Definitive answer to developer productivity v1 .2 Devworld.pptx

As given on DevWorld

1. Definitive answer to developer productvity Or..How good of a developer are you?

2. Bio Roy Wasse Bsc. Technical Computer Science Msc. Philosophy of Technology Co Founder OpenValue Dev & Management roles 15+ years of h iring experience

3. Is someone is a ‘good’ dev! Do you know YAGNI or HATEOS, SOLID? Explain balanced binary tree search algo? Use 20 best interview questions from internet Apply Open Hiring? Proxy variables e . g. , CV, experience, education, references, questionnaire Whiteboard coding Make (lengthy) assignment

4. Google Hiring process Assessments : You may be asked to do a brief online assessment , like a coding quiz, after you’ve submitted your resume. Short virtual chats : Before diving into more in-depth interviews, you’ll typically have one or two shorter conversations over phone or video. These will usually be with a recruiter and then with either the hiring manager or peer on the team, and are designed to assess key skills you’ll need for the role. Project work : We sometimes ask candidates to complete a small project prior to their in-depth interviews. This could range from prepping a case study to providing writing or code samples (don’t stress, they’re not that scary and we won’t spring this on you without warning), and helps us understand how you think and approach problems. We’ll let you know about any additional materials we’ll need early on. Interviews : We get excited about interviewing and take it seriously because, at the risk of sounding cliché, Google is what Googlers make it. Our process can be rigorous ( typically 3-4 interviews in one day, either over video or in person).

5. Can science help us and how? Let’s dive into literature And see how we can predict of someone will be a good dev! Who knows what works?!!

6. Use of Aptitude tests Measure if someone is dispositioned well to excel Popular in the 1970s (e.g., Alspaugh , 1972; Mazlack , 1980) due to lack of good results from these tests not used much anymore ( Bornat , Dehnadi , & Simon, 2008; Curtis, 1991; Mayer & Stalnaker, 1968) Also: in a sample of over 3,500 students, no incremental validity was found for specialized aptitude tests for programmers over the more general aptitude tests used by the United States Air Force ( Besetsny , Ree , & Earles , 1993)

7. Question Who thinks a computer science study is an important predictor for the expert level of a developer?

8. Role of education Large meta study (Roth, BeVier , Switzer, & Schippmann , 1996) reported a correlation of r = 0.16 Shortly after graduation, correlation between grades and job performance much higher 1 year, r = 0.23, n = 1.288 6 years, r = 0.05, n = 866

9. Question Who thinks a senior developer should have at least 5 years of experience?

10. Role of experience Studies show positive effect of having programming experience (Agarwal, Sinha & Tanniru , 1996; Arisholm & Sjøberg , 2004) Other studies have reported no effect of experience beyond the first few years (Jeffery & Lawrence, 1979) or no effect of experience ( Jørgensen , 1995; Wohlin , 2002, 2004)

11. Question Who thinks LLMs will replace us within the next few years for the better part of our job?

12. Role of LLMs Refactoring vs. Refuctering (Tornhill, Borg, Mones, 2024) Ideal for who? Generation works pretty well, but what about code deletion?

13. Question Who thinks Java/.Net/... certification programs are important for a developer?

14. Role of Programming knowledge Central predictor of skill is knowledge (Chi et al., 1988) Used in Java certification But…software development is a skill Acquired by practice

15. What is skill ? Ability (Ferguson, 1956) : “An ability is a trait defined by what an individual can do” “Skill is specialized type of ability, one that improves with practice and is well adjusted, well organized and goal oriented” (Fitts & Posner, 1967)

16. Model of skill acquisition “A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition”, (Dreyfus & Dreyfus , 1980) Flow is at level 5 Also think of driving / playing chess

17. It’s all about skill? Skill cannot by equated with performance “the individual performed well because of his high level of skill” is a generalization (Messick, 1994) 8 major components of job performance research show 3 direct determinants of performance on the job: knowledge , skill and motivation (Campbell, McCloy, Oppler & Sager, 1993)

18. Skill is just one..? Campbell’s work performance framework proposed eight work performance dimensions: job-specific task proficiency <= skill non–job-specific task proficiency written and oral communications demonstrating effort maintaining personal discipline facilitating peer and team performance supervision management and administration. According to Campbell, these eight dimensions are sufficient to describe the latent structure of performance at a general level. However, he also noted that the eight factors can have different patterns of subdimensions, and their content and salience can vary across jobs.

19. Technical skill is the most important one (McGill, 2008)

20. Skill & project success

21. What about teamwork Team performance is more complex to understand than individual performance (see, e.g., Baker & Salas, 1992; Volmer, 2006), individual skill or expertise is nevertheless a central component in team performance (Land, Wong, & Jeffery, 2003).

22. Importance of developers The claim of substantial programming performance differences on success is confirmed by additional data, reviews & experiments (Curtis, 1981; DeMarco & Lister, 1985, 1999; Trendowicz & Munch, 2009; Prechelt , 1999a) Variability in individual performance more important than technology or methods used to increase productivity In the book “Software Economics”, Boehm (1981) reported that the “ people factor ” was the largest of all investigated factors in the success of software projects and therefore concluded that developer attributes are by far the best opportunity for improving software productivity.

23. Pair experts for extra speed The benefits of pair programming has been studied from the perspective of forming the pair based on different levels of expertise (Lui & Chan, 2006) …or seniority ( Arisholm , Gallis , Dybå , & Sjøberg , 2007). A systematic literature review on pair programming found that, among students, this practice was most beneficial when the pair was comprised of individuals with a similar level of programming skill (Salleh, Mendes, & Grundy, 2011).

24. So common sense wins? A software developer is someone who usually holds “a bachelor’s degree in computer science and strong computer programming skills ” (US Bureau of Labor Statistics) So let’s hire based on skill level!!

25. Deep dive into skills And…how to objectively measure coding skills Soft skills

26. Research overview Relations between effort estimates, skill indicators, and measured programming skill. Magne Jørgensen , Gunnar R. Bergersen and Knut Liestøl (2021) Measuring Programming Skill: Construction and Validation of an Instrument for Evaluating Java Developers. Gunnar R. Bergersen, PhD thesis (2015) Construction and validation of an instrument for measuring programming skill. Gunnar R. Bergersen, Dag I. K. Sjøberg , and Tore Dybå (2014) Evaluating methods and technologies in software engineering with respect to developers’ skill level. Gunnar R. Bergersen and Dag I. K. Sjøberg (2012) Inferring skill from tests of programming performance: combining time and quality. Gunnar R. Bergersen, Jo E. Hannay , Dag I. K. Sjøberg , Tore Dybå , and Amela Karahasanovic (2011) Programming skill, knowledge, and working memory among professional developers from an investment theory perspective. Gunnar R. Bergersen and Jan-Eric Gustafsson (2011)

27. How to measure skill? Home work assignment (better rotate those) Whiteboard coding (under pressure) Online questionaires (assuming correlation between real world coding) All are subjective

28. Measure via self assessment Ratings of self may work better when people compare themselves relative to each other (people they know) rather than when providing absolute ratings (Kruger and Dunning (1999) Source: wikipedia

29. Measuring programming skill Research started in 50’s with punch card operators traits (McNamara & Hughes, 1955) 60’s use of test in selection researched (Perry & Cantley, 1965) Behavorial study of devs (Shneiderman, 1976) Theories concerning the comprehension of computer programs (Brooks, 1983)

30. Task performance is key >20 studies about job performance assessed Almost all frameworks mentioned task performance as an important dimension of individual work performance.

31. Predict skill Study of novice LISP programmers (Anderson & Jeffries, 1985) found that “ the best predictor of individual subject differences in errors on problems that involved one LISP concept was the number of errors on other problems that involved different concepts ” (Anderson, 1987, p. 203).

32. How to put this into a system Time (Bergersen, 2011) Quality (Ebert et. a l. 2005) Process improvement ( Dybå , 2000) Create a measurement model & test the model on new data (Bergers e n et al. , 2014)

33. Rasch model More than 60 years ago, Rasch (1960) used tests to study the reading performance of students. Rasch models have become a practical way to measure psychological abilities

34. Task construction Picked existing tasks from literature Reanalyzed data sets to combine time & quality Large control group with 255 developers from 9 countries used to score tasks

35. Collection of tasks VM in the cloud (Linux + IntelliJ / Eclipse Interchangeable Building a new standard to measure & analyze software engineering skills _

36. Detailed analytics of skills and meaningful comparisons with empirically validated norm groups

37. DEMO

38. OCA/OCP Makes you think like a compiler Checks for API knowledge But nothing about actual skill level Doesn’t measure skills like GrepS does So, what if we would consult the community for a GrepS test variant that uses both

39. Foojay community Over 1500 members globally 2 news articles every week day Podcast Conference information Tech questions And the certification initiative!

40. Foojay certified developer Multiple sessions with champions and 80+ community members Defined what should be tested and how Universal test Never do a test again! Or actually do test again to prove your progress! Current status = beta Want a free test, let me know (@roywasse)

41. Recap Coding is mostly about skills Research shows how to measure skills Let’s base Java certification this! Questions?

42. What about certification

43. Skill level & Estimates Jørgensen et. al, 2021

44. How does GrepS compare? _ USP’s: reusable + customizable