Dev2Next - How good of a developer are you.pdf

Dev2Next pres about how to recognize what makes a developer a good developer

1. How good of a
developer are you?
And can we objectively determine this?

2. Bio
•Roy Wasse
•Bsc. Technical
Computer Science
•Msc. Philosophy of
Technology
•Co Founder OpenValue
•Dev & Management roles
•15+ years of hiring
experience

3. Is someone is a
‘good’ dev!
•Do you know YAGNI or HATEOS, SOLID?
•Explain balanced binary tree search algo?
•Use 20 best interview questions from
internet
•Apply Open Hiring?
•Proxy variables e.g.,CV, experience,
education, references, questionnaire
•Whiteboard coding
•Make (lengthy) assignment

4. Google Hiring process
•Assessments: You may be asked to do a brief online assessment, like a coding quiz, after
you’ve submitted your resume.
•Short virtual chats: Before diving into more in -depth interviews, you’ll typically have
one or two shorter conversations over phone or video. These will usually be with a
recruiter and then with either the hiring manager or peer on the team, and are designed
to assess key skills you’ll need for the role.
•Project work: We sometimes ask candidates to complete a small project prior to their in-
depth interviews. This could range from prepping a case study to providing writing or
code samples (don’t stress, they’re not that scary and we won’t spring this on you
without warning), and helps us understand how you think and approach problems. We’ll let
you know about any additional materials we’ll need early on.
•Interviews: We get excited about interviewing and take it seriously because, at the risk
of sounding cliché, Google is what Googlers make it. Our process can be rigorous
(typically 3-4 interviewsin one day, either over video or in person).

5. •Can science help us and
how?
•Let’s dive into
literature
•And see how we can
predict of someone will
be a good dev!
Who knows what
works?!!

6. Use of Aptitude tests
•Measure if someone is dispositionedwell to excel
•Popular in the 1970s (e.g., Alspaugh, 1972; Mazlack,
1980)
•due to lack of good results from these tests not used
much anymore(Bornat, Dehnadi, & Simon, 2008; Curtis,
1991; Mayer & Stalnaker, 1968)
•Also: in a sample of over 3,500students, no
incremental validity was found for specializedaptitude
tests for programmers over the more generalaptitude
tests used by the United States Air Force ( Besetsny,
Ree, & Earles, 1993)

7. Question
•Who thinks a computer science study is an
important predictor for the expert level of a
developer?

8. Role of
education
•Large meta study (Roth,
BeVier, Switzer, &
Schippmann, 1996)
reported a correlationof
r = 0.16
•Shortly after graduation,
correlation between
grades and job
performance much higher
•1 year, r = 0.23,
n = 1.288
•6 years, r = 0.05,
n = 866

9. Question
•Who thinks a senior developer should have at
least 5 years of experience?

10. Role of
experience
•Studies show positiveeffect
of having programming
experience (Agarwal, Sinha &
Tanniru, 1996; Arisholm&
Sjøberg, 2004)
•Other studies have reported
no effect of experience
beyond the first few years
(Jeffery & Lawrence, 1979) or
no effect of experience
(Jørgensen, 1995; Wohlin,
2002, 2004)

11. Question
•Who thinks LLMs will replace us within the next
few years for the better part of our job?

12. Role of LLMs
•Refactoring vs. Refuctering (Tornhill, Borg, Mones, 2024)
•More or less important?
•Understanding syntax
•Framework knowledge
•Debugging & tracing
•Higher order skills
•Expressing intent, problem framing
•Architecture & design
•Understanding non functionals (sec, perf, ..)
•Coaching & sharing
•Supporting juniors
•Aligning with org (other teams)

13. GenAI changes required skills?
•Refactoring vs. Refuctering (Tornhill, Borg, Mones,
2024)
•More or less important?
•Understanding syntax
•Framework knowledge
•Debugging & tracing
•Higher order skills
•Expressing intent, problem framing
•Architecture & design
•Understanding non functionals (sec, perf, ..)
•Coaching & sharing
•Supporting juniors
•Aligning with org (other teams)

14. So..
•Is genAI not just another skill in our toolbox
•GenAI saves a lot of time (test, scaffolding,
regex, approaches).. But…how much time does an
enterprise developer actually code
•So, does genAI fundmentally changes the
required skills?
•Perhaps it changes the way the craft is applied

15. Question
•Who thinks Java/.Net/... certification programs
are important for a developer?

16. Role of
Programming
knowledge
•Central predictor of skill is
knowledge (Chi et al., 1988)
•Used in Java certification
•But…software development is a
skill
Acquired by practice

17. What is skill?
•Ability (Ferguson, 1956):
“An ability is a trait
defined by what an
individual can do”
•“Skill is specialized type
of ability, one that
improves with practice and
is well adjusted, well
organized and goal
oriented” (Fitts & Posner,
1967)

18. Model of skill acquisition
•“A Five-Stage Model of the Mental Activities
Involved in Directed Skill Acquisition”,
(Dreyfus & Dreyfus, 1980)
•Flow is at level 5
•Also think of driving / playing chess

19. It’s all about skill?
•Skill cannot by equated with performance
•“the individual performed well because of his
high level of skill” is a generalization
(Messick, 1994)
•8 major components of job performance research
show 3 direct determinants of performance on
the job: knowledge, skilland motivation
(Campbell, McCloy, Oppler & Sager, 1993)

20. Skill is just one..?
Campbell’s work performance framework proposed eight work performance
dimensions:
1.job-specific task proficiency <=skill
2.non–job-specific task proficiency
3.written and oral communications
4.demonstrating effort
5.maintaining personal discipline
6.facilitating peer and team performance
7.supervision
8.management and administration.
According to Campbell, these eight dimensions are sufficient to describe
the latent structure of performance at a general level. However, he also
noted that the eight factors can have different patterns of subdimensions,
and their content and salience can vary across jobs.

21. Technical skill is the most importantone
(McGill, 2008)

22. Skill & project success
emphasizing low price when selecting software providers in bidding
rounds increases the likelihood project will fail.” (Jørgensen, 2011)
Tested by analyzing a dataset of 4,791,067 bids for 785,326 small-
scale projects
Clients emphasizing a low price selected providers with lower skill,
resulted in 9% increase in failure rate

23. What about
teamwork
•Teamperformanceis more
complex to understand
than individual
performance (see, e.g.,
Baker & Salas, 1992;
Volmer, 2006),
individualskillor
expertise is
nevertheless a central
component in team
performance (Land, Wong,
& Jeffery, 2003).

24. Importance of developers
•The claim of substantial programming performance
differenceson success is confirmedby additional
data, reviews & experiments (Curtis, 1981;
DeMarco & Lister, 1985, 1999; Trendowicz& Munch,
2009; Prechelt, 1999a)
•Variabilityin individual performance more
important than technology or methods used to
increase productivity
•In the book “Software Economics”, Boehm (1981)
reported that the “people factor” was the largest
of all investigated factors in the success of
software projects and therefore concluded that
developer attributes are by far the best
opportunity for improving software productivity.

25. Pair experts for extra speed
•The benefits of pair programming has been studied
from the perspective of forming the pair based on
different levels of expertise (Lui & Chan, 2006)
•…or seniority (Arisholm, Gallis, Dybå, & Sjøberg,
2007).
•A systematic literature review on pair programming
found that, among students, this practice was most
beneficial when the pairwas comprised of
individualswith a similar level of programming
skill (Salleh, Mendes, & Grundy, 2011).

26. So common sense wins?
•A software developer is someone who usually
holds “a bachelor’s degree in computer science
and strongcomputer programmingskills” (US
Bureau of LaborStatistics)
•So let’s hirebased on skill level!!

27. Deep dive
into skills
•And…how to objectively
measure coding skills
•Soft skills

29. Research overview
•Relations between effort estimates, skill indicators, and measured
programming skill. MagneJørgensen, Gunnar R. Bergersen and Knut
Liestøl(2021)
•Measuring Programming Skill: Construction and Validation of an
Instrument for Evaluating Java Developers. Gunnar R. Bergersen,
PhD thesis (2015)
•Construction and validation of an instrument for measuring
programming skill. Gunnar R. Bergersen, Dag I. K. Sjøberg, and
Tore Dybå(2014)
•Evaluating methods and technologies in software engineering with
respect to developers’ skill level. Gunnar R. Bergersen and Dag I.
K. Sjøberg(2012)
•Inferring skill from tests of programming performance: combining
time and quality. Gunnar R. Bergersen, Jo E. Hannay, Dag I. K.
Sjøberg, Tore Dybå, and AmelaKarahasanovic(2011)
•Programming skill, knowledge, and working memory among
professional developers from an investment theory perspective.
Gunnar R. Bergersen and Jan -Eric Gustafsson (2011)

30. How to measure skill?
•Home work assignment (better rotate those)
•Whiteboard coding (under pressure)
•Online questionaires (assuming correlation
between real world coding)
•All are subjective

31. Measure via self
assessment
•Ratings of self may work
better when people compare
themselvesrelative to
each other (people they
know) ratherthan when
providing absolute ratings
(Kruger and Dunning (1999)
Source:
wikipedia

32. Measuring programming skill
•Research started in 50’s with punch card
operators traits (McNamara & Hughes, 1955)
•60’s use of test in selection researched (Perry
& Cantley, 1965)
•Behavorial study of devs (Shneiderman, 1976)
•Theories concerning the comprehension of
computer programs (Brooks, 1983)

33. Task performance is key
•>20 studies about job performance assessed
•Almost all frameworks mentioned task
performance as an important dimension of
individual work performance.

34. Predict skill
•Study of novice LISP
programmers (Anderson &
Jeffries, 1985) found that
“the best predictor of
individual subject
differences in errors on
problems that involved one
LISP concept was the number
of errors on other problems
that involved different
concepts” (Anderson, 1987,
p. 203).

35. How to put this into a system
•Time (Bergersen, 2011)
•Quality (Ebert et. al. 2005)
•Process improvement (Dybå, 2000)
Createa measurementmodel & test themodelon
newdata
(Bergersenet al., 2014)

36. Rasch model
•More than 60 years ago, Rasch (1960) used tests
to study the reading performance of students.
•Rasch models have become a practical way to
measure psychological abilities

37. Task construction
1.Picked existing tasks from literature
2.Reanalyzed data sets to combine time & quality
3.Large control group with 255 developers from 9
countries used to score tasks

38. •Collection of tasks
•VM in the cloud (Linux + IntelliJ / Eclipse
•Interchangeable
Building a new standard to measure & analyze
software engineering skills _

39. Detailed analytics of skills and
meaningful comparisons with empirically validated norm groups

40. DEMO

43. OCA/OCP
•Makes you think like a compiler
•Checks for API knowledge
•But nothing about actual skill level
•Doesn’t measure skills like GrepS does
•So, what if we would consult the community for
a GrepS test variant that uses both

44. Foojay community
•Over 1500 members globally
•2 news articles every week day
•Podcast
•Conference information
•Tech questions
•And the certification initiative!

46. Foojay certified developer
•Multiple sessions with champions and 80+
community members
•Defined what should be tested and how
•Universal test
•Never do a test again!
•Or actually do test again to prove your progress!
•Current status = beta
•Want a free test, let me know (@roywasse)

47. Recap
•Coding is mostly about skills
•Research shows how to measure skills
•Let’s base Java certification this!
•Questions?