HCI Research Methods - Lecture 7 - Human-Computer Interaction (1023841ANR)

This lecture forms part of a Human-Computer Interaction course given at the Vrije Universiteit Brussel.

1. 2 December 2005
Human-Computer Interaction
HCI Research Methods
Prof. Beat Signer
Department of Computer Science
Vrije Universiteit Brussel
beatsigner.com

2. Beat Signer -Department of Computer Science [email protected] 2November 10, 2023
Human-Computer Interaction
▪Human-Computer Interaction is a multidisciplinary field
▪Computer Science
▪Design
▪Cognitive Science
▪Psychology
▪…
Human-computer interaction is a discipline concerned
with the design, evaluation and implementation of
interactive computing systems for human use and with
the study of major phenomena surrounding them.
ACM SIGCHI Curricula for Human-Computer Interaction

3. Beat Signer -Department of Computer Science [email protected] 3November 10, 2023
HCI Research Contributions
▪Empirical contributions
▪collected quantitative or qualitative data
▪Artefact contributions (systems research)
▪development and evaluation of new artefacts
▪interfaces, toolkits, mock-ups, …
▪Methodological contributions
▪new method or modification of existing method, new metrics
▪Theoretical contributions
▪concepts and models as vehicles for thought
▪frameworks, design spaces, conceptual models
▪Dataset contributions
▪corpus for the benefit of the research community

4. Beat Signer -Department of Computer Science [email protected] 4November 10, 2023
HCI Research Contributions …
▪Literature survey contributions
▪review and synthesis of work done in a specific area
▪Opinion contributions
▪trying to persuade the readers to change their minds
▪HCI research as well as HCI research methods have
changed over time
▪web interfaces, user-generated content, touch screens,
collaboration, …
▪usability engineering, eye tracking, crowdsourcing, …

5. Beat Signer -Department of Computer Science [email protected] 5November 10, 2023
HCI Research
▪Most HCI researchers must collect their own data
▪in other domains (e.g. sociology or economics) data often
collected by large entities or government agencies
▪Studies with many participants (big data) can help us
determine correlations
▪Smaller studies might provide us with a deeper
understanding of the meaning of data
▪Longitudinal studies in HCI are rare
▪technology and tools change rapidly → often does not make
sense to compare over multiple years or decades
▪often seen as a shortcoming

6. Beat Signer -Department of Computer Science [email protected] 6November 10, 2023
Measurement
▪What to measure?
▪In early days of HCI (early 1980s) mainly human perfor-
mance for individual tasks (micro-HCI) measured in labs
▪task correctness
▪time performance
▪error rate
▪time to learn
▪user satisfaction
▪Broader level (macro-HCI) such as motivation, collabo-
ration or trust not easy to measure with existing metrics
▪use of multimethod approaches
-case studies, observations, interviews, data logging, …

7. Beat Signer -Department of Computer Science [email protected] 7November 10, 2023
Different Types of Research
Type of ResearchFocus General Claims Typical Methods
Descriptive Describe a
situation or a set
of events
X is happeningObservations,
field studies,
focus groups,
interviews
Relational Identify relations
between multiple
variables
(correlation)
X is related to YObservations,
field studies
surveys
Experimental Identify causes of
a situation or a
set of event
(causality)
X is responsible
for Y
Controlled
experiments

8. Beat Signer -Department of Computer Science [email protected] 8November 10, 2023
Experimental Research
▪Null hypothesis and alternative hypothesis (see lecture 6)
▪Independent variables
▪technology, e.g. typing vs. voice input, mouse vs. joystick, …
▪different types of design, e.g. pull-down vs. pop-up menu, font
size, colours, …
▪user related, e.g. age, gender, culture, computer experience, …
▪context, e.g. noise, lighting, temperature, …
Data
Hypothesis Study

9. Beat Signer -Department of Computer Science [email protected] 9November 10, 2023
Experimental Research
▪Dependent variables
▪efficiency
-time to complete task, speed (e.g. words per minute)
▪accuracy
-error rate
▪subjective satisfaction
-normally collected via Likert scale ratings (e.g. via questionnaires)
▪ease of learning or memorability and retention rate
-important for adoption of information technology
▪physical or cognitive demand
-how long can we interact without significant fatigue

10. Beat Signer -Department of Computer Science [email protected] 10November 10, 2023
Experimental Procedure
1.Identify a research hypothesis
▪how many independent variables?
▪how many values for each independent variable?
2.Specify experimental design (see lecture 6)
▪between-subjects design, within-subjects design
and pair-wise design
3.Run a pilot study
▪test design and study instruments
4.Recruit participants
5.Run the data collection sessions
6.Analyse the data (statistical analysis)
7.Report the results

11. Beat Signer -Department of Computer Science [email protected] 11November 10, 2023
Statistical Analysis
▪Preprocessing of data
▪cleaning up data and fixing errors if possible
-in anonymous studies we might not be able to contact participants for
clarification
-if error cannot be fixed, we have to remove the data item and treat it as
missing value in the analysis
▪coding data
-e.g. gender: “male” → 1 and “female” → 0
-e.g. degree: “high school” → 1, “university college” → 2 and “university” → 3
▪organising data
-bring data in layout / format for specific data processing software
(e.g. SPSS, available via the VUB software shop)

12. Beat Signer -Department of Computer Science [email protected] 12November 10, 2023
Descriptive Statistical Tests
▪Distribution of data points,
▪means, medians, variances, standard deviations and ranges
▪Comparing means
▪cannot just compare the means but also have to compute some
statistical significance tests (e.g. t tests or ANOVA)
▪Example
▪independent-samples t test for data on the next slide results in
a t value of 2.169
-higher than the t value (2.131) for the specific degree of freedom (df =15) at
the 95% confidence interval
-“An independent-samples t test suggests that there is significant difference in
the task completion time between the group who used the standard word-
processing software and the group who used word-processing software with
word prediction functions (t(15) = 2.169, p < 0.05).”

13. Beat Signer -Department of Computer Science [email protected] 13November 10, 2023
Data for Independent-Samples t Test

14. Beat Signer -Department of Computer Science [email protected] 14November 10, 2023
Data for Paired-Samples t Test

15. Beat Signer -Department of Computer Science [email protected] 15November 10, 2023
Identifying Relationships
▪Identify whether there is a relationship (correlation)
between various variables
▪compute Pearson’s product moment correlation coefficient
-SPSS will compute a correlation matrix between all variables
▪Pearson’s r value ranges from -1 to 1
- r =-1.0 means that there is a perfect negative linear relationship between two
variables → specific increase in one variable perfectly predicts decrease in the
other variable
-r =1.0 means that there is a perfect positive linear relationship between two
variable → specific increase in one variable perfectly predicts increase in the
other variable
-r =0 means that there is no linear relationship between the two variables
▪Note that correlation does not imply a causal relationship
▪might also be based on hidden (“intervening”) variable

16. Beat Signer -Department of Computer Science [email protected] 16November 10, 2023
Limitations of Experimental Research
▪Requires well-defined testable hypotheses with a limited
number of independent and dependant variables
▪many problems not clearly defined or involve a large number of
potential variables
▪Need strict control of factors that might influence the
dependent variables
▪not always possible to control these factors
▪Lab experiments might not be a good representation of a
user’s typical interaction behaviour
▪study participants might behave differently in lab-based
experiments
-stress of being observed, different environment, …

17. Beat Signer -Department of Computer Science [email protected] 17November 10, 2023
Questionnaires / Surveys
▪Closed as well as open questions (see lecture 3)
▪Pilot test to ensure validity and reliability
▪ensure that questions are clear, unambiguous and unbiased
▪Need higher response rates than surveys in interaction
design to get statistically relevant results
▪Data analysis
▪clean the data
▪statistical analysis of closed questions
-often just descriptive statistics (percentages etc.)
-inferential statistics by understanding the relationships between variables
▪Might be combined with other research methods
▪e.g. interviews, focus groups or diaries

18. Beat Signer -Department of Computer Science [email protected] 18November 10, 2023
Diaries
▪Individual maintaining regular recordings
▪fills the gap between observations in naturalistic settings
and lab studies
▪Advantages
▪good for understanding the “why” of user interaction with
technology
▪useful for technology that is used on the go
-difficult to do in lab setting or via observation
▪Disadvantages
▪participants might not record a sufficient number of entries
▪data analysis (mix of qualitative and quantitative) might take a
long time

19. Beat Signer -Department of Computer Science [email protected] 19November 10, 2023
Interviews
▪Open-ended (unstructured), semi-structured or
structured interviews (see lecture 3)
▪Content analysis
▪usually relies on qualitative methods for coding data
-try to find common structures and themes from qualitative data
-frequency of terms etc. (e.g. use MAXQDA)
▪if validity is a particular concern, then multiple researchers should
independently analyse the interviews
-individual analysts might have some bias
▪Might best be conducted as complements to other data
collection approaches

20. Beat Signer -Department of Computer Science [email protected] 20November 10, 2023
Grounded Theory
▪For news topic with limited literature to build on
▪no established theories to develop coding categories
▪Use emergent coding approach based on the notion of
grounded theory
▪qualitative research method that seeks to develop theory that is
“grounded in data systematically gathered and analysed”
▪might have multiple rounds of data collection and analysis to
allow the underlying theory to fully emerge
Data TheoryStudy

21. Beat Signer -Department of Computer Science [email protected] 21November 10, 2023
Ethnography
▪Combination of observation, interviews and participation
▪has its roots in anthropological studies
▪Examples
▪home settings
-country, culture and religion have a great impact on how technology is used in
homes
▪work settings
-e.g. London underground control centre (video ethnography)
-non-office based settings (e.g. vineyards and use of sensors)
▪educational settings
-use of technologies in learning activities
▪Virtual ethnography
▪use of web cams or videos

22. Beat Signer -Department of Computer Science [email protected] 22November 10, 2023
Ethnography …
▪Very useful in understanding the context of technology
usage
▪Often used as a first step to understand a group of users,
their problems, challenges, norms and processes
▪eventual goal of building some type of technology for them
or with them
▪More recently also ethnographic investigations of
ubiquitous computing environments
▪e.g. navigation needs of firefighters to find their way out of
hazardous, smoke-filled environments

23. Beat Signer -Department of Computer Science [email protected] 23November 10, 2023
Automated Data Collection Methods
▪Website access log analysers
▪Activity logging software
▪Custom software with reporting features

24. Beat Signer -Department of Computer Science [email protected] 24November 10, 2023
Measuring the Human
▪Eye tracking
▪Muscular and skeletal position sensing
▪Wii remote
▪smartwatches
▪Microsoft Kinect
▪Motion tracking
▪e.g. for interaction with large wall-sized displays
▪Physiological data
▪heart rate and blood volume / pressure
▪galvanic skin response
▪respiration
▪brain activity (EEG)

25. Beat Signer -Department of Computer Science [email protected] 25November 10, 2023
Online HCI Research
▪Observational online studies
▪usability and think-aloud studies
▪remote screen sharing
▪web cam live feed (audio/video)
▪remote keyboard/mouse control
▪potential recording of different streams
▪A / B testing for Internet business

26. Beat Signer -Department of Computer Science [email protected] 26November 10, 2023
Human Computation
▪Humans are often better in tasks requiring
detailed interpretation of complex input
▪Computers can be used to augment humans
(see NLS project in lecture 1), but humans can also be
used to augment computers → human computation
▪Task that is hard for a computer but easy for humans
▪ask multiple humans to complete small pieces of the task
▪e.g. reCAPTCHA

27. Beat Signer -Department of Computer Science [email protected] 27November 10, 2023
Conducting Human Computation Studies
▪Use crowdsourcing ser-
vices for large pool of
inexpensive study partici-
pants
▪e.g. Amazon’s Mechanical
Turk with specific APIs
▪potentially less bias and
increased validity
-participants do not directly
interact with researchers

28. Beat Signer -Department of Computer Science [email protected] 28November 10, 2023
Main HCI Conferences
▪CHI: ACM Conference on Human Factors in
Computing Systems
▪https://dl.acm.org/conference/chi/
▪UIST: ACM Symposium on User Interface Software
and Technology
▪https://dl.acm.org/conference/uist/
▪CSCW: ACM Conference on Computer-supported
Cooperative Work and Social Computing
▪https://dl.acm.org/conference/cscw/
▪IUI: Annual Conference on Intelligent User Interfaces
▪https://dl.acm.org/conference/iui/

29. Beat Signer -Department of Computer Science [email protected] 29November 10, 2023
Main HCI Conferences …
▪TEI: International Conference on Tangible,
Embedded, and Embodied Interaction
▪https://dl.acm.org/conference/tei/
▪EICS: International Conference on Engineering
Interactive Computing Systems
▪https://dl.acm.org/conference/eics/
▪DIS: ACM Conference on Designing Interactive Systems
▪https://dl.acm.org/conference/dis/
▪ICMI: International Conference on Multimodal Interaction
▪https://dl.acm.org/conference/icmi/

30. Beat Signer -Department of Computer Science [email protected] 30November 10, 2023
Exercise 7
▪Evaluation (Usability / User Experience)

31. Beat Signer -Department of Computer Science [email protected] 31November 10, 2023
Further Reading
▪Major parts of this lecture are based on
the book Research Methods in
Human-Computer Interaction
▪chapter 1–14

32. Beat Signer -Department of Computer Science [email protected] 32November 10, 2023
References
▪ Research Methods in Human-Computer
Interaction, Jonathan Lazar, Jinjuan Heidi Feng and
Harry Hochheiser, Morgan Kaufmann (2nd edition),
May 2019, ISBN-13: 978-0128053904
▪ Technology in Action (Learning in Doing: Social,
Cognitive and Computational Perspectives),
Christian Heath and Paul Luff, Cambridge Univer-
sity Press, June 2000, ISBN-13: 978-0521568692
▪ ▪SPSS (Statistical Package for the Social Sciences)
▪https://www.ibm.com/spss
▪available via VUB via Software Webshop

33. 2 December 2005
Next Lecture
Use Cases and Course Review