Heads-Up Multitasker: CHI 2024 Presentation.pdf

We developped a computationally rational model to model human's multitasking behaviors, especially in the context of reading on smart glasses while walking. Trying to predict a key aspect: attention distribution across digital content on smart glasses vs. items in the environment.

1. Heads-Up Multitasker: Simulating Attention
Switching On Optical Head-Mounted Displays
Bai Yunpeng, Aleksi Ikkala, Antti Oulasvirta*, Shengdong Zhao*, Lucia J. Wang, Pengzhi Yang, Peisen Xu

2. Background: Heads-Up Computing Paradigm
Convenient and
ubiquitous reading
Daily task
planning
Cooking
Exercising
Move and read on smart glasses
How to design optimal interfaces?

3. Background: Modeling Objectives
●Attention switches In the heart of an ancient forest, where
the sunlight gently filtered through a high
canopy, a small, forgotten pond mirrored
the sky so precisely that it
Reading on smart glasses
while walking
Walk control
●Walk speed
●Walk direction
We predict:
Vision attention control:
●Attention to the environment
●Text reading on smart glasses

4. Application: An Evaluation Tool
Which text spacing
is the best?
5 different
text spacings
Run our simulator to see!
Best Text
Spacing:
Layout 4
1
2
3
4
5

5. Application: An Evaluation Tool
Simple road read and walkCity read and walk Read, walk, see signs
Flexible to differnt design factors, users, scenarios, and tasks!
Handcrafted policies
Flexibly learn policies by
reinforcement learning

6. o, r
a
r
SC
r
SC
r
R
r
S







External state
(MuJoCo Simulation)
In the heart of an
ancient forest,
where the
sunlight gently
Our Model
FPV TPV
Read (R)
Scan (S)
Supervisory
Controller
(SC)
b
Locomotion
Control
(LC)
e

w

Oculomotor
Control
(OC)
Internal
state




a
o
Memory
Hierarchical
Reinforcement
Learning

7. External state
Oculomotor
Control
(OC)
Locomotion
Control
(LC)
o, r
a
Internal
state




Memory
a
o
Our Model
Supervisory
Controller
(SC)
Read (R)
Scan (S)
r
SC
r
SC
b
w

r
R
e

r
S
Assumptions:
●Decompose complex cognitive task to simpler subtasks in terms of
solvability and trainability.
●High-fidelity simulation of primitive actions – eye movement and walking.

8. Model – Supervisory Level – Supervisory Control (SC)
●State
○Reading progress.
○Walking status.
○Current task.
○Environmental situation.
●Action
○Attention deployment.
●Observation
○Time awareness.
○Current task.
○Reading progress.
○Walking speed.
●Reward Function
○r
t
= w
R
* r
R
+ w
R
* r
R
- w
A
* c
A
- c
T
●Transition
○Deterministic
Task Description:
When to switch attention?

Learning Objectives:
Minimize the environmental information loss
while maximizing the reading progress.
Github code

9. Model – Task Level – Read (R)
●State
○Target word should be read.
○Current fixation.
●Action
○The word index.
●Observation
○Time awareness.
○Current task.
○Belief: Probability representation on words.
●Reward Function
○r
t
= r
time cost
(-0.1) + Bonus (+-10)
●Transition
○Deterministic

Task Description:
Resume reading.

Learning Objectives:
Relocate the correct word quickly.
Github code

10. Model – Motor Level – Oculomotor Control (OC)
●State
○Target word should be fixated.
○Current fixation.
●Action
○The eyeball motor control: x and y rotations.
●Observation
○Time awareness.
○Vision perception (image captured in the simulator)
○Proprioception: Current fixation.
●Reward Function
○r
t
= 0.1 * (e
-10*d
- 1) + Bonus (+10)
●Transition
○Stochastic:
a
t+1
~ N (target fixation position, 0.08 * saccade amplitude)
Task Description:
Fixate on the target word.

Learning Objective:
Fixate on the target object ASAP under
the noisy perception and control
conditions.
Github code

11. Study Overview
Study 1: Attention switches.
Simulation data

Tasks
1
Study 2: Reading while walking.
Simulation data vs. Human data
2
Study 3: Resume reading after attention switches.
Simulation data vs. Human data
3
Study 4: Read, walk, and see env signs.
Simulation data vs. Human data
4

12. Study 1: Attention Switches Adapt to Agents and Walking Speed
Reward = w
read
⨉ r
1
+ w
walk
⨉ r
2
+ w
attention switch
⨉ r
3
●Higher w
read
, the agent priorities read more.
●Higher w
walk
, the agent priorities walk safety more.
We could train different agents by designing different reward functions.
Olaf
Env
events
Texts
Shakespeare Norman
Read walk attention swithces

13. Study 1: Attention Switches Adapt to Agents and Walking Speed
Other metrics:
●Reading interruption positions
●Reading speed
●Walking error rate
Number of attention switch

14. Study 2: Reading Speed Adapts to Walking
Simulation
Reading deterioration due
to head perturbations
Reading speed ratio = RS
walk
/ RS
stand

Results
(N=12)
HumanSim
Real world

15. Study 3: Reading Resumptions Adapt to Text Layouts
SimulationReal world
In the quiet town of
Willow Creek,
whispers of a
mysterious figure
Simulate the reading resumption adaptation to different text layouts.

16. Study 3: Reading Resumptions Adapt To Text Layouts
Layout 1
Layout 2
Layout 3
Completion Time (s)
N=12
Error Rate (%)
N=12

17. Study 3: Reading Resumptions Adapt To Text Layouts
Internal
state




Memory
Normal model
Internal
state




Ablation model:
No Memory ModuleVS.
Layout 1
Layout 2
Layout 3
Completion Time (s)
Ablation: huge discrepency

18. Study 4: Read, Walk, and See Env Signs
Task: Read, walk, see signs
Experiment setup: real world vs. simulation.
Route: a rectangle path

19. Study 4: Read, Walk, and See Env Signs

Percent
Simulation Human
N=12

20. Conclusion and Takeaways
A flexible simulator Hierarchical
POMDPs
Evaluate Extend Oculomotor Locomotion
●What is the task to optimize?
●How to describe user behaviors into
sequential decision-making processes?
●What cognitive processes to include?

21. Thank You!
Github codes and data Know more about me;
Contact me if you are looking for jobs
In the heart of an
ancient forest,
where the sunlight
gently filtered
through a