Engineering Touchscreen Input for 3-Way Displays: Taxonomy, Datasets, and Classification

Paper presented at EICS 2024: https://dl.acm.org/doi/10.1145/3660515.3661331
In the family of personal multi-display devices and environments, 3-way displays conveniently integrate into the conventional form factors of laptops and tablets, featuring both a central display area and two symmetrically expandable lateral sides. However, despite a large body of knowledge on touch input for single-display devices, little is known about users’ gesture preferences for 3-way displays. We propose a cross-display gesture taxonomy for future explorations of gesture input for multi-display devices, in which we position 3-way displays. Using a requirement elicitation, we report results from two gesture elicitation studies with a total of 48 participants, where a 3-way display was used as a remote control panel for a smart home environment (study #1) and a touchscreen interface for content manipulation performed both within and across displays (study #2). Based on these findings, we offer two consensus datasets of 3-way-display gestures that are consolidated into a larger classification of stroke-gesture input for 3-way displays.

1. Engineering Touchscreen Input for 3-Way Displays: Taxonomy, Datasets, and Classification Jean Vanderdonckt, Radu-Daniel Vatavu, Arthur Sluÿters

2. Engineering Touchscreen Input for 3-Way Displays: Taxonomy, Datasets, and Classification Jean Vanderdonckt LouRIM Université catholique de Louvain, Belgium Radu-Daniel Vatavu Machine Intelligence & Information Visualization Lab Stefan cel Mare University of Suceava , Romania Arthur Sluÿters LouRIM Université catholique de Louvain, Belgium

3. Introduction and Motivation The 3-way screen used in our study

4. 5 Cross-device taxonomy: for GUIs only Source: https://dl.acm.org/doi/10.1145/3290605.3300792

5. 6 Revisiting this taxonomy for gesture UIs Cross-device | Multi-device | Distributed Cross-display | Multi-display Multi-mobile 2-way MDEs Multi-monitor/screen Multi-slate/tablet Multi-display environment Cross-surface | Multi-surface | Trans-surface a b c d e Flipping gestures [45] Pull gestures [22] Across screens [22] Between screens [22] 3-way MDEs This paper Spatially-aware gestures [5,23] Spatially-agnostic gestures [23] Shortcut gestures [21] Motion gestures [26] Surface gestures [18,43] Spherical Gestures [33] Data-sharing gestures [6], Synchronous gestures [9,24], Connection gestures [12,14], Data transfer gestures [13], Auxiliary display [22], Cross-device Drag & Drop [31] Multi-touch [7], Swipe-hand open [22], Pick & Drop [25], Prevalent gestures [28], Multi-surface gestures [29] within across level of distribution Stitching gestures [10] Dexterous gestures [46] Object movement gestures [17], Cooperative gestures [17]

6. Second gesture elicitation study

7. 8 Revisiting this taxonomy for gesture UIs Cross-device | Multi-device | Distributed Cross-display | Multi-display Multi-mobile 2-way MDEs Multi-monitor/screen Multi-slate/tablet Multi-display environment Cross-surface | Multi-surface | Trans-surface a b c d e Flipping gestures [45] Pull gestures [22] Across screens [22] Between screens [22] 3-way MDEs This paper Spatially-aware gestures [5,23] Spatially-agnostic gestures [23] Shortcut gestures [21] Motion gestures [26] Surface gestures [18,43] Spherical Gestures [33] Data-sharing gestures [6], Synchronous gestures [9,24], Connection gestures [12,14], Data transfer gestures [13], Auxiliary display [22], Cross-device Drag & Drop [31] Multi-touch [7], Swipe-hand open [22], Pick & Drop [25], Prevalent gestures [28], Multi-surface gestures [29] within across level of distribution Stitching gestures [10] Dexterous gestures [46] Object movement gestures [17], Cooperative gestures [17]

8. 9 Resulting classification of gestures Single-tap gestures Multi-tap gestures Vertical swipes Horizontal swipes Geometrical shapes Symbolic shapes Stichting gestures Edge-center gestures Pinch-in center, pinch-out lateral In-out lateral gestures

9. 10 Conclusion Cross-device taxonomy for stroke-gesture interaction 2 Gesture Elicitation Studies Taxonomy: 20 classes of stroke gestures Most important: lateral in/out, stichting gestures Limitations No mid-air gestures No study of fatigue effect on gesture production Gesture vocabulary depends on the application ( here , only IoT and MDE)

10. Thank you very much for your attention