IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction in Immersive Environments

IVE 2024 short course on the Psychology of XR, Lecture13 on Neurotechnology for Enhanced Interaction in Immersive Environments.

This lecture was given by Hakim Si-Mohammed on July 17th 2024 at the University of South Australia.

1. Neurotechnology for Enhanced Interaction in
Immersive Environments
Hakim Si-Mohammed
Associate Professor
Univ. Lille, CRIStAL
Brain-Computer Interfaces team
Contact:[email protected]
Universityof South Australia2024

2. A few wordsaboutmyself
2
Computer Science Engineer
HigherNational Schoolof Computer Engineering
Supervision: Pr. Karima Benatchba, Pr. Yacine Challal
International mobility(Academic)
Graz Universityof Technology(3 months)
Supervision : Pr. Reinhold Scherer
Internationmobility(Industrial)
Microsoft Research (3 months)
Supervision : Dr. Andrew Wilson
PhD in Computer Science (Defended12/2019)
Inria/INSA Rennes
Supervision: Dr. Anatole Lécuyer, Dr. Ferran Argelaguet,
Pr. Géry Casiez
Associate Professor
Univ. Lille
June 2016
September2018
June 2019
October2016
September2020

3. The Brain-Computer Interfaces team
3

4. Agenda
4

5. ResearchTopic
5
Brain-Computer Interfaces (BCIs) Immersive Environments(AR/VR)

6. Brain-Computer Interfaces
Part I:

7. What isNOT a BCI
7

8. Let’splaya smallgame
8

9. Definitions
9
Interface: noun[C] a situation, method, or place
wheretwothingscome togetherand have an
effecton eachother. [Cambridge dictionary]
A BCIisa system thattranslates measuresof brain
activityintocommandsor messages for an
interactive application
J W p w EW W p w “Brain-Computer Interfaces: principles and practice” Ox U v s y P ss 2012
F. Lotte, L. Bougrain, M. Clerc, "Electroencephalography (EEG)-based Brain-Computer Interfaces",
Wiley Encyclopedia on Electrical and Electronics Engineering, 2015

10. How doesa BCI actuallywork
10
M. Clerc, L. Bougrain F. “Brain-Computer Interfaces 1: Foundations and
Methods", ISTE-Wiley, 2016

11. Historyof brainactivitymeasurement
11

12. Types of brainactivitymeasurementtechniques
12

13. Some widelyused brainactivitymeasurementtechniques
13

14. Electroencephalography
14

15. The notion of BCI paradigm
15
1: Modulation
2: Feedback
2: Modulation
3: Feedback
1: Stimulation
1: Monitoring
2: Feedback

16. Active BCI paradigms
16
1: Modulation
2: Feedback

17. Slow Cortical Potentials
17
1: Modulation
2: Feedback

18. MotorImagery
18
Penfield homonculus [Penfield54]
µ (~8-12 Hz) oscillations
β (~12-30 Hz) oscillations
Pfurtscheller & Neuper “M y b - p ”
Proceedings of the IEEE, 2001
1: Modulation
2: Feedback

19. Motorimageryapplications
19

20. ReactiveBCI paradigms
20

21. Steady-State Visual EvokedPotentials
21
Legeny
et al. 2013
Zhu et al. 2010

22. P300: aka Aha! Signal
22
[Stephanieet al. 2017]
P300S peller
Farwell & Donchin “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain
potentials” E p p y N p ys y 1988

23. Passive BCI paradigms
23
A passive BCI is a system estimating one or several mental states from
the user – without any voluntary action from this user – to adapt a
human-computer interaction accordingly
Zander, T. & Kothe . “T w s p ss v b -computer interfaces: applying brain-computer interface technology
to human- sys s ” J N E 2011

24. Error RelatedPotentials
24
Putze, F., Schünemann, M., Schultz, T., & Stuerzlinger, W. Automatic classification of auto-
correction errors in predictive text entry based on EEG and context information. ACM ICMI 2017
Correcting Auto-correct!

25. Error RelatedPotentials … For implicitcontrol
25
Zander, T. O., Krol, L. R., Birbaumer, N. P., & Gramann, K. Neuroadaptive technology enables implicit cursor
control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences, 2016

26. A few toolsfor BCI development
http://openvibe.inria.fr)
https://mne.tools/
26

27. Virtual Environments
Part II:

28. Digitally created 3D immersive environment
28
[Milgram and Kishino1994]

29. BCIsand AR/VR
Part III:

30. Combining BCI and VR/AR: A land of opportunities
30

31. The use of SSVEP in AR/VR
Part IV:
Legeny
et al. 2013

32. Is itpossible to exploit an SSVEP basedBCI in AR?
32
•Si-Mohammed et al., TowardsBCI-based Interfaces for AugmentedReality: Feasibility, Design and Evaluation, IEEE TVCG, 2018

33. How to integrateSSVEP stimulations in the AR environment?
33
•Si-Mohammed et al., TowardsBCI-based Interfaces for AugmentedReality: Feasibility, Design and Evaluation, IEEE TVCG, 2018

34. Robot control system based on AR and SSVEP
34
Si-Mohammed et al., Towards BCI-based Interfaces for Augmented Reality: Feasibility, Design and Evaluation,
IEEE TVCG, 2018

35. Integrated AR user interface
35
Collaboration:
Si-Mohammed, et al. "DesigningFunctionalPrototypes CombiningBCI and AR for Home Automation." International Conferenceon Virtual Reality and Mixed Reality.
Springer, Cham, 2022.
F. Bouchenak

36. How to improvethe user friendlinessof the SSVEP stimulations?
36
Collaboration:
A. Wilson
C. Holz
Si-Mohammed, et al. "On the Effect of Size and Contrast of the SSVEP Visual Stimuations on Classification Accuracy and User-
Friendliness in Virtual Reality." Winter BCI Conference. IEEE, Seoul, 2023.

37. How to improvethe user friendlinessof the SSVEP stimulations?
37
Classification
accuracy
Subjective
preference

38. Whatif wecouldgetridof the flickeringstimulations?
38
Patent:
•Using Real-World Inspired Animations to Create Pleasant SSVEP-Based Brain-Computer Interfaces, US Patent 407846-US-NP
Collaboration:
A. Wilson

39. Towardsgaze independentBCIs
39
Van Den Kerchove, et al. "Correcting for ERP latency jitter improves gaze-independent BCI decoding." Journal of Neural Engineering, 2024
A. Van Den Kerchove

40. CharacterizingUser
Experiencein VR usingEEG
Part V:

41. Anomalydetectionin VR usingErrorRelatedPotentials
41
Collaboration:
R. SchererC. Lopes-Dias
Si-Mohammed, Hakim, et al. "Detectingsystem errorsin virtualreality usingEEG througherror-relatedpotentials."2020 IEEE Conference
on Virtual Reality and 3D User Interfaces (VR). IEEE, 2020.

42. EEG-baseddetectionof cybersickness
42
•GENESIS: LeveraGing nEuromarkers for Next-gEneration immerSIve Systems
GENESIS

43. Identification of markers of Vection in VR using EEG
43
M. Naud

44. Conclusion and perspectives
Part III:

45. Towardsthe regulationof neurotechnologies
45

46. Open challenges
46

47. Summaryof the presentation
47

48. Thank you for your attention
Hakim Si-Mohammed
Associate Professor
Univ. Lille, CRIStAL
Brain-Computer Interfaces team
Contact:[email protected]
Universityof South Australia, 2024