Abstract—With the recent diversification of operating
devices, the demand for input operations that require
confirmation of the effect of differences in display response on
operability has increased. Regarding display response, previous
studies have investigated the threshold time and sense of agency
for a delayed response during device operation. However, these
studies only focused on subjective evaluations. Therefore, this
study aims to clarify the human motor characteristics and
activated brain regions based on the differences in display
response time during device operation. The target motion is the
rotational operation of the cylindrical rotary controller using the
index finger and thumb. The experimental conditions involve
four types of display response times (the duration from the
operation to the indicated response). We measured the brain
activity using near-infrared spectroscopy, the muscle activity
from a surface myoelectric potential measurement device, and
the force data of the index finger and thumb tip obtained from
two independent six-axis force/torque sensors. Although the
experimental results showed no significant difference in the
muscle activity and gripping force, a significant difference was
observed in the brain activity and the questionnaire survey by
the difference in display response time. This investigation reveals
that the difference in display response time affects brain activity
and subjective information, clarifying the relationship between
brain activity and subjective information.
I. INTRODUCTION
Recently, human–machine interface (HMI) devices, such
as touch screens and video presentation devices that transmit
information between humans and machines owing to the
development of virtual reality technology, have become more
complex and diversified. Thus, the development of HMI
devices that can be easily operated by humans is required.
Furthermore, the number of input operations that require
display response confirmation of the operations is increasing.
Various studies have focused on the display responses of
devices. For example, during device operations in
environments that assume an augmented reality or virtual
reality space, delays in display responses lead to discomfort
in humans, reducing immersiveness [1,2]. Therefore, a lower
display delay response characteristic is required during
operation in a highly immersive environment. Numerous
studies have also investigated the sense of agency (SoA),
which is the sense that “I am the one who is causing or
generating an action” [3] while operating the device [4].
These studies have reported that the SoA is reduced in
response without delay [5,6]. Other studies have investigated
the delay response that humans can perceive [7]. In
experiments concerning delay recognition of self-body
images, a response delay of 230 ms or more is recognizable
[8]; in touch screen operation, humans can recognize a delay
of 10 ms or less [9]. According to the results of these studies,
the delay recognition and SoA to the response of the device
differ based on the operation target and environment; a lower
delayed response to human operation sensation is not optimal.
Previous studies have also investigated brain activity, SoA,
and delayed recognition [10,11]. However, several human
operation sensations to the response of device operation have
been qualitatively evaluated through subjective information,
and few studies have examined quantitative evaluations based
on objective indicators.
In this study, we evaluated the effect of the difference in
display response time during device operation on human
subjective information and brain activity. Evaluating brain
activity during exercise is a useful quantitative evaluation
method because exercises, such as device operation, are
performed by processing sensory information in the brain and
transmitting commands to the muscles [12]. We
quantitatively evaluated the human operation sensation in
response to device operation by analyzing the activity state of
the cerebral cortex, subjective information on operability by a
questionnaire, muscle activity, and gripping force. The
purpose of this study was to clarify human motor
characteristics and brain region to be activated, resulting from
the difference in display response time. We aimed to elucidate
human perceptual characteristics and establish device design
indicators related to operational sensations; therefore,
quantitative evaluation of operational sensations is useful as
basic information in device development. As the target of the
operation device, we focus on the grip rotation operation by
the index finger and thumb of the cylindrical rotary
commander used to adjust the temperature and volume in the
car and select the menu of the car navigation display. Previous
studies have evaluated the steering operation [13] and in-
vehicle HMI using a driving simulator [14,15]. However,
studies regarding the response to car navigation displays have
not been conducted. In-vehicle devices, such as car navigation
systems, require complicated input operations because of the
Effect of Display Response Time on Brain Activity
in Human–Machine Interface Commander Operation
Kentaro Oshima, Toru Tsumugiwa, Member, IEEE, Ryuichi Yokogawa, Member, IEEE,
Mitsuhiro Narusue, Hiroto Nishimura, Yusaku Takeda, and Toshihiro Hara
This work was partially supported by JSPS KAKENHI Grant-in-Aid for
Scientific Research (C) Grant Number 20K04388, 2020 and 21K03973,
2021.
K. Oshima is with the Graduate School of Life and Medical Sciences,
Doshisha University, Kyoto, Japan.
T. Tsumugiwa is with the Department of Biomedical Engineering,
Faculty of Life and Medical Sciences, Doshisha University, Kyoto, Japan
(e-mail:
[email protected]).
R. Yokogawa is with the Department of Biomedical Engineering,
Faculty of Life and Medical Sciences, Doshisha University, Kyoto, Japan
(e-mail:
[email protected]).
M. Narusue is with the Technical Research Center, Mazda Motor
Corporation, Hiroshima, Japan.
H. Nishimura is with the Technical Research Center, Mazda Motor
Corporation, Hiroshima, Japan.
Y. Takeda is with the Technical Research Center, Mazda Motor
Corporation, Hiroshima, Japan.
T. Hara is with the Technical Research Center, Mazda Motor
Corporation, Hiroshima, Japan. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
September 27 - October 1, 2021. Prague, Czech Republic
978-1-6654-1714-3/21/$31.00 ©2021 IEEE 6204
2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) | 978-1-6654-1714-3/21/$31.00 ©2021 IEEE | DOI: 10.1109/IROS51168.2021.9636452
Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.