ReflecTrace: Hover Interface using Corneal Reflection Images Captured by Smartphone Front Camera

本研究では、スマートフォンに内蔵された前面カメラで取得した角膜反射画像を用いて、画面上の指のホバー入力を検出する手法を提案する。本手法は外部センサや追加ハードウェアを一切必要とせず、前面カメラでは直接撮影できない画面近傍空間における指位置の検出を可能にする。提案手法では、畳み込みニューラルネットワーク（CNN）を用いてホバーしている指の二次元位置を推定し、あらかじめ定義したグリッドのどのセル上に位置しているかを判定する。実験の結果、提案モデルは異なるグリッド設定においても二次元位置推定で高い精度を達成し、2×2のような粗いグリッドでは90％以上、最大5×5までの細かい分割でも80％以上を維持し、その頑健性と適応性を示した。

1. 1 2 3
4 5 6
7 8 9
10 11 12 1 2 3 4 5 6 1 2 3 4 5 6 78 9 10 11 12 1 2
34 5 6
Extension to gesture input in real-time operation
Improving model generalization and robustness to
external conditions
Hover Interface using Corneal Reflection Images
Captured by Smartphone Front Camera
ReflecTrace
Yudai Nakamura Kaori Ikematsu Naoto Takayanagi Kunihiro Kato Yuta Sugiura
1) 2) 1) 3) 1)
01
Introduction
Estimating the hovering finger’s area from corneal reflection images captured by front-facing camera
Background Problem Approach
Keio University LY Corporation Tokyo University of Technology
1) 2) 3)
Touch interaction faces several limitations
Glove use limits touch input
Sanitary concerns
Smartphones mainly rely on Touch Input
Hover Detection requires special hardware
Dependent on external sensors
Not applicable to general smartphones
Existing approaches are...
However, front camera cannot see the
hover space above the screen
Use corneal reflections to detect finger
shadows caused by hover
Hover position is inferred by a CNN
using predefined screen grid regions
02
ReflecTrace
Front Camera
Acquiring Facial
Image & Landmarks
Eye Image
Corneal Reflection
Estimating
Hover Position
Hover InputConvolution 2D Layer Max Pooling 2D Layer Convolution 2D Layer Flatten & Dense Dropout & Dense Convolution 2D Layer Max Pooling 2D Layer
Image Recognition
CNN Model Focus Region
03
Experiment
Dataset
14 participants (42,336 images)
4 body postures (sitting, standing, prone, supine)
3 display brightness levels (70%, 85%, 100%)
Inference Accuracy (4x5)
User-specific model: 85.73% (average)
All-subjects model: 82.05%
Accuracy ranged from 72.2% to 96.5%
04
Grid Evaluation
Accuracy declines with finer grids; 3×3 - 4×4 are practical
More classes require more data and model capacity
Errors are mostly adjacent, enabling smooth correction
05
Future Work
Application to swipe-hover input Lighting-induced iris variation
Contact: [email protected]