Artificial Passenger

1. ARTIFICIAL ARTIFICIAL
PASSENGERPASSENGER
(A Sleep Prevention Dialogue Based Car
System)
By
Kini Priyanka

2. IntroductionIntroduction
IBM (International business machines IBM (International business machines
corporation, NY) has developed a corporation, NY) has developed a
software that holds a conversation with the software that holds a conversation with the
driver to determine whether the driver can driver to determine whether the driver can
respond alertly enough, called “respond alertly enough, called “Artificial Artificial
PassengerPassenger”.”.
This was designed to make long solo This was designed to make long solo
journeys safer and more bearable.journeys safer and more bearable.

3. Why Such System?Why Such System?
According to a national survey in UK and USA, it According to a national survey in UK and USA, it
is observed that the driver fatigue annually is observed that the driver fatigue annually
causescauses
•10000 crashes10000 crashes
•1500 deaths 1500 deaths
•7100 injuries 7100 injuries
Majority of off-road accidents observed, were Majority of off-road accidents observed, were
caused by eye closure of half and even 2-3 caused by eye closure of half and even 2-3
seconds, where the normal human eye blinks at seconds, where the normal human eye blinks at
0.2-0.3 seconds.0.2-0.3 seconds.

4. What is an Artificial Passenger?What is an Artificial Passenger?
Natural language e-companion.Natural language e-companion.
Sleep preventive device in cars to Sleep preventive device in cars to
overcome drowsiness.overcome drowsiness.
Life safety system.Life safety system.

5. What Does It Do?What Does It Do?
Detects alarm conditions through sensors.Detects alarm conditions through sensors.
Broadcasts pre-stored voice messages Broadcasts pre-stored voice messages
over the speakers.over the speakers.
Captures images of the driver.Captures images of the driver.

6. Condition SensorCondition Sensor
Mobile Indicator Mobile Indicator
DeviceDevice

7. About Artificial PassengerAbout Artificial Passenger
The AP is an The AP is an Artificial IntelligenceArtificial Intelligence based based
companion that will be resident in software companion that will be resident in software
and chips embedded in the automobile and chips embedded in the automobile
dashboard. dashboard.
The system has a conversation planner The system has a conversation planner
that holds a profile of you, including details that holds a profile of you, including details
of your interests and profession. of your interests and profession.

8. A microphone picks up your answer and A microphone picks up your answer and
breaks it down into separate words with breaks it down into separate words with
speech-recognition software. speech-recognition software.
A camera built into the dashboard also A camera built into the dashboard also
tracks your lip movements to improve the tracks your lip movements to improve the
accuracy of speech recognition. accuracy of speech recognition.

9. A voice analyzer then looks for signs of A voice analyzer then looks for signs of
tiredness by checking to see if the answer tiredness by checking to see if the answer
matches your profile. Slow responses and matches your profile. Slow responses and
lack of attention are signs of fatigue.lack of attention are signs of fatigue.
If you reply quickly and clearly, the system If you reply quickly and clearly, the system
judges you to be alert and tells the judges you to be alert and tells the
conversation planner to continue the line conversation planner to continue the line
of questioning. of questioning.

10.  If your response is slow or doesn’t make If your response is slow or doesn’t make
sense, the voice analyzer assumes you sense, the voice analyzer assumes you
are dropping off and acts to get your are dropping off and acts to get your
attention.attention.
If driver displays signs of fatigue, the If driver displays signs of fatigue, the
artificial passenger might be programmed artificial passenger might be programmed
to open all the windows, sound a buzzer, to open all the windows, sound a buzzer,
increase background music volume, or increase background music volume, or
even spray the driver with ice water. even spray the driver with ice water.

11. Devices Used in Artificial Devices Used in Artificial
PassengerPassenger
Eye trackerEye tracker
Voice recognizer or speech recognizerVoice recognizer or speech recognizer
Natural language processorNatural language processor
Driver analyzerDriver analyzer
Conversational plannerConversational planner
AlarmAlarm
MicrophoneMicrophone
CameraCamera

12. Working ComponentsWorking Components
Eye-TrackerEye-Tracker
Collecting eye movement data requires both hardware and Collecting eye movement data requires both hardware and
software.software.
Hardware:: Head-Mounted systems or remote systemsHardware:: Head-Mounted systems or remote systems
Both systems measure the corneal reflection of the infrared LED, Both systems measure the corneal reflection of the infrared LED,
which illuminates and generates a reflection off the surface of the which illuminates and generates a reflection off the surface of the
eye.eye.
This action causes the pupil to appear as a bright disk in contrast to This action causes the pupil to appear as a bright disk in contrast to
surrounding iris and creates a small glint underneath the pupil which surrounding iris and creates a small glint underneath the pupil which
is used for calibration and tracking.is used for calibration and tracking.
Head mounted systemsHead mounted systems use multiple data points to record eye use multiple data points to record eye
movement and measures pupil glint from multiple angles.movement and measures pupil glint from multiple angles.
Remote systemsRemote systems ,in contrast, measure the orientation of the eye ,in contrast, measure the orientation of the eye
relative to a fixed unit such as camera.relative to a fixed unit such as camera.

13. Monitoring SystemMonitoring System

14. Software:Software:
Data collection and analysis is handled Data collection and analysis is handled
by eye-tracking software.by eye-tracking software.
Software catalogs eye-tracking in one of Software catalogs eye-tracking in one of
the two waysthe two ways
4.4.ERICA’s Eye Gaze software uses a ERICA’s Eye Gaze software uses a
small red x to represent eye movement.small red x to represent eye movement.
5.5.In other, data is stored as a series of x/y In other, data is stored as a series of x/y
coordinates related to specific grid points coordinates related to specific grid points
on the computer screenon the computer screen

15. Eye TrackerEye Tracker

16. Working Components( cont…)Working Components( cont…)
Automatic Speech RecognitionAutomatic Speech Recognition
There are two ASRs used in the system:There are two ASRs used in the system:
First one is “speaker independent” and used for First one is “speaker independent” and used for
decoding voice signals of the driver.decoding voice signals of the driver.
Second one operates with voice car media and Second one operates with voice car media and
decodes tapes, audio , telephones, mails etc.decodes tapes, audio , telephones, mails etc.

17. Working Components( contd…)Working Components( contd…)
Natural Language ProcessorNatural Language Processor
Processes the decoded signals of voice or text data provided Processes the decoded signals of voice or text data provided
from the ASR. from the ASR.

Identifies related meanings from the contents of the decoded Identifies related meanings from the contents of the decoded
messages.messages.
Produces variant of responses.Produces variant of responses.
This output goes to the driver analyzer as an input.This output goes to the driver analyzer as an input.

18. Working Components( contd…)Working Components( contd…)
Driver AnalyzerDriver Analyzer
Receives the textual data and voice data. Receives the textual data and voice data.
Measures the time response using a clock.Measures the time response using a clock.
Time responses, conclusions about driver’s Time responses, conclusions about driver’s
alertness will be passed on to the alertness will be passed on to the
conversational planner.conversational planner.

19. Working Components( contd…)Working Components( contd…)
Conversational PlannerConversational Planner
It is the heart of the system. It is the heart of the system.
Instructs the language generator to produce the Instructs the language generator to produce the
response.response.
If the driver is in perfect condition CP instructs the If the driver is in perfect condition CP instructs the
language generator to continue the conversation language generator to continue the conversation
otherwise it will be instructed to change the otherwise it will be instructed to change the
conversation.conversation.

20. The following is a typical scenario involving Artificial The following is a typical scenario involving Artificial
Passenger. Imagine, driver “Joe” returning home after late Passenger. Imagine, driver “Joe” returning home after late
nights. His head starts to nod nights. His head starts to nod
APAP Hey Joe, what did you get your daughter for her Hey Joe, what did you get your daughter for her
birthday?birthday?
Joe Joe It’s not her birthday!It’s not her birthday!
AP AP You seem a little tired. Want to play a game?You seem a little tired. Want to play a game?
Joe Joe Yes.Yes.
APAP You were a wiz at “Name that Tune” last time. I was You were a wiz at “Name that Tune” last time. I was
impressed. Want to try your hand at trivia?impressed. Want to try your hand at trivia?
JoeJoe ok. ok.
APAP Pick a category: Hollywood Stars, Magic Moments or Pick a category: Hollywood Stars, Magic Moments or
Hall of Fame?Hall of Fame?
JoeJoe Hall of Fame. Hall of Fame.

21. Working Components (contd…)Working Components (contd…)
AlarmAlarm
If the CP (conversational planner) receives If the CP (conversational planner) receives
information that the driver is about to fall asleep information that the driver is about to fall asleep
an alarm system is activated.an alarm system is activated.
MicrophoneMicrophone
For picking up words and separate For picking up words and separate
them by some internally used softwarethem by some internally used software
for conversation. for conversation.

22. Working Components (contd…)Working Components (contd…)
CameraCamera
This will track the lip movements of the This will track the lip movements of the
driver.driver.
Used to improve the accuracy of the Used to improve the accuracy of the
speech recognition.speech recognition.

23. ..
Showing the dashboard of the car where the whole artificial system is generally Showing the dashboard of the car where the whole artificial system is generally
attached.attached.

24. Steps Involved in TrackingSteps Involved in Tracking
Automatically initialize lips and eyes using Automatically initialize lips and eyes using
color predicates and connected color predicates and connected
components. components.
Track lip corners using dark line between Track lip corners using dark line between
lips and color predicate even through large lips and color predicate even through large
mouth movement like yawning. mouth movement like yawning.

25. Construct a bounding box of the head.Construct a bounding box of the head.
Determine rotation using distances Determine rotation using distances
between eye and lip feature points and between eye and lip feature points and
sides of the face. sides of the face.
Determine eye blinking and eye closing Determine eye blinking and eye closing
using the number and intensity of pixels in using the number and intensity of pixels in
the eye region.the eye region.
Determine driver vigilance level using all Determine driver vigilance level using all
acquired information.acquired information.

26. DetectingDetecting Driver VigilanceDriver Vigilance
Aiming a single camera at a head of the driver. Aiming a single camera at a head of the driver.
Detecting frequency of up and down nodding and left to Detecting frequency of up and down nodding and left to
right rotations of the head within a selected time period right rotations of the head within a selected time period
with the camera. with the camera.
Determining frequency of eye blinking and eye closing.Determining frequency of eye blinking and eye closing.
Determining frequency of yawning of the driver within the Determining frequency of yawning of the driver within the
selected time period with the cameraselected time period with the camera
Generating an alarm signal in real time if the frequency Generating an alarm signal in real time if the frequency
value of the up and down nodding, the left to right value of the up and down nodding, the left to right
rotations, the eye blinking, the eye closings, the yawning rotations, the eye blinking, the eye closings, the yawning
exceeds a selected threshold value. exceeds a selected threshold value.

27. ApplicationsApplications
Interface with Neighboring CarsInterface with Neighboring Cars - -Determines if a driver presents a Determines if a driver presents a
high safety risk for ex. falling asleep, tired, inexperienced or under high safety risk for ex. falling asleep, tired, inexperienced or under
the influence of alcohol and signals the cars nearby to be careful of the influence of alcohol and signals the cars nearby to be careful of
the driver.the driver.
Medical ApplicationMedical Application -The system can monitor a driver and detect if -The system can monitor a driver and detect if
they are sick ,for ex. having a stroke or heart attack.they are sick ,for ex. having a stroke or heart attack.
In any problem it alerts the vehicles near by , so the driver there can In any problem it alerts the vehicles near by , so the driver there can
become alert.become alert.
Opens and closes the doors and windows of the car automatically.Opens and closes the doors and windows of the car automatically.
It is also used for the entertainment.It is also used for the entertainment.
Also used in cabins in airplanes, trains, boats etcAlso used in cabins in airplanes, trains, boats etc

28. Future ImplementationFuture Implementation
Will provide us with shortest time routing Will provide us with shortest time routing
based on road conditions changing based on road conditions changing
because of weather and traffic, information because of weather and traffic, information
about the cars on the route, destination about the cars on the route, destination
requirement (as-flight has been delayed requirement (as-flight has been delayed
etc)etc)

29. ConclusionConclusion
Method for monitoring driver alertness Method for monitoring driver alertness
Sufficient time to avert an accident. Sufficient time to avert an accident.
Successful implementation of Artificial Successful implementation of Artificial
passenger would allow use of various passenger would allow use of various
services in car like reading emails, services in car like reading emails,
navigation, downloading music files, navigation, downloading music files,
voice games etc without compromising voice games etc without compromising
on driver safetyon driver safety

30. Questions?Questions?

31. Thank You…Thank You…