Agricultural Person Detection using Camera

Agricultural Person Detection Dataset

1. NREC Agricultural Person Detection Dataset

2. 2
Introductions
This project was funded by the Test Resource Management Center (TRMC) Test
and Evaluation / Science & Technology (T&E/S&T) Program through the U.S.
Army Program Executive Office for Simulation, Training and Instrumentation
(PEO STRI) under Contract No. W900KK-16-C-0006, “Robustness Inside-Out
Testing (RIOT).” NAVAIR Public Release 2018-165. Distribution Statement A –
“Approved for public release; distribution is unlimited”.
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection
Zachary Pezzementi, PhD
Lead Robotics Engineer
Trenton Tabor
Senior Robotics Engineer
Today’s
Speakers:
National Robotics Engineering Center (NREC)
Carnegie Mellon University
Pittsburgh, PA, USA

3. 3
Previous Work
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

4. 4
Previous Work
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

5. 5
Comparison to Other Datasets
First known benchmark for person detection in agricultural
environment or off-road in general
Existing datasets for person detection from vehicles focus on
urban environments and had neither as much nor as rich data:
Dataset # Img.
Train
# Img.
Val
# Img.
Test
StereoVideo Vehicle
Position
Environment
KITTI 7,481 - 7,518 ✔ ✔ - Urban
Caltech 34,893 784 4,025 - ✔ - Urban
Daimler 22,789 - 21,790 ✔* ✔* - Urban
NREC 48,37023,57723,950 ✔ ✔ ✔ Off-road,
Agricultural
BDD100K 70,000 - 30,000 - ✔ ✔ Urban
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection
* Daimler provides stereo and video for test set only.

6. 6
Environment comparison—HoG
visualization
Visualization using voc-release5, Urban image from Caltech pedestrian dataset
Example off-road encounter with person
Example urban encounter with person

7. 7
Talk Overview
•Platform and Environment
•Data Collection and Annotation Design
•Person Detection Evaluation Criteria
•Detection Results
-Performances on Data Slices
-Sample Detections
•Work Using the Dataset
-Simulating Challenging Conditions
-Evaluating Robustness
-Other Data Available
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

8. Dataset Introduction

9. 9
Orange Orchard: Devil’s Garden Grove
•Initial dataset collected in
Florida orange grove
•Leveraging previous
work, multiple data
collections including
-Static people in different
poses
-People in motion
-Various outfits
-Varied lighting/weather
-Varied poses
-Varied levels of occlusion
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

10. 10
Devil’s Garden Grove Context
3 Miles
2 Miles
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

11. 11
Platform
Off-road vehicle equipped for autonomy
Stereo camera pair
GPS
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

12. 12
Soergel’sApple Orchard Dataset
•New data set in local
apple orchard
•Covers same types of
poses and motions as
previous orange data
•Increases coverage of
motion and unusual
poses
•Evaluate generalization to
new environment, season
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

13. 13
SoergelsApple Orchard Overview
700 ft
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

14. 14
Apple Orchard Data Collection Vehicle
GPS
Stereo Cameras
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

15. 15
Camera Details
•Custom high- dynamic-
range cameras
•Resolution: 720x480
•Baseline: 20 cm
•Field of View:
-80
o
horizontal
-56
o
vertical
•Frame Rate
-7.5 Hz for orange data
-15 Hz for apple data
-Labeled at 7.5 Hz for
consistency
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

16. Data Collection Design

17. 17
Log Types
•Positive Logs
-Types
-Static Person
-Vehicle drives up to stationary standing/crouching person and comes to a
stop
-Moving Person
-Similar to above, but person is carrying out one of several motions
-Unusual Poses
-Person is in one of several more challenging and unconventional poses
-Capture a full set of these for a person wearing the same
outfit.
•Negative Logs
-Longer logs where vehicle drives around and no people are
in view
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

18. 18
Static Person Locations
•Person is standing in
each location within the
row shown at left.
•Varies the level of
visibility and occlusion at
different distances.
•Repeat for crouching.
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

19. 19
Static Person Location Sample Images
In Tree Row Edge Row Center Tree Gap
Apple
Orange
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

20. 20
Apple Orchard Sample Static Log
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

21. 21
Person in Motion Logs
•Typical paths for vehicle and personin moving person logs:
•(a) Awaysequence, where the vehicle follows behind a person walking along the row.
•(b), (c) show Toward, with the person approaching the moving vehicle along the edge/center of row.
•(d), (e) show Crossand Step, subject crosses or steps in front of the vehicle at different distances.
•(f) Get Up and Leave, seated subject stands up as vehicle approaches and exits to side of row.
•(g) shows a vehicle- person interaction at a row turn.
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

22. 22
Apple Orchard Sample Motion Log
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

23. 23
Unusual Pose Logs
•Features people lying down,
falling down, and climbing on
ladders.
•Intended to capture situations
where people are vulnerable,
but may not have been seen
often.
•These were collected using
mannequins for safety.
Falling Person
Lying along Row
On Ladder
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

24. 24
Sample (Challenging) Log from Cab
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

25. Evaluation

26. 26
Image Labeling
•Labeling format
mimics Caltech
dataset
•Bounding box in
left camera
image for visible
extent of person
•Interpolated
across a video
sequence to
speed up
process
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

27. 27
Train/Test/Val Breakdown
•Logs are broken into sets made up of a person in a
particular outfit.
•Entire set assigned to train, test, or validation.
•Selected to maintain ~50/25/25 balance, while maximizing
variety (appearance, time of day, etc.) within each subset.
•Perform allalgorithm development and tuning on train+val,
and run on test only for final result.
Train Val Test
# Labeled Images 48,370 23,950 23,577
# Positive Log Sets 23 10 13
# Positive Logs 403 201 220
# Negative Logs 52 26 24
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

28. 28
There are choices to be made
when converting bounding box
detections into correctness.
Usually use threshold on
overlap
Computed as the ratio of
Intersection (green area) to
Union (all area) of bounding
boxes
Appropriate choice depends on
task
Evaluation: Overlap Area Threshold
Considered bad detections with OA<0.5
Considered bad detections with OA<0.3
Ground Truth
CNN detections
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

29. 29
ROC Explanation
Ideal
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection
All ROCs are generated using required overlap of 0.5.

30. 30
Average Detection Rate Computation
•Overall rankings use
“average miss rate”,
which averages across
two quantities
-Sensitivity threshold
-Required overlap area
•Sample at vertical lines
and average across
curves
•Obtain detection rate as
1 –miss rate.
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

31. 31
Algorithms Evaluated
•Prior Work
-DetectNet
-Tao et al., 2016
-Baseline, generic, off-the-shelf method available from NVIDIA
-RPN+BF (Region Proposal Network + Boosted Forest)
-Zhang et al., 2016
-Leader on Caltech benchmark
-MSCNN (Multi- Scale CNN)
-Caiet al., 2016
-Leader on KITTI benchmark with public implementation
•Our novel detector
-MFC –Multiscale Foveal Context
-See paper for details
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

32. Results

33. 33
Metadata / Subsets
•Along with labeled data,
log names contain
significant metadata.
•Allows evaluation on
several interesting
subsets that highlight
different algorithms’
strengths/weaknesses.
Sample Log Names:
2015_11_16_1408_person_BlackHood_SoergelB_MiddaySunny_Row_StandingFrontalTireTracksPrimaryLog
2014_08_29_2220_movingPerson203SwaleMorningSunWeeds2ms_GreenPlaid_Row_MediumCross_PrimaryLog
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

34. 34
Results on Static Vs Motion
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

35. 35
Results on Occlusion levels
Clear/Partial
(70% visible)
Partial/Heavy
(30% visible)
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

36. 36
Results on Different Scales
Medium/Small
1300 px
Large/Medium
3500 px
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

37. 37
Results on Different Poses
Standing
Crouching
Lying
Falling
On Ladder
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

38. 38
Average Detection Rate Results by Subset
Subset RPN+BF MSCNN DetectNet MFC
Baseline 44.2 54.6 9.0 59.4
Env=Orange 46.0 55.0 11.0 59.1
Env=Apple 48.5 56.6 6.9 65.8
All 43.5 53.8 8.9 58.4
Occ=Clear 52.1 60.7 10.5 67.3
Occ=Partial 31.8 40.9 6.1 47.2
Occ=Heavy 9.9 7.3 0.3 16.8
Scale=Large 59.7 72.7 17.3 69.0
Scale=Medium 67.7 59.5 4.4 70.7
Scale=Small 15.7 30.3 1.7 47.6
Pose=Typical 48.8 59.5 10.2 62.7
Pose=Unusual 14.0 19.1 0.7 41.1
Motion=Static 40.5 53.0 10.5 56.4
Motion=Moving 57.9 60.0 7.8 70.4
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

39. 39
Test Set Detection Video Sample
Full video is nearly an hour long

40. Putting Image Manipulations in Context:
Robustness Testing for Safe Perception
Trenton Tabor, Senior Robotics Engineer
National Robotics Engineering Center
Zachary Pezzementi, Trenton Tabor, Samuel Yim, Jonathan K. Chang, Bill
Drozd, David Guttendorf, Michael Wagner and Philip Koopman
“Robustness Inside- Out Testing (RIOT).” NAVAIR Public Release 2018- 165.
Distribution Statement A –“Approved for public release; distribution is unlimited”

41. 41
Detectors Tested
•Many state- of-the-art
object detectors from
some of the most
popular deep network
frameworks
•Baseline Performance:
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

42. 42
Philosophy of Approach –
Robustness Testing for Perception
•Chose to partner with the robustness
testing group at our University
Robustness Testing is the process of
generating many queries of a system and
requires knowledge of what wrong behavior
is for these inputs.
This technique has found real, dangerous
bugs
-Previously tested 17 robotic systems
over several years
-An example, shown, is finding a
planner ignoring constraints, leading to
erratic behavior
Applying to perception…
Perception systems have such high dimensional inputs to make pure generation of new inputs
impractical.
We propose instead using physically grounded mutations of previously labeled data to create
exceptional inputs.
C. Hutchison et al. “Robustness testing of autonomy
software.” International Conference on Software
Engineering, 2018.
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

43. 43
Details of Approach
We propose using physically grounded mutations of
previously labeled data to create exceptional inputs.
•We built a list of degradations that occur in outdoor
imaging
•We implemented parameterized mutators modeling
some of these
•We used these mutated datasets to estimate robustness
of safety critical machine learning systems
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

44. 44
Basic Mutators
•All are literature backed degradations
•See paper for details and model references
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

45. 45
Mutators Using Geometry (“Contextual”)
•All are literature backed degradations
•See paper for details and model references
Haze
Defocus
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

46. 46
Striking Results
Small changes to images, even
physically realistic changes, can cause
a catastrophic change in classifier
performance
Raw Detection Strength
Ground Truth Label
Required False Positive Rate
Original
Mutated
MSCNN detections on original images and under moderate blur
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

47. 47
Sample Results: Channel Dropout
•Channel dropout is devastating to most detectors
Original Image Drop Channel Cb(YCbCr)
Better
Worse
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

48. 48
Sample Results: Haze
•There is variation in detector robustness to haze
Original Image Haze w/ 97.8m Visibility (????????????0.04)
Better
Worse
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

49. 49
Full Results
•Evaluated full combination
of mutators and detectors
•Allows analysis of general
robustness characteristics
of each detector
•Sometimes would change
choice of best detector,
depending on importance
of adverse conditions to
you
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

50. 50
Major Take- Aways
•Small image changes can have catastrophic effects on
safety critical perception.
•In fact, common image degradations can often cause
such failures for systems running over long periods.
•We demonstrate this on many state- of-art fieldable
systems within a framework for evaluating robustness in
adverse conditions.
Dataset download link:
http://www.nrec.ri.cmu.edu/projects/usdapersondetection

51. Other Data in Dataset

52. 52
Stereo
•Evaluated algorithms
only use left camera
image, but 3D
information is
available.
•Continuous video
also allows
application of scene
flow techniques.
•3D mesh generated
using PRSM
1
(KITTI
scene flow leader w/
public code) on one
stereo image pair.
1
Vogel et al. “3D Scene Flow Estimation with a
Piecewise Rigid Scene Model” IJCV, 2015
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

53. 53
Motion
•Evaluated algorithms process each image
independently
•Could incorporate motion information from previous
frames
•Could track or temporally filter detections
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

54. 54
Vehicle Motion Ground Truth
•Every frame of stereo
video is associated with
RTK GPS, with vehicle
geometry available.
•Available for evaluation of
visual odometryor
development of motion-
based algorithms
x0
(m)
x1
(m)
z0
(m)
z1
(m)
????????????
Apple 1.421.050.352.1522
o
Orange 1.79 0 0 3 29
o
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection

55. 55
Visual Odometry
•RTK GPS along with
stereo video
provides good data
for benchmarking
VO.
•Sample comparison
of VO algorithm to
logged pose over
time.
Birdseye View
Full SE(3) Pose

56. 56
Our Team
…and you? Internship,
post-doc, and full time
positions available
W. Drozd
D. GuttendorfM. Wagner
P. Koopman
Z. Pezzementi*T. Tabor* S. YimJ. K. Chang
P. Hu D. RamananB. P. W. Babu
C. Wellington
H. Herman
Interns
Staff
Campus Faculty
Dataset download link: http://www.nrec.ri.cmu.edu/projects/usdapersondetection