“Image Signal Processing Optimization for Object Detection,” a Presentation from Nextchip

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/image-signal-processing-optimization-for-object-detection-a-presentation-from-nextchip/

Young-Jun Yoo, Executive Vice President at Nextchip, presents the “Image Signal Processing Optimization for Object Detection” tutorial at the May 2024 Embedded Vision Summit.

This talk delves into the challenges and optimization strategies in image signal processing (ISP) for enhancing object detection in advanced driver-assistance systems (ADAS). Through real-world examples, Yoo explores the critical role of image tuning in addressing corner cases and improving detection accuracy.

The presentation covers ADAS sensing methodologies and verification processes, emphasizing the importance of image quality in sensor data. Yoo provides practical insights into image tuning techniques, including day-night transition handling and environment-based adjustments. He shares valuable knowledge on optimizing ISP to ensure robust object detection, enhancing safety and performance in automotive applications.

1. Young-Jun Yoo
EVP
NEXTCHIP CO., LTD.
Image Signal Processing Optimization
for Object Detection

2. NEXTCHIP Overview
•Developing & optimizing visioncore;
Image signal processing technology for 27 years
•Tuning know-how with various MP models
with globalOEMs and Tiers
•Tuning capability for human vision & machine mision
•Open architecture with various image sensors,
CFAs (color filter arrays)
World-class ISP In-house Core
Automotive Reliability
ASIC Design Technology
• Automotive process foundry experience;
14nm/28nm/55nm/60nm/95nm
Samsung/Global Foundries/USJC/TSMC
•ISO26262; Functional safety
•Cyber security
•CMMI Lv.-3
•A-Spice process
•AEC-Q100 Gr.2 lineup
© 2024 NEXTCHIP 2

3. 3
© 2024 NEXTCHIP
Image Signal Processing Optimization
for Object Detection
Chapter 1:What is the Difference? Human Vision vs. Machine Vision
Chapter 2:The Image Tuning Challenges for Human Vision
Chapter 3: The Image Tuning Challenges for Machine Vision

4. 4
Image Signal Processing Optimization
for Object Detection
© 2024 NEXTCHIP
Chapter 1:What is the Difference? Human Vision vs. Machine Vision
Chapter 2:The Image Tuning Challenges for Human Vision
Chapter 3: The Image Tuning Challenges for Machine Vision

5. Human Vision vs. Machine Vision
5© 2024 NEXTCHIP
• We asked this question to ChatGPT… It gave this image as an answer!
Do you feel the same way?

6. Image Tuning Needed for Both Types of Vision
6
ISPImage Sensor
Display
Lens
The key point of image sensor
Delivery of Robust “raw (bayer) data”
①Sensitivity (Pixel technology)
②Color (CFA-color filter array)
The key point of ISP
Processing “image signal data”
①Reproduction signal to vision
②Color/less noise
• What is image tuning? Why is it needed?
© 2024 NEXTCHIP

7. 7
Image Signal Processing Optimization
for Object Detection
© 2024 NEXTCHIP
Chapter 1:What is the Difference? Human Vision vs. Machine Vision
Chapter 2:The Image Tuning Challenges for Human Vision
Chapter 3: The Image Tuning Challenges for Machine Vision

8. Image Tuning Challenges for Human Vision
8
What is the challenge?
Make the image as similar as possible to one seen through a human eye
What is the key factor to tune for human vision?
• Color reproduction
• Lower noise level
• Brightness/edge/HDR (high dynamic range), etc.
Tuning under various environment, e.g., day & night
© 2024 NEXTCHIP

9. Image Tuning Challenges for Human Vision
9
Quantitative TEST Qualitative TEST
It presents you with
numerical value.
E.g., Δ-E, HDR dB, AE (auto
exposure) speed, etc.
It determines the user’s
motivation, comments, feeling,
etc. throughout the test process.
E.g., color balance and bright in a
sight
Color Accuracy
Field Test for Repeat
Indoor & Outdoor
Noise/Edge
Adjustment
How to do?
© 2024 NEXTCHIP

10. Image Tuning Challenges for Human Vision
10
02
Weakn
ess
Day Night
Day Result
①Tuning
②Test
③Tuning
④Repeat Test
Night Result
How to do?
© 2024 NEXTCHIP

11. Image Tuning Challenges for Human Vision
11
▶Problem
• Generally dark
• Too strong color
• Too strong edge level
© 2024 NEXTCHIP

12. Image Tuning Challenges for Human Vision
12
▶Tuning#1
• Brightness
• HDR & Contrast
• GCE
(global contrast enhancement)
© 2024 NEXTCHIP

13. Image Tuning Challenges for Human Vision
13
© 2024 NEXTCHIP
▶Tuning#2
• Color (hue, saturation)
• Color suppression

14. Image Tuning Challenges for Human Vision
14
▶Problem ▶Final tuned image
© 2024 NEXTCHIP

15. Image Tuning Challenges for Human Vision
15
▶Problem
• Generally dark
• Too strong color
• Too strong edge level
© 2024 NEXTCHIP

16. Image Tuning Challenges for Human Vision
16
▶Tuning#1
• Brightness
• HDR & Contrast
• GCE
(global contrast enhancement)
© 2024 NEXTCHIP

17. Image Tuning Challenges for Human Vision
17
▶Tuning#2
• Color (hue, saturation)
• Color suppress
© 2024 NEXTCHIP

18. Image Tuning Challenges for Human Vision
18
▶Problem ▶Final tuned image
© 2024 NEXTCHIP

19. 19
Image Signal Processing Optimization
for Object Detection
© 2024 NEXTCHIP
Chapter 1:What is the Difference? Human Vision vs. Machine Vision
Chapter 2:The Image Tuning Challenges for Human Vision
Chapter 3: The Image Tuning Challenges for Machine Vision

20. Image Tuning Challenges for Machine Vision
20
What is the challenge?
• Higher detection rate is needed
Methodsto increase detection rate such as:
• Retraining
• Changingtraining method
• Changing field of view andresolution
• Image tuning, etc.
© 2024 NEXTCHIP

21. Measure Factors for Test
21
Detection network
• YOLOv5s
Datasets
• Location : Pangyo, Korea
• Scene : Sunny, daytime & nighttime, rearview fisheye 190°
• Training image resolution : 640x360/ training images : 12,732
© 2024 NEXTCHIP

22. Test Dataset & Tuning
22
Test Dataset
• Quantitative experiments: Stationary object + Ground Truth
• Qualitative experiments: Driving scene
ISP Tuning
• Brightnesslevel: Auto exposure (AE)
• Edge sharpness level: Edge enhancement(EDGE)
• Noise level:Noise reduction (NR)
© 2024 NEXTCHIP

23. Quantitative Experiments –Metric
23
Metrics of best ISP for object detection
High Detection
Accuracy
Best ISP
High Confidence
Score
Precision
Consistency of
Detection Results
Frame(t-1)
Frame(t)
Detection Rate
Object Score
False Positive
© 2024 NEXTCHIP

24. Quantitative Experiments –ISP Tuning & Test Dataset
24
T1
Viewing Optimization Tuning
T2
All ISP OFF except color related
T4
AE Down, NR Up
T3
AE Up, EDGE Up
• 4 different ISP settings for the same scene
• About 3200 frames for each tuning point
Test dataset ISP tuning
• Brightnesslevel: Auto exposure (AE)
• Edge sharpness level: Edge enhancement(EDGE)
• Noise level:Noise reduction (NR)
© 2024 NEXTCHIP

25. Quantitative Experiments –Detection Accuracy
25
• An indicatorof recognition accuracy for each tuning point
High detectionaccuracy
© 2024 NEXTCHIP

26. Quantitative Experiments –Confidence Score
26
• Ascore which represents likelihood that the bounding box contains an object
High confidence score
© 2024 NEXTCHIP

27. Quantitative Experiments –Detection Consistency
27
• Anindicator of whether thesame object is consistently recognized
High consistencyscore
© 2024 NEXTCHIP
consistency

28. Quantitative Experiments –Precision
28
• An indicator of recognitionprecision
Precision
© 2024 NEXTCHIP

29. Quantitative Experiments –Result
29
• For all metrics, the higher the better
Totalevaluation results
Best
Second
Best
© 2024 NEXTCHIP

30. •EDGEhas the greatest impact ondetection performance
1.Too many EDGE Worse detection performance
2. More EDGE More false detections
•Darkerimage Reduced false detection rate and accuracy
• Need to fine the best ISP setting value between T2 and T4
Quantitative Experiments -Conclusion
T2
Result
T4
Result
Best ISP Point
© 2024 NEXTCHIP
30

31. T1 : Original Setting T5 : Edge Sharpness Off + Bright UpT6 : Edge Sharpness Off + Bright Up + NR Up
T8 : Edge Sharpness Off + Bright Down + NR UpT7 : Edge Sharpness Off + Bright Down
Qualitative Experiments –Evaluation Methods
Evaluation Methods
• Estimate the false detection rate
• Counting false positives (FP) for period in which false detection occurs in all tuning points
© 2024 NEXTCHIP
31

32. T1 : Original Setting T5 : Edge SharpnessOff + Bright UpT6 : Edge Sharpness Off + Bright Up + NR Up
T8 : Edge Sharpness Off + Bright Down + NR UpT7 : Edge Sharpness Off + Bright Down
• Additional 5 ISP settings for the same driving pathDaytime test
Qualitative Experiments –Best ISP for Object Detection
© 2024 NEXTCHIP
32

33. Daytime evaluation result
Qualitative Experiments –Best ISP for Object Detection
T8
: EDGE sharpness OFF + AE Down + NR Up
Best
© 2024 NEXTCHIP
33

34. T1 : Viewing Optimization Tuning T8: Edge Sharpness Off + AE Down + NR Up
Nighttime test • 2 ISP settings are applied for same driving path
Qualitative Experiments –Best ISP for Object Detection
© 2024 NEXTCHIP
34

35. Nighttime evaluation result
Sensing Optimization
Best
Qualitative Experiments –Best ISP for Object Detection
© 2024 NEXTCHIP
35

36. Quantitative Experiments –Conclusion
•Qualitatively, the detection rates are similar at all tuning points
• Datasets1 (Day time)
1. When noise level is high, reduces false detection rate
2. In daytime, brightness does not seem to have a significant effect on false detection
• Datasets2 (Night time)
1. T8 (Sensing) false detection rate is 0.1 better than T1 (viewing tuning)
2. At nighttime, when brightness level is low, reduced false detection rate
© 2024 NEXTCHIP
36

37. Future Works
The problem with current experiments
• Since the performance is evaluated only for specific points,
there are some limitations to estimate the tendency value for each tuning factor.
Further experiments
• We keepworking to analyze the trends while changing the AE (brightness), EDGE, and
the noise level in optimal ISP tuning.
© 2024 NEXTCHIP
37

38. Resources
•ChatGPT https://chatgpt.com/n
•Test by Nextchip Internal Standard of Image
Quantitative & Qualitative Test
2024 Embedded Vision Summit
●Booth#109
●Mr. Young-Jun Yoo
●[email protected]
© 2024 NEXTCHIP
38