Infrared simulation and processing on Nvidia platforms

Simulation and processing of Infrared images and video on Nvidia Platforms

1. Infrared simulation and processing On Nvidia Platforms

2. Neuronics Developing projects in Computer vision & AI 3D generation and simulation Nvidia Edge and server platforms and SDKs Defense / Military UAV / Autonomous vehicles Autonomous vassals robots Medical

3. Agenda Solve IR camera processing challenges

4. IR Camera chalanges Lack of Training data Resolution (Low) Physical connection Data variability between cameras Sensitive to compression Noise!

5. Camera connection Camera Link -> Capture card GMSL – Fiber ( for distanced camera) USB GIGE HDMI RAW T ransfer

6. Noise DSNU - Dark Signal Non-Uniformity ~1% bias PRNU - Photo-Response Non-Uniformity ~1% gain variability Thermal noise Sensor Gain Temp

7. Data problems Capture & Label Data – expensive / not possible Augmentation - Sensitive Just changing brightness like in RGB images wont work! Simulation 3D World simulation (Real, Blender, Omniverse, Unity) Thermal assignment (simple) Glitter simulation / Ray tracing Atmospheric simulation (Frequency, distance, angle)

8. Simulation – Basic Shared config

9. Sensor Simulation – Advance Temperature map Distance Map Glitter Map Per frame exposure Distance Map W ave Gliter to sensor out The simulation is not standalone since exposure is dynamic and changed by the processing pipe

10. Frame Processor Exposure estimation To sensor simulation

11. Target Acquisition

12. IR Recording and transfer We DON’T compress IR sensor output and pixel targets Store in raw format 8/10/12 bit!!! Sometimes we have problem i n streaming it Lossless JPEG, Lossless AV1 Process at the edge - > Jetsons!!! Stream Raw over fiber using NVIDIA Rivermax

13. Nvidia RiverMax Nvidia architecture for stream 10s-100s of GB direct to GPU Used to stream RAW 8/10/12bit video 4K Broadcasting quality

14. NVIDIA Video Analytics flow Sensors Actions Capture and Decode Pre- processing & Batching AI Inference Tracking Composition Business Rules and Analytics … … H.265 Less used for IR

15. Processing Pipe requirements Process stream/s of video (and audio/other data synchronously) Block based for each unit/function Easy connection between camera / socket / encoder / decoder / file / parser Ability to reuse and enhance existing modules (“Inheritance” + open source) Fix all the threads / buffers on its own Runs on Linux

16. Edge Processing Architecture DeepStream Nvidia Standard processing architecture – Not used Python CPP Processing Basic filtering using OpenCV/CUDA , RAPIDs Double Filter – DoG Inverse PSF Neural Networks Segmentation Binary Operations Tracking – MTT with Kalman/EKF/UKF

17. 10 Min questions time

18. Yossi Cohen Thank You