“Using Synthetic Data to Train Computer Vision Models,” a Presentation from Geisel Software

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/09/using-synthetic-data-to-train-computer-vision-models-a-presentation-from-geisel-software/

Brian Geisel, CEO of Geisel Software, presents the “Using Synthetic Data to Train Computer Vision Models” tutorial at the May 2024 Embedded Vision Summit.

Developers of machine-learning based computer vision applications often face difficulties obtaining sufficient data for training and evaluating models. In this talk, Geisel explores the use of synthetic data techniques to overcome these challenges. He explains the “simulation-to-reality” gap and the challenges it poses for realistic synthetic data generation.

Using an automated Mars exploration vehicle application, Geisel shares how his company generates synthetic data using the Unity simulation environment, and demonstrates how synthetic data can be effective—while also highlighting its limitations. He also touches on the potential for few-shot learning techniques to reduce the need for training data.

1. Using Synthetic Data to Train
Computer Vision Models
Brian Geisel, CEO
Geisel Software

2. Open World Simulation
© 2024 Geisel Software
2

3. Synthetic Data: When It’s Advisable and
When It’s Mandatory
© 2024 Geisel Software
3
Advisable
•Efficient and cost-effective way to train ML
•Safe testing of edge cases
Mandatory
•You have no access to the environment
•Data scarcity
•Generate diverse conditions

4. © 2024 Geisel Software
►Reality Gap
►Quality & Accuracy
►Overfitting
►Ethical & Legal Considerations
►Technical Complexity
►Validation Challenges
The Limitations of Synthetic Data
4

5. What is the Sim2Real Gap?
The discrepancy between simulated environments and real-world conditions when
developing and testing robots, algorithms, or machine learning models
Factors contributing to this gap:
►the fidelity of the physical world
►the complexity of real-world interactions
►the unpredictability of real environments
versus their simplified virtual counterparts
Sim2Real Gap Explained
© 2024 Geisel Software
5

6. © 2024 Geisel Software
Creating Synthetic Martian Environments
6

7. The Sim2Real gap is particularly crucial when it comes to Mars exploration:
►Unpredictable and Extreme Conditions
►Limited Training Set
►Limited Testing Opportunities
►Unable to Retrain While Deployed
►High Mission Stakes and Costs
►Limited Physical Access
The Mars Sim2Real Gap
© 2024 Geisel Software
7

8. Photorealistic Simulation Tools & Technologies
© 2024 Geisel Software
3D Modeling/Rendering
Software
Blender
QuixelMegascans
Maya
Unity
ArcGIS
QGIS
High Fidelity Rendering
Engine
Unity
Robotic Simulation
Frameworks
Gazebo
ROS
V-REP
Connection of Simulation
Environments
Unity/ROS Bridge
Generates Training Images
Machine Learning and
AI Platforms
TensorFlow
PyTorch
Drives Robotic Behavior
8
C
V

9. Creating Training Data from Simulated Environment
© 2024 Geisel Software
9
Footage from Mars Scanned Earth Objects CreateSimulation

10. Photorealistic Simulation
Demo –need file
Adjust Atmosphere Add Noise Shift Color
© 2024 Geisel Software

11. Training the Model with Synthetic Data
© 2024 Geisel Software
11
Original Object Instance Labeling Segmentation of
Objects of Interest

12. GIRAF RealSyncDigital Twinning Platform
© 2024 Geisel Software
12
►Integrates digital twins of robotic
systems
►Utilizes real-time data from physical
systems
►Highly adaptable and requiring minimal
specific system prerequisites
►Integrates seamlessly with existing
robotics systems, both physical and
simulated

13. Few-Shot Learning
© 2024 Geisel Software
13

14. Testing the Photorealism of the Simulation
© 2024 Geisel Software
14
Simulated Environment
Photorealistic Simulated Environment
Actual Atacama Desert

15. Resources
"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks"
by Chelsea Finn, Pieter Abbeel, and Sergey Levine
https://arxiv.org/abs/1703.03400
"Improved Synthetic Data for Deep Learning" by Lukas Tuggener, Ismail
Elezi, Jürgen Schmidhuber, ThiloStadelmann
https://arxiv.org/abs/2001.06630
Evaluation of Techniques for Sim2Real Reinforcement Learning
https://www.researchgate.net/publication/370625535_Evaluation_of
_Techniques_for_Sim2Real_Reinforcement_Learning
15
© 2024 Geisel Software

16. © 2024 Geisel Software
Thanks: Let’s Connect!
Brian Geisel
https://geisel.software
[email protected]
[phone]
16
LinkedIn