“Using MIPI CSI to Interface with Multiple Cameras,” a Presentation from Meta

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/10/using-mipi-csi-to-interface-with-multiple-cameras-a-presentation-from-meta/

Karthick Kumaran Ayyalluseshagiri Viswanathan, Staff Software Engineer at Meta, presents the “Using MIPI CSI to Interface with Multiple Cameras” tutorial at the May 2024 Embedded Vision Summit.

As demand rises for vision capabilities in robotics, virtual/augmented reality, drones and automotive, there’s a growing need for systems to incorporate multiple cameras. However, the limited number of camera interfaces on SoCs poses a challenge. In this presentation, Viswanathan explains a variety of techniques using the MIPI CSI standard to overcome SoC interface limitations and connect multiple high-resolution cameras to systems effectively.

1. Using MIPI CSI to Interface
with Multiple Cameras
Karthick Kumaran Ayyalluseshagiri
Viswanathan
Staff Software Engineer
Meta Platforms Inc

2. •Camera Interfaces
•Quick Recap –MIPI
•SoC –Camera Architecture
•MIPI CSI D-PHY
•MIPI CSI C-PHY
•Virtual Channel (VC)/ Data Type (DT) Aggregation
•Multi-Drop
•Summary
Topics
2© 2024 Meta Platforms Inc

3. •Growing number of advanced use cases in robotics, VR/AR, drones, and
automotive, demand multiple cameras
•Limited number of camera interfaces limits the number of cameras connected
to the SOC
•Will cover various techniques to overcome the limitations
Problem Statement
3© 2024 Meta Platforms Inc

4. •Parallel interface
•USB
•MIPI
I will be covering only the MIPI interface in this talk
Camera Interfaces
4© 2024 Meta Platforms Inc

5. •CSI for camera
•A-PHY (for automotive)
•C-PHY (trio)
•D-PHY (Differential phase)
•DSI for display
MIPI CSI-2
5© 2024 Meta Platforms Inc

6. •Two types of MIPI packets
•Short packet
•Long packet
•Short packet
•Packet header (PH)
•Long packet
•Packet header
•8-bit data identifier
•2-bit VC + 6-bit DT
•16-bit word count
•8-bit ECC
•Payload –Pixel data
•Packet footer
Quick Recap of MIPI CSI-2 Frame Format
6© 2024 Meta Platforms Inc

7. •Typically, SOCs have several number
of hardware blocks such as
•MIPI CSI PHY interfaces
•MIPI CSI decoders
•ISPs
•DMAs
•Number of hardware blocks vary
between SOCs
Architecture
7© 2024 Meta Platforms Inc

8. •CSI PHY used to interface
•Camera sensors
•Depth sensors
•DTOF / ITOF etc.,
•CSI decoder to decode the MIPI
protocol
•DMA to transfer the pixels to DDR
•ISP to perform image processing
Architecture
8© 2024 Meta Platforms Inc

9. •There are a fixed number of CSI PHY interfaces, CSI decoders and ISPs available on
embedded SoCs
•Number of CSI Interfaces and number of CSI decoders need not be always the same
•Similarly, number of CSI decoders and number of ISPs need not be always the same
•Color cameras need ISP to do demosaicand image quality enhancements
•Monochrome cameras –for computer vision / deep learning algorithms
•May not need any ISP processing
•Cross bar between CSI PHY interface and CSI decoder helps to map them arbitrarily
CSI PHY, CSI Decoder & ISP
9© 2024 Meta Platforms Inc

10. •ISP to perform image processing such as
•Demosaic, color space conversion (CSC), black level subtraction (BLS), gamma, scale, crop and
other IQ enhancements
•Inline ISP
•Pixels are passed to the ISP directly from the CSI decoder
•Less latency
•Less power consumption as it avoids round trip to the DDR
•Offline ISP
•Acts like M2M (Memory-to-Memory) device
•Higher latency than inline ISP
•Supports multiple instances
ISP
10© 2024 Meta Platforms Inc

11. •Up to 4 data lanes can be connected
to a single CSI PHY in D-PHY mode +
1 clock lane
•High resolution cameras use all the
4 data lanes
•Mid to low resolution cameras use
either 2 lanes or 1 lane
MIPI CSI D-PHY
11© 2024 Meta Platforms Inc

12. •Up to 3-trios can be connected to a
single CSI PHY in C-PHY mode
•High resolution, high fps cameras
use all the trios
•Lower resolution (less bandwidth)
cameras uses either 2 trios or single
trio
•Faster than D-PHY
MIPI CSI C-PHY
12© 2024 Meta Platforms Inc

13. •Two low resolution cameras can be
connected to a single CSI PHY
•2-lanes + 1-lane configuration
•1-lane + 1-lane configuration
•Dedicated CSI decoder required for each
camera to operate concurrently
•With one CSI decoder
•One of the camera can be enabled
•Allows switching cameras
CSI Split
13© 2024 Meta Platforms Inc

14. •Up to 3 low resolution cameras can be
connected to a single CSI PHY
•2-trio + 1-trio
•1-trio + 1-trio + 1-trio
•1-trio + 1-trio
•Dedicated CSI decoder required for
eachcamera to operate concurrently
•With one CSI decoder
•One of the camera can be enabled
•Allows switching cameras
CSI Split (continued)
14© 2024 Meta Platforms Inc

15. •Multiple cameras, up to 16, can use the
single CSI PHY interface
•Aggregator is needed
•Uses TDM (Time Division Multiplexing)
•Aggregator has to manage the TDM
•Uses MIPI virtual channels (VC)
•Single CSI decoder can decode MIPI
packets from all the cameras as they are
time multiplexed
Virtual Channel / Data Type Aggregation
15© 2024 Meta Platforms Inc

16. Multi-drop
16© 2024 Meta Platforms Inc
•Multiple sensors (usually 2-to-4)can be
connected to a single MIPI D-PHY port
•Limits the frame rate of each sensor
based on the data rate and the number
of lanes
•Same virtual channel for all the sensors
•Unique virtual channel for each sensor
•Works only with global shutter cameras

17. Multi-drop (continued)
17© 2024 Meta Platforms Inc
•Some cameras supports a feature
named “Multi-drop” which enables us
to connect more than 1 camera without
an aggregator
•Uses different exposure time / readout
time
•Can use different VCs for each cameras
•Can use same VCs for all the cameras
(super frame)

18. •MIPI CSI protocol provides lot of flexibility to connect multiple cameras to the CSI PHY
interfaces
•MIPI CSI protocol provides a mechanism to multiplex multiple camera sensor data in
time division multiplex mechanism
•Helps reduce the footprint of the SOC
Summary
18© 2024 Meta Platforms Inc

19. •MIPI CSI-2 Specification -https://www.mipi.org/specifications/csi-2
Resources
19© 2024 Meta Platforms Inc

20. Thank You
20© 2024 Meta Platforms Inc