Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): paving the way to 6G

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI)

1. tii.aeDSRC
Empowering NextGen Mobility
via Large Action Model Infrastructure (LAMI):
The Magic behind the curt(AI)n
Dr. Thierry Lestable,
Ph.D, Dipl-lng.
Chief Researcher, Digital Science Research Center
(DSRC)
Telecom & Cybersecurity Units

2. 2TII – Technology Innovation Institute
Towards Smart, Sustainable, Autonomous & Electrified INFRASTRUCTURE
Smart Cities depend on Intelligent Infrastructure !
COLLECTIVE INTELLIGENCE
Requires Very High Data rates, short
latency, Security, Sustainability
x10
X50
(Tbps)
x2
/10
Native AI
Backbone for advanced & autonomous city services (e.g. ITS), carbon-free
economies, closes Technology Divide, improves Productivity…
6G will natively enable the Collective/Collaborative Intelligence
6G is a Business Game Changer, thanks to Native AI, Open RAN
Trustworthy AI
1 2
3
4

3. TII – Technology Innovation Institute
US DoT :: ARPA-I
Time for action!...
$ 6 Million reward !

4. 4TII – Technology Innovation Institute
Smart Infrastructure :: Collective Intelligence
Design, Development and Roll-Out
Digital Twin

5. TII – Technology Innovation Institute
GenAI for trajectory prediction
Anticipating VRU + Drivers on the Road
With Inference time
improvements,
through Edge(Gen)AI,
real-time prediction of
trajectories will be
feasible.
Merging into crowded traffic
Merging into high-speed traffic Multiple Lane changing
Yielding Jaywalker
Multimodal Transformer
Social Relation
VRU: Vulnerable Road Users

6. TII – Technology Innovation InstituteDSRC, NextGen Telecoms & Security Innovation Summit, 25-26 Oct, Abu Dhabi, 2023. 6
What we are doing
LLM
Auto
nomous
Agent
Planning
Reasoning
Intent-driven
Autonomous Network
Intelligent Collective
Vehicle
System II AI
Semantic
Communication
Emergent Protocol
System II Communication
Collective Intelligence
+ => + =>
AutoGPT
CAMEL
GenAgent
GPT-4
Llama-2
Falcon
Effectiveness level
Semantic level
Logical inference
Path of thought
Autonomous, self-governed,
decentralized
Network for GenAI
➢Building effective communication and computing framework for LLM in network
➢Bringing collective intelligence through multi-agent interaction and collaboration
Telecom
Vertical
Towards Large Action Models (LAM)

7. 7TII – Technology Innovation Institute
Beyond LLMs…Towards LAMs
DSRC
Leveraging LLM/GenAI know-how, bringing LAM to Life…

8. TII – Technology Innovation InstituteDSRC, NextGen Telecoms & Security Innovation Summit, 25-26 Oct, Abu Dhabi, 2023. 8
Intent-driven Autonomous Network
Plan by rules => Plan by reasoning Standard Protocol => Emergent Protocol Automation => Autonomy
……
➢Limitations: Telecom domain knowledge and interfacing with Telecom platform Perception
LLM
Perceptor
image
text
SMF, UMF,…
CVX,…
KG
Actor
Perception
Memory
Knowledge
Base
Mechanism
RF
Vector DB
…
Message
Multi-Modal
Fusion
Feedback
Retrieval
Prune
Retrieval
LogInjection
Decision

9. TII – Technology Innovation InstituteDSRC, NextGen Telecoms & Security Innovation Summit, 25-26 Oct, Abu Dhabi, 2023. 9
Collective Intelligent ITS
Large Action Model (LAM)
Large Action Model (LAM)

10. TII – Technology Innovation Institute
LAM on Wheels & Infra….towards L5 ITS (1/2)
Paving the Road to Multi-Agents Collaborative LLMs…Boosting ITS End-to-End infrastructure
Collaboration with ARRC

11. TII – Technology Innovation Institute
LAM on Wheels & Infra….towards L5 ITS (2/2)
Paving the Road to Multi-Agents Collaborative LAMs…Boosting ITS End-to-End infrastructure
Nuro Driver - LLM case
As Nuro’s bot approaches the intersection, the model asks
autonomy stack: “What are you doing and why?”
ITS, and ADAS/AD stack solutions
will embrace LAM Models to better
capture surroundings, and forecast
behaviors & trajectories of Road
agents, including VRUs.
Ghost Autonomy - LLM case Wayve - LINGO - LLM case
(Raised +1.05 B USD)
“Ghost Autonomy Announces Investment from OpenAI Startup Fund
to Bring Multi-Modal LLMs to Autonomous Driving.
Ghost Autonomy, a pioneer in scalable autonomy software for
consumer cars, today announced a $5 million investment.”
Ambarella engineer Pai Wang gives EE Times an exclusive, live demo of
Llama2-13B running on the company’s CV3-AD, a 50-W autonomous
driving chip.
Ambarella Demos
LLM Inference on Autonomous Driving Chip

12. 12TII – Technology Innovation Institute
A new Era of AI Devices…
DSRC
LLM OS
LLM OS
LAM OS
Why Isn’t this just an app?”

13. 13TII – Technology Innovation Institute
Large Action Model Infrastructure: LAMI RSU
DSRC
Next steps == Actions
RSU: Road Side Unit

14. TII – Technology Innovation InstituteDSRC, NextGen Telecoms & Security Innovation Summit, 25-26 Oct, Abu Dhabi, 2023. 14
What we are (also) doing
Visual
Generation
Radio
Generation
Localization
Tracking
Detection
Recognition
Visualization
AIGC
Beamforming
Channel Estimation
Resource Allocation
Mobility
Management
Radio Environment
Map
Network Planning
RF
Signal
Image
Point-
cloud
Radar
signal
Cross
-
Modality Embedding
Unified Generative Model
=>
=>
Fine-tunePre-train
Shared Abstraction Space
PHY MAC NET
Static Object Mobile Object Environment
➢Building a foundation model with generative AI on multi-modal wireless data
➢Generalized to various applications and scenarios in wireless communications
GenAI for Network
Sensing
Communications
=>
=>
=>

15. Reconfigurable Intelligent Surfaces (RIS @28/3.5GHz Experiments)
An Appealing Energy Efficient Coverage Extender for the 5G Networks
3 . 5 G H Z R I S
AI Multi-Modal Radio Beam
Prediction.
LAM I will play Key role to
optimize NW.
The RIS successfully created a virtual Line of Sight (LoS) connection
with a data rate close to 1Gbps, equivalent to the LoS data rate.
Blockage/obstruction
RIS can be a good viable solution for Relay/Coverage Extension
1
2
3
15
Perception Layers
Multi-Modal inputs
Beam Steering Action
LAM MIMO

16. TII – Technology Innovation Institute 16
SecureFalcon – Safe Software (SW) Code Generation
Detect Vulnerabilities, Explainability of the vulnerability's nature, and Repairing Vulnerable Code
Digital Security Unit
92% Accuracy for Detection on 10 Top
Vulnerabilities (CWE)
CWE-119: Improper Restriction
CWE-120: Buffer Copy
CWE-476: NULL Pointer Dereference ,
CWE-122: Heap-based Buffer Overflow,
CWE-190 : Integer Overflow or Wraparound,
CWE-121 : Stack-based Buffer Overflow ,
CWE-78 : OS Command Injection,
CWE-787 : Out-of-bounds Write ,
CWE-20 : Improper Input Validation ,
CWE-762 : Mismatched Memory Management
Avoiding SW Code being generated with Vulnerabilities – a Cyber companion then Self-Healing platform

17. TII – Technology Innovation Institute 17
SecurityBERT : Network Intrusion Detection System (IDS)
An embedded tool to detect network attacks on connected devices using LLM
Detection:
98.2%
14 attacks
compact size: 16.7 MB
Outperform
traditional ML/DL
well suited for embedded devices == Being Deployed throughout Networks
Promising for Zero-Trust…
Fast (Inference Time)
1
2
3

18. 6G (IMT-2030) Usage Scenarios benefits…
Particularly interesting for ITS Roll-out:
-Additional sensing capabilities
-Positioning accuracy
-Distributing AI Load amongst different players/infrastructure
[Mobile operator, Grid, Transport]
Cyber Reasoning will be critical:
-Number of sensors (Massive MTC)
- Higher data rates (more complex attacks)
-FOTA/SOTA massive
-Trustworthy AI
-Post-Quantum Resistant

19. 19TII – Technology Innovation Institute
6G Momentum in UAE
DSRC
TDRA released first 6G Roadmap document.
National 6G committee being set-up.
6G Investment plan to be built soon.

20. TII – Technology Innovation Institute
Let’s make a Difference for a Better World!