Levels of AI Agents: from Rules to Large Language Models

AI agents are defined as artificial entities to perceive the environment, make decisions and take actions. Inspired by the 6 levels of autonomous driving by SAE (Society of Automotive Engineers), the AI agents are also categorized based on utilities and strongness, as the following levels: L0—no AI, with tools (with perception) plus actions; L1—use rule-based AI; L2—let rule-based AI replaced by IL/RL-based AI, with additional reasoning & decision making; L3—apply LLM-based AI instead of IL/RL-based AI, additionally setting up memory & reflection; L4—based on L3, facilitating autonomous learning & generalization; L5—based on L4, appending personality (emotion + character) and collaborative behavior (multi-agents).

1. Levels of AI Agents
Yu Huang
Roboraction.AI

2. Foundation Models and Large Language Models
•Foundation models are pre-trainedon a surrogate task and then
adapted to the downstream task of interest via fine-tuning.
•LLMsare trained on text data, to enable understanding natural
language, via text generation & comprehension.
•The emergent capabilities of LLMs: Prompting, In context learning
(ICL), Chain of thoughts (CoT) and Instruction following.
•Efficient Parameter Fine Tuning (EPFT): LoRA
•Alignment with human preference: RLHF
•LLMs are possible penetration of AI to AGI.

3. What is an AI Agent?
•Agent is an entity, that is able to perceive its environment and
execute actions.
•The AI Agent is the entity exhibiting intelligent behavior and
possessing capabilities like autonomy, reactivity, pro-activeness, and
social interactions.

4. Embodied AI
•Embodied AI is designed for enabling agents to display AI, not only in
virtual (such as cyber space) but also physical world, crucial for
realizing AGI.
•Multi-modal Large Models (MLMs) and world models are prominent
features of embodied AI.
•Some basic components:
•Embodied perception (including navigation)
•Embodied interaction (QA and embodied grasping)
•Embodied simulation (world model and adaptation)
•Embodied agent (MLMs, task and action planning, embodied control)
•Visual-Language-Action (VLA) model is a MLM in Embodied AI.

5. Levels of AGI“Levels of AGI: Operationalizing Progress on the Path to AGI”, arXiv2311.02462, 2023

6. OpenAI’sAGI Levelshttps://medium.com/@a.sale/chatgpt-5-and-beyond-openais-
five-level-roadmap-to-agi-unveiled-be09db42ca27
July 2024

7. Capabilities of AI Agents

8. Andrew Ngo on AI Agent Workflows

9. Levels of AI Agents

10. Perception and Action

11. Reasoning&Decision making

12. Memory and Reflection

13. Autonomous Learning and Generalization

14. Personality and Collaboration

15. Conclusion
Mo#vated by 5 levels of • autonomous driving by SAE, levels of AI
agents are classified based on intelligence u#li#es and power;
Mostly current AI Agents’ level lies on L2• -L3;
Some AI agents are on research and even developed for L4 or L5, but •
honestly, the system performance is not sa#sfying.
Challenging issues come from AI brain in the agent plaIorm, i.e. LLM, •
like hallucina#on, explainability, performance-cost (computa#on and
memory) tradeoff, safety & security, copyright and privacy etc.
Agent related issues: role playing• , catastrophic forgeQng, misuse,
threats to human race, collec#ve intelligence in agent society etc.