EIS-Webinar-Agent-Approaches-2024-08-21.pdf

There is significant excitement about Large Language Models, but the hype continues to outpace the reality. The latest entry into the AI lexicon that is getting increasing attention is that of agent-based approaches where autonomous agents operate and make decisions without human intervention. While this framework has promise in the future, and some thought leaders project billions of agents operating for organizations and individuals, there are some challenges that current approaches need to overcome before widespread adoption of more ambitious visions of highly functional, safe, reliable “agents for all” can be realized.

We will cover topics including:
What kinds of processes can LLMs automate?
What is the difference between Chatbots, Assistants and Agents?
Templated Prompts: Incorporating additional context.
What is an agent-based approach?
How does this differ from typical LLM powered applications?
How can agent based, and non-agent-based approaches be used for data remediation?
AI governance, and the role of ethical guidelines

1. www.earley.com
WEBINAR WEBINAR
Agent Based LLM Applications: Separating the
Hype from Practical Applications
SETH EARLEY
CEO & FOUNDER
EARLEY INFORMATION SCIENCE
Media Sponsor

2. www.earley.com
Today’s Speakers
[email protected]
https://www.linkedin.com/in/sethearley/
Seth Earley
Founder & CEO
Earley Information Science
Sanjay Mehta
Principal Solution Architect
Earley Information Science
[email protected]
https://www.linkedin.com/in/sanjaymehta/
Dominique Legault
Founder
ReliableGenius
Alexander Kline
Founding Partner
Arcana Concept
[email protected]
https://www.linkedin.com/in/alexander-kline-
futurist-/
[email protected]
https://www.linkedin.com/in/dominique-
legault-84824199//

3. www.earley.com
Before We Get Started
WE ARE RECORDING
SESSION WILL BE
50 MINUTES PLUS
10 MINUTES FOR
Q&A
YOUR INPUT IS
VALUED
Link to recording & slides
will be sent by email after
the webinar
Use the Q&A box to
submit questions
Participate in the polls
during the webinar
Feedback survey afterward
(~1.5 minutes)
Thank you to our media partners : VKTR
3

4. www.earley.com
About Earley Information Science
4
Proven methodologies to organize information and data.
SELL MORE
PRODUCT
SERVICE
CUSTOMERS
EFFICIENTLY
INNOVATE
FASTER
1994
YEAR FOUNDED.
Boston
HEADQUARTERED.
50+
SPECIALISTS & GROWING.

5. www.earley.com
Agenda
5
Agents, Assistants and Chatbots
LLMs and Process Automation
The role of Retrieval Augmented Generation (RAG)
Task agent –approach to iterative processing of data records
LLM Agent Engineering
Orchestration agent– “conductor” of agents
Contextualization of agents – data retrieval, system, process, function,
organization context, user preferences
Guidelines and guardrails – traceability, privacy, governance, IP protection
(respecting communication and role context)
Getting started

6. www.earley.com
Poll
6
1.Not on the radar
2.Planning stages for Agent based AI
3.Controlled experiments using Agent based AI
4.Agent based AI usage is currently banned
5.Implemented PoC’s (internal or externally facing)
6.Agent based AI applications deployed
7.None of the above
Where are you on your Agentic AI journey?

7. www.earley.com
Agents, Assistants and Chatbots
7
AI agentsact autonomously towards solving broad challenges. They can take
actions without or with limited human intervention.
AI assistantsserve a supporting role for specific human needs. Can handle a
narrow set of objectives without autonomy (decisions require human approval).
Chatbots handle simpler tasks – often information retrieval – in a
conversational format.

8. www.earley.com
LLMs and Process Automation
8
Customer service, content creation, data analysis, personal assistant
functions
Requirements:
•Do you understand the process in detail? (you can’t automate what you
don’t understand)
•Are the correct data sources available?
•Is data of sufficient quality?
•Is it structured for retrieval?

9. www.earley.com
Foundational Elements of an Agentic Solution
Capability
Tasks & Applications, Definition & Goals
(service assistant, sales assistant, operational task assistant...)
Governance
Rules, Guardrails, Evaluation
(security, restrictions, performance...)
Platform
Infrastructure, AI/LLM Model, Sources & Systems
(hosting, components, data, applications, information architecture)
Configuration
Interface, Prompting, Execution, Memory
(human-agent, agent-agent, ux, search, chat, navigation, data, analytics...)

10. www.earley.com
Agent Template Example
parameters
metadata
System Prompt 01
Tools/Functions03




Profiles
Identity / Security
Memory
Personalization


User04
Data 02
Information
Architecture
guardrails


Parsing
Indexing
Retrievers
Embeddings
Inputs ->
Outputs <-

11. Copyright © 2024 Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.com
Complex Analysis/Action
Product Support
Product Configuration
Complex Workflows
Domain Complexity
Transaction Support Knowledge Retrieval & Action
Information/
status inquiries/
order processing
Action Complexity
11
Action Complexity versus Domain Complexity
Multi Agent Workflows
Knowledge Bots or Configuration Agents
Transaction Bots or Agents

12. www.earley.com
Why LLM Agents?
12
•Improving the functions and
capabilities of LLMs
•Overcome intrinsic limitations of
LLMs around currency of
information, hallucinations,
weaknesses in non language tasks
(mathematics for example)
•Enable LLM to call a function that
executes an action
•Increasing focus on enabling LLMs to
learn to call API’s
https://arxiv.org/pdf/2302.04761

13. Copyright © 2024 Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.com 13
Orchestration
“conductor” of agents

14. Arcana Concept
•45 people from all different
backgrounds, skillsets and geographic
locations
•ideate, discuss world changing
inventions, businesses cases, prototype
& develop
•hardware + software + resources +
growth
lab : studio

15. Acceleration
Ideation, Research, Communication, Coordination, Presentation, Preservation

16. The Recipe
•Expert on Material Science, Lean startup
•Personality is a combination of the Merlin
Archetype and absent minded professor
•Speaks like you would expect a wizard to
speak
•Moves conversations forward
•Tracks concurrent projects
•Coordinates with his owl to fetch external
information
Purpose, Personality, Ability
The many iterations of Fizban

17. Who is Fizban?
•Tell us about his personality
•What does he do for you?
•What are the challenges and how
are they addressed?
Open AI GPT 4-Omni
Slack API integration
Custom Memory System
Search

18. Orchestration Agent
•Hands off tasks to specialized agents
•Manages and coordinates responses and
outputs
•Tracks multiple threads and conversations
•Manages short- and long-term memory
while respecting privacy and privileged
conversations
•Provides references and audit trails
through Retrieval Augmented Generation
Note: AI lies

19. www.earley.com 19
Agent LLM Engineering
Lessons Learned Developing AI Technology
Dominique Legault Aug 21
st
, 2024

20. Finding the balance between context
size, cost and efficiency
Context Relevance
1. Small Context Size & Low Accuracy (Bottom Left Quadrant)
•Insufficient Context: The LLM lacks necessary details, leading to vague or generic
responses. The cost is low, but so is accuracy.
•Example: A chatbot with minimal context providing incomplete or inaccurate answers.
2 .Small Context Size & High Accuracy (Top Left Quadrant)
•Optimized Small Context: With highly relevant but minimal context, the LLM provides
accurate responses efficiently. The cost remains low, and accuracy is high.
•Example: A well-tuned model with just enough context to deliver precise answers.
3. Large Context Size & Low Accuracy (Bottom Right Quadrant)
•Information Overload: The LLM is given too much context, including irrelevant
information. Accuracy drops as the model struggles to focus on relevant details. Cost is
high, but accuracy is low.
•Example: An AI overwhelmed by extensive data, leading to slower and less accurate
responses.
4. Large Context Size & High Accuracy (Top Right Quadrant)
•Optimally Managed Large Context: The LLM efficiently processes a large amount of
relevant information, providing highly accurate results. Cost is high, but so is accuracy.
•Example: A system with advanced context management techniques like pruning
irrelevant data to maintain accuracy.

21. Choosing the right workflow for your problemWorkflow Design
Single Shot Workflows:
•Definition: A single agent handles the entire task from start to finish, suitable for simple tasks.
•Advantages:
◦Simplicity: Easier to design and implement with minimal complexity.
◦Speed: Quick task completion with no need for agent coordination.
◦Consistency: More reliable outcomes since one model processes all information.
•Use Cases: Ideal for simple queries like answering specific questions or generating short text, and straightforward
processes like report generation.
Multi Agentic Workflows:
•Definition: Multiple agents collaborate, each specializing in different parts of a task.
•Advantages:
◦Specialization: Agents optimize for specific subtasks, increasing efficiency.
◦Scalability: Handles complex, multi-step tasks more effectively.
◦Flexibility: Adapts to dynamic tasks, activating different agents as needed.
•Use Cases: Best for complex problem solving, like multi-stage research or content creation, and dynamic
environments where tasks evolve, such as real-time data analysis.
Choosing the Right Workflow:
•Task Complexity: Single Shot Workflows are sufficient for simple tasks, while Multi Agentic Workflows excel in
complex scenarios.
•Resource Allocation: Single Shot Workflows use fewer resources; Multi Agentic Workflows offer greater flexibility
and scalability, ideal for complex environments.
•Outcome Reliability: Single Shot Workflows are consistent, while Multi Agentic Workflows provide tailored solutions
for more dynamic needs.
Impact on Business:
•Efficiency: Workflow choice affects business efficiency, particularly in automation and customer service.
•Scalability: Multi Agentic Workflows are better suited for scaling complex operations.

22. Decision Making Autonomy, Objective Alignment, and Impact on Workflow Efficiency
Agent Perspective
Autonomous Agents:
•Definition:Operate independently, making decisions based on pre-defined goals without human intervention.
•Characteristics:
◦Self-Sufficient: Function without continuous human oversight.
◦Goal-Oriented: Focus on achieving specific outcomes like process optimization.
◦Adaptable: Learn and adapt to changing conditions.
•Use Cases:Process automation, exploration, and research.
•Challenges:Trust and control, ethical considerations.
Personal Assistants:
•Definition:AI agents that assist a specific human, enhancing productivity and task management.
•Characteristics:
◦User-Centric: Tailor actions to individual preferences.
◦Context-Aware: Use personal data to provide relevant assistance.
◦Proactive/Reactive: Suggest actions or respond to commands.
•Use Cases:Executive support, personalized customer service.
•Challenges:Privacy, security, accuracy, and relevance.
Comparative Analysis:
•Decision-Making Autonomy:
◦Autonomous Agents: High autonomy, goal-driven.
◦Personal Assistants: Human-dependent, supports decision-making.
•Impact on Workflow:
◦Autonomous Agents: Automate processes, reduce human intervention.
◦Personal Assistants: Enhance human tasks, improve productivity.

23. Balancing Accessibility with Data Protection
Securing AI Agents: Public vs Private
Publicly Accessible Agents
•Definition:AI agents open to the public via websites, chatbots, or APIs.
•Key Traits:
◦Wide Accessibility:Designed for public interaction.
◦Limited Data Access:To mitigate risks, they handle non-sensitive information.
•Security Concerns:
◦Data Exposure:Potential leakage of sensitive information.
◦External Threats:Vulnerable to attacks like phishing.
◦Compliance Risks:Must adhere to privacy regulations like GDPR, CCPA.
•Best Practices:
◦Information Filtering:Ensure only public data is accessible.
◦Access Controls:Restrict access to sensitive data.
◦Continuous Monitoring:Regular checks for security breaches.
Internally Accessible Agents
•Definition:AI agents used exclusively within an organization.
•Key Traits:
◦Restricted Access:Available only to authorized personnel.
◦Deep Context Integration:Can access sensitive internal data for precise tasks.
•Security Concerns:
◦Internal Data Handling:Breaches could lead to severe consequences.
◦Insider Threats:Risks of misuse by internal staff.
◦Data Retention:Prevent unauthorized data retention.
•Best Practices:
◦Encryption & Logs:Secure data and monitor access.
◦RBAC:Limit data access based on roles.
◦Regular Audits:Ensure compliance with security policies.
Comparative Analysis:
•Risk Exposure:Public agents face external threats, while internal agents are prone to insider risks.
•Information Sensitivity:Public agents should limit data access; internal agents need strong protection for sensitive data.
•Deployment Strategy:Public agents require external safeguards, while internal agents need a robust internal security framework.
Organizational Impact:
•Data Governance:Clear policies for handling sensitive data.
•Incident Response:Quick action plans for breaches.
•Training & Awareness:Regular employee education on AI security.

24. Triggering Mechanisms and Action Execution
Optimizing LLM Agent Workflows
Triggering LLM Workflows:
•Manual:User commands initiate specific workflows (e.g., generating
reports).
•Automated:Event-based triggers like incoming emails or data changes.
•Scheduled:Time-based tasks (e.g., daily reports).
•Conditional:Rule-based actions triggered by predefined conditions.
Executing Actions:
•Text Generation:Creating content like emails, reports, or posts.
•Data Processing:Analyzing data to provide insights and
recommendations.
•Task Automation:Executing commands, updating systems, or sending
emails.
•Workflow Orchestration:Performing complex, multi-step processes.
Business Implications:
•Efficiency:Reduces manual tasks and boosts productivity.
•Scalability:Handles increased workloads effortlessly.
•Customization:Tailors workflows to business needs.
•Integration:Aligns with existing systems for seamless operations.

25. The role of Transparency and Debugging in AI-Driven Processes
Enhancing Trust and Efficiency in LLM Workflows`
1. Transparency in Workflows
•Building Trust: Transparency fosters user trust by clarifying LLM decision-making.
•Accountability: Transparent processes ensure errors and biases are addressed.
•Credibility: Referencing sources builds credibility and informed decision-making.
•Explainability: Clear reasoning enhances understanding in complex scenarios.
2. Debugging Multi-Agent Systems
•Challenges: Multi-agent systems can be complex, leading to issues like
miscommunication.
•Tool Frameworks: Tools like LangSmithoffer workflow visualization and step-by-
step debugging.
•Example: Tracing incomplete responses in customer inquiries.
3. Gaining Insight into LLM Behavior
•Performance Metrics: Monitor response time, accuracy, and relevance for fine-
tuning.
•Behavioral Analysis: Debugging tools reveal how LLMs interpret inputs and
outputs.
•Continuous Improvement: Feedback loops and regular analysis enhance LLM
alignment with user needs.
4. Business Impact
•Reliability & Efficiency: Transparent, well-debugged workflows reduce errors and
improve system performance.
•User Confidence: Regular monitoring and debugging increase user trust and
adoption of AI solutions.

26. www.earley.com 26
Task Agent
Data Remediation

27. Copyright © 2024Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.comwww.earley.com
Generation vs Retrieval
27
User query
Process query
using LLM to
understand user
intent
Generate
Retrieve
Process response
using LLM to provide
conversational
format
LLMs used to process query
and present results
Response
Proprietary information
-No exposure of IP
-Corporate knowledge
-Auditability
Publicly available
information
-Prone to hallucination
-Exposure of IP
-Missing audit trail

28. Copyright © 2024Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.comwww.earley.com
28Source: https://arxiv.org/abs/2312.10997
Retrieval Augmented Generation has Evolved

29. © 2024 Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.com
Source Data to be Enriched
Original Product Name
Original Description
Attribute Names
Original
Catalog Context
SCHEMA/
MDM/PIM/ERP
Related Artifacts
Web Pages/Content
Digital Assets (images,
diagrams)
Existing Search Index
Product Sheets,
Manuals
Knowledge Base
Target Audiences /
Segments
Feed, API, Crawl, Direct
Merge by key – series,
category or attribute code
Unified Document Repository
Rules & Governance
Standards Bodies (ISO,
TC, SC)
Restrictions
Style Guide
Branding
Examples (Positive &
Negative)
Existing Search Rules
3rd Party Intelligence
Google Knowledge
API & Graph
Competitor Websites &
Search Results
Large Language
Models
EIS MRO Knowledge
Graph
User context
Signals / Telemetry
Search Analytics
Clickstream Analytics
Transactional
Analytics
Performance Metrics
KPI's
Audience / Segment /
Profile
Prepare/process:
normalize, classify, tag
Generate Doc
Embeddings
Create Graph Index
Generate Contextual Embeddings
(Application, Audience, Behavioral)
Generate Query / Phrase /
Q&A Embeddings
Generate Industry &
Organization Specific
Ontological Embeddings
Create Vector Index
Generate LLM Context and
Metadata

30. Copyright © 2024Earley Information Science, Inc. All Rights Reserved.
Sample Output Data
30
Category Code: "E3107000000“
Category Name: "Jumper Bars“
Brand Code: “ACM“
Brand Name: “ACME“
Series Code: 110400166560
Series Name Original: "Dedicated Short Bar“
New Name: “ACME Shorting Bar for Electrical Terminal Blocks“
New Description: "A dedicated shorting bar designed for use with
electrical terminal blocks to create a secure electrical connection and
prevent short circuits. Suitable for automotive, manufacturing, and
automation applications.“
Keywords: "shorting bar", "electrical terminal blocks", "short circuit
prevention", "automotive", "manufacturing", "automation“
Relevant Categories: "Electrical Components", "Terminal Blocks“
Reasoning: "The new name and description provide a clear
understanding of the product's purpose and applications, while the
keywords and categories ensure it can be easily found by the target
audience.”
Original New
•New names, descriptions and normalized attribute values will draw from both internal (ACME) sources
and publicly available data
•Output results will conform to ACME style guidelines and industry standards
•Enriched SEO information & metadata (e.g. reasoning) will be provided all output records

31. www.earley.com 31
Guidelines and Guardrails

32. © 2024 Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.com
Guidelines and Guardrails
32
How do you define “reasoning ability”?
What is control flow and how is it managed?
Which agent reports to other agents? Is there a hierarchy?
How are agents making “decisions” and how are they being enabled?
What are the restrictions on what agents in the wild can do?
What are the governance controls and considerations?
How do enterprises need to think about agent automations and restrictions?

33. www.earley.com
Poll
33
1.They are not aware of the benefit/risk profile
2.They are aware but do not have the appetite for agent-
based approaches
3.They are aware and have appetite for agent based
approaches
4.The organization is already pursuing these approaches
5.Something else (let us know in comments)
What is the stance of your leadership in terms of agent based LLM
approaches?

34. www.earley.com
Getting Started
34
Consider high value processes that typically require human intervention
Map end to end data flows
Establish baselines for processes to measure impact of remediation
Consider guardrails for agent functionality

35. Copyright © 2024Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.comwww.earley.com
Requirements Discovery Session for Agent Based Approaches
The AI Agent Requirements sprint will quickly identify how the organization can deploy LLM Agents to
address issues and challenges that arise from LLM deployments that seek to improve efficiencies and
effectiveness of organizational processes.
The sprint consists of the following:
•Stakeholder interviews
•Education and alignment
•Current state maturity
•Review of systems and tools
•LLM Agent Proof of Value (PoV) plan

36. Copyright © 2023Earley Information Science, Inc. All Rights Reserved.
AI Agent Working Session Topics
Topic Overview Goals Questions to Address
Key Concepts and
Success Criteria
AI and LLM deployment considerations
including current state processes and
technology ecosystem
•Level set on key terms and foundational
understanding.
•How can the organization make
use of AI Agents?
•What is the role of LLMs and
automation?
Target Processes Overview of business objectives and
use cases for LLM Agents
•Orient client team to multiple agentic
approaches.
•Get initial response on which are most
important.
•Which business priorities must
the AI serve first?
•Which teams will be most
involved?
Current
Landscape
Definition
Map processes, knowledge, data and
content by people and systems to
identify how business processes are
currently supported.
•Develop more complete view of scope and
scale of systems.
•Map target process
•Develop domain model
•Where are key automation
leverage points/processes?
•Where are key areas of changes
re: people and process?
PoV Planning Plan and proposal for LLM Agent Proof
of Value.
•Plan to implement AI Agent powered
capability
•What is the level of effort
needed for infrastructure and
execution?

37. Copyright © 2024Earley Information Science, Inc. All Rights Reserved.www.earley.comwww.earley.comwww.earley.com
The EIS/Arcana AI Agent Working Session Can Help Get You There
Through a combination of interviews, questionnaires, surveys and working sessions, the EIS/Arcana AI
Agent Assessment:
1.Educates executives and stakeholders about AI technologies – capabilities and limitations
2.Evaluates business value and target use cases for Agentic Approaches
3.Outlines success factors and metrics
4.Examines critical areas of the enterprise for AI Agent readiness:
•Business alignment and process clarity
•Knowledge and Data readiness and technology infrastructure
•Ongoing governance, decision making and success measures
5.Summarizes the current state in an executive working session designed to identify gaps, set realistic goals and
prioritize actions for an AI Agent Proof of Value (PoV)
37
The output of the AI Agent Assessment is a roadmap for deployment
of an AI Agent PoV based on a target process for the organization
https://www.earley.com/agent-workflow-requirements-working-session

38. www.earley.com
Additional Reading
38
https://alexanderkline.substack.com/
https://www.earley.com/ama-article

39. www.earley.com
Contact
[email protected]
https://www.linkedin.com/in/sethearley/
Seth Earley
Founder & CEO
Earley Information Science
Sanjay Mehta
Principal Solution Architect
Earley Information Science
[email protected]
https://www.linkedin.com/in/sanjaymehta/
Dominique Legault
Founder
ReliableGenius
Alexander Kline
Founding Partner
Arcana Concept
[email protected]
https://www.linkedin.com/in/alexander-kline-
futurist-/
[email protected]
https://www.linkedin.com/in/dominique-
legault-84824199//

40. www.earley.com
40
We Make Information More Useable, Findable, And Valuable
Earley Information Science is a professional services firm headquartered in Boston and founded in 1994. With over
50+ specialists and growing, Earley focuses on architecting and organizing data – making it more findable, usable,
and valuable.
Our proven methodologies are designed to address product data, content assets, customer data, and corporate
knowledge bases. We deliver scalable solutions to the world’s leading brands, driving measurable business results.