EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today)
Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers.
Core Insights Include:
Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base.
Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes.
Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies.
Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively.
This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

1. The Key to Context: Prompt Engineering and Knowledge Engineering Seth Earley CEO & Founder Earley Information Science Media Sponsor Mike Doane Director, Content Delivery Cigna Healthcare SANJAY MEHTA Principal Solution Architect EARLEY INFORMATION SCIENCE

2. Today’s Speakers [email protected] https://www.linkedin.com/in/sethearley/ 2 Mike Doane Director, Content Delivery Cigna Healthcare [email protected] www.linkedin.com/in/mikedoane/ Seth Earley Founder & CEO Earley Information Science Sanjay Mehta Principal Solution Architect Earley Information Science [email protected] https://www.linkedin.com/in/sanjaymehta/ “I do not know of any books that have such useful and detailed advice on the relationship between data and successful conversational AI systems.” — Tom Davenport , President’s Distinguished Professor at Babson College, Research Fellow at MIT Initiative on the Digital Economy, and author of Only Humans Need Apply and The AI Advantage

3. Before We Get Started We are recording Session will be 50 minutes plus 10 minutes for Q&A Your input is valued Link to recording & slides will be sent by email after the webinar Use the Q&A box to submit questions Participate in the polls during the webinar Feedback survey afterward (~1.5 minutes) Thank you to our media partners : CMSWire 3

4. About Earley Information Science 4 Proven methodologies to organize information and data. Sell More Product Service Customers Efficiently Innovate Faster 1994 YEAR FOUNDED. Boston HEADQUARTERED. 5 0+ SPECIALISTS & GROWING.

5. Poll 5 Not on the radar Planning stages for Gen AI Controlled experiments using Gen AI Gen AI usage is currently banned Implemented PoC’s (internal or externally facing) Gen AI applications deployed None of the above Where are you on your Gen AI journey?

6. Agenda 6 There’s No AI Without IA Knowledge Engineering Taxonomies, Ontologies and Knowledge Graphs Knowledge Graphs and LLMs Content and Metadata Prompts as Metadata Deriving Prompts from Use Cases Building Standardization through Libraries of Use Cases Next Steps

7. Knowledge Engineering 7 How is taxonomy foundational? What is meant by ontologies being relational? Why is integration with content management critical? How is knowledge harvested? What is the role of Subject Matter Experts (SME’s)?

8. “There’s No AI without IA” Knowledge Architecture is Needed to Support Conversational and Cognitive Applications 8

9. Knowledge Architecture Knowledge engineering is a field of artificial intelligence (AI) that tries to emulate the judgment and behavior of a human expert in a given field.* *https://www.techtarget.com/searchenterpriseai/definition/knowledge-engineering Knowledge Engineering Knowledge architecture consists of the design artifacts and supporting technologies and processes that enable a contextualized information ecosystem .**

10. 10

11. Poll 11 No formal KM programs Early stages of KM KM is used at the departmental level KM is widely deployed and operationalized None of the above Where are you on your Knowledge Management (KM) journey?

12. Navigation versus Classification 12 Classification Hierarchies Allows for definition of “is-ness” (what is this thing?) and “about-ness” (what is it about then that helps me tell them apart?) Classification drives dynamic navigation via facets which leverage is-ness and about-ness (What is this? A sweater. Tell me about this sweater. It’s blue) Relationships between classification hierarchies defines the ontology (Products for Processes, Processes for Industry, etc.) Navigational Hierarchies What most people think of when they hear the term “taxonomy” Core structure of organizing principles for a collection of information Static navigational hierarchies (navigational taxonomies) is a dated approach for any but most rudimentary sites Dynamically driven by classification hierarchies We are not talking about navigational hierarchies (sometimes called “business taxonomies”) due to lack of adherence to classification rules Taxonomy is not the same as navigation

13. Taxonomies to Ontologies and Knowledge Graphs 13 Classification Hierarchies Define Ontologies Ontologies, when connected to data sources becomes a Knowledge Graph the knowledge scaffolding of the enterprise => Connect to Data => Knowledge Graph

14. Knowledge Graphs 14 What is the purpose of a Knowledge Graph? How are linguistics and analytics integrated? Knowledge Graphs connect the LLM to corporate knowledge and data. They are the source of truth for the LLM so that responses are appropriate to enterprise information

15. Generative AI 15 Creates new, original content Trained to learn the patterns and relationships Learned knowledge generates new content (Not a copy of the training data). This means that the knowledge of the organization needs to be referenced by the technology *Source: ChatGPT

16. Content Management Integration 16 Content (and Knowledge) Management is needed to: Ensure that the chatbot has access to a wide range of relevant information and can provide accurate and useful responses to user queries Identify knowledge and expertise needed for the chatbot to function effectively Design the chatbot's knowledge base and information architecture to support user needs.

17. Integration of Knowledge Graphs with LLMs 17 How do Knowledge Graphs Enhance Domain-specific Understanding? How can context be improved?

18. Generation vs Retrieval 18 User query Process query using LLM to understand user intent Generate response based on LLMs understanding of language patterns and concept relationships Retrieve response based on querying organizational knowledge and content Process response using LLM to provide conversational format Uses publicly available information Uses proprietary information and nonpublic IP Does not compromise or expose IP LLMs used to process query and present results Response Corp data sources Vector Store Corporate information is referenced through the knowledge graph

19. 19 Knowledge Graph as Ground Truth Process with LLM with question and relevant documents User query Search/ retrieval Response Corporate information retrieved through sources in knowledge graph Generated results based on provided documents

20. What is Context ? 20 The ability to understand who the user is and what they want. We can break the process down of referencing a manual: Scenario 1 – Access Here is the manual Scenario 2 – Generalized retrieval “Look in chapter 4” Scenario 3 – Specificity of the answer Here is the specific answer to your question from that manual Scenario 4 – Contextualized knowledge Here is the specific answer from the manual and related information based on your exact product configuration and context Manuals compile knowledge for technical support However … They require study And it takes too long to find answers to specific questions from large manuals . (RTFM – TLDR) What is the user’s context? How do people find information?

21. Customer Identity Graph How do we describe context? With metadata.

22. Explicit and Implicit Customer Metadata Where do we get metadata? By collecting signals instrumented throughout the user journey

23. User context, process context and content context 23 How do I set up my modem? Where is the installation guide? What does error code 50 mean? Questions and Answers Need Context

24. Product Manual – HM 2900 Series Modem Generative AI Reference Content Requires Context 24 Overview Set up options Settings Model 2960 Error code 50 Settings Model 2970 Error code 56 Installation Troubleshooting Hardware setup VPN requirements Factory settings Technical Specifications I need to install a modem. Which modem model do you have? Content type = Product Manual Content type = Troubleshooting Content type = Installation Model 2960 I am receiving an error code of 50 OK, here are the installation settings… That error requires the following troubleshooting steps: What context is required? Type of information, installation, product model, error code, etc. Metadata provides context Product name = HM 2900 Series Modem Model = 2960 Error code = 50

25. Prompt as Metadata Container 25 “I need to install a model 2960 modem, but I am receiving an error code of 50. Please provide the troubleshooting steps to complete my installation” Content type = Product Manual Content type = Troubleshooting Content type = Installation What context is required? Type of information, installation, product model, error code, etc. Prompt metadata provides context Product name = HM 2900 Series Modem Model = 2960 Error code = 50

26. Use Cases as Prompt 26 Use cases inform the knowledge architecture. Libraries of use cases allow for testing functionality, facet identification and baseline metric comparison Role metadata Entity metadata

27. Implications 27 Need reference prompts representing classes of use case Training material for how users need to ask questions Knowledge architecture provides clues as to how to ask questions to get content in context Analogous to faceted retrieval – an ambiguous question requires disambiguation through filtering

28. Types of Prompts 28 One shot – no context is needed – rote process (text summarization, sentiment analysis, spam classification). Few shot – Additional context provided through inputting relevant content with the prompt. Prompts become metadata – the role of the customer, technical vs business, the role of the bot “you are an expert in brand marketing”

29. Types of Prompts 29 Chain of thought prompting – Break the prompt into multiple steps and show intermediate results. Different lines of thinking and compares results Build the output based on sequential outputs of sub problems or steps Monitoring confidence and escalating for human review.

30. Prompt Metadata 30 Asking for outputs and then using knowledge to ask additional questions. “What questions should I ask about Generative AI and prompts?” Understanding the Prompt's Purpose: What is the main goal or desired outcome of this prompt? (e.g., generate creative text formats, translate languages, write different kinds of creative content) Who is the target audience for the generated output? (e.g., general public, specific professionals, children) In what context will the output be used? (e.g., marketing materials, educational content, personal entertainment) Crafting an Effective Prompt: What specific information or keywords must be included in the prompt? How can I phrase the prompt to ensure clarity and avoid ambiguity? How can you describe the output? (tone, mood, technical depth) How can you describe the audience? What is the expected artefact (plan, program, campaign, contract)? What additional metadata (features) can you provide? What is the use case?

31. 01 Role, Target, Purpose, Constraints Providing model with macro context (ex. Industry), organizational context, brand language,  objectives (outcomes/goals), role(s) & scope, access controls (ex. guard rails) Incorporate user specific context (ex. Preferences, history, timeframe) 02 Examples, Few Shot, Chain of Thought, Hints Provide model with scenarios and outcomes with reasoning steps.  Ex. Providing a problem & solution, a recipe , Q&A, format, directional stimulus, insights such as analytics. 03 Chaining, Synthetic Signals, Multi-Modal Ask the LLM generate an output to feed to another prompt, training data, examples.  Ex. Generate application specific keywords to feed to another prompt to generate a description.  Extract labels from an image and feed to prompt to generate a name. 04 Retrieval Augmented Generation Provide model with results from internal knowledge base, limit results to only what is within knowledge base or content provided using retrieval approaches such as vector, semantic, lexical, graph... 05 Signals & Evaluation Reinforcement context, outcome analysis, historical q&a  pairs, query outcome, KPI targets. LLM Integration With Enterprise Knowledge Base & Content

32. An Example Source: Rec Prompt https://arxiv.org/html/2312.10463v1

33. Poll 33 Forget leadership, its not known at any level Pockets of understanding throughout the organization Senior managers and leaders understand the connection KM programs are actively integrated with LLM projects through approaches like Retrieval Augmented Generation (RAG) None of the above Is there awareness at the leadership level of the connection of Knowledge Management to Gen AI?

34. Getting Started /Next Steps 34 Ensure adequate funding and executive support Define use cases Identify bodies of content needed to support use cases Gather baseline metrics for supported processes Build out strawman domain model Develop metadata structure for target use cases Tag content with taxonomy, ontology and reference knowledge graph Ingest tagged content into vector store as enriched embeddings Test against use cases with gold standard of responses Instrumentation of processes to show value and improvement Shampoo, Rinse, Repeat

35. Are you measuring the following? Quality and completeness of answers in knowledge base Usage of knowledge resources when handling calls Contributions to knowledge repositories by reps Consistency across knowledge sources Correlation of knowledge source quality with first call resolution, time per incident, CSAT, LTR, and other call center metrics Consistency and recency of taxonomies (compliant with best practices)? Call deflection to self service 35

36. KM for AI Readiness Sprint and Proof of Value Plan A KM for AI readiness sprint will quickly identify how the organization can deploy Generative AI to address issues and challenges that arise from knowledge capabilities that have not kept up with growth, acquisitions, changes in the marketplace or changes in products, technologies and the competitive landscape. The sprint consists of the following: Stakeholder interviews Education and alignment Current state maturity Review of knowledge systems and tools Generative AI Proof of Value (PoV) plan https://www.earley.com/km-ai-readiness-assessment

37. Maturity Model for Knowledge Management 1-Unpredictable 2-Aware 3-Competent 4-Synchronized 5-Choreographed Core Collaboration Rudimentary, random , haphazard Intentional, ineffective knowledge capture and codification Practices identified, formal harvesting and promotion of output Integrated into processes with creation, access and reuse mapped Seamless and habitual with collaboration processes integrated with business needs and downstream uses Expertise Location Who you know Word of mouth Key skills Identified and captured Formal expertise directories leveraged Expertise derivation through text analytics, community participation and group interactions Content Curation Haphazard and application limited Pockets of curated content, sub optimized across processes Content aligned with platform and process, governance driven information concierge Workflow driven integration with hybrid tagging and entity extraction processes Value added at each touch point, core metadata flows with content, lifecycles managed , retention enforced Information Architecture IA is Navigation, inconsistent metadata, few standards, poor usability Application and department localized taxonomies Classification structures applied to support dynamic knowledge Multi-channel, device and format independent cross application architecture Upstream supply chain and downstream syndication w/ partner & customer processes Infrastructure Foundational, little to no collaborative tools, CM rudimentary Foundational tools in place but with out of the box deployment Knowledge harvesting integrated with collaboration and ruse Expertise location, community management, intentional knowledge optimization Multiple tools mapped to detailed requirements and use cases with ongoing tuning and enhancement Search Integration Search as “Random Document Generator “ Some tuning of search ranking algorithm with content tagging Integration across structured and unstructured systems for content in context Expertise mapped to search domains and terms, reduced e-discovery risk Search driven integration across platforms, knowledge in task and process context Governance Non existent Initial attempts lead to fiefdoms Repeatable, defined KM governance Integrated, cross functional managed processes Business value driven, enterprise wide deployment Capability Maturity

38. Knowledge Management and AI Working Session Topics Topic Overview Goals Questions to Address Key Concepts and Success Criteria Generative AI and knowledge management definitions and success factors. Level set on key terms and foundational understanding. How can the organization make use of Generative AI? What is the role of knowledge? Knowledge in Context Overview of business objectives and use cases for knowledge application Orient client team to multiple knowledge taxonomies. Get initial response on which are most important (to client). Which business priorities must the AI serve first? Which teams will be most involved? Current Landscape Definition Map knowledge and content by people and systems to identify how business processes are currently supported. Develop more complete view of scope and scale of systems with knowledge and content. Map knowledge to process Develop domain model Where are key knowledge leverage (usage) points/processes? Where are key areas of changes re: people and process? PoV Planning Plan and proposal for Generative AI Proof of Value. Implement knowledge base with AI powered chat front end What are the costs and needed infrastructure for execution?

39. The EIS KM and AI Readiness Assessment Can Help Get You There Through a combination of interviews, questionnaires, surveys and working sessions, the EIS KM for AI Readiness Assessment: Educates executives and stakeholders about AI technologies – capabilities and limitations Evaluates business value and target use cases for Generative AI Outlines success factors and metrics Examines critical areas of the enterprise for Generative AI readiness: Business alignment and process clarity Knowledge and Data readiness and technology infrastructure Ongoing governance, decision making and success measures Summarizes the current state in an executive working session designed to identify gaps, set realistic goals and prioritize actions for a Generative AI Proof of Value (PoV) 39 The output of the EIS KM and AI Readiness Assessment is a roadmap for deployment of a Generative AI PoV based on corporate knowledge

40. 40 Need Clarity on Retrieval Augmented Generation (RAG)? https://www.earley.com/ama-article Great companion to The AI Powered Enterprise Download now: or you can request a physical reprint

41. Contact [email protected] https://www.linkedin.com/in/sethearley/ 41 Mike Doane Director, Content Delivery Cigna Healthcare [email protected] www.linkedin.com/in/mikedoane/ Seth Earley Founder & CEO Earley Information Science Dave Skrobela Client Partner Managing Director Earley Information Science [email protected] Sanjay Mehta Principal Solution Architect Earley Information Science [email protected] https://www.linkedin.com/in/sanjaymehta/ www.linkedin.com/in/skrobela/

42. We Make Information More Useable, Findable, And Valuable  Earley Information Science is a professional services firm headquartered in Boston and founded in 1994. With over 50+ specialists and growing , Earley focuses on architecting and organizing data – making it more findable, usable, and valuable. Our proven methodologies are designed to address product data, content assets, customer data, and corporate knowledge bases. We deliver scalable solutions to the world’s leading brands, driving measurable business results.