Introduction to LLMs and their relevance for Official Statistics

An introduction to large language models and their relevance for statistical offices , 2024 edition.
Can be downloaded at https://op.europa.eu/en/publication-detail/-/publication/f4a703b3-ea60-11ee-bf53-01aa75ed71a1/language-en
This manual is a straightforward resource for data professionals of Statistical Offices, introducing the use of Large Language Models (LLMs) in the field of Official Statistics. It outlines how LLMs can tackle complex data problems with their advanced language processing capabilities and integrates these models into current processes. This guide introduces LLMs, delineating their evolution, architecture, applications, and implications for future employment within the AI realm. Additionally, it emphasizes the need for ethical and responsible applications, blending research insights with practical industry examples to ensure professionals can maximize LLM benefits while maintaining trust and reliability in their work.

1. An introduction to Large Language Models and their relevance for Statistical Offices Dario Buono, Ph.D. Marius Felecan, MSE Cristiano Tessitore, Ph.D. An Eurostat AI paper by https://ec.europa.eu/eurostat/product?code=KS-TC-24-001

2. Hallucinate

3. Part One Framing the Context

4. GenAI Technology Large Language Models whats and hows.

5. AI definition 2018 “Artificial intelligence (AI) systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information, derived from this data and deciding the best action(s) to take to achieve the given goal . AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions” Independent High-Level Group on AI (hired by the European Commission), 2018

6. AI definition 2024 ‘AI system‘ is a machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments” Art. 3, EU AI Act

7. Large Language Models Architecture Domains Fields Transformers NLP Self Driving Tesla LLMs : Gemini ChatGPT Deep Learning Machine Learning Artificial Intelligence

8. Large Language Models - Training LLM ( parameters ) Corpus

9. Large Language Models – Fine tuning LLM ( parameters )

10. Large Language Models – Inference Mary had a little LLM lamb Prompt (Context)

11. Charting LLMs A Call for Standardization

12. Emerging & Disrupting Emerging - new and innovative development that significantly alters the current landscape of business and society. Disrupting - groundbreaking product or service that fundamentally changes the market or society.

13. Cars Prediction (1903) “The automobile is a fad, a novelty. Horses are here to stay.” President of Michigan Savings Bank

14. … forecasts about disruptors PC Prediction (mid-1970s) “There is no reason an individual would ever want a computer in their home.” Ken Olsen, founder of DEC digital photography, mobile computing, smart phone … Internet is now 35 years old! still reinventing itself

15. Not mature: Still progress to be made. Not well understood : Still the expectations are sometimes unrealistic. Emergent Disruptive Huge potential Chain reactions How fast? Hype vs. Reality

16. FOMO (fear of missing out) FOBO (fear of a better option) Fear of Being too Early high risk, high reward

17. Now & Near Future Some Strategies to Consider Education and Awareness Technology Adoption and Integration Workforce Reskilling and Upskilling Partnerships and Collaborations Ethical AI Use and Governance

18. Jeff Bezos on AI: Large language models are ‘not inventions, they’re discoveries’ Source: Jeff Bezos - Amazon and Blue Origin, Lex Fridman Podcast, YouTube

19. Questions ?

20. Part Two Use Cases

21. LLM4Statistics Interesting concepts and applications

22. Degrees of AI integration Assisted AI (Supportive) : AI systems that enhance human tasks without replacing human decision-making. Examples include analytical tools that provide insights for humans to interpret. Augmented AI (Collaborative) : These AI systems work alongside humans, enhancing their capabilities through suggestions or automating routine parts of tasks. This is seen in applications like medical diagnostics, where AI assists in analyzing data. Automated AI (Independent) : AI that operates fully autonomously, performing tasks without human intervention. Common uses include robotic process automation for repetitive tasks such as data entry. Autonomous AI (Self-sufficient) : The most advanced form, these systems operate independently in changing environments, making decisions without human input. Autonomous vehicles and intelligent drones are key examples.

23. Multitool vs Screwdriver

24. LLM vs SLM LLM SLM Parameters Several billions (even trillion) Few billions (1 to 3) Knowledge Wide To be specialized Languages Several English Fine tuning Expensive Cheap Hardware specs High Low Performances on benchmark High variable

25. Mixture of experts INPUT Router Expert 1 Expert 2 Expert 3 Expert 4 Expert 5 Additive combination OUTPUT LMSYS  Chatbot Arena Leaderboard as of 07/03/24  

26. Training Extremely expensive Fine tuning May be not that cheap RAG = Retrieval-Augmented Generation Solution for incorporating knowledge from external databases Dynamic | Cost-effective | Modular Training vs Fine tuning vs RAG systems

27. RAG systems Source: NVIDIA

28. RAG systems Source: Gao et al (2024)

29. Selected Use cases

30. GPT@JRC Cloud + Hosted Models Unified Graphical User Interface API available Internal Security and Privacy Rules Compliant European Commission Joint Research Centre GPTs Platform

31. Coding

32. Improved efficiency Our Use Case: Old, inherited codebase Documenting and Commenting

33. Boilerplate code Our recommendation: Use IDE integrated plugins, mature and efficient Comments: careful with prompts Code Generation

34. Test Case Generation Our experience: debugging works better for not so experienced developers Testing and Debugging

35. (Semi-)Automation Teams of GPTs Future developments

36. Research

37. Partner Assistant Critic (reviewer) Personas

38. ChatGPT with plugins Commercially available solutions Writefull - for academic and technical writing HeyGPT - Chat with PDFs Litmaps - Best Literature Search Jenni - Helps you write, edit, and cite with confidence. In-house custom developed scripts Literature review

39. (Semi-)Automation Exploring big context models Future developments

40. Model tuning Going beyond prompting

41. EUBERT is a pretrained BERT uncased model that has been trained on a vast corpus of documents registered by the European Publications Office . These documents span the last 30 years, providing a comprehensive dataset that encompasses a wide range of topics and domains. EU-BERT Model Text Classification, Question Answering, Named entity recognition, part-of-speech tagging, text generation ... https://huggingface.co/EuropeanParliament/EUBERT Using the EuroHPC Meluxina cluster, a core component of Europe's high-performance computing landscape: Hardware Type: 4 x GPUs 24GB GPU Days: 16 Model size: 94M params

42. Questions ?

43. Part Three IP and E thics

44. Generative AI tools are already pre-trained when available for use.  Most of the main models have been trained vast amounts of data available on the Internet , not always respecting copyright or intellectual property. The output of the model can infringe copyright The proprietorship of the output of GenAI is an issue, as copyright protection is only available for works created by human beings. What’s different on GenAI

45. Copyright issue – text

46. Copyright issue - image

47. Practical scenarios Intellectual Property No intention to use the output in any document; Intention to use the output in an internal document; Intention to use the output in an external document. Nature of the information Public information; Non-Public information; Nature of the system Public system; Trusted-cloud provider; Internal LLM.

48. [Rule n°1] Staff must never share any information that is not already in the public domain, nor personal data, with an online available generative AI model. [Rule n°2] Staff should always critically assess any response produced by an online available generative AI model for potential biases and factually inaccurate information. European Commission Guidelines

49. [Rule n°3] Staff should always critically assess whether the outputs of an online available generative AI model are not violating intellectual property rights, in particular copyright of third parties. [Rule n°4] Staff shall never directly replicate the output of a generative AI model in public documents, such as the creation of Commission texts, notably legally binding ones. [Rule n°5] Staff should never rely on online available generative AI models for critical and time-sensitive processes. European Commission Guidelines

50. Ethics

51. I t’s a wrap Not a magic wand but a useful tool

52. To be continued … https://cros.ec.europa.eu/dashboard/ntts-2025

53. Thank you © European Union 2024 CREDITS:

54. Dictionary.com

55. Gartner, https://www.gartner.com/en/articles/what-s-new-in-artificial-intelligence-from-the-2023-gartner-hype-cycle