AI Data Engineering for SMEs - some tricks and tools

A collection of tips and tools for using Generative AI that might be of use to SMEs and others. Use for data Analysis, summarising resources.

Presented as part of a workshop at AI for Business and Enterprise 16th May 2024 at Canterbury Christ Church University https://cccub2b.my.site.com/AIConference/s/

1. AI data engineering for SME’s Scott Turner Section Director Computing School of Engineering, Design and Technology bit.ly/ AIDataEng

2. Split of Session Gemini to extract from websites Use Tree of Thoughts as a way to see different perspectives. Gemini and ChatGPT4 using Chain of thought to find questions and potential answers Use of Data Analyst Using Data Analyst Documents Chain of Density for richer summaries Generating Social media posts from text.

3. Gemini Read, convert to plain text and consolidate information from the following sites https://www.gov.uk/government/publications/national-disability-strategy/part-2-disabled-peoples-everyday-experience-at-the-heart-of-government-policy-making-and-service-delivery and https://www.gov.uk/government/publications/national-disability-strategy/part-1-practical-steps-now-to-improve-disabled-peoples-everyday-lives and https://www.gov.uk/government/publications/national-disability-strategy/part-3-a-cross-government-effort-to-transform-disabled-peoples-everyday-lives When done present in table form three key points of each website and key points of similarity

4. Strategies Use personas as much as possible Tree of Thoughts – use the personas to bring together thoughts of ‘experts’ to a group plan Example: Randomly chose a company Got three persona sales, production and finance to go through the three websites and come up with a plan and actions with ideas informed by the others.

5. Using the extracted contents of the three websites. In the context of the company with this website https://www.careco.co.uk/about. Imagine that these three experts Sales, Production and Finance are going review and draw insights from these for opportunities for the organisation offered by the websites. They will work on one step at a time and share their steps with each other as they proceed. The experts will write down each step of their thinking and share it with the group. The experts will take a moment to examine each other’s steps and compare the stated steps. An expert can change their opinion based on seeing what another expert stated. Then all experts will go on to the next step. They will integrate through the process each time and refine the answers. At the very end, the experts are to reach a final decision based on having seen each other’s stated steps throughout the problem-solving process, each expert will give an action to take away.

6. Expand and find some sources Let’s say one of the actions was “ Action (Production): Research government grants, programs, and potential partnerships related to homecare for disabled people. Explore collaboration opportunities.” We could expand it and get some potential sources of funding Prompt: “Taking Action (Production) expand it and search for potential UK funding sources to make it happen”

7. If you want to explore this a bit. The Prompts and the Responses are available at: https://llmapplied.blogspot.com/2024/05/taking-to-yourselves-with-chatgpt-and.html

8. Chain of Thought Decompose a questions into a series of self-contained subquestions and answer them. Prompt below adapted from: https://www.forbes.com/sites/lanceeliot/2023/07/31/new-prompt-engineering-technique-pumps-up-chain-of-thought-with-factored-decomposition-and-spurs-exciting-uplift-when-using-generative-ai/?sh=5ccf938e19cb I’m going to ask you a question. I want you to decompose it into a series of subquestions . Each subquestion should be self-contained with all the information necessary to solve it. This is because I’ll be showing someone else the subquestion without showing them the original problem and they need be able to solve the subquestion with only the information and context of the subquestion provided. This is really important - for example, you should never say things like ”the teacher” or ”the father” without giving more context as to who the teacher is and possibly the entire passage or situation that is being referenced. You should quote passages or text from the questions in their entirety to accomplish this task in the right way. For each subquestion provide an answer and then provide a final answer.

9. Sentiment Analysis Using ChatGPT4o Load in the file Prompt 1: Read in, convert to plain text and then extract the contents. Produce a summary approximately 250 words. Produce a sentiment analysis of the document identifying is it positive, negative or neutral document Prompt 2: Quantify the levels of sentiment into positive, negative and neutral and produce a visualization of these Prompt 3: Produce and show three word clouds for the negative, positive and neutral words Prompt 4: Produce Produce a new word document called senti2.docx with all the analysis and images https://chat.openai.com/share/6a08a000-6220-4fe8-bef1-26a28b09f0be

10. Using ChatGPT4 and Data Analyst Prompt 1: dataset_kaggel.zip Zip Archive Using this file, extract the file and analyse it. Produce three interesting presentations of key data. Prompt 2: Produce a single report made up of the visualisations and the analysis around them. Save the report as word document called "world.docx“ Prompt 3: looking at just United Kigdom and France data produce a table comparing the two Prompt4: Produce a graph that visualises the french and UK data with all appropriate scaling and legends Prompt 5: Fit the best model to fit the data of the two countries and show these on the graph Prompt 6: using the models produced predict the inflation rate in 2025 for France and the Uk

11. Better summaries A lot of people are putting documents through these tools. Research suggests an Approach called Chain of Density prompts where the tool goes through multiple times and makes the summary richer. The original paper can be found at Structure/Prompt itself. You will generate increasingly concise entity-dense summarises of the article (Article)loaded in. Repeat the following 2 steps 5 times Step 1. Identity 1-3 informative Entities (“;” delimited)from the article which are missing from the previous generated summary. Step 2. Write a new denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities. A Missing Entity is:- -Relevant: to the main article. -Specific: descriptive yet concise (5 words or fewer). -Faithful: present in the Article. -Anywhere: located anywhere in the Article. Guidelines: -The first summary should be long (8-10 sentences, ~250 words) yet highly non-specific, containing little information beyond the entities marked as missing. Use overly verbose language and fillers ( e,g . “this articles discusses”) to reach ~250 words. -Make every word count: rewrite the previous summary to improve flow and make space for additional entities. -Make space with fusion, compression, and removal of uninformative phrases like “the article discusses”. -The summaries should become highly dense and yet self-contained, e.g. easily understood without the Article. -Missing entities can appear anywhere in the new summary. -Never drop entities from the previous summary. If space cannot be made, add fewer new entities. Remember , use exact same number of words for each summary. https://doi.org/10.48550/arXiv.2309.04269

12. First go. Summary 1: This article discusses the Kent & Medway Local Skills Improvement Plan (LSIP), which aims to enhance local skills and training in response to employer needs and labor market demands. It builds on a large evidence base, including insights from employer surveys and engagement with local businesses. The LSIP highlights key sectors such as construction, manufacturing, engineering, and health and social care. It also emphasizes the importance of cross-cutting themes like decarbonization, digitalization, and SME engagement. The plan outlines specific actions and objectives for improving perceptions of the construction industry, modernizing training offers, and upskilling and reskilling the existing workforce. Additionally, it addresses the need for better alignment between training providers and employer needs, and the importance of fostering a culture of continual learning and innovation. The LSIP aims to support the local economy by ensuring a skilled workforce that can meet future demands and contribute to economic growth. The plan has been approved by the Secretary of State in accordance with the Skills and Post-16 Education Act 2022. Missing Entities: Trailblazer LSIPs Strategic Development Fund (SDF) Employer Representative Bodies

13. Summary – final version Summary 5: The Kent & Medway Local Skills Improvement Plan (LSIP) aligns skills training with employer needs, using evidence from surveys and engagements. It focuses on construction, manufacturing, engineering, and health and social care, addressing decarbonization, digitalization, and SME engagement. The LSIP outlines actions to improve construction industry perceptions, modernize training, and upskill the workforce, emphasizing alignment between training providers and employers. Approved under the Skills and Post-16 Education Act 2022, it builds on Trailblazer LSIPs and is supported by the Strategic Development Fund (SDF). Employer Representative Bodies coordinate efforts. The Kent Construction Focus Group (KCFG) highlights construction skills needs. The Skills for Jobs White Paper sets the reform context, and the Employer Skills Survey (ESS) provides data on skill gaps. SME engagement challenges, Modern Methods of Construction (MMC), and decarbonization skills are key focus areas. Digital skills for construction, upskilling and reskilling pathways, and collaboration with the Careers Hub and National Career Service are integral to the plan's success. https://chat.openai.com/share/99ad1cfc-1ad0-4522-9148-c1de9251d186

14. Social media generation Prompt 1: Read the contents of https://www.canterbury.ac.uk/study-here/courses/postgraduate/data-intelligence, convert to plain text. Using this information create five posssible twitter posts to promote this course to grduates Prompt 2: Repeat but add the course webpage url in as well Prompt 3: Now repeat but this time for LinkedIn posts

15. Some useful sources Tree of Thoughts – Strategy https://www.forbes.com/sites/lanceeliot/2023/09/08/prompt-engineering-embraces-tree-of-thoughts-as-latest-new-technique-to-solve-generative-ai-toughest-problems/ Chain of Thought : https://www.forbes.com/sites/lanceeliot/2023/07/31/new-prompt-engineering-technique-pumps-up-chain-of-thought-with-factored-decomposition-and-spurs-exciting-uplift-when-using-generative-ai/?sh=5ccf938e19cb Chain of Density: https://doi.org/10.48550/arXiv.2309.04269 Kaggle Data https://www.kaggle.com/datasets/sazidthe1/global-inflation-data?resource=download https://kentemployerskillsplan.org/resources/kent-medway-local-skills-improvement-plan-2023/ Earlier version of slides bit.ly/ AIDataEng