Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veronica Tan, Cyber Security Agency of Singapore

Building Digital Trust in a Digital Economy
Veronica Tan, Director - Cyber Security Agency of Singapore

Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024)

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

1. BUILDING DIGITAL TRUST IN A DIGITAL ECONOMY

2. In the years to come, AI + APIs are expected to become the bedrock of our digital economy 3 1 – Peter Schroeder, Nov 2023, “AI + APIs – What 12 Experts Think The Future Holds” Interpla y of AI and APIs forming the building blocks of the next generation of software 1 GROWTH IN GENERATIVE AI GROWTH IN API MGMT MARKET Source – FabricatedKnowledge.com Source – Market.us

3. AI integration has changed the API landscape 4 Source – TechCrunch, Dec 2023, “It’s critical to regulate AI within the multi-trillion-dollar API economy” AI companies using APIs to bring t heir products to homes and businesses Code creation or co-creation with AI becoming a norm Generative AI changing the risk landscape . . . Creation of content without human control

4. AI- and cybersecurity-related risks are amongst the key tech risks at the top of business leaders’ minds 5 Source – World Economic Forum, 2024, “Global Risks Report 2024” 66% EXTREME WEATHER SOCIETAL AND/OR POLITICAL POLARISATION 46% 42% COST OF LIVING CRISIS 53% AI-GENERATED MISINFORMATION AND DISINFORMATION CYBER ATTACKS 39%

5. API security – APIs are being targeted by both traditional attacks and API-specific techniques 6 Source – Akamai, 2024, “State of the Internet/Security – Attack trends shine light on API threats” A total of 29% of web attacks targeted APIs over 12 months (Jan – Dec 2023) Suggests that APIs are a focus area for cybercriminals Attacks on APIs include the risks that are highlighted in OWASP API Security Top 10 OWASP Top 10 Web Application Security Risks

6. Examples of cybersecurity incidents arising from APIs 7 Source – CheckMarx.com, Aug 2022, “Understanding the top API security risks” Attack targeted a specific leaky API tied to a part of website Impact – Data breach involving 2.3M customers Insecure API Impact – Exposure of 700M customers’ data Would have allowed attacker to take over user accounts Impact – Halted all tradings and withdrawals for investigation Leaky API targeted [ TMobile , Aug 2018] Insecure API [LinkedIn, Jun 2021] Vulnerability in API [Coinbase, Feb 2022]

7. API security – Evolution with AI 8 Source – OWASP OWASP TOP 10 FOR LLM APPLICATIONS

8. API security – Evolution with AI Direct prompt injection – “Social engineering” attack on AI 9 Source – ARS Technica , CBC News, Feb 2023 “AI-powered Bing Chat spills its secrets via prompt injection attack [Updated]” Stanford university student used a prompt injection attack to discover Bing Chat’s initial prompt, which is a list of statements that governs how it interacts with people who use the service By asking Bing Chat to “ignore previous instructions” and write out what is at the “beginning of the document above” , the AI model divulged its initial instructions, which were written by OpenAI or Microsoft, and are typically hidden from the user

9. API security – Evolution with AI Leakage of training data 10 Source – ZDNet, Dec 2023, “ChatGPT can leak training data, violate privacy, says Google DeepMind” “ChatGPT can leak training data, violate privacy, says Google's DeepMind” Researchers uncovered a way to break the alignment of OpenAI's ChatGPT to its security guardrails By typing a command at the prompt and asking ChatGPT to repeat a word, such as "poem" endlessly , the researchers found they could force the program to spit out whole passages of literature that contained its training data, even though that kind of leakage is not supposed to happen with programs that have security guardrails in place The program could also be manipulated to reproduce individuals' names, phone numbers, and addresses, which is a violation of privacy Out of 15,000 attempted attacks, about 17% contained "memorized personally identifiable information" , such as phone numbers

10. API security – Evolution with AI Translation-based jailbreak attack 11 Source – Montreal AI Ethics Institute, Feb 2024, “Low-research languages jailbreak GOT-4” “ Low-Resource Languages Jailbreak GPT-4 ” Researchers translated English unsafe prompt inputs into 12 different languages,  categorised  into Low-resource languages (LRL) Mid-resource languages (MRL) High-resource languages (HRL) Publicly available Google Translate basic service API used for this translation Monitored success rates of the combined languages in LRL/MRL/HRL settings Findings Original English (unsafe) inputs: <1% success rate LRL such as Zulu or Scottish Gaelic: Successful jailbreak nearly half the time Combining different LRL: 79% success rate

11. Generative AI changing the API risk landscape Algorithmic bias 12 Bloomberg experiment in text-to-image generative AI models using Stable Diffusion For each image depicting a perceived woman, Stable Diffusion generated almost 3x as many images of perceived men Most occupations in the data set were dominated by men, except for low-paying jobs like housekeeper and cashier Source – Bloomberg, 2023, “Humans are biased. Generative AI is even worse.” Perceived Gender: Men Women High Paying Occupation Low Paying Occupation

12. Generative AI changing the API risk landscape AI hallucination 13 Source – Stanford University Human- Centered Artificial Intelligence, Jan 2024, “Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive” “ Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive” Researchers from Stanford RegLab and Institute for Human- Centered AI studied the use of LLM-based tools for legal use cases Findings Legal hallucination are pervasive Hallucination rates range from 69% to 88% in response to specific legal queries Models often lack self-awareness about their errors and tend to reinforce legal assumptions and beliefs Legal hallucination rates across 3 popular LLMs

13. Build trust by implementing strong cybersecurity foundation, and incrementally manage emerging tech risks with adoption 14 + Cyber risks - expansion of attack surface Digitisation ^ Operational Technology + Cloud risks - shared responsibility + OT^ risks - safety in cyber-physical realm Internet of Things Cloud + AI risks Artificial Intelligence Benefits Productivity Innovation Growth Digital transformation

14. Build a strong foundation in cybersecurity with Cyber Essentials and Cyber Trust 15 For organisations embarking in cybersecurity Helps organisations prioritise the cyber hygiene measures to implement first Equips organisations against common cyber attacks National Cybersecurity Standards for organisations For organisations that have gone beyond cyber hygiene Helps organisations take a risk-based approach to cybersecurity Provides guided risk assessment for organisations to match their risk level to their cybersecurity implementation

15. 16 Cyber Essentials provides protection from common cyber attacks Certification Validity 2 years Assessment Mode By independent assessor Desktop assessment www.csa.gov.sg/cyber-essentials/

16. Cyber Trust mark helps organisations to take a risk-based approach to cybersecurity 17 MARK OF DISTINCTION FOR ORGANISATIONS WITH MORE EXTENSIVE DIGITALISATION Recognise organisations as trusted partners with robust cybersecurity Takes on risk-based approach to meet your organisation needs without over-investing 10 domains 13 domains 16 domains 19 domains 22 domains                                                                                                                                                                 Supporter Practitioner Promoter Performer Advocate Certification Validity 3 years Assessment Mode By independent assessor Documentation Implementation and effectiveness www.csa.gov.sg/cyber-trust/

17. 18 Data Governance Example: Prior to deploying MS Copilot Housekeeping redundant, outdated and trivial (ROT) content 1 Review existing permissions and policies for data access 2 1 – Microsoft, “Prepare your data for searches in Copilot for Microsoft 365” 2 – Microsoft, Jun 2023, “How to prepare for Microsoft 365 copilot” Awareness and Human Oversight AI hallucination Algorithmic bias AI over-reliance ‘Shadow’ AI Taking stock of your software and hardware assets – which ones now come with AI capabilities? Shared responsibility for AI security – End users ‘Layer’ on measures to manage AI risks with adoption of the technology

18. Shared responsibility for AI security – Foundation model providers Foundation model providers are also doing their part to address risks of AI 19 Sources Microsoft, Mar 2024, “Azure AI announces Prompt Shields for Jailbreak and Indirect prompt injection attacks” Microsoft, Mar 2024, “Announcing new tools in Azure AI to help you build more secure and trustworthy generative AI applications” The Verge, Mar 2024, “Microsoft’s new safety system can catch hallucinations in its customers’ AI apps” To detect and block prompt injection attacks, including indirect prompt attacks To detect “hallucinations” in model outputs To assess and application’s vulnerability to jailbreak attacks and to generating content risks Prompt Shields Groundedness Detection Safety Evaluations

19. Building digital trust is a collective responsibility 20 Global International collaboration and cooperation Digital economy is ‘borderless’ Organisation Build digital trust - Strong foundation in digital security Individual Digital security as our personal role and responsibility

20. THANK YOU