Accessibility Testing and VPATs - UX Scotland, May 2024

On first impression, the Voluntary Product Accessibility Template (VPAT) looks dry, dull, and a right old chore.

Roger will be sharing what he's learned on his own accessibility journey and wants to persuade you to see past that – to recognize the VPAT as an eye-opening opportunity for an enjoyable and fruitful challenge that carries an exciting opportunity to advocate for improved accessibility in your organisation.

With the right mindset, once you've done an accessibility audit for one product, you'll want to do it for all of them!

Takeaways:
- What are VPATs, and how they differ
What you should know before doing an accessibility audit
- The audit process
- Why the language you use in the VPAT matters
- Analyzing the VPAT results and understanding what 'good' looks like
- And most importantly… why completing the VPAT is not the goal here

Themes:
Digital accessibility, Inclusivity and accessibility, UX and product design, Measurability and metrics

1. Roger Attrill, UX Specialist (and Accessibility Advocate) IQVIA @think_ui / rogerattrill UX Scotland 29 May 2024 A11y you need to know about accessibility audits and VPATs

2. About me Image created by DALL-E 3 via Bing Copilot (This is not me!)

3. Introduction

4. Introduction Image created by DALL-E 3 via Bing Copilot Sharing what I’ve learnt on my own accessibility auditing journey…

5. Introduction Companion article published today: https://tbc Some terms I’ll use a LOT! Different types of accessibility audit Why the VPAT?

6. Terms I’m going to use a LOT! WCAG (commonly pronounced as WuhCAG ) is a series of web content accessibility guidelines published by the Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C). (Digital) accessibility is a practice and a mindset of designing and building websites, web applications, and web-based documents in a way that removes barriers that prevent interaction or access. VPAT® is a Voluntary Product Accessibility Template. It’s a statement of the accessibility conformance of your product, mainly against WCAG.

7. WCAG 2.2 Success Criteria

8. WCAG 2.1 Success Criteria

9. WCAG 2.0 Success Criteria

10. Different types of accessibility audit Assess design system or mock-ups to communicate accessibility requirements for developers Design audit Usability testing Screen reader Risk evaluation Detailed audit VPAT (Voluntary Product Accessibility Template) Accessibility Conformance Statement Level of Effort Validation User testing with inclusive participants to understand what’s usable or not in the real world, including people with disabilities Assess user workflows and scenarios with screen readers and/or other assistive technologies on different browsers and platforms Identify critical accessibility issues that real world users will experience In-depth discovery exercise to identify accessibility issues using automated tools and manual testing, with remediation recommendations A standardized procurement report detailing instances where the product doesn’t support accessibility guidelines (WCAG, Section 508) Describes the level of detail to which a product or service provides support for people with disabilities, so they know what to expect Uses product knowledge to assess the magnitude and/or cost of remediating accessibility issues across the product or website Reassessment of product based on previous audit findings to determine success of remediation exercise

11. Different types of accessibility audit Design Audit Assess design system or mock-ups to communicate accessibility requirements for developers Risk evaluation Identify critical accessibility issues that real world users will experience Usability testing User testing with inclusive participants to understand what’s usable or not in the real world, including people with disabilities Detailed audit An in-depth discovery exercise to identify accessibility issues using automated tools and manual testing, with remediation recommendations Screen reader Assess user workflows and scenarios with screen readers and/or other assistive technologies on different browsers and platforms VPAT (Voluntary Product Accessibility Template) A standardized procurement report detailing instances where the product doesn’t support accessibility guidelines (WCAG, Section 508) Accessibility Conformance Statement Describes the level of detail to which a product or service provides support for people with disabilities, so they know what to expect Level of Effort Uses product knowledge to assess the magnitude and/or cost of remediating accessibility issues across the product or website Validation Reassessment of product based on previous audit findings to determine success of remediation exercise

12. Different types of accessibility audit

13. Accessibility statement (TfL Go) How accessible the app is We know some parts of TfL Go across iOS and Android are not fully accessible: iOS Some text has colour contrast issues Some headings aren't labelled as headings Some elements don't receive focus when using an external keyboard …

14. Different types of accessibility audit

15. Introduction Article published today: https://www.article Some terms I’ll use a LOT! Different types of accessibility audit Why the VPAT?

16. The process includes activities from other audits: Risk evaluation: identifying the critical issues that will block users Screen reader testing: using screen readers as a tool to check for issues Detailed audit: you’re providing a complete report identifying any instance of non-conformance Accessibility conformance statements: you’re identifying the key issues that end-users will need to be aware of within the product, before they use it Level of effort: you can use the results of the VPAT to inform the product team about the effort required to resolve issues

17. Where to find the official VPAT®? You can download the latest official templates from the Information Technology Industry Council (ITI) website: https://www.itic.org/policy/accessibility/vpat

18. Different VPAT editions for VPAT v2.5 https://www.itic.org/policy/accessibility/vpat Section 508 edition is for Section 508 – the U.S. Federal accessibility standard - and references WCAG 2.0  EU edition caters for EN 301 549 – the EU’s “Accessibility requirements suitable for public procurement of ICT products and services in Europe” – and covers WCAG 2.0 and WCAG 2.1 WCAG edition caters for WCAG 2.0 , WCAG 2.1 , and WCAG 2.2 International edition incorporates all three

19. Preparation

20. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

21. Get the mindset Image created by DALL-E 3 via Bing Copilot Be the change

22. Get the mindset * Check out the W3C course “Introduction to Web Accessibility” from edX: https://www.w3.org/WAI/courses/foundations-course/ Understand the needs of people with disabilities * Understand what the accessibility guidelines require of the product Get an audit under your belt Learn along the way Improve on the next one Be the change

23. Get the mindset Any accessibility audit is better than none!

24. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

25. Journey Image created by DALL-E 3 via Bing Copilot Not the destination It’s about what happens during and after the process.

26. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

27. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

28. Detective mode Image created by DALL-E 3 via Bing Copilot Look for trouble

29. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

30. Which screen reader? https://webaim.org/projects/screenreadersurvey10/#primary One that people use in the real world!

31. Screen readers… Image created by DALL-E 3 via Bing Copilot They are hard work for noobs!

32. Screen readers… Image created by DALL-E 3 via Bing Copilot Locate the emergency exit first

33. Screen readers: Emergency exits * NVDA: Insert + Q (Ctrl + Alt + N to start ) JAWS: Insert + F4 VoiceOver : Cmd + F5 to stop/start Narrator: Caps Lock + Esc (Windows + Ctrl + Enter to stop/start) * ‘Ctrl’ temporarily silences all readers More screen reader shortcuts and gestures including cheat sheets (from Deque): https://dequeuniversity.com/screenreaders/

34. Screen reader and braille display users

35. Each node in the accessibility tree has component info, e.g. Name Description Role State Value Action The accessibility tree But only if the information is there in the first place!

36. Docs for accessibility inspectors Chrome https://developer.chrome.com/docs/devtools/accessibility/reference Firefox https://firefox-source-docs.mozilla.org/devtools-user/accessibility_inspector/index.html Apple https://developer.apple.com/documentation/accessibility/accessibility-inspector

37. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

38. Big task? Image created by DALL-E 3 via Bing Copilot Understand the scope first

39. Understand the scope of your audit You only need to review content that you are responsible for You don’t need to audit absolutely everything You’ll probably only be auditing the level A and AA criteria …but what you do audit needs to be done in detail

40. Understand the scope of your audit You only need to review content that you are responsible for You don’t need to audit absolutely everything You’ll probably only be auditing the level A and AA criteria …but what you do audit needs to be done in detail

41. Google GMail VPAT Tested Pages: - Send a quick short message - Send an important formal message - Understand a conversation - I want to find information from a message received earlier. - Clean up or get rid of the messages - Don't miss any important messages - Share content with someone - Remember to get back to a message later - Keep my Gmail account secure

42. Understand the scope of your audit You only need to review content that you are responsible for You don’t need to audit absolutely everything You’ll probably only be auditing the level A and AA criteria …but what you do audit needs to be done in detail

43. Understand the scope of your audit You only need to review content that you are responsible for You don’t need to audit absolutely everything You’ll probably only be auditing the level A and AA criteria …but what you do audit needs to be done in detail

44. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

45. https://bit.ly/wcag-info https://docs.google.com/spreadsheets/d/1kbX5rIf7-1K8WTn2rBXg71vvigQjNiShxe0yZ5oRPOI/edit The WCAG Success Criteria

46. https://intopia.digital/resources/ The WCAG Success Criteria More Resources https://www.levelaccess.com/resources/must-have-wcag-2-1-checklist/ https://www.w3.org/WAI/WCAG22/quickref/?versions=2.1 https://nhsdigital.github.io/accessibility-checklist/checklist-full/

47. Preparing to do an audit Get the mindset - be the change - become the expert Know that the value is in the journey not the destination Give yourself time Put your detective hat on Break down the scope of your audit Get familiar with the WCAG success criteria Learn how to use a screen reader (and the accessibility tree)

48. The audit process

49. Prepared? Image created by DALL-E 3 via Bing Copilot Now for the auditing process

50. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results

51. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results (In WCAG # order, not A, AA, AAA)

52. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results

53. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results

54. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results

55. Example

56. Example 1.1.1 Non-text Content (A) https://www.w3.org/WAI/WCAG22/Understanding/

57. Example 1.1.1 Non-text Content (A) ?? https://www.w3.org/WAI/WCAG22/Understanding/

58. Example: 1.1.1 Non-text Content (A) All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for (some situations...)

59. Example: 1.1.1 Non-text Content (A) All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for (some situations...)

60. Example: 1.1.1 Non-text Content (A) All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for (some situations...)

61. Example: 1.1.1 Non-text Content (A) All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for (some situations...)

62. Example: 1.1.1 Non-text Content (A) Using examples from the Web Accessibility Initiative: https://www.w3.org/WAI/tutorials/images/ Standalone images Icon buttons and links Other meaningful graphics Decorative images Other content… Looking for issues…

63. Example: 1.1.1 Non-text Content (A) 1) Standalone images https://www.w3.org/WAI/tutorials/images/

64. Example: 1.1.1 Non-text Content (A) 2) Icon buttons and icon links https://www.w3.org/WAI/tutorials/images/

65. Example: 1.1.1 Non-text Content (A) 3) Other meaningful graphics https://www.w3.org/WAI/tutorials/images/

66. Example: 1.1.1 Non-text Content (A) 4) Decorative images https://www.w3.org/WAI/tutorials/images/

67. Example: 1.1.1 Non-text Content (A) 5) Other content https://www.w3.org/WAI/tutorials/images/

68. Semi-automated tools to help WAVE (Web tool or browser extension): https://wave.webaim.org/

69. Semi-automated tools to help Web Developer (browser extension): https://chrispederick.com/work/web-developer/

70. Semi-automated tools to help ANDI (Bookmarklet): https://www.ssa.gov/accessibility/andi/help/install.html

71. Semi-automated tools to help Semi-automated means they do half the job. You do the rest!

72. Working document Introduction WCAG criteria Summary of results A working document For each WCAG criteria description from WAI links for more details example failure checks notes and testing tips tools to use audit notes Audit notes explicitly state if not being evaluated date of testing each page or module tested notes and observations in passing possible issues to investigate actual issues found possible resolutions wording to use in VPAT result (Supports, Partially Supports, etc) statement and summary of issues Summary of results: conformance by A, AA,… conformance by severity conformance by criteria prioritized issues VPAT Introduction Product info WCAG criteria results Plain English please 

73. Look at other VPATS to learn by example e.g. 1.1.1 Non-text Content (A) Google Drawings Web Application: Partially Supports Documentation: Partially Supports Google Drawings web application properly labels and announces all non-text elements during navigation, with some exceptions: When adding a shape to the drawing canvas, NVDA users are not able to perceive the available options, as shapes are not reported by screen reader. When inserting a diagram, there is an unlabeled image present in the diagram sidebar. Documentation: The documentation properly labels and announces all non-text elements during navigation, with one exception: There is a graphic embedded in the link “Visit the learning center ” that has no alt text. Google Meet Partially Supports Most of the non-text content have text alternatives or has a text alternate that serve equivalent purpose. However, there is an exception: Some decorative images are not hidden from screen readers on the Meet Landing page in the ‘Join/Create - Talk to my colleagues’ user journey.

74. Look at other VPATS to learn by example e.g. 1.1.1 Non-text Content (A) Google Drawings Web Application: Partially Supports Documentation: Partially Supports Google Drawings web application properly labels and announces all non-text elements during navigation, with some exceptions: When adding a shape to the drawing canvas, NVDA users are not able to perceive the available options, as shapes are not reported by screen reader. When inserting a diagram, there is an unlabeled image present in the diagram sidebar. Documentation: The documentation properly labels and announces all non-text elements during navigation, with one exception: There is a graphic embedded in the link “Visit the learning center ” that has no alt text. Google Meet Partially Supports Most of the non-text content have text alternatives or has a text alternate that serve equivalent purpose. However, there is an exception: Some decorative images are not hidden from screen readers on the Meet Landing page in the ‘Join/Create - Talk to my colleagues’ user journey.

75. Reporting back

76. Reporting back to your team Be wary of suggesting fixes - be descriptive rather than prescriptive. Don’t flood the bug tracker Be ready for questions Have some metrics to hand - what does ‘good’ look like A few tips…

77. Reporting back to your team Be wary of suggesting fixes - be descriptive rather than prescriptive. Don’t flood the bug tracker Be ready for questions Have some metrics to hand - what does ‘good’ look like A few tips…

78. Reporting issues Image created by DALL-E 3 via Bing Copilot Don’t flood the bug tracker

79. Reporting back to your team Be wary of suggesting fixes - be descriptive rather than prescriptive. Don’t flood the bug tracker Be ready for questions Have some metrics to hand - what does ‘good’ look like A few tips…

80. Be ready for questions like… What does a good result look like? How does the team know what to do next? What is the remediation effort involved? What can we get away with not doing? ( Ewww !) Should we have been expecting all criteria to be supported? Or even… 

81. Analysing results

82. Reporting issues Image created by DALL-E 3 via Bing Copilot What does good look like?

83. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion:

84. Google VPATS https://about.google/belonging/ accessibility-conformance-reports/ Microsoft VPATS https://www.microsoft.com/en-us/ accessibility/conformance-reports Apple VPATS https://support.apple.com/accessibility/vpat

85. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion:

86. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion: For example: Understanding the type of user for which an issue occurs Does the issue also match evidence/feedback from real users? Does the issue block an essential activity? Does the issue occur in a very frequent activity? Is a less common activity made just a bit harder? Are there workarounds?

87. https://nhsdigital.github.io/accessibility-checklist/checklist-full/

88. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion:

89. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion: Some criteria are simply more prominent! For example: 4.1 Use of Color (Level A) 4.3 Contrast (Minimum) (Level AA) 1.1 Keyboard (Level A)

90. Unexpectedness of failure Metrics to understand what ‘Good’ looks like Severity Level of effort required to resolve Perceived importance Number of criteria supported And for each criterion: Some criteria are simply more prominent! For example: 4.1 Use of Color (Level A) 4.3 Contrast (Minimum) (Level AA) 1.1 Keyboard (Level A) Other issues are reported as most problematic https://webaim.org/projects/screenreadersurvey10/#problematic

91. Conclusion

92. The VPAT… …is done It’s all over. Right? Image created by DALL-E 3 via Bing Copilot

93. The VPAT… Image created by DALL-E 3 via Bing Copilot …is NOT the goal!

94. What are the benefits of an accessibility audit?

95. What are the benefits of an accessibility audit?

96. What are the benefits of an accessibility audit?

97. Think big Image created by DALL-E 3 via Bing Copilot Look for your North Star

98. Think big Image created by DALL-E 3 via Bing Copilot Look for your North Star Be the change (The accessibility advocate, not accessibility police! )

99. Image created by DALL-E 3 via Bing Copilot Thank you! / rogerattrill

100. Image created by DALL-E 3 via Bing Copilot Questions? / rogerattrill