How to Measure Microsoft 365 Copilot Success and Manage AI Risk: Advanced Strategies for Secure, Compliant Adoption

Microsoft 365 Copilot: Measuring Success, Managing Risk
Presenter | Nikki Chapple, Microsoft MVP, Principal Cloud Architect, CloudWay
Event | South Coast Summit 2025
Format | Conference Breakout
Location | Farnborough International Exhibition & Conference Centre
Date | 18 October 2025

Are you ready to answer your CIO’s questions about the value, security, and compliance of Microsoft 365 Copilot and generative AI? This session gives IT leaders, architects, and compliance professionals a practical framework for measuring Copilot adoption, business value, and risk, moving beyond basic usage metrics.
Generative AI is transforming the workplace, but it brings new risks and unknowns. With 75% of knowledge workers using AI at work, and most bringing their own tools, organisations face major governance and compliance challenges. This session shows how to gain the visibility and insight needed to manage these risks and maximise value.

To measure Microsoft 365 Copilot and generative AI success, use a layered approach:

- For visibility, track shadow AI apps and traffic volume to identify unmanaged risk and prevent data leakage. Use Defender for Cloud Apps for shadow AI assessment.
- For insight, monitor sensitive prompts and risky usage trends to detect and mitigate compliance breaches. Use Purview Data Security Posture Management (DSPM) for AI to gain data security insights.
- For action, measure prompts per user, active days, retention, and licences to gauge adoption health and user engagement. Also, compare enabled versus active users and adoption growth rate to track rollout success and licence ROI. Use Microsoft 365 Admin Center for Copilot and agent usage, and adoption reports.
- For business value and risk reduction, assess time saved, behavioural change, and sentiment to prove ROI, productivity gains, and cultural adoption. Use Viva Insights for Copilot analytics and impact reporting.

This framework ensures you can demonstrate value, manage risk, and drive continuous improvement in your AI journey.

Why this matters:
With the rapid rise of generative AI, visibility is non-negotiable. You can’t manage what you can’t measure. The strategies and tools covered align with current trends in AI adoption, regulatory scrutiny, and the need for proactive, data-driven governance. Attendees will leave with actionable insights to drive adoption, reduce risk, and clearly demonstrate the business impact of Microsoft 365 Copilot and generative AI.

1. Microsoft 365
Copilot: Measuring
Success, Managing
Risk
Nikki Chapple, Microsoft MVP
Principal Cloud Architect, CloudWay
Data Security & Compliance Specialist

2. Nikki Chapple
Principal Cloud Architect, CloudWay
Microsoft MVP
All Things
M365
Compliance
LinkedIn
Blog

3. Agenda
Why Measure
Microsoft 365
Copilot and 3rd-
party Gen AI
Usage?
Microsoft 365
Admin Center
Viva Insights
Copilot Analytics
Purview DSPM
for AI
Summary

5. #SCS2025
You Can’t Manage What You Can’t Measure

6. Microsoft 365 Copilot:
Measuring Success,
Managing Risk
Business
Value & Risk
Reduction
Action
Insight
Visibility

7. Level KPI PillarWhat to MeasureWhy It Matters Where to Measure
Visibility
Safety &
Compliance
Shadow AI apps
discovered, traffic
volume
Identify unmanaged
risk and prevent data
leakage
Defender for Cloud
Apps →Shadow AI
Assessment
Insight
Safety &
Compliance
Sensitive prompts,
risky usage trend
Detect and mitigate
compliance breaches
Purview DSPM for AI
→Data Security
Insights
Action
Adoption &
Engagement
Prompts per user,
active days,
retention, licences
Gauge adoption
health and user
engagement
Microsoft 365 Admin
Center →Copilot &
Agent Usage Reports
Adoption &
Engagement
Enabled vs active
users, adoption
growth rate
Track rollout success
and licence ROI
Microsoft 365 Admin
Center →Copilot
Adoption Reports
Business
Value &
Risk
Reduction
Impact &
Value
Time saved,
behavioural
change, sentiment
Prove ROI,
productivity gains, &
cultural adoption
Viva Insights →Copilot
Analytics & Impact

8. From Insight to Action: Your Measurement
Toolkit
•Visibility
Defender for Cloud
Apps
•Insight
Purview DSPM for AI
Hub
•Action
Microsoft 365 Admin
Center
•Business Value & Risk ReductionViva Insights

9. Visibility -Defender
for Cloud Apps
Track Copilot and Gen AI interactions

10. View Gen AI App Catalog 1123 Apps

12. Assess the risk of an App

17. Discover what Gen Ai Apps are being used

21. Purview Data
Security Posture
Management for AI
(DSPM for AI)
Track Copilot and Gen AI interactions

22. DSPM for AI Copilot Experiences Reports
Prerequisites
Audit enabled
Copilot
licenses
assigned
Configuration
None
Optional
Sensitivity
labels
RBAC
Global reader
or Security
reader –view

23. DSPM for A:Out of The Box (OOB) Experience

24. DSPM for AI: OOB Copilot Reports

27. DSPM for AI: OOB Apps and Agents Reports

28. #SCS2025

29. #SCS2025

30. #SCS2025

31. #SCS2025

32. DSPM for AI: OOB Agents Reports
#SCS202
5

33. Problem: We Do Not See 3
rd
Party Gen AI
Activity

34. 75% of knowledge
workers have tried
or use AI at work
and 78% of those
AI users are
bringing their own
AI tools to the office
2024 Work Trend Index Annual Report from Microsoft and LinkedIn

35. Configure DSPM for AI for 3rd party Gen AI
Usage Reports (E5 Compliance)
Prerequisites
Audit enabled
Devices
onboarded
Purview Extension
deployed
Licensing
E5 Compliance
Copilot licenses
not required
One-Click
policies
eDLP Polices
IRM policies
Compliance
Admin –
configure policies
Security reader –
view
IRM –view Risky
user info

36. DSPM for AI: Prerequisites To See AI activity in
Microsoft Copilot, agents and other AI apps

37. DSPM for AI: Configure the One-Click Policies
to See More Insights

38. DSPM for AI: Track Other AI Interactions

39. Over 450 domains

40. DSPM for AI -Track Risky Behaviours

41. DSPM for AI: Track Agents

42. DSPM for AI: Activity Explorer

43. DSPM for AI: Activity Explorer Filters

44. Recap: DSPM For Microsoft Copilot, Agents,
and other AI apps
Activity
Reports
•Microsoft
Copilot and
agents
interactions
•Other AI
Apps
interactions
•Enterprise
app
interactions
Data Reports
•Sensitive
interactions
per AI app
•Unethical AI
interactions
•Sensitivity
labels
referenced
•Interactions
by
department
User Reports
•Insider Risk
Severity
•Insider risk
severity per
AI app
•Potential
risky AI
usage
Apps &
Agents
Reports
•Copilot
Experiences
•Copilot
Studio
•Enterprise
Apps
•Agents
Activity
Explorer
•User level
AI activity
•Prompt and
Responses

45. Microsoft 365 Admin
Center: Usage and
Adoption Reports
Copilot & Agent Usage and
Adoption Insights

46. M365 Admin Center: Copilot Control System

47. Copilot Control System -Adoption & Licencing Overview

48. Copilot Control System -Health
#SCS202
5

49. M365 Admin Center: Usage Reports –Copilot Readiness

50. #SCS2025

51. M365 Admin Center: Usage Reports –Copilot Usage
#SCS202
5

52. #SCS2025

53. M365 Admin Center: Usage Reports –Copilot
Agents
#SCS202
5

54. M365 Admin Center: Usage Reports –Copilot Chat
#SCS202
5

55. #SCS2025

56. M365 Admin Center: Adoption Score

57. M365 Admin Center: Adoption Score –AI Adoption
#SCS202
5

58. AI Adoption Score & Peer Benchmark

59. Quick Links to Adoption Resources
#SCS202
5

60. Send Organisational Messages
#SCS202
5

61. Copilot Adoption
#SCS202
5

62. AI Adoption –Current and Trend Information
#SCS202
5

63. #SCS2025

64. Recap: Microsoft 365 Admin Center Reporting
Copilot Control
System
•Recommendations
•Licencing
•Health
Usage Reports
•Microsoft 365
Copilot readiness
•Microsoft 365
Copilot usage
•Microsoft 365
Copilot Chat
usage (users
without M365
Copilot)
•Agent usage
Adoption Reports
•AI adoption Score
•Peer benchmark
•AI adoption
resources
•Organizational
messages
•Insights and Trend
Analysis
•Breakdown by
department

65. Copilot Analytics by
Viva Insights

66. Copilot Analytics -Features Depends on
Licences and Number of Users
Copilot Licences
Assigned
Viva Insights
Licences Assigned
Feature Availability
≥ 50 Copilot
licences
Not required Full dashboard features
Not applicable
≥ 50 Viva Insights
licences
Full dashboard features
< 50 Copilot
licences
Not required Limited dashboard features
0 Copilot licences
0 Viva Insights
licences
No dashboard data available

67. Historic: Microsoft Copilot Dashboard: In the
Viva Insights Teams App

68. Microsoft Copilot Dashboard Is Moving To
The Viva Insights Web App

70. Filters Reports
(Over 50 Copilot or
Viva Insights Licenses)

71. Microsoft Copilot Dashboard: Adoption Page

72. #SCS2025

74. Microsoft Copilot Dashboard: Impact Page
(Qualitative sentiment data)

75. Microsoft Copilot Dashboard: Impact Page

76. Microsoft Copilot Dashboard -Learning

77. Microsoft Copilot Dashboard: Delegate Access

78. Copilot Analytics

79. Copilot Adoption Power BI Report

80. Copilot Impact Power BI Report

81. Copilot Business Impact Power BI Report

82. Copilot Studio Agents Power BI Report

83. Copilot Analytics Custom Reporting –Five
Base Templates

84. Example Custom Report Using R and Python

85. Copilot Analytics: Shared Reports

87. Copilot Analytics Settings
AI Administrator
•Manage user access
•Manage access to Viva
Insight web app
•Delegate access to the
dashboard
•Set minimum group size
for the dashboard
•Create an exclusion list
Global Administrator
•Upload organizational
data for the dashboard
•Upload aggregated
survey results for the
dashboard

88. Summary

89. Continuous Improvement in AI Measurement
Measure
Analyse
Decide
Act
Validate
Defender for
Cloud Apps
Purview DSPM for
AI Hub
Microsoft 365
Admin
Viva Insights

90. Key Takeaways
Visibility is
Non-
Negotiable
Adoption
Metrics Go
Beyond
Licences
Risk
Management
Requires
Proactive
Monitoring
Business
Value Must
Be
Quantified
Continuous
Improvement
is Essential

91. FEEDBACK
SOUTHCOASTSUMMIT
ACLOUDTECHNOLOGYCOMMUNITYCONFERENCE #SCS2025

92. Thank you
•Please provide feedback

93. Resources
•Nikki Chapple | Microsoft MVP | Data Security + Governance
•All Things M365 Compliance -YouTube