CS 8-5_Audit and Control of Continuous Monitoring Programs and Artificial Intelligence Role (1).pdf

CS 8-5_Audit and Control of Continuous Monitoring Programs and Artificial Intelligence Role (1)

2. Audit and Control of Continuous
Monitoring Programs and
Artificial Intelligence Role

3. Tim Grace
CIA, CISA, CISM, CRISC, CDPSE, CISSP,
MBA/MIS
Tim is a leader in the Cyber Security, Privacy, Audit, and Compliance
field for CypraComp Lc. With more than 30 years of experience
leading efforts in the information technology, internal audit,
cybersecurity and governance, risk and compliance fields, working
within the financial services, healthcare, manufacturing, education,
insurance, retail and government industries. Tim has been a Chief
Audit Executive, Chief Information Security Officer / Technology
Officer, and graduate level university Professor. Tim has led the
implementation, buildout, audit, and assessment of numerous
organizations Governance, Risk and Compliance functions, including
internal audit. He holds an MBA / MIS, and has several industry
certifications including CIA, CISSP, CISM, CISA, CRISC, and
CDPSE.

4. More
My First Computer (80 bytes - about
20 words)
My First External Storage Device
electronic numerical integrator and
computer (ENIAC) Circa. 1946
My first set of work papers
External Storage Device (5MB) – Circa. 1956
Approximately 4GB of Punch Cards
Storage today –
2TB SD Cards
are now available
1946 - 80 bytes
2024 – 2TB – 2,199,023,255,552 bytes
1 PB - 1,125,899,906,842,624 petabytes

5. Continuous
Monitoring

6. Definitions
Continuous Monitoring - Maintaining ongoing awareness of financial, operational, and information technology vulnerabilities,
threats, anomalies, discrepancies, and controls to support organizational risk management decisions. A continuous monitoring
program defines, establishes, implements, and operates the various aspects of an organizations risk process to provide the
information necessary to make risk-based decisions regarding organizational status at all levels.
Effectiveness - To be effective, a continuous monitoring program should have the following elements:
•Addresses the assessment of controls for design, effectiveness and status monitoring.
•Promotes the concept of near real- time risk management and ongoing authorization of the control process through implementation
of robust, organization-wide continuous monitoring processes.
•Incorporates processes to ensure that response actions are taken in accordance with findings and organizational risk tolerances
and that they have the intended effects.
Comprehensiveness - A comprehensive continuous monitoring program serves as a risk management and decision support tool
used at each level of an organization. Strategies and business objectives at the organizational level direct activities needed at the
mission and business levels as well as the financial, operational and technology level functions implemented in support of continuous
monitoring.
Adapted from the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-137A – “Assessing Information Security Continuous
Monitoring (ISCM) Programs” May 2020

7. Fundamentals of a Continuous Monitoring
Program
Define the organization-wide continuous monitoring strategies, based on
organizational risk tolerance, that maintain clear visibility into assets,
awareness of vulnerabilities, up-to-date threat information, and
mission/business impacts.
Establish the organization-wide continuous monitoring program, determine
metrics, status monitoring frequencies, control assessment frequencies, and
the continuous monitoring technical architecture.
Implement the continuous monitoring program and collect the related
information required for metrics, assessments, and reporting. Automate
collection, analysis, and reporting of data where possible.
Analyze the data collected, report findings, and determine the appropriate
response. It may be necessary to collect additional information to clarify or
supplement existing monitoring data.
Respond to findings with technical, management, and operational risk-
mitigating activities, or accept, transfer/share, or avoid/reject the risk.
Review / Update the continuous monitoring program, adjusting the strategy at
the applicable level, and mature measurement capabilities to increase
visibility into assets and awareness of vulnerabilities, further enable data-
driven control of the organization’s information infrastructure, and increase
organizational resilience.
Adapted from the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-137A – “Assessing Information Security Continuous
Monitoring (ISCM) Programs” May 2020

8. Potential Benefits of Continuous Monitoring
Programs
•Timely identification of problems, weaknesses and risks.
•Quick corrective action, which can help reduce the cost of any required periodic financial, regulatory, and operational reviews.
•Cost reduction.
•Continuous monitoring supports:
oContinuous audit
ocontinuous controls monitoring
ocontinuous transaction inspection
•Examination of 100% of transactions and data processed in different systems, applications and databases.
•Can reduce/detect fraud.
•Can reduce inconsistencies, duplications, errors, policy violations, missing approvals, incomplete data, dollar or volume limit
errors, or other possible breakdowns in internal controls.

9. Systems Where Continuous Monitoring Can Be
Used
•Accounts Receivable
•Accounts Payable
•Travel and Expense
•Payroll
•Payment/Purchasing Cards
•Inventory Transactions/Systems
•Bank Accounts/Records
•Human Resource Systems
•Learning Management Systems
•Detection and Response (XDR)
•Security Information and Event Monitoring (SIEM)
•Security Orchestration, Automation and Response (SOAR)
•Physical Security
•HVAC Equipment
•Environmental Control Systems
•Access Control Systems
•Medical Devices
•Mobile Devices
•Industrial Control Systems
•Supervisory Control and Data Acquisition (SCADA)
•Business Applications
•Closed Circuit Television (CCTV)
•Identity and Access Management
•Security Awareness Systems
•Third-Party Vendor Systems
•Stock Brokerage Systems

10. Risk Management Framework (RMF)
Determine control design and
effectiveness
ASSESS
Target Controls
Define criticality/sensitivity of
information according to
potential worst-case, adverse
impact to mission/business.
CATEGORIZE
Target Process
Continuously track changes to the
enterprise/organization that may
affect risk and reassess control
design and effectiveness.
MONITOR
Target State
AUTHORIZE
Information System
Determine risk to organizational
operations and assets, individuals,
other organizations, and the Nation;
if acceptable, authorize operation.
Implement controls within the
enterprise/organization using sound
control practices.
IMPLEMENT
Target
Controls
SELECT
Target
Controls
Select baseline controls; apply
tailoring guidance and supplement
controls as needed based on risk
assessment.
Audit Efficiency – Utilizes financial, operational,
and technology control baseline frameworks,
focusing on processes and controls that are critical
components to financial statement audit success
.
Operational Efficiency – Automation of the
process will provide executive leadership the
necessary information to make cost-effective, risk-
based decisions with regards to the enterprise /
organization.
Cost/Resource Savings – Continuous monitoring
process promotes reciprocity and reuse of test
results and assessment documentation. Ensures
leveraging of numerous industry mandates,
regulations, and best practices
Process Improvement / Optimization -
Incorporates consistent implementation and
assessment of controls to create resource and time
efficiencies (common controls). Implements a risk-
based mission and risk decisions throughout the
lifecycle process.
PREPARE
process Initiation
Execute the RMF from an
organization / functional level
perspective.
Adapted from the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-37, Revision 2 - “Risk Management Framework for
Information Systems and Organizations” December 2018

11. Continuous Auditing

12. Definitions
Continuous Monitoring - a management process that monitors on an ongoing basis whether controls are operating effectively. It is
used to detect compliance and risk issues associated with an organization’s technology, financial and operational environment.
Continuous Auditing - the combination of technology-enabled ongoing risk and control assessments. Designed to enable the
internal auditor to report on subject matter within a much shorter timeframe than under the traditional retrospective approach. An
automated method used to perform auditing activities on a more frequent basis.
Continuous Assurance - performed by internal audit, continuous assurance is a combination of continuous auditing and testing of
first and second lines of defense continuous monitoring.
Continuous Reporting - the release of financial and non-financial information on a real-time or near real-time basis. The purpose is
to allow external parties access to information as underlying events take place, rather than waiting for end-of-period reports.

Ongoing Control Assessment - the ongoing evaluation of internal controls against a baseline condition and subsequent changes to
control configurations, through the use of technology-based audit techniques.
Ongoing Risk Assessment - the ongoing identification and assessment of risks to the achievement of business objectives through
the use of technology- based audit techniques.
Technology-based Audit Techniques – any automated audit tool, such as generalized audit software, test data generators,
computerized audit programs, specialized audit utilities, and Computer-assisted Audit Techniques (CAATs).
Reference – Global Technology Audit Guide (GTAG) 3 – “Continuous Auditing: Coordinating Continuous Auditing and Monitoring to Provide Continuous
Assurance” 2
nd
Edition, March 2015

13. Foundational Continuous Assurance Framework
Reference – Global Technology Audit Guide (GTAG) 3 – “Continuous Auditing: Coordinating Continuous Auditing and Monitoring to Provide Continuous
Assurance” 2
nd
Edition, March 2015
Ongoing risk and control assessments enabled by
technology- based audit techniques:
•Generalized audit software
•Spreadsheet software
•Scripts
•Specialized audit utilities
•CAATs
•Commercial packaged solutions
•Custom developed production systems
Flexible and scalable to ensure optimization of: •Timely identification of exceptions and anomalies
•Analysis of patterns and trends
•Detailed transaction analysis against cut-off thresholds
•Testing of controls
•Comparative analysis among peers
When deployed effectively:
•Focus is on audit objectives and assertions
•Detection of emerging risks and control weaknesses

14. Continuous Auditing/Continuous Monitoring
Relationship
Reference – Global Technology Audit Guide (GTAG) 3 – “Continuous Auditing: Coordinating Continuous Auditing and Monitoring to Provide Continuous
Assurance” 2
nd
Edition, March 2015
There is an inverse relationship between continuous auditing and continuous monitoring. All three
lines of defense contribute to measuring and strengthening the effectiveness of risk management
and control. Internal audit should adjust the extent of its continuous auditing work based on the
adequacy and consistency of the continuous monitoring management deploys. If continuous
monitoring deployed by the first and second lines of defense is lacking or inconsistent, internal audit
should increase its continuous auditing efforts accordingly.
In areas where management has not implemented continuous monitoring, auditors should extend
detailed testing using continuous auditing techniques. Where the first or second line of defense
performs continuous monitoring on a comprehensive basis across end-to-end business process
areas, internal audit may not need to perform the same detailed techniques as would otherwise be
applied under continuous auditing. Instead, auditors should perform procedures to determine
whether the continuous monitoring process is reliable. Such procedures include:
•Review of detected anomalies and management’s response
•Review of management’s resolve to enact and sustain remediation
•Review and testing of controls over continuous monitoring process itself, such as:
•Security
•Change control
•IT operations

15. Practical Applications for Continuous Auditing
Reference – Global Technology Audit Guide (GTAG) 3 – “Continuous Auditing: Coordinating Continuous Auditing and Monitoring to Provide Continuous
Assurance” 2
nd
Edition, March 2015

16. Continuous Auditing Implementation
Reference – Global Technology Audit Guide (GTAG) 3 – “Continuous Auditing: Coordinating Continuous Auditing and Monitoring to Provide Continuous
Assurance” 2
nd
Edition, March 2015
The sequence of activities may vary, and other
activities not identified may need to be
performed when developing continuous
auditing to support a specific audit.
Examples of Continuous Auditing Indicators:
•Collection / analysis of data supporting key
business processes and high-risk areas.
•Collaborate with business owners and IT
professionals to develop risk indicators.
•Leverage risk assessment results.
Key Risk Indicators could include:
•Focus on the extent of change
experienced by the entity over time.
•Process- based leading indicators and
symptomatic lagging indicators.
•Identify a sufficient number so when
routinely compared will isolate outlier
entities.

17. •Used as a data analysis process - auditors report errors and exceptions and do not tie these back to risks and control
weaknesses.
•Initial cost setup can be expensive – time, resources, process creation.
•Lack of ‘C’ Level buy-in. Who wants to be continuously audited?
•Management looks to the Internal Audit team to find the risks.
•Lack of follow up
Failures of a Continuous Auditing program
What else?

18. History – Continuous Auditing
The first application of continuous auditing was developed at AT&T Bell Laboratories in 1989. Known as a continuous process auditing
system (CPAS), the system provided measurement, monitoring, and analysis of the company's billing information. Key concepts such
as metrics, analytics, and alarms pertaining to financial information were also introduced. – Source - Wikipedia

19. History – Continuous Monitoring
Continuous monitoring can be traced back to its roots in traditional auditing processes. It goes further than a traditional periodic
snapshot audit by putting in place continuous monitoring of transactions and controls so that weak or poorly designed or implemented
controls can be corrected or replaced sooner rather than later. – Source - Wikipedia

20. Artificial Intelligence

21. The theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual
perception, speech recognition, decision-making, and translation between languages.
Definition
AI is the backbone of innovation in modern computing, unlocking value for individuals and businesses. For example, optical
character recognition (OCR) uses AI to extract text and data from images and documents, turns unstructured content into business-
ready structured data, and unlocks valuable insights.
4 main types of artificial intelligence
•Reactive machines - AI systems that have no memory and are task specific, meaning that an input always delivers the same
output. Examples – Machine Learning Models, IBM’s Deep Blue, Netflix’s recommendation’s engine
•Limited memory machines – Imitates the way a human brains’ neurons work. It gets smarter as it receives more data. Examples
– Self Driving Cars
•Theory of mind – Theoretical concept whereas the AI could have the potential to understand the world and how other entities
have thoughts and emotions. Which would mean it will affect how and what they believe based on the world around them.
Examples – Empathy, truth vs lies, knowing of human emotions. (Data from Star Trek The Next Generation).
•Self-awareness – Systems designed with a sense of self awareness, a conscious understanding or their existence. I am hungry
or I like to pepperoni on my pizza, and I hate mushrooms. (also Data from Star Trek).

22. •Chatbots
•Language Translation
•Personal Assistants
•Virtual Assistants
•Navigation and Travel
•Social Media
•Health Monitoring
•Banking and Finance
•Shopping
•Marketing
•Entertainment
•Transportation
•Image Recognition
•Education
•E-Commerce
Current Artificial Intelligence Applications
•Fraud Detection
•Autonomous Vehicles
•Customer Service Help
•Facial Recognition
•Medical Diagnosis
•Investing
•Administrative Tasks
•Creating Content
•Voice assistance
•Voice Response
•Personalized Learning
•Spam Filters
•Robotics
•Cleaning Offices
•Cleaning Large Equipment
•Human Resources
•Inventory Management
•Agriculture
•Gaming
•Astronomy
•Space Travel
•Data Security
•Identification of Threats
•Flaw Identification
•Threat Prevention
•Threat Response
•Shipping
•Traffic
•Route Planning
•Ride-Sharing
•Supply Chain
•Car Insurance
•Inspections
•Quality Control
•Music and Media
Streaming
•Smart Speakers
•Video Games
•Physical Security
•Surveillance
•Adaptive Battery
Charging
•Journalism
•Legal Research
•Fixing Tim’s PowerPoint
Error’s

23. What Artificial
Intelligence
Applications should
Auditors be
concerned with?

24. •Chatbots
•Language Translation
•Personal Assistants
•Virtual Assistants
•Navigation and Travel
•Social Media
•Health Monitoring
•Banking and Finance
•Shopping
•Marketing
•Entertainment
•Transportation
•Image Recognition
•Education
•E-Commerce
All of the Above
•Fraud Detection
•Autonomous Vehicles
•Customer Service Help
•Facial Recognition
•Medical Diagnosis
•Investing
•Administrative Tasks
•Creating Content
•Voice assistance
•Voice Response
•Personalized Learning
•Spam Filters
•Robotics
•Cleaning Offices
•Cleaning Large Equipment
•Human Resources
•Inventory Management
•Agriculture
•Gaming
•Astronomy
•Space Travel
•Data Security
•Identification of Threats
•Flaw Identification
•Threat Prevention
•Threat Response
•Shipping
•Traffic
•Route Planning
•Ride-Sharing
•Supply Chain
•Car Insurance
•Inspections
•Quality Control
•Music and Media
Streaming
•Smart Speakers
•Video Games
•Physical Security
•Surveillance
•Adaptive Battery
Charging
•Journalism
•Legal Research
•Fixing Tim’s PowerPoint
Error’s

25. Continuous
Monitoring and
Artificial Intelligence

26. An organization that is planning to implement Artificial Intelligence
of any kind should consider
•Compliance Environment (Regulations, Laws, Industry
Standards, Privacy, Frameworks)
•Organizational Policies, Standards, and Procedures
•Technology in use
•Governance Structure
•Risk Management
•Mission of the organization
•Business Needs
•Project Management
•Accountability
•In other words, treat Artificial Intelligence projects like any other
project
Auditing Artificial Intelligence
Attending to the Risk of Artificial Intelligence
•Dependence
•Privacy
•Bias and Discrimination
•Ethics
•Security (logical & physical)
•Transparency
•Malicious Use
•Misinformation and Manipulation
•Compliance
•Job Displacement

27. • Chatbots
• Language Translation
• Personal Assistants
• Virtual Assistants
• Navigation and Travel
• Social Media
• Health Monitoring
• Banking and Finance
• Shopping
• Marketing
• Entertainment
• Transportation
• Image Recognition
• Education
• E-Commerce
• Fraud Detection
• Autonomous Vehicles
• Customer Service Help
• Facial Recognition
• Medical Diagnosis
• Investing
• Administrative Tasks
• Creating Content
• Voice assistance
• Voice Response
• Personalized Learning
• Spam Filters
• Robotics
• Cleaning Offices
• Cleaning Large Equipment
• Human Resources
• Inventory Management
• Agriculture
• Gaming
• Astronomy
• Space Travel
• Data Security
• Identification of Threats
• Flaw Identification
• Threat Prevention
• Threat Response
• Shipping
• Traffic
• Route Planning
• Ride-Sharing
•Supply Chain
•Car Insurance
•Inspections
•Quality Control
•Music and Media Streaming
•Smart Speakers
•Video Games
•Physical Security
•Surveillance
•Adaptive Battery Charging
•Journalism
•Legal Research
•Fixing Tim’s PowerPoint Error’s
Anything that can be automated can be continuously audited & anything that can be automated will have metrics
Continuous Auditing of Artificial Intelligence
Systems

28. Discussion Questions
Where do you see AI being continuously
monitored in your industry?
How can AI help with the Audit process?
How do you think AI will change the face
of Audit in the future?

29. Tim Grace
[email protected]
314 623 8216
Thank you for your
time