Optimizing Cloud Native Workflows with Even-Driven Architecture.pdf

This is the presentation which was held on AWS Community Day Bangladesh 2024.

1. Optimizing Cloud Native Workflows
With Event-driven Architecture
Md. Mostafa Al Mahmud
AWS Community Builder (Serverless)
Senior Software Engineer, Cogwheel Analytics


@ 2024, Md. Mostafa Al Mahmud

2. https://mdmostafa.com/
https://www.linkedin.com/in/md-mostafa/
https://twitter.com/md__mostafa


Senior Software Engineer
Cogwheel Analytics
Md. Mostafa Al Mahmud
@ 2024, Md. Mostafa Al Mahmud

3. Worked in the projects of some valuable brands
@ 2024, Md. Mostafa Al Mahmud

4. Community Activities
@ 2024, Md. Mostafa Al Mahmud

5. Agenda

●Understanding EDA: overview and key concepts
●Core benefits of EDA and it’s benefits
●Exploring Common usage patterns in EDA
●Orchestration vs Choreography: Choosing the Right
Approach
●Essential Considerations for Building an EDA app
●Best practices
●Demo - Error Handling Strategies
●Helpful resources for Further Learning
@ 2024, Md. Mostafa Al Mahmud

6. EDA - Powering Agility and Scalability

●Events are inclusive in modern applications
●EDA drives innovation across industries
●A central point to modern applications
●Communication between services and
development teams
●Many comprehensive toolkits availability
@ 2024, Md. Mostafa Al Mahmud

7. Core Values of Event-Driven Architectures

●Independent feature build and
development
●Enhanced Scalability and Resilience
●Effortless feature integration in the current
app
●Loose coupling and modularity
●Asynchronous processing
@ 2024, Md. Mostafa Al Mahmud

8. Why to choose EDA

●Real-time reactivity
●Effortless scalability
●Targeted communication
●Simplified integration
●Enhanced security and privacy
●Optimized resource utilization
@ 2024, Md. Mostafa Al Mahmud

9. Key Concepts of Event-Driven Architectures
There are several building blocks
●Events: Signals of state changes
●Immutable and observable
●Producers: Publish events
●Consumers: Downstream components that
react to events
●Event Brokers: Centralized mediate
communication
○Event Routers: Push events to targets
○Event Stores: Consumers pull events
@ 2024, Md. Mostafa Al Mahmud

10. Tight Coupling vs. Loose Coupling

●Drawbacks of tight coupling
○Development Challenges
○Scalability issues
○And so on
●E-commerce Example: Tightly coupled
services (orders, billing, shipping, inventory)
create a fragile flow
@ 2024, Md. Mostafa Al Mahmud

11. Tight Coupling vs. Loose Coupling

●Loose coupling: the power of Events
@ 2024, Md. Mostafa Al Mahmud

12. Idempotency

●Have no unintended consequences upon
retries
●Crucial concept for EDA due to inherent retry
mechanisms
●Techniques for achieving idempotency:
○Building idempotent services.
○Utilizing idempotency keys in events.
@ 2024, Md. Mostafa Al Mahmud

13. Common Patterns of EDA

Point-to-Point Messaging
●Producers send messages to a single
consumer.
●Messaging queues serve as event
brokers.
●Messages persist until consumed,
ensuring reliability.
●"Dumb pipes" in microservices
communication
@ 2024, Md. Mostafa Al Mahmud

14. Common Patterns of EDA

Publish-Subscribe Messaging
●Producers send the same message to one
or many consumers.
●Utilizes event routers instead of queues.
●Generally lacks event persistence.
●Event bus (another event router type)
@ 2024, Md. Mostafa Al Mahmud

15. Common Patterns of EDA

Event streaming
●Continuous flows of events or data
●Consumers typically poll for new events
●Event streams can be processed
individually or collectively over time.
●Data streams interpret data over time,
often used for real-time analytics or data
persistence.
@ 2024, Md. Mostafa Al Mahmud

16. Common Patterns of EDA
@ 2024, Md. Mostafa Al Mahmud

17. Common Patterns of EDA

Choreography
●Ideal for communication between Bounded
Contexts.
●Producers focus on event delivery
(fire-and-forget).
●Event schema ensures message clarity.
●Reduces dependencies between contexts
(loose coupling).
●Use EventBridge as event bus
https://martinfowler.com/bliki/BoundedContext.html

@ 2024, Md. Mostafa Al Mahmud

18. Common Patterns of EDA

Orchestration
●Manages service integration sequence
●Maintains application state for
complex workflows
●Handles errors and retries
●Use AWS Step Functions, Amazon
MWAA
@ 2024, Md. Mostafa Al Mahmud

19. When to Use Choreography and Orchestration Together
@ 2024, Md. Mostafa Al Mahmud

20. Understanding Combining Patterns
Fan-out Pattern
Distributing a
single event to
multiple
subscribers
@ 2024, Md. Mostafa Al Mahmud

21. Understanding Combining Patterns
Event Filtering &
Routing Pattern
Directing events
to specific
targets
@ 2024, Md. Mostafa Al Mahmud

22. Understanding Combining Patterns

Event and
Message
Buffering
@ 2024, Md. Mostafa Al Mahmud

23. When to Use Combining Patterns
https://aws.a
mazon.com/bl
ogs/compute/i
ntroducing-th
e-amazon-ev
entbridge-ser
vice-integratio
n-for-aws-ste
p-functions/

Workflow
Orchestration
@ 2024, Md. Mostafa Al Mahmud

24. Step Functions
●Serverless orchestration service for EDA
workflows
●Visually define workflows
●Workflow components:
○State machines: entire workflows
○States: individual steps
○Task states: utilize other AWS
services (e.g., Lambda functions)
to perform tasks
●Standard vs. Express workflows
https://catalog.us-east-1.prod.workshops.aws/wo
rkshops/9e0368c0-8c49-4bec-a210-8480b51a34
ac/en-US/development/error-handling/step-4
@ 2024, Md. Mostafa Al Mahmud

25. Step Functions - Use Cases
Dynamic Parallelism
Function orchestration
Error handling Parallel processing
Human in the loop
Branching
https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
@ 2024, Md. Mostafa Al Mahmud

26. Step Functions - Real-life Use Cases
●Data Processing Workflows
○Orchestrate end-to-end ETL processes, coordinating different sources and destinations
○Refine and reduce large data sets into useful formats, run multi-step analytics, and ML workflows
●DevOps and IT Automation
○Build tools for CI/CD pipelines
○Create event-driven applications that automatically respond to changes in infrastructure
○Synchronize data between source and destination S3 buckets
●E-commerce Automation
○Automate e-com business processes, such as order fulfillment and inventory tracking, etc.
●Web Application Workflows
○User registration processes, sign-on authentication, and so on
●Media Processing
○Extract data from PDF or images for processing
○Build a serverless video transcoding pipeline using Amazon MediaConvert
@ 2024, Md. Mostafa Al Mahmud

27. Demo
Error Handling
@ 2024, Md. Mostafa Al Mahmud

28. Best Practices of EDA

●Event identifications with event storming
●Naming convention
●ECST events
●Notification events
●Conformist pattern
●ACL
●OHS
●Event-first thinking
●Idempotency
●Ordering (order/unorder events)
@ 2024, Md. Mostafa Al Mahmud

29. Resources

●https://aws.amazon.com/blogs/compute/introducing-the-amazon-eventbridge-service-integration-for-a
ws-step-functions/
●https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is-how-it-works-concepts.html
●https://d1.awsstatic.com/SMB/aws-modernization-intro-to-eda-guide-2022-smb-build-websites-and-ap
ps-resource.pdf
●https://aws.amazon.com/event-driven-architecture/
●https://theburningmonk.com/2020/08/choreography-vs-orchestration-in-the-land-of-serverless/
●https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
●https://catalog.us-east-1.prod.workshops.aws/workshops/9e0368c0-8c49-4bec-a210-8480b51a34ac/e
n-US/development/error-handling
●https://d1.awsstatic.com/psc-digital/2023/gc-300/build-eda-on-aws/Build-EDA-AWS-eBook.pdf
●https://aws.amazon.com/step-functions/use-cases/
Resources
@ 2024, Md. Mostafa Al Mahmud

30. Be connected with me
@ 2024, Md. Mostafa Al Mahmud

31. Thank you
@ 2024, Md. Mostafa Al Mahmud