Building a Scalable Event-Driven Architecture for Open Finance Brasil by Thiago Maciel, Thiare Costa, and Thomas Rabelo

Learn about PCM's journey to an event-driven, serverless architecture that empowers Open Finance Brasil to process billion daily reports with cost efficiency using AWS SQS, DynamoDB, S3, Firehose, ElasticCache, EKS with KEDA and Karpenter, Node.js, and Nest.js.

1. A ScyllaDB Community

Building a Scalable Event-Driven
Architecture for Open Finance Brasil

Thiago Maciel
Senior Software Engineer
Thiare Costa
Head of Data and Engineer
Thomas Rabelo
Tech Lead

2. Thomas Rabelo (he/him)
■16+ years in software engineering and technical
leadership
■Specialized in software architecture and distributed
systems systems
■Academic contributor with scientific publications and
teaching experience
■Open Finance Brasil Tech Lead

3. Thiago Maciel (he/him)
■15+ years building enterprise-scale solutions
■Full-Stack technical expertise
■Specialized in high-performance distributed systems
■Cross-industry experience in major Brazilian sectors
■Focus on scalability, resilience and cost-effectiveness

4. Thiare Costa (he/him)
■18+ years in financial industry and banking
■Specialized in leading Data Platforms teams
■Focused on high-performance distributed systems
■Expert in agile methodologies and system design
■Passionate about inspiring leadership and talent
development

5. ■Open Finance Brasil Context
■Problems Faced
■Rethinking the Architecture
■Results & Achievements
■Conclusion
■Next Steps
Presentation Agenda

6. Open Finance Brasil Context

7. ■Largest Open Finance system globally
■30M+ customers sharing financial data
■900+ participating institutions
Open Finance Brasil: A Financial Revolution




■Foster competition and reduce banking
spreads
■Democratize access to financial products
■Digitize the financial system for efficiency
and innovation

8. Ensure quality and reliability in data
sharing

Create the most stable and functional
Open Finance in the world

Open Finance Brasil’s Vision

9. Metadata Collection Platform
(PCM)

10. The Heart of Open Finance Brasil Monitoring
What is PCM?
■Central monitoring platform for the Open Finance ecosystem
■Collects metadata from all financial institutions' interactions
■Monitor efficiency and compliance of data sharing APIs

Strategic importance
■Critical infrastructure for ecosystem stability
■Enables regulatory oversight by Banco Central do Brasil (Central Bank)

11. Problems Faced

12. Old Architecture: The Perfect Storm
PostgreSQL as
the primary
storage for all
history data
Dual caching
layer with
MongoDB &
Redis
RabbitMQ stream
& poor queue
design
No reliance on
built-in cloud
services
Mixed
technology
stack
High operation &
infrastructure costs

13. The Case for Change
Actual Demand
System Performance Threshold
?????? 500M+ reports never processed
⚠ Up to 100M reports handled daily
?????? Critical business insights impacted

14. Rethinking the Architecture

15. Architecture Decision Drivers
Reduce Costs
■Eliminate unnecessary services
■Optimize resource usage
■Pay-per-use model
Keep it Simple (KISS)
■Minimize complexity
■Reduce failure points
■Easier to debug and maintain
Team centric Design
■Design, build and care by 3 Engineers
■Less cognitive load
■Focus on business value

16. Architecture: Overview

17. Architecture: EDA

18. Scalability: EKS + Karpenter + Keda

19. Right tool for the right job
Old Architecture
❌ Multiple databases
PostgreSQL + MongoDB + Redis
❌ Complex Data Management
Multiple sync points
Redundant storage
❌ High Operational
Overhead
Multiple maintenance points
Complex backup strategy
Resource-intensive
New Architecture
✅ Purpose-built Solutions
DynamoDB + S3
✅Simplified Data Flow
Clear data lifecycle
Cost-effective storage
✅Optimize Operations
Managed services
Native scalability
Pay-per-use model

20. Results & Achievements

21. Results
10x
Increase in daily
reports
(100 million to 1
billion)
70%
Cost reduction
($7 to $2)
per million
reports
Eliminated
Processing
bottlenecks

Observability
FinOps & SRE

22. Message Volume
1.5B messages/day
└ Equivalent to ~17,000 messages
per second



PCM by Numbers
DynamoDB Load
50M requests/day
└ Supporting critical business operations
Data Processing
1.4 TB daily firehose
└ Comparable to 580 HD movies
per day
Request Rate
Peak: 22K requests/second
Avg: 12K requests/second
└ Consistent high-performance delivery

23. Conclusion

24. Unified technology stack simplifies operations
Event-driven architecture enables scalability and resilience
Dynamic scaling (e.g, Keda, Karpenter) optimize costs
Integrated observability ensures operational excellence
Conclusion

25. Next steps

26. Open Finance expansion with new services (e.g, Pix by proximity,
credit portability)
Anticipated growth to 2 billion reports/day by 2025
PoC of Temporal IO for:
■Simplifying distributed systems
■Ensuring reliability in workflows
■Reducing operational complexity
Future steps

27. Stay in Touch
Thiago Maciel
[email protected]
https://www.linkedin.com/in/thiagomaciel/
Thiarê Costa
thiare.costa@openfinancebrasil.com.br
https://www.linkedin.com/in/thiare/
Thomas Rabelo
[email protected]
https://www.linkedin.com/in/thomas-rabelo-13632339/