Mainframe 3270 2 1960s Legacy Data Infrastructure is Not Fit for the Future
3270 Branch Teller System Branch Server ATM Tandem - Base24 Website Mainframe MQ ISO 8583 60’s - 90s Legacy Data Infrastructure is Not Fit for the Future
4 ATM Tandem - Base24 Contact Center Core enabled branches Internet Banking Mobile Banking Account Origination Enterprise Data Warehousing Liquidity Management Customer 360 CRM Doc Mgmt. Campaign Mgmt. Business Analytics ESB/Integration Layer Databases Mainframes MQs SOA Layer First decade of 2000s ... Legacy Data Infrastructure is Not Fit for the Future
Second decade of 2000s 5 ATM Tandem - Base24 Contact Center Core enabled branches Internet Banking Mobile Banking Microservices Ecosystem External API layer Customer microservice Product Notification Cross-Sell Payment Personalisation Customer 360 Gov Services ... ESB Layer Streaming Caching NoSQL Databases Mainframes MQs Data Lake ... Legacy Data Infrastructure is Not Fit for the Future
Your Enterprise Data Architecture is a GIANT MESS 6 LINE OF BUSINESSES ECOSYSTEM PUBLIC CLOUD
Implication: Brittle, Complex Interconnections LINE OF BUSINESS 01 LINE OF BUSINESS 02 PUBLIC CLOUD Data architecture is rigid, complicated, and expensive - making it too hard, time consuming and cost-prohibitive to digitally transform 7
... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Real-time Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
Paradigm for Data in Motion: Event Streams 9 Rich customer experiences Real-time events Real-time Event Streams A Sale A shipment A Trade A Customer Experience Data driven operations
Confluent Enables Endless Applications and Use Cases Hybrid & Multi-Cloud Messaging & Mainframe Modernization Streaming Analytics Event Driven Microservices CDC Patterns from Systems Of Records Corporate & Investment Banking, Capital Markets Trade Processing (Equities, FICC, Derivatives...) Real Time Payments and Payments Tracking Risk Analytics Market, Reference, & Security Master Data Distribution Trading System Integrations & Automation CTO - Technology Modernization Finance, Risk, Compliance, IT, Cyber Credit & Market Risk (CCAR, BCBS 239, FRTB ) OATS / CAT reporting Operational Log Hub IT Observability Cyber Security | SIEM Modernization Retail Banking, Wealth & Asset Management Fraud Detection Open Banking Customer 360 (omni channel banking, alerts & notifications) Client Advisor Workstations Data and Analytics for Asset Managers
Real Time Offers in Retail/Consumer Banking - Sample examples Line of Business Types of Offers Generalised benefits Credit Card Business Real-time location based offers Real-time increase in credit limit EPP plans at POS or e-POS E-commerce tranx and payment plan Increase in uptake of merchant offers. Increase in revenue from merchant promos Increase in revenues from more activity on credit cards Increase in revenue from merchant fees. Increase in NII Assets Pre-login loan/mortgage pages to actual conversion Open API based bundled offers on lending plus insurance Telesales gets to work on hot leads and higher conversion %. Higher fee based income due to insurance bundling. Unsecured lending marketplace participation especially for e-commerce transactions. Increase in NII Liabilities Real time fee based income from upsell/cross-sell offers on mobile and internet channels. Increase loyalty based transactions Capture brokerage and FX drop-offs in real-time and leverage other channels to capture opportunity Increase in fee based income Higher NPS and also better engagement scores. Covid allowed for higher FX and brokerage fees Lower account origination charges
Confluent - A Few Use Cases
Financial Use Cases - Customer Examples IT Modernization Mainframe offload, bridging to the cloud, microservices & improved developer velocity Customer Experience I ncrease digital engagement & improve customer experience Fraud Detection Account verification, integration across retail channels, real-time credit card detection Cyber Security High volume log ingest & SIEM optimization Regulatory Compliance Kafka serves as a commit-log and keeps record of change in state, access, and location for governance Market & Credit Risk Consolidates data across dozens of disparate risk systems
CUSTOMER BEHAVIOR Settlements Customer Segmentation Corporate Data Warehouse CORE SYSTEMS  CUSTOMER 360 Truth, understanding, and accelerated growth Social ATM Call Center Website Mobile Payments Wealth Management Retail Events Wholesale Events Market Data Trade Data Customer Com
Applied insights, event-driven bank or fund CUSTOMER 360 - Typical Architecture Pattern Kafka Event Bus Data L ake Consumers STREAMING PLATFORM SoR Business-Relevant Events Raw Events Producers Customer Communication Streaming Engine Rules | Models | Machine Learning Relevance Engine
High level logical design for real time offers ( CitiBank ) Core Banking Systems Credit Card Systems CIF Wealth mgmt systems MQ/CDC MQ/CDC MQ/CDC REST/json Data Lake/ Data Warehouse Merchant System/Static Data Model build & analytics system Marketing system - Hosting the offer palate Microservices Consumer Banking Apps Digital Services APIs Analytics Data Platform POS machines Transaction event capture Create Offer Update Offer take-up/ failure Cache/ Persistent Store - Capture timestamp, privacy, identify customer segment, identify earlier offer given, identifity location KSQL REST/json
Citibank Leverages Confluent for Global Event Streaming Platform
Business Outcomes for Citibank
Financial Use Cases - Customer Examples IT Modernization Mainframe offload, bridging to the cloud, microservices & improved developer velocity Customer Experience I ncrease digital engagement & improve customer experience Fraud Detection Account verification, integration across retail channels, real-time credit card detection Cyber Security High volume log ingest & SIEM optimization Regulatory Compliance Kafka serves as a commit-log and keeps record of change in state, access, and location for governance Market & Credit Risk Consolidates data across dozens of disparate risk systems
Mainframe Pain Points : High Cost of Operations: Mainframe backing user interactions Cost incurred for every user click All transactions written to mainframe Delayed Reporting Read transactions from Mainframe Read descriptions from other systems Reconcile overnight to serve user queries Confluent Solution: Mainframe Offload CQRS Pattern Writes to mainframe thru Kafka Reads from Kafka Majority of load (i.e. cost) moved off Mainframe Real Time reporting Read transactions from Kafka topic Read descriptions from another Kafka topic Simple microservices that reconciles transactions and descriptions to be served from search IT Modernization - Mainframe Modernization (Offloading)
IT Modernization - Mainframe Modernization Benefits 22
Kafka Kafka Core Messaging Applications Mobile Applications Access layer Confluent Schema Registry Kafka Connect Kafka Connect IBM MQ Queue HDFS MYSQL ODS layer IBM MQ Queue Solution Logical Architecture
IT Modernization - Microservices Fulfillment service Stock service Order service Return service Payment service UI service GUI Why build with Confluent: Completely decoupled microservices Single inter-communication standard Inherently fast - millions of calls/sec Asynchronous services development Stateful, distributed, and scalable platform Maintains version compatibility Distributed and highly scalable Process data in flight and real-time
Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Transaction Data Vendor Description Starbucks Coffee Walmart Blu-Ray Transaction Description Schema Microservices Client profiles Mainframe MIPS = $$ Mainframe offloading to reduce MIPS rate Legacy MQ Communication with App Kafka for decoupling between MQ and App Direct communication via Kafka (no MQ anymore)
AWS VPC Peering Hybrid Cloud Architecture Confluent Schema Registry Confluent Kafka “Central Data Source cluster” Data Center Replication Confluent Connectors Legacy Systems (Mainframes) Confluent Cloud VPC Private VPC Stream Processing Apps Confluent Cloud AWS Aurora RDBMS AWS Direct Connect Confluent Cloud PROD STAGING CDC AWS Fargate Stream Processing Apps AWS Fargate PROD STAGING / DEV AWS VPC Peering Confluent Replicator
British mutual financial institution, the seventh largest cooperative financial institution and the largest building society in the world with over 15 million members.A major provider of mortgages, loans, savings and current accounts in the UK and launched the first Internet Banking Service in 1997. Use Cases: -Mainframe offloading - Streaming ETL -Microservices Challenges Competition from digital first banks driving disruption and modernization Digital disruption efforts including open banking, regulatory requirements and expose data through APIs High and unpredictable data volumes, 24x7 SLA and availability requirements, Protect core Systems of Record (SORs) from the external loads / applications. Drive Cloud adoption and IT modernization strategy Solution Developed an event based real-time data platform on Confluent called “Speed Layer” Speed Layer - preferred source of data for high-volume read-only data requests and event sourcing. Delivered secure, near real time customer, account and transaction information from back end systems to front end systems with speed and resilience. Microservices architectures to onboard new use cases quickly and easily Maintain service availability despite unprecedented demand, agility and autonomy in digital development teams
World’s Leading Digital Bank Leverages Confluent for Unique Revenue generating experiences Producers ADA microservices digibank/ PayLah/ iWeb microservices ADA microservices Core SOI services cAPI microservices Digital Nervous System microservices ADA microservices microservices Connectors REST kSQL Consumers ADA microservices digibank/ PayLah/ iWeb microservices ADA microservices Core SOI services cAPI microservices Capabilities at a glance Real-time event based marketing on all-channels leveraging SAILOR & ADA platform. Microservices and API data request fulfillment using REST and Connector methods. Creating a foundation for a real-time digital nervous system for the bank. Benefits accrued Revenue improvement: Using the ADA and SAILOR platform real-time contextual offers on all channels. Improves offtake and conversion ratio on owned as well as served platforms. Real-time KYC & Fraud Management: Provides the capabilities to reduce fraud especially in the credit card area in real-time as opposed to end of day. Better risk mgmt. & fine grained control. Customer experience : Seamless experience on self-as well as served platforms through APIs. Eg. Buying insurance through wallet and servicing a cross-sell bundle through the same interaction. Regulatory position improvement : Suppression of notifications based on multiple dimensions and type of merchant transactions results in better regulatory observance of MAS & HKMA rules. Reduced cost of operations : Reduction in number of calls to mainframe and SOI has provided the bank greater scalability at lower per transaction costs.
Worlds best digital bank - Real Time contextual marketing Core Banking Systems (Finacle & Mainframe) Credit Card Systems CIF Wealth mgmt systems Avaloq MQ MQ/CDC SOI MQ Data Lake/ Data Warehouse (ADA/Hadoop) SAILOR Model build & analytics system Marketing system - Hosting the offer palate (UNICA) Microservices Consumer Banking Apps Digital Services APIs Analytics Data Platform POS machines Transaction event capture Create Offer Update Offer take-up/ failure Cache/ Persistent Store - Capture timestamp, privacy, identify customer segment, identify earlier offer given, identifity location KSQL REST/json Learning Path R, SAS, TensorFlow
World’s leading digital bank real-time offer platform Central Confluent Kafka Cluster Multiple Individual Apache Kafka Clusters Confluent Replicator (for Aggregation) Microservices Consumer Banking Apps Digital Services ML Platform Services Analytics Data Platform Kafka Connect Cluster Kafka Connect Cluster Multiple Sources RDBMS NoSQL Custom Apps Data Lake …..
Business Outcomes for world’s leading digital bank 1000’s of real-time contextual offers through served and self-owned channels resulting in improved revenue performance for retail wealth and distribution business. Improvement in liquidity position leveraging real-time data resulting in newer payment products on offer to institutional banking clients. Significant decrease in calls to Finacle reducing complexity and improving time to market. Reduction in cost and processing time for post trade settlement. Ability to introduce new security testing methods like chaos engineering and reduce false positives in overall log monitoring. This reduces security incidents and improves code coverage. Self service advanced analytics platform.
Payment System modernization ( Fintech Usecase )
Payment System modernization: Payments Model The future payments model will be measured by the totality of the customer relationship Speed + Scale Are critical as they signify volume Accessibility + Holistic Risk Management Have become baseline requirements given customer expectations Personalization + Insight Compete basis Differentiation in their Products and Service offerings Source: Global Payments Remade by Covid-19 , Deloitte
Opportunity for Improvement (Payment System modernization) Source: Global Payments Remade by Covid-19 , Deloitte
Vision Match Technical Debt & App modularity Evolving Regulatory Compliance need Launch of New Products & Services ? Reference Architecture - Future Payment Systems Source : McKinsey Financial Services Insights 1 Bidirectional Encoder Representations from Transformers. A type of deep learning model used to effectively summarize sequential data such as natural language and payment transactions 2 Banking Industry Architecture Network, is a banking standards body launched by several global banks in 2008. What should be my Key considerations for Modernization of Architecture?
Common Use Cases at a Glance Microservices Architecture Legacy Systems (MF, Database etc.) Modern Apps (Cloud Svcs, SAAS..) Legacy Messaging & Middleware (Tibco, MQ…) Data Sources Operational Data Modernization Apps Integration Messaging Modernization Confluent Platform (CP) Confluent Cloud (CC) Confluent Platform (CP) MS 1 MS 2 MS 3 MS 4 MS 5 Streaming ETL ETL RT Analytics RT Predictions Hybrid Cloud / Multi Region Hybrid Cloud / Multi Region Data Analytics Data Warehouse Modernization Analytics & Visualization Legacy / Modern Apps Integration Customer Systems Platforms (Payment, Trading, Banking, Lending, Market Data) Personalized Recommendations Promotion Next Best Offer Fraud Detection Data Lakes Business Analytics Reporting Open Banking Customer 360 Customer Communication Notification & Alerts Customer Activity Tracking - Web, branch, mobile… Regulatory Compliance Hub Cyber Security CUSTOMER 3RD PARTY Security Incident & Event Mgmt
Payment Gateway Instant Payments Anywhere, Anytime: Clearing & Settlement Branch Mobile Phone Internet Service App Channels Settlement Service Payment Gateway Clearing Clearing Request Clearing Notification Settlement Request Settlement Notification Balance Report Settlement Notification Balance Report Payer Payee Payment Gateway Branch Mobile Phone Internet Service App Channels
Adjustments Instant Payments Anywhere, Anytime: Reconciliation Customer makes payment 1 Calculate Differences 3 Update Balance 4 5 Bank File APP Bank statement 2 Cash book 2
Reference Data Microservices events Payments Accounts On-prem SaaS Apps Mobile Microservices Databases KYC Data warehouse Payment Portals Initiation Compliance Clearing Routing Confirmation Exception Fax/ Phone Files Mobile 3rd Party SWIFT Network ACH/BACS Network RIGS Network Card Network Payments Systems Direct connectivity to clearing and settlement networks ACH Wire RTGS Real-time Instant Cross-border Payment Instructions and Acknowledgements Bank System Interfaces End to End Payment processing Microservices - Pre-Processing, Processing & Distribution Payment format transformation(e.g. ISO20022) Integration with OFAC/FX etc. platforms Ingest, Aggregate, Enrich and Transform Payment Data in Real-Time Next Generation Payment Hub
Enable Real-time Fraud Scoring And Detection Payments Credit Card Transactions Debit Card Transactions Credit Card Applications Mortgage Applications Real-time Fraud Scoring ML Model Enhancement Fraud Case Management Historical Analysis
Real-time Pipeline for Fraud Detection Top 10 US Bank (>$25B+) 1 st Generation was a batch process using MapReduce, Python, MongoDB and alerts were too slow Customers now get alerts before putting card away Kafka used to queue events to avoid overwhelming downstream customer messaging system Future enhancements will involve AI for further personalization
Anomaly Detection: Aggregate data to identify patterns and anomalies in real-time 44 Aggregate data 1 … per 30-sec windows 2 CREATE TABLE possible_fraud AS SELECT card_number, COUNT (*) FROM authorization_attempts WINDOW TUMBLING (SIZE 30 SECONDS) GROUP BY card_number HAVING COUNT (*) >=3 EMIT CHANGES;
Detect unusual credit card activity https://developer.confluent.io/tutorials/credit-card-activity/confluent.html Understand user behavior with clickstream data https://developer.confluent.io/tutorials/clickstream/confluent.html Build customer loyalty programs https://developer.confluent.io/tutorials/loyalty-rewards/confluent.html Monitor security threats by analyzing and filtering audit logs https://developer.confluent.io/tutorials/audit-logs/confluent.html Create personalized banking promotions https://developer.confluent.io/tutorials/next-best-offer/confluent.html Automate instant payment verifications https://developer.confluent.io/tutorials/payment-status-check/confluent.html ksqlDB Examples
Rule 3 Aggregated Payment Amount to same Account > intra-day threshold Rule 2 Count of Payments to same Account > permissible threshold Rule 1 Check Diff Opening Balance and Closing Balance > intra-day threshold Source Account Transactions AML using Event Streaming 1 Create Transactions Stream ksqlDB Cdc Source Connector Source Payments App Create Payments Stream 1 2 Transaction Type Check Confluent Cloud 2 Link Payments to Accounts 3 Account Balance Calculation Schema Registry