Route It Like It’s Hot: Scaling Payments Routing at American Express by Benjamin Cane

Join the American Express Payment Acquiring and Network team as they share insights on building their Global Transaction Router, which powers payment routing at Amex scale. Learn how they design, build, and operate it to handle record-breaking shopping days, ticket sales, and unpredictable demand.

1. A ScyllaDBCommunity
Route It Like It’s Hot
Scaling Payments Routing at American Express
Benjamin Cane
Distinguished Engineer

2. ▪Distinguished Engineer at American Express.
▪I work on our core payments platforms.
▪Throughout my career I’ve held roles in both
infrastructure and software engineering.
▪Building fast, scalable, and reliable distributed
systems is my passion.
Introductions: Benjamin Cane

3. In 2018, American Express started an
initiative to rebuild its payment network
from the ground up.
We wanted to build a platform that could
adapt to the future of payments.
We designed the system to be Cloud-ready,
Flexible, Secure, Resilient, Low-Latency,
and Scalable.
Payment Network Modernization

4. An essential component of this new network is the Global Transaction Router. It manages
connectivity and routes payment transactions to and from partner financial institutions. Of
course, it must perform this role at the scale of American Express Payments.
American Express Global Transaction Router

5. Sitting at the edge of our Payment Network presents
several challenges.
Challenges:
▪Managing and load balancing long-lived TCP sessions.
▪Instant, rapid-fire, asynchronous ISO8583 messages.
▪ISO8583 is an international standard for payment
messaging.
▪Unexpected behavior and traffic spikes.
5
Unique Challenges

6. Design
SECURE, RESILIENT, LOW-LATENCY & SCALABLE

7. There is a lot to like about Go as a language. It’s easy to
learn, has excellent tooling, has a strong standard library
(especially for networking-based applications), and is a
language built for software engineering.
It also fits nicely with challenges commonly faced
by Payments Platforms:
▪Concurrency: Enables us to handle high volumes of
connections and transactions.
▪Ahead-of-time-compilation: Enables us to be ready to
handle volume immediately without warm-up time.
7
Design: Selection of Go

8. While we support multiple protocols and message
formats externally (like any good payment switch)
internally, we opted for gRPC.
We selected gRPC because it offers the convenience of
a modern protocol without sacrificing performance.
▪Protobuf: Binary format is more efficient during
transmission and serialization.
▪HTTP/2: Provides asynchronous communications
with a well-understood protocol.
8
Design: gRPC for Internal Communications

9. Logging can be a silent killer.
▪We leverage Asynchronous
instead of Synchronous
Logging.
▪We reduced the logs per
Transaction while balancing
manageability.
▪Transaction-based logs are
always Debug, which is off by
default.
9
Design: Asynchronous Logging

10. Build
A GREAT DESIGN CAN BE POORLY IMPLEMENTED

11. As we build the platform, we leverage tools like PProf and Go’s
Benchmark Tests to incrementally improve the system.
Doing so has led us to find opportunities for optimization:
▪Replacing Mutex with Reader/Writer Mutex.
▪Reducing goroutine creation in key areas.
▪Avoiding Tickers in favor of Afterfunc.
12
Build: Incremental Optimizations

12. 14
Build: Reader/Writer Mutex vs. Mutex

13. 15
Build: Overusing Channels

14. Operate
POORLY MANAGED PLATFORMS DEGRADE OVER TIME,
WELL MANAGED PLATFORMS IMPROVE OVER TIME

15. We continuously monitor and improve the system to ensure our
payment router stays scalable and performant.
With every release, we:
▪Performance test to identify any degradation.
▪Chaos test to identify any resiliency challenges and verify
failover and recovery capabilities.
▪Simulate bad behavior to ensure we can handle unexpected
use of the system.
17
Operate: Continuous Improvement

16. When building a high-performance, low-latency transaction
router, every design decision and implementation detail matters.
But the key is not to get it right the first time; it's about
constantly iterating, testing, and fine-tuning the system to stay
ahead of challenges.
In our world, milliseconds matter. Payments don't wait.
18
Conclusion

17. Benjamin Cane – Distinguished Engineer
EOF
Does any of this sound interesting? American Express is hiring! Check out our open positions at
americanexpress.com/techcareers
bencane.com
linkedin.com/in/bencane