Unconventional Methods to Identify Bottlenecks in Low-Latency and High-Throughput Data Pipelines

In this presentation, we explore how standard profiling and monitoring methods may fall short in identifying bottlenecks in low-latency data ingestion workflows. Instead, we showcase the power of simple yet clever methods that can uncover hidden performance limitations.

Attendees will discover unconventional techniques, including clever logging, targeted instrumentation, and specialized metrics, to pinpoint bottlenecks accurately. Real-world use cases will be presented to demonstrate the effectiveness of these methods. By the end of the session, attendees will be equipped with alternative approaches to identify bottlenecks and optimize their low-latency data ingestion workflows for high throughput.

1. Unconventional Methods to Identify Bottlenecks in Low-Latency and High-Throughput Data Pipelines Zamir Paltiel Head of Engineering at Hyperspace

2. Zamir Paltiel Head of Engineering at Hyperspace Performance enthusiast F ocused on search performance in recent years Love building things from scratch Father to 4 children

3. Intro - performance optimization process How developers measure their pipelines today The issues one might miss Case study at Hyperspace A new approach to find bottlenecks Agenda

4. Intro - performance optimization process

5. How to increase speed of your code Run test Fix/Optimize Analyze Measure Repeat

6. Pareto Principle

7. T he bottleneck syndrome

8. What should I measure? CPU time Cycles Context switches Cache misses Memory Disk Network Mutexes

9. How devs measure code speed today

10. CPU Profilers CPU Profiler shows what functions consume what percent of CPU time. Event based profilers - use built-in hooks that the runtime framework provides Applicable to managed code like Java, Go, C# Instrumentation profilers - add code to the program that collects required info Inject in runtime or d uring compilation phase Adds overhead to code execution that sometimes creates distortion of reality Sampling profilers - collects call stack of the process in timely intervals Less intrusive - the code runs as usual Very good approximation of the amount of CPU time each function took in percentage No knowledge about the actual time an operation took - missing off cpu time

11. Flame Graph

12. T he issues one might miss

13. Off CPU Time

14. Case study at Hyperspace

15. Sample Ingestion Pipeline

16. Let’s profile it!

17. Flame Graph Analysis JSON parsing seems like the most significant part - can be replaced by binary serialization Challenge - Using Top command we see that CPU utilization is 2/8 CPUs - seems like there is a bottleneck that prevents us from reaching maximum ingestion speed

18. A new approach to find bottlenecks

19. Finding the real bottleneck

20. What we found If you have an off CPU bottleneck - the more load you add, it will take higher percentage of the time. Redundant lock - turns that FPGA SDK is thread safe. Usage of a single file descriptor to communicate with PCI interface Underutilizing the speed of the PCI bus - no usage of multithreading Not utilizing the 4 channels that exists in the PCI

21. Increase of ~30% in ingestion speed

22. Summary There are important factors to measure apart from CPU time We reviewed a new methodology to find off-cpu bottlenecks This new process yielded significant improvement in Hyperspace ingestion speed

23. LinkedIn: Zamir Paltiel W ebsite : hyper-space.io Thank you! Let’s connect.