Statistical Analysis of DNS Latencies.pdf

Short lightning talk on measuring and comparing DNS latency.

1. Ondřej Surý <[email protected]>
Internet Systems Consortium
2024-05-21

2. Measuring DNS Latency

3. DNS Performance Metrics (quick intro)Measuring DNS Latency
•Performance under normal conditions
‣
•The data is right-skewed
‣The usual descriptive statistics are useless (average, mean, …)
‣Most of the queries are answered very quickly
‣In fact, 95% of the queries are answered under 2 milliseconds
‣The tails makes it interesting
Ondřej Surý <[email protected]>
2024-05-21 1 / 8

4. Logarithmic Percentile Histogram Measuring DNS LatencyOndřej Surý <[email protected]>
2024-05-21 2 / 8

5. Logarithmic Percentile Histogram Measuring DNS Latency
•Both axes are logarithmic
‣x-axis: slowest percentile
‣y-axis: average latency
•It makes the tail more visible
•Variant of Complementary Cumulative Distribution Function
•Very robust, can be used for monitoring (1% slowest percentile)
•Introduced by the good folks at PowerDNS
See more: https://blog.powerdns.com/2017/11/02/dns-performance-
metrics-the-logarithmic-percentile-histogram
Ondřej Surý <[email protected]>
2024-05-21 3 / 8

6. Compare DNS Latencies

7. DNS Performance for Developers Compare DNS Latencies
•Comparing two branches of BIND 9
‣Did we improve the code?
‣Did we made things worse?
‣Currently, we compare the graphs by looking at them;
‣And then running more tests;
‣And then some wishful thinking…
•Sending thanks to Python’s numpy and scipy developers!
Ondřej Surý <[email protected]>
2024-05-21 4 / 8

8. Pick the right statistics Compare DNS Latencies
•The distribution is not normal
•Non-parametrical test then?
‣Kolmogorov-Smirnov test didn’t really work
•Normalize the data?
‣Box Cox Transformation didn’t really work
•Maybe look only at the tail then?
Ondřej Surý <[email protected]>
2024-05-21 5 / 8

9. Looking at the tail Compare DNS Latencies
•Pick the 95% (99%) percentile complement
‣Either return the lowest bucket needed for 5% of responses
‣Or count the answers in (1.9-2.0 second buckets)
•Have at least 3 runs for each group
•Yay! The data are normal and the group variances are equal
‣Shapiro-Wilk test
–
first group (??????=
0.905
, ??????=
0.436
)
–
second group (??????=
0.970
, ??????=
0.874
)
‣
Brown-Forsyth test (??????=
0.070
, ??????=
0.798
)
Ondřej Surý <[email protected]>
2024-05-21 6 / 8

10. Parametrical test (ANOVA) Compare DNS Latencies
•We can test more than two branches
•One-way ANOVA reports difference between branches
‣
??????=
9244.090
, ??????<.
001
•Two-sample T-Test (for confirmation)
‣
??????=−
96.146
, ??????<.
001
Ondřej Surý <[email protected]>
2024-05-21 7 / 8

11. More ideas?

12. Other tests? More ideas?
•Is this even correct? Or am I crazy? (I’m not a statistician)
•Can we just compare two data sets (1x baseline with 1x branch)?
•Can we use the full (right-skewed) population?
•Are there any other non-parametrical tests I can try/use?
•Are there any other suitable statistical methods?
•Is this useful for other Internet measurements?
Ondřej Surý <[email protected]>
2024-05-21 8 / 8

13. Thank you!