[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse

Computation is increasingly constrained by power. With each advancement in the manufacturing process, a decreasing percentage of the CPU can operate at full capacity, leading to the emergence of the term 'dark silicon'. This trend necessitates techniques that utilize chip area to optimize power efficiency through specialized accelerators.

The presentation will outline key concepts that led to the dark silicon such as Moore’s law and breakdown of Dennard scaling, followed by an overview of current and upcoming CPU accelerators. The focus will then shift to vector units and the specifics of vector programming. Attendees will be introduced to registers, a range of vector operations, and methods to develop branchless algorithms such as sorting networks. The session will conclude with an overview of the new Java Vector API and how it was already picked up by projects to do AI inference (Llama 2) and vector search (AstraDB and Cassandra).

1. HOW I LEARNED TO STOP
WORRYING AND LOVE THE DARK
SILICON APOCALYPSE
HTTP://SLI.DO
#GEECON
ROOM 11

2. How I learned to stop worrying and love
the dark silicon apocalypse

3. How I learned to stop worrying and love
the dark silicon apocalypse

4. @TKOWALCZ
TKOWALCZ
Tomasz Kowalczewski

5. for(int i = 0; i<100; i++) {
...
}

6. for(int i = 0; i<100; i++) {
...
}
.back:
cmp rcx, 100
je .outside
...
inc rcx
jmp .back
.outside:

7. M3 Max
Graviton 4
AMD EPYC Genoa-X

8. Today: 1000 lights/m @ 100W
2024: 2000 lights/m @ 200W
2028: 4000 lights/m @ 400W

9. Dennard scaling
As transistors get smaller, their power density stays
constant, so that the power use stays in proportion
with area; both voltage and current scale
(downward) with length
ROBERT H. DENNARD, IBM

10. S
S

11. CPU
CPU
x2
x2
2 EPOCHS

12. 75% DARK AFTER 2 GENERATIONS
93% DARK AFTER 4 GENERATIONS
CPU
x2
x2
x4
x4
CPU
4 EPOCHS

14. MICHAEL B. TAYLOR, IS DARK SILICON USEFUL?

15. MORE CORES, MORE BETTER
MORE ACCELERATORS, MORE BETTER
(C) PATRICK KENNEDY, SERVETHEHOME.COM

16. FJCVTZS
Floating-point Javascript Convert to Signed fixed-
point, rounding toward Zero.

17. PCMPESTRI
Packed Compare Explicit Length Strings, Return
Index

18. PICTURE CREDIT TO @FRITZCHENSFRITZ, ANNOTATED BY @GPUSAREMAGIC

19. PICTURE CREDIT TO @FRITZCHENSFRITZ, ANNOTATED BY @GPUSAREMAGIC

20. VECTOR REGISTER

21. VECTOR REGISTER

22. VECTOR REGISTER

24. VECTOR REGISTER

25. 1 2 3 4 5 6 7 8
VECTOR OPERATIONS
+
10 20 30 40 50 60 70 80
=
11 22 33 44 55 66 77 88

26. VECTOR REGISTER
add
subtract
multiply
shuffle
blend
load
BITS? BYTES? ELEMENTS?

27. Integer Integer Integer Integer Integer Integer Integer Integer
VECTOR REGISTERS
SHAPE
+
ELEMENT TYPE
=
SPECIES
SHAPE
64 bit
128 bit
256 bit
Long Long Long Long
Double Double Double Double

28. Vector cosine similarity
text-embedding-3-large is our new next generation
larger embedding model and creates embeddings
with up to 3072 dimensions.
PICTURE CREDIT OPENAI

29. Vector cosine similarity
text-embedding-3-large is our new next generation
larger embedding model and creates embeddings
with up to 3072 dimensions.

30. Origin Destination Flight No. Carrier Codeshare Cabin
KRK DFW 1022 LH ... ...
KRK BOS 933 LH ... ...
KRK LHR 123 BA ... ...
Row vs. Column layout in databases

31. Origin Destination Flight No. Carrier Codeshare Cabin
KRK DFW 1022 LH ... ...
KRK BOS 933 LH ... ...
KRK LHR 123 BA ... ...
Row vs. Column layout in databases
1022 933 123 ...IntVector

32. Origin Destination Flight No. Carrier Codeshare Cabin
KRK DFW 1022 LH ... ...
KRK BOS 933 LH ... ...
KRK LHR 123 BA ... ...
1022 933 123 ...
Row vs. Column layout in databases
IntVector

33. Row vs. Column layout in databases
KRK KRK KRK ... ... ...
DFW BOS LHR ... ... ...
1022 933 123 ... ... ...
Origin
Destination
Flight No.

34. Row vs. Column layout in databases
KRK KRK KRK ... ... ...
DFW BOS LHR ... ... ...
1022 933 123 ... ... ...
Origin
Destination
Flight No.
1022 933 123 ...IntVector

35. Row vs. Column layout in databases
3xKRK 13xDFW 42xLHR ... ... ...
18xDFW 22xBOS 2xLHR ... ... ...
1022 -89 -810 ... ... ...
Origin
Destination
Flight No.
1022 933 123 ...IntVector
?

36. SOURCE: HTTPS://EN.M.WIKIPEDIA.ORG/WIKI/FILE:TABLE_OF_X86_REGISTERS_SVG.SVG
K0 K1 K2 K3 K4 K5 K6 K7

37. 1 2 3 4
1 1 1 1 0 0 0 0
1 2 3 4 0 0 0 0
MEMORY
MASK
Vector masks
V1
LOAD

38. 1 2 3 4 5 6 7 8
1 1 1 1 0 0 0 0
1 2 3 4 0 0 0 0
V1
MASK
Vector masks
MEMORY
STORE
Page boundary, fault suppression

39. 1 2 3 4 5 6 7 8
7 6 5 4 3 2 1 0
8 7 6 5 4 3 2 1
V1
SHUFFLE
Vector shuffles
V2
REARRANGE

40. VECTOR OPERATIONS

41. Sorting networks

42. Sorting networks
1
5
1
5
1
5 1
5

43. Sorting networks
A
B
C
1
5
1
5

44. Sorting networks
A
B
C
D
A
B
C

45. Sorting networks
A
B
C
D
D
B
A
4

46. 5 4 1 2 5 8 7 6
MAX(V1, V2)
2 6 3 7 2 5 1 9
5 6 3 7 5 8 7 9
V1
V2
VECTOR MIN/MAX
MIN(V1, V2)
2 4 1 2 2 5 1 6

47. Sorting networks
1
5 1
5
5
1
1
5
shuffle
1
1
5
5
min
max
1
5
blend
blend

48. Sorting networks
BERENGER BRAMAS,
A NOVEL HYBRID QUICKSORT ALGORITHM VECTORIZED USING AVX-512 ON INTEL SKYLAKE

49. NEED FOR UPFRONT DESGIN OF DATA STRUCTURES AND MEMORY
LAYOUTS
COMPLICATED ALGORITHMS
NEED TO EMPLOY ELABORATE CODE TACTICS TO NOT BREAK (RE)BOXING
FAILURE TO OPTIMISE LEADS TO CATASTROPHIC DEGRADATION OF
PERFORMANCE
WILL INCUBATE UNTIL PROJECT VALHALLA BECOMES AVAILABLE
NEED TO MEASURE, MEASURE, MEASURE ON TARGET HARDWARE

50. FAST AND ROBUST VECTORIZED IN-PLACE SORTING OF PRIMITIVE TYPES
INTEL® INTRINSICS GUIDE
JVECTOR SIMDOPS
PERFORMANCE SPEED LIMITS
DESIGNING IN 2023 - 10 PROBLEMS TO SOLVE (JIM KELLER)
JLLAMA