PCI Express to Hell from lorem ipsum dolor sit ames.pdf

pcie lorem ipsum dolor sit ames

2. PCI Express
To Hell
28th July, 2023
Gynvael Coldwind

3. About your speaker
3
Managing Director @ HexArcana Cybersecurity GmbH

4. But today we're meeting because...
4

5. ...I like playing with PCs a bit too much
5

6. Disclaimers
6

7. Disclaimers
7
Won't be in-depth protocol-wise

Focused on user desktops
(not datacenter/enterprise hardware)

Some information might be outdated
(most of my hardware is PCIe 2.0, 3.0 and 4.0)

At the end of a day I'm just a hobbyist

8. Connectors
8

9. This is a PCI Express slot
9
PCI Express slots x8 and x16 (photo: Miha Ulanov, CC BY-SA 3.0)

10. This is a PCI Express slot
10
PCI Express slots x8 and x16 ISA 8 and 6-bit slots (photo: Miha Ulanov, CC BY-SA 3.0)

11. This is a PCI Express slot
11
PCI Express slots x16 (photo: JulianVilla26, CC BY-SA 4.0)

12. This is a PCI Express slot
12
PCI Express slots x16 (photo: JulianVilla26, CC BY-SA 4.0)

13. This is a PCI Express slot
13
PCI Express slots x1

14. This is a PCI Express slot
14
PCI Express slots x1 (yes, really, I'll get back to it later)

15. This is a PCI Express slot
15
PCI Express x1 and x16 slots

16. PCIe canonical connectors
16
4 types of slots
number of lanes
x4
x8
x16
x1
Power + SMBus + JTAG
data lanes

17. Other PCIe connectors
17
Obviously M.2
(from x1 to x4 depending on key) MiniPCI Express
(used in laptops/etc)

18. Other PCIe connectors
18
ExpressCard
34mm / 54mm variants
(had USB and PCIe modes)
AT-2874SX ExpressCard/34 (gigabit ethernet fiber)
© AndyEdits / CC BY-SA 4.0
(via Wikimedia Commons)
SD cards*
(from version 7.0)
© D-Kuru / CC BY-SA 4.0
(via Wikimedia Commons)
CFexpress cards*
* not sure if that's only the NVMe protocol over PCIe or full PCIe

19. Other PCIe connectors
19
© Amin / CC BY-SA 4.0
(via Wikimedia Commons)
Thunderbolt 1, 2, 3, 4
USB4
© Raimond Spekking / CC BY-SA 4.0
(via Wikimedia Commons)
mini DP connector USB Type-C connector
PCIe OCuLink
(PCI Express Over Cuprum Link)

20. Other PCIe connectors
20
Source:
https://global.icydock.com/resources/icy_tips_251.html
U.2 (used in server land?)
Basically PCIe in SATA Express
CY U.2 U2 SFF-8639 NVME PCIe SSD Cable

21. Other PCIe connectors
21
Likely there are other connectors out there as well,
especially in the server world ??????

22. Lanes
22

23. Serial Experiment Lanes
23
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
PCI Express version/gen/rev
6.0 ...
We are here

24. Serial Experiment Lanes
24
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
number of lanes
6.0 ...

25. Serial Experiment Lanes
25
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
x2 There are no x2 physical connectors.
But there are devices which use x2 lane
configurations.
6.0 ...
x5
x13

26. 26
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
GT/s is GigaTransfers per second
x2
2.5 GT/s
5 GT/s
10 GT/s
20 GT/s
40 GT/s
6.0 ...
Serial Experiment Lanes

27. 27
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
Rule of thumb:
- for 1.0 and 2.0 assume 10 bits per byte
2.5GT/s → 250 MB/s

- for anything other assume standard 8 bits per byte*
8GT/s → 8 GB/s
* kinda sorta close enough (128b/130b, then it gets funny from 6.0, i.e. PAM-4 and friends)
x2
2.5 GT/s
5 GT/s
10 GT/s
20 GT/s
40 GT/s
6.0 ...
5 GT/s 8 GT/s 16 GT/s 32 GT/s 64 GT/s
(32 GBd)
Serial Experiment Lanes

28. 28
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
x2
2.5 GT/s
5 GT/s
10 GT/s
20 GT/s
40 GT/s
6.0 ...
5 GT/s 8 GT/s 16 GT/s 32 GT/s 64 GT/s
(32 GBd)
10 GT/s
20 GT/s
16 GT/s
Fast enough for 10GbE cards
Serial Experiment Lanes

29. 29
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
x2
2.5 GT/s
5 GT/s
10 GT/s
6.0 ...
5 GT/s 8 GT/s 16 GT/s 32 GT/s 64 GT/s
(32 GBd)
10 GT/s
20 GT/s
16 GT/s
32 GT/s
32 GT/s
64 GT/s
64 GT/s
128 GT/s
Older to modern m.2 NVMe SSD zone Very "rule-of-thumbish"
(check SSD specs!)
You can likely get an SSD
that uses x16 ofc, but
that's not M.2 then.
Serial Experiment Lanes

30. 30
x4
x8
x16
x1
1.0 2.0 3.0 4.0 5.0
x2
2.5 GT/s
5 GT/s
10 GT/s
6.0 ...
5 GT/s 8 GT/s 16 GT/s 32 GT/s 64 GT/s
(32 GBd)
10 GT/s
20 GT/s
16 GT/s
32 GT/s
32 GT/s
64 GT/s
64 GT/s
128 GT/s
RTX 4999 TI zone
20 GT/s
40 GT/s
40 GT/s
80 GT/s
64 GT/s
128 GT/s
128 GT/s
256 GT/s
256 GT/s
512 GT/s
Why is this orange?
Serial Experiment Lanes

31. Protip
31
Always calculate required and
available bandwidth

Remember to take into account PCIe version of both the
host and the device.

32. Device vs Host

32

33. Device vs Host
33
PCIe is compatible in all directions

34. Device vs Host
34
PCIe is compatible in all directions
PCIe 2.0 x1 into PCIe 4.0 x16?
VS

35. Device vs Host
35
PCIe is compatible in all directions
PCIe 2.0 x1 into PCIe 4.0 x16?
VS
✓

36. Device vs Host
36
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 2.0 x16?
VS

37. Device vs Host
37
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 2.0 x16?
VS
✓

38. Device vs Host
38
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 1.0 x1?
VS

39. Source: chalk210
https://www.techpowerup.com/forums/threads/hd4350-
a-pci-e-x1-slot-and-a-dremel-tons-of-pics-warning.8879
1/page-5#post-1944178
Device vs Host
39
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 1.0 x1?
VS
✓

40. Source: (original author unknown)
https://forums.anandtech.com/threads/so-a-pcie-x16-ca
rd-works-in-an-x1-slot-right.2391834/
Source: chalk210
https://www.techpowerup.com/forums/threads/hd4350-
a-pci-e-x1-slot-and-a-dremel-tons-of-pics-warning.8879
1/page-5#post-1944178
Device vs Host
40
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 1.0 x1?
VS
✓

41. Source: (original author unknown)
https://forums.anandtech.com/threads/so-a-pcie-x16-ca
rd-works-in-an-x1-slot-right.2391834/
Source: chalk210
https://www.techpowerup.com/forums/threads/hd4350-
a-pci-e-x1-slot-and-a-dremel-tons-of-pics-warning.8879
1/page-5#post-1944178
Device vs Host
41
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 1.0 x1?
VS
✓
Technically
PRSNT1# (Side A Pin 1)
should be bridged with
PRSNT2# (Side B Pin 17)

42. Device vs Host
42
PCIe is compatible in all directions
PCIe 4.0 x16 into PCIe 1.0 x1?✓
StarTech.com PCIe x1 to x16 adapter
PCIe Extender/Riser Cable

43. Device vs Host
43
PCIe is compatible in all directions...kinda
Above 4G Decoding
Resizable BARs
Intel Xeon Phi

44. Device vs Host
44
PCIe is compatible in all directions...kinda
RaspberryPi 4's PCIe vs GPUs
Just go and watch Jeff Geerling's videos

45. Protip
45
Physical dimensions also matters
(a big GPU will block other slots)

Adapters are a thing, but ideally don't use them unless you
have to.

46. Welcome to hell
46

47. Physical size vs electrical connections vs lanes
47

48. Physical size vs electrical connections vs lanes
48
Note lack of electrical connections beyond x4
Slot is actually floating here

49. Physical size vs electrical connections vs lanes
49
Note lack of electrical connections beyond x8

50. 10GbE PCIe card
50
ASUS XG-C100C
(bought on 24.06.2022)
ASUS XG-C100C
(bought on 25.01.2023)
x4 x4
Chipset: Aquantia Corp. AQC107
PCIe 3.0 (8GT/s) x4

51. 10GbE PCIe card
51
ASUS XG-C100C
(bought on 24.06.2022)
ASUS XG-C100C
(bought on 25.01.2023)
x4 x4
Chipset: Aquantia Corp. AQC107
PCIe 3.0 (8GT/s) x4
Chipset: Aquantia Corp. AQC113CS

52. Chipset: Aquantia Corp. AQC107
PCIe 3.0 (8GT/s) x4
(great for lanes, bad for PCIe 2.0 slots)
10GbE PCIe card
52
ASUS XG-C100C
(bought on 24.06.2022)
ASUS XG-C100C v2
(bought on 25.01.2023)
x4 x4
Chipset: Aquantia Corp. AQC113CS
PCIe 4.0 (16GT/s) x2
x2
x4
x1
2.0
x2
5 GT/s 4 GB/s
10 GT/s 8 GB/s
20 GT/s 16 GB/s

53. (great for lanes, bad for PCIe 2.0 slots)
10GbE PCIe card
53
ASUS XG-C100C
(bought on 24.06.2022)
ASUS XG-C100C v2
(bought on 25.01.2023)
x4 x4
Chipset: Aquantia Corp. AQC107
PCIe 3.0 (8GT/s) x4
Chipset: Aquantia Corp. AQC113CS
PCIe 4.0 (16GT/s) x2
x2
uff

54. Slot identification

54
(In) consistency in MSI X99A Xpower Gaming Titanium
Note: Some manufacturers do get this right (as far as that's possible).

55. 55
PCIe is a tree

56. Source: manual of MSI X99A Xpower Gaming Titanium
You see, it's a tree (with detachable branches)

56
CPU and PCH provide
PCIe lanes

i7-6900K provides 40 3.0
lanes.

X99 provides 8 2.0 lanes.

Total 48 lanes, nice!
Kinda sorta.
x40 lanes*
x8 lanes

57. Source: manual of MSI X99A Xpower Gaming Titanium
You see, it's a tree (with detachable branches)

57
DMI is basically PCIe

Here we have DMI 2.0 x4
(it's most likely 2.0, but
might be 3.0?)

Usually DMI cuts into CPU's
total lane count.
So that's 36 left.

DMI is a bottleneck.
x40 lanes*
(actual x36)
x8 lanes

58. Source: manual of MSI X99A Xpower Gaming Titanium
You see, it's a tree (with detachable branches)

58
x40 lanes*
(actual x36)
x8 lanes
On-board devices
eat PCIe lanes for
breakfast
(but usually only PCH
ones)

Switches and BIOS
options (RTFM).



2.0 x2 or 3.0 x1

59. On-board devices
eat PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.



Source: manual of MSI X99A Xpower Gaming Titanium
You see, it's a tree (with detachable branches)

59
x40 lanes*
(actual x36)
x8 lanes

60. You see, it's a tree (with detachable branches)

On-board devices
eat PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.




Unless we rip-out that
Wi-Fi module ofc.

60

61. You see, it's a tree (with detachable branches)

On-board devices
ear PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.




Unless we rip-out that
Wi-Fi module ofc.

61

62. You see, it's a tree (with detachable branches)

On-board devices
ear PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.

Or desolder a chip (see
Tomasz Mloduchowski's
blogpost on PCIe on
RPi4).

62
(c) Tomasz Mloduchowski

63. (c) Tomasz Mloduchowski
You see, it's a tree (with detachable branches)

63
(c) Tomasz Mloduchowski
On-board devices
ear PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.

Or desolder a chip (see
Tomasz Mloduchowski's
blogpost on PCIe on
RPi4).

64. You see, it's a tree (with detachable branches)

64
(c) Tomasz Mloduchowski
PCIe "extended" through
"USB" socket
On-board devices
ear PCIe lanes for
breakfast
(but usually only PCH
ones)

In this case we get only
one x1 2.0 slot from PCH.

Or desolder a chip (see
Tomasz Mloduchowski's
blogpost on PCIe on
RPi4).

65. 65
Count
Lanes

66. Source: manual of MSI X99A Xpower Gaming Titanium
Lane counting
66
x40 lanes*
(actual x36)
What about the
slots?

Total lanes in "CPU" slots:

x16
+ x8
+ x8
+ x16
+ x4 or x8
======
x52 or x56

67. Source: manual of MSI X99A Xpower Gaming Titanium
Lane counting
67
x40 lanes*
(actual x36)
What about the
slots?

Total lanes in "CPU" slots:

x16
+ x8
+ x8
+ x16
+ x4 or x8
======
x52 or x56
We cannot use all slots to
their full extent at the same
time.
Not enough minerals lanes.

68. Protip
68
Slots are a lie

Don't take them at face value

69. 69
But wait! There's more!
Source: manual of MSI X99A Xpower Gaming Titanium
x16
x16
x4
x36 lanes used total!
Any problems?

70. 70
But wait! There's more!
Source: manual of MSI X99A Xpower Gaming Titanium
x16
x16
x4
Either the x4 device won't work at all...

71. 71
But wait! There's more!
Source: manual of MSI X99A Xpower Gaming Titanium
x16 capped at x8
x16
x4
Or x4 will work, but first device will be
capped and last device won't.

72. 72
But wait! There's more!
Source: manual of MSI X99A Xpower Gaming Titanium
x16 capped at x8
x16
x4
Or x4 won't work and first device will still
be capped at x8.

73. 73
But wait! There's more!
CPU/PCH/actual slots work only in specific
pre-defined configurations!
Motherboard's manual will have some details.
Source: manual of MSI X99A Xpower Gaming Titanium

74. 74
Proper configuration for this mobo
Source: manual of MSI X99A Xpower Gaming Titanium
x16
x16
x4
Solution: Move x4 device to PCI_E6
(also no M.2 SSD on 3.0 PCIe for you, sorry)
x4

75. 75
Furthermore...
3 ports using the same lanes!
Source: manual of MSI X99A Xpower Gaming Titanium

76. Protip
76
Know your motherboard

You cannot route lanes as you please (sadly).

Read the manual.
Seriously.
It will save you time.

Also read about the chipset (lane configuration).

77. 77
But...
Don't worry!

78. 78
But...
Don't worry!
Most CPUs don't have 40 lanes!

(wait, that's actually a bad thing)

79. Gallery of different
configurations
79

80. AMD Athlon 3000G
80
PCIe CPU lanes: 6
Config: 1x4+1x2

81. AMD Athlon 3000G
81
PCIe CPU lanes: 6 4
Config: 1x4+1x2 (chipset)

E.g. B450 chipset adds x6 2.0 lanes.

82. Intel i3-6100 + Gigabyte B250-FinTech
82
PCIe CPU lanes: 16
Config: 1x16, 2x8, 1x8+2x4

Chipset B250 adds 12 PCIe 3.0
lanes!

All 11 x1 slots are chipset slots.
Only the x16 is a CPU slot.

83. Protip
83
Make sure your CPU and mobo
have enough slots and lanes

Will you get more cards in the future?

Also, don't be me-from-10-years-ago,
don't fill every slot just because you can.

84. M.2 and PCIe
84

85. Unkeyed M.2
(doesn't exists)
Quick digression: M.2 and PCIe
85

86. Key A (Notch at pins 8-15)
2 * PCIe x1, USB 2.0, I
2
C, DP x4
Quick digression: M.2 and PCIe
86

87. Key B (notch at pins 12-19)
PCIe x2, SATA, USB 2.0 and 3.0,
audio, UIM, HSIC, SSIC, I
2
C, SMBus
Quick digression: M.2 and PCIe
87

88. Key E (notch at pins 24-31)
2 * PCIe x1, USB 2.0, I
2
C, SDIO, UART,
PCM and CNVi
Quick digression: M.2 and PCIe
88

89. Key M (notch at pins 59-66)
PCIe x4, SATA, SMBus
your typical NVMe key
Quick digression: M.2 and PCIe
89
That one is M key

90. Remember this one?
90
Two PCI Express slots x1

91. It was a weird M.2 Key E
91
Two PCI Express slots x1
Note the
double
notches!
https://gynvael.coldwind.pl/?id=757

92. M.2 to PCIe adapters exist and are cheap
92
Delock PCI Express x1 to M.2 Key M Adapter
(yes, you stick it in the slot vertically)
Digitus M.2 NVMe SSD PCI Express 3.0 (x16) Add-On Card
(no, I don't know why it's x16)
Digitus M.2 NGFF / NVMe SSD PCI Express 3.0 (x4)
Add-On Card

93. M.2 to PCIe adapters offer only 1 slot!
93
Digitus M.2 NGFF / NVMe SSD PCI Express 3.0 (x4)
Add-On Card
Key B
(SATA only!)
Key M
(PCIe x4)
But what about x16 → 4 * x4?

94. x16 → 4 times x4

94

95. x16 → 4 times x4
95
CAD projection image for BIFURCATOR project by ARBER FACA
(https://grabcad.com/library/bifurcator-1)
Problem:
I have an x16 PCIe port
I want to split it into 4 * x4

96. x16 → 4 times x4
96
SYBA SI-PEX40161
PCIe 3.0 x16 → 4 * x4 M.2
($35 on ebay)
ASUS Hyper M.2 x16 PCIe 4.0 X4
($80 on amazon)
You buy one of these

97. x16 → 4 times x4
97
SYBA SI-PEX40161
PCIe 3.0 x16 → 4 * x4 M.2
($35 on ebay)
ASUS Hyper M.2 x16 PCIe 4.0 X4
($80 on amazon)
IT DOESN'T WORK!

98. x16 → 4 times x4
98
SYBA SI-PEX40161
PCIe 3.0 x16 → 4 * x4 M.2
($35 on ebay)
ASUS Hyper M.2 x16 PCIe 4.0 X4
($80 on amazon)
IT DOESN'T WORK!
"motherboard support required"
I.e. your motherboard has to support...

99. Bifurcation
99
CAD projection image for BIFURCATOR project by ARBER FACA
(https://grabcad.com/library/bifurcator-1)
Problem:
I have an x16 PCIe port
I want to split it into 4 * x4

Answer:
Bifurcation

100. Bifurcation
100
PCIe switch chips are expensive!
PEX 8734 → $200 PEX 8747 → $140

101. Bifurcation
101
PCIe switch chips are expensive!
PEX 8747 based PCIe 3.0 x8 →4 * x4 M.2 card ($262 on ebay)

102. Protip
102
If it's cheap,
there's no bifurcation chip

But maybe your motherboard has it?

103. Or you can go all out...
103
Apex Storage X21 (21 [sic] M.2 disks, 100 PCIe 4.0 Lanes, $2800)

There's also Apex Storage X16 PCIe 4.0 for $1800 btw.
These are two boards screwed together btw

104. These exist btw
104

105. StarTech PCIe PCI host card
105
???

106. Non-PCIe cards?
106

107. BLIKVM PCIe
107
Only used for
power

108. TerASIC OSK-GT (FPGA)
108
PCIe is optional
Bitstreams needs to support PCIe.
Typical for FPGA cards.

109. Windows PCIe debugging
(UH, SORRY,
NO WINDOWS SETUP)
109

110. Linux PCIe debugging
(NO SLIDES, SHOW
COMMANDS)
110

111. Final notes
111

112. Always RTFM and check device specs

Count your lanes

Know which slots are CPU/PCH-driven

Calculate bottlenecks

You need enough slots, but slots by
themselves aren't enough

More CPU lanes == less problems

Don't trust ChatGPT
Final notes
112
And don't worry, modern USB-C is even worse. It's like M.2 connector but no
physical key to say which protocols are supported.

113. Step by step algorithm
113
When building a new PC (or modifying an existing one):

1.Make an inventory of each PCIe device you want to put in
a.Note the PCIe version (e.g. 4.0).
b.Note the physical connector (e.g. PCIe x8 or M.2 key M).
c.Note the actual number of lanes required and optimal (e.g. x4 is enough, but x8 is optimal).
Do bandwidth calculation and check if the device can actually reach that bandwidth with less lanes.
d.Note if you need this device to work on full efficiency or whether performance can be sacrificed.
2.Find a CPU that has enough lanes
Ideally it's the CPU that should provide the lanes. The chipset is usually slower (lower PCIe version) and
oversells lanes (CPU↔chipset connection is usually x4, so it will bottleneck if all devices start using their
lanes).
Remember to deduct lanes used by the chipset from CPU lanes (double-check if that wasn't already done
by the manufacturer).
3.Find a motherboard for that CPU that has enough slots
Important: consult the motherboard manual to be sure PCIe/M.2 slots can work in the lane configuration
you need them. Ask support or online if needed.
Make sure there's also enough physical space.

KEEP these notes and make a habit of updating them through device's lifetime!

114. THANK YOU
THE END
gynvael.coldwind.pl/discord
gynvael.coldwind.pl
pagedout.institute
@gynvael
youtube.com/c/GynvaelEN Q&A (if we have time)
...and questions are simple
go/pcie-hell