mtcawsc2019_marjanovic_pcie_xilinx_and_fpga_tool

This is presentation for pcie driver on FPGA

1. DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 0/60
DMA transfer, PCIe Driver
and FPGA Tools.
Jan Marjanovic (MTCA Tech Lab/DESY)
MTCA/ATCA Workshop China 2019
at IHEP, Beijing

2. Introduction
IMicroTCA and PCI Express
IPCI Express protocol
IFPGA side
ISoftware side
ITroubleshooting techniques
IFurther resources and summary
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 2/60

3. MicroTCA and PCI Express
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 3/60

4. MicroTCA backplane
A typical ADC system in MicroTCA is shown here:
from PICMGRMicroTCA.4 Enhancements for Rear I/O and Timing R1.0
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 4/60

5. Advanced Mezzanine Cards
A typical board in such a system is shown here:
from PICMGRMicroTCA.4 Enhancements for Rear I/O and Timing R1.0
talk tomorrow:
MMC Stamp and its applications
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 5/60

6. Ports on an Advanced Mezzanine Cards
AMC standard provides some guidance on how the ports should be assigned:
from PICMGRAdvanced Mezzanine Card AMC.O Specication R2.0
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 6/60

7. MicroTCA backplane
MicroTCA backplane connections as specied by MicroTCA.4 standard:
REQ 6-6Port 0shouldbe used for base Ethernet interface. Port 1 is connected
to the optional (redundant) MCH.
REQ 6-7
Port 4 to 7shouldbe used for PCIe.
REQ 6-8Port 12 to 15maybe used for application specic wire-ring.
REQ 6-9Ports 17- 20shouldbe wired as a bus according Section 6.4.
REQ 6-10FCLKA (Clk3)shouldbe used for PCIe clock distribution as dened in
MTCA.O R1.0
...
from PICMGRSpecication MTCA.4 Revision 1.0: MicroTCA Enhancements for Rear I/O and Precision Timing
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 7/60

8. MicroTCA backplane
MicroTCA backplane connections as specied by MicroTCA.4 standard:
REQ 6-6Port 0shouldbe used for base Ethernet interface. Port 1 is connected
to the optional (redundant) MCH.
REQ 6-7
Port 4 to 7shouldbe used for PCIe.
REQ 6-8Port 12 to 15maybe used for application specic wire-ring.
REQ 6-9Ports 17- 20shouldbe wired as a bus according Section 6.4.
REQ 6-10FCLKA (Clk3)shouldbe used for PCIe clock distribution as dened in
MTCA.O R1.0
...
from PICMGRSpecication MTCA.4 Revision 1.0: MicroTCA Enhancements for Rear I/O and Precision Timing
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 7/60

9. Reasons to use PCI Express
IHigh-throughput (up to 64 Gbps (gen 3, x8 link))
ILow-latency
ISoftware model (memory-based transaction, method to detect devices on the bus)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 8/60

10. Reasons to use PCI Express
A lot of boards available on the market use PCI Express as main/only communication protocol.
FPGAs also allow implementation of other protocols (e.g. 10 and 40 Gb Ethernet and Serial Rapid IO).
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 9/60

11. PCI Express protocol
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 10/60

12. PCI Express
PCIe express is high-speed protocol for PC extension cards. Different form-factors are
also available. Link out of 1, 2, 4, 8 or 16 lanes, lane rate 2.5 GT/s, 5.0 GT/s, 8.0 GT/s and
16.0 GT/s.
inspired by Figure 2-12 from PCI Express Technology 3.0
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 11/60

13. PCI Express
Concrete example:
* real PCIe link in MicroTCA system includes a PCIe switch in MicroTCA Carrier Hub (MCH)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 12/60

14. PCIe TLP transaction types
Packets can be one of several types: Memory Read, Memory Write, Completion,
Completion with Data, Messages, Conguration Accesses and some legacy types.
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 13/60

15. Getting the data from FPGA to CPU
IOften we need to access a large quantity of data (e.g. readout from ADCs)
IOne option is to use data 64 bit at a time (using MRd command)
IExperimental results show that we can get around 2.5 MB/s with kernel-space
for the method and the results)
memcpy()
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 14/60

16. Getting the data from FPGA to CPU
memcpy()
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 15/60

17. Bus Mastering and DMA
Bus Mastering means that endpoint (device) can send read and write packets.
Usually driver enables bus mastering on a device when the device gets congured. Result
can be observed with:
$ sudo l sp ci -s 0 c :00 - vv
0 c :0 0. 0 S i g n a l p r o c e s s i n g c o n t r o l l e r : X i l i n x C o r p o r a t i o n D e v i c e 0038 ( rev 25)
S u b s y s t e m : X i l i n x C o r p o r a t i o n D e v i c e 0007
P h y s i c a l Slot : 12
C o n t r o l : I /O - Mem +B u s M a s t e r+ SpecCycle - MemWINV - VGASnoop - ParErr - Stepping -
SERR - FastB2B - D i s I N T x +
S t a t u s : Cap + 66 MHz - UDF - FastB2B - ParErr - D E V S E L = fast > TAbort - < TAbort -
< MAbort - > SERR - < PERR - INTx -
[. . .]
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 16/60

18. PCIe protocol summary
IPacket-oriented protocol, memory-based
IThree important packet types to transfer data
IMemory Read
IMemory Write
ICompletion with Data
IRoot port (i.e. CPU) and endpoints (FPGA devices, typically)
IBus Mastering needed for DMA from device (endpoint)
Ilspci
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 17/60

19. FPGA side
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 18/60

20. Implementation of PCIe in FPGA
Modern FPGAs from Xilinx and Intel include hard IP cores to handle lower layers of
protocol.
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 19/60

21. Implementation of PCIe in FPGA
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 20/60

22. Implementation of PCIe in FPGA
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 21/60

23. Implementation of PCIe in FPGA
On top of that, both Xilinx and Intel provide IP cores to also handle the Transaction Layer
protocol
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 22/60

24. PCIe sub-system
This is how the Xilinx DMA Subsystem for PCI Express looks in Vivado:
master AXI4 port
DMA port - burst transfer
master AXI4-lite port
access to regs
xcvr ports
to AMC port 4-7
100 MHz clk
to AMC FCLKA
usrirq
from app logic
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 23/60

25. Let's build a system, step by step
1. We open Vivado, start a new project, create a Block Diagram and place an xdma IP:
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 24/60

26. Let's build a system, step by step
2. We then connect external connections:
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 25/60

27. Let's build a system, step by step
3. We then proceed to add a GPIO, to blink the LEDs
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 26/60

28. Let's build a system, step by step
4. Let's not forget to assign the addresses
Mini tip: place at
a know address (e.g. at 0x0)
ident register (e.g. 0xde30b02d)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 27/60

29. Let's build a system, step by step
5. We add a DDR4 controller (not a trivial step, a lot of conguration) and connect the
DMA port of PCIe to DDR4 controller:
second port
for application logic
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 28/60

30. Let's build a system, step by step
6. some magic (aka months of work on application logic)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 29/60

31. Typical PCIe-based system in FPGA
7. This is how a typical PCIe-based system looks
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 30/60

32. Summary of the FPGA part
IModern FPGAs provide a lot of hard IP
ITypical ow involves conguring IPs and connecting them together
ICustom logic required for application-specic part
ITwo-step transfer: rst to on-board memory, then to CPU
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 31/60

33. Software side
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 32/60

34. CPU board
From ”FPGA side” we have solved the left part, now it is time to look at the right part:
ITypically there is one (or more) CPU AMCs is MicroTCA systems
IUsually (in experimental physics) the CPU runs GNU/Linux
IDESY runs Ubuntu 16.04 LTS, Ubuntu 18.04 LTS is also in use
IPCIe hotplug was modernized in Linux kernel 4.19[1]!available in Ubuntu 20.04 LTS
[1]https://lwn.net/Articles/767885/
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 33/60

35. Driver for Xilinx DMA subsystem
IXilinx provides a Linux driver for their DMA subsystem for PCIe
IAvailabe on GitHub:
https://github.com/Xilinx/dma_ip_drivers/tree/master/XDMA/
IPreviously available in Xilinx Answer Record #65444:
https://www.xilinx.com/support/answers/65444.html
IMicroTCA Tech Lab mantains an internal fork with some improvements!
discussion with our legal department on licensing ongoing
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 34/60

36. Linux pci subsystem
ILinux has a subsystem responsible for managing PCI/PCIe device driver
IHot-swap module (pciehp) is also part of Linux kernel
IDevice-specic driver register by providing driver registerdriver()
function
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 35/60

37. Linux pci subsystem
https://elixir.bootlin.com/linux/v4.15/source/include/linux/pci.h#L744
s t r u c t p c i _ d r i v e r {
s t r u c t l i s t _ h e a d node ;
co n st char * name ;
co n st s t r u c t p c i _ d e v i c e _ i d * i d _ t a b l e ; /* must be non - NULL for p r ob e to be c a l l e d */
int (* pr ob e ) ( s t r u c t p c i _ d e v * dev , c on st s t r u c t p c i _ d e v i c e _ i d * id ) ; /* New d e v i c e
i n s e r t e d */
void (* r e m o v e ) ( s t r u c t p c i _ d e v * dev ) ; /* D e v i c e r e m o v e d ( NULL if not a hot - plug
c a p a b l e d r i v e r ) */
int (* s u s p e n d ) ( s t r u c t p c i _ d e v * dev , p m _ m e s s a g e _ t s ta te ) ; /* D e v i c e s u s p e n d e d */
int (* s u s p e n d _ l a t e ) ( s t r u c t p c i _ d e v * dev , p m _ m e s s a g e _ t s ta te ) ;
int (* r e s u m e _ e a r l y ) ( s t r u c t p c i _ d e v * dev ) ;
int (* r e s u m e ) ( s t r u c t p c i _ d e v * dev ) ; /* D e v i c e w ok en up */
void (* s h u t d o w n ) ( s t r u c t p c i _ d e v * dev ) ;
int (* s r i o v _ c o n f i g u r e ) ( s t r u c t p c i _ d e v * dev , int n u m _ v f s ) ; /* PF pdev */
c on st s t r u c t p c i _ e r r o r _ h a n d l e r s * e r r _ h a n d l e r ;
c on st s t r u c t a t t r i b u t e _ g r o u p ** g r o u p s ;
s t r u c t d e v i c e _ d r i v e r d r i v e r ;
s t r u c t p c i _ d y n i d s d y n i d s ;
};
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 36/60

38. Xilinx driver
Xilinx driver only provides
fromhttps://github.com/Xilinx/dma_ip_drivers/blob/8b8c70b697f049649d5fa99be9c6bc4302d89ac9/
XDMA/linux-kernel/xdma/xdma_mod.c#L316:
s t a t i c s t r u c t p c i _ d r i v e r p c i _ d r i v e r = {
. name = D R V _ M O D U L E _ N A M E ,
. i d _ t a b l e = pci_ids ,
. p ro be = probe_one ,
. r e m o v e = r em ov e_ on e ,
. e r r _ h a n d l e r = & x d m a _ e r r _ h a n d l e r ,
};
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 37/60

39. Interface to user space
On the other side (towards the user space), Xilinx DMA driver provides several char
devices:
$ ll / dev | grep xdma
drwxr - xr - x 3 root root 60 Jun 20 17:19 xdma
crw - rw - rw - 1 root root 238 , 36 Jun 20 17:19 xdma0_c2h_0
crw - rw - rw - 1 root root 238 , 1 Jun 20 17:19 xdma0_control
crw - rw - rw - 1 root root 238 , 10 Jun 20 17:19 xdma0_events_0
crw - rw - rw - 1 root root 238 , 11 Jun 20 17:19 xdma0_events_1
[...]
crw - rw - rw - 1 root root 238 , 24 Jun 20 17:19 xdma0_events_14
crw - rw - rw - 1 root root 238 , 25 Jun 20 17:19 xdma0_events_15
crw - rw - rw - 1 root root 238 , 32 Jun 20 17:19 xdma0_h2c_0
crw - rw - rw - 1 root root 238 , 0 Jun 20 17:19 xdma0_user
crw - rw - rw - 1 root root 238 , 2 Jun 20 17:19 xdma0_xvc
h2c
AXI4 port (DMA to device)
user
AXI4-Lite port
c2h
AXI4 port (DMA from device)
eventsN
AXI4 port (DMA from device)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 38/60

40. Interface to user space
On the other side (towards the user space), Xilinx DMA driver provides several char
devices:
$ ll / dev | grep xdma
drwxr - xr - x 3 root root 60 Jun 20 17:19 xdma
crw - rw - rw - 1 root root 238 , 36 Jun 20 17:19 xdma0_c2h_0
crw - rw - rw - 1 root root 238 , 1 Jun 20 17:19 xdma0_control
crw - rw - rw - 1 root root 238 , 10 Jun 20 17:19 xdma0_events_0
crw - rw - rw - 1 root root 238 , 11 Jun 20 17:19 xdma0_events_1
[...]
crw - rw - rw - 1 root root 238 , 24 Jun 20 17:19 xdma0_events_14
crw - rw - rw - 1 root root 238 , 25 Jun 20 17:19 xdma0_events_15
crw - rw - rw - 1 root root 238 , 32 Jun 20 17:19 xdma0_h2c_0
crw - rw - rw - 1 root root 238 , 0 Jun 20 17:19 xdma0_user
crw - rw - rw - 1 root root 238 , 2 Jun 20 17:19 xdma0_xvc
h2c
AXI4 port (DMA to device)
user
AXI4-Lite port
c2h
AXI4 port (DMA from device)
eventsN
AXI4 port (DMA from device)
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 38/60

41. Interface to user application
To access the status and control registers (on AXI4-Lite interface), C variant:
#
int
if
p e r r o r
r e t u r n
}
void
if
p e r r o r
r e t u r n
}
u i n t 3 2 _ t
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 39/60

42. Interface to user application
To access the status and control registers (on AXI4-Lite interface), Python variant
c la ss
def
u s e r _ f i l e n a m e
self
self
def
self
os
def
bs
r e t u r n
def
bs
self
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 40/60

43. Hot-plug and hot-swap
IHot-plug requires that virtual memory is additional kernel arguments:
pci = realloc , assign - busses , hpmemsize =32 M
IListing of a hot-plug procedure is shown on the next slide
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 41/60

44. Hot-plug
[ 1 1 . 7 1 9 8 5 4 ] xdma : l o a d i n g out - of - tree m o d u l e t a i n t s k e r n e l .
[ 1 1 . 7 1 9 9 0 2 ] xdma : m o d u l e v e r i f i c a t i o n f a i l e d : s i g n a t u r e and / or r e q u i r e d key m i s s i n g - t a i n t i n g k e r n e l
[ 1 1 . 7 2 0 6 3 0 ] xdma : x d m a _ m o d _ i n i t : X i l i n x XDMA R e f e r e n c e D r i v e r xdma v 2 01 8 .3 .5 0
[ 1 1 . 7 2 0 6 3 1 ] xdma : x d m a _ m o d _ i n i t : d e s c _ b l e n _ m a x : 0 x f f f f f f f / 2 6 8 4 3 5 4 5 5 , s g d m a _ t i m e o u t : 10 sec .
[ 1 2 . 9 3 8 9 6 4 ] JAN : plug in AMC and push in the h a n d l e
[ 2 6 . 3 4 7 9 5 6 ] p c i e h p 0 0 0 0 : 0 4 : 0 8 . 0 : p c i e 2 0 4 : Slot (5): A t t e n t i o n b u t t o n p r e s s e d
[ 2 6 . 3 4 7 9 7 8 ] p c i e h p 0 0 0 0 : 0 4 : 0 8 . 0 : p c i e 2 0 4 : Slot (5) P o w e r i n g on due to b u t t o n pr es s
[ 2 6 . 5 0 5 5 0 6 ] p c i e h p 0 0 0 0 : 0 4 : 0 8 . 0 : p c i e 2 0 4 : Slot (5): Link Up
[ 2 7 . 4 2 0 1 9 4 ] pci 0 0 0 0 : 0 7 : 0 0 . 0 : [10 ee : 7 0 2 4 ] type 00 c la ss 0 x 0 7 0 0 0 1
...
[ 2 7 . 4 2 0 4 7 1 ] pci 0 0 0 0 : 0 7 : 0 0 . 0 : BAR 0: a s s i g n e d [ mem 0 xa6000000 -0 x a 6 7 f f f f f ]
[ 2 7 . 4 2 0 4 7 7 ] pci 0 0 0 0 : 0 7 : 0 0 . 0 : BAR 1: a s s i g n e d [ mem 0 xa6800000 -0 x a 6 8 0 f f f f ]
[ 2 7 . 4 2 0 4 8 2 ] p c i e p o r t 0 0 0 0 : 0 4 : 0 8 . 0 : PCI b r i d g e to [ bus 07]
[ 2 7 . 4 2 0 4 8 5 ] p c i e p o r t 0 0 0 0 : 0 4 : 0 8 . 0 : b r i d g e w i n d o w [ io 0 x4000 -0 x 4f f f ]
[ 2 7 . 4 2 0 4 8 9 ] p c i e p o r t 0 0 0 0 : 0 4 : 0 8 . 0 : b r i d g e w i n d o w [ mem 0 xa6000000 -0 x a 9 f f f f f f ]
[ 2 7 . 4 2 0 4 9 2 ] p c i e p o r t 0 0 0 0 : 0 4 : 0 8 . 0 : b r i d g e w i n d o w [ mem 0 x96000000 -0 x 9 7 f f f f f f 64 bit pref ]
...
[ 2 7 . 4 2 0 9 4 4 ] xdma : m a p _ s i n g l e _ b a r : BAR0 at 0 x a 6 0 0 0 0 0 0 m a p p e d at 0 x 0 0 0 0 0 0 0 0 4 c 4 8 9 4 e 6 , l e n g t h = 8 3 8 8 6 0 8 ( / 8 3 8 8 6 0 8 )
[ 2 7 . 4 2 0 9 7 7 ] xdma : m a p _ s i n g l e _ b a r : BAR1 at 0 x a 6 8 0 0 0 0 0 m a p p e d at 0 x 0 0 0 0 0 0 0 0 b 3 1 4 f a 4 2 , l e n g t h = 6 5 5 3 6 ( / 6 5 5 3 6 )
[ 2 7 . 4 2 0 9 8 0 ] xdma : m a p _ b a r s : c o n f i g bar 1 , pos 1.
[ 2 7 . 4 2 0 9 8 1 ] xdma : i d e n t i f y _ b a r s : 2 BARs : c o n f i g 1 , user 0 , b y p a s s -1.
[ 2 7 . 4 2 1 0 3 8 ] xdma : p r o b e _ o n e : 0 0 0 0 : 0 7 : 0 0 . 0 xdma0 , pdev 0 x 0 0 0 0 0 0 0 0 a 8 5 0 8 4 2 8 , xdev 0 x 0 0 0 0 0 0 0 0 a f f c a b 9 a , 0 x 0 0 0 0 0 0 0 0 9 f a 6 7 5 e 1 , usr 16 , ch 1 ,1.
[ 2 7 . 4 2 1 8 3 3 ] xdma : c d e v _ x v c _ i n i t : x cd ev 0 x 0 0 0 0 0 0 0 0 b 8 5 1 3 8 f 3 , bar 0 , o f f s e t 0 x 4 0 0 0 0 .
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 42/60

45. Summary of the software part
IDrivers are provided by FPGA vendors
IEasy-to-use user-space interface (for register access, DMA and interrupts)
IHot-swap facilitates development and improves up-time
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 43/60

46. Troubleshooting techniques
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 44/60

47. Three possible roles
INot all techniques are needed in all cases
IFollowing slides are marked with icons to indicate the target roles
IHere we distinguish between three different roles
Hardware Developer Application Developer System Integrator
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 45/60

48. Troubleshooting conguration
Linux command, expected output:
$ lspci
00:00.0 Host bridge : Intel Corporation Xeon E3 -1200 v6 /7 th Gen Core Processor Host Bridge / DRAM Registers ( rev 05)
00:01.0 PCI bridge : Intel Corporation Xeon E3 -1200 v5 / E3 -1500 v5 /6 th Gen Core Processor PCIe Controller ( x16 ) ( rev 05)
...
01:00.1 System peripheral : PLX Technology , Inc . Device 87 d0 ( rev ca )
01:00.2 System peripheral : PLX Technology , Inc . Device 87 d0 ( rev ca )
01:00.3 System peripheral : PLX Technology , Inc . Device 87 d0 ( rev ca )
01:00.4 System peripheral : PLX Technology , Inc . Device 87 d0 ( rev ca )
02:01.0 PCI bridge : PLX Technology , Inc . Device 8725 ( rev ca )
02:02.0 PCI bridge : PLX Technology , Inc . Device 8725 ( rev ca )
02:08.0 PCI bridge : PLX Technology , Inc . Device 8725 ( rev ca )
...
03:00.0 PCI bridge : PLX Technology , Inc . PEX 8748 48 - Lane , 12 - Port PCI Express Gen 3 (8 GT / s ) Switch , 27 x 27 mm FCBGA ( rev ca )
04:01.0 PCI bridge : PLX Technology , Inc . PEX 8748 48 - Lane , 12 - Port PCI Express Gen 3 (8 GT / s ) Switch , 27 x 27 mm FCBGA ( rev ca )
...
07:00.0 Serial controller : Xilinx Corporation Device 7024
0 d :00.0 Ethernet controller : Intel Corporation Ethernet Controller X710 for 10 GbE SFP + ( rev 02)
...
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 46/60

49. Troubleshooting conguration
Linux command
Expected output:
$ lspci
...
07:00.0 Serial controller : Xilinx Corporation Device 7024
...
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 47/60

50. Troubleshooting conguration
Linux command
More information
$ sudo lspci -s 07:00.0 -v
07:00.0 Serial controller : Xilinx Corporation Device 7024 ( prog - if 01 [16450])
Subsystem : Xilinx Corporation Device 0007
Physical Slot : 5
Flags : bus master , fast devsel , latency 0 , IRQ 17
Memory at 92800000 (32 - bit , non - prefetchable ) [ size =8 M ]
Memory at 93000000 (32 - bit , non - prefetchable ) [ size =64 K ]
Capabilities : [40] Power Management version 3
Capabilities : [48] MSI : Enable - Count =1/1 Maskable - 64 bit +
Capabilities : [60] Express Endpoint , MSI 00
Capabilities : [100] Device Serial Number 00 -00 -00 -00 -00 -00 -00 -00
Kernel driver in use : xdma
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 48/60

51. xdma driver
To enable more verbose output from the driver, un-comment line with LIBXDMADEBUG
in xdma/Makele:
EXTRA_CFLAGS := - I$ ( topdir )/ include $ ( XVC_FLAGS )
# EXTRA_CFLAGS += - D __ LIB XD MA _D EB UG __
# EXTRA_CFLAGS += - DINTE RNAL_T ESTIN G
or ne-grained control in libxdma/libxdma.h (enable printout for individual fuction, e.g.
interrupt handler) DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 49/60

52. Driver
Dmesg output:
[ 1332.118599] xdma : char_ sgdma_ llsee k : char _sgdma _llsee k : pos =2147483648
[ 1332.125051] xdma : c h a r _ s g d m a _ r e a d _ w r i t e : file 0 xffffffc87a6ea900 , priv 0 xffffffc875f3e0e8 , buf 0 x0000000020d21000 ,4096 , pos 2147483648 , W 0 , 0 - C2H0 - MM .
[ 1332.138547] xdma : x d m a _ r e q u e s t _ c b _ d u m p : request 0 xffffffc87a6a5e00 , total 4096 , ep 0 x80000000 , sw_desc 1 , sgt 0 xf fffff8 00fec bd68 .
[ 1332.150105] xdma : sgt_dump : sgt 0 xffffff800fecbd68 , sgl 0 xffffffc87161c980 , nents 1/1.
[ 1332.157929] xdma : sgt_dump : 0 , 0 xffffffc87161c980 , pg 0 xffffffbf1d772838 ,0+4096 , dma 0 x86b301000 ,4096.
[ 1332.167141] xdma : x d m a _ r e q u e s t _ c b _ d u m p : 0/1 , 0 x86b301000 , 4096.
[ 1332.172967] xdma : xdma_xfer_submit : 0 - C2H0 - MM , len 4096 sg cnt 1.
[ 1332.178964] xdma : transfer_init : transfer - > desc_bus = 0 x701b0000 .
[ 1332.184964] xdma : transfer_build : sw desc 0/1: 0 x86b301000 , 0 x1000 , ep 0 x80000000 .
[ 1332.192437] xdma : transfer_init : transfer 0 xfff fffc87 a6a5e1 8 has 1 descriptors
[ 1332.199566] xdma : xdma_xfer_submit : xfer , 4096 , ep 0 x80001000 , done 0 , sg 1/1.
[ 1332.206692] xdma : transfer_dump : xfer 0 xffffffc87a6a5e18 , state 0 x0 , f 0 x1 , dir 2 , len 4096 , last 1.
[ 1332.215729] xdma : transfer_dump : transfer 0 xffffffc87a6a5e18 , desc 1 , bus 0 x701b0000 , adj 1.
[ 1332.224070] xdma : dump_desc : 0 xfff fff800 fbc200 0 /0 x00 : 0 xad4b0013 0 xad4b0013 magic | extra_adjacent | control
[ 1332.233455] xdma : dump_desc : 0 xfff fff800 fbc200 4 /0 x04 : 0 x00001000 0 x00001000 bytes
[ 1332.240841] xdma : dump_desc : 0 xfff fff800 fbc200 8 /0 x08 : 0 x80000000 0 x80000000 src_addr_lo
[ 1332.248756] xdma : dump_desc : 0 xfff fff800 fbc200 c /0 x0c : 0 x00000000 0 x00000000 src_addr_hi
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 50/60

53. MCH conguration
MCH provides an overview of the link stati per individual port:
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 51/60

54. Link state
Monitor LTSSM with Integrated Logic Analyzer (ILA)
Shown above is the behavior of a faulty link - link should stay in state L0 (0x16).
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 52/60

55. Eye diagram with oscilloscope
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 53/60

56. Protocol decoding with oscilloscope
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 54/60

57. Application logic - unit tests
Fromhttps://github.com/j-marjanovic/chisel-stuff/tree/master/example-5-gunzip/vivado_tb_project
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 55/60

58. Summary
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 56/60

59. Additional resources
IL. Petrosyan, ”MicroTCA and PCIe HotSwap under Linux”,
http://webcast.desy.de/?m=201312&lang=en,
at MTCA Workshop for Industry and Research 2013, Hamburg
IJ. Marjanovic, ”Direct Memory Access with FPGA”,
https://github.com/MicroTCA-Tech-Lab/damc-tck7-fpga-bsp/blob/
demo-mtcaws2018/docs/demo/mtcaws2018_marjanovic_dma.pdf
at MTCA Workshop for Industry and Research 2018, Hamburg
(similar to this one, but more hands-on - Vivado project and SW available)
IJ. Corbet, A. Rubini, G. Kroah-Hartman, ”Linux Device Drivers, 3rd Edition”,
(quite old at this point, 4th Edition long over-due)
IM. Jackson, R. Budruk, ”PCI Express Technology 3.0”
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 57/60

60. Summary
IPCI Express is a protocol of choice for many institutes and vendors
IProvides low latency and high throughput
IFPGA vendors provide a lot of IP, drivers, ...
IModern FPGA development is focused on connecting IP cores together
IThere are a lot of ”tools” to troubleshoot PCIe-based systems
IMicroTCA Tech Lab provides BSP with PCIe for some of the boards, more BSPs (e.g.
for Zynq UltraScale MPSoc) to follow
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 58/60

61. MicroTCA Technology Lab
TRANSFER MTCA TO RESEARCH
AND INDUSTRY
ICustom developments
IHigh-end test & measurement services
ISystem conguration & integration
ILLRF design
Marketing.
Services & Support.
Tech-Shop.
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 59/60

62. Thank you
谢谢谢谢谢谢
https://techlab.desy.de
Deutsches Elektronen-Synchrotron DESY
A Research Centre of the Helmholtz Association
Notkestr. 85, 22607 Hamburg, Germany
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 60/60

63. Backup slides
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 1/10

64. memcpy meas: memcpy implementation
Inspired bydrivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c#L486
s t a t i c
s t r u c t
u32
void
u32
) {
void
u32
len
ds t3 2
wh il e
*
a d d r e s s
d st 32
len
}
}
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 2/10

65. memcpy meas: time measurement
see alsohttps://www.kernel.org/doc/html/v4.15/driver-api/device-io.html
long
s t r u c t
s t r u c t
u64
B U G _ O N
B U G _ O N
lro
B U G _ O N
B U G _ O N
p r i n t k nn
t0 _n s
x d m a _ m e m c p y _ f r o m _ d e v
t1 _n s
p r i n t k nn
d u r a t i o n _ n s
p r i n t k nn
d u r a t i o n _ n s
r e t u r n
}
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 3/10

66. memcpy meas: CPU 1
On Concurrent Tech AM900x
kernel 4.4.0-139-generic
Intel(R) Core(TM) i7-3612QE CPU @ 2.10GHz
[ 5 5 7 7 5 . 1 3 2 9 7 7 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 5 5 7 7 5 . 4 9 8 4 1 1 ] JAN : m e m c p y done
[ 5 5 7 7 5 . 4 9 8 4 1 4 ] JAN : d u r a t i o n 5 6 0 0 4 9 2 3 1 ns ( t1 = 1 1 1 3 1 2 9 4 5 4 1 9 9 6 2 , t0 = 1 1 1 3 1 2 3 8 5 3 7 0 7 3 1 )
[ 5 5 7 9 3 . 4 7 8 8 3 3 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 5 5 7 9 3 . 8 4 4 3 3 3 ] JAN : m e m c p y done
[ 5 5 7 9 3 . 8 4 4 3 3 5 ] JAN : d u r a t i o n 4 6 8 0 4 1 1 4 4 ns ( t1 = 1 1 1 3 5 2 2 5 6 8 7 5 6 5 0 , t0 = 1 1 1 3 5 1 7 8 8 8 3 4 5 0 6 )
[ 5 5 8 1 9 . 0 8 3 8 7 0 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 5 5 8 1 9 . 4 4 9 5 6 2 ] JAN : m e m c p y done
[ 5 5 8 1 9 . 4 4 9 5 6 4 ] JAN : d u r a t i o n 5 2 0 0 4 5 7 1 5 ns ( t1 = 1 1 1 4 0 1 6 0 5 2 1 3 6 3 3 , t0 = 1 1 1 4 0 1 0 8 5 1 6 7 9 1 8 )
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 4/10

67. memcpy meas: CPU 2
On ADLINK AMC-1000
kernel 4.15.0-39-generic
Intel(R) Core(TM)2 Duo CPU L7400 @ 1.50GHz
[ 3 5 7 . 3 3 3 2 4 8 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 3 5 7 . 7 9 2 1 5 2 ] JAN : m e m c p y done
[ 3 5 7 . 7 9 2 1 5 7 ] JAN : d u r a t i o n 4 5 8 8 9 8 8 9 1 ns ( t1 = 3 5 7 7 9 2 1 5 0 6 5 9 , t0 = 3 5 7 3 3 3 2 5 1 7 6 8 )
[ 3 6 0 . 0 6 1 1 5 1 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 3 6 0 . 5 2 0 0 6 8 ] JAN : m e m c p y done
[ 3 6 0 . 5 2 0 0 7 3 ] JAN : d u r a t i o n 4 5 8 9 1 0 7 6 2 ns ( t1 = 3 6 0 5 2 0 0 6 6 2 1 8 , t0 = 3 6 0 0 6 1 1 5 5 4 5 6 )
[ 3 6 2 . 6 4 7 2 2 5 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 3 6 3 . 1 0 6 2 6 2 ] JAN : m e m c p y done
[ 3 6 3 . 1 0 6 2 6 8 ] JAN : d u r a t i o n 4 5 9 0 3 0 0 5 1 ns ( t1 = 3 6 3 1 0 6 2 5 9 2 3 2 , t0 = 3 6 2 6 4 7 2 2 9 1 8 1 )
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 5/10

68. memcpy meas: CPU 3
On external CPU
kernel 4.15.0-39-generic
Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
[ 2 2 5 . 6 7 8 0 6 6 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 2 2 6 . 1 2 8 7 7 4 ] JAN : m e m c p y done
[ 2 2 6 . 1 2 8 7 7 5 ] JAN : d u r a t i o n 4 5 0 7 0 2 9 6 5 ns ( t1 = 2 2 5 8 3 1 9 1 5 2 1 6 , t0 = 2 2 5 3 8 1 2 1 2 2 5 1 )
[ 2 2 9 . 4 9 3 6 4 6 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 2 2 9 . 9 4 4 7 4 2 ] JAN : m e m c p y done
[ 2 2 9 . 9 4 4 7 4 3 ] JAN : d u r a t i o n 4 5 1 0 9 0 8 8 5 ns ( t1 = 2 2 9 6 4 7 8 5 4 4 6 1 , t0 = 2 2 9 1 9 6 7 6 3 5 7 6 )
[ 2 3 2 . 2 8 5 6 4 9 ] JAN : m e m c p y started , size : 1 0 4 8 5 7 6
[ 2 3 2 . 7 3 6 6 2 6 ] JAN : m e m c p y done
[ 2 3 2 . 7 3 6 6 2 7 ] JAN : d u r a t i o n 4 5 0 9 7 1 6 4 7 ns ( t1 = 2 3 2 4 3 9 7 1 6 8 4 6 , t0 = 2 3 1 9 8 8 7 4 5 1 9 9 )
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 6/10

69. xdma tools: regrw
regrw tool, e.g. read
$ regrw / dev / xdma / card0 / user 0x10038 w
argc = 4
device : / dev / xdma / card0 / user
address : 0x00010038
access type : w r i t e
access width given .
access width : word (32b i t s )
character device / dev / xdma / card0 / user opened .
Memory mapped at address 0x7f34356fb000 .
Read 32b i t value at address 0x00010038 (0 x7f343570b038 ) : 0x000107ff
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 7/10

70. xdma tools: regrw
regrw tool, e.g. write
$ regrw / dev / xdma / card0 / user 0x10038 w 0 x 1 0 7 f f
argc = 5
device : / dev / xdma / card0 / user
address : 0x00010038
access type : w r i t e
access width given .
access width : word (32b i t s )
character device / dev / xdma / card0 / user opened .
Memory mapped at address 0x7f1675e65000 .
Write 32b i t s value 0x000107ff to 0x00010038 (0 x0x7f1675e75038 )
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 8/10

71. Driver
Dmesg output:
[ 1 3 3 2 . 1 1 8 5 9 9 ] x d m a : c h a r _ s g d m a _ l l s e e k : c h a r _ s g d m a _ l l s e e k : p o s = 2 1 4 7 4 8 3 6 4 8
[ 1 3 3 2 . 1 2 5 0 5 1 ] x d m a : c h a r _ s g d m a _ r e a d _ w r i t e : f i l e 0 x f f f f f f c 8 7 a 6 e a 9 0 0 , p r i v 0 x f f f f f f c 8 7 5 f 3 e 0 e 8 , b u f 0 x 0 0 0 0 0 0 0 0 2 0 d 2 1 0 0 0 , 4 0 9 6 , p o s 2 1 4 7 4 8 3 6 4 8 , W 0 , 0 - C 2 H 0 - M M .
[ 1 3 3 2 . 1 3 8 5 4 7 ] x d m a : x d m a _ r e q u e s t _ c b _ d u m p : r e q u e s t 0 x f f f f f f c 8 7 a 6 a 5 e 0 0 , t o t a l 4 0 9 6 , e p 0 x 8 0 0 0 0 0 0 0 , s w _ d e s c 1 , s g t 0 x f f f f f f 8 0 0 f e c b d 6 8 .
[ 1 3 3 2 . 1 5 0 1 0 5 ] x d m a : s g t _ d u m p : s g t 0 x f f f f f f 8 0 0 f e c b d 6 8 , s g l 0 x f f f f f f c 8 7 1 6 1 c 9 8 0 , n e n t s 1 / 1 .
[ 1 3 3 2 . 1 5 7 9 2 9 ] x d m a : s g t _ d u m p : 0 , 0 x f f f f f f c 8 7 1 6 1 c 9 8 0 , p g 0 x f f f f f f b f 1 d 7 7 2 8 3 8 , 0 + 4 0 9 6 , d m a 0 x 8 6 b 3 0 1 0 0 0 , 4 0 9 6 .
[ 1 3 3 2 . 1 6 7 1 4 1 ] x d m a : x d m a _ r e q u e s t _ c b _ d u m p : 0 / 1 , 0 x 8 6 b 3 0 1 0 0 0 , 4 0 9 6 .
[ 1 3 3 2 . 1 7 2 9 6 7 ] x d m a : x d m a _ x f e r _ s u b m i t : 0 - C 2 H 0 - MM , l e n 4 0 9 6 s g c n t 1 .
[ 1 3 3 2 . 1 7 8 9 6 4 ] x d m a : t r a n s f e r _ i n i t : t r a n s f e r - > d e s c _ b u s = 0 x 7 0 1 b 0 0 0 0 .
[ 1 3 3 2 . 1 8 4 9 6 4 ] x d m a : t r a n s f e r _ b u i l d : s w d e s c 0 / 1 : 0 x 8 6 b 3 0 1 0 0 0 , 0 x 1 0 0 0 , e p 0 x 8 0 0 0 0 0 0 0 .
[ 1 3 3 2 . 1 9 2 4 3 7 ] x d m a : t r a n s f e r _ i n i t : t r a n s f e r 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 h a s 1 d e s c r i p t o r s
[ 1 3 3 2 . 1 9 9 5 6 6 ] x d m a : x d m a _ x f e r _ s u b m i t : x f e r , 4 0 9 6 , e p 0 x 8 0 0 0 1 0 0 0 , d o n e 0 , s g 1 / 1 .
[ 1 3 3 2 . 2 0 6 6 9 2 ] x d m a : t r a n s f e r _ d u m p : x f e r 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 , s t a t e 0 x0 , f 0 x1 , d i r 2 , l e n 4 0 9 6 , l a s t 1 .
[ 1 3 3 2 . 2 1 5 7 2 9 ] x d m a : t r a n s f e r _ d u m p : t r a n s f e r 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 , d e s c 1 , b u s 0 x 7 0 1 b 0 0 0 0 , a d j 1 .
[ 1 3 3 2 . 2 2 4 0 7 0 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 0 0 / 0 x 0 0 : 0 x a d 4 b 0 0 1 3 0 x a d 4 b 0 0 1 3 m a g i c | e x t r a _ a d j a c e n t | c o n t r o l
[ 1 3 3 2 . 2 3 3 4 5 5 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 0 4 / 0 x 0 4 : 0 x 0 0 0 0 1 0 0 0 0 x 0 0 0 0 1 0 0 0 b y t e s
[ 1 3 3 2 . 2 4 0 8 4 1 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 0 8 / 0 x 0 8 : 0 x 8 0 0 0 0 0 0 0 0 x 8 0 0 0 0 0 0 0 s r c _ a d d r _ l o
[ 1 3 3 2 . 2 4 8 7 5 6 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 0 c / 0 x 0 c : 0 x 0 0 0 0 0 0 0 0 0 x 0 0 0 0 0 0 0 0 s r c _ a d d r _ h i
[ 1 3 3 2 . 2 5 6 6 6 7 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 1 0 / 0 x 0 0 : 0 x 6 b 3 0 1 0 0 0 0 x 6 b 3 0 1 0 0 0 d s t _ a d d r _ l o
[ 1 3 3 2 . 2 6 4 5 7 8 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 1 4 / 0 x 0 4 : 0 x 0 0 0 0 0 0 0 8 0 x 0 0 0 0 0 0 0 8 d s t _ a d d r _ h i
[ 1 3 3 2 . 2 7 2 4 9 2 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 1 8 / 0 x 0 8 : 0 x 0 0 0 0 0 0 0 0 0 x 0 0 0 0 0 0 0 0 n e x t _ a d d r
[ 1 3 3 2 . 2 8 0 2 2 8 ] x d m a : d u m p _ d e s c : 0 x f f f f f f 8 0 0 f b c 2 0 1 c / 0 x 0 c : 0 x 0 0 0 0 0 0 0 0 0 x 0 0 0 0 0 0 0 0 n e x t _ a d d r _ p a d
[ 1 3 3 2 . 2 8 8 3 1 0 ] x d m a : d u m p _ d e s c :
[ 1 3 3 2 . 2 9 1 1 8 3 ] x d m a : t r a n s f e r _ q u e u e : t r a n s f e r _ q u e u e ( t r a n s f e r = 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 ) .
[ 1 3 3 2 . 2 9 8 3 0 8 ] x d m a : t r a n s f e r _ q u e u e : t r a n s f e r _ q u e u e ( ) : s t a r t i n g 0 - C 2 H 0 - M M e n g i n e .
[ 1 3 3 2 . 3 0 5 4 3 0 ] x d m a : e n g i n e _ s t a r t : e n g i n e _ s t a r t ( 0 - C 2 H 0 - M M ) : t r a n s f e r = 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 .
[ 1 3 3 2 . 3 1 3 1 5 6 ] x d m a : e n g i n e _ s t a r t : i o w r i t e 3 2 ( 0 x 7 0 1 b 0 0 0 0 t o 0 x f f f f f f 8 0 0 f b a 5 0 8 0 ) ( f i r s t _ d e s c _ l o )
[ 1 3 3 2 . 3 2 1 4 0 4 ] x d m a : _ _ w r i t e _ r e g i s t e r : e n g i n e _ s t a r t : w r e g 0 x f f f f f f b 7 9 5 8 c 6 6 e 0 ( 0 x f f f f f f 8 0 0 f b a 5 0 8 0 ) , 0 x 7 0 1 b 0 0 0 0 .
[ 1 3 3 2 . 3 3 1 0 3 9 ] x d m a : e n g i n e _ s t a r t : i o w r i t e 3 2 ( 0 x 0 0 0 0 0 0 0 0 t o 0 x f f f f f f 8 0 0 f b a 5 0 8 4 ) ( f i r s t _ d e s c _ h i )
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 9/10

72. Driver
Dmesg output:
[ 1 3 3 2 . 3 3 9 2 8 6 ] x d m a : _ _ w r i t e _ r e g i s t e r : e n g i n e _ s t a r t : w r e g 0 x f f f f f f b 7 9 5 8 c 6 6 e 4 ( 0 x f f f f f f 8 0 0 f b a 5 0 8 4 ) , 0 x 0 .
[ 1 3 3 2 . 3 4 8 3 1 4 ] x d m a : e n g i n e _ s t a r t : i o w r i t e 3 2 ( 0 x 0 0 0 0 0 0 0 0 t o 0 x f f f f f f 8 0 0 f b a 5 0 8 8 ) ( f i r s t _ d e s c _ a d j a c e n t )
[ 1 3 3 2 . 3 5 7 0 8 2 ] x d m a : _ _ w r i t e _ r e g i s t e r : e n g i n e _ s t a r t : w r e g 0 x f f f f f f b 7 9 5 8 c 6 6 e 8 ( 0 x f f f f f f 8 0 0 f b a 5 0 8 8 ) , 0 x 0 .
[ 1 3 3 2 . 3 6 6 1 0 9 ] x d m a : e n g i n e _ s t a r t : i o r e a d 3 2 ( 0 x f f f f f f 8 0 0 f b a 1 0 4 0 ) ( d u m m y r e a d f l u s h e s w r i t e s ) .
[ 1 3 3 2 . 3 7 4 1 8 3 ] x d m a : e n g i n e _ s t a r t _ m o d e _ c o n f i g : i o w r i t e 3 2 ( 0 x 0 0 f 8 3 e 1 f t o 0 x f f f f f f 8 0 0 f b a 1 0 0 4 ) ( c o n t r o l )
[ 1 3 3 2 . 3 8 2 9 5 1 ] x d m a : _ _ w r i t e _ r e g i s t e r : e n g i n e _ s t a r t _ m o d e _ c o n f i g : w r e g 0 x f f f f f f b 7 9 5 8 c 2 6 6 c ( 0 x f f f f f f 8 0 0 f b a 1 0 0 4 ) , 0 x f 8 3 e 1 f .
[ 1 3 3 2 . 3 9 3 4 5 7 ] x d m a : e n g i n e _ s t a r t _ m o d e _ c o n f i g : i o r e a d 3 2 ( 0 x f f f f f f 8 0 0 f b a 1 0 4 0 ) = 0 x 0 0 0 0 0 0 0 1 ( d u m m y r e a d f l u s h e s w r i t e s ) .
[ 1 3 3 2 . 3 9 3 4 7 7 ] x d m a : x d m a _ i s r : ( i r q = 5 0 , d e v 0 x f f f f f f c 8 7 a 2 d e 0 0 0 ) < < < < I S R .
[ 1 3 3 2 . 3 9 3 4 8 2 ] x d m a : x d m a _ i s r : c h _ i r q = 0 x 0 0 0 0 0 0 0 2
[ 1 3 3 2 . 3 9 3 4 8 5 ] x d m a : _ _ w r i t e _ r e g i s t e r : c h a n n e l _ i n t e r r u p t s _ d i s a b l e : w r e g 0 x 2 0 0 0 ( 0 x f f f f f f 8 0 0 f b a 2 0 1 8 ) , 0 x 2 .
[ 1 3 3 2 . 3 9 3 4 8 9 ] x d m a : x d m a _ i s r : u s e r _ i r q = 0 x 0 0 0 0 0 0 0 0
[ 1 3 3 2 . 3 9 3 4 9 1 ] x d m a : x d m a _ i s r : s c h e d u l e _ w o r k , 0 - C 2 H 0 - M M .
[ 1 3 3 2 . 4 3 3 3 1 0 ] x d m a : e n g i n e _ s t a r t : 0 - C 2 H 0 - M M e n g i n e 0 x f f f f f f c 8 7 a 2 d e 9 7 8 n o w r u n n i n g
[ 1 3 3 2 . 4 4 0 5 1 4 ] x d m a : t r a n s f e r _ q u e u e : t r a n s f e r = 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 s t a r t e d 0 - C 2 H 0 - M M e n g i n e w i t h t r a n s f e r 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 .
[ 1 3 3 2 . 4 5 1 2 7 7 ] x d m a : t r a n s f e r _ q u e u e : e n g i n e - > r u n n i n g = 1
[ 1 3 3 2 . 4 5 6 2 2 6 ] x d m a : e n g i n e _ s e r v i c e _ w o r k : e n g i n e _ s e r v i c e ( ) f o r 0 - C 2 H 0 - M M e n g i n e f f f f f f c 8 7 a 2 d e 9 7 8
[ 1 3 3 2 . 4 6 4 6 4 9 ] x d m a : e n g i n e _ s e r v i c e _ s h u t d o w n : e n g i n e j u s t w e n t i d l e , r e s e t t i n g R U N _ S T O P .
[ 1 3 3 2 . 4 7 2 3 8 1 ] x d m a : x d m a _ e n g i n e _ s t o p : x d m a _ e n g i n e _ s t o p ( e n g i n e = f f f f f f c 8 7 a 2 d e 9 7 8 )
[ 1 3 3 2 . 4 7 9 4 1 3 ] x d m a : x d m a _ e n g i n e _ s t o p : S t o p p i n g S G D M A 0 - C 2 H 0 - M M e n g i n e ; w r i t i n g 0 x 0 0 f 8 3 e 1 e t o 0 x f f f f f f 8 0 0 f b a 1 0 0 4 .
[ 1 3 3 2 . 4 8 9 3 9 6 ] x d m a : _ _ w r i t e _ r e g i s t e r : x d m a _ e n g i n e _ s t o p : w r e g 0 x f f f f f f b 7 9 5 8 c 2 6 6 c ( 0 x f f f f f f 8 0 0 f b a 1 0 0 4 ) , 0 x f 8 3 e 1 e .
[ 1 3 3 2 . 4 9 9 2 0 4 ] x d m a : x d m a _ e n g i n e _ s t o p : x d m a _ e n g i n e _ s t o p ( 0 - C 2 H 0 - M M ) d o n e
[ 1 3 3 2 . 5 0 5 4 5 7 ] x d m a : e n g i n e _ s e r v i c e : d e s c _ c o u n t = 1
[ 1 3 3 2 . 5 0 9 9 6 9 ] x d m a : e n g i n e _ s e r v i c e : h e a d o f q u e u e t r a n s f e r 0 x f f f f f f c 8 7 a 6 a 5 e 1 8 h a s 1 d e s c r i p t o r s
[ 1 3 3 2 . 5 1 8 3 8 9 ] x d m a : e n g i n e _ s e r v i c e : E n g i n e c o m p l e t e d 1 d e s c , 1 n o t y e t d e q u e u e d
[ 1 3 3 2 . 5 2 5 4 2 1 ] x d m a : e n g i n e _ s e r v i c e _ f i n a l _ t r a n s f e r : e n g i n e 0 - C 2 H 0 - M M c o m p l e t e d t r a n s f e r
[ 1 3 3 2 . 5 3 3 0 6 0 ] x d m a : e n g i n e _ s e r v i c e _ f i n a l _ t r a n s f e r : C o m p l e t e d t r a n s f e r I D = 0 x f f f f f f c 8 7 a 6 a 5 e 1 8
[ 1 3 3 2 . 5 4 1 3 0 7 ] x d m a : e n g i n e _ s e r v i c e _ f i n a l _ t r a n s f e r : * p d e s c _ c o m p l e t e d = 1 , t r a n s f e r - > d e s c _ n u m = 1
[ 1 3 3 2 . 5 4 1 3 1 1 ] x d m a : e n g i n e _ s e r v i c e _ r e s u m e : n o p e n d i n g t r a n s f e r s , 0 - C 2 H 0 - M M e n g i n e s t a y s i d l e .
[ 1 3 3 2 . 5 5 7 6 3 6 ] x d m a : _ _ w r i t e _ r e g i s t e r : c h a n n e l _ i n t e r r u p t s _ e n a b l e : w r e g 0 x 2 0 0 0 ( 0 x f f f f f f 8 0 0 f b a 2 0 1 4 ) , 0 x 2 .
[ 1 3 3 2 . 5 6 6 7 5 9 ] x d m a : x d m a _ x f e r _ s u b m i t : t r a n s f e r f f f f f f c 8 7 a 6 a 5 e 1 8 , 4 0 9 6 , e p 0 x 8 0 0 0 0 0 0 0 c o m p l , + 0 .
DMA transfer, PCIe Driver and FPGA Tools
Jan Marjanovic (MTCA Tech Lab/DESY), 2019-06-24 Page 10/10