the FC7 AMC for DAQ & control applications in CMS

FC7 project

introduction
- concept
- motivations
- specifications hardware overview
- highlights
- high frequency performance acceptance testing & manufacturing
- test stands
- manufacturing & production experience uses in CMS summary

1. project
the FC7 AMC for
DAQ & control applications in CMS
Mark Pesaresi (Imperial College), Paschalis Vichoudis (CERN)

Magnus Hansen, Manoel Barros Marin, Francois Vasey (CERN)
Greg Iles, Sarah Greenwood, Andrew Rose, Geoff Hall (Imperial College)

2. | FC7 | Mark Pesaresi | TWEPP 2014 2
content
introduction
- concept
- motivations
- specifications

hardware overview
- highlights
- high frequency performance

acceptance testing & manufacturing
- test stands
- manufacturing & production experience

uses in CMS

summary

3. | FC7 | Mark Pesaresi | TWEPP 2014 3
concept
the FC7 AMC is a flexible, uTCA compatible card for generic CMS data acquisition & system control uses

the FC7 is targeted for
o both production systems or prototyping / bench-top use
o uTCA for CMS applications (compatibility with AMC13) or standard uTCA
o users of high speed optical links up to 10Gbps

4. Imperial MP7
CERN GLIB
| FC7 | Mark Pesaresi | TWEPP 2014 4
design motivations (I)
basing on existing hardware helps to reduce development time to maturity
o especially with regards to reuse of firmware and software components
o fewer nasty surprises!
designed to be an evolution of
o the Gigabit Link Interface Board (GLIB) – a highly
successful project filling a similar role, with emphasis on GBT
link testing



using design components, novel features and layout guidelines from
o the MP7 – a high optical density O(Tbps) processing card to
be used in the upgraded CMS Calorimeter Trigger

5. | FC7 | Mark Pesaresi | TWEPP 2014 5
design motivations (II)
principally designed with two CMS users in mind

o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions

o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability

6. | FC7 | Mark Pesaresi | TWEPP 2014 6
design motivations (II)
principally designed with two CMS users in mind

o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions

o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability


- compatibility with uTCA for CMS (AMC13)
- low jitter clock distribution & deterministic latency
- capability to run 10Gbps SERDES links

7. | FC7 | Mark Pesaresi | TWEPP 2014 7
design motivations (II)
principally designed with two CMS users in mind

o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC
control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow
commands, reception of synchronous detector status, centralised trigger rules, more partitions

o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the
upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased
readout bandwidth to DAQ, robustness to readout errors, increased processing capability


- compatibility with uTCA for CMS (AMC13)
- low jitter clock distribution & deterministic latency
- capability to run 10Gbps SERDES links
additionally: continued capability to run GBT (4.8Gbps) systems

8. | FC7 | Mark Pesaresi | TWEPP 2014 8
FC7: top view
looks like a
simple board…

what’s the
fuss?

9. | FC7 | Mark Pesaresi | TWEPP 2014 9
FC7: bottom view
OK

that might
explain why
it’s so heavy…

10. | FC7 | Mark Pesaresi | TWEPP 2014 10
main features
- full size, double width AMC module for uTCA

- Kintex 7 FPGA supporting serial links up to 10Gbps

- 4Gb DDR3 RAM (128M x 32b)

- 2 FMC sockets fully compatible with Low Pin Count FMCs
o additionally supports up to 20 high speed serial
lanes to the front panel
o supports dual-width, legacy & non-standard FMCs

- flexible backplane configuration
o high speed connectivity on up to 12 ports
o fully compatible with uTCA for CMS and AMC13
dual QSFP+
8 x SFP+
2x 14b A/D, 2x 16b D/A

11. | FC7 | Mark Pesaresi | TWEPP 2014 11
development history
November 2012
June 2013
July /August 2013
September 2013
November 2013
January 2014
June/July 2014
Today
Spec approved
First prototypes (R0a) submitted
First prototypes received
Majority of design validated, one minor design issue
Some serious manufacturing issues uncovered
Begin assembly of test system & firmware
Informal price enquiry to trial new manufacturers
Pre-production series (R0b) submitted with two manufacturers
Pre-production batches received
Excellent manufacturing quality with one supplier
Order for 70 pieces placed
70 production boards received (TCDS)
Minor design update (R1) submitted
Order of 120 pieces in production (pixFED)
development

production

12. | FC7 | Mark Pesaresi | TWEPP 2014 12
feature highlights
Kintex 7
FPGA
high range Kintex 7 XC7K420T

pin compatible with XC7K480T
(difference only in logic)
FMC 1
FC7 is FMC spec compliant
but can also accommodate
non-standard mezzanines &
stacking heights
‘-2’ speed grade – MGTs support
10Gbps (10.3125Gbps max)

supports almost any MGT protocol
to FMCs (e.g. GBT, 10GbE, custom)
12 MGT pairs
68 user IO
(34 differential)
I/O operates up to 3.3V signalling -
supports legacy FMCs

13. | FC7 | Mark Pesaresi | TWEPP 2014 13
feature highlights
switched JTAG header
FPGA, microcontroller, FMCs
backplane JTAG option
unique laser etched QR code
unique EEPROM h/w address

automatic logging & inventory,
configuration with CERN assigned
MAC address on acceptance
external 12V power header

desktop/crate mode switch
(requires additional h/w for GbE control)

14. | FC7 | Mark Pesaresi | TWEPP 2014 14
feature highlights
DDR3
SRAM
AMC connector

4Gb DDR3 specified
up to 480MHz
30Gbps maximum
bandwidth (R+W)
designed to be compatible
with both uTCA for CMS and
other AMC.x standards

supports range of MGT
protocols across backplane
Port 0 GbE
Port 1 GbE or AMC13-DAQ*
Port 2 SATA/SAS
Port 3 SATA/SAS or AMC13-TTC/S (LVDS)
Ports 4-7 PCIe/SRIO*
Ports 8-11 Ext Fat-Pipe (SRIO/10GbE XAUI)*
Ports 12-15 LVDS I/O
Ports 17-20 unused
FCLKA AMC13-TTC or PCIe clocks
TCLKA/TCLKC telecoms clock inputs
TCLKB/TCLKD telecoms clock outputs

15. | FC7 | Mark Pesaresi | TWEPP 2014 15
feature highlights
flash
uC
uSD
firmware image repository including a
fallback inviolate ‘Golden’ image

allows fast write (~10s) speeds & instant
FPGA reconfiguration over IPBus
(compared to 5min for JTAG->SPI flash)
ATMEL microcontroller runs
imperial_mmc v1.6 code
(shared with MP7)

16. | FC7 | Mark Pesaresi | TWEPP 2014 16
feature highlights
16 layer PCB
Nelco N4000 13-EPSI
- low loss tangent for HF
- low dielectric constant
-misaligned with PCB weave

strict constraint on thickness

17. | FC7 | Mark Pesaresi | TWEPP 2014 17
prototype performance testing
electrical TX testing at 10Gbps
XILINX SERDES electrical
breakout FMC (XM104)
optical TX & loopback testing at 10Gbps
Faster Technology
8x SFP+ FMC (FM-S18)
20ps total jitter
1.7ps ch-ch spread

specification <27ps
BER <10
-12
(PRBS31)

performance limited by
optical margin/SFPs

18. | FC7 | Mark Pesaresi | TWEPP 2014 18
production acceptance testing (I)
GPIB-controllable
power supply
GPIB-USB controller
ATMEL programmer
(microcontroller
configuration)
XILINX programmer
(CPLD configuration)
SD card
(FPGA configuration)

19. | FC7 | Mark Pesaresi | TWEPP 2014 19
production acceptance testing (II)
AMC extender card (CERN)
Ports 1-15 loopback
Port 0 IPBus control
electrical loopback FMCs
(commercial)

20. | FC7 | Mark Pesaresi | TWEPP 2014 20
production acceptance testing (III)
- visual inspection
- scan QR code
- connect to the testbench

1.Power up
2.Configure CPLD
3.Configure microcontroller
4.Power cycle to load FPGA (from SD)
5.Check ethernet link with FPGA (ping)
6.Verify on-board voltages (via the monitoring circuitry)
7.Identify the board (i2c readable S/N)
8.Assign MAC address (from LUT based on label)
9.Run DDR3 memory test (@960MT/s)
10.Run FMC loopback test (standard I/O)
11.Increase current limit for SERDES tests
12.Run FMC loopback test (SERDES @ 10Gb/s)
13.Run AMC loopback test (SERDES @ 5Gb/s)
14.Power down

21. | FC7 | Mark Pesaresi | TWEPP 2014 21
production acceptance testing (III)
- visual inspection
- scan QR code
- connect to the testbench

1.Power up
2.Configure CPLD
3.Configure microcontroller
4.Power cycle to load FPGA (from SD)
5.Check ethernet link with FPGA (ping)
6.Verify on-board voltages (via the monitoring circuitry)
7.Identify the board (i2c readable S/N)
8.Assign MAC address (from LUT based on label)
9.Run DDR3 memory test (@960MT/s)
10.Run FMC loopback test (standard I/O)
11.Increase current limit for SERDES tests
12.Run FMC loopback test (SERDES @ 10Gb/s)
13.Run AMC loopback test (SERDES @ 5Gb/s)
14.Power down

22. | FC7 | Mark Pesaresi | TWEPP 2014 22
manufacturing experiences
submitted prototype and pre-production series runs with two different manufacturers

o one well known to CERN/UK, one with no prior experience of
o experimented with two PCB materials compatible with the impedances & high speed
signal integrity required



large fraction of development period dedicated to tracking down & resolving manufacturing issues

o two manufacturers extremely useful to identify manufacturing vs. design faults (whether
due to designer error or board complexity)
o have learnt the importance of good, two-way communication with your supplier

23. | FC7 | Mark Pesaresi | TWEPP 2014 23
example 1: PCB issues
for 10Gbps we can’t use simple FR4…


Nelco N4000 laminate (Park) has very good high frequency characteristics
but now known to be susceptible to delamination & registration defects
o also an expensive material, low yield
o requires careful handling during manufacture & assembly;
e.g. contamination, humidity, life time, temperature profiles
o working guidelines between PCB/assembly manufacturers compiled

TU-872 SLK laminate (TUC) trialled successfully but also tends to suffer
similar registration defects – special measures during construction


even more fancy HF materials now available but even less well known
o key is building communicative relationship with supplier
real x-ray representations

24. | FC7 | Mark Pesaresi | TWEPP 2014 24
example 2: assembly issues
concerning assembly, one of the suppliers demonstrated an excellent
quality of work

continuous improvement in the assembly process

o detailed written feedback provided after each batch

o use X-ray imaging to identify soldering issues

o identified washing process behind an issue with a mechanical
switch – assembly process changed accordingly

o improved packaging guidelines after damage during transport

25. | FC7 | Mark Pesaresi | TWEPP 2014 25
pre & production yield
best manufacturer chosen for supply of production FC7s

total 80 pre-production and production boards manufactured between Nov ’13 - Jun ‘14


70 passed all tests first time around, 8 returned to manufacturer

78 passed tests second time around
o 1 board damaged in transit
o 1 board with unidentified issue on backplane

final yield >97%

26. | FC7 | Mark Pesaresi | TWEPP 2014 26
firmware
mature firmware
development
- based on IPBus


system related logic is
segregated from user


reuse of GLIB f/w blocks,
similar user logic structure


suite of scripts for
configuration over IPBus
high level f/w diagram

27. | FC7 | Mark Pesaresi | TWEPP 2014 27
FC7 for CMS TCDS
with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS
a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards
a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs
AMC13
local partition
managers
partition
interface cards
P5

28. | FC7 | Mark Pesaresi | TWEPP 2014 28
FC7 for CMS TCDS
with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS
a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards
a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs
AMC13
local partition
managers
partition
interface cards
TTC/TTS to-
from AMC13
to DAQ
TTC/TTS
TTC/TTS to-
from FEDs
P5

29. | FC7 | Mark Pesaresi | TWEPP 2014 29
FC7 for CMS pixFED
with a single custom FMC, the FC7 can fully satisfy the role of the pixel FED

2x 12 channel parallel pluggable optical receivers (Fitel)
- running at 400Mbps, reads in data directly from detector

SFP+/QSFP+ required for DAQ (not pictured)
- 10Gbps readout, to be prototyped

total 70 boards required for pixFED, possibly more


with the TCDS FMC, the FC7 could also satisfy the role
of the pixel Front End Controller (FEC)

nothing special required, R&D ongoing
~30 boards required for pixFEC

30. | FC7 | Mark Pesaresi | TWEPP 2014 30
summary
flexible Kintex 7 based uTCA compatible AMC developed for generic DAQ/detector control uses in CMS
o suitable both for small scale test systems or production systems , compatible with a range of FMCs
o able to operate up to 10Gbps line rates across the front panel
o advanced FPGA configuration features

successful production of 80 pieces to date, another batch of 120 currently in production
o short development timescale
o mature system firmware, isolated from user firmware, test script suite

stress the importance of good communication with your supplier
o especially helpful when requesting non standard PCB constructions
o automated acceptance testing for quickly identifying manufacture issues

FC7s already being installed in P5
o TCDS ready for operation in 2015
o pixel FED development under way

31. | FC7 | Mark Pesaresi | TWEPP 2014 31
links
https://espace.cern.ch/project-FC7

firmware & software:
https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/

public documentation:
https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/trunk/fc7_r0/doc/
project
thank you for your attention
with special thanks to: Jan Troska, Jeroen Hegeman, Karl Gill