N7K Hardware Architecture and it's components

Cisco Nexus 7000 Switch Architecture
BRKARC-3470
Ron Fuller, CCIE#5851 (R&S/Storage)
Technical Marketing Engineer

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 3
Session Abstract
This session presents an in-depth study of the architecture of the latest
generation of Nexus 7000 and Nexus 7700 data centreswitches. Topics include
supervisors, fabrics, I/O modules, forwarding engines, and physical design
elements, as well as a discussion of key hardware-enabled features that combine
to implement high-performance data centrenetwork services.

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 4
Session Goal
To provide a thorough understanding of the Nexus 7000 / Nexus 7700
switching architecture, supervisor, fabric, and I/O module design, packet flows,
and key forwarding engine functions
This session will examine the Nexus 7700 system, as well as the latest
additions to the Nexus 7000
This session will not examine NX-OS software architecture or other Nexus
platform architectures
4

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 5
What Is Nexus 7000?
Data-centreclass Ethernet switch designed to deliver high performance, high availability,
system scale, and investment protection
Nexus 7000 designed for general-purpose Data Centre deployments, focused on 10G
density plus 40G/100G
I/O Modules
Supervisor Engines
Fabrics
Chassis

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 6
What Is Nexus 7700?
Data-centreclass Ethernet switch designed to deliver high performance, high availability,
system scale, and investment protection
Nexus 7700 designed for SP and MSDC Data Centre deployments, focused on high-
density 40G/100G
I/O Modules
Supervisor Engine
Fabrics
Chassis

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 7
Nexus 7000
General purpose DC switching w/10/40/100G
Nexus 7700
Targeted at Densest 40G/100G deployments
Common Foundation
•Same release vehicles, versioning, feature-sets
•Common configuration model
•Common operational model
•Common fabric ASICs (Fab2) and architecture
•Same central arbitration model
•Same VOQ/QoS model
•Identical forwarding ASICs (F2E, F3)
•Consistent hardware feature sets
•Parallel evolution of hardware capability/scale
Nexus 7000 / Nexus 7700 –Common Foundation

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 8
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 9
Nexus 7000 Chassis Family
Front Rear
21RU
N7K-C7010
25RU
Front Rear
N7K-C7018
Front Rear
N7K-C7009
14RU
NX-OS 4.1(2) and later
NX-OS 5.2(1) and later
Nexus 7010 Nexus 7018
Nexus 7009
Front
N7K-C7004
7RU
NX-OS 6.1(2) and later
Rear
Nexus 7004
Front
Back
Side Side
Side Side Side
Back

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 10
Nexus 7700 Chassis Family
Front Rear
26RU
N77-C7718
Nexus 7718
Front Rear
14RU
N77-C7710
Nexus 7710
Front Rear
9RU
N77-C7706
Nexus 7706
NX-OS 6.2(6) and later
NX-OS 6.2(2) and later
NX-OS 6.2(2) and later
Front
Back
Front
Back
Front
Back

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 11
Key Chassis Components
Nexus 7000
Common components:
–Supervisor engines
–I/O modules
–Power supplies (except 7004)
Chassis-specific components:
–Fabric modules
–Fan trays
Nexus 7700
Common components:
–Supervisor engines
–I/O modules
–Power supplies
Chassis-specific components:
–Fabric modules
–Fan trays
Common hardware components between Nexus 7000 and Nexus 7700:
NONE
No interchangeable hardware components between
Nexus 7000 and Nexus 7700

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 12
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 13
Next generation supervisors providing control plane and management functions
Connects to fabric via 1G inband interface
Interfaces with I/O modules via 1G switched EOBC
Second-generation dedicated central arbiter ASIC
–Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
Supervisor Engine 2 / 2E
Console Port
Management
Ethernet
N7K-SUP2/N7K-SUP2E
USB Host
Ports
ID and Status
LEDs
Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000/ Nexus 7700)
Base performance High performance
One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM
USB Log
Flash
USB Expansion
Flash
N77-SUP2E
ID and Status
LEDs
Console PortManagement
Ethernet
USB Expansion
Flash

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 14
Nexus 7000 / 7700 I/O Module Families
M1 1G and 10G
M2 10G / 40G / 100G
F1 10G F2 10G
F2E 10G F3 40G
F2E 10G
F3 10G / 40G / 100G
F3 closes the
F/M feature gap!

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 15
10G / 40G / 100G M2 I/O modules
Share common hardware architecture
Two integrated forwarding engines (120Mpps)
–Support for “XL” forwarding tables (licensed)
Distributed L3 multicast replication
802.1AE LinkSec on all ports
N7K-M224XP-23L
Nexus 7000 M2 I/O Modules
N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
Supported in NX-OS release 6.1(1) and later
N7K-M206FQ-23L
N7K-M202CF-22L
Module PortDensity OpticsBandwidth
M2 10G 24 x 10G (plus Nexus 2000 FEX support)SFP+ 240G
M2 40G 6 x 40G (or up to 24x 10G via breakout)QSFP+ 240G
M2 100G2 x 100G CFP 200G

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 16
Nexus 7000 M2 I/O Module Architecture
N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
LinkSec +
12 X 10G MAC -or-
3 X 40G MAC-or-
1 X 100G MAC
Forwarding
Engine
VOQs
Fabric 2 ASIC
To Fabric Modules
Replication
Engine
Replication
Engine
Front Panel Ports
LC
CPU
EOBC
VOQs
LinkSec +
12 X 10G MAC-or-
3 X 40G MAC-or-
1 X 100G MAC
Forwarding
Engine
VOQs
Replication
Engine
Replication
Engine
VOQs
To Central Arbiters
Arbitration
Aggregator
…

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 18
Nexus 7000 / 7700 F2E I/O Modules
N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E
7000: Supported in NX-OS release 6.1(2) and later
7700: Supported in NX-OS release 6.2(2) and later
N7K-F248XP-25E
N7K-F248XT-25E48-port 1G/10G with SFP/SFP+ transceivers
480G full-duplex fabric connectivity
System-on-chip (SoC) forwarding engine design
–12 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services
(ACL/QoS)
Interoperability with M1/M2, in Layer 2 mode on
Nexus 7000
–Proxy routing for inter-VLAN/L3 traffic
LinkSec support*
–Last 8 ports (SFP+)
–All 48 ports (Copper)
Supports Nexus 2000 (FEX) connections
* Roadmap item
N77-F248XP-23E

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 19
Nexus 7000 F2E Module Architecture
N7K-F248XP-25E / N7K-F248XT-25E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
24
LC
CPU
EOBC
To Central Arbiters
Arbitration
Aggregator…
4 X 10G
SoC
68
4 X 10G
SoC
1012
4 X 10G
SoC
1416
4 X 10G
SoC
1820
4 X 10G
SoC
2224
4 X 10G
SoC
2628
4 X 10G
SoC
3032
4 X 10G
SoC
3436
4 X 10G
SoC
3840
4 X 10G
SoC
4244
4 X 10G
SoC
4648
13 57 91113151719212325272931 3335373941434547
LinkSec-capable (F2E fibre)
LinkSec-capable (F2E copper)

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 20
Nexus 7700 F2E Module Architecture
N77-F248XP-23E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
24
LC
CPU
EOBC
To Central Arbiters
Arbitration
Aggregator…
4 X 10G
SoC
68
4 X 10G
SoC
1012
4 X 10G
SoC
1416
4 X 10G
SoC
1820
4 X 10G
SoC
2224
4 X 10G
SoC
2628
4 X 10G
SoC
3032
4 X 10G
SoC
3436
4 X 10G
SoC
3840
4 X 10G
SoC
4244
4 X 10G
SoC
4648
13 57 91113151719212325272931 3335373941434547
LinkSec-capable
Fabric 2
To Fabric Modules

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 22
Nexus 7000 F3 40G Module
12-port 40G QSFP+ module
480G full-duplex fabric connectivity
SoC forwarding engine design
–6 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4
services (ACL/QoS) and advanced
features
Fabric Services Accelerator (FSA) CPU
Breakout cable support
Requires Supervisor Engine 2 / 2E
N7K-F312FQ-25

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 23
Nexus 7000 12-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 1
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
Fabric ASIC
LC Inband
23 4 5 6 7 8 9101112
…
x 6
to FSA
CPUto ARB
x 6
1G switch
x 6
…

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 24
FSA CPU
Fabric Services Accelerator (FSA)
High-performance module CPU
with on-board acceleration
engines
–6Gbps inband connectivity from
SOCs to FSA
–Multi-Mpps packet processing
–2GB dedicated DRAM
Performance/scale boost for
distributed fabric services,
including BFD and sampled
NetFlow (roadmap)
Other potential applications
include distributed ARP/ping
processing, data plane packet
analysis (wireshark), network
probing, etc.
6 x 1Gbps
Module Inband
I/O
2GB
DRAM
Dual-Core
LC CPU
Acceleration
Engines
2GB
DRAM
EOBC

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 25
Nexus 7700 F3 48-Port 1G/10G Module
48-port 1G/10G with SFP/SFP+
transceivers
480G full-duplex fabric connectivity
SoC-based forwarding engine design
–6 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4
services (ACL/QoS) and advanced
features
Fabric Services Accelerator (FSA) CPU
LinkSec support (last 8 ports)*
Supports Nexus 2000 (FEX)
connections
N77-F348XP-23
* Roadmap item

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 26
8 X 10G
SoC 1
Nexus 7700 F3 48-Port 1G/10G Module Architecture
To Fabric Modules
To Central Arbiters
Arbitration
Aggregator
8 X 10G
SoC 2
8 X 10G
SoC 3
8 X 10G
SoC 4
8 X 10G
SoC 5
8 X 10G
SoC 6
Fabric ASIC Fabric ASIC
…
x 6
1
Front Panel Ports (SFP/SFP+)
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
LinkSec-capable
to FSA
CPUto ARB
FSA
CPU
EOBC
LC Inband
x 6
1G switch
x 6
…

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 27
Nexus 7700 F3 40G and 100G Modules
24-port 40G QSFP+ module /
12-port 100G CPAK module
960G/1.2T full-duplex fabric connectivity
SoC forwarding engine design
–12 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services
(ACL/QoS) and advanced features
Fabric Services Accelerator (FSA) CPU
40G breakout cable support*
N77-F324FQ-25
N77-F312CK-26
* Roadmap item

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 28
2 X 40G
SoC 1
Nexus 7700 F3 24-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
2 X 40G
SoC 7
2 X 40G
SoC 8
2 X 40G
SoC 9
2 X 40G
SoC 10
2 X 40G
SoC 11
2 X 40G
SoC 12
Fabric ASIC Fabric ASIC
LC Inband
23 4 5 6 7 8 9101112131415161718192021222324
1G switch
…
…
x 12
to FSA
CPUto ARB
x 12
x 6

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 29
Nexus 7700 F3 12-Port 100G Module Architecture
Front Panel Ports (CPAK)
To Fabric Modules
To Central Arbiters
Arbitration
Aggregator
1 X 100G
SoC 2
2
1 X 100G
SoC 3
3
1 X 100G
SoC 4
4
1 X 100G
SoC 5
5
1 X 100G
SoC 6
6
1 X 100G
SoC 7
1 X 100G
SoC 8
1 X 100G
SoC 9
1 X 100G
SoC 10
1 X 100G
SoC 11
Fabric ASIC Fabric ASIC
7 8 9 10 11
1 X 100G
SoC 12
12
1 X 100G
SoC 1
1
FSA
CPU
EOBC
LC Inband
1G switch
…
…
x 12
to FSA
CPUto ARB
x 12
x 6

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 30
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 31
M-Series Forwarding Engine Hardware
Two hardware forwarding engines
integrated on every M2 I/O module
120Mpps (60Mpps per forwarding
engine) Layer 2 bridging with
hardware MAC learning
120 Mpps (60Mpps per forwarding
engine) Layer 3 IPv4
60Mpps (30Mpps per forwarding
engine) Layer 3 IPv6 unicast
Layer 3 IPv4 and IPv6 multicast
support (SM, SSM, Bidir)
MPLS/VPLS/EoMPLS
OTV
RACL/VACL/PACL
QoS remarking and policing
policies
Policy-based routing (PBR)
Unicast RPF check and IP source
guard
IGMP snooping
Ingress and egress NetFlow (full
and sampled)
Hardware Table M-Series Modules
without Scale License
M-Series Modules with
Scale License
MAC Address Table 128K 128K
FIB TCAM 128K IPv4 / 64K IPv6900K IPv4 / 350K IPv6
Classification TCAM (ACL/QoS) 64K 128K
NetFlow Table 1M 1M

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 32
From I/O Module
Replication Engines
To I/O Module
Replication Engines
M-Series Forwarding Engine Architecture
L2 Engine
Ingress Parser
MAC
Table L2 Lookup (pre-L3)
L2 Lookup (post-L3)
Final Results
L3 Engine
Classification
(ACL/QoS)
NetFlow
Layer 3 FIB
Policing
FIB TCAM/
ADJ
CL TCAM
FE Daughter Card
Ingress lookup pipeline
Egress lookup
pipeline
Egress NetFlow collection
Ingress MAC table lookups
Port-channel hash result
Ingress IGMP snooping
lookups
FIB TCAM and adjacency table
lookups for Layer 3 forwarding
ECMP hashing
Multicast RPF check
Ingress policing
Egress MAC lookups
Egress IGMP snooping
lookups
PKT
HDR
Egress ACL/QoS classification
Ingress NetFlow collection
Ingress ACL/QoS classification
Egress policing

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 33
F2E Forwarding Engine Hardware
Each SoC forwarding engine services 4
front-panel 10G ports (12 SoCs per module)
60Mpps per SoC Layer 2 bridging with
hardware MAC learning
60Mpps per forwarding engine Layer 3
IPv4/ IPv6 unicast
Layer 3 IPv4 and IPv6 multicast support
(SM, SSM, Bidir*)
RACL/VACL/PACL
QoS remarking and policing policies
Policy-based routing (PBR)
Unicast RPF check and IP source guard
IGMP snooping
FabricPath forwarding
FCoE (with Sup2 / Sup2E)
–Roadmap on Nexus 7700
Ingress sampled NetFlow
Hardware Table Per F2E SoC Per F2E Module
MAC Address Table 16K 192K*
FIB TCAM 32K IPv4/16K IPv632K IPv4/16K IPv6
Classification TCAM (ACL/QoS) 16K 192K*
* Assumes specific configuration to scale SoC resources
* Roadmap item

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 34
F3 Forwarding Engine Hardware
Each SoC forwarding engine services:
–8 front-panel 10G ports
–2 front-panel 40G ports
–1 front-panel 100G port
148Mpps per SoC Layer 2 bridging with
hardware MAC learning
148Mpps per forwarding engine Layer 3
IPv4/ IPv6 unicast
Layer 3 IPv4 and IPv6 multicast support
(SM, SSM, Bidir*)
RACL/VACL/PACL
QoS remarking and policing policies
Policy-based routing (PBR)
Unicast RPF check and IP source guard
IGMP snooping
FabricPath forwarding
Overlay Transport Virtualisation(OTV)
MPLS/VPLS/EoMPLS, LISP, VXLAN,
GRE, FCoE*
Ingress/egress* sampled NetFlow
Hardware Table Per F3 SoC Per F3 Module
MAC Address Table 64K 384K/768K**
FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6
Classification TCAM (ACL/QoS) 16K 96K/192K**
** Assumes specific configuration to scale SoC resources
* Roadmap items

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 35
F3 Forwarding Engine
Decision Engine
Layer 3 Lookups
QoS / ACL
Ingress Parser
MAC
Table
FIB/ADJ
CL
L2 Lookup (post-L3)
Front-panel
To/From Central
Arbiter To Fabric From Fabric
Ingress
Buffer (VOQ)
Virtual output
queues
L2 Lookup (pre-L3)
Egress Parser
F3 SoC
Ingress and egress
forwarding decisions
(L2/L3 lookups,
ACL/QoS, features etc.)
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
Forwarding
tables
1G / 10G / 40G / 100G
1G / 10G / 40G / 100G
capable interface MAC
Egress
Buffer
Egress fabric
receive buffer
HDR
PKT HDR
PKT
PKT HDR

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 36
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 37
Crossbar Switch Fabric Modules
Provide interconnection of I/O modules
Each installed fabric increases available per-payload slot bandwidth
Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC
Different I/O modules leverage different amount of available fabric bandwidth
Access to fabric bandwidth controlled using QoS-aware central arbitration with
VOQ
N7K-C7018-FAB-2
N7K-C7010-FAB-2
N7K-C7009-FAB-2
Fabric Module Supported Chassis
Per-fabric module
bandwidth
Max fabric
modules
Total bandwidth per
slot
Nexus 7000Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot
Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot
N77-C7718-FAB-2
N77-C7710-FAB-2
N77-C7706-FAB-2

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 38
110G
(2 x 55G)
Ingress Module Egress Module
Multistage Crossbar
Nexus 7000/ Nexus 7700 implement 3-stage crossbar switch fabric
Stages 1 and 3 on I/O modules
Stage 2 on fabric modules
1st stage Egress
Module
2nd stage
Ingress
Module
3rd stage
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASICFabric ASIC Fabric ASIC
Fabric Modules
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
1
Fabric
ASIC
2 3 4 5
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
6
Fabric
ASIC
1.32T
1st stage
3rd stage
550G
110G
(2 x 55G)
1Fabric
ASIC
2 3 4 5Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric Modules
Nexus 7000 Nexus 7700

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 39
110Gbps220Gbps330Gbps440Gbps550Gbps
Local Fabric 2
(480G)
Local Fabric 2
(240G)
I/O Module Capacity –Nexus 7000
One fabric:
Any port can pass traffic to any
other port in VDC
Three fabrics:
240G M2 module has maximum
bandwidth
Five fabrics:
480G F2E/F3module has maximum
bandwidth
Fabric 2Modules
1
Fabric 2
ASIC
2
Fabric 2
ASIC
3
Fabric 2
ASIC
4
Fabric 2
ASIC
5
Fabric 2
ASIC
per slot bandwidth

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 40
What About Nexus 7004?
Nexus 7004 has no fabric modules
I/O modules have local fabric with 10 available fabric channels
–I/O modules connect “back-to-back” via 8 fabric channels
–Two fabric channels “borrowed” to connect supervisor engines
Sup Slot 2Sup Slot 1
M2/F2E/F3
Module 4
M2/F2E/F3
Module 3
Fabric
ASIC
Fabric 2
ASIC
Fabric 2
ASIC
Fabric
ASIC
2 * 55G
fabric channels
8 * 55G local fabric channels
interconnect I/O modules (440G)

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 41
220Gbps440Gbps660Gbps880Gbps1100Gbps1320Gbps
Local Fab2
#1 (480G)
Local Fab2
#1 (960G)
Local Fab2
#1 (1.2T)
Fab2
#2
Fab2
#2
Fab2
#2
I/O Module Capacity –Nexus 7700
One fabric:
Any port can pass traffic to any other
port in VDC
Three fabrics:
480G F2E/F3 10G module has maximum
bandwidth
Five fabrics:
960G F340G module has maximum
bandwidth
Six fabrics:
1.2T F3 100G module has maximum
bandwidth
per slot bandwidth
Fabric 2Modules
1
Fabric 2
ASICs
2
Fabric 2
ASICs
3
Fabric 2
ASICs
4
Fabric 2
ASICs
5
Fabric 2
ASICs
6
Fabric 2
ASICs

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 42
Fabric, VOQ, and Arbitration
Crossbar fabric–Provides dedicated, high-bandwidth interconnects between
ingress and egress I/O modules
Virtual Output Queues (VOQs)–Provide bufferingand queuingfor ingress-
buffered switch architecture
Central arbitration–Controls scheduling of traffic into fabric based on
fairness, priority, and bandwidth availability at egress ports
Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for
packet transport inside switch

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 43
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 44
Buffering, Queuing, and Scheduling
Buffering–storing packets in memory
–Needed to absorb bursts, manage congestion
Queuing–buffering packets according to traffic class
–Provides dedicated buffer for packets of different priority
Scheduling–controlling the order of transmission of buffered packets
–Ensures preferential treatment for packets of higher priority and fair treatment for
packets of equal priority
Nexus 7000 / Nexus 7700 use queuing policiesand network-QoS policiesto
define buffering, queuing, and scheduling behaviour
Defaultqueuing and network-QoS policies always in effect in absence of any
user configuration

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 45
I/O Module Buffering Models
Buffering model varies by I/O module family
–M-series modules: hybrid model combining ingress VOQ-buffered
architecture with egress port-buffered architecture
–F-series modules: pure ingress VOQ-buffered architecture

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 46Egress ModuleIngress Module
VOQ 0
Port ASIC 0
M2 –Hybrid Ingress/Egress Buffered
DWRR
VOQ 1RE 1
RE 0
…
DWRR
Port 1
Port 12
FABRIC
VOQ 0
VOQ 1 RE 1
RE 0
Port ASIC 0
Port 1
Port 12
Ingress port buffer –Manages congestion of
ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
INGRESS QUEUING POLICIES
10G module used as example
Diagram represents half
of each I/O module
8 ingress
queues

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 47Egress ModuleIngress Module
VOQ 0
1
2
3
4
5
6
SP234
VOQ
Buffer
Source
Priority
Port ASIC 0
M2 –Hybrid Ingress/Egress Buffered
DWRR
VOQ 1RE 1
RE 0
…
DWRR
Port 1
Port 12
FABRIC
VOQ 0
VQI 1DWRR
SP
…
VQI 6DWRR
SP
VOQ 1
Sources 7-12 VQIs 7-12
RE 1
RE 0
Port ASIC 0
Port 1
Port 12
Ingress port buffer –Manages congestion of
ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
Ingress VOQ buffer –Manages
congestion toward egress
destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives
frames from fabric
Scheduling
FABRIC-QOS POLICY
10G module used as example
Diagram represents half
of each I/O module
Shared buffer
carved by source
and priority
4 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 48Egress ModuleIngress Module
VOQ 0
1
2
3
4
5
6
SP234
VOQ
Buffer
Source
Priority
Port ASIC 0
DWRR
VOQ 1RE 1
RE 0
…
DWRR
Port 1
Port 12
FABRIC
VOQ 0
VQI 1DWRR
SP
…
VQI 6DWRR
SP
VOQ 1
Sources 7-12 VQIs 7-12
RE 1
RE 0
Port ASIC 0
Port 1
Port 12
DWRR
SP
…
DWRR
SP
Ingress port buffer –Manages congestion of
ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
Ingress VOQ buffer –Manages
congestion toward egress
destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives
frames from fabric
Scheduling
Egress port buffer –
Manages congestion at egress
physical interface
Buffering / queuing / scheduling
EGRESS QUEUING POLICIES
10G module used as example
Diagram represents half
of each I/O module
8 egress
queues
M2 –Hybrid Ingress/Egress Buffered

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 49
Egress SOCIngress SOC
Ingress VOQ
F2E –Ingress Buffered (Nexus 7000)
FABRIC
Egress VOQ
VQI 1DWRR
PQ
…
VQI 4DWRR
PQ
Ingress VOQ buffer –Manages congestion toward
egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives frames from
fabric
Scheduling
1
2
3
4
hi
VOQ
Buffer
lo
hi
lo
hi
lo
hi
lo
10G Port 1
10G Port 2
10G Port 3
10G Port 4
10G Port 1
10G Port 2
10G Port 3
10G Port 4
Diagram represents
one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
2 or 4 ingress
queues per port
4 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 50
Egress SOCIngress SOC
Ingress VOQ
F3 10G –Ingress Buffered (Nexus 7700)
FABRIC
Egress VOQ
VQI 1
…
VQI 8
Ingress VOQ buffer –Manages congestion toward
egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives frames from
fabric
Scheduling
2
4
6
8
VOQ
Buffer
hi
lo
1
3
5
7
10G Port 1
10G Port 3
10G Port 5
10G Port 2
Diagram represents
one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
10G Port 4
10G Port 6
10G Port 7
10G Port 8
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
10G Port 1
10G Port 2
10G Port 3
10G Port 4
10G Port 5
10G Port 6
10G Port 7
10G Port 8
DWRR
PQ
DWRR
PQ
2 or 4 ingress
queues per port
8 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 51
Egress SOCIngress SOC
Ingress VOQ
F3 40G –Ingress Buffered (Nexus 7000)
FABRIC
Egress VOQ
VQI 1DWRR
PQ
VQI 2DWRR
PQ
Ingress VOQ buffer –Manages congestion toward
egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives frames from
fabric
Scheduling
1
2
VOQ
Buffer
lo
hi
lo
hi
40G Port 1
40G Port 2
40G Port 1
Diagram represents
one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
40G Port 2
2 or 4 ingress
queues per port
4 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 52
Egress SOCIngress SOC
Ingress VOQ
F3 40G –Ingress Buffered (Nexus 7700)
FABRIC
Egress VOQ
VQI 1
VQI 2
Ingress VOQ buffer –Manages congestion toward
egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives frames from
fabric
Scheduling
40G Port 1
Diagram represents
one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
40G Port 2
DWRR
PQ
DWRR
PQ
1
2
VOQ
Buffer
lo
hi
lo
hi
40G Port 1
40G Port 2
2 or 4 ingress
queues per port
8 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 53
Egress SOCIngress SOC
Ingress VOQ
F3 100G –Ingress Buffered (Nexus 7700)
FABRIC
Egress VOQ
VQI 1
Ingress VOQ buffer –Manages congestion toward
egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer –Receives frames from
fabric
Scheduling
1
VOQ
Buffer
hi
lo
100G Port 1
Diagram represents
one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
DWRR
PQ
100G Port 1
2 or 4 ingress
queues per port
8 priority
levels

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 54
FAQ: What Is a VQI?
VQI = Virtual Queuing Index
“A Destination Across the Fabric”
For M2 / F2E / F3 10G modules, VQI == 10G interface
For M2 40/100G ports, uses multiple 10G VQIs
For F3 40/100G ports, uses single 40/100G VQI

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 55
40G Port
Ingress Modules
10G10G 40G 40G 100G
Spines
Spines
Spines
Spines
Fabrics
M2 Module 40G and 100G Flow Limits
Each Virtual Queuing Index (VQI) sustains 10G
traffic flow
All packets in given 5-tuple flow hash to single VQI
Single-flow limit is 10G
Packets split into 66-bit “code words”
Four code words transmitted in parallel, one on each
physical Txfibre
No per-flow limit imposed –splitting occurs at physical layer
Egress Interfaces
Destination
VQIs
1 VQI1 VQI 4 VQIs4 VQIs 10 VQIs
Internal to Nexus 7000 System
n…4321
64 bits
1 packet
On the Wire (40G)
Tx 1
Tx 2
Tx 3
Tx 4
66 bits
15
2
3
4
6
…
64/66B Encoding

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 56
Ingress Modules
10G10G 40G 40G 100G
Spines
Spines
Spines
Spines
Fabrics
F3 Module 40G and 100G Flow Limits
Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow
based on destination interface type
No single-flow limit –full 40G/100G flow support
Egress Interfaces
Destination
VQIs
1 VQI1 VQI 1 VQI 1 VQI 1 VQI
Internal to Nexus 7000 / 7700 System

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 57
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 58
Hardware Layer 2 Forwarding Process
Layer 2 forwarding –traffic steering based on destination MAC address
MAC table lookup drives Layer 2 forwarding
Source MAC and destination MAC lookups performed for each frame, based
on {VLAN,MAC} pairs
Source MAC lookup drives new learns and refreshes aging timers
Destination MAC lookup dictates outgoing switchport

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 59
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L2 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
ACL/QoS/
NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result –
destination +
hash result
Credit grant for
fabric access
Egress
port QoS
LinkSec
encryption
Static or hash-
based RE uplink
selection
Hash-based uplink
and VQI selection
Round-robin
transmit to VQI
Static
downlink
selection
L2 SMAC/ DMAC
lookups
Port-channel hash
result
HDR= Packet HeadersDATA= Packet Data = Internal SignallingCTRL

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 60
SoC
VOQ
SoC
DE
F2E / F3 L2 Packet Flow
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
VOQ arbitration
Credit grant for
fabric access
Receive from fabric
Return
credit
to pool
Transmit
packet on
wire
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
Transmit
to fabric
VOQ
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Ingress L2 SMAC/ DMAC
lookups, ACL/QoS lookups,
NetFlow sampling
Return result –
destination
Submit packet headers for lookup
Egress port QoS
(Scheduling)
Return buffer credit
HDR= Packet HeadersDATA= Packet Data = Internal SignallingCTRL

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 61
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 62
Layer 3 Forwarding
Nexus 7000 decouples control plane and data plane
Forwarding tables built on control plane using routing protocols or static
configuration
–OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
Tables downloaded to forwarding engine hardware for data plane forwarding
–FIB TCAM contains IP prefixes
–Adjacency table contains next-hop information

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 63
Hardware Layer 3 Forwarding Process
FIB TCAM lookup based on longest-match destination prefix comparison
FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)
Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow
lookups, affecting final forwarding result

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 64
10.1.1.2
10.1.1.3
10.10.0.10
10.10.0.100
10.10.0.33
10.1.1.4
10.1.2.xx
10.1.3.xx
10.1.1.xx
10.100.1.xx
10.10.0.xx
10.100.1.xx
10.10.100.xx
IP FIB TCAM Lookup
FIB TCAM
Generate
Lookup Key
10.1.1.10
Generate TCAM lookup key
(destination IP address)
Forwarding Engine
FIB DRAM
Load-Sharing
Hash
Adjacency Table
Next-hop 4 (IF, MAC)
Next-hop 6 (IF, MAC)
Next-hop 7 (IF, MAC)
Next-hop 5 (IF, MAC)
Next-hop 3 (IF, MAC)
Next-hop 1 (IF, MAC)
Next-hop 2 (IF, MAC)
10.1.1.xx
Ingress
unicast IP
packet header
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Hit in FIB
returns result
in FIB DRAM
Adjacency
index identifies
ADJ block to
use
Modulo function
selects exact
next hop entry
to use
Offset
Compare
lookup key
Return lookup
result
# next-
hops
Flow
Data
Result
HIT!
Adj Index
mod

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 65
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L3 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
L3 FIB/ADJ lookup
Ingress and egress
ACL/QoS/NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result –
destination +
hash result
Credit grant for
fabric access
Egress
port QoS
LinkSec
encryption
Static or Hash-based
uplink selection
Hash-based uplink
(and VQI) selection
Round-robin
transmit to VOQ
Static RE
downlink
selection
L2 ingress and egress
SMAC/ DMAC lookups
Port-channel hash result
HDR= Packet HeadersDATA= Packet Data = Internal SignallingCTRL

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 66
SoC
VOQ
SoC
DE
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
VOQ
F2E / F3 L3 Packet Flow
HDR= Packet HeadersDATA= Packet Data = Internal SignallingCTRL
VOQ arbitration
Credit grant for
fabric access
Return
credit
to pool
Transmit
packet on
wire
Transmit
to fabric
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Return result –
destination
Submit packet headers for lookup
L2 ingress and egress SMAC/
DMAC lookups
L3 FIB/ADJ lookup
Ingress and egress ACL/QoS
lookups, NetFlow sampling
Receive from fabric
Egress port QoS
(Scheduling)
Return buffer credit

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 67
Layer 3 Forwarding –Module Interoperability Models
Two interoperability models for L3 forwarding:
“Proxy Forwarding”
“Ingress Forwarding” with Lowest Common Denominator

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 68
From F1/F2Eperspective, Router MAC reachable through giant port-channel
All packets destined to Router MAC forwarded through fabric toward one
“member port” in that channel
Proxy Forwarding Model –Conceptual
All F1/F2E modules
All M1/M2 modules
Up to 128 “links”10.1.10.100
vlan 10
10.1.20.100
vlan 20
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 69
Proxy ForwardingModel–Actual
10.1.10.100
vlan 10
e1/1
Fabric
F1/F2E
SoC
FE
e2/1
Fabric
F1/F2E
SoC
FE
10.1.20.100
vlan 20
Replication
Engine
e3/1
e3/2
M1/M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e3/3
e3/4
e3/5
e3/6
e3/7
e3/8
Replication
Engine
e4/1
e4/2
M1/M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e4/3
e4/4
e4/5
e4/6
e4/7
e4/8
Fabric
Fabric
Modules
Fabric
…
VLAN DMAC Dest Port
10 router_mac →internal_channel (e3/1 -8,e4/1-8)
EtherChannel Hash Function
hash_input (from packet) →select_member_port
Ingress MAC:
VLAN DMAC Dest Port
10 router_mac →L3_lookup
Routing:
DIP Next Hop
10.1.20.100 →server_2_mac (v20)
Egress MAC:
VLAN DMAC Dest Port
20 server_2_mac →e2/1
1
2
3
4
6
5
7
8
9
10
Programming of all
M1/M2 forwarding engines
Programming of all
F1/F2E forwarding engines
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24
Can be up to 128
M1/M2 VQIs
Mod 1
Mod 2
Mod 4
Mod 3

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 71
Ingress Forwarding with Lowest Common
Denominator Model
F3 module interoperability always Ingress Forwarding –NOproxy forwarding
with F3
–Essentially equivalent to current M1 + M2 interoperability model
–The ingressmodule makes all the forwarding decisions
Supported feature set based on Lowest Common Denominator
–Feature available if all modules support the feature
VDC Type Layer 2 Layer 3 vPC
Fabric
Path
VXLAN FEX MPLS OTV LISP FCoE Table Sizes
F3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F3 size
M2+ F3 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ F3 size
F2/F2E + F3 ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ F2E size
Not all features
supported by
software today

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 73
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 74
What Is Classification?
Matching packets
–Layer 2, Layer 3, and/or Layer 4 information
Used to decide whether to apply a particular policy to a packet
–Enforce security, QoS, or other policies
Some examples:
–Match TCP/UDP source/destination port numbers to enforce security policy
–Match destination IP addresses to apply policy-based routing (PBR)
–Match 5-tuple to apply marking policy
–Match protocol-type to apply Control Plane Policing (CoPP)
–etc.

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 75
CL TCAM Lookup –ACL
ip access-list example
permit ip any host 10.1.2.100
deny ip any host 10.1.68.44
deny ip any host 10.33.2.25
permit tcp any any eq 22
deny tcp any any eq 23
deny udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
xxxxxxx|10.1.2.100 |xx |xxx |xxx
xxxxxxx |10.1.68.44 |xx |xxx |xxx
xxxxxxx |10.33.2.25 |xx |xxx |xxx
xxxxxxx |xxxxxxx|tcp |xxx|22
xxxxxxx |xxxxxxx| tcp | xxx| 23
xxxxxxx |xxxxxxx|tcp |xxx|80
xxxxxxx | xxxxxxx| udp | xxx |161
xxxxxxx |xxxxxxx|udp |xxx|514
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
CL TCAM
Generate
Lookup Key
Generate TCAM
lookup key
CL SRAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx| 10.2.2.2 | xx| xxx | xxx
xxxxxxx |xxxxxxx |tcp|xxx |80
SIP | DIP | Pr | SP | DP
Compare lookup
key to CL TCAM
entries
Comparisons
(X = “Mask”)
Hit in CL TCAM
returns result in
CL SRAM
Security ACL
Forwarding Engine
Result
Return
lookup
result
Result affects
final packet
handling
Permit
Permit
Permit
Permit
Deny
Deny
Deny
Deny
HIT!
Results
SIP | DIP | Pr | SP | DP

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 76
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
Result affects
final packet
handling
Generate
Lookup Key
Forwarding Engine
xxxxxxx|10.3.3.xx|xx |xxx |xxx
xxxxxxx | 10.4.12.xx|xx |xxx |xxx
10.1.1.xx| xxxxxxx |udp|xxx |xxx
10.1.1.xx |xxxxxxx|tcp |xxx|xxx
xxxxxxx |10.5.5.xx| tcp | xxx| 23
CL TCAM Lookup –QoS
ip access-list police
permit ip any 10.3.3.0/24
permit ip any 10.4.12.0/24
ip access-list remark-dscp-32
permit udp 10.1.1.0/24 any
ip access-list remark-dscp-40
permit tcp 10.1.1.0/24 any
ip access-list remark-prec-3
permit tcp any 10.5.5.0/24 eq 23
CL TCAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx| 10.2.2.xx| xx| xxx | xxx
10.1.1.xx| xxxxxxx| tcp | xxx| xxx
HIT!
CL SRAM
QoS Classification ACLs
Generate
TCAM lookup
key
SIP | DIP | Pr | SP | DP
Compare
lookup
key
Hit in CL TCAM
returns result in
CL SRAM
Result
Return
lookup
result
Policer ID 1
Policer ID 1
Remark DSCP 32
Remark DSCP 40
Remark IP Prec 3
SIP | DIP | Pr | SP | DP
Comparisons
(X = “Mask”)
Results

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 77
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 78
NetFlow
NetFlow collects flow data for packets traversing the switch
Each module maintains independent NetFlow table
M2 F2E / F3
Per-interface NetFlow Yes Yes
NetFlow direction Ingress/Egress Ingress only
Full NetFlow Yes No
Sampled NetFlow Yes Yes
FSA Assist for Sampled NetFlow No F3 only (future)
Bridged NetFlow Yes Yes
Hardware Cache Yes No
Software Cache No Yes
Hardware Cache Size
512K entries per
forwarding engine
N/A
NDE (v5/v9) Yes Yes

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 79
Full vs. Sampled NetFlow
NetFlowcollects full or sampledflow data
Full NetFlow: Accounts for every packet of every flow on interface
–Available on M-Series modules only
–Flow data collection up to capacity of hardware NetFlow table
Sampled NetFlow: Accounts for M in N packets on interface
–Available on both M2 (ingress/egress) and F2E/F3 (ingress only)
–M2: Flow data collection up to capacity of hardwareNetFlow table
–F2E/F3: Flow data collection for up to ~1000pps per module
–F3 (future): Increased per-module sampling rate leveraging on-board Fabric
Services Accelerator (FSA) complex

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 81
NetFlow on M2 Modules
Fabric
ASIC
VOQs
Mgmt
Enet
Supervisor
Engine
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Hardware
Flow Creation
Hardware
Flow Creation
Hardware
Flow Creation
Aged Flow Info
Aged Flow Info
Aged Flow Info
Generate NetFlow v5
or v9 export packets
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 82
Sampled NetFlow on F2E/F3
Modules
F3 Module
FSA
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
F3 Module
Fabric
ASIC
VOQs
Mgmt
Enet
Supervisor
Engine
FSA
CPU
SoC
Decision
Engine
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
DRAM
NetFlow
Cache
Populate cache based
on received samples
Age flows and
generate NetFlow v5
or v9 export packets
F2E Module
LC
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
Data Flow
Data Flow
Data Flow
via Module
Inband
via Module
Inband
via Module
Inband
Sampled
Packets
Sampled
Packets
Sampled
Packets
Aged
Flows
Aged
Flows
Aged
Flows

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 83
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 85
Conclusion
You should now have a thorough understanding of the
Nexus 7000/ Nexus 7700switching architecture, I/O module
design, packet flows, and key forwarding engine functions…
Any questions?
85

Q & A

© 2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 88
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco Live 2014 Polo Shirt!
Complete your Overall Event Survey and 5 Session
Evaluations.
Directly from your mobile device on the Cisco Live
Mobile App
By visiting the Cisco Live Mobile Site
www.ciscoliveaustralia.com/mobile
Visit any Cisco Live Internet Station located
throughout the venue
Polo Shirts can be collected in the World of Solutions
on Friday 21 March 12:00pm -2:00pm
Learn online with Cisco Live!
Visit us online after the conference for full access
to session videos and presentations.
www.CiscoLiveAPAC.com

N7K Hardware Architecture and it's components

About This Presentation

Slide Content

Tags

Categories

Download

Quick Actions

Statistics

Related Slideshows

N7K Hardware Architecture and it&#39;s components

About This Presentation

Slide Content

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

Slide 17

Slide 18

Slide 19

Slide 20

Slide 21

Slide 22

Slide 23

Slide 24

Slide 25

Slide 26

Slide 27

Slide 28

Slide 29

Slide 30

Slide 31

Slide 32

Slide 33

Slide 34

Slide 35

Slide 36

Slide 37

Slide 38

Slide 39

Slide 40

Slide 41

Slide 42

Slide 43

Slide 44

Slide 45

Slide 46

Slide 47

Slide 48

Slide 49

Slide 50

Slide 51

Slide 52

Slide 53

Slide 54

Slide 55

Slide 56

Slide 57

Slide 58

Slide 59

Slide 60

Slide 61

Slide 62

Slide 63

Slide 64

Slide 65

Slide 66

Slide 67

Slide 68

Slide 69

Slide 70

Slide 71

Slide 72

Slide 73

Slide 74

Slide 75

Slide 76

Slide 77

Slide 78

N7K Hardware Architecture and it's components