MPLS Transport network for power utilities

WAN technology & migration from TDM to Carrier Ethernet

1. The POWERful Choice – Carrier Ethernet or MPLS For Power Utilities Yaakov (J) Stein CTO

2. SONET/SDH is being phased out SONET technology is widely deployed, but SONET technology is aging SONET equipment is becoming obsolete and hard to find SONET is hard to maintain (parts hard to obtain and expensive) finding staff with SONET expertise is becoming ever more difficult no new rates/functionality/standards/applications are being developed for SONET Modern packet-based networks (based on Ethernet, MPLS, and IP) are the present and future are broadband and becoming even more so are less expensive (both CAPEX and OPEX) and more flexible are being actively extended (e.g., migration to 61850) But there are open questions can all the relevant services be migrated to packet (e.g., teleprotection, synchrophasors) ? which packet-based network to choose ?

3. The options Carrier Ethernet Based on most popular technology in the world Look and feel similar to SONET/SDH networks Mature carrier-grade technology Support for synchronization Network security mechanisms available MPLS Core network technology Inherits rich IP control plane Deterministic paths available (MPLS-TE) Has no inherent network security MPLS-TP Based on MPLS, but adds mechanisms patterned after Carrier Ethernet OAM and protection switching (including rings) Look and feel similar to SONET/SDH networks Does not require IP forwarding or control plane Has no inherent network security

4. What is Carrier Ethernet ? (1) Ethernet started out as a LAN technology LAN networks are small and operated by consumer and hence are easily managed When Ethernet left the LAN environment new mechanisms were needed, e.g. scalability (to reach 100s of thousands of end-points) OAM ( F ault M anagement, P erformance M onitoring) deterministic (Connection-Oriented) connections support for various topologies (e.g., point-point, rings, trees) resilience mechanisms (e.g., Automatic Protection Switching) support for synchronization Carrier Ethernet (CE) adds carrier-grade features to Ethernet so that it can replace SONET/SDH as a transport network Metcalf’s original sketch of Ethernet Blue means Ethernet

5. What is Carrier Ethernet ? (2) Mature Technology widely deployed by service providers promoted and maintained by Metro Ethernet Forum (MEF) Deterministic and C onnection O riented (unlike connectionless IP) provisioning through management system (not routing) support for point-point, multipoint-multipoint, ring, tree, … topologies Support for Q uality o f S ervice ( up to 8 C lasses o f S ervice) enforcement of bandwidth profiles (dual token bucket shaping/policing) color (conformance) marking Carrier-grade operations mechanisms : service activation testing (Y.1564) F ault M anagement (802.1ag, Y.1731) P erformance M onitoring (Y.1731) A utomatic P rotection S witching (G.8031, G.8032) Synchronization <timing distribution> (SyncE, 1588) Network security mechanisms : access authorization (802.1X) source authentication, integrity and optional encryption (MACSec)

6. What is MPLS ? (1) MPLS started out as a technology to accelerate IP forwarding by setting up tunnels to transport IP other traffic can be transported via pseudowires MPLS defined by the IETF, and inherits the rich IP protocol suite like all IETF protocols, MPLS does not define layer 2 or below MPLS is a mature technology for core IP networks full T raffic E ngineering is available, but not traffic conditioning (policing/shaping) supports mesh topologies uses local Fast ReRoute (not protection switching) for resilience no network security mechanisms (since core elements are trusted) A new MPLS version ( MPLS-TP ) takes MPLS out of the core network into the transport domain WARNING: there are two non-interoperable versions (from IETF and ITU-T) Red means MPLS

7. What is MPLS ? (2) We can now distinguish four distinct flavors of MPLS: best effort MPLS (usually with LDP, perhaps with RSVP-TE for FRR) not true CO – pinned to route not to Network Elements used in Internet core MPLS for L3VPN services (RFC 4364 <ex-2547> using BGP) used to deliver VPN services to business users traffic engineered MPLS-TE (currently with RSVP-TE) true CO with resource reservation used when strict SLA guarantees must be given (banks, government, …) transport profile - MPLS-TP (with management or RSVP-TE) does not assume the existence of IP forwarding plane does not require the IP control plane (can work with management systems) implements OAM and APS functionality (based on Carrier Ethernet) supports ring topologies still in initial phases of deployment (little interop testing has been performed) does not add network security features (still susceptible to attack)

8. The battlefront Ethernet started in the local network (LAN) and for many years has moved into transport networks MPLS started in the core network (WAN) and is now trying to conquer transport networks with MPLS-TP local network TRANSPORT NETWORK core network ETHERNET MPLS first mile last mile

9. Technical Comparison

10. Features in common Both Ethernet and MPLS (all flavors) : can natively transport IP traffic Ethernet can natively transport other traffic types (EtherType) MPLS can transport other traffic types via pseudowire technology can be transported over SONET/SDH and OTN are being actively developed (by multiple standards organizations) Ethernet by the IEEE, MEF, ITU, … MPLS by the IETF, ITU-T, … may exhibit very high or very low transit delays (and everything in-between) (unlike SONET/SDH which has constant switching latency) very high delay when packets need to wait in a queue very low delay (much lower than SONET/SDH ) for prioritorized traffic Both CE and MPLS-TP : typically use network management systems for configuration define FM/PM OAM and diagnostic tests support rings and define APS

11. 1 st reason for differences – format Ethernet packet headers are self-describing a globally unique source address a globally unique destination address an optional connection identifier (VLAN) optional Class of Service and Drop Eligibility Indicator a payload protocol type identifier (EtherType) MPLS packet headers are only locally meaningful no unique addresses a locally meaningful label (stack) a TTL field (to avoid packet looping) optionally a Traffic Class (TC) field DA (6B) SA (6B) T/L (2B) VT (2B) VLAN (2B) Label (20b) TC (3b) S (1b) TTL (8b)

12. 2 nd reason for differences – control Ethernet was zero-touch in broadcast domain LANs CE uses network management to support large networks Ethernet does define L2 control protocols (STP, LACP, LLDP, …) but does not define a routing protocol (neglecting TRILL, E-VPN, etc.) Best effort MPLS tunnels according to topology found by IP routing protocols So best effort MPLS: does not require sophisticated management system does requires the full logistics of an IP network MPLS-TE requires both IP routing and a sophisticated management system MPLS-TP is the only flavor of MPLS that does not require IP routing but when routing is not used, configuration management is required (basically equivalent to Carrier Ethernet)

13. Additional differences Ethernet defines physical (L1) layers (but may run over MPLS as a PW) MPLS requires a server layer to transport it (which is usually Ethernet) Ethernet can not tolerate forwarding loops Carrier Ethernet supports rings with G.8032 and Industrial Ethernet supports them with H igh-availability S eamless R edundancy MPLS can (since it contains a TTL field) Carrier Ethernet supports bandwidth profiles (bucketing) Ethernet supports IEEE 1588 timing distribution over packet and defines a physical layer to support Synchronous Ethernet MPLS may obtain support for 1588 (work ongoing in IETF) but since MPLS does not a physical layer it can not provide physical layer synchronization support Ethernet has network security mechanisms (MACsec, 802.1X, SNMPv3) MPLS does not define any standardized network security mechanisms and since MPLS has no source address it can not provide source authentication

14. The new trend – SDN Distributed routing protocols are limited to finding simple connectivity minimizing number of hops but can not perform more sophisticated operations optimizing paths under constraints (e.g., delay, security) setting up backup paths integrating networking functionalities (e.g., NAT, firewall) into paths Lately, a new paradigm has arisen – S oftware D erived N etworking, which: removes control protocols from network elements replaces distributed routing with centralized path computation configures the forwarding actions of the switches from a central site SDN sees the IP/MPLS control plane as a disadvantage and adopts the Carrier Ethernet / MPLS-TP approach New SDN tools can optimally manage operational networks SDN services can be added and modified at the speed of software SDN should lead to significant OPEX reductions

15. Why not use both ? (1) We have seen that MPLS is missing several critical features in particular, synchronization and network security So, why not use both Ethernet and MPLS taking the best features of each ? In fact, MPLS does not define its own physical layer and the most common physical layer supporting MPLS is Ethernet although MPLS can be transported over other physical layers, e.g., SDH or OTN So the real question is whether to maintain an Ethernet network or an MPLS network in addition to an Ethernet network ! ETHERNET MPLS

16. Why not use both ? (2) How many networks are there ? Ethernet defines its own physical layer although Ethernet can be transported over other physical layers When transporting IP over Ethernet there are actually 2 or 3 networks 3 IP 2 Ethernet 1 Ethernet or optionally SONET/SDH or OTN MPLS does not define its own physical layer When transporting IP over MPLS there are actually 3 or 4 networks 3 IP 2 .5 MPLS 2 Ethernet 1 Ethernet or optionally SONET/SDH or OTN Do we care how many networks there are ?

17. Why not use both ? (3) Yes, because maintaining networks is never trivial or expense-free! Attempts to design a network to use Ethernet as a dumb pipe under MPLS usually end up using a large number of Ethernet mechanisms For example, when running MPLS over Ethernet, one usually needs : staff trained in Ethernet technologies and staff trained in IP/MPLS technologies to be able to run Ethernet OAM and MPLS diagnostic tools to maintain an Ethernet NMS and MPLS management screens Network management is the core business of a network service provider and for them it may be reasonable to maintain duplicate staff, tools, operations centers, etc. Network maintenance is not the core business of a power utility and the duplication and added complexity is usually not justifiable

18. Operational Comparison

19. Utilities network requirements Traffic types (not an exhaustive list) SCADA operational traffic teleprotection traffic synchrophasor traffic surveillance video general TCP/IP and there is a growing demand for bandwidth Determinism (CO behavior) best effort / nondeterministic (Internet-like) behavior is not acceptable Resilience (critical infrastructures must be highly reliable) Low (and constant) end-end delay (for SCADA and teleprotection applications) Management networks presently employ centralized management end-to-end provisioning and maintenance are must s Synchronization Network security (merits discussion in a separate section) cyber security is a growing concern regulatory requirements are appearing

20. Traffic types SONET/SDH was designed to transport certain traffic types and rates mapping new traffic types is difficult and complex transport of most traffic rates is inefficient no higher rates are being defined for SONET/SDH Ethernet was designed to transport arbitrary traffic types and rates EtherType mechanism to indicate payload types pseudowire technology may also be used no rate constraints higher rates being defined (presently 100Gbps) MPLS was designed to transport IP traffic pseudowire technology enables transport of arbitrary traffic types MPLS imposes no rate constraints or limitations So, regarding traffic, SONET/SDH is reaching End-of-Life while Ethernet and MPLS are future proof!

21. Determinism Networks are deterministic when traffic consistently flows through the network in the same way With nondeterministic networks (e.g., IP and best effort MPLS ) each packet may take a different route through the network, thus enabling intermittent faults (only when the packets happen to go there) complicating troubleshooting (where did the packets go?) excluding the reservation of resources or specific processing at particular network elements (you can’t be sure the packets will go where you want …) SONET/SDH networks are C ircuit S witched, and thus completely deterministic CE and some types of MPLS (TE, TP) are C onnection O riented and thus relatively deterministic traffic consistently takes the same path through the network but does not always take precisely the same time to traverse So, due to lack of determinism, best effort MPLS is not a reasonable candidate for a power utility operational network

22. Resilience SONET/SDH is well-known for its Automatic Protection Switching gold standard 1:1 APS supports < 50 millisecond protection switching time 1+1 APS can provide hitless switching (at the cost of increased bandwidth) Best effort MPLS relies on slow rerouting for recovery MPLS with F ast R e R oute performs local detours around failures at the expense of loss of determinism CE and MPLS-TP support several types of APS CE’s G.8031 and G.8032 and MPLS-TP’s RFC 6378, 6974, ITU-T G.8131 1+1 pseudowire redundancy achieves hitless switching at the cost of increased bandwidth consumption So, from the point of view of resilience CE and MPLS-TP are as good as SONET/SDH !

23. End-end delay and delay consistency Some operational traffic require low and consistent delay For example, teleprotection’s end-end delay budget may be 6 milliseconds SONET/SDH latency is typically sufficiently low (e.g., under 2 msec.) is constant is independent of SONET/SDH rate (whether OC3 or OC192) Carrier Ethernet and MPLS may have much lower transit latencies prioritorized packets only wait for the packet already exiting the switch for the worst case (1500B packet that just started) this latency is: 1 2 3 m sec at 100 Mbps (about the same as a SONET/SDH frame) 12.3 m sec at 1 Gbps 1.23 m sec at 10 Gbps TDM pseudowire traffic requires a jitter buffer eliminates delay variation adds additional latency (under 1 msec for prioritorized, low PDV, traffic) So, delay considerations actually favor CE and MPLS over SONET/SDH !

24. What about delay asymmetry ? For some bi-directional applications the delay must be symmetric (the same in both directions) SONET/SDH ADM rings have constant delay asymmetry (without “spatial reuse” management) teleprotection mechanisms compensate for this CE and MPLS CE is always co-routed and thus symmetric best effort MPLS may not be co-routed but MPLS-TE and MPLS-TP can be TDM pseudowire may introduce buffer asymmetry correct implementation keeps this very low So, delay asymmetry considerations actually favor CE and MPLS-TP over SONET/SDH ! SONET/SDH– Delay asymmetry CE or MPLS Symmetric delay

25. Management SONET/SDH networks typically are typically supported by sophisticated management platforms ( O peration S upport S ystems, N etwork M anagement S ystems) developed by vendors or users over decades Carrier Ethernet was developed to replace SONET/SDH in service provider networks and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel MPLS-TP was developed to be functionally equivalent to previously developed CE and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel So, from the point of view of management SONET/SDH , CE and MPLS-TP are exceptionally similar while best-effort MPLS is completely different

26. Synchronization Synchronization (AKA timing) the ability to transfer highly accurate frequency or time over a network (obviating reliance on GPS) While timing may not be a requirement in present-day utilities networks it is crucial to support some imminent applications such as new teleprotection mechanisms and synchrophasors SONET/SDH has native support for frequency transfer as it requires highly accurate frequency for its own operation but does not support time transfer Ethernet fully supports both time and frequency transfer by use of Synchronous Ethernet (ITU-T G.8261/2/4) for physical layer support and support for IEEE 1588 Precision Time Protocol for packet layer distribution MPLS does not currently support timing at all work in IETF-TICTOC is progressing to provide some support for IEEE 1588 having no physical layer, MPLS will never support physical layer frequency distribution So, regarding synchronization CE is the best alternative followed by SONET/SDH ( and MPLS has no support )

27. Summary (so far) So far we have compared CE , MPLS , and MPLS-TP to SONET/SDH , and found Traffic types and growing demand for bandwidth Determinism SONET/SDH , CE and MPLS-TP are all acceptable best effort MPLS is unacceptable for critical operational networks Resilience CE and MPLS-TP (but not non-TP MPLS ) are as good as SONET/SDH Delay (including consistency and asymmetry) favors CE and MPLS (for asymmetry only MPLS-TP ) over SONET/SDH Management CE and MPLS-TP (but not non-TP MPLS ) are equivalent to SONET/SDH Synchronization CE has full support, SONET/SDH supports frequency, MPLS is deficient In the final section we will discuss Network Security and discover further differences between Carrier Ethernet and MPLS

28. Network Security for Power Utilities

29. Security highlights MPLS was invented for core networks where network elements are in secure locations, and therefore trusted and was thus designed without any security mechanisms In particular, the MPLS forwarding plane can not be source authenticated (no source address!) has no standardized integrity mechanism and t he MPLS control plane uses soft-state protocols Ethernet was designed for untrusted network elements CE does not suffer from most of these ailments since Ethernet ports can be: Authorized (by 802.1X) and Ethernet packets can be Source authenticated (by MACsec) Integrity (and replay) tested (by MACsec) and CE uses a security-enabled management plane (instead of a control plane) Let’s see why this is important !

30. MPLS data plane DoS (injection ) attack Once a packet is inside an MPLS network it can not be blocked (no authentication) If an attacker gains physical access to an MPLS network node (e.g., by using a free port) he/she can inject fake MPLS packets (guessing until a valid label is found) At high rates this injection can overwhelm forwarding resources MPLS Core Substation RTU LAN TPR Central Site DMS/EMS Data Center Connect to any free MPLS port PE CE can block this attack using 802.1X authorization LSP LSP PE PE PE Data Plane

31. Central Site DMS/EMS Data Center MPLS man in the middle attack Tampering means falsifying SCADA RTU/IED <-> control station data Can be implemented by owning the switch or by inserting an evil SFP into a port MPLS has no integrity mechanisms to detect tampering Result can be power disruption and/or physical damage to equipment MPLS Core Substation RTU LAN TPR LSP LSP PE Data Plane CE can block this attack using MACSec’s integrity check

32. Central Site DMS/EMS Data Center MPLS LSP swap attack The attacker exchanges the internal labels belonging to 2 substations Implemented by owning the switch or via an Evil SFP MPLS has no source authentication mechanisms The Central Site control systems now believe that indications from substation A belong to substation B (and vice versa ) MPLS Core Substation A RTU LAN TPR LSP PE Data Plane CE can block this attack using MACSec’s source authentication Data A Substation B RTU LAN TPR PE Data B Data A Data B Data A Data B

33. MPLS control plane attack Not relevant for MPLS-TP w/o control plane MPLS control protocols (e.g., LDP and RSVP-TE) are soft-state (when contact with a peer is lost, LSPs are withdrawn) Intermittently deleting consecutive few heartbeat packets causes massive denial of service A more complex attack can poison the Label Information Base Substation RTU LAN TPR Central Site DMS/EMS Data Center MPLS Core LDP or RSVP-TE LSP PE Control Plane Attack is not applicable to CE which doesn’t use a Control Plane

34. Summary (final this time) In our previous summary we saw that Carrier Ethernet and MPLS-TP (but not MPLS ) were as good as, or even better than, SONET/SDH on most accounts and had the further advantage of being future proof Best effort MPLS is nondeterministic and should not be considered for operational networks Concerning synchronization (crucial for up-and-coming applications) Carrier Ethernet has full support while MPLS has none (thus diminishing its status as being future proof ) Now we have seen that Regarding Network Security MPLS is highly vulnerable while Carrier Ethernet possesses mechanisms to fight off attacks These facts should be taken into account when planning future transport networks

35. Yaakov (J) Stein CTO [email protected]