Workshop on
Synchronization and
Timing Systems

June 18-21, 2018
San Jose, CA
DoubleTree San Jose

The ATIS Interview

Network Synchronization: A Changing Paradigm on the Way to 5G

Richard Linzy, Senior Data/Facility Architecture Engineer, Windstream
Interviewed by Pat Diamond, Principal, Diamond Consulting

Pat Diamond: Let’s start by discussing the topic of Ethernet-based sync services. We hear a lot lately about IEEE 1588-2008, also known as PTP Version 2, as well as SyncE. How do you see end-customers using these services to help with their IP networks?

Richard Linzy: Next-generation networks will have quite a transformative impact on both the end users as well as the carrier-to-carrier network services.  This is because bandwidth is king. We started out many years ago with the belief and the mindset that Ethernet did not require synchronization.  That was true when we were running 100 MG, 1 G, and even into the 10 G realm.  As an industry, however, when we began to look at larger pipes, such as 100 GB links and 500 GB links, along with the manufacturing industry, we recognized that the requirement for precision phase and time of day (TOD) services had to be provided to the layer 3 routers. This is because as we increase the speed of these bandwidths, we have to be able to provide a precision TOD time mark and a precision phase for quality of service QoS-type services so that the cache buffering on those machines is properly controlled and managed on the control plane and the throughput of data  is expedited through. This thereby reduces overall latency in the carrier network.

Diamond: Aren’t there services whereby a transmitting element will send to another element a timestamp that says when to send?

Linzy: Exactly. This is where we actually begin to build intelligent routing algorithms that we can actually take and put a QoS stamp level plus a TOD level stamp that says “Okay handle these packets first by QoS stamp.”  Then the TOD stamp actually prioritizes that traffic in and out through that router for prioritization of traffic throughput.

Diamond: So that says that all of the network devices that participate in this process must have the same value of time within a certain level of accuracy of alignment, machine to machine to machine?

Linzy: Exactly. With the 10 GB services and the 100 GB services it was adequate to run 100 microseconds in accuracy from unit to unit, router to router. Moving forward, more pure optical 0TU2 0TU4 network architecture is being deployed in order to minimize latency across networks. With those types of optical networks we were improving our latency.  However, now we’re also able to add new capacity to the networks. We’re putting 200 GB or 500 GB through a pipe. Suddenly, if you look at the MEF and IEEE work taking place, they’re looking at this in terms of 1.5 microseconds peer-to-peer accuracy. The only way to accomplish that today is with the PTP V2 standards for phase and TOD.  NTP no longer meets that requirement. However, a great deal of SONET or SDH-type traffic is still being routed across today’s IP networks using pseudowire. So frequency synchronization is still a very big proponent of running these legacy TDM services across the Ethernet pipes. That’s where SyncE comes in to play.

Diamond: So SyncE will give you traceability according to the ITU G.8262 grouping of recommendations of frequency accuracy equivalent to Stratum 2?

Linzy: That is correct.

Diamond: It’s interesting to think today that 15 years ago packet over SONET were the rage. There were large pipes where you’d shove an Ethernet frame into an OC48. Now with pseudowire and precision timing at the physical layer you can take SONET and SDH frames and put them into Ethernet packets.

Linzy: That is correct.

Diamond: An interesting change in paradigms.

Linzy: The industry changed from the old model.  In the carrier world, a gentleman named Mr. Hertzog who was the CEO of ITT in 1984, once noted:  In the telecom world and the carrier world, our economic rule is to be able to squeeze 600 times the value of the equipment and 1,500 times the earnings off of every invested dollar. We had to wait until the SONET transport equipment had outlived itself from a value standpoint — not a technology standpoint but a value standpoint. Once it had paid for itself in terms of ROI, the networks began to move more toward Ethernet-based architect networks. I believe that in another 20 to 30 perhaps years we won’t see SDH or SONET anywhere in the network. There will be a pure Ethernet or optical play network.  This will take time. Today, we’re starting to see a transition from the more TDM-based networks to a more Ethernet-centric network. Now we’re reversing the model and throwing those services across Ethernet packets.  That’s a challenge in itself because we have to be able to guarantee synchronization at a frequency level still with the TDM-type services.

Diamond: With 5G looming on the horizon, how will this change the backhaul approach operators have implemented to date?

Linzy: Industry is still discussing that question.  We have probably 80 percent of the facts we need now on the topic. Many of the 5G standards are still being developed by standard bodies and driven by organizations such as ATIS, and are being rolled out. In the past, on 4G LTE and 3G radio networks for cellular services, we were shooting for a bandwidth allocation to each handheld of about 2 MB up to 6 MB of service. With 5G they’re actually looking at being able to push up to 1 GB to every handheld in your pocket.  With this additional traffic ability, the delivered transport layer to the towers — which used to be based on 100 MB pipes or 1 GB pipes at the largest — is now being looked at as 10 GB pipes or 100 GB pipes. That is the case in large tier 1 markets. With 5G radio going to the larger Ethernet pipes, we will actually be able to bring a 1 GB throughput to each user’s handheld device. This enables the ability to have much larger streaming video services across 5G platforms. In streaming video today we shoot for 30 frames per second. True video, as it is known, is 60 frames per second. With 5G radio, they’re looking to be able to do true video to the handheld.

Diamond: To paraphrase:  The original design paradigm, or early design paradigm, for 3G and LTE was to deliver, let’s just say, a typical 2 MB per-second per-user interface per-device. Now you’re talking about a gigabit to that same user. That’s a 500 times throughput increase from the radio out. That means that the backhaul is going to have to carry statistically 500 times more traffic.

Linzy: Yes, if everyone maximized their 1 GB, architecturally speaking. Even though we’re providing the 1 GB possibility to each handheld through the 5G architecture, each user on the network will be using a max of 10 to 15 percent of their utilization at any given point in the concurrent base.

Diamond: You’re not sending a gigabit of continuous traffic; you’re sending each one a gigabit’s worth of traffic for 5 milliseconds. It takes their handheld device that long to process that volume of information and present it to them intelligently.

Linzy: Correct. With this increase in bandwidth, however, even though they may not be utilizing all that bandwidth to their handheld, this is how synchronization comes into play. We need to have a true synchronization to that handheld 24 hours of a day. This means that upon a request of information from the cloud, they’re fully synchronized back to the tower, back to the small cell, and back to the cloud at all times. This now means every handheld in the field will now have to have a synchronization service to it to guarantee the performance of that streaming video or the large content data packet downloads, in terms of what they looking to receive. Today we’re synchronizing to the radio, and the radio has its own inherent synchronization down to the handhelds. This is not very precise, but it represents a best effort. With 5G we’re now saying synchronization not only has to get to the tower to the radio but it also has to be passed from that radio all the way down to every handheld. Now the timing requirement will be to have 1.5 microseconds of accuracy of phase and TOD to every phone.

Diamond: You are saying that the phase requirements to date out to the base stations and the radios have phase aligned the lobes on the antennas to be transmitting within 1.5 microseconds or less of each other. But you’re saying that the handsets themselves are going to have to start participating in that timing paradigm in order for them to be able to process the time signals that are being sent to them as well as to be able to respond.

Linzy: That is correct.

Diamond: Interesting.  5G has a profound impact, possibly, on the backhaul approach — not just from the standpoint of the amount of bandwidth that is going to be required to the base station, but in the architecture of the backhaul.  Possibly from a proximity standpoint too. If you’re trying to do this and you want to do it with a minimum of propagation delay you’re going to have to bring very, very high-capacity sources much closer to the end points.

Linzy: Correct. This leads us into a timing as a service issue. As we put out timing as a service to these carriers to be able to support this new architecture, we’re having to reexamine how we’re actually routing our traffic across these Ethernet pipes for synchronization. In the past we have looked at synchronization and said if we give it a high enough QoS label then we can give it a priority routing across the network. However, latency and packet delay variation (PDV) are synchronization killers across an Ethernet network.

With Ethernet traffic, one of the real benefits is what we call its “self-healing properties.” In other words, if we get a congested path or we get a degraded path we route around that problem automatically.  However, there are issues with synchronization if you reroute that traffic.  Let’s say we have a path that goes from Atlanta to North Carolina to Concord on its nearest path. Say we’ve turned up synchronization. We validated PDV and we validated the latency across that link. Then, suddenly, due to degradation, they ship that traffic load to reroute. Now it’s actually loading from Atlanta, to Fort Lauderdale back out to Lexington, Kentucky, and then back to Concord.  You’ve now increased the latency of your path. You’ve also increased the PDV of the path. Now synchronization is no longer working or acceptable at a quality of service level for that particular rerouted path. So in the world of Ethernet we have to look at the synchronization traffic a little bit differently than we do data traffic, or video traffic, or voice traffic. We look at synchronization traffic and actually do what’s called a “weighted route” so that we have predefined paths of rerouting that traffic across the network in the event of a degradation event or an over traffic utilization or subscription to that path.

Diamond: What is the biggest issue regarding service level agreements for timing as a service? Every time you’re offering a service the service has to be against a set of measurements and metrics. These are not only the ones the provider service uses, but also the ones that the consumer of the service has a way of measuring.  Given what you just said, providing timing as a service, which is an inevitable, what is the biggest issue to service level agreements for that to take place?

Linzy: There are actually three different points of interest in that question:

  • The first is for the end-users as well as the providers to have a common customer portal interface with real-time measurements and metrics that they both use and can see. These enable them to measure for SLA purpose as well as to be able to react in the event of a degraded situation.
  • The second part of that is, with synchronization as a service there’s a diversity element. When we go out to one of our customers who’s asking for a timing stream, normally they want that as a diversified stream off of two different grandmasters somewhere in our network. So we come from grandmaster A to their site and we come from grandmaster B to their site across from two different Ethernet pipes or paths. Eventually, we need  a real-time monitoring system that can measure and monitor within a 50 nanosecond window of true time, the two streams coming in to validate that against a known variable
  • That known variable is the third issue, which is the big question. Do we use an atomic clock? And if so, how do we get back to the true atomic clock time point to make that measurement?  Do we use the GPS which is 10 nanoseconds accuracy and use a time mark off of it to be able to use it as a base reference point?  That’s the real question as we have seen in conversation in the industry. What do we use for that point of reference? What time mark do we use throughout the international market?

Diamond: Thank you so much for your insights into this issue, Richard. In closing, is there anything else you would like to add?

Linzy: On behalf of Windstream,  I would like to thank ATIS for this opportunity. I would also like to state the importance of the ATIS-NIST Workshop on Synchronization and Timing Systems (WSTS). Synchronization is playing an increasingly important role in network stabilization, in terms of being able to provide a true precision TOD stamp. Not only for transaction stamping and record keeping for our customers, but also for the ability to do event-correlation and troubleshooting across all networks.

From the customer’s perspective, and with 5G radio coming out in the wireless side, we already see today the use of handheld tablets and other wireless devices that are tied back to their cloud. You’ll find examples throughout the medical industry as well as in the office executive wings of everyday financial groups. Individuals are processing information in real time off of these tablets, and doing business in real time. As this paradigm advances, synchronization both in terms of TOD and phase, all the way down to these handheld devices becomes much more critical.

As new technology and paradigm shifts such as these that I’ve discussed in this interview take place, it is important that our end-users and carrier partners educate themselves on both today’s state of the art as well as the evolving future of the technology in standards, such as the new 1588v3 that is coming out and being ratified hopefully by the end of this year. It’s important to know how these forces move us forward into being able to meet the requirements of 5G networks as well as the high-speed optical and ether services. In terms of the market, these topics are high-impact. WSTS keeps information exchange on these topics active and vital.