Time-synchronization

This article is written for those interested in latency, especially those using the Wireless Endpoint or across multiple ByteBlower Servers. The quality of the latency measurements strongly depends on having a high-quality time reference.

This is especially visible in the one-way latency measurements of ByteBlower server and Wireless Endpoint. Yet this tool provides invaluable information on network under test. With FrameBlasting you receive throughput and latency information for each direction and for each traffic-flow.

• This makes any asymmetry in either direction clearly visible.
• It allows you to evaluate the active queue management (AQM) behavior.
• With sufficient Endpoints, individual hops can be characterized.

Yet measuring such latency values across distributed setups poses a particular challenge. To this end this text explains how to set up a qualitative time synchronization that solves these difficulties.

Latency of the traffic is the time-interval between transmitting the packet and next receiving somewhere else. Measuring this duration requires a sufficient fine timestamp to of each. In case of a single system, with a single clock, this calculation is straightforward. It is in shown in the picture below.

The time-axis at the center represents the clock on the ByteBlower Server. The packet is transmitted at the left side of the figure, it passes through the “Network under Test” and is received further down the time-axis.

As can be seen above, although this calculation requires precise timestamps, any time-offset is unimportant. Such values are canceled out in the subtraction.

This time-offset becomes important in case multiple systems are involved, e.g. a Wireless Endpoint and a ByteBlower Server. This situation is shown in the next situation.

In the above case, the lack of time synchronization between the ByteBlower and Wireless Endpoint causes them to continuously drift compared to each other. These relative time-offsets have a large impact on the latency measurement, in some cases even resulting in negative latency values.

This can be improved with time synchronization, its goal is to eliminate the drift between these different systems. In the figure below, a time references keeps both Wireless Endpoint and ByteBlower synchronized to each other. This is sufficient to fix the latency measurement issues.

PTP, Precision Time Protocol is a time synchronization protocol intended for local networks. In this it complements NTP that offers time synchronization in wider networks, even the internet as whole. Furthermore, as an IEEE standard (IEEE 1588), PTP has considerable hardware support even in commercial-off-the-shelf hardware.

These attributes make it the ideal protocol for the time synchronization required for accurate latency measurements.. In the next sections we’ll show how to achieve a time sync with offsets of mere microseconds.

PTP is natively supported by the 4100 and 2100 series. It requires dedicated cabling, more information found in the following article. For the other models we recommend falling back to NTP.

In this section we will detail how to setup a new time sync master. Of course if you’re lab already provides PTP, then it does make sense to try using those first.

The guide on the Intel NUC platform. These small barebones that are available for a modest prices. All steps were tested on the “Intel® NUC 11 Lite”. Even this lower-end model was sufficient for a proper time synchronization. This is the same Intel NUC as used for the Golden Client.

In addition to the Intel NUC, many other systems will also serve well. In general following guidelines help you selecting

• Pick physical hardware, avoid virtual machines.
• Disabling CPU frequency scaling tends to improve stability of timesync. This is especially true on older hardware.
• Keep load on the system light, prefer dedicating the system to the time synchronization.

Step 1. Install the OS (Debian)

Just like the Golden Client, Debian is chosen as base OS for the PTP master. This Linux distribution has considerable open source support and will stay relevant for the time being.

An ISO image with the installer is available from the link below. This binary should be copied onto a USB stick. Several tools are available for performing this copy, balena is on that is available on a large number of platforms.

• Installers: https://www.debian.org/distrib/netinst 

This PTP master does not require a desktop environment. If you’re comfortable with managing such a headless system, the desktop can be omitted. For Intel NUC platform both headless and desktop should work well, no large performance differences are expected.

Step 2 PTP Software

After installing Debian, the PTP software is next. Linux has two leading PTP implementations: ptpd and LinuxPTP. This last one is the more recent one of the two.  The software is available in the package manager. It can be installed as follows

• apt install linuxptp

The two following programs in this package are of importance:

• ptp4l
  This program performs the PTP communication with the other nodes in the network. It can work in both MASTER or CLIENT, default it will listen first and only become MASTER if no other node is found. For accurate timestamping, it relies on the hardware Clock on the NIC. This is where the next program becomes important.
• phc2sys
  Unlike the previous, this program is purely local. It provides clock synchronization between the OS clock and the clock on the NIC.
  Keeping these two clocks synchronized is important for hardware timestamping since in this case it's the NIC that performs all the timestamping.

Finally, the linuxPTP package contain also an third tool: Timemaster. This tool is intended for PTP SLAVE systems and not useful in setting up a MASTER instance.

PTP Configuration

Of the above two programs, only ptp4l needs a minimal configfile, mainly to specifiy the PTP domain number. On Debian, this file is stored at /etc/linuxptp/ptp4l.conf. Only the super-user (root) can make changes here.

The PTP domain ensures that multiple PTP services can operate in parallel on the same LAN. Values up to 127 are possible, the example below uses the domain 5.

• domainNumber 5

All systems joining the PTP sync should have the same domain number. This same number should be configured on the Golden Client and/or ByteBlower 4100 system.

As mentioned earlier, default ptp4l first scans the network for other PTP MASTERs. This can be avoided with the following configuration.

• slaveOnly 0
  masterOnly 1

Finally, when the system support its, also enable hardware timestamping in the configuration. This of course requires running phc2sys (shown next).

• time_stamping hardware or time_stamping software
  

Running the PTP programs

Both ptp4l and phc2sys can run from the command-line. This is an easy way to get started with the setup. The list below shows how to run both.

• /usr/sbin/ptp4l -i enp2s0 -f /etc/linuxptp/ptp4l.conf
  This line starts the ptp4l on the network interface enp2s0. It uses the configuration found in the /etc/linuxptp/ptp4l.conf .
  Default no output is printed, all logging is written to /var/log/messages. This can be overriden with the option -m, this is sometimes easier during debugging.
  
  
• /usr/sbin/phc2sys -a -rr -n5
  Using this command phc2sys is instructed to automatically select a clock source (-a) to synchronize the NIC (-rr) for PTP domain 5 (-n5). Like ptp4l, the option -m can be used to print logging to stdout.

Starting everything up at boot

The previous section started the programs from the command line. Including them in the startup of the system makes them more useful. On Debian, and many other linux distributions, this is done by Systemd.

Systemd uses services and associated service files. A new one is created below for each of the above two command-line commands. These files are stored in /etc/systemd/system/ . They were named ptp_master.service and nic_clock_sync.service.

[Unit]
Description=Master PTP server

[Service]
User=root
ExecStart=/usr/sbin/ptp4l -i enp2s0 -f /etc/linuxptp/ptp4l.conf
Restart=always

[Install]
WantedBy=multi-user.target

[Unit]
Description=Sync the Clock on the NIC to improve PTP accuracy

[Service]
User=root
ExecStart=/usr/sbin/phc2sys -a -rr -n5
Restart=always

[Install]
WantedBy=multi-user.target

Finally, Systemd needs to be instructed to also enable the above services at startup. This is done with following commands:

systemctl enable ptp_master.service
systemctl enable nic_clock_sync.service

Step3 Install NTP (optional)

The above steps configure the PTP Master. This provides a most accurate time-synchronization for the local network but without a global time-reference the PTP-synchronized systems will still slowly drift away from the wall-clock time. Altough too small to have a significant impact on the latency measurements, it can make it hard to compare logs and timestamps.

In addition, while PTP is perfect for the Golden Client and ByteBlower 4100 systems, the protocol is not yet supported on ByeBlower 5100, 3x00 and 1x00. These systems require NTP for time-synchronization. To minimize time-offsets between ByteBlower servers and Golden Client it helps to keep latency to the NTP server also to a minimum.

Multiple NTP implementations are available; the steps below use Chrony. This is a more recent implementation that performs well compared to the others.  Installing Chrony on Debian is easy:

• apt install Chrony

Unlike the earlier mentioned PTP programs, on Debian this package will also configure Chrony to start at boot. The status is visible with the following command

• systemctl status chrony

To use Chrony also as NTP reference, the keyword allow should be added in /etc/chrony/chrony.conf. Like before, only the super-user (root) can modify these files. 

To evaluate the time sync master, we recommend creating a small setup with minimal latency between the synchronized systems. Measuring the latency over this network provides lower-bound where further tests can be compared to.

For example,

• Between 2x 10Gbit ByteBlower servers.
  These systems can be connected directly with a fiber cable.
• Between a 10Gbit ByteBlower server and Golden Client
  Connect the wired Ethernet of the Golden Client to the ByteBlower server via the ByteBlower switch.

Trial testruns can be done with the ByteBlower GUI. As shown below, configure a latency flow in either direction.

Running the above configuration on the earlier described timesync master gives the results shown below.

As can ben seen in both the tabular and the graphs, both ByteBlower 4100 servers never deviated farther than a couple microseconds. Such level of accuracy is sufficient for Wi-Fi, Docsis, (XS)GPON and 4G/5G testing.