Time-synchronization Configuration

In this section we will detail how to setup a new time sync master. Of course if your lab already provides PTP, then it does make sense to try using those first. PTP is natively supported by the 4100 and 2100 series, the protocol is not yet supported on ByeBlower 5100, 3x00 and 1x00. These systems require NTP for time-synchronization. (NTP section below)

The guide on the Intel NUC platform. These small barebones that are available for a modest prices. All steps were tested on the “Intel® NUC 11 Lite”. Even this lower-end model was sufficient for a proper time synchronization. This is the same Intel NUC as used for the Golden Client.

In addition to the Intel NUC, many other systems will also serve well. In general following guidelines help you selecting

• Pick physical hardware, avoid virtual machines.
• Disabling CPU frequency scaling tends to improve stability of timesync. This is especially true on older hardware.
• Keep load on the system light, prefer dedicating the system to the time synchronization.

Step 1. Install the OS (Debian)

Just like the Golden Client, Debian is chosen as base OS for the PTP master. This Linux distribution has considerable open source support and will stay relevant for the time being.

An ISO image with the installer is available from the link below. This binary should be copied onto a USB stick. Several tools are available for performing this copy, balena is on that is available on a large number of platforms.

• Installers: https://www.debian.org/distrib/netinst 

This PTP master does not require a desktop environment. If you’re comfortable with managing such a headless system, the desktop can be omitted. For Intel NUC platform both headless and desktop should work well, no large performance differences are expected.

Step 2 PTP Software

After installing Debian, the PTP software is next. Linux has two leading PTP implementations: ptpd and LinuxPTP. This last one is the more recent one of the two.  The software is available in the package manager. It can be installed as follows

apt-get install linuxptp

The two following programs in this package are of importance:

• ptp4l
  This program performs the PTP communication with the other nodes in the network. It can work in both MASTER or CLIENT, default it will listen first and only become MASTER if no other node is found. For accurate timestamping, it relies on the hardware Clock on the NIC. This is where the next program becomes important.
• phc2sys
  Unlike the previous, this program is purely local. It provides clock synchronization between the OS clock and the clock on the NIC.
  Keeping these two clocks synchronized is important for hardware timestamping since in this case it's the NIC that performs all the timestamping.

Finally, the linuxPTP package contain also an third tool: Timemaster. This tool is intended for PTP SLAVE systems and not useful in setting up a MASTER instance.

PTP Configuration

Of the above two programs, only ptp4l needs a minimal configfile, mainly to specifiy the PTP domain number. On Debian, this file is stored at /etc/linuxptp/ptp4l.conf. Only the super-user (root) can make changes here.

The PTP domain ensures that multiple PTP services can operate in parallel on the same LAN. Values up to 127 are possible, the example below uses the domain 5.

domainNumber 5

All systems joining the PTP sync should have the same domain number. This same number should be configured on the Golden Client and/or ByteBlower 4100 system.

As mentioned earlier, default ptp4l first scans the network for other PTP MASTERs. This can be avoided with the following configuration.

slaveOnly 0
masterOnly 1

Finally, when the system support its, also enable hardware timestamping in the configuration. This of course requires running phc2sys (shown next).

Either:

time_stamping hardware

or:

time_stamping software

Running the PTP programs

Both ptp4l and phc2sys can run from the command-line. This is an easy way to get started with the setup. The list below shows how to run both.

• ​/usr/sbin/ptp4l -i enp2s0 -f /etc/linuxptp/ptp4l.conf​
  This line starts the ptp4l on the network interface enp2s0. It uses the configuration found in the /etc/linuxptp/ptp4l.conf .
  Default no output is printed, all logging is written to /var/log/messages. This can be overriden with the option -m, this is sometimes easier during debugging.
  
  
• ​/usr/sbin/phc2sys -a -rr -n5​
  Using this command phc2sys is instructed to automatically select a clock source (-a) to synchronize the NIC (-rr) for PTP domain 5 (-n5). Like ptp4l, the option -m can be used to print logging to stdout.

Starting everything up at boot

The previous section started the programs from the command line. Including them in the startup of the system makes them more useful. On Debian, and many other linux distributions, this is done by Systemd.

Systemd uses services and associated service files. A new one is created below for each of the above two command-line commands. These files are stored in /etc/systemd/system/ . They were named ptp_master.service and nic_clock_sync.service.

[Unit]
Description=Master PTP server

[Service]
User=root
ExecStart=/usr/sbin/ptp4l -i enp2s0 -f /etc/linuxptp/ptp4l.conf
Restart=always

[Install]
WantedBy=multi-user.target

[Unit]
Description=Sync the Clock on the NIC to improve PTP accuracy

[Service]
User=root
ExecStart=/usr/sbin/phc2sys -a -rr -n5
Restart=always

[Install]
WantedBy=multi-user.target

Finally, Systemd needs to be instructed to also enable the above services at startup. This is done with following commands:

systemctl enable ptp_master.service
systemctl enable nic_clock_sync.service

The above steps configure the PTP Master. This provides a most accurate time-synchronization for the local network but without a global time-reference the PTP-synchronized systems will still slowly drift away from the wall-clock time. Altough too small to have a significant impact on the latency measurements, it can make it hard to compare logs and timestamps.

In addition, while PTP is perfect for the Golden Client and ByteBlower 4100 systems, the protocol is not yet supported on ByeBlower 5100, 3x00 and 1x00. These systems require NTP for time-synchronization. To minimize time-offsets between ByteBlower servers and Golden Client it helps to keep latency to the NTP server also to a minimum.

Multiple NTP implementations are available; the steps below use Chrony. This is a more recent implementation that performs well compared to the others.  Installing Chrony on Debian is easy:

apt-get install Chrony

Unlike the earlier mentioned PTP programs, on Debian this package will also configure Chrony to start at boot. The status is visible with the following command

systemctl status chrony.service

To use Chrony also as NTP reference, the keyword allow should be added in /etc/chrony/chrony.conf. Like before, only the super-user (root) can modify these files. 

To evaluate the time sync master, we recommend creating a small setup with minimal latency between the synchronized systems. Measuring the latency over this network provides lower-bound where further tests can be compared to.

For example,

• Between 2x 10Gbit ByteBlower servers.
  These systems can be connected directly with a fiber cable.
• Between a 10Gbit ByteBlower server and Golden Client
  Connect the wired Ethernet of the Golden Client to the ByteBlower server via the ByteBlower switch.

Trial testruns can be done with the ByteBlower GUI. As shown below, configure a latency flow in either direction.

Running the above configuration on the earlier described timesync master gives the results shown below.

As can ben seen in both the tabular and the graphs, both ByteBlower 4100 servers never deviated farther than a couple microseconds. Such level of accuracy is sufficient for Wi-Fi, Docsis, (XS)GPON and 4G/5G testing.