Background: Calculating network throughput on layered networks
Posted by Tim De Backer, Last modified by Dries Decock on 03 April 2018 02:26 PM
This is a background article on throughput calculation on layered packet-based networks. For information about throughput values in ByteBlower GUI reports, see here.
Throughput in packet-based networks
Calculating throughput is very straightforward. We simple have to calculate how much of something is handled in a second. In the case of networking, this boils down to how much data is transmitted or received in a second:
To calculate the throughput of something, we have to define that something. In other words, we should define what traffic is part of the
Furthermore, within the context of packet-based networks, the relevant data may be counted or measured in two different units.
The resulting throughput will thus be measured in packets per seconds (pps) or bits per second (bps). These two measurements are obviously closely related.
The throughput in packets is typically called the packet rate or (depending on the protocol in question) frame rate, segment rate,....
After deciding what data is counted and in what unit (packets or bits) it is measured, there is only one thing left to decide: what does it mean to transmit or receive data. In layered networks such as TCP/IP, this differs from layer to layer.
Throughput in layered packet-based networks
In layered networks, the networking functionality is divided in multiple layers. For example, the TCP/IP stack consists of the following layers:
Network protocols implement the functionality of such a layer. Examples are:
Each layer provides a service to the layer above. For example, layer 3 offers unreliable delivery over a network to the end-to-end layer 4 transport protocol. On the other hand, each protocols uses a combination of internal logic and services of the layer below to operate. For example, layer 3 delivers messages over a network by determining the direction of the destination (layer 3 logic) and sending the packet to the next hop in that direction (through a layer 2 service). This is clarified in the figure below:
Due to the different scope and functionality of network layers, the definition of transmitting (TX) and receiving (RX) changes as well. For example, from a layer 5 point of view, transmission of a network message may be reliable end-to-end delivery of that message, while layer 2 sees transmission as taking data across a single network link.
To define a measure of throughput, we still needed to define what TX and RX ment. In the context of a layered network stack, it comes down to this rule:
This means that we can calculate both the throughput in packets per second and the throughput in bits per second for each of the layers. This is put in practice for the TCP/IP stack in the next section.
TCP/IP network stack
The figure below shows a typical TCP/IP network stack when used over Ethernet and a typical network packet that comes through that stack. It consists of the following protocols:
When a layer uses a service of the layer below, this service takes up time and resources. In a stacked setup, bottlenecks throughout the stack may limit the throughput. Possible limitating factors are:
Furthermore, much of this functionality has in itself impact on the throughput. This will be examplified in the following subsections.
Layer 2 throughput: Ethernet
The Ethernet protocol spans both the data-link layer (layer 2) and the physical layer (layer 1) and there is no clear interface between those two parts. Because there is no single interface, there is no single defintion of throughput, because the definition depends on that interface (see above).
Frames per second (framerate)
To calculate the throughput in Ethernet frames per second, we simply have to count the number of frames in the relevant period of time.
Bits per second
On the other hand, the throughput in bits per second depends on whether we consider none, some or all of the following functionality as part of the layer 2 protocol (data link transfer) or the layer 1 protocol (physical transmission):
Functionality that is considered part of layer 2 is located above the inter-layer interface used in the throughput definition. Therefore, the corresponding bytes in the network packet are also part of the layer 2 header and thus part of the data moving accross the interface to and from layer 1. This is clarified in picture X.
By placing more and more functions at layer 2, we get the following configurations:
The effect on the throughput becomes clear when looking at an example. Consider a layer 2 payload of 1000 bytes and a frame rate (i.e. throughput in pps) of 1000 frames per second. The throughput in bits per second is then calculated follows:
Notice the effect of the configuration is limited. The results are only as the results only 2.3 percent apart. However, this effect becomes much larger when packets are smaller. Consider the same example with the minimal layer 2 payload size for ethernet, which is 46 bytes:
In this case the difference in throughput between the two extremes has risen to 40 percent!
Note: minimum size and padding
The Ethernet specification determines that the minimal size for a layer 2 Ethernet frame (including the MAC header but excluding the rest) is 60 bytes. This means that the minimal payload size is 46 bytes. When the layer 2 payload (typically an IP datagram) is less then 46 bytes, Ethernet will fill up the remaining bytes with padding to create a 60 byte frame.
This padding is therefore part of the layer 2 payload, but not of the layer 3 packet.
Layer 3 throughput: IP
Packets per second (packet rate)
When IP runs on top of ethernet, calculating the layer 3 throughput in packets per second is typically quite straightforward. An IP datagram corresponds exactly with the payload of a single Ethernet frame.
If the IP protocol receives a network segment (TCP) or datagram (UDP) from layer 4 that is too large for the interface to layer 2 it will either:
In any case, the number of layer 3 IP datagrams is the same as the number of layer 2 Ethernet frames.
Bits per second
To calculate the throughput in bits per second, simply strip the Ethernet frame from all layer 2 information. This includes all Ethernet II header information and the padding. Use the resulting IP datagram size to calculate the throughput in bits per second.
Note: other protocols
The calculation for ethernet is straighforward because a single IP datagram always matches a single Ethernet frame. Notice that for other layer 2 protocols, things may not be that easy.
For example, running IP over ATM causes a single IP datagram to be chopped up to fit the fixed 48-byte payload size of ATM cells. At the receiving side of the link, ATM reassembles the IP datagram before throwing it over the inter-layer interface towards layer 3. This may influence the throughput in a number of ways:
Layer 4 throughput: TCP or UDP
The throughput definition is just as valid on layer 4: the amount of relevant data per second that is passed up or down the interface to layer 3 (IP). The packets passed down to or up from layer 3 are typically called segments in the case of TCP and segments or datagrams in the case of UDP. The layer 4 protocol may decide on their size.
The fragmentation and reassembly of both IPv4 and IPv6 (described above) is transparent to the layer 4 protocol running at the end-hosts. From the layer 4 point of view, the TCP segment or UDP datagram passed down at the sending host is received unchanged at the destination host.
However, the fragmentation at layer 3 may have impact on the layer 4 throughput:
Therefore, layer 4 will typically avoid layer 3 fragmentation by providing packets of the correct size. This may be done through end-to-end path Maximum Transmission Unit (MTU) discovery.
Segments or datagrams per second
For the connectionless UDP, defining the number of packets per second is trivial. The packet throughput is only limited by the performance of the underlying layers.
For the connection-oriented TCP all transferred segments such as ACK messages and possible retransmissions are included. The segment throughput may be influenced by:
Due to this complexity, layer 4 throughput is not very interesting. If we want to know the end-user throughput, it is better to calculate the layer 5 throughput, which takes and end-user application point of view.
Bits per second
Once the number of TCP segments or UDP datagrams per second is defined, the throughput in bits per second can be easily calculated.
Layer 5 throughput: HTTP, FTP, ...
From the standpoint of an application protocol, such as HTTP, all functionality of the network stack is abstracted away.
The interface between layer 4 and layer 5 is typically called a socket. The socket interface completely hides the packet based network. Instead it acts as a buffered input or output stream:
The rule of thumb for layer 5 thus boils down to the amount of data moving through the socket and may depend on many factors:
Since the notion of packets is gone at layer 5, only the throughput in bits per second is defined.