Personal tools
\\\\\\\\\
Toolbox /////////
KAREN wiki
KIWI ADVANCED RESEARCH AND EDUCATION NETWORK
Jumbo Frames
From KAREN wiki
Contents |
Jumbo Frames
The Technical Centre section of the KAREN website contains information regarding Jumbo Frames.
Another Interesting read is Gigabit Ethernet Jumbo Frames, by Phil Dykstra. His advice is basically every host in the campus should have a path between it and the WAN that:
- does not reduce the host link bandwidth
- does not reduce the MTU
A 3rd bullet point pertinant to our community would be:
- includes seemless and unrestrained access to KAREN routes
REANNZ Jumbo Frames Testing
We are currently performing a series of tests/experiments to demonstrably quantify the benefits of enabling Jumbo Frames, details and updates follow below...
Testing Environment
Getting Started
The first challenge encountered was ensuring that every device on the path was correctly configured for Jumbo Frames, including the hosts. It is safe to assume that the host device does not support Jumbo Frames unless explicitly stated as doing so. For example, we encountered some difficulty confirming the NIC in our standard office laptop Jumbo Frames support status. Alternative hosts with Jumbo Frame support needed to be sourced as a result. Also noteworthy is all hosts involved had 64-bit PCIe LAN controllers. 32-bit PCI LAN Controllers significanlty reduce the maximum throughput of the NIC.
A large packet ping with the 'do not fragment' bit set was used to confirm all the devices in the path supported Jumbo Frames and were correctly configured:
C:\ping ndt.karen.ac.nz -l 4000 -f
Pinging beacon.karen.ac.nz [210.7.46.246] with 4000 bytes of data:
Reply from 210.7.46.246: bytes=4000 time=3ms TTL=62
Reply from 210.7.46.246: bytes=4000 time=1ms TTL=62
Reply from 210.7.46.246: bytes=4000 time=1ms TTL=62
Reply from 210.7.46.246: bytes=4000 time=1ms TTL=62
Ping statistics for 210.7.46.246:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 3ms, Average = 1ms
A certain amount of tweaking was required on the various hosts involved to ensure maximum throughput was achieved. Some detailed and easy to follow information can be found at the Pittsburg Supercomputing Centre. Of particular interest was the Bandwidth Delay Product calculation. This showed that with just 1ms delay between our Wellington based hosts in our test enviromnent, an optimal buffer size would be 1.25MB. If one end of our testing environment had been in Auckland, with 11ms delay, then the optimal buffer size would be 13.75MB. This highlights the fact that Jumbo Frames are most beneficial on Long Fat Networks and that our test environment was not ideal for demonstrating this.
NDT Server
The KAREN Network Diagnostic Tool (NDT) Server was developed by Internet2 to assist users of research and education (R&E) networks to troubleshoot and measure network performance. Throughput tests from both our XP and MacBook Pro hosts quickly showed that the NDT server was not suitable as a test platform for Jumbo Frames. The NDT results actually degraded after enabling Jumbo Frames. As a result we began to concetrate on alternative applications for testing. Perfsonar looks to include suitable tools for testing throughput over time, we will be investigating this further...
FTP Server
Our initial experiances with FTP have been frustrating, standard FTP clients seem incapable of using the maximum MTU size (Jumbo Frame) to inprove throughput. A recent packet capture showed a typical MTU of 60Bytes. We are exploring alternative FTP Applications...
IPERF
Iperf 2.0.2-2 was installed on an Ubuntu server in the hosting facility. This was configured as a Server with the following command line:
iperf -s -m -M 100000 -w 1M -l 24K
Iperf 1.7.0 was installed on the WinXP workstation. This was configured as the client with the following command line and result:
C:\>iperf -c beacon.karen.ac.nz -M 100000 -w 1M -l 24K ------------------------------------------------------------ Client connecting to beacon.karen.ac.nz, TCP port 5001 TCP window size: 1.00 MByte ------------------------------------------------------------ [1820] local 210.7.40.246 port 1260 connected with 210.7.46.250 port 5001 [ ID] Interval Transfer Bandwidth [1820] 0.0-10.0 sec 1.15 GBytes 987 Mbits/sec
With Jumbo Frames Disabled:
C:\>iperf -c beacon.karen.ac.nz -M 100000 -w 1M -l 24k ------------------------------------------------------------ Client connecting to beacon.karen.ac.nz, TCP port 5001 TCP window size: 1.00 MByte ------------------------------------------------------------ [1908] local 210.7.40.246 port 1044 connected with 210.7.46.250 port 5001 [ ID] Interval Transfer Bandwidth [1908] 0.0-10.0 sec 1.03 GBytes 881 Mbits/sec
The Iperf test results above demonstrate an increase of over 10% in throughput, by enabling Jumbo Frames. It is important to note that with the test hosts so close togeather (1ms) that this represents the minimum opportunity for performance gains. It would be great to try the same test over a much longer distance, with delay of 10ms+.
TFTP
Xxx
