Perfsonar

Article

Perfsonar is a toolset for ative performance meassurements on high speed networks. You will find more detailed information at https://www.perfsonar.net/

From their webpage: "PerfSONAR is a network measurement toolkit designed to provide federated coverage of paths, and help to establish end-to-end usage expectations.  There are 1000s of perfSONAR instances deployed world wide, many of which are available for open testing of key measures of network performance.  This global infrastructure helps to identify and isolate problems as they happen, making the role of supporting network users easier for engineering teams, and increasing productivity when utilizing network resources. perfSONAR has been developed through an international collaboration led by Internet2, ESnet, Indiana University, and GEANT."

Current implementations use a server directly connected to the device that represents the meassurement point. Using the switchdev environment it should be possible to install perfsonar directly onto the device and by this avoid having to use external hardware. This might also allow to install perfsonar in an ad hoc mod when needing additionall meassurement points for better debugging of an issue. 

We will focus our experiments on the perfsonar tools, not the backend / frontend of the full suite, as it does not make sense to have this functionallity on the switch hardware.

Our proof of concept install will look in particular at three question:

1. Does the install run into any problems not seen on other platforms? Additionally, we will provide the information needed to install it on the switchdev environment.

2. What is the performance we can achieve, taking into account that the connection between CPU and dataplane might be a bottleneck.

3. Do some test meassurements against public test points to verify correct functionallity

In this example we will use a switchdev environment to install perfsonar directly on a 100G switch. As a switchdev environment is a standard Linux environment with a driver that exposes the 100G network ports / the highspeed dataplane ASIC as a standard linux ethernet interface, there should be no special treatment (like having to compile source code on the platform) needed. We will use the standard dfn/yum install process using the OS rpm repositories. We are following the install procedure as published at the perfsonar website:

https://www.perfsonar.net/deploy/installation-and-configuration/

First, we need to install some prerequisites, like epel and the Internet2 repository that holds the perfsonar rpms: 

[root@localhost hwmon1]# 

/epel-release-6-8.noarch.rpm -hUv https://dl.fedoraproject.org/pub/epel/6/x86_64
Retrieving https://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
warning: /var/tmp/rpm-tmp.NXFbOC: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:epel-release-6-8 ################################# [100%]
main/RPMS/Internet2-repo-0.6-1.noarch.rpmsoftware.internet2.edu/rpms/el6/x86_64/
Retrieving http://software.internet2.edu/rpms/el6/x86_64/main/RPMS/Internet2-repo-0.6-1.noarch.rpm
warning: /var/tmp/rpm-tmp.5GbrKM: Header V4 DSA/SHA1 Signature, key ID 9d7b9686: NOKEY
error: Failed dependencies:
yum is needed by Internet2-repo-0.6-1.noarch
[root@localhost hwmon1]# dnf install yum
Extra Packages for Enterprise Linux 6 - x86_64 7.8 MB/s | 11 MB 00:01
Last metadata expiration check: 0:00:09 ago on Wed Mar 1 21:59:15 2017.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
pyliblzma x86_64 0.5.3-16.fc25 fedora 54 k
python-pycurl x86_64 7.43.0-4.fc25 fedora 242 k
python-six noarch 1.10.0-3.fc25 fedora 35 k
python-urlgrabber noarch 3.10.1-9.fc25 fedora 112 k
python2-iniparse noarch 0.4-20.fc25 fedora 46 k
python2-pygpgme x86_64 0.3-18.fc25 fedora 90 k
python2-rpm x86_64 4.13.0.1-1.fc25 updates 105 k
pyxattr x86_64 0.5.3-8.fc25 fedora 34 k
yum noarch 3.4.3-510.fc25 fedora 1.2 M
yum-metadata-parser x86_64 1.1.4-17.fc25 fedora 39 k

Transaction Summary
================================================================================
Install 10 Packages

Total download size: 2.0 M
Installed size: 8.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/10): pyliblzma-0.5.3-16.fc25.x86_64.rpm 97 kB/s | 54 kB 00:00
(2/10): python-urlgrabber-3.10.1-9.fc25.noarch. 161 kB/s | 112 kB 00:00
(3/10): python2-iniparse-0.4-20.fc25.noarch.rpm 216 kB/s | 46 kB 00:00
(4/10): pyxattr-0.5.3-8.fc25.x86_64.rpm 259 kB/s | 34 kB 00:00
(5/10): python2-pygpgme-0.3-18.fc25.x86_64.rpm 446 kB/s | 90 kB 00:00
(6/10): yum-metadata-parser-1.1.4-17.fc25.x86_6 199 kB/s | 39 kB 00:00
(7/10): python-six-1.10.0-3.fc25.noarch.rpm 194 kB/s | 35 kB 00:00
(8/10): python-pycurl-7.43.0-4.fc25.x86_64.rpm 580 kB/s | 242 kB 00:00
(9/10): yum-3.4.3-510.fc25.noarch.rpm 819 kB/s | 1.2 MB 00:01
(10/10): python2-rpm-4.13.0.1-1.fc25.x86_64.rpm 270 kB/s | 105 kB 00:00
--------------------------------------------------------------------------------
Total 869 kB/s | 2.0 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Installing : python2-rpm-4.13.0.1-1.fc25.x86_64 1/10
Installing : python-six-1.10.0-3.fc25.noarch 2/10
Installing : python2-iniparse-0.4-20.fc25.noarch 3/10
Installing : python-pycurl-7.43.0-4.fc25.x86_64 4/10
Installing : python-urlgrabber-3.10.1-9.fc25.noarch 5/10
Installing : yum-metadata-parser-1.1.4-17.fc25.x86_64 6/10
Installing : pyxattr-0.5.3-8.fc25.x86_64 7/10
Installing : python2-pygpgme-0.3-18.fc25.x86_64 8/10
Installing : pyliblzma-0.5.3-16.fc25.x86_64 9/10
Installing : yum-3.4.3-510.fc25.noarch 10/10
Verifying : yum-3.4.3-510.fc25.noarch 1/10
Verifying : pyliblzma-0.5.3-16.fc25.x86_64 2/10
Verifying : python-urlgrabber-3.10.1-9.fc25.noarch 3/10
Verifying : python2-iniparse-0.4-20.fc25.noarch 4/10
Verifying : python2-pygpgme-0.3-18.fc25.x86_64 5/10
Verifying : pyxattr-0.5.3-8.fc25.x86_64 6/10
Verifying : yum-metadata-parser-1.1.4-17.fc25.x86_64 7/10
Verifying : python-pycurl-7.43.0-4.fc25.x86_64 8/10
Verifying : python-six-1.10.0-3.fc25.noarch 9/10
Verifying : python2-rpm-4.13.0.1-1.fc25.x86_64 10/10

Installed:
pyliblzma.x86_64 0.5.3-16.fc25 python-pycurl.x86_64 7.43.0-4.fc25
python-six.noarch 1.10.0-3.fc25 python-urlgrabber.noarch 3.10.1-9.fc25
python2-iniparse.noarch 0.4-20.fc25 python2-pygpgme.x86_64 0.3-18.fc25
python2-rpm.x86_64 4.13.0.1-1.fc25 pyxattr.x86_64 0.5.3-8.fc25
yum.noarch 3.4.3-510.fc25 yum-metadata-parser.x86_64 1.1.4-17.fc25

Complete!
main/RPMS/Internet2-repo-0.6-1.noarch.rpmsoftware.internet2.edu/rpms/el6/x86_64/m
Retrieving http://software.internet2.edu/rpms/el6/x86_64/main/RPMS/Internet2-repo-0.6-1.noarch.rpm
warning: /var/tmp/rpm-tmp.EP4JTH: Header V4 DSA/SHA1 Signature, key ID 9d7b9686: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:Internet2-repo-0.6-1 ################################# [100%]
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)
error: /etc/pki/rpm-gpg/RPM-GPG-KEY-Internet2: key 1 import failed.
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)
error: /etc/pki/rpm-gpg/RPM-GPG-KEY-Internet2-testing: key 1 import failed.
main/RPMS/Internet2-repo-0.6-1.noarch.rpmsoftware.internet2.edu/rpms/el6/x86_64/m
Retrieving http://software.internet2.edu/rpms/el6/x86_64/main/RPMS/Internet2-repo-0.6-1.noarch.rpm
warning: /var/tmp/rpm-tmp.5HHLOz: Header V4 DSA/SHA1 Signature, key ID 9d7b9686: NOKEY
Preparing... ################################# [100%]
package Internet2-repo-0.6-1.noarch is already installed
[root@localhost hwmon1]#


We additionally need to edit the Internet2.repo file. The mirrorlist entry does not work, so we comment it out, and instead we add a direct entry for the baseurl. You can see the working configuration below:

-------------------

# Name: Internet2 RPM Repository
# URL: http://software.internet2.edu
[Internet2]
name = Internet2 RPM Repository - software.internet2.edu - main
baseurl=http://software.internet2.edu/rpms/el6/$ARCH/main/
#mirrorlist = http://software.internet2.edu/rpms/el6/mirrors-Toolkit-Internet2
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Internet2
gpgcheck = 0
--------------------------------

Now we can install and use the perfsonar tools:

[root@localhost yum.repos.d]# dnf install perfsonar-tools.noarch
Last metadata expiration check: 0:44:23 ago on Tue Feb 28 23:04:05 2017.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
I2util x86_64 1.5-1 Internet2 54 k
autogen-libopts x86_64 5.18.10-1.fc25 fedora 71 k
bwctl-client x86_64 1.6.1-1.el6 Internet2 97 k
bwctl-server x86_64 1.6.1-1.el6 Internet2 118 k
iperf x86_64 2.0.8-6.fc25 fedora 325 k
ndt-client x86_64 3.7.0.2-1.el6 Internet2 46 k
ntp x86_64 4.2.6p5-43.fc25 updates 551 k
ntpdate x86_64 4.2.6p5-43.fc25 updates 90 k
nuttcp x86_64 7.2.1-1 Internet2 70 k
owamp-client x86_64 3.5.0-1.el6 Internet2 133 k
owamp-server x86_64 3.5.0-1.el6 Internet2 109 k
paris-traceroute x86_64 0.92-9.fc24 fedora 69 k
perfsonar-tools noarch 3.5.1-1 Internet2 2.2 k
perl-Sys-Syslog x86_64 0.35-1.fc25 fedora 48 k

Transaction Summary
================================================================================
Install 14 Packages

Total download size: 1.7 M
Installed size: 4.0 M
Is this ok [y/N]: y
Is this ok [y/N]: y
Downloading Packages:
(1/14): perfsonar-tools-3.5.1-1.noarch.rpm 22 kB/s | 2.2 kB 00:00
(2/14): owamp-client-3.5.0-1.el6.x86_64.rpm 777 kB/s | 133 kB 00:00
(3/14): owamp-server-3.5.0-1.el6.x86_64.rpm 624 kB/s | 109 kB 00:00
(4/14): ndt-client-3.7.0.2-1.el6.x86_64.rpm 878 kB/s | 46 kB 00:00
(5/14): bwctl-client-1.6.1-1.el6.x86_64.rpm 774 kB/s | 97 kB 00:00
(6/14): I2util-1.5-1.x86_64.rpm 1.4 MB/s | 54 kB 00:00
(7/14): bwctl-server-1.6.1-1.el6.x86_64.rpm 489 kB/s | 118 kB 00:00
(8/14): nuttcp-7.2.1-1.x86_64.rpm 400 kB/s | 70 kB 00:00
(9/14): perl-Sys-Syslog-0.35-1.fc25.x86_64.rpm 78 kB/s | 48 kB 00:00
(10/14): paris-traceroute-0.92-9.fc24.x86_64.rp 88 kB/s | 69 kB 00:00
(11/14): ntpdate-4.2.6p5-43.fc25.x86_64.rpm 331 kB/s | 90 kB 00:00
(12/14): ntp-4.2.6p5-43.fc25.x86_64.rpm 1.2 MB/s | 551 kB 00:00
(13/14): autogen-libopts-5.18.10-1.fc25.x86_64. 163 kB/s | 71 kB 00:00
(14/14): iperf-2.0.8-6.fc25.x86_64.rpm 247 kB/s | 325 kB 00:01
--------------------------------------------------------------------------------
Total 708 kB/s | 1.7 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Installing : iperf-2.0.8-6.fc25.x86_64 1/14
Installing : bwctl-client-1.6.1-1.el6.x86_64 2/14
Installing : autogen-libopts-5.18.10-1.fc25.x86_64 3/14
Installing : ntpdate-4.2.6p5-43.fc25.x86_64 4/14
Installing : ntp-4.2.6p5-43.fc25.x86_64 5/14
Installing : nuttcp-7.2.1-1.x86_64 6/14
Installing : perl-Sys-Syslog-0.35-1.fc25.x86_64 7/14
Installing : I2util-1.5-1.x86_64 8/14
Installing : bwctl-server-1.6.1-1.el6.x86_64 9/14
Installing : paris-traceroute-0.92-9.fc24.x86_64 10/14
Installing : ndt-client-3.7.0.2-1.el6.x86_64 11/14
Installing : owamp-client-3.5.0-1.el6.x86_64 12/14
Installing : owamp-server-3.5.0-1.el6.x86_64 13/14
Installing : perfsonar-tools-3.5.1-1.noarch 14/14
Verifying : perfsonar-tools-3.5.1-1.noarch 1/14
Verifying : owamp-server-3.5.0-1.el6.x86_64 2/14
Verifying : owamp-client-3.5.0-1.el6.x86_64 3/14
Verifying : bwctl-client-1.6.1-1.el6.x86_64 4/14
Verifying : ndt-client-3.7.0.2-1.el6.x86_64 5/14
Verifying : paris-traceroute-0.92-9.fc24.x86_64 6/14
Verifying : bwctl-server-1.6.1-1.el6.x86_64 7/14
Verifying : I2util-1.5-1.x86_64 8/14
Verifying : perl-Sys-Syslog-0.35-1.fc25.x86_64 9/14
Verifying : nuttcp-7.2.1-1.x86_64 10/14
Verifying : iperf-2.0.8-6.fc25.x86_64 11/14
Verifying : ntp-4.2.6p5-43.fc25.x86_64 12/14
Verifying : ntpdate-4.2.6p5-43.fc25.x86_64 13/14
Verifying : autogen-libopts-5.18.10-1.fc25.x86_64 14/14

Installed:
I2util.x86_64 1.5-1 autogen-libopts.x86_64 5.18.10-1.fc25
bwctl-client.x86_64 1.6.1-1.el6 bwctl-server.x86_64 1.6.1-1.el6
iperf.x86_64 2.0.8-6.fc25 ndt-client.x86_64 3.7.0.2-1.el6
ntp.x86_64 4.2.6p5-43.fc25 ntpdate.x86_64 4.2.6p5-43.fc25
nuttcp.x86_64 7.2.1-1 owamp-client.x86_64 3.5.0-1.el6
owamp-server.x86_64 3.5.0-1.el6 paris-traceroute.x86_64 0.92-9.fc24
perfsonar-tools.noarch 3.5.1-1 perl-Sys-Syslog.x86_64 0.35-1.fc25

Complete!
[root@localhost yum.repos.d]#

As we see this looks exactly as installs on a standard Linux machine looks, there is nothing special happening in regards to us using the hardware accelerated ASIC.

Now we are able to run perfsonar tests against other machines in the net as needed, for example we run bwctl against albq-pt1.es, a publicily accessible test endpoint provided by esnet.


[root@localhost hwmon1]# bwctl -c albq-pt1.es.net
bwctl: NTP: STA_NANO should be set. Make sure ntpd is running, and your NTP configuration is good.
bwctl: Using tool: iperf3
bwctl: 47 seconds until test results available

SENDER START
Connecting to host 198.129.254.6, port 5486
[ 12] local 156.56.6.146 port 35067 connected to 198.129.254.6 port 5486
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 12] 0.00-1.00 sec 48.8 MBytes 409 Mbits/sec 0 5.49 MBytes
[ 12] 1.00-2.00 sec 79.3 MBytes 665 Mbits/sec 0 5.49 MBytes
[ 12] 2.00-3.00 sec 79.8 MBytes 669 Mbits/sec 0 5.49 MBytes
[ 12] 3.00-4.00 sec 80.2 MBytes 673 Mbits/sec 0 5.49 MBytes
[ 12] 4.00-5.00 sec 79.1 MBytes 664 Mbits/sec 0 5.49 MBytes
[ 12] 5.00-6.00 sec 80.0 MBytes 671 Mbits/sec 0 5.49 MBytes
[ 12] 6.00-7.00 sec 80.0 MBytes 671 Mbits/sec 0 5.49 MBytes
[ 12] 7.00-8.00 sec 80.0 MBytes 671 Mbits/sec 0 5.49 MBytes
[ 12] 8.00-9.00 sec 79.2 MBytes 665 Mbits/sec 0 5.49 MBytes
[ 12] 9.00-10.00 sec 80.0 MBytes 671 Mbits/sec 0 5.49 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 12] 0.00-10.00 sec 767 MBytes 643 Mbits/sec 0 sender
[ 12] 0.00-10.00 sec 765 MBytes 642 Mbits/sec receiver

iperf Done.

SENDER END
[root@localhost hwmon1]#

bwctl uses iperf(3) for testing by default. When using iperf3 on one of the dataplane ports there is an additional bottleneck that influences the performance we can achieve. The interface between the CPU where the iperf packets are generated and the dataplane ASIC is a PCI3 bus not providing the full dataplane linerate. Also the CPU is not fast enough to 100G data streams. We did some example testing of the maximum transfer speeds that we can expect in this environment. Without any tuning, iperf3 was able to transfer about 5G/s to a directly connected iperf3 process on a server. When setting the MTU to 9000, we were able to get around 8G/s bandwidth in our setup. So testing on 10G links seems to be reasonably achieveable on this platform, but higher speeds will be challenging. On this platform the vendor is currently working to enable dataplane side generation of test and debugging packets, which might provide a way to do higher speed tests in the future avoiding the CPU and CPU to dataplane connection bottlenecks. As soon as this functionallity is available we will do some more in depth testing of what performance can be achieved.