Network I/O Performance
English | 简体中文
Spiderpool can be used with Macvlan, SR-IOV, and IPvlan to implement a complete network solution. This article will compare it with the mainstream network CNI plug-ins on the market ( Such as cilium, calico) Network Latency and Throughput in various scenarios
ENV
This test contains performance benchmark data for various scenarios. All tests were performed between containers running on two different bare metal nodes with 10 Gbit/s network interfaces.
- Kubernetes:
v1.28.2 - container runtime:
containerd 1.6.24 - OS:
ubuntu 23.04 - kernel:
6.2.0-35-generic - NIC:
Mellanox Technologies MT27800 Family [ConnectX-5]
| Node | Role | CPU | Memory |
|---|---|---|---|
| master1 | control-plane, worker | 56C | 125Gi |
| worker1 | worker | 56C | 125Gi |
Test object
This test uses macvlan with Spiderpool as the test solution, and selected Calico, Cilium For comparison, two common network solutions are as follows. The following is the relevant version and other information:
| Test object | illustrate |
|---|---|
| Spiderpool based macvlan datapath | Spiderpool version v0.8.0 |
| Calico | Calico version v3.26.1, based on iptables datapath and no tunnels |
| Cilium | Cilium version v1.14.3, based on full eBPF acceleration and no tunneling |
sockperf network latency test
Sockperf is a network benchmarking tool that can be used to measure network latency. It allows you to evaluate the performance of your network by testing the latency between two endpoints. We can use it to separately test Pod's cross-node access to Pod and Service. When testing access to Service's cluster IP, there are two scenarios: kube-proxy or cilium + kube-proxy replacement.
- Cross-node Pod latency testing for Pod IP purposes.
Use sockperf pp --tcp -i <Pod IP> -p 12345 -t 30 to test the latency of cross-node Pod access to the Pod IP. The data is as follows.
| Test object | latency |
|---|---|
| Calico based on iptables datapath and tunnelless | 51.3 usec |
| Cilium based on full eBPF acceleration and no tunneling | 29.1 usec |
| Spiderpool Pod on the same subnet based on macvlan | 24.3 usec |
| Spiderpool Pod across subnets based on macvlan | 26.2 usec |
| node to node | 32.2 usec |
- Cross-node Pod latency test for cluster IP purpose.
Use sockperf pp --tcp -i <Cluster IP> -p 12345 -t 30 to test the latency of cross-node Pod access to the cluster IP. The data is as follows.
| Test object | latency |
|---|---|
| Calico based on iptables datapath and tunnelless | 51.9 usec |
| Cilium based on full eBPF acceleration and no tunneling | 30.2 usec |
| Spiderpool Pod based on macvlan on the same subnet and kube-proxy | 36.8 usec |
| Spiderpool Pod based on macvlan on the same subnet and fully eBPF accelerated | 27.7 usec |
| node to node | 32.2 usec |
netperf performance test
netperf is a widely used network performance testing tool that allows you to measure various aspects of network performance, such as throughput. We can use netperf to test Pod's cross-node access to Pod and Service respectively. When testing access to Service's cluster IP, there are two scenarios: kube-proxy or cilium + kube-proxy replacement.
- Netperf testing of cross-node Pods for Pod IP purposes.
Use netperf -H <Pod IP> -l 10 -c -t TCP_RR -- -r100,100 to test the throughput of cross-node Pod access to Pod IP. The data is as follows.
| Test object | Throughput (rps) |
|---|---|
| Calico based on iptables datapath and tunnelless | 9985.7 |
| Cilium based on full eBPF acceleration and no tunneling | 17571.3 |
| Spiderpool Pod on the same subnet based on macvlan | 19793.9 |
| Spiderpool Pod across subnets based on macvlan | 19215.2 |
| node to node | 47560.5 |
- Netperf testing across node Pods for cluster IP purposes.
Use netperf -H <cluster IP> -l 10 -c -t TCP_RR -- -r100,100 to test the throughput of cross-node Pods accessing the cluster IP. The data is as follows.
| Test object | Throughput (rps) |
|---|---|
| Calico based on iptables datapath and tunnelless | 9782.2 |
| Cilium based on full eBPF acceleration and no tunneling | 17236.5 |
| Spiderpool Pod based on macvlan on the same subnet and kube-proxy | 16002.3 |
| Spiderpool Pod based on macvlan on the same subnet and fully eBPF accelerated | 18992.9 |
| node to node | 47560.5 |
iperf network performance test
iperf is a popular network performance testing tool that allows you to measure network bandwidth between two endpoints. It is widely used to evaluate the bandwidth and performance of network connections. In this chapter, we use it to test Pod's cross-node access to Pod and Service. When testing access to Service's cluster IP, there are two scenarios: kube-proxy or cilium + kube-proxy replacement.
- iperf testing of cross-node Pods for Pod IP purposes.
Use iperf3 -c <Pod IP> -d -P 1 to test the performance of cross-node Pod access to Pod IP. Use the -P parameter to specify threads 1, 2, and 4 respectively. The data is as follows.
| Test object | Number of threads 1 | Number of threads 2 | Number of threads 4 |
|---|---|---|---|
| Calico based on iptables datapath and tunnelless | 3.26 Gbits/sec | 4.56 Gbits/sec | 8.05 Gbits/sec |
| Cilium based on full eBPF acceleration and no tunneling | 9.35 Gbits/sec | 9.36 Gbits/sec | 9.39 Gbits/sec |
| Spiderpool Pod on the same subnet based on macvlan | 9.36 Gbits/sec | 9.37 Gbits/sec | 9.38 Gbits/sec |
| Spiderpool Pod across subnets based on macvlan | 9.36 Gbits/sec | 9.37 Gbits/sec | 9.38 Gbits/sec |
| node to node | 9.41 Gbits/sec | 9.40 Gbits/sec | 9.42 Gbits/sec |
- iperf testing of cross-node Pods for cluster IP purposes.
Use iperf3 -c <cluster IP> -d -P 1 to test the performance of cross-node Pod access to cluster IP. Use the -P parameter to specify threads 1, 2, and 4 respectively. The data is as follows.
| Test object | Number of threads 1 | Number of threads 2 | Number of threads 4 |
|---|---|---|---|
| Calico based on iptables datapath and tunnelless | 3.06 Gbits/sec | 4.63 Gbits/sec | 8.02 Gbits/sec |
| Cilium based on full eBPF acceleration and no tunneling | 9.35 Gbits/sec | 9.35 Gbits/sec | 9.38 Gbits/sec |
| Spiderpool Pod based on macvlan on the same subnet and kube-proxy | 3.42 Gbits/sec | 6.75 Gbits/sec | 9.24 Gbits/sec |
| Spiderpool Pod based on macvlan on the same subnet and fully eBPF accelerated | 9.36 Gbits/sec | 9.38 Gbits/sec | 9.39 Gbits/sec |
| node to node | 9.41 Gbits/sec | 9.40 Gbits/sec | 9.42 Gbits/sec |
redis-benchmark performance test
redis-benchmark is designed to measure the performance and throughput of a Redis server by simulating multiple clients and executing various Redis commands. We used redis-benchmark to test Pod's cross-node access to the Pod and Service where the Redis service is deployed. When testing access to Service's cluster IP, there are two scenarios: kube-proxy or cilium + kube-proxy replacement.
- Cross-node Pod redis-benchmark testing based on Pod IP.
Use redis-benchmark -h <Pod IP> -p 6379 -d 1000 -t get,set to test the performance of cross-node Pod access to Pod IP. The data is as follows.
| Test object | get | set |
|---|---|---|
| Calico based on iptables datapath and tunnelless | 45682.96 rps | 46992.48 rps |
| Cilium based on full eBPF acceleration and no tunneling | 59737.16 rps | 59988.00 rps |
| Spiderpool Pod on the same subnet based on macvlan | 66357.00 rps | 66800.27 rps |
| Spiderpool Pod across subnets based on macvlan | 67444.45 rps | 67783.67 rps |
- Cross-node Pod redis-benchmark testing for cluster IP purposes.
Use redis-benchmark -h <cluster IP> -p 6379 -d 1000 -t get,set to test the performance of cross-node Pod access to cluster IP. The data is as follows.
| Test object | get | set |
|---|---|---|
| Calico based on iptables datapath and tunnelless | 46082.95 rps | 46728.97 rps |
| Cilium based on full eBPF acceleration and no tunneling | 60496.07 rps | 58927.52 rps |
| Spiderpool Pod based on macvlan on the same subnet and kube-proxy | 45578.85 rps | 46274.87 rps |
| Spiderpool Pod based on macvlan on the same subnet and fully eBPF accelerated | 63211.12 rps | 64061.50 rps |
Same node eBPF acceleration test
Spiderpool can achieve same-node communication acceleration with the help of the istio-tcpip-bypass project. Run the service on one node of the cluster and not on the other node. Conduct a performance test through Sockperf between Pods on the same node. The data is as follows.
| Test object | latency |
|---|---|
| Node enables eBPF acceleration | 7.643 usec |
| Node is not enabled for eBPF acceleration | 17.335 usec |
Summary
When Spiderpool is used as an underlay network solution, its IO performance is ahead of Calico and Cilium in most scenarios.