Problem: Your fastest NIC is only 10Gbps and you need more speed
Solution: Use two of them! Linux Bonding saves the day
Then...: What if want even more speed?
No problem, just add more NICs!
(numbers are from memory and sloppy notes, so...)
3 interfaces ~ 30G (mixed packet sizes)
4 interfaces ~ 33G (mixed packet sizes)
~ 4.8Mpps (64B packets)
The router is too fast, it routes more than we can send with the current pktgen host (old 48-core AMD)
Change from routing to generating
Kernel update to Bifrost 3.17
4*10G saturated with mixed packet sizes
RX -------------------------- TX -------------------------- lo 0 bit/s 0 pps 0 bit/s 0 pps eth0 936 bit/s 0 pps 9518.3 M bit/s 1325 k pps eth1 936 bit/s 0 pps 9518.7 M bit/s 1323 k pps eth2 936 bit/s 0 pps 9518.5 M bit/s 1324 k pps eth3 936 bit/s 0 pps 9518.4 M bit/s 1324 k pps eth4 1.3 k bit/s 1 pps 9518.7 M bit/s 1323 k pps eth5 976 bit/s 0 pps 9518.8 M bit/s 1323 k pps bond0 5.9 k bit/s 6 pps 25331.6 M bit/s 7.945 M pps
6*10G saturated (mixed packet size)
RX -------------------------- TX -------------------------- lo 0 bit/s 0 pps 0 bit/s 0 pps eth0 936 bit/s 0 pps 9529.2 M bit/s 1292 k pps eth1 936 bit/s 0 pps 9528.8 M bit/s 1294 k pps eth2 936 bit/s 0 pps 9529.1 M bit/s 1293 k pps eth3 936 bit/s 0 pps 9528.2 M bit/s 1298 k pps eth4 936 bit/s 0 pps 9529.0 M bit/s 1293 k pps eth5 936 bit/s 0 pps 9529.0 M bit/s 1293 k pps eth6 936 bit/s 0 pps 9528.5 M bit/s 1296 k pps eth7 1.0 k bit/s 1 pps 9529.0 M bit/s 1294 k pps bond0 7.4 k bit/s 7 pps 12661.9 M bit/s 10 M pps
8*10G saturated (mixed packet size)
It gets a bit messy...
RX -------------------------- TX -------------------------- lo 0 bit/s 0 pps 0 bit/s 0 pps eth0 952 bit/s 0 pps 1560.7 M bit/s 3.329 M pps eth1 952 bit/s 0 pps 1560.8 M bit/s 3.329 M pps eth2 952 bit/s 0 pps 1559.5 M bit/s 3.326 M pps eth3 952 bit/s 0 pps 1561.5 M bit/s 3.331 M pps eth4 936 bit/s 0 pps 1561.6 M bit/s 3.331 M pps eth5 1.3 k bit/s 1 pps 1556.0 M bit/s 3.319 M pps bond0 5.9 k bit/s 6 pps 9360.1 M bit/s 19 M pps
RX -------------------------- TX -------------------------- lo 0 bit/s 0 pps 0 bit/s 0 pps eth0 928 bit/s 0 pps 1151.0 M bit/s 2.455 M pps eth1 928 bit/s 0 pps 1151.6 M bit/s 2.456 M pps eth2 816 bit/s 0 pps 1152.0 M bit/s 2.457 M pps eth3 928 bit/s 0 pps 1149.4 M bit/s 2.452 M pps eth4 936 bit/s 0 pps 1149.9 M bit/s 2.453 M pps eth5 928 bit/s 0 pps 1152.9 M bit/s 2.459 M pps eth6 1.1 k bit/s 1 pps 1151.2 M bit/s 2.455 M pps eth7 936 bit/s 0 pps 1149.7 M bit/s 2.452 M pps bond0 7.4 k bit/s 7 pps 9207.6 M bit/s 19 M pps
seems we have a pps limit of ~ 19Mpps
------------------------------------------------------------------------------ PerfTop: 177963 irqs/sec kernel:98.8% [100000 cycles], (all, 24 CPUs) ------------------------------------------------------------------------------ samples pcnt kernel function _______ _____ _______________ 54932.00 - 15.6% : _raw_spin_lock 32755.00 - 9.3% : ixgbe_xmit_frame_ring [ixgbe] 15617.00 - 4.4% : ip_send_check 14003.00 - 4.0% : ixgbe_poll [ixgbe] 12475.00 - 3.5% : __dev_queue_xmit 10628.00 - 3.0% : do_gettimeofday 10486.00 - 3.0% : build_skb 10178.00 - 2.9% : __memset 10139.00 - 2.9% : skb_put 9534.00 - 2.7% : __netdev_alloc_frag 8868.00 - 2.5% : irq_entries_start 7890.00 - 2.2% : kmem_cache_free 7749.00 - 2.2% : bond_3ad_xor_xmit [bonding] 6534.00 - 1.9% : kmem_cache_alloc 6131.00 - 1.7% : bond_xmit_hash [bonding]
HP ProLiant DL380pGen8 passing through
2 x Xeon E5-2630V2 / 2.6 GHz
8 x 2 GB 10600
4 * Dual port Intel (82599)
OS (kernel): Our 3.17, Patches. Jesper and others. Net-next.
(faster CPUs and memory exists)
soon out of PCIe slots too...
We need better hardware for generating
40G / 100G cards seems to (soon?) be available
That's all for now
Questions