10G, what now

Uppsala universitet

10G, what now

Problem: Your fastest NIC is only 10Gbps and you need more speed

Solution: Use two of them! Linux Bonding saves the day

Then...: What if want even more speed?

No problem, just add more NICs!

Testing Problems

(numbers are from memory and sloppy notes, so...)

3 interfaces ~ 30G (mixed packet sizes)

4 interfaces ~ 33G (mixed packet sizes)

~ 4.8Mpps (64B packets)

The router is too fast, it routes more than we can send with the current pktgen host (old 48-core AMD)

Swap places

Change from routing to generating

Kernel update to Bifrost 3.17

4*10G saturated with mixed packet sizes

ifstat2 needs an update...

             RX --------------------------   TX --------------------------
lo                0   bit/s        0   pps        0   bit/s        0   pps
eth0            936   bit/s        0   pps   9518.3 M bit/s     1325 k pps
eth1            936   bit/s        0   pps   9518.7 M bit/s     1323 k pps
eth2            936   bit/s        0   pps   9518.5 M bit/s     1324 k pps
eth3            936   bit/s        0   pps   9518.4 M bit/s     1324 k pps
eth4            1.3 k bit/s        1   pps   9518.7 M bit/s     1323 k pps
eth5            976   bit/s        0   pps   9518.8 M bit/s     1323 k pps
bond0           5.9 k bit/s        6   pps  25331.6 M bit/s    7.945 M pps

6*10G saturated (mixed packet size)

add another nic, 2 more interfaces

             RX --------------------------   TX --------------------------
lo                0   bit/s        0   pps        0   bit/s        0   pps
eth0            936   bit/s        0   pps   9529.2 M bit/s     1292 k pps
eth1            936   bit/s        0   pps   9528.8 M bit/s     1294 k pps
eth2            936   bit/s        0   pps   9529.1 M bit/s     1293 k pps
eth3            936   bit/s        0   pps   9528.2 M bit/s     1298 k pps
eth4            936   bit/s        0   pps   9529.0 M bit/s     1293 k pps
eth5            936   bit/s        0   pps   9529.0 M bit/s     1293 k pps
eth6            936   bit/s        0   pps   9528.5 M bit/s     1296 k pps
eth7            1.0 k bit/s        1   pps   9529.0 M bit/s     1294 k pps
bond0           7.4 k bit/s        7   pps  12661.9 M bit/s       10 M pps

8*10G saturated (mixed packet size)

lots of fibers

It gets a bit messy... cables

What about pps

             RX --------------------------   TX --------------------------
lo                0   bit/s        0   pps        0   bit/s        0   pps
eth0            952   bit/s        0   pps   1560.7 M bit/s    3.329 M pps
eth1            952   bit/s        0   pps   1560.8 M bit/s    3.329 M pps
eth2            952   bit/s        0   pps   1559.5 M bit/s    3.326 M pps
eth3            952   bit/s        0   pps   1561.5 M bit/s    3.331 M pps
eth4            936   bit/s        0   pps   1561.6 M bit/s    3.331 M pps
eth5            1.3 k bit/s        1   pps   1556.0 M bit/s    3.319 M pps
bond0           5.9 k bit/s        6   pps   9360.1 M bit/s       19 M pps

             RX --------------------------   TX --------------------------
lo                0   bit/s        0   pps        0   bit/s        0   pps
eth0            928   bit/s        0   pps   1151.0 M bit/s    2.455 M pps
eth1            928   bit/s        0   pps   1151.6 M bit/s    2.456 M pps
eth2            816   bit/s        0   pps   1152.0 M bit/s    2.457 M pps
eth3            928   bit/s        0   pps   1149.4 M bit/s    2.452 M pps
eth4            936   bit/s        0   pps   1149.9 M bit/s    2.453 M pps
eth5            928   bit/s        0   pps   1152.9 M bit/s    2.459 M pps
eth6            1.1 k bit/s        1   pps   1151.2 M bit/s    2.455 M pps
eth7            936   bit/s        0   pps   1149.7 M bit/s    2.452 M pps
bond0           7.4 k bit/s        7   pps   9207.6 M bit/s       19 M pps

seems we have a pps limit of ~ 19Mpps

and some perf (from last test)

------------------------------------------------------------------------------
   PerfTop:  177963 irqs/sec  kernel:98.8% [100000 cycles],  (all, 24 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

            54932.00 - 15.6% : _raw_spin_lock
            32755.00 -  9.3% : ixgbe_xmit_frame_ring    [ixgbe]
            15617.00 -  4.4% : ip_send_check
            14003.00 -  4.0% : ixgbe_poll       [ixgbe]
            12475.00 -  3.5% : __dev_queue_xmit
            10628.00 -  3.0% : do_gettimeofday
            10486.00 -  3.0% : build_skb
            10178.00 -  2.9% : __memset
            10139.00 -  2.9% : skb_put
             9534.00 -  2.7% : __netdev_alloc_frag
             8868.00 -  2.5% : irq_entries_start
             7890.00 -  2.2% : kmem_cache_free
             7749.00 -  2.2% : bond_3ad_xor_xmit        [bonding]
             6534.00 -  1.9% : kmem_cache_alloc
             6131.00 -  1.7% : bond_xmit_hash   [bonding]

HW/SW info

HP ProLiant DL380pGen8 passing through

2 x Xeon E5-2630V2 / 2.6 GHz

8 x 2 GB 10600

4 * Dual port Intel (82599)

OS (kernel): Our 3.17, Patches. Jesper and others. Net-next.

(faster CPUs and memory exists)

server inside

soon out of PCIe slots too... inside

Whats next

We need better hardware for generating

40G / 100G cards seems to (soon?) be available

Closing time

That's all for now

Questions