Linux low-latency

June 19, 2018 | 3 Minute Read

Linux low-latency related knowledge

An attempt to gather resources on low-latency linux. Musicians use low-latency linux for recording and live performance. In stock market ticker data streaming one will need low-latency on per packet processing.
A resource on preparing UbuntuStudio specially for musicians.

First things first, if you need lower latency, get away from ubuntu and use Debian or Redhat (small footprint) OS image + kernel.

Zero-copy Zero copy lets you avoid redundant data copies between intermediate buffers and reduces the number of context switches between user-space and kernel-space. Ideal Zero copy (zero cpu copy) is possible when your hardware (disk drive, network card, graphic card, sound card) supports DMA (Direct Memory Acess).

File system zero-copy: reduce user-space, kernel-space interaction for file copying Check this code in github Filesystem zero-copy in C/Java.

Q: How to install the low-latency kernel in Ubuntu 16.04? following might work
sudo apt-get install linux-image-lowlatency

Playing with sysctl flags (offered by kernel to configure)</b>
Two servers located in two different data center. Both server deals with a lot of concurrent large file transfersa. But network performance is very poor for large files and performance degradation take place with a large files. How do I tune TCP under Linux to solve this problem? Documentation Here are some of the sysctl flags to adjust kernel behavior.

--Default maximum Linux TCP buffer size
cat /proc/sys/net/ipv4/tcp_mem
--The default and maximum amount for the receive socket memory
cat /proc/sys/net/core/rmem_default
cat /proc/sys/net/core/rmem_max
--The default and maximum amount for the send socket memory:
cat /proc/sys/net/core/wmem_default
cat /proc/sys/net/core/wmem_max
--The maximum amount of option memory buffers:
cat /proc/sys/net/core/optmem_max


--Tune values
echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf
--You also need to set minimum size, initial size, and maximum size in bytes:
echo 'net.ipv4.tcp_rmem= 10240 87380 12582912' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem= 10240 87380 12582912' >> /etc/sysctl.conf
--Turn on window scaling which can be an option to enlarge the transfer window:
echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
--Enable timestamps as defined in RFC1323:
echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
--Enable select acknowledgments:
echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
--By default, TCP saves various connection metrics in the route cache when the connection closes, so that connections established in the near future can use these to set initial conditions. Usually, this increases overall performance, but may sometimes cause performance degradation. If set, TCP will not cache metrics on closing connections.
echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf

--Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them.
echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf

--Now reload the changes:
sysctl -p

--Use tcpdump to view changes for eth0:
tcpdump -ni eth0

--done_

These are some simple guidelines on selecting the right kernel that fits your need (in a particularorder)–by ubuntu fan askubuntu

  • If you do not require low latency for your system then please use the -generic kernel.
  • If you need a low latency system (e.g. for recording audio) then please use the -preempt kernel as a first choice. This reduces latency but doesnt sacrifice power saving features. It is available only for 64 bit systems (amd64).
  • If the -preempt kernel does not provide enough low latency for your needs (or you have an 32 bit system) then you should try the -lowlatency kernel.
  • If the -lowlatency kernel isn't enough then you should try the -rt kernel
  • If the -rt kernel isn't enough stable for you then you should try the -realtime kernel

The most relevant kernel options if you want to recompile your kernel yourself to have a low-latency desktop:
PREEMPT=y and: CONFIG_1000_HZ=y To add some powersaving check this one: CONFIG_NO_HZ=y

Additionally, preemt, lowlatency or the rt kernel wont make your system faster (for general tasks?). They are slightly slower than generic kernel.

Good references

  • https://blog.cloudflare.com/how-to-achieve-low-latency/
  • redhat tuning: Redhat low-latency config guide
  • Linux tcp tuning: https://www.cyberciti.biz/faq/linux-tcp-tuning/

© 2021 Abu Obaida