Browse code

doc/tutorials/tcp_tunning: note about network buffers size

- they may need tunning when dealing with large data

Daniel-Constantin Mierla authored on 15/01/2019 12:02:08
Showing 1 changed files
... ...
@@ -19,22 +19,22 @@ For now it deals only with Linux specific optimizations.
19 19
 
20 20
 net.core.somaxconn             -  limit of the listen() backlog, default 128
21 21
 net.ipv4.tcp_max_syn_backlog   -  default 1024 or 128
22
-net.ipv4.tcp_timestamps        - default on., should be on (along with 
22
+net.ipv4.tcp_timestamps        - default on., should be on (along with
23 23
                                  tcp_tw_recycle and timestamp supporting
24 24
                                  peers allows for fast connections rates)
25 25
 
26
-1.2 connection in close_wait
26
+1.2 Connection in close_wait
27 27
 
28
- Connection should stay as little as possible
29
- in close_wait to quickly free the fd/resources for new connections attempts
30
- WARNING: this could break normal TCP use, use it only if you know what you are
28
+  Connection should stay as little as possible
29
+  in close_wait to quickly free the fd/resources for new connections attempts
30
+  WARNING: this could break normal TCP use, use it only if you know what you are
31 31
   doing
32
- 
32
+
33 33
 net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
34 34
                               (the default seems to be ok)
35 35
 net.ipv4.tcp_tw_recycle     - enables fast time wait sockets recycling (default
36 36
                               off), should be enabled if you have lots of short
37
-                              lived connections 
37
+                              lived connections
38 38
                               WARNING: see the above warning
39 39
 net.ipv4.tcp_tw_reuse       - allows reusing of time-wait sockets (default off)
40 40
                               WARNING: see above
... ...
@@ -49,10 +49,26 @@ net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)
49 49
 1.4 Open file descriptors
50 50
 
51 51
 fs.file-max                 - maximum number of fds that will be allocated
52
-                              (you probably need to increase it, default 
52
+                              (you probably need to increase it, default
53 53
                                depends on installed memory)
54 54
 
55
-1.5 other sysctl that might affect tcp connection rate or the maximum number
55
+1.5 Buffers size
56
+
57
+  When dealing with large data, check the values for network sockets buffers
58
+  to be sufficient, among them:
59
+
60
+net.core.rmem_max
61
+net.core.wmem_max
62
+net.core.rmem_default
63
+net.core.wmem_default
64
+net.ipv4.tcp_mem
65
+net.ipv4.tcp_rmem
66
+net.ipv4.tcp_wmem
67
+
68
+  Note: sysctl -w net.ipv4.route.flush=1 - enusre that immediatly subsequent
69
+  connections use the values
70
+
71
+1.6 Other sysctl that might affect tcp connection rate or the maximum number
56 72
     of open connections
57 73
 
58 74
 fs.epoll.max_user_instances - maximum number of devices - per user (2.6.27.8+)
Browse code

doc: reorganized the content of doc folder

Daniel-Constantin Mierla authored on 07/12/2016 13:54:42
Showing 1 changed files
1 1
new file mode 100644
... ...
@@ -0,0 +1,103 @@
1
+Kamailio TCP Tunning/monitoring for lots of open connections
2
+==============================================================
3
+
4
+0. Introduction
5
+----------------
6
+
7
+This document describes very briefly various settings that should improve
8
+Kamailio TCP performance for sites handling a lot of TCP traffic (> 1000 open
9
+connections or very high connection/disconnection rates).
10
+
11
+For now it deals only with Linux specific optimizations.
12
+
13
+
14
+1. Useful Linux kernel settings
15
+-------------------------------
16
+
17
+1.1 Connection rate/pending connections: by default the connection rate is
18
+ too small
19
+
20
+net.core.somaxconn             -  limit of the listen() backlog, default 128
21
+net.ipv4.tcp_max_syn_backlog   -  default 1024 or 128
22
+net.ipv4.tcp_timestamps        - default on., should be on (along with 
23
+                                 tcp_tw_recycle and timestamp supporting
24
+                                 peers allows for fast connections rates)
25
+
26
+1.2 connection in close_wait
27
+
28
+ Connection should stay as little as possible
29
+ in close_wait to quickly free the fd/resources for new connections attempts
30
+ WARNING: this could break normal TCP use, use it only if you know what you are
31
+  doing
32
+ 
33
+net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
34
+                              (the default seems to be ok)
35
+net.ipv4.tcp_tw_recycle     - enables fast time wait sockets recycling (default
36
+                              off), should be enabled if you have lots of short
37
+                              lived connections 
38
+                              WARNING: see the above warning
39
+net.ipv4.tcp_tw_reuse       - allows reusing of time-wait sockets (default off)
40
+                              WARNING: see above
41
+
42
+net.ipv4.tcp_syncookies     - default off, in this case it's probably better to
43
+                              keep it off
44
+
45
+1.3 Port range
46
+
47
+net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)
48
+
49
+1.4 Open file descriptors
50
+
51
+fs.file-max                 - maximum number of fds that will be allocated
52
+                              (you probably need to increase it, default 
53
+                               depends on installed memory)
54
+
55
+1.5 other sysctl that might affect tcp connection rate or the maximum number
56
+    of open connections
57
+
58
+fs.epoll.max_user_instances - maximum number of devices - per user (2.6.27.8+)
59
+fs.epoll.max_user_watches   - maximum number of "watched" fds - per user
60
+                              (2.6.27.8+)
61
+
62
+net.ipv4.tcp_max_orphans    - might be worth a look if things go wrong
63
+net.core.netdev_max_backlog - maximum device backlog
64
+
65
+Related applications
66
+--------------------
67
+ifconfig <dev> txqueuelen <val> - set device transmission queue len
68
+
69
+iptables  - remove the ip_conntrack module (it limits the maximum tcp
70
+  connections, adds extra overhead (slow)). It's probably better to remove
71
+  all the iptables modules.
72
+
73
+
74
+2. Monitoring (values to watch for)
75
+-----------------------------------
76
+
77
+2.1 File descriptors
78
+
79
+fs.dentry-state  - format: nr. dentries, nr. unused, age_limit, want_pages
80
+fs.file-nr       - format: allocated, unused, max (==fs.file-max)
81
+fs.inode-state  -  format: nr. allocated, nr. free, preshrink
82
+
83
+2.2 TCP
84
+
85
+/proc/net/netstat - the TW, TWRecycled, TWKilled, PAWPassive, PAWActive,
86
+                    PASWEstab fields
87
+                   ( cat /proc/net/netstat |cut -d" " -f12-17 ;
88
+                     cat /proc/net/sockstat)
89
+/proc/net/sockstat
90
+
91
+
92
+3. Kamailio settings
93
+----------------------
94
+
95
+- Don't forget to increase tcp_max_connections and the amount of shared memory
96
+- You should increase the number of Kamailio "tcp_children" processes (-N no)
97
+  As a rule of thumb, (maximum simultaneous connections)/2000 should be ok
98
+- You might have to decrease TCP_BUF_SIZE to a smaller value (e.g 8K)
99
+- You might want to increase PKG_MEM_POOL_SIZE (for large queues)
100
+
101
+- You might need to increase the maximum open fds limit before starting Kamailio
102
+  (e.g. ulimit -n 1000000)
103
+