Browse code

- tcp tunning hints for runnig ser with lots of tcp connections (for now only linux is discussed)

Andrei Pelinescu-Onciul authored on 17/01/2007 22:14:43
Showing 1 changed files
1 1
new file mode 100644
... ...
@@ -0,0 +1,96 @@
0
+# $Id$
1
+#
2
+# History:
3
+# --------
4
+# 2006-01-26  created by andrei
5
+
6
+TCP Tunning/monitoring for lots of open connections
7
+
8
+0. Introduction
9
+----------------
10
+
11
+This document describes very briefly various settings that should improve
12
+ ser+tcp performance for sites handling a lot of tcp traffic (> 1000 open
13
+  connections or very high connection/disconnection rates).
14
+For now it deals only with linux specific optimizations.
15
+
16
+
17
+1. Usefull kernel settings
18
+---------------------------
19
+
20
+1.1 connection rate/pending connections: by default the connection rate is
21
+ too small
22
+
23
+net.core.somaxconn             -  limit of the listen() backlog, default 128
24
+net.ipv4.tcp_max_syn_backlog   -  default 1024 or 128
25
+net.ipv4.tcp_timestamps        - default on., should be on (along with 
26
+                                 tcp_tw_recycle and timestamp supporting
27
+                                 peers allows for fast connections rates)
28
+
29
+1.2 connection in close_wait: connection should stay as little as possible
30
+ in close_wait to quickly free the fd/resources for new connections attempts
31
+ WARNING: this could break normal tcp use, use it only if you know what you are
32
+  doing
33
+ 
34
+net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
35
+                              (the default seems to be ok)
36
+net.ipv4.tcp_tw_recycle     - enables fast time wait sockets recycling (default
37
+                              off), should be enabled if you have lots of short
38
+                              lived connections 
39
+                              WARNING: see the above warning
40
+net.ipv4.tcp_tw_reuse       - allows reusing of time-wait sockets (default off)
41
+                              WARNING: see above
42
+
43
+net.ipv4.tcp_syncookies     - default off, in this case it's probably better to
44
+                              keep it off
45
+
46
+1.3 port range
47
+net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)
48
+
49
+1.4  open fds
50
+fs.file-max                 - maximum number of fds that will be allocated
51
+                              (you probably need to increase it, default 
52
+                               depends on installed memory)
53
+
54
+1.5 other sysctl that might affect tcp connection rate or the maximum number
55
+    of open connections
56
+
57
+iptables  - remove the ip_conntrack module (it limits the maximum tcp
58
+ connections, adds extra overhead (slow)). It's probably better to remove
59
+  all the iptables modules.
60
+
61
+net.ipv4.tcp_max_orphans    - might be worth a look if things go wrong
62
+net.core.netdev_max_backlog - maximum device backlog
63
+
64
+ifconfig <dev> txqueuelen <val> - set device transmission queue len
65
+
66
+
67
+2. Monitoring (values to watch for)
68
+-----------------------------------
69
+
70
+2.1 FDs
71
+
72
+fs.dentry-state  - format: nr. dentries, nr. unused, age_limit, want_pages
73
+fs.file-nr       - format: allocated, unused, max (==fs.file-max)
74
+fs.inode-state  -  format: nr. allocated, nr. free, preshrink
75
+
76
+2.2 TCP
77
+
78
+/proc/net/netstat - the TW, TWRecycled, TWKilled, PAWPassive, PAWActive,
79
+                    PASWEstab fields
80
+                   ( cat /proc/net/netstat |cut -d" " -f12-17 ;
81
+                     cat /proc/net/sockstat)
82
+/proc/net/sockstat
83
+
84
+
85
+3. Ser settings
86
+
87
+- don't forget to increase tcp_max_connections and the amount of shared memory
88
+- you should increase the number of ser "tcp_children" processes (-N no)
89
+As a rule of thumb, (maximum simultaneous connections)/2000 should be ok
90
+- you might have to decrease TCP_BUF_SIZE to a smaller value (e.g 8K)
91
+- you might want to increase PKG_MEM_POOL_SIZE (for large queues)
92
+
93
+- you might need to increase the maximum open fds limit before starting ser
94
+ (e.g. ulimit -n 1000000)
95
+