Browse code

Merge remote branch 'origin/andrei/tcp_tls_changes'

Asynchronous TLS support and various TCP and io_wait fixes
(especially on BSDs).

* origin/andrei/tcp_tls_changes: (67 commits)
tls: fix partial write on write-wants-read queue flush
tls: more config vars displayed by the tls.options RPC
tls: fix trailing space in new modparams
tls: verbose debugging for SSL_ERROR_WANT_WRITE
tls: add lib64 to LIBS path
tls: doc - notes about enabling debugging
tls: added debug log level modparam
tls: modparams for ct write queue params
tls: doc - new & async related config options
tls: no tls_bio debugging by default
tls: change read_ahead, buffers and freelist defaults
tcp: verbose and safer close()
tls: enable PARTIAL_WRITE by default
tls: partial SSL_write support when reading (tls_read_f)
tls: don't report SSL protocol errors as bugs
tls: more consistent low memory checking
io_wait: kqueue: use the entire array during too many errors fallback
tcp: fix dispatching closed connections to tcp readers
tcp: more complete error messages
tls: support for partial encoding and reseting send_flags
tcp: support for tls partial encoding
tls: update & fix repeated send & delayed send
tcp: change tls send callback interface
tsend: s/char*/const char*/ in function params.
tls: very verbose debug logging
tls: fix tls_send out-of-mem on new connection
tcp: force eof after read if write side hangup
tcp: don't reset read_flags on RD_CONN_REPEAT_READ
tls: deal with internal openssl buffering
tls: fix initial state error handling
tcp: more consistent IO_FD_CLOSING usage
io_wait: kqueue: use a bigger array
io_wait: kqueue: handle ENOENT and more robust error handling
io_wait: fix kqueue io_wait_add & POLLIN
io_wait: don't update FD watched status on error
io_wait: fix kqueue and too many errors in changelist
io_wait: fix: check for EV_ERROR for kqueue()
tcp: fix fd passing bug
tls: config option for sending close notify alerts
tls: SSL_shutdown() only fully established connections
tls: ssl_flush() fix and re-worked error reporting
tls: tls.list rpc: fix timeout & ip display
tls: fix queue accounting
tls: rpc: tls.list and tls.options update
tls: config options for the internal queues
tls: fix wrong wbio usage
tls: fix empty files treatment
tls: added tls.options rpc
tls: migrated to the runtime cfg framework
db_flatstore: updated get_abs_pathname use
core: get_abs_pathname() uses now pkg_malloc()
core: str.h - s/NULL/0/
tls: doc - removed handshake_timeout and send_timeout
tls: removed handshake_timeout and send_timeout
tls: s/tls_cfg/tls_domains_cfg
tls: added tls_info rpc
tls: fix unregistered rpc commands
tls: async support (major tls core rewrite)
tls: tls_bio ctrl cmd support, fixes and debug
tls: clear text write queue implementation
tls: added a minimum overhead shm buffer queue
tls: safer destroy_cfg
tcp: new tls hooks interface and async tls changes
tls: added custom memory based bio
tcp: minor cleanups & spelling
tcp: tcp_send() split in 3 smaller functions
tcp: comments & new internal command

Andrei Pelinescu-Onciul authored on 16/08/2010 00:18:57
Showing 48 changed files
... ...
@@ -34,6 +34,7 @@ core:
34 34
      compiled, use ser -V |grep --color RAW_SOCKS or for a running
35 35
      ser: sercmd core.udp4_raw_info.
36 36
      See udp4_raw, udp4_raw_mtu and udp4_raw_ttl below.
37
+  - asynchronous TLS support
37 38
   - onreply_route {...} is now equivalent with onreply_route[0] {...}
38 39
   - global, per protocol blacklist ignore masks (via extended send_flags).
39 40
     See dst_blacklist_udp_imask a.s.o (dst_blacklist_*_imask).
... ...
@@ -100,11 +101,39 @@ modules:
100 101
            blst_rpl_clear_ignore(mask): like blst_rpl_ignore(mask), but
101 102
             clears instead of setting.
102 103
    - tls:
103
-           new options for better tuning memory usage for modern openssl
104
-            versions: ssl_release_buffers, ssl_freelist_max_len,
105
-            ssl_max_send_fragment, ssl_read_ahead. For more info see
106
-            modules/doc/tls/README.
107
-           compression is now disabled by default. To enable it set
104
+          asynchronous TLS support
105
+          new TLS RPCs (tls.info, tls.options), tls.list more detailed.
106
+          removed handshake_timeout and send_timeout module parameters /
107
+            config variables. The values from tcp are used instead
108
+            (tcp_connect_timeout and tcp_send_timeout).
109
+          runtime config support
110
+          more config options:
111
+            send_close_notify - enables/disables sending close notify
112
+              alerts prior to closing the corresponding TCP connection.
113
+              Sending the close notify prior to tcp shutdown is "nicer"
114
+              from a TLS point of view, but it has a measurable
115
+              performance impact. Default: off. Can be set at runtime
116
+              (tls.send_close_notify).
117
+            con_ct_wq_max - per connection tls maximum clear text write
118
+              queue size.  The TLS clear-text write queues are used when a
119
+              send attempt has to be delayed due to an on-going TLS level
120
+              renegotiation. Can be set at runtime (tls.con_ct_wq_max).
121
+              Default: 65536 (64 Kb).
122
+            ct_wq_max - maximum total for all the tls clear text write
123
+              queues (summed). Can be set at runtime (tls.ct_wq_max).
124
+              Default: 10485760 (10 Mb).
125
+            ct_wq_blk_size - internal TLS pre-write (clear-text) queue
126
+              minimum block size (advance tunning or debugging).
127
+              Can be set at runtime (tls.ct_wq_blk_size).
128
+              Default: 4096 (4 Kb).
129
+          verbose debug messages can be enable by re-compiling with
130
+            -DTLS_RD_DEBUG (for the read path) and -DTLS_WR_DEBUG
131
+            (for the write path).
132
+          new options for better tuning memory usage for modern openssl
133
+            versions: ssl_release_buffers (default 1), ssl_freelist_max_len
134
+            (default 0), ssl_max_send_fragment, ssl_read_ahead (default 0).
135
+            For more info see modules/doc/tls/README.
136
+          compression is now disabled by default. To enable it set
108 137
             tls_disable_compression to 0, but note that memory usage will
109 138
             increase dramatically especially for large number of
110 139
             connections (>1000).
... ...
@@ -640,7 +640,7 @@ cfg_parser_t* cfg_parser_init(str* filename)
640 640
 		goto error;
641 641
 	}
642 642
 
643
-	free(pathname);
643
+	pkg_free(pathname);
644 644
 
645 645
 	st->file = base;
646 646
 	st->line = 1;
... ...
@@ -653,7 +653,7 @@ cfg_parser_t* cfg_parser_init(str* filename)
653 653
 		pkg_free(st);
654 654
 	}
655 655
 	if (base) pkg_free(base);
656
-	if (pathname) free(pathname);
656
+	if (pathname) pkg_free(pathname);
657 657
 	return NULL;
658 658
 }
659 659
 
... ...
@@ -555,13 +555,21 @@ int init_io_wait(io_wait_h* h, int max_fd, enum poll_types poll_method)
555 555
 #endif
556 556
 #ifdef HAVE_KQUEUE
557 557
 		case POLL_KQUEUE:
558
-			h->kq_array=local_malloc(sizeof(*(h->kq_array))*h->max_fd_no);
558
+			h->kq_changes_size=KQ_CHANGES_ARRAY_SIZE;
559
+			/* kevent returns different events for read & write
560
+			   => to get all the possible events in one call we
561
+			   need twice the number of added fds + space
562
+			   for possible changelist errors.
563
+			   OTOH if memory is to be saved at all costs, one can
564
+			   decrease the array size.
565
+			 */
566
+			h->kq_array_size=2 * h->max_fd_no + h->kq_changes_size;
567
+			h->kq_array=local_malloc(sizeof(*(h->kq_array))*h->kq_array_size);
559 568
 			if (h->kq_array==0){
560 569
 				LOG(L_CRIT, "ERROR: init_io_wait: could not alloc"
561 570
 							" kqueue event array\n");
562 571
 				goto error;
563 572
 			}
564
-			h->kq_changes_size=KQ_CHANGES_ARRAY_SIZE;
565 573
 			h->kq_changes=local_malloc(sizeof(*(h->kq_changes))*
566 574
 										h->kq_changes_size);
567 575
 			if (h->kq_changes==0){
... ...
@@ -570,7 +578,8 @@ int init_io_wait(io_wait_h* h, int max_fd, enum poll_types poll_method)
570 578
 				goto error;
571 579
 			}
572 580
 			h->kq_nchanges=0;
573
-			memset((void*)h->kq_array, 0, sizeof(*(h->kq_array))*h->max_fd_no);
581
+			memset((void*)h->kq_array, 0,
582
+						sizeof(*(h->kq_array))*h->kq_array_size);
574 583
 			memset((void*)h->kq_changes, 0,
575 584
 						sizeof(*(h->kq_changes))* h->kq_changes_size);
576 585
 			if (init_kqueue(h)<0){
... ...
@@ -1,6 +1,6 @@
1
-/* 
1
+/*
2 2
  * $Id$
3
- * 
3
+ *
4 4
  * Copyright (C) 2005 iptelorg GmbH
5 5
  *
6 6
  * Permission to use, copy, modify, and distribute this software for any
... ...
@@ -31,9 +31,9 @@
31 31
  *                 this assumption)
32 32
  *     local_malloc (defaults to pkg_malloc)
33 33
  *     local_free   (defaults to pkg_free)
34
- *  
34
+ *
35 35
  */
36
-/* 
36
+/*
37 37
  * History:
38 38
  * --------
39 39
  *  2005-06-13  created by andrei
... ...
@@ -45,6 +45,7 @@
45 45
  *  2007-11-29  support for write (POLLOUT); added io_watch_chg() (andrei)
46 46
  *  2008-02-04  POLLRDHUP & EPOLLRDHUP support (automatically enabled if POLLIN
47 47
  *               is set) (andrei)
48
+ *  2010-06-17  re-enabled & enhanced the EV_ERROR for kqueue (andrei)
48 49
  */
49 50
 
50 51
 
... ...
@@ -78,8 +79,8 @@
78 79
 #endif
79 80
 #ifdef HAVE_SELECT
80 81
 /* needed on openbsd for select*/
81
-#include <sys/time.h> 
82
-#include <sys/types.h> 
82
+#include <sys/time.h>
83
+#include <sys/types.h>
83 84
 #include <unistd.h>
84 85
 /* needed according to POSIX for select*/
85 86
 #include <sys/select.h>
... ...
@@ -108,7 +109,7 @@ extern int _os_ver; /* os version number, needed to select bugs workarrounds */
108 109
 
109 110
 #if 0
110 111
 enum fd_types; /* this should be defined from the including file,
111
-				  see tcp_main.c for an example, 
112
+				  see tcp_main.c for an example,
112 113
 				  0 has a special meaning: not used/empty*/
113 114
 #endif
114 115
 
... ...
@@ -146,8 +147,10 @@ struct io_wait_handler{
146 147
 	enum poll_types poll_method;
147 148
 	int flags;
148 149
 	struct fd_map* fd_hash;
149
-	int fd_no; /*  current index used in fd_array and the passed size for 
150
-				   ep_array & kq_array*/
150
+	int fd_no; /*  current index used in fd_array and the passed size for
151
+				   ep_array (for kq_array at least
152
+				    max(twice the size, kq_changes_size) should be
153
+				   be passed). */
151 154
 	int max_fd_no; /* maximum fd no, is also the size of fd_array,
152 155
 						       fd_hash  and ep_array*/
153 156
 	/* common stuff for POLL, SIGIO_RT and SELECT
... ...
@@ -169,6 +172,7 @@ struct io_wait_handler{
169 172
 	struct kevent* kq_array;   /* used for the eventlist*/
170 173
 	struct kevent* kq_changes; /* used for the changelist */
171 174
 	size_t kq_nchanges;
175
+	size_t kq_array_size;   /* array size */
172 176
 	size_t kq_changes_size; /* size of the changes array */
173 177
 #endif
174 178
 #ifdef HAVE_DEVPOLL
... ...
@@ -218,7 +222,7 @@ static inline struct fd_map* hash_fd_map(	io_wait_h* h,
218 222
  *          events - combinations of POLLIN, POLLOUT, POLLERR & POLLHUP
219 223
  *          idx    - index in the fd_array (or -1 if not known)
220 224
  * return: -1 on error
221
- *          0 on EAGAIN or when by some other way it is known that no more 
225
+ *          0 on EAGAIN or when by some other way it is known that no more
222 226
  *            io events are queued on the fd (the receive buffer is empty).
223 227
  *            Usefull to detect when there are no more io events queued for
224 228
  *            sigio_rt, epoll_et, kqueue.
... ...
@@ -242,10 +246,11 @@ int handle_io(struct fd_map* fm, short events, int idx);
242 246
  *       and EVFILT_WRITE, EV_ADD for the same fd).
243 247
  * returns: -1 on error, 0 on success
244 248
  */
245
-static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag, 
249
+static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag,
246 250
 								void* data)
247 251
 {
248 252
 	int n;
253
+	int r;
249 254
 	struct timespec tspec;
250 255
 
251 256
 	if (h->kq_nchanges>=h->kq_changes_size){
... ...
@@ -256,11 +261,35 @@ static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag,
256 261
 		tspec.tv_nsec=0;
257 262
 again:
258 263
 		n=kevent(h->kq_fd, h->kq_changes, h->kq_nchanges, 0, 0, &tspec);
259
-		if (n==-1){
260
-			if (errno==EINTR) goto again;
261
-			LOG(L_ERR, "ERROR: io_watch_add: kevent flush changes "
262
-						" failed: %s [%d]\n", strerror(errno), errno);
263
-			return -1;
264
+		if (unlikely(n == -1)){
265
+			if (unlikely(errno == EINTR)) goto again;
266
+			else {
267
+				/* for a detailed explanation of what follows see
268
+				   io_wait_loop_kqueue EV_ERROR case */
269
+				if (unlikely(!(errno == EBADF || errno == ENOENT)))
270
+					BUG("kq_ev_change: kevent flush changes failed"
271
+							" (unexpected error): %s [%d]\n",
272
+							strerror(errno), errno);
273
+					/* ignore error even if it's not a EBADF/ENOENT */
274
+				/* one of the file descriptors is bad, probably already
275
+				   closed => try to apply changes one-by-one */
276
+				for (r = 0; r < h->kq_nchanges; r++) {
277
+retry2:
278
+					n = kevent(h->kq_fd, &h->kq_changes[r], 1, 0, 0, &tspec);
279
+					if (n==-1) {
280
+						if (unlikely(errno == EINTR))
281
+							goto retry2;
282
+					/* for a detailed explanation of what follows see
283
+						io_wait_loop_kqueue EV_ERROR case */
284
+						if (unlikely(!(errno == EBADF || errno == ENOENT)))
285
+							BUG("kq_ev_change: kevent flush changes failed:"
286
+									" (unexpected error) %s [%d] (%d/%d)\n",
287
+										strerror(errno), errno,
288
+										r, h->kq_nchanges);
289
+						continue; /* skip over it */
290
+					}
291
+				}
292
+			}
264 293
 		}
265 294
 		h->kq_nchanges=0; /* changes array is empty */
266 295
 	}
... ...
@@ -395,7 +424,7 @@ inline static int io_watch_add(	io_wait_h* h,
395 424
 #ifdef HAVE_SIGIO_RT
396 425
 		case POLL_SIGIO_RT:
397 426
 			fd_array_setup(events);
398
-			/* re-set O_ASYNC might be needed, if not done from 
427
+			/* re-set O_ASYNC might be needed, if not done from
399 428
 			 * io_watch_del (or if somebody wants to add a fd which has
400 429
 			 * already O_ASYNC/F_SETSIG set on a duplicate)
401 430
 			 */
... ...
@@ -472,7 +501,7 @@ again2:
472 501
 		case POLL_KQUEUE:
473 502
 			if (likely( events & POLLIN)){
474 503
 				if (unlikely(kq_ev_change(h, fd, EVFILT_READ, EV_ADD, e)==-1))
475
-				goto error;
504
+					goto error;
476 505
 			}
477 506
 			if (unlikely( events & POLLOUT)){
478 507
 				if (unlikely(kq_ev_change(h, fd, EVFILT_WRITE, EV_ADD, e)==-1))
... ...
@@ -480,8 +509,8 @@ again2:
480 509
 					if (likely(events & POLLIN)){
481 510
 						kq_ev_change(h, fd, EVFILT_READ, EV_DELETE, 0);
482 511
 					}
512
+					goto error;
483 513
 				}
484
-				goto error;
485 514
 			}
486 515
 			break;
487 516
 #endif
... ...
@@ -516,7 +545,7 @@ again_devpoll:
516 545
 		pf.events=events;
517 546
 check_io_again:
518 547
 		n=0;
519
-		while(e->type && ((n=poll(&pf, 1, 0))>0) && 
548
+		while(e->type && ((n=poll(&pf, 1, 0))>0) &&
520 549
 				(handle_io(e, pf.revents, idx)>0) &&
521 550
 				(pf.revents & (e->events|POLLERR|POLLHUP)));
522 551
 		if (unlikely(e->type && (n==-1))){
... ...
@@ -531,20 +560,20 @@ error:
531 560
 	if (e) unhash_fd_map(e);
532 561
 	return -1;
533 562
 #undef fd_array_setup
534
-#undef set_fd_flags 
563
+#undef set_fd_flags
535 564
 }
536 565
 
537 566
 
538 567
 
539 568
 #define IO_FD_CLOSING 16
540
-/* parameters:    h - handler 
569
+/* parameters:    h - handler
541 570
  *               fd - file descriptor
542 571
  *            index - index in the fd_array if known, -1 if not
543 572
  *                    (if index==-1 fd_array will be searched for the
544
- *                     corresponding fd* entry -- slower but unavoidable in 
573
+ *                     corresponding fd* entry -- slower but unavoidable in
545 574
  *                     some cases). index is not used (no fd_array) for epoll,
546 575
  *                     /dev/poll and kqueue
547
- *            flags - optimization flags, e.g. IO_FD_CLOSING, the fd was 
576
+ *            flags - optimization flags, e.g. IO_FD_CLOSING, the fd was
548 577
  *                    or will shortly be closed, in some cases we can avoid
549 578
  *                    extra remove operations (e.g.: epoll, kqueue, sigio)
550 579
  * returns 0 if ok, -1 on error */
... ...
@@ -600,7 +629,6 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
600 629
 		goto error;
601 630
 	}
602 631
 	events=e->events;
603
-	unhash_fd_map(e);
604 632
 	
605 633
 	switch(h->poll_method){
606 634
 		case POLL_POLL:
... ...
@@ -614,13 +642,12 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
614 642
 				FD_CLR(fd, &h->master_wset);
615 643
 			if (unlikely(h->max_fd_select && (h->max_fd_select==fd)))
616 644
 				/* we don't know the prev. max, so we just decrement it */
617
-				h->max_fd_select--; 
645
+				h->max_fd_select--;
618 646
 			fix_fd_array;
619 647
 			break;
620 648
 #endif
621 649
 #ifdef HAVE_SIGIO_RT
622 650
 		case POLL_SIGIO_RT:
623
-			fix_fd_array;
624 651
 			/* the O_ASYNC flag must be reset all the time, the fd
625 652
 			 *  can be changed only if  O_ASYNC is reset (if not and
626 653
 			 *  the fd is a duplicate, you will get signals from the dup. fd
... ...
@@ -629,17 +656,18 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
629 656
 			 */
630 657
 			/*if (!(flags & IO_FD_CLOSING)){*/
631 658
 				/* reset ASYNC */
632
-				fd_flags=fcntl(fd, F_GETFL); 
633
-				if (unlikely(fd_flags==-1)){ 
634
-					LOG(L_ERR, "ERROR: io_watch_del: fnctl: GETFL failed:" 
635
-							" %s [%d]\n", strerror(errno), errno); 
636
-					goto error; 
637
-				} 
638
-				if (unlikely(fcntl(fd, F_SETFL, fd_flags&(~O_ASYNC))==-1)){ 
639
-					LOG(L_ERR, "ERROR: io_watch_del: fnctl: SETFL" 
640
-								" failed: %s [%d]\n", strerror(errno), errno); 
641
-					goto error; 
642
-				} 
659
+				fd_flags=fcntl(fd, F_GETFL);
660
+				if (unlikely(fd_flags==-1)){
661
+					LOG(L_ERR, "ERROR: io_watch_del: fnctl: GETFL failed:"
662
+							" %s [%d]\n", strerror(errno), errno);
663
+					goto error;
664
+				}
665
+				if (unlikely(fcntl(fd, F_SETFL, fd_flags&(~O_ASYNC))==-1)){
666
+					LOG(L_ERR, "ERROR: io_watch_del: fnctl: SETFL"
667
+								" failed: %s [%d]\n", strerror(errno), errno);
668
+					goto error;
669
+				}
670
+			fix_fd_array; /* only on success */
643 671
 			break;
644 672
 #endif
645 673
 #ifdef HAVE_EPOLL
... ...
@@ -648,7 +676,7 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
648 676
 			/* epoll doesn't seem to automatically remove sockets,
649 677
 			 * if the socket is a duplicate/moved and the original
650 678
 			 * is still open. The fd is removed from the epoll set
651
-			 * only when the original (and all the  copies?) is/are 
679
+			 * only when the original (and all the  copies?) is/are
652 680
 			 * closed. This is probably a bug in epoll. --andrei */
653 681
 #ifdef EPOLL_NO_CLOSE_BUG
654 682
 			if (!(flags & IO_FD_CLOSING)){
... ...
@@ -698,7 +726,7 @@ again_devpoll:
698 726
 				if (write(h->dpoll_fd, &pfd, sizeof(pfd))==-1){
699 727
 					if (errno==EINTR) goto again_devpoll;
700 728
 					LOG(L_ERR, "ERROR: io_watch_del: removing fd from "
701
-								"/dev/poll failed: %s [%d]\n", 
729
+								"/dev/poll failed: %s [%d]\n",
702 730
 								strerror(errno), errno);
703 731
 					goto error;
704 732
 				}
... ...
@@ -706,10 +734,11 @@ again_devpoll:
706 734
 #endif
707 735
 		default:
708 736
 			LOG(L_CRIT, "BUG: io_watch_del: no support for poll method "
709
-					" %s (%d)\n", poll_method_str[h->poll_method], 
737
+					" %s (%d)\n", poll_method_str[h->poll_method],
710 738
 					h->poll_method);
711 739
 			goto error;
712 740
 	}
741
+	unhash_fd_map(e); /* only on success */
713 742
 	h->fd_no--;
714 743
 	return 0;
715 744
 error:
... ...
@@ -719,12 +748,12 @@ error:
719 748
 
720 749
 
721 750
 
722
-/* parameters:    h - handler 
751
+/* parameters:    h - handler
723 752
  *               fd - file descriptor
724 753
  *           events - new events to watch for
725 754
  *              idx - index in the fd_array if known, -1 if not
726 755
  *                    (if index==-1 fd_array will be searched for the
727
- *                     corresponding fd* entry -- slower but unavoidable in 
756
+ *                     corresponding fd* entry -- slower but unavoidable in
728 757
  *                     some cases). index is not used (no fd_array) for epoll,
729 758
  *                     /dev/poll and kqueue
730 759
  * returns 0 if ok, -1 on error */
... ...
@@ -781,14 +810,14 @@ inline static int io_watch_chg(io_wait_h* h, int fd, short events, int idx )
781 810
 	
782 811
 	add_events=events & ~e->events;
783 812
 	del_events=e->events & ~events;
784
-	e->events=events;
785 813
 	switch(h->poll_method){
786 814
 		case POLL_POLL:
815
+			fd_array_chg(events
787 816
 #ifdef POLLRDHUP
788
-			/* listen to POLLRDHUP by default (if POLLIN) */
789
-			events|=((int)!(events & POLLIN) - 1) & POLLRDHUP;
817
+							/* listen to POLLRDHUP by default (if POLLIN) */
818
+							| (((int)!(events & POLLIN) - 1) & POLLRDHUP)
790 819
 #endif /* POLLRDHUP */
791
-			fd_array_chg(events);
820
+						);
792 821
 			break;
793 822
 #ifdef HAVE_SELECT
794 823
 		case POLL_SELECT:
... ...
@@ -882,7 +911,7 @@ again_devpoll1:
882 911
 				if (unlikely(write(h->dpoll_fd, &pfd, sizeof(pfd))==-1)){
883 912
 					if (errno==EINTR) goto again_devpoll1;
884 913
 					LOG(L_ERR, "ERROR: io_watch_chg: removing fd from "
885
-								"/dev/poll failed: %s [%d]\n", 
914
+								"/dev/poll failed: %s [%d]\n",
886 915
 								strerror(errno), errno);
887 916
 					goto error;
888 917
 				}
... ...
@@ -892,18 +921,21 @@ again_devpoll2:
892 921
 				if (unlikely(write(h->dpoll_fd, &pfd, sizeof(pfd))==-1)){
893 922
 					if (errno==EINTR) goto again_devpoll2;
894 923
 					LOG(L_ERR, "ERROR: io_watch_chg: re-adding fd to "
895
-								"/dev/poll failed: %s [%d]\n", 
924
+								"/dev/poll failed: %s [%d]\n",
896 925
 								strerror(errno), errno);
926
+					/* error re-adding the fd => mark it as removed/unhash */
927
+					unhash_fd_map(e);
897 928
 					goto error;
898 929
 				}
899 930
 				break;
900 931
 #endif
901 932
 		default:
902 933
 			LOG(L_CRIT, "BUG: io_watch_chg: no support for poll method "
903
-					" %s (%d)\n", poll_method_str[h->poll_method], 
934
+					" %s (%d)\n", poll_method_str[h->poll_method],
904 935
 					h->poll_method);
905 936
 			goto error;
906 937
 	}
938
+	e->events=events; /* only on success */
907 939
 	return 0;
908 940
 error:
909 941
 	return -1;
... ...
@@ -912,7 +944,7 @@ error:
912 944
 
913 945
 
914 946
 
915
-/* io_wait_loop_x style function 
947
+/* io_wait_loop_x style function.
916 948
  * wait for io using poll()
917 949
  * params: h      - io_wait handle
918 950
  *         t      - timeout in s
... ...
@@ -953,11 +985,11 @@ again:
953 985
 				/* repeat handle_io if repeat, fd still watched (not deleted
954 986
 				 *  inside handle_io), handle_io returns that there's still
955 987
 				 *  IO and the fd is still watched for the triggering event */
956
-				while(fm->type && 
988
+				while(fm->type &&
957 989
 						(handle_io(fm, h->fd_array[r].revents, r) > 0) &&
958 990
 						repeat && ((fm->events|POLLERR|POLLHUP) &
959 991
 													h->fd_array[r].revents));
960
-				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd) 
992
+				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd)
961 993
 										  array shifting */
962 994
 			}
963 995
 		}
... ...
@@ -1002,9 +1034,9 @@ again:
1002 1034
 			if (unlikely(revents)){
1003 1035
 				h->crt_fd_array_idx=r;
1004 1036
 				fm=get_fd_map(h, h->fd_array[r].fd);
1005
-				while(fm->type && (fm->events & revents) && 
1037
+				while(fm->type && (fm->events & revents) &&
1006 1038
 						(handle_io(fm, revents, r)>0) && repeat);
1007
-				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd) 
1039
+				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd)
1008 1040
 										  array shifting */
1009 1041
 				n--;
1010 1042
 			}
... ...
@@ -1028,7 +1060,7 @@ again:
1028 1060
 			if (errno==EINTR) goto again; /* signal, ignore it */
1029 1061
 			else{
1030 1062
 				LOG(L_ERR, "ERROR:io_wait_loop_epoll: "
1031
-						"epoll_wait(%d, %p, %d, %d): %s [%d]\n", 
1063
+						"epoll_wait(%d, %p, %d, %d): %s [%d]\n",
1032 1064
 						h->epfd, h->ep_array, h->fd_no, t*1000,
1033 1065
 						strerror(errno), errno);
1034 1066
 				goto error;
... ...
@@ -1054,7 +1086,7 @@ again:
1054 1086
 					;
1055 1087
 			if (likely(revents)){
1056 1088
 				fm=(struct fd_map*)h->ep_array[r].data.ptr;
1057
-				while(fm->type && ((fm->events|POLLERR|POLLHUP) & revents) && 
1089
+				while(fm->type && ((fm->events|POLLERR|POLLHUP) & revents) &&
1058 1090
 						(handle_io(fm, revents, -1)>0) && repeat);
1059 1091
 			}else{
1060 1092
 				LOG(L_ERR, "ERROR:io_wait_loop_epoll: unexpected event %x"
... ...
@@ -1075,55 +1107,123 @@ inline static int io_wait_loop_kqueue(io_wait_h* h, int t, int repeat)
1075 1107
 	int n, r;
1076 1108
 	struct timespec tspec;
1077 1109
 	struct fd_map* fm;
1110
+	int orig_changes;
1111
+	int apply_changes;
1078 1112
 	int revents;
1079 1113
 	
1080 1114
 	tspec.tv_sec=t;
1081 1115
 	tspec.tv_nsec=0;
1116
+	orig_changes=h->kq_nchanges;
1117
+	apply_changes=orig_changes;
1118
+	do {
1082 1119
 again:
1083
-		n=kevent(h->kq_fd, h->kq_changes, h->kq_nchanges,  h->kq_array,
1084
-					h->fd_no, &tspec);
1120
+		n=kevent(h->kq_fd, h->kq_changes, apply_changes,  h->kq_array,
1121
+					h->kq_array_size, &tspec);
1085 1122
 		if (unlikely(n==-1)){
1086
-			if (errno==EINTR) goto again; /* signal, ignore it */
1087
-			else{
1088
-				LOG(L_ERR, "ERROR: io_wait_loop_kqueue: kevent:"
1123
+			if (unlikely(errno==EINTR)) goto again; /* signal, ignore it */
1124
+			else {
1125
+				/* for a detailed explanation of what follows see below
1126
+				   the EV_ERROR case */
1127
+				if (unlikely(!(errno==EBADF || errno==ENOENT)))
1128
+					BUG("io_wait_loop_kqueue: kevent: unexpected error"
1089 1129
 						" %s [%d]\n", strerror(errno), errno);
1090
-				goto error;
1130
+				/* some of the FDs in kq_changes are bad (already closed)
1131
+				   and there is not enough space in kq_array to return all
1132
+				   of them back */
1133
+				apply_changes = h->kq_array_size;
1134
+				goto again;
1091 1135
 			}
1092 1136
 		}
1093
-		h->kq_nchanges=0; /* reset changes array */
1137
+		/* remove applied changes */
1138
+		h->kq_nchanges -= apply_changes;
1139
+		if (unlikely(apply_changes < orig_changes)) {
1140
+			orig_changes -= apply_changes;
1141
+			memmove(&h->kq_changes[0], &h->kq_changes[apply_changes],
1142
+									sizeof(h->kq_changes[0])*h->kq_nchanges);
1143
+			apply_changes = (orig_changes < h->kq_array_size) ? orig_changes :
1144
+								h->kq_array_size;
1145
+		} else {
1146
+			orig_changes = 0;
1147
+			apply_changes = 0;
1148
+		}
1094 1149
 		for (r=0; r<n; r++){
1095 1150
 #ifdef EXTRA_DEBUG
1096 1151
 			DBG("DBG: kqueue: event %d/%d: fd=%d, udata=%lx, flags=0x%x\n",
1097 1152
 					r, n, h->kq_array[r].ident, (long)h->kq_array[r].udata,
1098 1153
 					h->kq_array[r].flags);
1099 1154
 #endif
1100
-#if 0
1101
-			if (unlikely(h->kq_array[r].flags & EV_ERROR)){
1102
-				/* error in changes: we ignore it, it can be caused by
1103
-				   trying to remove an already closed fd: race between
1104
-				   adding something to the changes array, close() and
1105
-				   applying the changes */
1106
-				LOG(L_INFO, "INFO: io_wait_loop_kqueue: kevent error on "
1107
-							"fd %ld: %s [%ld]\n", h->kq_array[r].ident,
1155
+			if (unlikely((h->kq_array[r].flags & EV_ERROR) ||
1156
+							 h->kq_array[r].udata == 0)){
1157
+				/* error in changes: we ignore it if it has to do with a
1158
+				   bad fd or update==0. It can be caused by trying to remove an
1159
+				   already closed fd: race between adding something to the
1160
+				   changes array, close() and applying the changes (EBADF).
1161
+				   E.g. for ser tcp: tcp_main sends a fd to child for reading
1162
+				    => deletes it from the watched fds => the changes array
1163
+					will contain an EV_DELETE for it. Before the changes
1164
+					are applied (they are at the end of the main io_wait loop,
1165
+					after all the fd events were processed), a CON_ERR sent
1166
+					to tcp_main by a sender (send fail) is processed and causes
1167
+					the fd to be closed. When the changes are applied =>
1168
+					error for the EV_DELETE attempt of a closed fd.
1169
+					Something similar can happen when a fd is scheduled
1170
+					for removal, is close()'ed before being removed and
1171
+					re-opened(a new sock. get the same fd). When the
1172
+					watched fd changes will be applied the fd will be valid
1173
+					(so no EBADF), but it's not already watch => ENOENT.
1174
+					We report a BUG for the other errors (there's nothing
1175
+					constructive we can do if we get an error we don't know
1176
+					how to handle), but apart from that we ignore it in the
1177
+					idea that it is better apply the rest of the changes,
1178
+					rather then dropping all of them.
1179
+				*/
1180
+				/*
1181
+					example EV_ERROR for trying to delete a read watched fd,
1182
+					that was already closed:
1183
+					{
1184
+						ident = 63,  [fd]
1185
+						filter = -1, [EVFILT_READ]
1186
+						flags = 16384, [EV_ERROR]
1187
+						fflags = 0,
1188
+						data = 9, [errno = EBADF]
1189
+						udata = 0x0
1190
+					}
1191
+				*/
1192
+				if (h->kq_array[r].data != EBADF &&
1193
+						h->kq_array[r].data != ENOENT)
1194
+					BUG("io_wait_loop_kqueue: kevent unexpected error on "
1195
+							"fd %ld udata %lx: %s [%ld]\n",
1196
+							(long)h->kq_array[r].ident,
1197
+							(long)h->kq_array[r].udata,
1108 1198
 							strerror(h->kq_array[r].data),
1109 1199
 							(long)h->kq_array[r].data);
1110
-			}else{ 
1111
-#endif
1200
+			}else{
1112 1201
 				fm=(struct fd_map*)h->kq_array[r].udata;
1113 1202
 				if (likely(h->kq_array[r].filter==EVFILT_READ)){
1114
-					revents=POLLIN | 
1115
-						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP);
1116
-					while(fm->type && (fm->events & revents) && 
1203
+					revents=POLLIN |
1204
+						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP) |
1205
+						(((int)!((h->kq_array[r].flags & EV_EOF) &&
1206
+								 	h->kq_array[r].fflags != 0) - 1)&POLLERR);
1207
+					while(fm->type && (fm->events & revents) &&
1117 1208
 							(handle_io(fm, revents, -1)>0) && repeat);
1118 1209
 				}else if (h->kq_array[r].filter==EVFILT_WRITE){
1119
-					revents=POLLOUT | 
1120
-						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP);
1121
-					while(fm->type && (fm->events & revents) && 
1210
+					revents=POLLOUT |
1211
+						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP) |
1212
+						(((int)!((h->kq_array[r].flags & EV_EOF) &&
1213
+								 	h->kq_array[r].fflags != 0) - 1)&POLLERR);
1214
+					while(fm->type && (fm->events & revents) &&
1122 1215
 							(handle_io(fm, revents, -1)>0) && repeat);
1216
+				}else{
1217
+					BUG("io_wait_loop_kqueue: unknown filter: kqueue: event "
1218
+							"%d/%d: fd=%d, filter=%d, flags=0x%x, fflags=0x%x,"
1219
+							" data=%lx, udata=%lx\n",
1220
+					r, n, h->kq_array[r].ident, h->kq_array[r].filter,
1221
+					h->kq_array[r].flags, h->kq_array[r].fflags,
1222
+					(long)h->kq_array[r].data, (long)h->kq_array[r].udata);
1123 1223
 				}
1124
-			/*} */
1224
+			}
1125 1225
 		}
1126
-error:
1226
+	} while(unlikely(orig_changes));
1127 1227
 	return n;
1128 1228
 }
1129 1229
 #endif
... ...
@@ -1207,14 +1307,14 @@ again:
1207 1307
 			 *  POLLIN|POLLRDNORM|POLLMSG (=POLL_MSG),
1208 1308
 			 *  POLLERR (=POLL_ERR),
1209 1309
 			 *  POLLPRI|POLLRDBAND (=POLL_PRI),
1210
-			 *  POLLHUP|POLLERR (=POLL_HUP) 
1310
+			 *  POLLHUP|POLLERR (=POLL_HUP)
1211 1311
 			 *  [linux 2.6.22 fs/fcntl.c:447]
1212 1312
 			 */
1213 1313
 #ifdef EXTRA_DEBUG
1214 1314
 			DBG("io_wait_loop_sigio_rt: siginfo: signal=%d (%d),"
1215 1315
 					" si_code=%d, si_band=0x%x,"
1216 1316
 					" si_fd=%d\n",
1217
-					siginfo.si_signo, n, siginfo.si_code, 
1317
+					siginfo.si_signo, n, siginfo.si_code,
1218 1318
 					(unsigned)sigio_band,
1219 1319
 					sigio_fd);
1220 1320
 #endif
... ...
@@ -1227,7 +1327,7 @@ again:
1227 1327
 				/* fix revents==POLLPRI case */
1228 1328
 				revents |= (!(revents & POLLPRI)-1) & POLLIN;
1229 1329
 				/* we can have queued signals generated by fds not watched
1230
-			 	 * any more, or by fds in transition, to a child 
1330
+			 	 * any more, or by fds in transition, to a child
1231 1331
 				 * => ignore them */
1232 1332
 				if (fm->type && ((fm->events|POLLERR|POLLHUP) & revents))
1233 1333
 					handle_io(fm, revents, -1);
... ...
@@ -1243,7 +1343,7 @@ again:
1243 1343
 			}
1244 1344
 		}
1245 1345
 	}else{
1246
-		/* signal queue overflow 
1346
+		/* signal queue overflow
1247 1347
 		 * TODO: increase signal queue size: 2.4x /proc/.., 2.6x -rlimits */
1248 1348
 		LOG(L_WARN, "WARNING: io_wait_loop_sigio_rt: signal queue overflowed"
1249 1349
 					"- falling back to poll\n");
... ...
@@ -70,7 +70,7 @@ int flat_uri(db_uri_t* uri)
70 70
 
71 71
  error:
72 72
 	if (furi) {
73
-		if (furi->path.s) free(furi->path.s);
73
+		if (furi->path.s) pkg_free(furi->path.s);
74 74
 		db_drv_free(&furi->drv);
75 75
 		pkg_free(furi);
76 76
 	}
... ...
@@ -10,8 +10,10 @@ auto_gen=
10 10
 NAME=tls.so
11 11
 
12 12
 DEFS+= -I$(LOCALBASE)/ssl/include
13
-LIBS+= -L$(LOCALBASE)/lib -L$(LOCALBASE)/ssl/lib -lssl  -lcrypto \
14
-	$(TLS_EXTRA_LIBS)
13
+LIBS+=	-L$(LOCALBASE)/lib -L$(LOCALBASE)/ssl/lib \
14
+		-L$(LOCALBASE)/lib64 -L$(LOCALBASE)/ssl/lib64 \
15
+		-lssl  -lcrypto \
16
+		$(TLS_EXTRA_LIBS)
15 17
 # NOTE: depending on the way in which libssl was compiled you might
16 18
 #       have to add -lz -lkrb5   (zlib and kerberos5).
17 19
 #       E.g.: make TLS_HOOKS=1 TLS_EXTRA_LIBS="-lz -lkrb5"
... ...
@@ -14,42 +14,48 @@ Andrei Pelinescu-Onciul
14 14
    1.3. Important Notes
15 15
    1.4. Compiling the TLS Module
16 16
    1.5. TLS and Low Memory
17
-   1.6. Known Limitations
18
-   1.7. Quick Certificate Howto
19
-   1.8. Parameters
20
-
21
-        1.8.1. tls_method (string)
22
-        1.8.2. certificate (string)
23
-        1.8.3. private_key (string)
24
-        1.8.4. ca_list (string)
25
-        1.8.5. verify_certificate (boolean)
26
-        1.8.6. verify_depth (integer)
27
-        1.8.7. require_certificate (boolean)
28
-        1.8.8. cipher_list (string)
29
-        1.8.9. send_timeout (int)
30
-        1.8.10. handshake_timeout (int)
31
-        1.8.11. connection_timeout (int)
32
-        1.8.12. tls_disable_compression (boolean)
33
-        1.8.13. ssl_release_buffers (integer)
34
-        1.8.14. ssl_free_list_max_len (integer)
35
-        1.8.15. ssl_max_send_fragment (integer)
36
-        1.8.16. ssl_read_ahead (boolean)
37
-        1.8.17. tls_log (int)
38
-        1.8.18. low_mem_threshold1 (integer)
39
-        1.8.19. low_mem_threshold2 (integer)
40
-        1.8.20. tls_force_run (boolean)
41
-        1.8.21. config (string)
42
-
43
-   1.9. Functions
44
-
45
-        1.9.1. is_peer_verified()
46
-
47
-   1.10. History
17
+   1.6. TLS Debugging
18
+   1.7. Known Limitations
19
+   1.8. Quick Certificate Howto
20
+   1.9. Parameters
21
+
22
+        1.9.1. tls_method (string)
23
+        1.9.2. certificate (string)
24
+        1.9.3. private_key (string)
25
+        1.9.4. ca_list (string)
26
+        1.9.5. verify_certificate (boolean)
27
+        1.9.6. verify_depth (integer)
28
+        1.9.7. require_certificate (boolean)
29
+        1.9.8. cipher_list (string)
30
+        1.9.9. send_timeout (int)
31
+        1.9.10. handshake_timeout (int)
32
+        1.9.11. connection_timeout (int)
33
+        1.9.12. tls_disable_compression (boolean)
34
+        1.9.13. ssl_release_buffers (integer)
35
+        1.9.14. ssl_free_list_max_len (integer)
36
+        1.9.15. ssl_max_send_fragment (integer)
37
+        1.9.16. ssl_read_ahead (boolean)
38
+        1.9.17. send_close_notify (boolean)
39
+        1.9.18. con_ct_wq_max (integer)
40
+        1.9.19. ct_wq_max (integer)
41
+        1.9.20. ct_wq_blk_size (integer)
42
+        1.9.21. tls_log (int)
43
+        1.9.22. tls_debug (int)
44
+        1.9.23. low_mem_threshold1 (integer)
45
+        1.9.24. low_mem_threshold2 (integer)
46
+        1.9.25. tls_force_run (boolean)
47
+        1.9.26. config (string)
48
+
49
+   1.10. Functions
50
+
51
+        1.10.1. is_peer_verified()
52
+
53
+   1.11. History
48 54
 
49 55
 1.1. Overview
50 56
 
51 57
    This module implements the TLS transport for SIP-router using the
52
-   Openssl library (http://www.openssl.org). To enable the TLS support
58
+   OpenSSL library (http://www.openssl.org). To enable the TLS support
53 59
    this module must be loaded and enable_tls=yes must be added to the
54 60
    SIP-router config file
55 61
 
... ...
@@ -103,12 +109,15 @@ route{
103 109
    significantly slow down the TLS connection handshake, thus limiting the
104 110
    maximum SIP-router TLS connection rate.
105 111
 
106
-   Compression is fully supported and used by default, if you have a new
107
-   enough Openssl version (starting with 0.9.8). Although there are some
108
-   problems with zlib compression in currently deployed Openssl versions
109
-   (up to and including 0.9.8d, see openssl bug #1468), the TLS module
110
-   will automatically switch to its own fixed version. There's no need to
111
-   force-disable the compression.
112
+   Compression is fully supported if you have a new enough Openssl version
113
+   (starting with 0.9.8). Although there are some problems with zlib
114
+   compression in currently deployed Openssl versions (up to and including
115
+   0.9.8d, see openssl bug #1468), the TLS module will automatically
116
+   switch to its own fixed version. Note however that starting with sr 3.1
117
+   compression is not enabled by default, due to the huge extra memory
118
+   consumption that it causes (about 10x more memory). To enable it use
119
+   modparam("tls", "tls_disable_compression", 0) (see
120
+   tls_disable_compression).
112 121
 
113 122
    The TLS module includes workarounds for the following known openssl
114 123
    bugs: openssl #1204 (disable SS_OP_TLS_BLOCK_PADDING_BUG if compression
... ...
@@ -122,11 +131,10 @@ route{
122 131
 1.4. Compiling the TLS Module
123 132
 
124 133
    In most case compiling the TLS module is as simple as:
125
-make modules modules=modules/tls
134
+make -C modules/tls
126 135
 
127 136
    or
128
-cd modules/tls
129
-make
137
+make modules modules=modules/tls
130 138
 
131 139
    or (compiling whole SIP-router and the tls module)
132 140
 make all include_modules=tls
... ...
@@ -157,7 +165,21 @@ make TLS_EXTRA_LIBS="-lkrb5 -lz" all include_modules=tls
157 165
    reduce openssl memory usage it to disable compression (see
158 166
    tls_disable_compression).
159 167
 
160
-1.6. Known Limitations
168
+1.6. TLS Debugging
169
+
170
+   Debugging messages can be selectively enabled by recompiling the tls
171
+   module with a combination of the following defines:
172
+     * TLS_WR_DEBUG - debug messages for the write/send part.
173
+     * TLS_RD_DEBUG - debug messages for the read/receive part.
174
+     * TLS_BIO_DEBUG - debug messages for the custom BIO.
175
+
176
+   Example 2. Compiling TLS with Debug Messages
177
+make -C modules/tls extra_defs="-DTLS_WR_DEBUG -DTLS_RD_DEBUG"
178
+
179
+   To change the level at which the debug messages are logged, change the
180
+   tls_debug module parameter.
181
+
182
+1.7. Known Limitations
161 183
 
162 184
    The private key must not encrypted (SIP-router cannot ask you for a
163 185
    password on startup).
... ...
@@ -173,10 +195,16 @@ make TLS_EXTRA_LIBS="-lkrb5 -lz" all include_modules=tls
173 195
    TLS specific config reloading is not safe, so for now better don't use
174 196
    it, especially under heavy traffic.
175 197
 
176
-   This documentation is incomplete. The select framework and rpc sections
177
-   are completely missing.
198
+   This documentation is incomplete. The RPCs are not documented here, but
199
+   in doc/rpc_list/rpc_tls.txt or
200
+   http://sip-router.org/docbook/sip-router/branch/master/rpc_list/rpc_lis
201
+   t.html#rpc_exports.tls. The provided selects are not documented. A list
202
+   with all the ones implemented by the tls module can be seen under
203
+   doc/select_list/select_tls.txt or or
204
+   http://sip-router.org/docbook/sip-router/branch/master/select_list/sele
205
+   ct_list.html#select_list.tls.
178 206
 
179
-1.7. Quick Certificate Howto
207
+1.8. Quick Certificate Howto
180 208
 
181 209
    Revision History
182 210
    Revision $Revision$ $Date$
... ...
@@ -260,12 +288,12 @@ fg:
260 288
                 modparam("tls", "require_certificate", 1)
261 289
         (for more information see the module parameters documentation)
262 290
 
263
-1.8. Parameters
291
+1.9. Parameters
264 292
 
265 293
    Revision History
266 294
    Revision $Revision$ $Date$
267 295
 
268
-1.8.1. tls_method (string)
296
+1.9.1. tls_method (string)
269 297
 
270 298
    Sets the SSL/TLS protocol method. Possible values are:
271 299
      * TLSv1 - only TLSv1 connections are accepted. This is the default
... ...
@@ -283,12 +311,12 @@ fg:
283 311
    If rfc3261 conformance is desired, TLSv1 must be used. For
284 312
    compatibility with older clients SSLv23 is a good option.
285 313
 
286
-   Example 2. Set tls_method parameter
314
+   Example 3. Set tls_method parameter
287 315
 ...
288 316
 modparam("tls", "tls_method", "TLSv1")
289 317
 ...
290 318
 
291
-1.8.2. certificate (string)
319
+1.9.2. certificate (string)
292 320
 
293 321
    Sets the certificate file name. The certificate file can also contain
294 322
    the private key in PEM format.
... ...
@@ -299,12 +327,12 @@ modparam("tls", "tls_method", "TLSv1")
299 327
 
300 328
    The default value is [SER_CFG_DIR]/cert.pem.
301 329
 
302
-   Example 3. Set certificate parameter
330
+   Example 4. Set certificate parameter
303 331
 ...
304 332
 modparam("tls", "certificate", "/usr/local/etc/ser/my_certificate.pem")
305 333
 ...
306 334
 
307
-1.8.3. private_key (string)
335
+1.9.3. private_key (string)
308 336
 
309 337
    Sets the private key file name.
310 338
 
... ...
@@ -314,12 +342,12 @@ modparam("tls", "certificate", "/usr/local/etc/ser/my_certificate.pem")
314 342
 
315 343
    The default value is [SER_CFG_DIR]/cert.pem.
316 344
 
317
-   Example 4. Set private_key parameter
345
+   Example 5. Set private_key parameter
318 346
 ...
319 347
 modparam("tls", "private", "/usr/local/etc/ser/my_pkey.pem")
320 348
 ...
321 349
 
322
-1.8.4. ca_list (string)
350
+1.9.4. ca_list (string)
323 351
 
324 352
    Sets the CA list file name. This file contains a list of all the
325 353
    trusted CAs certificates. If a signature in a certificate chain belongs
... ...
@@ -332,12 +360,12 @@ modparam("tls", "private", "/usr/local/etc/ser/my_pkey.pem")
332 360
    certificate in the PEM format to one file, e.g.: for f in
333 361
    trusted_cas/*.pem ; do cat "$f" >> ca_list.pem ; done .
334 362
 
335
-   Example 5. Set ca_list parameter
363
+   Example 6. Set ca_list parameter
336 364
 ...
337 365
 modparam("tls", "ca_list", "/usr/local/etc/ser/ca_list.pem")
338 366
 ...
339 367
 
340
-1.8.5. verify_certificate (boolean)
368
+1.9.5. verify_certificate (boolean)
341 369
 
342 370
    If enabled it will force certificate verification. For more information
343 371
    see the verify(1) openssl man page.
... ...
@@ -349,12 +377,12 @@ modparam("tls", "ca_list", "/usr/local/etc/ser/ca_list.pem")
349 377
 
350 378
    By default the certificate verification is off.
351 379
 
352
-   Example 6. Set verify_certificate parameter
380
+   Example 7. Set verify_certificate parameter
353 381
 ...
354 382
 modparam("tls", "verify_certificate", 1)
355 383
 ...
356 384
 
357
-1.8.6. verify_depth (integer)
385
+1.9.6. verify_depth (integer)
358 386
 
359 387
    Sets how far up the certificate chain will the certificate verification
360 388
    go in the search for a trusted CA.
... ...
@@ -363,12 +391,12 @@ modparam("tls", "verify_certificate", 1)
363 391
 
364 392
    The default value is 9.
365 393
 
366
-   Example 7. Set verify_depth parameter
394
+   Example 8. Set verify_depth parameter
367 395
 ...
368 396
 modparam("tls", "verify_depth", 9)
369 397
 ...
370 398
 
371
-1.8.7. require_certificate (boolean)
399
+1.9.7. require_certificate (boolean)
372 400
 
373 401
    When enabled it will require a certificate from a client. If the client
374 402
    does not offer a certificate and verify_certificate is on, the
... ...
@@ -376,12 +404,12 @@ modparam("tls", "verify_depth", 9)
376 404
 
377 405
    The default value is off.
378 406
 
379
-   Example 8. Set require_certificate parameter
407
+   Example 9. Set require_certificate parameter
380 408
 ...
381 409
 modparam("tls", "require_certificate", 1)
382 410
 ...
383 411
 
384
-1.8.8. cipher_list (string)
412
+1.9.8. cipher_list (string)
385 413
 
386 414
    Sets the list of accepted ciphers. The list consists of cipher strings
387 415
    separated by colons. For more information on the cipher list format see
... ...
@@ -390,54 +418,46 @@ modparam("tls", "require_certificate", 1)
390 418
    The default value is not set (all the Openssl supported ciphers are
391 419
    enabled).
392 420
 
393
-   Example 9. Set cipher_list parameter
421
+   Example 10. Set cipher_list parameter
394 422
 ...
395 423
 modparam("tls", "cipher_list", "HIGH")
396 424
 ...
397 425
 
398
-1.8.9. send_timeout (int)
399
-
400
-   Sets the maximum interval of time after which SIP-router will give up
401
-   trying to send a message over TLS (time after a TLS send will be
402
-   aborted and the corresponding TLS connection closed). The value is in
403
-   seconds.
404
-
405
-   The default value is 120 s.
406
-
407
-   Example 10. Set send_timeout parameter
408
-...
409
-modparam("tls", "send_timeout", 1)
410
-...
411
-
412
-1.8.10. handshake_timeout (int)
426
+1.9.9. send_timeout (int)
413 427
 
414
-   Sets the maximum interval of time after which SIP-router will give up
415
-   trying to accept a TLS connection or connect to a TLS peer. The value
416
-   is in seconds.
428
+   This parameter is obsolete and cannot be used in newer TLS versions (>
429
+   sip-router 3.0). In these versions the send_timeout is replaced by
430
+   tcp_send_timeout (common with all the tcp connections).
417 431
 
418
-   The default value is 120 s.
432
+1.9.10. handshake_timeout (int)
419 433
 
420
-   Example 11. Set handshake_timeout parameter
421
-...
422
-modparam("tls", "handshake_timeout", 1)
423
-...
434
+   This parameter is obsolete and cannot be used in newer TLS versions (>
435
+   sip-router 3.0). In these versions the handshake_timeout is replaced by
436
+   tcp_connect_timeout (common with all the tcp connections).
424 437
 
425
-1.8.11. connection_timeout (int)
438
+1.9.11. connection_timeout (int)
426 439
 
427 440
    Sets the amount of time after which an idle TLS connection will be
428
-   closed. This is similar to tcp_connection_lifetime. The value is
429
-   expressed in seconds.
441
+   closed, if no I/O ever occured after the initial open. If an I/O event
442
+   occurs, the timeout will be extended with tcp_connection_lifetime. The
443
+   value is expressed in seconds.
430 444
 
431 445
    The default value is 10 min.
432 446
 
433 447
    If the value set is -1, the connection will never be close on idle.
434 448
 
435
-   Example 12. Set connection_timeout parameter
449
+   It can be changed also at runtime, via the RPC interface and config
450
+   framework. The config variable name is tls.connection_timeout.
451
+
452
+   Example 11. Set connection_timeout parameter
436 453
 ...
437 454
 modparam("tls", "connection_timeout", 60)
438 455
 ...
439 456
 
440
-1.8.12. tls_disable_compression (boolean)
457
+   Example 12. Set tls.connection_timeout at runtime
458
+ $ sercmd cfg.set_now_int tls connection_timeout 180
459
+
460
+1.9.12. tls_disable_compression (boolean)
441 461
 
442 462
    If set compression over SSL/TLS will be disabled. Note that compression
443 463
    uses a lot of memory (about 10x more then with the compression
... ...
@@ -451,18 +471,19 @@ modparam("tls", "connection_timeout", 60)
451 471
 modparam("tls", "tls_disable_compression", 0) # enable
452 472
 ...
453 473
 
454
-1.8.13. ssl_release_buffers (integer)
474
+1.9.13. ssl_release_buffers (integer)
455 475
 
456 476
    Release internal OpenSSL read or write buffers as soon as they are no
457 477
    longer needed. Combined with ssl_free_list_max_len has the potential of
458 478
    saving a lot of memory ( ~ 32k per connection in the default
459
-   configuration, or 16k + ssl_max_send_fragment).
479
+   configuration, or 16k + ssl_max_send_fragment). For sr versions > 3.0
480
+   it makes little sense to disable it (0) since the tls module already
481
+   has its own internal buffering.
460 482
 
461 483
    A value of -1 would not change this option from its openssl default.
462 484
    Use 0 or 1 for enable/disable.
463 485
 
464
-   By default the value is -1 (the openssl default, which at least in
465
-   openssl 1.0.0 is 0/disabled).
486
+   By default the value is 1 (enabled).
466 487
 
467 488
 Note
468 489
 
... ...
@@ -472,7 +493,7 @@ Note
472 493
    Example 14. Set ssl_release_buffers parameter
473 494
 modparam("tls", "ssl_release_buffers", 1)
474 495
 
475
-1.8.14. ssl_free_list_max_len (integer)
496
+1.9.14. ssl_free_list_max_len (integer)
476 497
 
477 498
    Sets the maximum number of free memory chunks, that OpenSSL will keep
478 499
    per connection. Setting it to 0 would cause any unused memory chunk to
... ...
@@ -482,10 +503,10 @@ modparam("tls", "ssl_release_buffers", 1)
482 503
    Should be combined with ssl_release_buffers.
483 504
 
484 505
    A value of -1 has a special meaning: the OpenSSL default will be used
485
-   (no attempt on changing the value will be made).
506
+   (no attempt on changing the value will be made). For OpenSSL 1.0 the
507
+   internal default is 32.
486 508
 
487
-   By default the value is -1 (the OpenSSL default, which at least in
488
-   OpenSSL 1.0.0 is 32).
509
+   By default the value is 0 (no freelist).
489 510
 
490 511
 Note
491 512
 
... ...
@@ -495,7 +516,7 @@ Note
495 516
    Example 15. Set ssl_freelist_max_len parameter
496 517
 modparam("tls", "ssl_freelist_max_len", 0)
497 518
 
498
-1.8.15. ssl_max_send_fragment (integer)
519
+1.9.15. ssl_max_send_fragment (integer)
499 520
 
500 521
    Sets the maximum number of bytes (from the clear text) sent into one
501 522
    TLS or SSL record. Valid values are between 512 and 16384. Note however
... ...
@@ -530,39 +551,148 @@ Note
530 551
    Example 16. Set ssl_max_send_fragment parameter
531 552
 modparam("tls", "ssl_max_send_fragment", 4096)
532 553
 
533
-1.8.16. ssl_read_ahead (boolean)
554
+1.9.16. ssl_read_ahead (boolean)
534 555
 
535
-   Enables read ahead, reducing the number of read() system calls done
536
-   internally by the OpenSSL library.
556
+   Enables read ahead, reducing the number of internal OpenSSL BIO read()
557
+   calls. This option has only debugging value, in normal circumstances it
558
+   should not be changed from the default.
537 559
 
538
-   When disabled OpenSSL will make at least 2 read() sytem calls per
560
+   When disabled OpenSSL will make at least 2 BIO read() calls per
539 561
    received record: one to get the record header and one to get the rest
540 562
    of the record.
541 563
 
564
+   The TLS module buffers internally all read()s and defines its own fast
565
+   BIO so enabling this option would only cause more memory consumption
566
+   and a minor slow-down (extra memcpy).
567
+
542 568
    A value of -1 has a special meaning: the OpenSSL default will be used
543 569
    (no attempt on changing the value will be made).
544 570
 
545
-   By default the value is 1 (enabled).
571
+   By default the value is 0 (disabled).
546 572
 
547 573
    Example 17. Set ssl_read_ahead parameter
548 574
 modparam("tls", "ssl_read_ahead", 1)
549 575
 
550
-1.8.17. tls_log (int)
576
+1.9.17. send_close_notify (boolean)
577
+
578
+   Enables/disables sending close notify alerts prior to closing the
579
+   corresponding TCP connection. Sending the close notify prior to tcp
580
+   shutdown is "nicer" from a TLS point of view, but it has a measurable
581
+   performance impact. Default: off. Can be set at runtime
582
+   (tls.send_close_notify).
583
+
584
+   The default value is 0 (off).
585
+
586
+   It can be changed also at runtime, via the RPC interface and config
587
+   framework. The config variable name is tls.send_close_notify.
588
+
589
+   Example 18. Set send_close_notify parameter
590
+...
591
+modparam("tls", "send_close_notify", 1)
592
+...
593
+
594
+   Example 19. Set tls.send_close_notify at runtime
595
+ $ sercmd cfg.set_now_int tls send_close_notify 1
596
+
597
+1.9.18. con_ct_wq_max (integer)
598
+
599
+   Sets the maximum allowed per connection clear-text send queue size in
600
+   bytes. This queue is used when data cannot be encrypted and sent
601
+   immediately because of an ongoing TLS/SSL level renegotiation.
602
+
603
+   The default value is 65536 (64 Kb).
604
+
605
+   It can be changed also at runtime, via the RPC interface and config
606
+   framework. The config variable name is tls.con_ct_wq_max.
607
+
608
+   Example 20. Set con_ct_wq_max parameter
609
+...
610
+modparam("tls", "con_ct_wq_max", 1048576)
611
+...
612
+
613
+   Example 21. Set tls.con_ct_wq_max at runtime
614
+ $ sercmd cfg.set_now_int tls con_ct_wq_max 1048576
615
+
616
+1.9.19. ct_wq_max (integer)
617
+
618
+   Sets the maximum total number of bytes queued in all the clear-text
619
+   send queues. These queues are used when data cannot be encrypted and
620
+   sent immediately because of an ongoing TLS/SSL level renegotiation.
621
+
622
+   The default value is 10485760 (10 Mb).
623
+
624
+   It can be changed also at runtime, via the RPC interface and config
625
+   framework. The config variable name is tls.ct_wq_max.
626
+
627
+   Example 22. Set ct_wq_max parameter
628
+...
629
+modparam("tls", "ct_wq_max", 4194304)
630
+...
631
+
632
+   Example 23. Set tls.ct_wq_max at runtime
633
+ $ sercmd cfg.set_now_int tls ct_wq_max 4194304
634
+
635
+1.9.20. ct_wq_blk_size (integer)
636
+
637
+   Minimum block size for the internal clear-text send queues (debugging /
638
+   advanced tunning). Good values are multiple of typical datagram sizes.
639
+
640
+   The default value is 4096.
641
+
642
+   It can be changed also at runtime, via the RPC interface and config
643
+   framework. The config variable name is tls.ct_wq_blk_size.
644
+
645
+   Example 24. Set ct_wq_blk_size parameter
646
+...
647
+modparam("tls", "ct_wq_blk_size", 2048)
648
+...
649
+
650
+   Example 25. Set tls.ct_wq_max at runtime
651
+ $ sercmd cfg.set_now_int tls ct_wq_blk_size 2048
652
+
653
+1.9.21. tls_log (int)
551 654
 
552 655
    Sets the log level at which TLS related messages will be logged.
553 656
 
554
-   The default value is 3.
657
+   The default value is 3 (L_DBG).
555 658
 
556
-   Example 18. Set tls_log parameter
659
+   It can be changed also at runtime, via the RPC interface and config
660
+   framework. The config variable name is tls.log.
661
+
662
+   Example 26. Set tls_log parameter
557 663
 ...
558 664
 # ignore TLS messages if SIP-router is started with debug less than 10
559 665
 modparam("tls", "tls_log", 10)
560 666
 ...
561 667
 
562
-1.8.18. low_mem_threshold1 (integer)
668
+   Example 27. Set tls.log at runtime
669
+ $ sercmd cfg.set_now_int tls log 10
670
+
671
+1.9.22. tls_debug (int)
672
+
673
+   Sets the log level at which TLS debug messages will be logged. Note
674
+   that TLS debug messages are enabled only if the TLS module is compiled
675
+   with debugging enabled (e.g. -DTLS_WR_DEBUG, -DTLS_RD_DEBUG or
676
+   -DTLS_BIO_DEBUG).
677
+
678
+   The default value is 3 (L_DBG).
679
+
680
+   It can be changed also at runtime, via the RPC interface and config
681
+   framework. The config variable name is tls.debug.
682
+
683
+   Example 28. Set tls_debug parameter
684
+...
685
+# ignore TLS debug messages if SIP-router is started with debug less than 10
686
+modparam("tls", "tls_debug", 10)
687
+...
688
+
689
+   Example 29. Set tls.debug at runtime
690
+ $ sercmd cfg.set_now_int tls debug 10
563 691
 
564
-   Sets the minimal free memory from which new TLS connection will start
565
-   to fail. The value is expressed in KB.
692
+1.9.23. low_mem_threshold1 (integer)
693
+
694
+   Sets the minimal free memory from which attempts to open or accept new
695
+   TLS connections will start to fail. The value is expressed in KB.
566 696
 
567 697
    The default value depends on whether the openssl library used handles
568 698
    well low memory situations (openssl bug #1491). As of this writing this
... ...
@@ -577,14 +707,20 @@ modparam("tls", "tls_log", 10)
577 707
      * -1 - use the default value
578 708
      * 0 - disable (TLS connections will not fail preemptively)
579 709
 
710
+   It can be changed also at runtime, via the RPC interface and config
711
+   framework. The config variable name is tls.low_mem_threshold1.
712
+
580 713
    See also low_mem_threshold2.
581 714
 
582
-   Example 19. Set low_mem_threshold1 parameter
715
+   Example 30. Set low_mem_threshold1 parameter
583 716
 ...
584 717
 modparam("tls", "low_mem_threshold1", -1)
585 718
 ...
586 719
 
587
-1.8.19. low_mem_threshold2 (integer)
720
+   Example 31. Set tls.low_mem_threshold1 at runtime
721
+ $ sercmd cfg.set_now_int tls low_mem_threshold1 2048
722
+
723
+1.9.24. low_mem_threshold2 (integer)
588 724
 
589 725
    Sets the minimal free memory from which TLS operations on already
590 726
    established TLS connections will start to fail preemptively. The value
... ...
@@ -603,14 +739,20 @@ modparam("tls", "low_mem_threshold1", -1)
603 739
      * -1 - use the default value
604 740
      * 0 - disable (TLS operations will not fail preemptively)
605 741
 
742
+   It can be changed also at runtime, via the RPC interface and config
743
+   framework. The config variable name is tls.low_mem_threshold2.
744
+
606 745
    See also low_mem_threshold1.
607 746
 
608
-   Example 20. Set low_mem_threshold2 parameter
747
+   Example 32. Set low_mem_threshold2 parameter
609 748
 ...
610 749
 modparam("tls", "low_mem_threshold2", -1)
611 750
 ...
612 751
 
613
-1.8.20. tls_force_run (boolean)
752
+   Example 33. Set tls.low_mem_threshold2 at runtime
753
+ $ sercmd cfg.set_now_int tls low_mem_threshold2 1024
754
+
755
+1.9.25. tls_force_run (boolean)
614 756
 
615 757
    If enabled SIP-router will start even if some of the openssl sanity
616 758
    checks fail (turn it on at your own risk).
... ...
@@ -626,12 +768,12 @@ modparam("tls", "low_mem_threshold2", -1)
626 768
 
627 769
    By default tls_force_run is disabled.
628 770
 
629
-   Example 21. Set tls_force_run parameter
771
+   Example 34. Set tls_force_run parameter
630 772
 ...
631 773
 modparam("tls", "tls_force_run", 11)
632 774
 ...
633 775
 
634
-1.8.21. config (string)
776
+1.9.26. config (string)
635 777
 
636 778
    Sets the name of the TLS specific config file.
637 779
 
... ...
@@ -657,7 +799,7 @@ modparam("tls", "tls_force_run", 11)
657 799
    client when it initiates a new connection by itself (it connects to
658 800
    something).
659 801
 
660
-   Example 22. Short config file
802
+   Example 35. Short config file
661 803
 [server:default]
662 804
 method = TLSv1
663 805
 verify_certificate = no
... ...
@@ -683,29 +825,36 @@ ca_list = local_ca.pem
683 825
    For a more complete example check the tls.cfg distributed with the
684 826
    SIP-router source (sip_router/modules/tls/tls.cfg).
685 827
 
686
-   Example 23. Set config parameter
828
+   Example 36. Set config parameter
687 829
 ...
688 830
 modparam("tls", "config", "/usr/local/etc/ser/tls.cfg")
689 831
 ...
690 832
 
691
-1.9. Functions
833
+   It can be changed also at runtime. The new config will not be loaded
834
+   immediately, but after the first tls.reload RPC call.
835
+
836
+   Example 37. Change and reload tls config at runtime
837
+ $ sercmd cfg.set_now_string tls config "/usr/local/etc/ser/new_tls.cfg"
838
+ $ sercmd tls.reload
839
+
840
+1.10. Functions
692 841
 
693 842
    Revision History
694 843
    Revision $Revision$ $Date$
695 844
 
696
-1.9.1. is_peer_verified()
845
+1.10.1. is_peer_verified()
697 846
 
698 847
    Returns true if the connection on which the message was received is TLS
699 848
    , the peer presented an X509 certificate and the certificate chain
700 849
    verified ok. It can be used only in a request route.
701 850
 
702
-   Example 24. is_peer_verified usage
851
+   Example 38. is_peer_verified usage
703 852
         if (proto==TLS && !is_peer_verified()){
704 853
                 sl_send_reply("400", "No certificate or verification failed");
705 854
                 drop;
706 855
         }
707 856
 
708
-1.10. History
857
+1.11. History
709 858
 
710 859
    Revision History
711 860
    Revision $Revision$ $Date$
... ...
@@ -719,5 +868,9 @@ modparam("tls", "config", "/usr/local/etc/ser/tls.cfg")
719 868
    multiple domains, a tls specific config, config reloading and a tls
720 869
    specific select framework.
721 870
 
871
+   For ser/sr 3.1 most of the TLS specific code was completely re-written
872
+   to add support for asynchrounous TLS and fix several long standing
873
+   bugs.
874
+
722 875
    The code is currently maintained by Andrei Pelinescu-Onciul
723 876
    <andrei@iptel.org>.
... ...
@@ -16,6 +16,11 @@
16 16
 		<para>
17 17
 			This module was put together by Jan Janak <email>jan@iptel.org</email> from code  from the experimental tls core addon (<ulink url="http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/experimental/tls/">http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/experimental/tls/</ulink>), code originally written by Peter Griffiths and later maintained by Cesc Santasusana and from an iptelorg tls code addon, written by Andrei Pelinescu-Onciul <email>andrei@iptel.org</email>. Jan also added support for multiple domains, a tls specific config, config reloading and a tls specific select framework.
18 18
 		</para>
19
+		<para>