Browse code

Merge remote branch 'origin/andrei/tcp_tls_changes'

Asynchronous TLS support and various TCP and io_wait fixes
(especially on BSDs).

* origin/andrei/tcp_tls_changes: (67 commits)
tls: fix partial write on write-wants-read queue flush
tls: more config vars displayed by the tls.options RPC
tls: fix trailing space in new modparams
tls: verbose debugging for SSL_ERROR_WANT_WRITE
tls: add lib64 to LIBS path
tls: doc - notes about enabling debugging
tls: added debug log level modparam
tls: modparams for ct write queue params
tls: doc - new & async related config options
tls: no tls_bio debugging by default
tls: change read_ahead, buffers and freelist defaults
tcp: verbose and safer close()
tls: enable PARTIAL_WRITE by default
tls: partial SSL_write support when reading (tls_read_f)
tls: don't report SSL protocol errors as bugs
tls: more consistent low memory checking
io_wait: kqueue: use the entire array during too many errors fallback
tcp: fix dispatching closed connections to tcp readers
tcp: more complete error messages
tls: support for partial encoding and reseting send_flags
tcp: support for tls partial encoding
tls: update & fix repeated send & delayed send
tcp: change tls send callback interface
tsend: s/char*/const char*/ in function params.
tls: very verbose debug logging
tls: fix tls_send out-of-mem on new connection
tcp: force eof after read if write side hangup
tcp: don't reset read_flags on RD_CONN_REPEAT_READ
tls: deal with internal openssl buffering
tls: fix initial state error handling
tcp: more consistent IO_FD_CLOSING usage
io_wait: kqueue: use a bigger array
io_wait: kqueue: handle ENOENT and more robust error handling
io_wait: fix kqueue io_wait_add & POLLIN
io_wait: don't update FD watched status on error
io_wait: fix kqueue and too many errors in changelist
io_wait: fix: check for EV_ERROR for kqueue()
tcp: fix fd passing bug
tls: config option for sending close notify alerts
tls: SSL_shutdown() only fully established connections
tls: ssl_flush() fix and re-worked error reporting
tls: tls.list rpc: fix timeout & ip display
tls: fix queue accounting
tls: rpc: tls.list and tls.options update
tls: config options for the internal queues
tls: fix wrong wbio usage
tls: fix empty files treatment
tls: added tls.options rpc
tls: migrated to the runtime cfg framework
db_flatstore: updated get_abs_pathname use
core: get_abs_pathname() uses now pkg_malloc()
core: str.h - s/NULL/0/
tls: doc - removed handshake_timeout and send_timeout
tls: removed handshake_timeout and send_timeout
tls: s/tls_cfg/tls_domains_cfg
tls: added tls_info rpc
tls: fix unregistered rpc commands
tls: async support (major tls core rewrite)
tls: tls_bio ctrl cmd support, fixes and debug
tls: clear text write queue implementation
tls: added a minimum overhead shm buffer queue
tls: safer destroy_cfg
tcp: new tls hooks interface and async tls changes
tls: added custom memory based bio
tcp: minor cleanups & spelling
tcp: tcp_send() split in 3 smaller functions
tcp: comments & new internal command

Andrei Pelinescu-Onciul authored on 16/08/2010 00:18:57
Showing 48 changed files
... ...
@@ -34,6 +34,7 @@ core:
34 34
      compiled, use ser -V |grep --color RAW_SOCKS or for a running
35 35
      ser: sercmd core.udp4_raw_info.
36 36
      See udp4_raw, udp4_raw_mtu and udp4_raw_ttl below.
37
+  - asynchronous TLS support
37 38
   - onreply_route {...} is now equivalent with onreply_route[0] {...}
38 39
   - global, per protocol blacklist ignore masks (via extended send_flags).
39 40
     See dst_blacklist_udp_imask a.s.o (dst_blacklist_*_imask).
... ...
@@ -100,11 +101,39 @@ modules:
100 100
            blst_rpl_clear_ignore(mask): like blst_rpl_ignore(mask), but
101 101
             clears instead of setting.
102 102
    - tls:
103
-           new options for better tuning memory usage for modern openssl
104
-            versions: ssl_release_buffers, ssl_freelist_max_len,
105
-            ssl_max_send_fragment, ssl_read_ahead. For more info see
106
-            modules/doc/tls/README.
107
-           compression is now disabled by default. To enable it set
103
+          asynchronous TLS support
104
+          new TLS RPCs (tls.info, tls.options), tls.list more detailed.
105
+          removed handshake_timeout and send_timeout module parameters /
106
+            config variables. The values from tcp are used instead
107
+            (tcp_connect_timeout and tcp_send_timeout).
108
+          runtime config support
109
+          more config options:
110
+            send_close_notify - enables/disables sending close notify
111
+              alerts prior to closing the corresponding TCP connection.
112
+              Sending the close notify prior to tcp shutdown is "nicer"
113
+              from a TLS point of view, but it has a measurable
114
+              performance impact. Default: off. Can be set at runtime
115
+              (tls.send_close_notify).
116
+            con_ct_wq_max - per connection tls maximum clear text write
117
+              queue size.  The TLS clear-text write queues are used when a
118
+              send attempt has to be delayed due to an on-going TLS level
119
+              renegotiation. Can be set at runtime (tls.con_ct_wq_max).
120
+              Default: 65536 (64 Kb).
121
+            ct_wq_max - maximum total for all the tls clear text write
122
+              queues (summed). Can be set at runtime (tls.ct_wq_max).
123
+              Default: 10485760 (10 Mb).
124
+            ct_wq_blk_size - internal TLS pre-write (clear-text) queue
125
+              minimum block size (advance tunning or debugging).
126
+              Can be set at runtime (tls.ct_wq_blk_size).
127
+              Default: 4096 (4 Kb).
128
+          verbose debug messages can be enable by re-compiling with
129
+            -DTLS_RD_DEBUG (for the read path) and -DTLS_WR_DEBUG
130
+            (for the write path).
131
+          new options for better tuning memory usage for modern openssl
132
+            versions: ssl_release_buffers (default 1), ssl_freelist_max_len
133
+            (default 0), ssl_max_send_fragment, ssl_read_ahead (default 0).
134
+            For more info see modules/doc/tls/README.
135
+          compression is now disabled by default. To enable it set
108 136
             tls_disable_compression to 0, but note that memory usage will
109 137
             increase dramatically especially for large number of
110 138
             connections (>1000).
... ...
@@ -640,7 +640,7 @@ cfg_parser_t* cfg_parser_init(str* filename)
640 640
 		goto error;
641 641
 	}
642 642
 
643
-	free(pathname);
643
+	pkg_free(pathname);
644 644
 
645 645
 	st->file = base;
646 646
 	st->line = 1;
... ...
@@ -653,7 +653,7 @@ cfg_parser_t* cfg_parser_init(str* filename)
653 653
 		pkg_free(st);
654 654
 	}
655 655
 	if (base) pkg_free(base);
656
-	if (pathname) free(pathname);
656
+	if (pathname) pkg_free(pathname);
657 657
 	return NULL;
658 658
 }
659 659
 
... ...
@@ -555,13 +555,21 @@ int init_io_wait(io_wait_h* h, int max_fd, enum poll_types poll_method)
555 555
 #endif
556 556
 #ifdef HAVE_KQUEUE
557 557
 		case POLL_KQUEUE:
558
-			h->kq_array=local_malloc(sizeof(*(h->kq_array))*h->max_fd_no);
558
+			h->kq_changes_size=KQ_CHANGES_ARRAY_SIZE;
559
+			/* kevent returns different events for read & write
560
+			   => to get all the possible events in one call we
561
+			   need twice the number of added fds + space
562
+			   for possible changelist errors.
563
+			   OTOH if memory is to be saved at all costs, one can
564
+			   decrease the array size.
565
+			 */
566
+			h->kq_array_size=2 * h->max_fd_no + h->kq_changes_size;
567
+			h->kq_array=local_malloc(sizeof(*(h->kq_array))*h->kq_array_size);
559 568
 			if (h->kq_array==0){
560 569
 				LOG(L_CRIT, "ERROR: init_io_wait: could not alloc"
561 570
 							" kqueue event array\n");
562 571
 				goto error;
563 572
 			}
564
-			h->kq_changes_size=KQ_CHANGES_ARRAY_SIZE;
565 573
 			h->kq_changes=local_malloc(sizeof(*(h->kq_changes))*
566 574
 										h->kq_changes_size);
567 575
 			if (h->kq_changes==0){
... ...
@@ -570,7 +578,8 @@ int init_io_wait(io_wait_h* h, int max_fd, enum poll_types poll_method)
570 570
 				goto error;
571 571
 			}
572 572
 			h->kq_nchanges=0;
573
-			memset((void*)h->kq_array, 0, sizeof(*(h->kq_array))*h->max_fd_no);
573
+			memset((void*)h->kq_array, 0,
574
+						sizeof(*(h->kq_array))*h->kq_array_size);
574 575
 			memset((void*)h->kq_changes, 0,
575 576
 						sizeof(*(h->kq_changes))* h->kq_changes_size);
576 577
 			if (init_kqueue(h)<0){
... ...
@@ -1,6 +1,6 @@
1
-/* 
1
+/*
2 2
  * $Id$
3
- * 
3
+ *
4 4
  * Copyright (C) 2005 iptelorg GmbH
5 5
  *
6 6
  * Permission to use, copy, modify, and distribute this software for any
... ...
@@ -31,9 +31,9 @@
31 31
  *                 this assumption)
32 32
  *     local_malloc (defaults to pkg_malloc)
33 33
  *     local_free   (defaults to pkg_free)
34
- *  
34
+ *
35 35
  */
36
-/* 
36
+/*
37 37
  * History:
38 38
  * --------
39 39
  *  2005-06-13  created by andrei
... ...
@@ -45,6 +45,7 @@
45 45
  *  2007-11-29  support for write (POLLOUT); added io_watch_chg() (andrei)
46 46
  *  2008-02-04  POLLRDHUP & EPOLLRDHUP support (automatically enabled if POLLIN
47 47
  *               is set) (andrei)
48
+ *  2010-06-17  re-enabled & enhanced the EV_ERROR for kqueue (andrei)
48 49
  */
49 50
 
50 51
 
... ...
@@ -78,8 +79,8 @@
78 78
 #endif
79 79
 #ifdef HAVE_SELECT
80 80
 /* needed on openbsd for select*/
81
-#include <sys/time.h> 
82
-#include <sys/types.h> 
81
+#include <sys/time.h>
82
+#include <sys/types.h>
83 83
 #include <unistd.h>
84 84
 /* needed according to POSIX for select*/
85 85
 #include <sys/select.h>
... ...
@@ -108,7 +109,7 @@ extern int _os_ver; /* os version number, needed to select bugs workarrounds */
108 108
 
109 109
 #if 0
110 110
 enum fd_types; /* this should be defined from the including file,
111
-				  see tcp_main.c for an example, 
111
+				  see tcp_main.c for an example,
112 112
 				  0 has a special meaning: not used/empty*/
113 113
 #endif
114 114
 
... ...
@@ -146,8 +147,10 @@ struct io_wait_handler{
146 146
 	enum poll_types poll_method;
147 147
 	int flags;
148 148
 	struct fd_map* fd_hash;
149
-	int fd_no; /*  current index used in fd_array and the passed size for 
150
-				   ep_array & kq_array*/
149
+	int fd_no; /*  current index used in fd_array and the passed size for
150
+				   ep_array (for kq_array at least
151
+				    max(twice the size, kq_changes_size) should be
152
+				   be passed). */
151 153
 	int max_fd_no; /* maximum fd no, is also the size of fd_array,
152 154
 						       fd_hash  and ep_array*/
153 155
 	/* common stuff for POLL, SIGIO_RT and SELECT
... ...
@@ -169,6 +172,7 @@ struct io_wait_handler{
169 169
 	struct kevent* kq_array;   /* used for the eventlist*/
170 170
 	struct kevent* kq_changes; /* used for the changelist */
171 171
 	size_t kq_nchanges;
172
+	size_t kq_array_size;   /* array size */
172 173
 	size_t kq_changes_size; /* size of the changes array */
173 174
 #endif
174 175
 #ifdef HAVE_DEVPOLL
... ...
@@ -218,7 +222,7 @@ static inline struct fd_map* hash_fd_map(	io_wait_h* h,
218 218
  *          events - combinations of POLLIN, POLLOUT, POLLERR & POLLHUP
219 219
  *          idx    - index in the fd_array (or -1 if not known)
220 220
  * return: -1 on error
221
- *          0 on EAGAIN or when by some other way it is known that no more 
221
+ *          0 on EAGAIN or when by some other way it is known that no more
222 222
  *            io events are queued on the fd (the receive buffer is empty).
223 223
  *            Usefull to detect when there are no more io events queued for
224 224
  *            sigio_rt, epoll_et, kqueue.
... ...
@@ -242,10 +246,11 @@ int handle_io(struct fd_map* fm, short events, int idx);
242 242
  *       and EVFILT_WRITE, EV_ADD for the same fd).
243 243
  * returns: -1 on error, 0 on success
244 244
  */
245
-static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag, 
245
+static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag,
246 246
 								void* data)
247 247
 {
248 248
 	int n;
249
+	int r;
249 250
 	struct timespec tspec;
250 251
 
251 252
 	if (h->kq_nchanges>=h->kq_changes_size){
... ...
@@ -256,11 +261,35 @@ static inline int kq_ev_change(io_wait_h* h, int fd, int filter, int flag,
256 256
 		tspec.tv_nsec=0;
257 257
 again:
258 258
 		n=kevent(h->kq_fd, h->kq_changes, h->kq_nchanges, 0, 0, &tspec);
259
-		if (n==-1){
260
-			if (errno==EINTR) goto again;
261
-			LOG(L_ERR, "ERROR: io_watch_add: kevent flush changes "
262
-						" failed: %s [%d]\n", strerror(errno), errno);
263
-			return -1;
259
+		if (unlikely(n == -1)){
260
+			if (unlikely(errno == EINTR)) goto again;
261
+			else {
262
+				/* for a detailed explanation of what follows see
263
+				   io_wait_loop_kqueue EV_ERROR case */
264
+				if (unlikely(!(errno == EBADF || errno == ENOENT)))
265
+					BUG("kq_ev_change: kevent flush changes failed"
266
+							" (unexpected error): %s [%d]\n",
267
+							strerror(errno), errno);
268
+					/* ignore error even if it's not a EBADF/ENOENT */
269
+				/* one of the file descriptors is bad, probably already
270
+				   closed => try to apply changes one-by-one */
271
+				for (r = 0; r < h->kq_nchanges; r++) {
272
+retry2:
273
+					n = kevent(h->kq_fd, &h->kq_changes[r], 1, 0, 0, &tspec);
274
+					if (n==-1) {
275
+						if (unlikely(errno == EINTR))
276
+							goto retry2;
277
+					/* for a detailed explanation of what follows see
278
+						io_wait_loop_kqueue EV_ERROR case */
279
+						if (unlikely(!(errno == EBADF || errno == ENOENT)))
280
+							BUG("kq_ev_change: kevent flush changes failed:"
281
+									" (unexpected error) %s [%d] (%d/%d)\n",
282
+										strerror(errno), errno,
283
+										r, h->kq_nchanges);
284
+						continue; /* skip over it */
285
+					}
286
+				}
287
+			}
264 288
 		}
265 289
 		h->kq_nchanges=0; /* changes array is empty */
266 290
 	}
... ...
@@ -395,7 +424,7 @@ inline static int io_watch_add(	io_wait_h* h,
395 395
 #ifdef HAVE_SIGIO_RT
396 396
 		case POLL_SIGIO_RT:
397 397
 			fd_array_setup(events);
398
-			/* re-set O_ASYNC might be needed, if not done from 
398
+			/* re-set O_ASYNC might be needed, if not done from
399 399
 			 * io_watch_del (or if somebody wants to add a fd which has
400 400
 			 * already O_ASYNC/F_SETSIG set on a duplicate)
401 401
 			 */
... ...
@@ -472,7 +501,7 @@ again2:
472 472
 		case POLL_KQUEUE:
473 473
 			if (likely( events & POLLIN)){
474 474
 				if (unlikely(kq_ev_change(h, fd, EVFILT_READ, EV_ADD, e)==-1))
475
-				goto error;
475
+					goto error;
476 476
 			}
477 477
 			if (unlikely( events & POLLOUT)){
478 478
 				if (unlikely(kq_ev_change(h, fd, EVFILT_WRITE, EV_ADD, e)==-1))
... ...
@@ -480,8 +509,8 @@ again2:
480 480
 					if (likely(events & POLLIN)){
481 481
 						kq_ev_change(h, fd, EVFILT_READ, EV_DELETE, 0);
482 482
 					}
483
+					goto error;
483 484
 				}
484
-				goto error;
485 485
 			}
486 486
 			break;
487 487
 #endif
... ...
@@ -516,7 +545,7 @@ again_devpoll:
516 516
 		pf.events=events;
517 517
 check_io_again:
518 518
 		n=0;
519
-		while(e->type && ((n=poll(&pf, 1, 0))>0) && 
519
+		while(e->type && ((n=poll(&pf, 1, 0))>0) &&
520 520
 				(handle_io(e, pf.revents, idx)>0) &&
521 521
 				(pf.revents & (e->events|POLLERR|POLLHUP)));
522 522
 		if (unlikely(e->type && (n==-1))){
... ...
@@ -531,20 +560,20 @@ error:
531 531
 	if (e) unhash_fd_map(e);
532 532
 	return -1;
533 533
 #undef fd_array_setup
534
-#undef set_fd_flags 
534
+#undef set_fd_flags
535 535
 }
536 536
 
537 537
 
538 538
 
539 539
 #define IO_FD_CLOSING 16
540
-/* parameters:    h - handler 
540
+/* parameters:    h - handler
541 541
  *               fd - file descriptor
542 542
  *            index - index in the fd_array if known, -1 if not
543 543
  *                    (if index==-1 fd_array will be searched for the
544
- *                     corresponding fd* entry -- slower but unavoidable in 
544
+ *                     corresponding fd* entry -- slower but unavoidable in
545 545
  *                     some cases). index is not used (no fd_array) for epoll,
546 546
  *                     /dev/poll and kqueue
547
- *            flags - optimization flags, e.g. IO_FD_CLOSING, the fd was 
547
+ *            flags - optimization flags, e.g. IO_FD_CLOSING, the fd was
548 548
  *                    or will shortly be closed, in some cases we can avoid
549 549
  *                    extra remove operations (e.g.: epoll, kqueue, sigio)
550 550
  * returns 0 if ok, -1 on error */
... ...
@@ -600,7 +629,6 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
600 600
 		goto error;
601 601
 	}
602 602
 	events=e->events;
603
-	unhash_fd_map(e);
604 603
 	
605 604
 	switch(h->poll_method){
606 605
 		case POLL_POLL:
... ...
@@ -614,13 +642,12 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
614 614
 				FD_CLR(fd, &h->master_wset);
615 615
 			if (unlikely(h->max_fd_select && (h->max_fd_select==fd)))
616 616
 				/* we don't know the prev. max, so we just decrement it */
617
-				h->max_fd_select--; 
617
+				h->max_fd_select--;
618 618
 			fix_fd_array;
619 619
 			break;
620 620
 #endif
621 621
 #ifdef HAVE_SIGIO_RT
622 622
 		case POLL_SIGIO_RT:
623
-			fix_fd_array;
624 623
 			/* the O_ASYNC flag must be reset all the time, the fd
625 624
 			 *  can be changed only if  O_ASYNC is reset (if not and
626 625
 			 *  the fd is a duplicate, you will get signals from the dup. fd
... ...
@@ -629,17 +656,18 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
629 629
 			 */
630 630
 			/*if (!(flags & IO_FD_CLOSING)){*/
631 631
 				/* reset ASYNC */
632
-				fd_flags=fcntl(fd, F_GETFL); 
633
-				if (unlikely(fd_flags==-1)){ 
634
-					LOG(L_ERR, "ERROR: io_watch_del: fnctl: GETFL failed:" 
635
-							" %s [%d]\n", strerror(errno), errno); 
636
-					goto error; 
637
-				} 
638
-				if (unlikely(fcntl(fd, F_SETFL, fd_flags&(~O_ASYNC))==-1)){ 
639
-					LOG(L_ERR, "ERROR: io_watch_del: fnctl: SETFL" 
640
-								" failed: %s [%d]\n", strerror(errno), errno); 
641
-					goto error; 
642
-				} 
632
+				fd_flags=fcntl(fd, F_GETFL);
633
+				if (unlikely(fd_flags==-1)){
634
+					LOG(L_ERR, "ERROR: io_watch_del: fnctl: GETFL failed:"
635
+							" %s [%d]\n", strerror(errno), errno);
636
+					goto error;
637
+				}
638
+				if (unlikely(fcntl(fd, F_SETFL, fd_flags&(~O_ASYNC))==-1)){
639
+					LOG(L_ERR, "ERROR: io_watch_del: fnctl: SETFL"
640
+								" failed: %s [%d]\n", strerror(errno), errno);
641
+					goto error;
642
+				}
643
+			fix_fd_array; /* only on success */
643 644
 			break;
644 645
 #endif
645 646
 #ifdef HAVE_EPOLL
... ...
@@ -648,7 +676,7 @@ inline static int io_watch_del(io_wait_h* h, int fd, int idx, int flags)
648 648
 			/* epoll doesn't seem to automatically remove sockets,
649 649
 			 * if the socket is a duplicate/moved and the original
650 650
 			 * is still open. The fd is removed from the epoll set
651
-			 * only when the original (and all the  copies?) is/are 
651
+			 * only when the original (and all the  copies?) is/are
652 652
 			 * closed. This is probably a bug in epoll. --andrei */
653 653
 #ifdef EPOLL_NO_CLOSE_BUG
654 654
 			if (!(flags & IO_FD_CLOSING)){
... ...
@@ -698,7 +726,7 @@ again_devpoll:
698 698
 				if (write(h->dpoll_fd, &pfd, sizeof(pfd))==-1){
699 699
 					if (errno==EINTR) goto again_devpoll;
700 700
 					LOG(L_ERR, "ERROR: io_watch_del: removing fd from "
701
-								"/dev/poll failed: %s [%d]\n", 
701
+								"/dev/poll failed: %s [%d]\n",
702 702
 								strerror(errno), errno);
703 703
 					goto error;
704 704
 				}
... ...
@@ -706,10 +734,11 @@ again_devpoll:
706 706
 #endif
707 707
 		default:
708 708
 			LOG(L_CRIT, "BUG: io_watch_del: no support for poll method "
709
-					" %s (%d)\n", poll_method_str[h->poll_method], 
709
+					" %s (%d)\n", poll_method_str[h->poll_method],
710 710
 					h->poll_method);
711 711
 			goto error;
712 712
 	}
713
+	unhash_fd_map(e); /* only on success */
713 714
 	h->fd_no--;
714 715
 	return 0;
715 716
 error:
... ...
@@ -719,12 +748,12 @@ error:
719 719
 
720 720
 
721 721
 
722
-/* parameters:    h - handler 
722
+/* parameters:    h - handler
723 723
  *               fd - file descriptor
724 724
  *           events - new events to watch for
725 725
  *              idx - index in the fd_array if known, -1 if not
726 726
  *                    (if index==-1 fd_array will be searched for the
727
- *                     corresponding fd* entry -- slower but unavoidable in 
727
+ *                     corresponding fd* entry -- slower but unavoidable in
728 728
  *                     some cases). index is not used (no fd_array) for epoll,
729 729
  *                     /dev/poll and kqueue
730 730
  * returns 0 if ok, -1 on error */
... ...
@@ -781,14 +810,14 @@ inline static int io_watch_chg(io_wait_h* h, int fd, short events, int idx )
781 781
 	
782 782
 	add_events=events & ~e->events;
783 783
 	del_events=e->events & ~events;
784
-	e->events=events;
785 784
 	switch(h->poll_method){
786 785
 		case POLL_POLL:
786
+			fd_array_chg(events
787 787
 #ifdef POLLRDHUP
788
-			/* listen to POLLRDHUP by default (if POLLIN) */
789
-			events|=((int)!(events & POLLIN) - 1) & POLLRDHUP;
788
+							/* listen to POLLRDHUP by default (if POLLIN) */
789
+							| (((int)!(events & POLLIN) - 1) & POLLRDHUP)
790 790
 #endif /* POLLRDHUP */
791
-			fd_array_chg(events);
791
+						);
792 792
 			break;
793 793
 #ifdef HAVE_SELECT
794 794
 		case POLL_SELECT:
... ...
@@ -882,7 +911,7 @@ again_devpoll1:
882 882
 				if (unlikely(write(h->dpoll_fd, &pfd, sizeof(pfd))==-1)){
883 883
 					if (errno==EINTR) goto again_devpoll1;
884 884
 					LOG(L_ERR, "ERROR: io_watch_chg: removing fd from "
885
-								"/dev/poll failed: %s [%d]\n", 
885
+								"/dev/poll failed: %s [%d]\n",
886 886
 								strerror(errno), errno);
887 887
 					goto error;
888 888
 				}
... ...
@@ -892,18 +921,21 @@ again_devpoll2:
892 892
 				if (unlikely(write(h->dpoll_fd, &pfd, sizeof(pfd))==-1)){
893 893
 					if (errno==EINTR) goto again_devpoll2;
894 894
 					LOG(L_ERR, "ERROR: io_watch_chg: re-adding fd to "
895
-								"/dev/poll failed: %s [%d]\n", 
895
+								"/dev/poll failed: %s [%d]\n",
896 896
 								strerror(errno), errno);
897
+					/* error re-adding the fd => mark it as removed/unhash */
898
+					unhash_fd_map(e);
897 899
 					goto error;
898 900
 				}
899 901
 				break;
900 902
 #endif
901 903
 		default:
902 904
 			LOG(L_CRIT, "BUG: io_watch_chg: no support for poll method "
903
-					" %s (%d)\n", poll_method_str[h->poll_method], 
905
+					" %s (%d)\n", poll_method_str[h->poll_method],
904 906
 					h->poll_method);
905 907
 			goto error;
906 908
 	}
909
+	e->events=events; /* only on success */
907 910
 	return 0;
908 911
 error:
909 912
 	return -1;
... ...
@@ -912,7 +944,7 @@ error:
912 912
 
913 913
 
914 914
 
915
-/* io_wait_loop_x style function 
915
+/* io_wait_loop_x style function.
916 916
  * wait for io using poll()
917 917
  * params: h      - io_wait handle
918 918
  *         t      - timeout in s
... ...
@@ -953,11 +985,11 @@ again:
953 953
 				/* repeat handle_io if repeat, fd still watched (not deleted
954 954
 				 *  inside handle_io), handle_io returns that there's still
955 955
 				 *  IO and the fd is still watched for the triggering event */
956
-				while(fm->type && 
956
+				while(fm->type &&
957 957
 						(handle_io(fm, h->fd_array[r].revents, r) > 0) &&
958 958
 						repeat && ((fm->events|POLLERR|POLLHUP) &
959 959
 													h->fd_array[r].revents));
960
-				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd) 
960
+				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd)
961 961
 										  array shifting */
962 962
 			}
963 963
 		}
... ...
@@ -1002,9 +1034,9 @@ again:
1002 1002
 			if (unlikely(revents)){
1003 1003
 				h->crt_fd_array_idx=r;
1004 1004
 				fm=get_fd_map(h, h->fd_array[r].fd);
1005
-				while(fm->type && (fm->events & revents) && 
1005
+				while(fm->type && (fm->events & revents) &&
1006 1006
 						(handle_io(fm, revents, r)>0) && repeat);
1007
-				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd) 
1007
+				r=h->crt_fd_array_idx; /* can change due to io_watch_del(fd)
1008 1008
 										  array shifting */
1009 1009
 				n--;
1010 1010
 			}
... ...
@@ -1028,7 +1060,7 @@ again:
1028 1028
 			if (errno==EINTR) goto again; /* signal, ignore it */
1029 1029
 			else{
1030 1030
 				LOG(L_ERR, "ERROR:io_wait_loop_epoll: "
1031
-						"epoll_wait(%d, %p, %d, %d): %s [%d]\n", 
1031
+						"epoll_wait(%d, %p, %d, %d): %s [%d]\n",
1032 1032
 						h->epfd, h->ep_array, h->fd_no, t*1000,
1033 1033
 						strerror(errno), errno);
1034 1034
 				goto error;
... ...
@@ -1054,7 +1086,7 @@ again:
1054 1054
 					;
1055 1055
 			if (likely(revents)){
1056 1056
 				fm=(struct fd_map*)h->ep_array[r].data.ptr;
1057
-				while(fm->type && ((fm->events|POLLERR|POLLHUP) & revents) && 
1057
+				while(fm->type && ((fm->events|POLLERR|POLLHUP) & revents) &&
1058 1058
 						(handle_io(fm, revents, -1)>0) && repeat);
1059 1059
 			}else{
1060 1060
 				LOG(L_ERR, "ERROR:io_wait_loop_epoll: unexpected event %x"
... ...
@@ -1075,55 +1107,123 @@ inline static int io_wait_loop_kqueue(io_wait_h* h, int t, int repeat)
1075 1075
 	int n, r;
1076 1076
 	struct timespec tspec;
1077 1077
 	struct fd_map* fm;
1078
+	int orig_changes;
1079
+	int apply_changes;
1078 1080
 	int revents;
1079 1081
 	
1080 1082
 	tspec.tv_sec=t;
1081 1083
 	tspec.tv_nsec=0;
1084
+	orig_changes=h->kq_nchanges;
1085
+	apply_changes=orig_changes;
1086
+	do {
1082 1087
 again:
1083
-		n=kevent(h->kq_fd, h->kq_changes, h->kq_nchanges,  h->kq_array,
1084
-					h->fd_no, &tspec);
1088
+		n=kevent(h->kq_fd, h->kq_changes, apply_changes,  h->kq_array,
1089
+					h->kq_array_size, &tspec);
1085 1090
 		if (unlikely(n==-1)){
1086
-			if (errno==EINTR) goto again; /* signal, ignore it */
1087
-			else{
1088
-				LOG(L_ERR, "ERROR: io_wait_loop_kqueue: kevent:"
1091
+			if (unlikely(errno==EINTR)) goto again; /* signal, ignore it */
1092
+			else {
1093
+				/* for a detailed explanation of what follows see below
1094
+				   the EV_ERROR case */
1095
+				if (unlikely(!(errno==EBADF || errno==ENOENT)))
1096
+					BUG("io_wait_loop_kqueue: kevent: unexpected error"
1089 1097
 						" %s [%d]\n", strerror(errno), errno);
1090
-				goto error;
1098
+				/* some of the FDs in kq_changes are bad (already closed)
1099
+				   and there is not enough space in kq_array to return all
1100
+				   of them back */
1101
+				apply_changes = h->kq_array_size;
1102
+				goto again;
1091 1103
 			}
1092 1104
 		}
1093
-		h->kq_nchanges=0; /* reset changes array */
1105
+		/* remove applied changes */
1106
+		h->kq_nchanges -= apply_changes;
1107
+		if (unlikely(apply_changes < orig_changes)) {
1108
+			orig_changes -= apply_changes;
1109
+			memmove(&h->kq_changes[0], &h->kq_changes[apply_changes],
1110
+									sizeof(h->kq_changes[0])*h->kq_nchanges);
1111
+			apply_changes = (orig_changes < h->kq_array_size) ? orig_changes :
1112
+								h->kq_array_size;
1113
+		} else {
1114
+			orig_changes = 0;
1115
+			apply_changes = 0;
1116
+		}
1094 1117
 		for (r=0; r<n; r++){
1095 1118
 #ifdef EXTRA_DEBUG
1096 1119
 			DBG("DBG: kqueue: event %d/%d: fd=%d, udata=%lx, flags=0x%x\n",
1097 1120
 					r, n, h->kq_array[r].ident, (long)h->kq_array[r].udata,
1098 1121
 					h->kq_array[r].flags);
1099 1122
 #endif
1100
-#if 0
1101
-			if (unlikely(h->kq_array[r].flags & EV_ERROR)){
1102
-				/* error in changes: we ignore it, it can be caused by
1103
-				   trying to remove an already closed fd: race between
1104
-				   adding something to the changes array, close() and
1105
-				   applying the changes */
1106
-				LOG(L_INFO, "INFO: io_wait_loop_kqueue: kevent error on "
1107
-							"fd %ld: %s [%ld]\n", h->kq_array[r].ident,
1123
+			if (unlikely((h->kq_array[r].flags & EV_ERROR) ||
1124
+							 h->kq_array[r].udata == 0)){
1125
+				/* error in changes: we ignore it if it has to do with a
1126
+				   bad fd or update==0. It can be caused by trying to remove an
1127
+				   already closed fd: race between adding something to the
1128
+				   changes array, close() and applying the changes (EBADF).
1129
+				   E.g. for ser tcp: tcp_main sends a fd to child for reading
1130
+				    => deletes it from the watched fds => the changes array
1131
+					will contain an EV_DELETE for it. Before the changes
1132
+					are applied (they are at the end of the main io_wait loop,
1133
+					after all the fd events were processed), a CON_ERR sent
1134
+					to tcp_main by a sender (send fail) is processed and causes
1135
+					the fd to be closed. When the changes are applied =>
1136
+					error for the EV_DELETE attempt of a closed fd.
1137
+					Something similar can happen when a fd is scheduled
1138
+					for removal, is close()'ed before being removed and
1139
+					re-opened(a new sock. get the same fd). When the
1140
+					watched fd changes will be applied the fd will be valid
1141
+					(so no EBADF), but it's not already watch => ENOENT.
1142
+					We report a BUG for the other errors (there's nothing
1143
+					constructive we can do if we get an error we don't know
1144
+					how to handle), but apart from that we ignore it in the
1145
+					idea that it is better apply the rest of the changes,
1146
+					rather then dropping all of them.
1147
+				*/
1148
+				/*
1149
+					example EV_ERROR for trying to delete a read watched fd,
1150
+					that was already closed:
1151
+					{
1152
+						ident = 63,  [fd]
1153
+						filter = -1, [EVFILT_READ]
1154
+						flags = 16384, [EV_ERROR]
1155
+						fflags = 0,
1156
+						data = 9, [errno = EBADF]
1157
+						udata = 0x0
1158
+					}
1159
+				*/
1160
+				if (h->kq_array[r].data != EBADF &&
1161
+						h->kq_array[r].data != ENOENT)
1162
+					BUG("io_wait_loop_kqueue: kevent unexpected error on "
1163
+							"fd %ld udata %lx: %s [%ld]\n",
1164
+							(long)h->kq_array[r].ident,
1165
+							(long)h->kq_array[r].udata,
1108 1166
 							strerror(h->kq_array[r].data),
1109 1167
 							(long)h->kq_array[r].data);
1110
-			}else{ 
1111
-#endif
1168
+			}else{
1112 1169
 				fm=(struct fd_map*)h->kq_array[r].udata;
1113 1170
 				if (likely(h->kq_array[r].filter==EVFILT_READ)){
1114
-					revents=POLLIN | 
1115
-						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP);
1116
-					while(fm->type && (fm->events & revents) && 
1171
+					revents=POLLIN |
1172
+						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP) |
1173
+						(((int)!((h->kq_array[r].flags & EV_EOF) &&
1174
+								 	h->kq_array[r].fflags != 0) - 1)&POLLERR);
1175
+					while(fm->type && (fm->events & revents) &&
1117 1176
 							(handle_io(fm, revents, -1)>0) && repeat);
1118 1177
 				}else if (h->kq_array[r].filter==EVFILT_WRITE){
1119
-					revents=POLLOUT | 
1120
-						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP);
1121
-					while(fm->type && (fm->events & revents) && 
1178
+					revents=POLLOUT |
1179
+						(((int)!(h->kq_array[r].flags & EV_EOF)-1)&POLLHUP) |
1180
+						(((int)!((h->kq_array[r].flags & EV_EOF) &&
1181
+								 	h->kq_array[r].fflags != 0) - 1)&POLLERR);
1182
+					while(fm->type && (fm->events & revents) &&
1122 1183
 							(handle_io(fm, revents, -1)>0) && repeat);
1184
+				}else{
1185
+					BUG("io_wait_loop_kqueue: unknown filter: kqueue: event "
1186
+							"%d/%d: fd=%d, filter=%d, flags=0x%x, fflags=0x%x,"
1187
+							" data=%lx, udata=%lx\n",
1188
+					r, n, h->kq_array[r].ident, h->kq_array[r].filter,
1189
+					h->kq_array[r].flags, h->kq_array[r].fflags,
1190
+					(long)h->kq_array[r].data, (long)h->kq_array[r].udata);
1123 1191
 				}
1124
-			/*} */
1192
+			}
1125 1193
 		}
1126
-error:
1194
+	} while(unlikely(orig_changes));
1127 1195
 	return n;
1128 1196
 }
1129 1197
 #endif
... ...
@@ -1207,14 +1307,14 @@ again:
1207 1207
 			 *  POLLIN|POLLRDNORM|POLLMSG (=POLL_MSG),
1208 1208
 			 *  POLLERR (=POLL_ERR),
1209 1209
 			 *  POLLPRI|POLLRDBAND (=POLL_PRI),
1210
-			 *  POLLHUP|POLLERR (=POLL_HUP) 
1210
+			 *  POLLHUP|POLLERR (=POLL_HUP)
1211 1211
 			 *  [linux 2.6.22 fs/fcntl.c:447]
1212 1212
 			 */
1213 1213
 #ifdef EXTRA_DEBUG
1214 1214
 			DBG("io_wait_loop_sigio_rt: siginfo: signal=%d (%d),"
1215 1215
 					" si_code=%d, si_band=0x%x,"
1216 1216
 					" si_fd=%d\n",
1217
-					siginfo.si_signo, n, siginfo.si_code, 
1217
+					siginfo.si_signo, n, siginfo.si_code,
1218 1218
 					(unsigned)sigio_band,
1219 1219
 					sigio_fd);
1220 1220
 #endif
... ...
@@ -1227,7 +1327,7 @@ again:
1227 1227
 				/* fix revents==POLLPRI case */
1228 1228
 				revents |= (!(revents & POLLPRI)-1) & POLLIN;
1229 1229
 				/* we can have queued signals generated by fds not watched
1230
-			 	 * any more, or by fds in transition, to a child 
1230
+			 	 * any more, or by fds in transition, to a child
1231 1231
 				 * => ignore them */
1232 1232
 				if (fm->type && ((fm->events|POLLERR|POLLHUP) & revents))
1233 1233
 					handle_io(fm, revents, -1);
... ...
@@ -1243,7 +1343,7 @@ again:
1243 1243
 			}
1244 1244
 		}
1245 1245
 	}else{
1246
-		/* signal queue overflow 
1246
+		/* signal queue overflow
1247 1247
 		 * TODO: increase signal queue size: 2.4x /proc/.., 2.6x -rlimits */
1248 1248
 		LOG(L_WARN, "WARNING: io_wait_loop_sigio_rt: signal queue overflowed"
1249 1249
 					"- falling back to poll\n");
... ...
@@ -70,7 +70,7 @@ int flat_uri(db_uri_t* uri)
70 70
 
71 71
  error:
72 72
 	if (furi) {
73
-		if (furi->path.s) free(furi->path.s);
73
+		if (furi->path.s) pkg_free(furi->path.s);
74 74
 		db_drv_free(&furi->drv);
75 75
 		pkg_free(furi);
76 76
 	}
... ...
@@ -10,8 +10,10 @@ auto_gen=
10 10
 NAME=tls.so
11 11
 
12 12
 DEFS+= -I$(LOCALBASE)/ssl/include
13
-LIBS+= -L$(LOCALBASE)/lib -L$(LOCALBASE)/ssl/lib -lssl  -lcrypto \
14
-	$(TLS_EXTRA_LIBS)
13
+LIBS+=	-L$(LOCALBASE)/lib -L$(LOCALBASE)/ssl/lib \
14
+		-L$(LOCALBASE)/lib64 -L$(LOCALBASE)/ssl/lib64 \
15
+		-lssl  -lcrypto \
16
+		$(TLS_EXTRA_LIBS)
15 17
 # NOTE: depending on the way in which libssl was compiled you might
16 18
 #       have to add -lz -lkrb5   (zlib and kerberos5).
17 19
 #       E.g.: make TLS_HOOKS=1 TLS_EXTRA_LIBS="-lz -lkrb5"
... ...
@@ -14,42 +14,48 @@ Andrei Pelinescu-Onciul
14 14
    1.3. Important Notes
15 15
    1.4. Compiling the TLS Module
16 16
    1.5. TLS and Low Memory
17
-   1.6. Known Limitations
18
-   1.7. Quick Certificate Howto
19
-   1.8. Parameters
20
-
21
-        1.8.1. tls_method (string)
22
-        1.8.2. certificate (string)
23
-        1.8.3. private_key (string)
24
-        1.8.4. ca_list (string)
25
-        1.8.5. verify_certificate (boolean)
26
-        1.8.6. verify_depth (integer)
27
-        1.8.7. require_certificate (boolean)
28
-        1.8.8. cipher_list (string)
29
-        1.8.9. send_timeout (int)
30
-        1.8.10. handshake_timeout (int)
31
-        1.8.11. connection_timeout (int)
32
-        1.8.12. tls_disable_compression (boolean)
33
-        1.8.13. ssl_release_buffers (integer)
34
-        1.8.14. ssl_free_list_max_len (integer)
35
-        1.8.15. ssl_max_send_fragment (integer)
36
-        1.8.16. ssl_read_ahead (boolean)
37
-        1.8.17. tls_log (int)
38
-        1.8.18. low_mem_threshold1 (integer)
39
-        1.8.19. low_mem_threshold2 (integer)
40
-        1.8.20. tls_force_run (boolean)
41
-        1.8.21. config (string)
42
-
43
-   1.9. Functions
44
-
45
-        1.9.1. is_peer_verified()
46
-
47
-   1.10. History
17
+   1.6. TLS Debugging
18
+   1.7. Known Limitations
19
+   1.8. Quick Certificate Howto
20
+   1.9. Parameters
21
+
22
+        1.9.1. tls_method (string)
23
+        1.9.2. certificate (string)
24
+        1.9.3. private_key (string)
25
+        1.9.4. ca_list (string)
26
+        1.9.5. verify_certificate (boolean)
27
+        1.9.6. verify_depth (integer)
28
+        1.9.7. require_certificate (boolean)
29
+        1.9.8. cipher_list (string)
30
+        1.9.9. send_timeout (int)
31
+        1.9.10. handshake_timeout (int)
32
+        1.9.11. connection_timeout (int)
33
+        1.9.12. tls_disable_compression (boolean)
34
+        1.9.13. ssl_release_buffers (integer)
35
+        1.9.14. ssl_free_list_max_len (integer)
36
+        1.9.15. ssl_max_send_fragment (integer)
37
+        1.9.16. ssl_read_ahead (boolean)
38
+        1.9.17. send_close_notify (boolean)
39
+        1.9.18. con_ct_wq_max (integer)
40
+        1.9.19. ct_wq_max (integer)
41
+        1.9.20. ct_wq_blk_size (integer)
42
+        1.9.21. tls_log (int)
43
+        1.9.22. tls_debug (int)
44
+        1.9.23. low_mem_threshold1 (integer)
45
+        1.9.24. low_mem_threshold2 (integer)
46
+        1.9.25. tls_force_run (boolean)
47
+        1.9.26. config (string)
48
+
49
+   1.10. Functions
50
+
51
+        1.10.1. is_peer_verified()
52
+
53
+   1.11. History
48 54
 
49 55
 1.1. Overview
50 56
 
51 57
    This module implements the TLS transport for SIP-router using the
52
-   Openssl library (http://www.openssl.org). To enable the TLS support
58
+   OpenSSL library (http://www.openssl.org). To enable the TLS support
53 59
    this module must be loaded and enable_tls=yes must be added to the
54 60
    SIP-router config file
55 61
 
... ...
@@ -103,12 +109,15 @@ route{
103 103
    significantly slow down the TLS connection handshake, thus limiting the
104 104
    maximum SIP-router TLS connection rate.
105 105
 
106
-   Compression is fully supported and used by default, if you have a new
107
-   enough Openssl version (starting with 0.9.8). Although there are some
108
-   problems with zlib compression in currently deployed Openssl versions
109
-   (up to and including 0.9.8d, see openssl bug #1468), the TLS module
110
-   will automatically switch to its own fixed version. There's no need to
111
-   force-disable the compression.
106
+   Compression is fully supported if you have a new enough Openssl version
107
+   (starting with 0.9.8). Although there are some problems with zlib
108
+   compression in currently deployed Openssl versions (up to and including
109
+   0.9.8d, see openssl bug #1468), the TLS module will automatically
110
+   switch to its own fixed version. Note however that starting with sr 3.1
111
+   compression is not enabled by default, due to the huge extra memory
112
+   consumption that it causes (about 10x more memory). To enable it use
113
+   modparam("tls", "tls_disable_compression", 0) (see
114
+   tls_disable_compression).
112 115
 
113 116
    The TLS module includes workarounds for the following known openssl
114 117
    bugs: openssl #1204 (disable SS_OP_TLS_BLOCK_PADDING_BUG if compression
... ...
@@ -122,11 +131,10 @@ route{
122 122
 1.4. Compiling the TLS Module
123 123
 
124 124
    In most case compiling the TLS module is as simple as:
125
-make modules modules=modules/tls
125
+make -C modules/tls
126 126
 
127 127
    or
128
-cd modules/tls
129
-make
128
+make modules modules=modules/tls
130 129
 
131 130
    or (compiling whole SIP-router and the tls module)
132 131
 make all include_modules=tls
... ...
@@ -157,7 +165,21 @@ make TLS_EXTRA_LIBS="-lkrb5 -lz" all include_modules=tls
157 157
    reduce openssl memory usage it to disable compression (see
158 158
    tls_disable_compression).
159 159
 
160
-1.6. Known Limitations
160
+1.6. TLS Debugging
161
+
162
+   Debugging messages can be selectively enabled by recompiling the tls
163
+   module with a combination of the following defines:
164
+     * TLS_WR_DEBUG - debug messages for the write/send part.
165
+     * TLS_RD_DEBUG - debug messages for the read/receive part.
166
+     * TLS_BIO_DEBUG - debug messages for the custom BIO.
167
+
168
+   Example 2. Compiling TLS with Debug Messages
169
+make -C modules/tls extra_defs="-DTLS_WR_DEBUG -DTLS_RD_DEBUG"
170
+
171
+   To change the level at which the debug messages are logged, change the
172
+   tls_debug module parameter.
173
+
174
+1.7. Known Limitations
161 175
 
162 176
    The private key must not encrypted (SIP-router cannot ask you for a
163 177
    password on startup).
... ...
@@ -173,10 +195,16 @@ make TLS_EXTRA_LIBS="-lkrb5 -lz" all include_modules=tls
173 173
    TLS specific config reloading is not safe, so for now better don't use
174 174
    it, especially under heavy traffic.
175 175
 
176
-   This documentation is incomplete. The select framework and rpc sections
177
-   are completely missing.
176
+   This documentation is incomplete. The RPCs are not documented here, but
177
+   in doc/rpc_list/rpc_tls.txt or
178
+   http://sip-router.org/docbook/sip-router/branch/master/rpc_list/rpc_lis
179
+   t.html#rpc_exports.tls. The provided selects are not documented. A list
180
+   with all the ones implemented by the tls module can be seen under
181
+   doc/select_list/select_tls.txt or or
182
+   http://sip-router.org/docbook/sip-router/branch/master/select_list/sele
183
+   ct_list.html#select_list.tls.
178 184
 
179
-1.7. Quick Certificate Howto
185
+1.8. Quick Certificate Howto
180 186
 
181 187
    Revision History
182 188
    Revision $Revision$ $Date$
... ...
@@ -260,12 +288,12 @@ fg:
260 260
                 modparam("tls", "require_certificate", 1)
261 261
         (for more information see the module parameters documentation)
262 262
 
263
-1.8. Parameters
263
+1.9. Parameters
264 264
 
265 265
    Revision History
266 266
    Revision $Revision$ $Date$
267 267
 
268
-1.8.1. tls_method (string)
268
+1.9.1. tls_method (string)
269 269
 
270 270
    Sets the SSL/TLS protocol method. Possible values are:
271 271
      * TLSv1 - only TLSv1 connections are accepted. This is the default
... ...
@@ -283,12 +311,12 @@ fg:
283 283
    If rfc3261 conformance is desired, TLSv1 must be used. For
284 284
    compatibility with older clients SSLv23 is a good option.
285 285
 
286
-   Example 2. Set tls_method parameter
286
+   Example 3. Set tls_method parameter
287 287
 ...
288 288
 modparam("tls", "tls_method", "TLSv1")
289 289
 ...
290 290
 
291
-1.8.2. certificate (string)
291
+1.9.2. certificate (string)
292 292
 
293 293
    Sets the certificate file name. The certificate file can also contain
294 294
    the private key in PEM format.
... ...
@@ -299,12 +327,12 @@ modparam("tls", "tls_method", "TLSv1")
299 299
 
300 300
    The default value is [SER_CFG_DIR]/cert.pem.
301 301
 
302
-   Example 3. Set certificate parameter
302
+   Example 4. Set certificate parameter
303 303
 ...
304 304
 modparam("tls", "certificate", "/usr/local/etc/ser/my_certificate.pem")
305 305
 ...
306 306
 
307
-1.8.3. private_key (string)
307
+1.9.3. private_key (string)
308 308
 
309 309
    Sets the private key file name.
310 310
 
... ...
@@ -314,12 +342,12 @@ modparam("tls", "certificate", "/usr/local/etc/ser/my_certificate.pem")
314 314
 
315 315
    The default value is [SER_CFG_DIR]/cert.pem.
316 316
 
317
-   Example 4. Set private_key parameter
317
+   Example 5. Set private_key parameter
318 318
 ...
319 319
 modparam("tls", "private", "/usr/local/etc/ser/my_pkey.pem")
320 320
 ...
321 321
 
322
-1.8.4. ca_list (string)
322
+1.9.4. ca_list (string)
323 323
 
324 324
    Sets the CA list file name. This file contains a list of all the
325 325
    trusted CAs certificates. If a signature in a certificate chain belongs
... ...
@@ -332,12 +360,12 @@ modparam("tls", "private", "/usr/local/etc/ser/my_pkey.pem")
332 332
    certificate in the PEM format to one file, e.g.: for f in
333 333
    trusted_cas/*.pem ; do cat "$f" >> ca_list.pem ; done .
334 334
 
335
-   Example 5. Set ca_list parameter
335
+   Example 6. Set ca_list parameter
336 336
 ...
337 337
 modparam("tls", "ca_list", "/usr/local/etc/ser/ca_list.pem")
338 338
 ...
339 339
 
340
-1.8.5. verify_certificate (boolean)
340
+1.9.5. verify_certificate (boolean)
341 341
 
342 342
    If enabled it will force certificate verification. For more information
343 343
    see the verify(1) openssl man page.
... ...
@@ -349,12 +377,12 @@ modparam("tls", "ca_list", "/usr/local/etc/ser/ca_list.pem")
349 349
 
350 350
    By default the certificate verification is off.
351 351
 
352
-   Example 6. Set verify_certificate parameter
352
+   Example 7. Set verify_certificate parameter
353 353
 ...
354 354
 modparam("tls", "verify_certificate", 1)
355 355
 ...
356 356
 
357
-1.8.6. verify_depth (integer)
357
+1.9.6. verify_depth (integer)
358 358
 
359 359
    Sets how far up the certificate chain will the certificate verification
360 360
    go in the search for a trusted CA.
... ...
@@ -363,12 +391,12 @@ modparam("tls", "verify_certificate", 1)
363 363
 
364 364
    The default value is 9.
365 365
 
366
-   Example 7. Set verify_depth parameter
366
+   Example 8. Set verify_depth parameter
367 367
 ...
368 368
 modparam("tls", "verify_depth", 9)
369 369
 ...
370 370
 
371
-1.8.7. require_certificate (boolean)
371
+1.9.7. require_certificate (boolean)
372 372
 
373 373
    When enabled it will require a certificate from a client. If the client
374 374
    does not offer a certificate and verify_certificate is on, the
... ...
@@ -376,12 +404,12 @@ modparam("tls", "verify_depth", 9)
376 376
 
377 377
    The default value is off.
378 378
 
379
-   Example 8. Set require_certificate parameter
379
+   Example 9. Set require_certificate parameter
380 380
 ...
381 381
 modparam("tls", "require_certificate", 1)
382 382
 ...
383 383
 
384
-1.8.8. cipher_list (string)
384
+1.9.8. cipher_list (string)
385 385
 
386 386
    Sets the list of accepted ciphers. The list consists of cipher strings
387 387
    separated by colons. For more information on the cipher list format see
... ...
@@ -390,54 +418,46 @@ modparam("tls", "require_certificate", 1)
390 390
    The default value is not set (all the Openssl supported ciphers are
391 391
    enabled).
392 392
 
393
-   Example 9. Set cipher_list parameter
393
+   Example 10. Set cipher_list parameter
394 394
 ...
395 395
 modparam("tls", "cipher_list", "HIGH")
396 396
 ...
397 397
 
398
-1.8.9. send_timeout (int)
399
-
400
-   Sets the maximum interval of time after which SIP-router will give up
401
-   trying to send a message over TLS (time after a TLS send will be
402
-   aborted and the corresponding TLS connection closed). The value is in
403
-   seconds.
404
-
405
-   The default value is 120 s.
406
-
407
-   Example 10. Set send_timeout parameter
408
-...
409
-modparam("tls", "send_timeout", 1)
410
-...
411
-
412
-1.8.10. handshake_timeout (int)
398
+1.9.9. send_timeout (int)
413 399
 
414
-   Sets the maximum interval of time after which SIP-router will give up
415
-   trying to accept a TLS connection or connect to a TLS peer. The value
416
-   is in seconds.
400
+   This parameter is obsolete and cannot be used in newer TLS versions (>
401
+   sip-router 3.0). In these versions the send_timeout is replaced by
402
+   tcp_send_timeout (common with all the tcp connections).
417 403
 
418
-   The default value is 120 s.
404
+1.9.10. handshake_timeout (int)
419 405
 
420
-   Example 11. Set handshake_timeout parameter
421
-...
422
-modparam("tls", "handshake_timeout", 1)
423
-...
406
+   This parameter is obsolete and cannot be used in newer TLS versions (>
407
+   sip-router 3.0). In these versions the handshake_timeout is replaced by
408
+   tcp_connect_timeout (common with all the tcp connections).
424 409
 
425
-1.8.11. connection_timeout (int)
410
+1.9.11. connection_timeout (int)
426 411
 
427 412
    Sets the amount of time after which an idle TLS connection will be
428
-   closed. This is similar to tcp_connection_lifetime. The value is
429
-   expressed in seconds.
413
+   closed, if no I/O ever occured after the initial open. If an I/O event
414
+   occurs, the timeout will be extended with tcp_connection_lifetime. The
415
+   value is expressed in seconds.
430 416
 
431 417
    The default value is 10 min.
432 418
 
433 419
    If the value set is -1, the connection will never be close on idle.
434 420
 
435
-   Example 12. Set connection_timeout parameter
421
+   It can be changed also at runtime, via the RPC interface and config
422
+   framework. The config variable name is tls.connection_timeout.
423
+
424
+   Example 11. Set connection_timeout parameter
436 425
 ...
437 426
 modparam("tls", "connection_timeout", 60)
438 427
 ...
439 428
 
440
-1.8.12. tls_disable_compression (boolean)
429
+   Example 12. Set tls.connection_timeout at runtime
430
+ $ sercmd cfg.set_now_int tls connection_timeout 180
431
+
432
+1.9.12. tls_disable_compression (boolean)
441 433
 
442 434
    If set compression over SSL/TLS will be disabled. Note that compression
443 435
    uses a lot of memory (about 10x more then with the compression
... ...
@@ -451,18 +471,19 @@ modparam("tls", "connection_timeout", 60)
451 451
 modparam("tls", "tls_disable_compression", 0) # enable
452 452
 ...
453 453
 
454
-1.8.13. ssl_release_buffers (integer)
454
+1.9.13. ssl_release_buffers (integer)
455 455
 
456 456
    Release internal OpenSSL read or write buffers as soon as they are no
457 457
    longer needed. Combined with ssl_free_list_max_len has the potential of
458 458
    saving a lot of memory ( ~ 32k per connection in the default
459
-   configuration, or 16k + ssl_max_send_fragment).
459
+   configuration, or 16k + ssl_max_send_fragment). For sr versions > 3.0
460
+   it makes little sense to disable it (0) since the tls module already
461
+   has its own internal buffering.
460 462
 
461 463
    A value of -1 would not change this option from its openssl default.
462 464
    Use 0 or 1 for enable/disable.
463 465
 
464
-   By default the value is -1 (the openssl default, which at least in
465
-   openssl 1.0.0 is 0/disabled).
466
+   By default the value is 1 (enabled).
466 467
 
467 468
 Note
468 469
 
... ...
@@ -472,7 +493,7 @@ Note
472 472
    Example 14. Set ssl_release_buffers parameter
473 473
 modparam("tls", "ssl_release_buffers", 1)
474 474
 
475
-1.8.14. ssl_free_list_max_len (integer)
475
+1.9.14. ssl_free_list_max_len (integer)
476 476
 
477 477
    Sets the maximum number of free memory chunks, that OpenSSL will keep
478 478
    per connection. Setting it to 0 would cause any unused memory chunk to
... ...
@@ -482,10 +503,10 @@ modparam("tls", "ssl_release_buffers", 1)
482 482
    Should be combined with ssl_release_buffers.
483 483
 
484 484
    A value of -1 has a special meaning: the OpenSSL default will be used
485
-   (no attempt on changing the value will be made).
485
+   (no attempt on changing the value will be made). For OpenSSL 1.0 the
486
+   internal default is 32.
486 487
 
487
-   By default the value is -1 (the OpenSSL default, which at least in
488
-   OpenSSL 1.0.0 is 32).
488
+   By default the value is 0 (no freelist).
489 489
 
490 490
 Note
491 491
 
... ...
@@ -495,7 +516,7 @@ Note
495 495
    Example 15. Set ssl_freelist_max_len parameter
496 496
 modparam("tls", "ssl_freelist_max_len", 0)
497 497
 
498
-1.8.15. ssl_max_send_fragment (integer)
498
+1.9.15. ssl_max_send_fragment (integer)
499 499
 
500 500
    Sets the maximum number of bytes (from the clear text) sent into one
501 501
    TLS or SSL record. Valid values are between 512 and 16384. Note however
... ...
@@ -530,39 +551,148 @@ Note
530 530
    Example 16. Set ssl_max_send_fragment parameter
531 531
 modparam("tls", "ssl_max_send_fragment", 4096)
532 532
 
533
-1.8.16. ssl_read_ahead (boolean)
533
+1.9.16. ssl_read_ahead (boolean)
534 534
 
535
-   Enables read ahead, reducing the number of read() system calls done
536
-   internally by the OpenSSL library.
535
+   Enables read ahead, reducing the number of internal OpenSSL BIO read()
536
+   calls. This option has only debugging value, in normal circumstances it
537
+   should not be changed from the default.
537 538
 
538
-   When disabled OpenSSL will make at least 2 read() sytem calls per
539
+   When disabled OpenSSL will make at least 2 BIO read() calls per
539 540
    received record: one to get the record header and one to get the rest
540 541
    of the record.
541 542
 
543
+   The TLS module buffers internally all read()s and defines its own fast
544
+   BIO so enabling this option would only cause more memory consumption
545
+   and a minor slow-down (extra memcpy).
546
+
542 547
    A value of -1 has a special meaning: the OpenSSL default will be used
543 548
    (no attempt on changing the value will be made).
544 549
 
545
-   By default the value is 1 (enabled).
550
+   By default the value is 0 (disabled).
546 551
 
547 552
    Example 17. Set ssl_read_ahead parameter
548 553
 modparam("tls", "ssl_read_ahead", 1)
549 554
 
550
-1.8.17. tls_log (int)
555
+1.9.17. send_close_notify (boolean)
556
+
557
+   Enables/disables sending close notify alerts prior to closing the
558
+   corresponding TCP connection. Sending the close notify prior to tcp
559
+   shutdown is "nicer" from a TLS point of view, but it has a measurable
560
+   performance impact. Default: off. Can be set at runtime
561
+   (tls.send_close_notify).
562
+
563
+   The default value is 0 (off).
564
+
565
+   It can be changed also at runtime, via the RPC interface and config
566
+   framework. The config variable name is tls.send_close_notify.
567
+
568
+   Example 18. Set send_close_notify parameter
569
+...
570
+modparam("tls", "send_close_notify", 1)
571
+...
572
+
573
+   Example 19. Set tls.send_close_notify at runtime
574
+ $ sercmd cfg.set_now_int tls send_close_notify 1
575
+
576
+1.9.18. con_ct_wq_max (integer)
577
+
578
+   Sets the maximum allowed per connection clear-text send queue size in
579
+   bytes. This queue is used when data cannot be encrypted and sent
580
+   immediately because of an ongoing TLS/SSL level renegotiation.
581
+
582
+   The default value is 65536 (64 Kb).
583
+
584
+   It can be changed also at runtime, via the RPC interface and config
585
+   framework. The config variable name is tls.con_ct_wq_max.
586
+
587
+   Example 20. Set con_ct_wq_max parameter
588
+...
589
+modparam("tls", "con_ct_wq_max", 1048576)
590
+...
591
+
592
+   Example 21. Set tls.con_ct_wq_max at runtime
593
+ $ sercmd cfg.set_now_int tls con_ct_wq_max 1048576
594
+
595
+1.9.19. ct_wq_max (integer)
596
+
597
+   Sets the maximum total number of bytes queued in all the clear-text
598
+   send queues. These queues are used when data cannot be encrypted and
599
+   sent immediately because of an ongoing TLS/SSL level renegotiation.
600
+
601
+   The default value is 10485760 (10 Mb).
602
+
603
+   It can be changed also at runtime, via the RPC interface and config
604
+   framework. The config variable name is tls.ct_wq_max.
605
+
606
+   Example 22. Set ct_wq_max parameter
607
+...
608
+modparam("tls", "ct_wq_max", 4194304)
609
+...
610
+
611
+   Example 23. Set tls.ct_wq_max at runtime
612
+ $ sercmd cfg.set_now_int tls ct_wq_max 4194304
613
+
614
+1.9.20. ct_wq_blk_size (integer)
615
+
616
+   Minimum block size for the internal clear-text send queues (debugging /
617
+   advanced tunning). Good values are multiple of typical datagram sizes.
618
+
619
+   The default value is 4096.
620
+
621
+   It can be changed also at runtime, via the RPC interface and config
622
+   framework. The config variable name is tls.ct_wq_blk_size.
623
+
624
+   Example 24. Set ct_wq_blk_size parameter
625
+...
626
+modparam("tls", "ct_wq_blk_size", 2048)
627
+...
628
+
629
+   Example 25. Set tls.ct_wq_max at runtime
630
+ $ sercmd cfg.set_now_int tls ct_wq_blk_size 2048
631
+
632
+1.9.21. tls_log (int)
551 633
 
552 634
    Sets the log level at which TLS related messages will be logged.
553 635
 
554
-   The default value is 3.
636
+   The default value is 3 (L_DBG).
555 637
 
556
-   Example 18. Set tls_log parameter
638
+   It can be changed also at runtime, via the RPC interface and config
639
+   framework. The config variable name is tls.log.
640
+
641
+   Example 26. Set tls_log parameter
557 642
 ...
558 643
 # ignore TLS messages if SIP-router is started with debug less than 10
559 644
 modparam("tls", "tls_log", 10)
560 645
 ...
561 646
 
562
-1.8.18. low_mem_threshold1 (integer)
647
+   Example 27. Set tls.log at runtime
648
+ $ sercmd cfg.set_now_int tls log 10
649
+
650
+1.9.22. tls_debug (int)
651
+
652
+   Sets the log level at which TLS debug messages will be logged. Note
653
+   that TLS debug messages are enabled only if the TLS module is compiled
654
+   with debugging enabled (e.g. -DTLS_WR_DEBUG, -DTLS_RD_DEBUG or
655
+   -DTLS_BIO_DEBUG).
656
+
657
+   The default value is 3 (L_DBG).
658
+
659
+   It can be changed also at runtime, via the RPC interface and config
660
+   framework. The config variable name is tls.debug.
661
+
662
+   Example 28. Set tls_debug parameter
663
+...
664
+# ignore TLS debug messages if SIP-router is started with debug less than 10
665
+modparam("tls", "tls_debug", 10)
666
+...
667
+
668
+   Example 29. Set tls.debug at runtime
669
+ $ sercmd cfg.set_now_int tls debug 10
563 670
 
564
-   Sets the minimal free memory from which new TLS connection will start
565
-   to fail. The value is expressed in KB.
671
+1.9.23. low_mem_threshold1 (integer)
672
+
673
+   Sets the minimal free memory from which attempts to open or accept new
674
+   TLS connections will start to fail. The value is expressed in KB.
566 675
 
567 676
    The default value depends on whether the openssl library used handles
568 677
    well low memory situations (openssl bug #1491). As of this writing this
... ...
@@ -577,14 +707,20 @@ modparam("tls", "tls_log", 10)
577 577
      * -1 - use the default value
578 578
      * 0 - disable (TLS connections will not fail preemptively)
579 579
 
580
+   It can be changed also at runtime, via the RPC interface and config
581
+   framework. The config variable name is tls.low_mem_threshold1.
582
+
580 583
    See also low_mem_threshold2.
581 584
 
582
-   Example 19. Set low_mem_threshold1 parameter
585
+   Example 30. Set low_mem_threshold1 parameter
583 586
 ...
584 587
 modparam("tls", "low_mem_threshold1", -1)
585 588
 ...
586 589
 
587
-1.8.19. low_mem_threshold2 (integer)
590
+   Example 31. Set tls.low_mem_threshold1 at runtime
591
+ $ sercmd cfg.set_now_int tls low_mem_threshold1 2048
592
+
593
+1.9.24. low_mem_threshold2 (integer)
588 594
 
589 595
    Sets the minimal free memory from which TLS operations on already
590 596
    established TLS connections will start to fail preemptively. The value
... ...
@@ -603,14 +739,20 @@ modparam("tls", "low_mem_threshold1", -1)
603 603
      * -1 - use the default value
604 604
      * 0 - disable (TLS operations will not fail preemptively)
605 605
 
606
+   It can be changed also at runtime, via the RPC interface and config
607
+   framework. The config variable name is tls.low_mem_threshold2.
608
+
606 609
    See also low_mem_threshold1.
607 610
 
608
-   Example 20. Set low_mem_threshold2 parameter
611
+   Example 32. Set low_mem_threshold2 parameter
609 612
 ...
610 613
 modparam("tls", "low_mem_threshold2", -1)
611 614
 ...
612 615
 
613
-1.8.20. tls_force_run (boolean)
616
+   Example 33. Set tls.low_mem_threshold2 at runtime
617
+ $ sercmd cfg.set_now_int tls low_mem_threshold2 1024
618
+
619
+1.9.25. tls_force_run (boolean)
614 620
 
615 621
    If enabled SIP-router will start even if some of the openssl sanity
616 622
    checks fail (turn it on at your own risk).
... ...
@@ -626,12 +768,12 @@ modparam("tls", "low_mem_threshold2", -1)
626 626
 
627 627
    By default tls_force_run is disabled.
628 628
 
629
-   Example 21. Set tls_force_run parameter
629
+   Example 34. Set tls_force_run parameter
630 630
 ...
631 631
 modparam("tls", "tls_force_run", 11)
632 632
 ...
633 633
 
634
-1.8.21. config (string)
634
+1.9.26. config (string)
635 635
 
636 636
    Sets the name of the TLS specific config file.
637 637
 
... ...
@@ -657,7 +799,7 @@ modparam("tls", "tls_force_run", 11)
657 657
    client when it initiates a new connection by itself (it connects to
658 658
    something).
659 659
 
660
-   Example 22. Short config file
660
+   Example 35. Short config file
661 661
 [server:default]
662 662
 method = TLSv1
663 663
 verify_certificate = no
... ...
@@ -683,29 +825,36 @@ ca_list = local_ca.pem
683 683
    For a more complete example check the tls.cfg distributed with the
684 684
    SIP-router source (sip_router/modules/tls/tls.cfg).
685 685
 
686
-   Example 23. Set config parameter
686
+   Example 36. Set config parameter
687 687
 ...
688 688
 modparam("tls", "config", "/usr/local/etc/ser/tls.cfg")
689 689
 ...
690 690
 
691
-1.9. Functions
691
+   It can be changed also at runtime. The new config will not be loaded
692
+   immediately, but after the first tls.reload RPC call.
693
+
694
+   Example 37. Change and reload tls config at runtime
695
+ $ sercmd cfg.set_now_string tls config "/usr/local/etc/ser/new_tls.cfg"
696
+ $ sercmd tls.reload
697
+
698
+1.10. Functions
692 699
 
693 700
    Revision History
694 701
    Revision $Revision$ $Date$
695 702
 
696
-1.9.1. is_peer_verified()
703
+1.10.1. is_peer_verified()
697 704
 
698 705
    Returns true if the connection on which the message was received is TLS
699 706
    , the peer presented an X509 certificate and the certificate chain
700 707
    verified ok. It can be used only in a request route.
701 708
 
702
-   Example 24. is_peer_verified usage
709
+   Example 38. is_peer_verified usage
703 710
         if (proto==TLS && !is_peer_verified()){
704 711
                 sl_send_reply("400", "No certificate or verification failed");
705 712
                 drop;
706 713
         }
707 714
 
708
-1.10. History
715
+1.11. History
709 716
 
710 717
    Revision History
711 718
    Revision $Revision$ $Date$
... ...
@@ -719,5 +868,9 @@ modparam("tls", "config", "/usr/local/etc/ser/tls.cfg")
719 719
    multiple domains, a tls specific config, config reloading and a tls
720 720
    specific select framework.
721 721
 
722
+   For ser/sr 3.1 most of the TLS specific code was completely re-written
723
+   to add support for asynchrounous TLS and fix several long standing
724
+   bugs.
725
+
722 726
    The code is currently maintained by Andrei Pelinescu-Onciul
723 727
    <andrei@iptel.org>.
... ...
@@ -17,6 +17,11 @@
17 17
 			This module was put together by Jan Janak <email>jan@iptel.org</email> from code  from the experimental tls core addon (<ulink url="http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/experimental/tls/">http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/experimental/tls/</ulink>), code originally written by Peter Griffiths and later maintained by Cesc Santasusana and from an iptelorg tls code addon, written by Andrei Pelinescu-Onciul <email>andrei@iptel.org</email>. Jan also added support for multiple domains, a tls specific config, config reloading and a tls specific select framework.
18 18
 		</para>
19 19
 		<para>
20
+			For ser/sr 3.1 most of the TLS specific code was completely
21
+			re-written to add support for asynchrounous TLS and fix several