From 0218ac3b17cf22f44895cd06e8501c93e922407a Mon Sep 17 00:00:00 2001
From: Rainer Gerhards <rgerhards@adiscon.com>
Date: Thu, 31 Jan 2008 08:25:28 +0000
Subject: updated to new 3.11.0 features

---
 doc/rsyslog_high_database_rate.html | 50 ++++++++++++++++++++++++++++++++++---
 1 file changed, 46 insertions(+), 4 deletions(-)

diff --git a/doc/rsyslog_high_database_rate.html b/doc/rsyslog_high_database_rate.html
index 54961f5d..5b0a29ae 100644
--- a/doc/rsyslog_high_database_rate.html
+++ b/doc/rsyslog_high_database_rate.html
@@ -14,7 +14,7 @@
 
 		<a href="http://www.gerhards.net/rainer">Rainer 
 
-		Gerhards</a> (2008-01-28)</i></small></P>
+		Gerhards</a> (2008-01-31)</i></small></P>
 
 <h2>Abstract</h2>
 
@@ -72,16 +72,54 @@ restarts, so even a reboot of the system will not cause any message loss.</p>
 disk queue and then configure your action. There is nothing else to do. With the 
 following simple config file, you log anything you receive to a MySQL database 
 and have buffering applied automatically.</p>
-<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL)
-$ModLoad imudp.so # network reception
-$UDPServerRun 514 # start a udp server at port 514
+<textarea rows="11" cols="80">$ModLoad ommysql.so  # load the output driver (use ompgsql.so for PostgreSQL)
+$ModLoad imudp.so    # network reception
+$UDPServerRun 514    # start a udp server at port 514
 $ModLoad imuxsock.so # local message reception
 
 $WorkDirectory /rsyslog/work # default location for work (spool) files
 $MainMsgQueueFileName mainq  # set file name, also enables disk mode
 
+$ActionResumeRetryCount -1   # infinite retries on insert failure
+# for PostgreSQL replace :ommysql: by :ompgsql: below:
 *.*       :ommysql:hostname,dbname,userid,password;
 </textarea>
+<p>The simple setup above has one drawback: the write database action is 
+executed together with all other actions. Typically, local files are also 
+written. These local file writes are now bound to the speed of the database 
+action. So if the database is down, or threre is a large backlog, local files 
+are also not (or late) written.</p>
+<p><b>There is an easy way to avoid this with rsyslog.</b> It involves a 
+slightly more complicated setup. In rsyslog, each action can utilize its own 
+queue. If so, messages are simply pulled over from the main queue and then the 
+action queue handles action processing on its own. This way, main processing and 
+the action are de-coupled. In the above example, this means that local file 
+writes will happen immediately while the database writes are queued. As a 
+side-note, each action can have its own queue, so if you would like to more than 
+a single database or send messages reliably to another host, you can do all of 
+this on their own queues, de-coupling their processing speeds.</p>
+<p>The configuration for the de-coupled database write involves just a few more 
+commands:</p>
+<textarea rows="11" cols="80">$ModLoad ommysql.so  # load the output driver (use ompgsql.so for PostgreSQL)
+$ModLoad imudp.so    # network reception
+$UDPServerRun 514    # start a udp server at port 514
+$ModLoad imuxsock.so # local message reception
+
+$WorkDirectory /rsyslog/work # default location for work (spool) files
+
+$ActionQueueType LinkedList # use asynchronous processing
+$ActionQueueFileName dbq    # set file name, also enables disk mode
+$ActionResumeRetryCount -1  # infinite retries on insert failure
+# for PostgreSQL replace :ommysql: by :ompgsql: below:
+*.*       :ommysql:hostname,dbname,userid,password;
+</textarea>
+<p><b>This is the recommended configuration for this use case.</b> It requires 
+rsyslog 3.11.0 or above.</p>
+<p>In this example, the main message queue is NOT disk-assisted (there is no 
+$MainMsgQueueFileName directive). We still could do that, but have not done it 
+because there seems to be no need. The only slow running action is the database 
+writer and it has its own queue. So there is no real reason to use a large main 
+message queue (except, of course, if you expect *really* heavy traffic bursts).</p>
 <p>Note that you can modify a lot of queue performance parameters, but the above
 config will get you going with default values. If you consider using this on a real
 busy server, it is strongly recommended to invest some time in setting the tuning
@@ -102,6 +140,10 @@ comments or find bugs (I *do* bugs - no way... ;)), please
 	<li>2008-01-28 * 
 
 	<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li>
+	<li>2008-01-28 * 
+
+	<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> 
+	* Updated to new v3.11.0 capabilities</li>
 
 </ul>
 <h2>Copyright</h2>
-- 
cgit v1.2.3