diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/Makefile.am | 3 | ||||
| -rw-r--r-- | doc/build_from_repo.html | 29 | ||||
| -rw-r--r-- | doc/debug.html | 2 | ||||
| -rw-r--r-- | doc/highperf.txt | 4 | ||||
| -rw-r--r-- | doc/imsolaris.html | 47 | ||||
| -rw-r--r-- | doc/imtcp.html | 28 | ||||
| -rw-r--r-- | doc/manual.html | 3 | ||||
| -rw-r--r-- | doc/module_workflow.png | bin | 0 -> 14749 bytes | |||
| -rw-r--r-- | doc/msgflow.txt | 56 | ||||
| -rw-r--r-- | doc/rsconf1_escape8bitcharsonreceive.html | 44 | ||||
| -rw-r--r-- | doc/rsyslog_conf_global.html | 7 | ||||
| -rw-r--r-- | doc/rsyslog_conf_modules.html | 94 | ||||
| -rw-r--r-- | doc/rsyslog_conf_templates.html | 46 | ||||
| -rw-r--r-- | doc/src/module_workflow.dia | bin | 0 -> 1700 bytes | |||
| -rw-r--r-- | doc/src/tls.dia | bin | 4656 -> 5201 bytes | |||
| -rw-r--r-- | doc/status.html | 42 | ||||
| -rw-r--r-- | doc/troubleshoot.html | 54 |
17 files changed, 414 insertions, 45 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index f1c3f871..3d900cac 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -41,6 +41,7 @@ html_files = \ imtcp.html \ imgssapi.html \ imrelp.html \ + imsolaris.html \ imuxsock.html \ imklog.html \ professional_support.html \ @@ -66,6 +67,7 @@ html_files = \ rsconf1_actionresumeinterval.html \ rsconf1_allowedsender.html \ rsconf1_controlcharacterescapeprefix.html \ + rsconf1_escape8bitcharsonreceive.html \ rsconf1_debugprintcfsyslinehandlerlist.html \ rsconf1_debugprintmodulelist.html \ rsconf1_debugprinttemplatelist.html \ @@ -123,6 +125,7 @@ html_files = \ grfx_files = \ rsyslog_confgraph_complex.png\ rsyslog_confgraph_std.png \ + module_workflow.png \ direct_queue0.png \ direct_queue1.png \ direct_queue2.png \ diff --git a/doc/build_from_repo.html b/doc/build_from_repo.html index 8d3b20fe..a06863e1 100644 --- a/doc/build_from_repo.html +++ b/doc/build_from_repo.html @@ -43,12 +43,37 @@ you downloaded an official distribution tarball (see the <a href="install.html">rsyslog install guide</a>, starting at step 2, for further details about that). +<h2>Special Compile-Time Options</h2> +<p>On some platforms, compile-time issues occur, like the one shown below: +<p><pre><code> +make[2]: Entering directory `/home/az/RSyslog/rsyslog-5.5.0/tools' + CCLD rsyslogd +rsyslogd-omfile.o: In function `getClockFileAccess': +/home/az/RSyslog/rsyslog-5.5.0/tools/omfile.c:91: undefined reference to `__sync_fetch_and_add_8' +/home/az/RSyslog/rsyslog-5.5.0/tools/omfile.c:91: undefined reference to `__sync_fetch_and_add_8' +/home/az/RSyslog/rsyslog-5.5.0/tools/omfile.c:91: undefined reference to `__sync_fetch_and_add_8' +</code></pre> +<p>Note that the exact error messages can be different. These type of errors stem down to +atomic instruction support in GCC, which is somewhat depending on the machine architecture it +compiles code for. Very old machines (like the original i386) do not even at all provide support +for these instructions. +<p>The availability of atomic instructions is vital for rsyslog - it can not be built without them. +Consequently, there is a configure check included for them. But under some circumstances, +GCC seems to report they are available, but does not provide implementations for +all of them (at least this is my observation...). The simple cure is to make sure that +GCC generates code for a modern-enough architecture. This, for example, can be done as +follows: +<p><pre><code> +./configure CFLAGS="-march=i586 -mcpu=i686" --enable-imfile ... (whatever you need) +</code></pre> +<p>These settings should resolve the issue . + <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 1.2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> diff --git a/doc/debug.html b/doc/debug.html index 46759986..6aeb7975 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -138,7 +138,7 @@ instance of rsyslogd can be aborted by pressing ctl-c. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/highperf.txt b/doc/highperf.txt new file mode 100644 index 00000000..5f9481e1 --- /dev/null +++ b/doc/highperf.txt @@ -0,0 +1,4 @@ +links to high performance papers: + +- http://www.kegel.com/c10k.html +- http://pl.atyp.us/content/tech/servers.html (**) diff --git a/doc/imsolaris.html b/doc/imsolaris.html new file mode 100644 index 00000000..ce0e7e84 --- /dev/null +++ b/doc/imsolaris.html @@ -0,0 +1,47 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Solaris Input Module (imsolaris)</title> + +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Solaris Input Module</h1> +<p><b>Module Name: imsolaris</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Reads local Solaris log messages including the kernel log.</p> +<p>This module is specifically tailored for Solaris. Under Solaris, there +is no special kernel input device. Instead, both kernel messages as well as +messages emitted via syslog() are received from a single source. +<p>This module obeys the Solaris door() mechanism to detect a running syslogd +instance. As such, only one can be active at one time. If it detects another +active intance at startup, the module disables itself, but rsyslog will +continue to run. +<p><b>Configuration Directives</b>:</p> +<ul> +<li><strong>$IMSolarisLogSocketName <name></strong><br> +This is the name of the log socket (stream) to read. If not given, /dev/log +is read. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>None currently known. For obvious reasons, works on Solaris, only (and compilation +will most probably fail on any other platform). +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the default log source +<br> +</p> +<textarea rows="15" cols="60">$ModLoad imsolaris +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imtcp.html b/doc/imtcp.html index 434b3903..422bbd55 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -23,7 +23,7 @@ versions do NOT support it. </p> <p><b>Configuration Directives</b>:</p> <ul> -<li>$InputTCPServerAddtlFrameDelimiter <Delimiter><br> +<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -43,28 +43,36 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. -<li>$InputTCPServerNotifyOnConnectionClose [on/<b>off</b>] (available since 4.5.5)<br> +<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames. However, +some users brought up the case that it may be useful to define a different delimiter and +totally disable LF as a delimiter (the use case named were multi-line messages). This mode +is non-standard and will probably come with a lot of problems. However, as there is need +for it and it is relatively easy to support, we do so. Be sure to turn this setting to +"on" only if you exactly know what you are doing. You may run into all sorts of troubles, +so be prepared to wrangle with that! +<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> instructs imtcp to emit a message if the remote peer closes a connection.<br> <b>Important:</b> This directive is global to all listeners and must be given right after loading imtcp, otherwise it may have no effect.</li> -<li>$InputTCPServerRun <port><br> +<li><b>$InputTCPServerRun</b> <port><br> Starts a TCP server on selected port</li> -<li>$InputTCPMaxListeners <number><br> +<li><b>$InputTCPMaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li>$InputTCPMaxSessions <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> -<li>$InputTCPServerStreamDriverMode <number><br> +<li><b>$InputTCPMaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>$InputTCPServerStreamDriverMode</b> <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> -<li>$InputTCPServerInputName <name><br> +<li><b>$InputTCPServerInputName</b> <name><br> Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a name is not strictly necessary, but can be useful to apply filtering based on which input the message was received from. -<li>$InputTCPServerStreamDriverAuthMode <mode-string><br> +<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> -<li>$InputTCPServerStreamDriverPermittedPeer <id-string><br> +<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> -<li>$InputTCPServerBindRuleset <ruleset><br> +<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/manual.html b/doc/manual.html index c744c3fa..b9085daf 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 5.4.0 (v5-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 5.5.5 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -71,6 +71,7 @@ syslog sender over NAT</a> (online only)</li> <li><a href="debug.html">debug support in rsyslog</a></li> <li>Developer Documentation <ul> + <li><a href="build_from_repo.html">building rsyslog from the source repository</a></li> <li><a href="dev_oplugins.html">writing rsyslog output plugins</a></li> <li><a href="dev_queue.html">the rsyslog message queue object (developer's view)</a></li> </ul></li> diff --git a/doc/module_workflow.png b/doc/module_workflow.png Binary files differnew file mode 100644 index 00000000..e1a72e96 --- /dev/null +++ b/doc/module_workflow.png diff --git a/doc/msgflow.txt b/doc/msgflow.txt new file mode 100644 index 00000000..ebee18f8 --- /dev/null +++ b/doc/msgflow.txt @@ -0,0 +1,56 @@ +flow of messages (in terms of functions) after they have +been pulled off the main queue. + +Functions are listed in the order they are (usually) called +if there are branches in processing flow, this is explicitely +stated. + +as of: 2010-06-08, master branch (v5) + +syslogd.c/msgConsumer +syslogd.c/msgConsumeOne + if ACLcheck needed: + net.cvthname, + net.isAllowedSinder2 + MsgSetRcvFromStr + MsgSetRcvFromIPStr + if NEEDS_PARSING: + parser.ParseMsg +ruleset.ProcessBatch (loops through ruleset) +ruleset.c/processMsgDoRules (for each rule in ruleset) +rule.c/processMsg +1:rule.c/shouldProcessThisMessage + (evaluates filters, optimize via ALL-Filter) +if to be processed, loop through associated actions -> +2:rule.c/processMsgsDoAction +action.c/actionCallAction (LOCKs action object!) +action.c/doActionCallAction (does duplicate message reduction) +action.c/actionWriteToAction + limits based on iExecEveryNthOccur + generates "message repeated..." string if necessary + limits based on iSecsExecOnceInterval +! **qqueueEnqObj** + This means, we are done processing the action at this + stage. The queue may run async, but usually does not + do so (in default settings). + + +Now looking at processing of the action queue. If the queue is +in direct mode, remember that the action object is still +be locked (this may also be a potential bug in non-direct mode, as +it looks like we need this prequisite!). + +action.c/processBatchMain (queue Consumer, LOOK mutActExec) +action.c/processAction + (calls finishBatch at the end, but not so important + for our analysis) +action.c/submitBatch (recursive submit/retry loop for messages) +action.c/tryDoAction (submits a [potentially partial] batch) +action.c/actionProcessMessage + (action.c/actionPrepare (utility to set status/TX mode)) +action.c/actionCallDoAction +1: action.c/prepareDoActionParams +1: template.c/tplToString-tplToArray + string buffer is cached in action object +2:<output Module>/doAction + diff --git a/doc/rsconf1_escape8bitcharsonreceive.html b/doc/rsconf1_escape8bitcharsonreceive.html new file mode 100644 index 00000000..408851c1 --- /dev/null +++ b/doc/rsconf1_escape8bitcharsonreceive.html @@ -0,0 +1,44 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h2>$Escape8BitCharactersOnReceive</h2> +<p><b>Type:</b> global configuration directive</p> +<p><b>Default:</b> off</p> +<p><b>Available Since:</b> 5.5.2</p> +<p><b>Description:</b></p> +<p>This directive instructs rsyslogd to replace non US-ASCII characters (those that +have the 8th bit set) during reception of the message. +This may be useful for some systems. +Please note that this escaping breaks Unicode and many other encodings. Most importantly, +it can be assumed that Asian and European characters will be rendered hardly readable by +this settings. However, it may still be useful when the logs themself are primarily +in English and only occasionally contain local script. +If this option is turned on, all control-characters are converted to a 3-digit octal number and be prefixed with the $ControlCharacterEscapePrefix character (being '#' by default). +<p><b>Warning:</b></p> +<ul> + <li>turning on this option most probably destroys non-western character sets + (like Japanese, Chinese and Korean) as well as European character sets.</li> + <li>turning on this option destroys digital signatures if such exists inside + the message</li> + <li>if turned on, the drop-cc, space-cc and escape-cc + <a href="property_replacer.html">property replacer</a> options do not work + as expected because control characters are already removed upon message + reception. If you intend to use these property replacer options, you must + turn off $Escape8BitCharactersOnReceive.</li> +</ul> +<p><b>Sample:</b></p> +<p><code><b>$Escape8BitCharactersOnReceive on</b></code></p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index ca0c9941..b71fc761 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -140,7 +140,9 @@ our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a <li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> <li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> <li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> +<li><a href="rsconf1_escape8bitcharsonreceive.html">$Escape8BitCharactersOnReceive</a></li> <li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> +<li><b>$EscapeControlCharactersOnReceive</b> [<b>on</b>|off] - escape USASCII HT character</li> <li>$ErrorMessagesToStderr [<b>on</b>|off] - direct rsyslogd error message to stderr (in addition to other targets)</li> <li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> <li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> @@ -276,6 +278,9 @@ system is stripped. If set to on, full names are always used.</li> <li><a href="droppriv.html">$PrivDropToGroupID</a></li> <li><a href="droppriv.html">$PrivDropToUser</a></li> <li><a href="droppriv.html">$PrivDropToUserID</a></li> +<li><b>$Sleep</b> <seconds> - puts the rsyslog main thread to sleep for the specified +number of seconds immediately when the directive is encountered. You should have a +good reason for using this directive!</li> <li><a href="rsconf1_umask.html">$UMASK</a></li> </ul> <p><b>Where <size_nbr> is specified above,</b> @@ -294,7 +299,7 @@ point of view, "1,,0.0.,.,0" also has the value 1000. </p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 4ce62b38..57146e88 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -19,8 +19,17 @@ modules solve your need, you may consider writing one or have one written for you by <a href="http://www.rsyslog.com/professional-services">Adiscon's professional services for rsyslog</a> </b>(this often is a very cost-effective and efficient way of getting what you need). +<p>There exist different classes of loadable modules: +<ul> +<li><a href="rsyslog_conf_modules.html#im">Input Modules</a> +<li><a href="rsyslog_conf_modules.html#om">Output Modules</a> +<li><a href="rsyslog_conf_modules.html#pm">Parser Modules</a> +<li><a href="rsyslog_conf_modules.html#mm">Message Modification Modules</a> +<li><a href="rsyslog_conf_modules.html#sm">String Generator Modules</a> +<li><a href="rsyslog_conf_modules.html#lm">Library Modules</a> +</ul> -<h2>Input Modules</h2> +<a name"im"></a><h2>Input Modules</h2> <p>Input modules are used to gather messages from various sources. They interface to message generators. <ul> @@ -32,10 +41,11 @@ to message generators. <li>immark - support for mark messages</li> <li><a href="imklog.html">imklog</a> - kernel logging</li> <li><a href="imuxsock.html">imuxsock</a> - unix sockets, including the system log socket</li> +<li><a href="imsolaris.html">imsolaris</a> - input for the Sun Solaris system log source</li> <li><a href="im3195.html">im3195</a> - accepts syslog messages via RFC 3195</li> </ul> -<h2>Output Modules</h2> +<a name"om"></a><h2>Output Modules</h2> <p>Output modules process messages. With them, message formats can be transformed and messages be transmitted to various different targets. <ul> @@ -52,20 +62,96 @@ SQLLite, Ingres, Oracle, mSQL)</li> <li><a href="ommail.html">ommail</a> - permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omoracle.html">omoracle</a> - output module for Oracle (native OCI interface)</li> +<li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> </ul> -<h2>Library Modules</h2> +<a name="pm"></a><h2>Parser Modules</h2> +<p>Parser modules are used to parse message content, once the message has been +received. They can be used to process custom message formats or invalidly formatted +messages. For details, please see the <a href="messageparser.html">rsyslog +message parser documentation</a>. +<p>The current modules are currently provided as part of rsyslog: +<ul> +<li>pmrfc5424 - parses RFC5424-formatted messages (the new syslog standard) +<li>pmrfc3164 - the traditional/legacy syslog parser +</ul> + +<a name="mm"></a><h2>Message Modification Modules</h2> +<p>Message modification modules are used to change the content of messages being processed. +They can be implemented using either the output module or the parser module interface. +From the rsyslog core's point of view, they actually are output or parser modules, it is their +implementation that makes them special. +<p>Currently, there do not exist any such modules, but could be written with +the methods the engine provides. They could be used, for example, to: +<ul> +<li>anonymize message content +<li>add dynamically computed content to message (fields) +</ul> + +<a name="lm"></a><h2>String Generator Modules</h2> +<p>String generator modules are used, as the name implies, to generate strings based +on the message content. They are currently tightly coupled with the template system. +Their primary use is to speed up template processing by providing a native C +interface to template generation. These modules exist since 5.5.6. To get an idea +of the potential speedup, the default file format, when generated by a string generator, +provides a roughly 5% speedup. For more complex strings, especially those that include +multiple regular expressions, the speedup may be considerably higher. +<p>String generator modules are written to a quite simple interface. However, a word of +caution is due: they access the rsyslog message object via a low-level interface. +That interface is not guaranteed yet to stay stable. So it may be necessary to +modify string generator modules if the interface changes. Obviously, we will not do that +without good reason, but it may happen. +<p>Rsyslog comes with a set of core, build-in string generators, which are used +to provide those default templates that we consider to be time-critical: +<ul> +<li>smfile - the default rsyslog file format +<li>smfwd - the default rsyslog (network) forwarding format +<li>smtradfile - the traditional syslog file format +<li>smfwd - the traditional syslog (network) forwarding format +</ul> +<p>Note that when you replace these defaults be some custom strings, you will +loose some performance (around 5%). For typical systems, this is not really relevant. +But for a high-performance systems, it may be very relevant. To solve that issue, create +a new string generator module for your custom format, starting out from one of the +default generators provided. If you can not do this yourself, you may want to +contact <a href="mailto:info%40adiscon.com">Adiscon</a> as we offer custom development +of string generators at a very low price. +<p>Note that string generator modules can be dynamically loaded. However, the default +ones provided are so important that they are build right into the executable. But this +does not need to be done that way (and it is straightforward to do it dynamic). + + +<a name="lm"></a><h2>Library Modules</h2> <p>Library modules provide dynamically loadable functionality for parts of rsyslog, most often for other loadable modules. They can not be user-configured and are loaded automatically by some components. They are just mentioned so that error messages that point to library moduls can be understood. No module list is provided. +<h2>Where are the modules integrated into the Message Flow?</h2> +<p>Depending on their module type, modules may access and/or modify messages at +various stages during rsyslog's processing. Note that only the "core type" (e.g. input, +output) but not any type derived from it (message modification module) specifies when +a module is called. +<p>The simplified workflow is as follows: +<p align="center"> +<img src="module_workflow.png" alt"rsyslog: loadable modules and message flow"> +<p>As can be seen, messages are received by input modules, then passed to one or many +parser modules, which generate the in-memory representation of the message and may +also modify the message itself. The, the internal representation is passed to +output modules, which may output a message and (with the interfaces newly introduced +in v5) may also modify messageo object content. +<p>String generator modules are not included inside this picture, because they are +not a required part of the workflow. If used, they operate "in front of" the +output modules, because they are called during template generation. +<p>Note that the actual flow is much more complex and depends a lot on queue and +filter settings. This graphic above is a high-level message flow diagram. + <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index baa4ce29..23a02049 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -16,6 +16,35 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> +<p>Starting with 5.5.6, there are actually two differnt types of template: +<ul> +<li>string based +<li>string-generator module based +</ul> +<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: +<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> +(Of course, you must have first loaded the module via $ModLoad). +<p>The important part is the equal sign: it tells the rsyslog config parser that +no string follows but a strgen module name. +<p>There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +<p>So for most use cases, string-generator module based templates are <b>not</b> +the route to take. Usually, us use <b>string based templates</b> instead. +This is what the rest of the documentation now talks about. + <p>A template consists of a template directive, a name, the actual template text and optional options. A sample is:</p> <blockquote><code>$template MyTemplateName,"\7Text @@ -136,6 +165,23 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> +<h3>String-based Template Samples</h3> +<p>This section provides some sample of what the default formats would +look as a text-based template. Hopefully, their description is self-explanatory. +Note that each $Template statement is on a <b>single</b> line, but probably broken +accross several lines for display purposes by your browsers. Lines are separated by +empty lines. +<p><code> +$template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +<br><br> +$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +<br><br> +$template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +<br><br> +$template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +<br><br> +$template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL +</code></p> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] diff --git a/doc/src/module_workflow.dia b/doc/src/module_workflow.dia Binary files differnew file mode 100644 index 00000000..178571f4 --- /dev/null +++ b/doc/src/module_workflow.dia diff --git a/doc/src/tls.dia b/doc/src/tls.dia Binary files differindex 77e5d185..d7c9811d 100644 --- a/doc/src/tls.dia +++ b/doc/src/tls.dia diff --git a/doc/status.html b/doc/status.html index ff056489..6c183868 100644 --- a/doc/status.html +++ b/doc/status.html @@ -2,37 +2,30 @@ <html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2009-11-05.</p> +<p>This page reflects the status as of 2010-05-20.</p> <h2>Current Releases</h2> -<p><b>v5 development:</b> 5.3.4 [2009-11-04] - -<a href="http://www.rsyslog.com/Article423.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml">download</a> +<p><b>development:</b> 5.5.5 [2010-05-20] - +<a href="http://www.rsyslog.com/Article461.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-204.phtml">download</a> <br> -<!-- not at the moment! -<b>v4 development:</b> 4.5.1 [2009-07-15] - -<a href="http://www.rsyslog.com/Article388.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml">download</a></p> ---> +<b>v4 development:</b> 4.7.2 [2010-05-03] - +<a href="http://www.rsyslog.com/Article457.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-202.phtml">download</a></p> <!-- not at the moment! -<br><b>v5-beta:</b> 5.1.6 [2009-10-15] - -<a href="http://www.rsyslog.com/Article413.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml">download</a> +<p><b>beta:</b> 5.3.7 [2010-01-27] - +<a href="http://www.rsyslog.com/Article437.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-192.phtml">download</a></p> --> -<br><b>v4-beta:</b> 4.5.6 [2009-11-05] - -<a href="http://www.rsyslog.com/Article425.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-186.phtml">download</a></p> - -<p><b>v5 stable:</b> 5.2.0 [2009-11-02] (recommended to use -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml">5.3.3</a> instead) - -<a href="http://www.rsyslog.com/Article421.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-184.phtml">download</a> +<p><b>v5 stable:</b> 5.4.0 [2010-03-08] - +<a href="http://www.rsyslog.com/Article447.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-197.phtml">download</a> -<br><b>v4 stable:</b> 4.4.2 [2009-10-09] - -<a href="http://www.rsyslog.com/Article409.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml">download</a> +<br><b>v4 stable:</b> 4.6.2 [2010-03-26] - +<a href="http://www.rsyslog.com/Article449.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-198.phtml">download</a> <br><b>v3 stable:</b> 3.22.1 [2009-07-02] - <a href="http://www.rsyslog.com/Article381.phtml">change log</a> - @@ -57,7 +50,8 @@ help port rsyslog to other platforms. As such, platform list is now maintained inside the rsyslog wiki</a>. Platform maintainers perhaps have posted extra information there. If you do platform-specific work, feel free to add information to the wiki.</p> -<p>Rsyslog is the default syslogd in Fedora 8 and above.</p> +<p>Rsyslog is the default syslogd on almost all modern Linuxes. +It is also available under BSD and Sun Solaris.</p> <h2>Additional information</h2> <p><b>Currently supported features are listed on the <a href="features.html">rsyslog features page</a>.</b></p> <ul> diff --git a/doc/troubleshoot.html b/doc/troubleshoot.html index a8855fd4..0f0c7fca 100644 --- a/doc/troubleshoot.html +++ b/doc/troubleshoot.html @@ -102,13 +102,63 @@ comes without any guarantees, include no guarantee on confidentiality [aka "we don't want to be sued for work were are not even paid for ;)]. <b>So if you submit debug logs, do so at your sole risk</b>. By submitting them, you accept this policy. +<p><b>Segmentation Faults</b> +<p>Rsyslog has a very rapid development process, complex capabilities and now gradually gets +more and more exposure. While we are happy about this, it also has some bad effects: some +deployment scenarios have probably never been tested and it may be impossible to test +them for the development team because of resources needed. So while we try to avoid this, +you may see a serious problem during deployments in demanding, non-standard, environments +(hopefully not with a stable version, but chances are good you'll run into troubles with +the development versions). +<p>Active support from the user base is very important to help us track down those things. +Most often, serious problems are the result of some memory misadressing. During development, +we routinely use valgrind, a very well and capable memory debugger. This helps us to create +pretty clean code. But valgrind can not detect everything, most importantly not code pathes +that are never executed. So of most use for us is information about aborts and abort locations. +<p>Unforutnately, faults rooted in adressing errors typically show up only later, so the +actual abort location is in an unrelated spot. To help track down the original spot, +<a href="http://www.gnu.org/software/hello/manual/libc/Heap-Consistency-Checking.html">libc +later than 5.4.23 offers support</a> for finding, and possible temporary relief from it, +by means of the MALLOC_CHECK_ environment variable. Setting it to 2 is a useful troubleshooting +aid for us. It will make the program abort as soon as the check routines detect anything +suspicious (unfortunately, this may still not be the root cause, but hopefully closer to it). +Setting it to 0 may even make some problems disappear (but it will NOT fix them!). +With functionality comes cost, and so exporting MALLOC_CHECK_ without need comes at +a performance penalty. However, we strongly recommend adding this instrumentation to your +test environment should you see any serious problems. Chances are good it will help us +interpret a dump better, and thus be able to quicker craft a fix. +<p>In order to get useful information, we need some backtrace of the abort. First, you need +to make sure that a core file is created. Under Fedora, for example, that means you need +to have an "ulimit -c unlimited" in place. +<p>Now let's assume you got a core file (e.g. in /core.1234). So what to do next? Sending a +core file to us is most often pointless - we need to have the exact same system configuration in +order to interpret it correctly. Obviously, chances are extremely slim for this to be. So we would +appreciate if you could extract the most important information. This is done as follows: +<ul> +<li>$gdb /path/to/rsyslogd +<li>$info thread +<li>you'll see a number of threads (in the range 0 to n with n being the highest number). For + <b>each</b> of them, do the following (let's assume that i is the thread number): + <ul> + <li>$ thread i (e.g. thread 0, thread 1, ...) + <li>$bt + </ul> +<li>then you can quit gdb with "$q" +</ul> +<p>Then please send all information that gdb spit out to the development team. It is best to first +ask on the forum or mailing list on how to do that. The developers will keep in contact with you +and, I fear, will probably ask for other things as well ;) +<p>Note that we strive for highest reliability of the engine even in unusual deployment scenarios. +Unfortunately, this is hard to achieve, especially with limited resources. So we are depending on +cooperation from users. This is your chance to make a big contribution to the project without the +need to program or do anything else except get a problem solved ;) <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> |
