diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/Makefile.am | 2 | ||||
| -rw-r--r-- | doc/debug.html | 5 | ||||
| -rw-r--r-- | doc/imklog.html | 96 | ||||
| -rw-r--r-- | doc/impstats.html | 100 | ||||
| -rw-r--r-- | doc/imptcp.html | 11 | ||||
| -rw-r--r-- | doc/imtcp.html | 11 | ||||
| -rw-r--r-- | doc/imudp.html | 7 | ||||
| -rw-r--r-- | doc/lookup_tables.html | 205 | ||||
| -rw-r--r-- | doc/manual.html | 8 | ||||
| -rw-r--r-- | doc/mmjsonparse.html | 45 | ||||
| -rw-r--r-- | doc/mmnormalize.html | 46 | ||||
| -rw-r--r-- | doc/omfile.html | 16 | ||||
| -rw-r--r-- | doc/omfwd.html | 2 | ||||
| -rw-r--r-- | doc/omlibdbi.html | 46 | ||||
| -rw-r--r-- | doc/ommongodb.html | 60 | ||||
| -rw-r--r-- | doc/omudpspoof.html | 173 | ||||
| -rw-r--r-- | doc/rainerscript.html | 7 | ||||
| -rw-r--r-- | doc/rsconf1_rulesetparser.html | 2 | ||||
| -rw-r--r-- | doc/rsyslog_conf_modules.html | 3 | ||||
| -rw-r--r-- | doc/v7compatibility.html | 47 |
20 files changed, 753 insertions, 139 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index e8cdba55..6501cf6c 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -70,6 +70,7 @@ html_files = \ tls_cert_client.html \ tls_cert_scenario.html \ rainerscript.html \ + lookup_tables.html \ rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ @@ -115,6 +116,7 @@ html_files = \ gssapi.html \ licensing.html \ mmnormalize.html \ + mmjsonparse.html \ ommail.html \ omuxsock.html \ omrelp.html \ diff --git a/doc/debug.html b/doc/debug.html index 6aeb7975..8b104d80 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -49,6 +49,11 @@ FileTrace=vm.c FileTrace=expr.c"</li> <li><b>Debug</b> - if present, turns on the debug system and enables debug output <li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>OutputTidToStderr</b> - if present, makes rsyslog output information about +the thread id (tid) of newly create processesto stderr. Note that not necessarily +all new threads are reported (depends on the code, e.g. of plugins). This is +only available under Linux. This usually does NOT work when privileges have +been dropped (that's not a bug, but the way it is). <li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> </ul> </ul> diff --git a/doc/imklog.html b/doc/imklog.html index 05292ddf..1f195b16 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -15,7 +15,10 @@ syslog engine.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><strong>$KLogInternalMsgFacility +<li><strong>LogPath</strong><br> +The path to the Kernel log. This value should only be changed if you really know what +you are doing.</li> +<li><strong>InternalMsgFacility <facility></strong><br> The facility which messages internally generated by imklog will have. imklog generates some messages of itself (e.g. on problems, startup and @@ -26,13 +29,54 @@ need to specify this configuratin directive - it is included primarily for few limited cases where it is needed for good reason. Bottom line: if you don't have a good idea why you should use this setting, do not touch it.</li> -<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility -[on/<span style="font-style: italic;">off</span>]<br> -</span>At least under BSD the kernel log may contain entries +<li><b>PermitNonKernelFacility [on/<i>off</i>]</b><br> +At least under BSD the kernel log may contain entries with non-kernel facilities. This setting controls how those are handled. The default is "off", in which case these messages are ignored. Switch it to on to submit non-kernel messages to rsyslog -processing.<span style="font-weight: bold;"></span></li> +processing.</li> +<li><b>ParseKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled and the kernel creates a timestamp for its log messages, this timestamp will be +parsed and converted into regular message time instead to use the receive time of the kernel +message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the +clock used by the kernel to create the timestamps is not supposed to be as accurate as the +monotonic clock required to convert it. Depending on the hardware and kernel, it can result +in message time differences between kernel and system messages which occurred at same time. +<li><b>KeepKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin +of in each message rather than to remove it, when it could be parsed and converted into +local time for use as regular message time. Only used when <b>ParseKernelTimestamp</b> is on. +<li><b>ConsoleLogLevel</b> [<i>number</i>] +(former klogd -c option) -- sets the console log level. If specified, only messages with +up to the specified level are printed to the console. The default is -1, which means that +the current settings are not modified. To get this behavior, do not specify +ConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time +it is changed, the previous definition is re-set. The one activate will be that one that is +active when imklog actually starts processing. In short words: do not specify this +directive more than once! +</ul> +<b>Caveats/Known Bugs:</b> +<p>This is obviously platform specific and requires platform +drivers. +Currently, imklog functionality is available on Linux and BSD.</p> +<p>This module is <b>not supported on Solaris</b> and not needed there. +For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the kernel log. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling kernel messages.<br> +</p> +<textarea rows="4" cols="60">module(load="imklog") +</textarea> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><strong>$KLogInternalMsgFacility +<facility></strong><br> +equivalent to: InternalMsgFacility</li> +<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility +[on/<span style="font-style: italic;">off</span>]<br> +equivalent to: PermitNonKernelFacility</li> <li><span style="font-weight: bold;"></span>$DebugPrintKernelSymbols [on/<b>off</b>]<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -50,14 +94,7 @@ it except if you have a very good reason. If you have one, let us know because otherwise new versions will no longer support it.<br> Linux only, ignored on other platforms (but may be specified)</li> <li><b>$klogConsoleLogLevel</b> [<i>number</i>] -(former klogd -c option) -- sets the console log level. If specified, only messages with -up to the specified level are printed to the console. The default is -1, which means that -the current settings are not modified. To get this behavior, do not specify -$klogConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time -it is changed, the previous definition is re-set. The one activate will be that one that is -active when imklog actually starts processing. In short words: do not specify this -directive more than once! -<br><b>Linux only</b>, ignored on other platforms (but may be specified)</li> +<br>equivalent to: ConsoleLogLevel</li> <li><b>$klogUseSyscallInterface</b> [on/<b>off</b>] -- former klogd -s option<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -65,40 +102,17 @@ Linux only, ignored on other platforms (but may be specified)</li> former klogd -2 option<br> Linux only, ignored on other platforms (but may be specified)<br style="font-weight: bold;"> </li> -<li><b>$klogParseKernelTimestamp</b> [on/<b>off</b>] -If enabled and the kernel creates a timestamp for its log messages, this timestamp will be -parsed and converted into regular message time instead to use the receive time of the kernel -message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the -clock used by the kernel to create the timestamps is not supposed to be as accurate as the -monotonic clock required to convert it. Depending on the hardware and kernel, it can result -in message time differences between kernel and system messages which occurred at same time. -</li> -<li><b>$klogKeepKernelTimestamp</b> [on/<b>off</b>] -If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin -of in each message rather than to remove it, when it could be parsed and converted into -local time for use as regular message time. Only used, when $klogParseKernelTimestamp is on. -</li> +<li><b>$klogParseKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: ParseKernelTimeStamp</li> +<li><b>$klogKeepKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: KeepKernelTimeStamp</li> </ul> -<b>Caveats/Known Bugs:</b> -<p>This is obviously platform specific and requires platform -drivers. -Currently, imklog functionality is available on Linux and BSD.</p> -<p>This module is <b>not supported on Solaris</b> and not needed there. -For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> -<p><b>Sample:</b></p> -<p>The following sample pulls messages from the kernel log. All -parameters are left by default, which is usually a good idea. Please -note that loading the plugin is sufficient to activate it. No directive -is needed to start pulling kernel messages.<br> -</p> -<textarea rows="15" cols="60">$ModLoad imklog -</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/impstats.html b/doc/impstats.html index 64b04a30..8db9c6f6 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -16,26 +16,66 @@ availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version). <p>The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). -Be sure that your configuration records the counter messages (default is syslog.info). +Be sure that your configuration records the counter messages (default is syslog.=info). +Besides logging to the regular syslog stream, the module can also be configured to +write statistics data into a (local) file. <p>Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be noticable (for high-load environments). <p>The rsyslog website has an updated overview of available <a href="http://rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a>. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Module Confguration Parameters</b>:</p> +<p>This module supports module parameters, only. <ul> -<li>$PStatInterval <Seconds><br> -Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the -actual interval may be a bit longer. We do not try to be precise and so the interval is -actually a sleep period which is entered after generating all messages. So the actual -interval is what is configured here plus the actual time required to generate messages. -In general, the difference should not really matter. -<li>$PStatFacility <numerical facility><br> -The numerical syslog facility code to be used for generated messages. Default -is 5 (syslog).This is useful for filtering messages.</li> -<li>$PStatSeverity <numerical severity><br> -The numerical syslog severity code to be used for generated messages. Default -is 6 (info).This is useful for filtering messages.</li> + <li><strong>interval </strong>[seconds] (default 300 [5minutes])<br> + Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the + actual interval may be a bit longer. We do not try to be precise and so the interval is + actually a sleep period which is entered after generating all messages. So the actual + interval is what is configured here plus the actual time required to generate messages. + In general, the difference should not really matter. + <br></li> + <li><strong>facility </strong>[templateName]<br> + The numerical syslog facility code to be used for generated messages. Default + is 5 (syslog). This is useful for filtering messages. + <br></li> + <li><strong>severity </strong>[templateName]<br> + The numerical syslog severity code to be used for generated messages. Default + is 6 (info).This is useful for filtering messages. + <br></li> + <li><strong>format </strong>[json/cee/<b>legacy</b>](rsyslog v6.3.8+ only)<br> + Specifies the format of emitted stats messages. The default of "legacy" is + compatible with pre v6-rsyslog. The other options provide support for + structured formats (note the "cee" is actually "project lumberack" logging). + <br></li> + <li><strong>log.syslog </strong>[<b>on</b>/off] - available since 7.3.6<br> + This is a boolean setting specifying if data should be sent + to the usual syslog stream. This is useful if custom formatting + or more elaborate processing is desired. However, output is placed + under the same restrictions as regular syslog data, especially in + regard to the queue position (stats data may sit for an extended + period of time in queues if they are full).<br></li> + <li><strong>log.file </strong>[file name] - available since 7.3.6<br> + If specified, statistics data is written the specified file. For + robustness, this should be a local file. The file format cannot be + customized, it consists of a date header, followed by a colon, + followed by the actual statistics record, all on one line. Only + very limited error handling is done, so if things go wrong stats + records will probably be lost. Logging to file an be a useful + alternative if for some reasons (e.g. full queues) the regular + syslog stream method shall not be used solely. Note that turning + on file logging does NOT turn of syslog logging. If that is desired + log.syslog="off" must be explicitely set. + <br></li> + +</ul> +<p><b>Legacx Configuration Directives</b>:</p> +A limited set of parameters can also be set via the legacy configuration +syntax. Note that this is intended as an upward compatibilit layer, so +newer features are intentionally <b>not</b> available via legacy directives. +<ul> +<li>$PStatInterval <Seconds> - same as the "interval" parameter. +<li>$PStatFacility <numerical facility> - same as the "facility" parameter. +<li>$PStatSeverity <numerical severity> - same as the "severity" parameter. <li>$PStatJSON <on/<b>off</b>> (rsyslog v6.3.8+ only)<br> If set to on, stats messages are emitted as structured cee-enhanced syslog. If set to off, legacy format is used (which is compatible with pre v6-rsyslog). @@ -45,23 +85,45 @@ set to off, legacy format is used (which is compatible with pre v6-rsyslog). <ul> <li>This module MUST be loaded right at the top of rsyslog.conf, otherwise stats may not get turned on in all places.</li> -<li>experimental code</li> </ul> -<p><b>Sample:</b></p> +<p><b>Samples:</b></p> <p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:<br> </p> -<textarea rows="8" cols="60">$ModLoad impstats +<textarea rows="5" cols="60">module(load="impstats" interval="600" severity="7") + +# to actually gather the data: +syslog.=debug /var/log/rsyslog-stats +</textarea> +<p><b>Legacy Sample:</b></p> +<p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:</p> +<textarea rows="6" cols="60">$ModLoad impstats $PStatInterval 600 $PStatSeverity 7 -syslog.debug /var/log/rsyslog-stats +syslog.=debug /var/log/rsyslog-stats </textarea> +<p>In the next sample, the default interval of 5 minutes is used. However, this time +stats data is NOT emitted to the syslog stream but to a local file instead. +<p> +<textarea rows="3" cols="70">module(load="impstats" interval="600" severity="7" + log.syslog="off" /* need to turn log stream logging off! */ + log.file="/path/to/local/stats.log") +</textarea> +<p>And finally, we log to both the regular syslog log stream as well as a file. +Within the log stream, we forward the data records to another server: +<p> +<textarea rows="4" cols="70">module(load="impstats" interval="600" severity="7" + log.file="/path/to/local/stats.log") + +syslog.=debug @central.example.net +</textarea> + <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imptcp.html b/doc/imptcp.html index 107dd306..b5bd0970 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -13,11 +13,9 @@ <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via plain TCP syslog. This is a specialised input plugin tailored for high performance on Linux. It will -probably not run on any other platform. Also, it does no provide TLS services. +probably not run on any other platform. Also, it does not provide TLS services. Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a>. <p>This module has no limit on the number of listeners and sessions that can be used. -<p>Multiple receivers may be configured by -specifying $InputPTCPServerRun multiple times. </p> <p><b>Configuration Directives</b>:</p> @@ -95,6 +93,13 @@ the message was received from. Binds specified ruleset to next server defined. <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imtcp.html b/doc/imtcp.html index feb0bdd7..b9f0b056 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -17,10 +17,6 @@ Encryption is natively provided by selecting the approprioate network stream driver and can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> (an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> -<p>Multiple receivers may be configured by specifying -$InputTCPServerRun multiple times. This is available since version 4.3.1, earlier -versions do NOT support it. -</p> <p><b>Configuration Directives</b>:</p> <p><b>Global Directives</b>:</p> @@ -100,6 +96,13 @@ activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imudp.html b/doc/imudp.html index 1ebce4d7..f2d04d65 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -47,6 +47,13 @@ default 514, start UDP server on this port. Either a single port can be specifie <br>Array of ports: Port=["514","515","10514","..."]</li> <li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html new file mode 100644 index 00000000..d72810f1 --- /dev/null +++ b/doc/lookup_tables.html @@ -0,0 +1,205 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Lookup Tables</title> +</head> + +<body> +<h1>Lookup Tables</h1> + +<p><b><font color="red">NOTE: this is</font> proposed functionality, which is +<font color="red">NOT YET IMPLEMENTED</font>!</b> + +<p><b>Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices).</b> +<p>The base idea is to use a message variable as an index into a table which then +returns another value. For example, $fromhost-ip could be used as an index, with +the table value representing the type of server or the department or remote office +it is located in. A main point with lookup tables is that the lookup is very fast. +So while lookup tables can be emulated with if-elseif constructs, they are generally +much faster. Also, it is possible to reload lookup tables during rsyslog runtime without +the need for a full restart. +<p>The lookup tables itself exists in a separate configuration file (one per table). This +file is loaded on rsyslog startup and when a reload is requested. +<p>There are different types of lookup tables: +<ul> +<li><b>string</b> - the value to be looked up is an arbitrary string. Only exact +some strings match. +<li><b>array</b> - the value to be looked up is an integer number from a consequtive set. +The set does not need to start at zero or one, but there must be no number missing. So, for example +5,6,7,8,9 would be a valid set of index values, while 1,2,4,5 would not be (due to missing +2). +A match happens if the requested number is present. +<li><b>sparseArray</b> - the value to be looked up is an integer value, but there may +be gaps inside the set of values (usually there are large gaps). A typical use case would +be the matching of IPv4 address information. A match happens on the first value that is +less than or equal to the requested value. +</ul> +<p>Note that index integer numbers are represented by unsigned 32 bits. +<p>Lookup tables can be access via the lookup() built-in function. The core idea is to +set a local variable to the lookup result and later on use that local variable in templates. +<p>More details on usage now follow. +<h2>Lookup Table File Format</h2> +<p>Lookup table files contain a single JSON object. This object contains of a header and a +table part. +<h3>Header</h3> +<p>The header is the top-level json. It has paramters "version", "nomatch", and "type". +The version parameter +must be given and must always be one for this version of rsyslog. The nomatch +parameter is optional. If specified, it contains the value to be used if lookup() +is provided an index value for which no entry exists. The default for +"nomatch" is the empty string. Type specifies the type of lookup to be done. +<h3>Table</h3> +This must be an array of elements, even if only a single value exists (for obvious +reasons, we do not expect this to occur often). Each array element must contain two +fields "index" and "value". +<h3>Example</h3> +<p>This is a sample of how an ip-to-office mapping may look like: +<pre> +{ "version":1, "nomatch":"unk", "type":"string", + "table":[ {"index":"10.0.1.1", "value":"A" }, + {"index":"10.0.1.2", "value":"A" }, + {"index":"10.0.1.3", "value":"A" }, + {"index":"10.0.2.1", "value":"B" }, + {"index":"10.0.2.2", "value":"B" }, + {"index":"10.0.2.3", "value":"B" } + ] +} +</pre> +Note: if a different IP comes in, the value "unk" +is returend thanks to the nomatch parameter in +the first line. +<p> +<h2>RainerScript Statements</h2> +<h3>lookup_table() Object</h3> +<p>This statement defines and intially loads a lookup table. Its format is +as follows: +<pre> +lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (mandatory)<br> + Defines the name of lookup table for further reference + inside the configuration. Names must be unique. Note that + it is possible, though not advisible, to have different + names for the same file. + <li><b>file</b> (mandatory)<br> + Specifies the full path for the lookup table file. This file + must be readable for the user rsyslog is run under (important + when dropping privileges). It must point to a valid lookup + table file as described above. + <li><b>reloadOnHUP</b> (optional, default "on")<br> + Specifies if the table shall automatically be reloaded + as part of HUP processing. For static tables, the + default is "off" and specifying "on" triggers an + error message. Note that the default of "on" may be + somewhat suboptimal performance-wise, but probably + is what the user intuitively expects. Turn it off + if you know that you do not need the automatic + reload capability. +</ul> + +<h3>lookup() Function</h3> +<p>This function is used to actually do the table lookup. Format: +<pre> +lookup_table("name", indexvalue) +</pre> +<h4>Parameters</h4> +<ul> + <li><b>return value</b><br> + The function returns the string that is associated with the + given indexvalue. If the indexvalue is not present inside the + lookup table, the "nomatch" string is returned (or an empty string + if it is not defined). + <li><b>name</b> (constant string)<br> + The lookup table to be used. Note that this must be specificed as a + constant. In theory, variable table names could be made possible, but + their runtime behaviour is not as good as for static names, and we do + not (yet) see good use cases where dynamic table names could be useful. + <li><b>indexvalue</b> (expression)<br> + The value to be looked up. While this is an arbitrary RainerScript expression, + it's final value is always converted to a string in order to conduct + the lookup. For example, "lookup(table, 3+4)" would be exactly the same + as "lookup(table, "7")". In most cases, indexvalue will probably be + a single variable, but it could also be the result of all RainerScript-supported + expression types (like string concatenation or substring extraction). + Valid samples are "lookup(name, $fromhost-ip & $hostname)" or + "lookup(name, substr($fromhost-ip, 0, 5))" as well as of course the + usual "lookup(table, $fromhost-ip)". +</ul> + + +<h3>load_lookup_table Statement</h3> + +<p><b>Note: in the final implementation, this MAY be implemented as an action. +This is a low-level decesion that must be made during the detail development +process. Parameters and semantics will remain the same of this happens.</b> + +<p>This statement is used to reload a lookup table. It will fail if +the table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight performance +hit during the load phase. It is assume that always a triggering condition +is used to load the table. +<pre> +load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (string)<br> + The lookup table to be used. + <li><b>errOnFail</b> (boolean, default "on")<br> + Specifies whether or not an error message is to be emitted if + there are any problems reloading the lookup table. + <li><b>valueOnFail</b> (optional, string)<br> + This parameter affects processing if the lookup table cannot + be loaded for some reason: If the parameter is not present, + the previous table will be kept in use. If the parameter is + given, the previous table will no longer be used, and instead + an empty table be with nomath=valueOnFail be generated. In short, + that means when the parameter is set and the reload fails, + all matches will always return what is specified in valueOnFail. +</ul> + +<h3>Usage example</h3> +<p>For clarity, we show only those parts of rsyslog.conf that affect +lookup tables. We use the remote office example that an example lookup +table file is given above for. +<pre> +lookup_table(name="ip2office" file="/path/to/ipoffice.lu" + reloadOnHUP="off") + + +template(name="depfile" type="string" + string="/var/log/%$usr.dep%/messages") + +set $usr.dep = lookup("ip2office", $fromhost-ip); +action(type="omfile" dynfile="depfile") + +# support for reload "commands" +if $fromhost-ip == "10.0.1.123" + and $msg contains "reload office lookup table" + then + load_lookup_table(name="ip2office" errOnFail="on") +</pre> + +<p>Note: for performance reasons, it makes sense to put the reload command into +a dedicated ruleset, bound to a specific listener - which than should also +be sufficiently secured, e.g. via TLS mutual auth. + +<h2>Implementation Details</h2> +<p>The lookup table functionality is implemented via highly efficient algorithms. +The string lookup is based on a parse tree and has O(1) time complexity. The array +lookup is also O(1). In case of sparseArray, we have O(log n). +<p>To preserve space and, more important, increase cache hit performance, equal +data values are only stored once, no matter how often a lookup index points to them. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/manual.html b/doc/manual.html index 78b5d006..1d8e1f19 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -13,13 +13,13 @@ It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its <a href="features.html"> advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_tls.html">encryption protected syslog</a> relay chains while at the same time being very easy to setup for the -novice user. And as we know what enterprise users really need, there is -also <a href="http://www.rsyslog.com/professional-services">professional -rsyslog support</a> available directly from the source!</p> +novice user. And as we know what enterprise users really need, there are +also <a href="http://www.rsyslog.com/professional-services"> rsyslog +professional services</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 7.2.5 (v7-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 7.3.5 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might diff --git a/doc/mmjsonparse.html b/doc/mmjsonparse.html new file mode 100644 index 00000000..c2c862d7 --- /dev/null +++ b/doc/mmjsonparse.html @@ -0,0 +1,45 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>CEE/lumberjack JSON support Module (mmjsonparse)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Log Message Normalization Module</h1> +<p><b>Module Name: mmjsonparse</b></p> +<p><b>Available since: </b>6.6.0+ +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides support for parsing structured log messages +that follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked +and, if present, the JSON-encoded structured message content is parsed. +The properties are than available as original message properties. +</p> +<p><b>Action specific Configuration Directives</b>:</p> +<p>currently none +<ul> +<p><b>Legacy Configuration Directives</b>:</p> +<p>none +<b>Caveats/Known Bugs:</b> +<p>None known at this time. +</ul> +<p><b>Sample:</b></p> +<p>This activates the module and applies normalization to all messages:<br> +</p> +<textarea rows="2" cols="60">module(load="mmjsonparse") +action(type="mmjsonparse") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="2" cols="60">$ModLoad mmjsonparse +*.* :mmjsonparse: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2012 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html index 82f9b6a2..787bd957 100644 --- a/doc/mmnormalize.html +++ b/doc/mmnormalize.html @@ -11,37 +11,53 @@ <p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> <p><b>Description</b>:</p> <p>This module provides the capability to normalize log messages via -<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to libee, unstructured text, +<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to liblognorm, unstructured text, like usually found in log messages, can very quickly be parsed and put into -a normal form. This is done so quickly, that it usually should be possible +a normal form. This is done so quickly, that it should be possible to normalize events in realtime. -<p>This module is implemented via the output module interface. That means that +<p>This module is implemented via the output module interface. This means that mmnormalize should be called just like an action. After it has been called, -the normalized message properties are avaialable and can be access. These properties -are called the "CEE" properties, because liblognorm creates a format that is -inspired by the CEE approach. +the normalized message properties are avaialable and can be accessed. These properties +are called the "CEE/lumberjack" properties, because liblognorm creates a format that is +inspired by the CEE/lumberjack approach. +<p><b>Please note:</b> CEE/lumberjack properties are different from regular properties. +They have always "$!" prepended to the property name given in the rulebase. Such a +property needs to be called with <b>%$!propertyname%</b>. <p>Note that mmnormalize should only be called once on each message. Behaviour is -undifined if multiple calls to mmnormalize happen for the same message. +undefined if multiple calls to mmnormalize happen for the same message. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> -<li>$mmnormalizeRuleBase <rulebase-file><br> -Specifies which rulebase file is to use. This file is loaded. If there are +<li><b>ruleBase</b> [word]<br> +Specifies which rulebase file is to use. If there are multiple mmnormalize instances, each one can use a different file. However, a single instance can use only a single file. This parameter MUST be given, -because normalization can only happen based on a rulebase. -<li>$mmnormalizeUseRawMsg <on/off><br> +because normalization can only happen based on a rulebase. It is recommended +that an absolute path name is given. Information on how to create the rulebase +can be found in the <a href="http://www.liblognorm.com/files/manual/index.html">liblognorm manual</a>. +<li><b>useRawMsg</b> [boolean]<br> Specifies if the raw message should be used for normalization (on) or just the MSG part of the message (off). Default is "off". </ul> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li>$mmnormalizeRuleBase <rulebase-file> - equivalent to the "ruleBase" +parameter. +<li>$mmnormalizeUseRawMsg <on/off> - equivalent to the "useRawMsg" +parameter. +</ul> <b>Caveats/Known Bugs:</b> <p>None known at this time. </ul> <p><b>Sample:</b></p> <p>This activates the module and applies normalization to all messages:<br> </p> -<textarea rows="8" cols="60">$ModLoad mmnormalize -$mmnormalizeRuleBase rulebase.rb +<textarea rows="2" cols="60">module(load="mmnormalize") +action(type="mmnormalize" ruleBase="/path/to/rulebase.rb") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="3" cols="60">$ModLoad mmnormalize +$mmnormalizeRuleBase /path/to/rulebase.rb *.* :mmnormalize: </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] @@ -49,7 +65,7 @@ $mmnormalizeRuleBase rulebase.rb <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2010-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omfile.html b/doc/omfile.html index 23ecc034..75ac5f49 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -13,14 +13,14 @@ <p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> <p> </p> -<p><b>Global Configuration Directives</b>:</p> +<p><b>Module Configuration Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> sets a new default template for file actions.<br></li> </ul> <p> </p> -<p><b>Action specific Configuration Directives</b>:</p> +<p><b>Action Confguration Parameters</b>:</p> <ul> <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -28,6 +28,16 @@ <li><strong>ZipLevel </strong>0..9 [default 0]<br> if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + <li><b>VeryReliableZip</b> [<b>on</b>/off] (v7.3.0+) - if ZipLevel is greater 0, + then this setting controls if extra headers are written to make the resulting file + extra hardened against malfunction. If set to off, data appended to previously unclean + closed files may not be accessible without extra tools. + Note that this risk is usually expected to be bearable, and thus "off" is the default mode. + The extra headers considerably + degrade compression, files with this option set to "on" may be four to five times as + large as files processed in "off" mode. + </li><br> + <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -160,7 +170,7 @@ $FileCreateMode 0644 index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omfwd.html b/doc/omfwd.html index 5599ae39..fb9145e1 100644 --- a/doc/omfwd.html +++ b/doc/omfwd.html @@ -16,7 +16,7 @@ <p><b>Global Configuration Directives</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li> + sets a non-standard default template for this module.<br></li> </ul> <p> </p> diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html index 008dcb81..e47c7f57 100644 --- a/doc/omlibdbi.html +++ b/doc/omlibdbi.html @@ -54,7 +54,23 @@ dlopen()ed plugin (as omlibdbi is). So in short, you probably save you a lot of headache if you make sure you have at least libdbi version 0.8.3 on your system. </p> -<p><b>Action Parameters</b>:</p> +<p><b>Module Parameters</b></p> +<ul> +<li><b>template</b><br> +The default template to use. This template is used when no template is +explicitely specified in the action() statement. +<li><b>driverdirectory</b><br> +Path to the libdbi drivers. Usually, +you do not need to set it. If you installed libdbi-drivers at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <b>not</b> use this configuration directive. +Usually, everything works just fine. +Note that this was an action() paramter in rsyslog versions below 7.3.0. +However, only the first action's driverdirectory parameter was actually used. +This has been cleaned up in 7.3.0, where this now is a module paramter. +</li> +</ul> +<p><b>Action Parameters</b></p> <ul> <li><b>server</b><br>Name or address of the MySQL server <li><b>db</b><br>Database to use @@ -68,24 +84,18 @@ writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", "freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use ompgsql instead).</li> -<li><b>driverdirectory</b><br> -Path to the libdbi drivers. Usually, -you do not need to set it. If you installed libdbi-drivers at a -non-standard location, you may need to specify the directory here. If -you are unsure, do <b>not</b> use this configuration directive. -Usually, everything works just fine.</li> </ul> <p><b>Legacy (pre-v6) Configuration Directives</b>:</p> +<p>It is strongly recommended NOT to use legacy format. <ul> -<li><b>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</b> +<li><i>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</i> - like the driverdirectory action parameter. -<li><strong>$ActionLibdbiDriver drivername</strong><br> - like the drivername action parameter. -<li><span style="font-weight: bold;">$ActionLibdbiHost hostname</span> - like the server action parameter -The host to connect to.</li> -<li><b>$ActionLibdbiUserName user</b> - like the uid action parameter -<li><b>$ActionlibdbiPassword</b> - like the pwd action parameter -<li><b>$ActionlibdbiDBName db</b> - like the db action parameter -<li><b>selector line: :omlibdbi:<i>;template</i></b><br> +<li><i>$ActionLibdbiDriver drivername</i> - like the drivername action parameter +<li><i>$ActionLibdbiHost hostname</i> - like the server action parameter +<li><i>$ActionLibdbiUserName user</i> - like the uid action parameter +<li><i>$ActionlibdbiPassword</i> - like the pwd action parameter +<li><i>$ActionlibdbiDBName db</i> - like the db action parameter +<li><i>selector line: :omlibdbi:<code>;template</code></i><br> executes the recently configured omlibdbi action. The ;template part is optional. If no template is provided, a default template is used (which is currently optimized for MySQL - sorry, folks...)</li> @@ -114,14 +124,14 @@ database "syslog_db" on mysqlsever.example.com. The server is MySQL and being accessed under the account of "user" with password "pwd" (if you have empty passwords, just remove the $ActionLibdbiPassword line).<br> </p> -<textarea rows="5" cols="60">$ModLoad omlibdbi +<textarea rows="5" cols="60">module(load="omlibdbi") *.* action(type="omlibdbi" driver="mysql" server="mysqlserver.example.com" db="syslog_db" uid="user" pwd="pwd" </textarea> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>The same as above, but in legacy config format (pre rsyslog-v6): -<textarea rows="10" cols="60">$ModLoad omlibdbi +<textarea rows="8" cols="60">$ModLoad omlibdbi $ActionLibdbiDriver mysql $ActionLibdbiHost mysqlserver.example.com $ActionLibdbiUserName user diff --git a/doc/ommongodb.html b/doc/ommongodb.html new file mode 100644 index 00000000..a6112642 --- /dev/null +++ b/doc/ommongodb.html @@ -0,0 +1,60 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>MongoDB Output Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>MongoDB Output Module</h1> +<p><b>Module Name: ommongodb</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides native support for logging to MongoDB. +</p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>server</b><br>Name or address of the MongoDB server +<li><b>serverport</b><br>Permits to select +a non-standard port for the MongoDB server. The default is 0, which means the +system default port is used. There is no need to specify this parameter unless +you know the server is running on a non-standard listen port. +<li><b>db</b><br>Database to use +<li><b>collection</b><br>Collection to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +</ul> +<p>Note rsyslog contains a canned default template to write to the MongoDB. It +will be used automatically if no other template is specified to be used. This template is: +<p> +<textarea rows="5" cols="80">template(name="BSON" type="string" string="\"sys\" : \"%hostname%\", \"time\" : \"%timereported:::rfc3339%\", \"time_rcvd\" : \"%timegenerated:::rfc3339%\", \"msg\" : \"%msg%\", \"syslog_fac\" : \"%syslogfacility%\", \"syslog_sever\" : \"%syslogseverity%\", \"syslog_tag\" : \"%syslogtag%\", \"procid\" : \"%programname%\", \"pid\" : \"%procid%\", \"level\" : \"%syslogpriority-text%\"") +</textarea> +<p>This creates the BSON document needed for MongoDB if no template is specified. The default +schema is aligned to CEE and project lumberjack. As such, the field names are standard +lumberjack field names, and <b>not</b> +<a href="property_replacer.html">rsyslog property names</a>. When specifying templates, be sure +to use rsyslog property names as given in the table. If you would like to use lumberjack-based +field names inside MongoDB (which probably is useful depending on the use case), you need to +select fields names based on the lumberjack schema. +If you just want to use a subset of the fields, but with lumberjack names, you can look up the +mapping in the default template. For example, the lumberjack field "level" contains the rsyslog +property "syslogpriority-text". +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog" and into the collection "log" on mongosever.example.com. The server is +being accessed under the account of "user" with password "pwd". +</p> +<textarea rows="5" cols="80">module(load="ommongodb") +*.* action(type="ommongodb" server="mongoserver.example.com" db="syslog" collection="log" uid="user" pwd="pwd") +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the ASL 2.0.</font></p> +</body></html> diff --git a/doc/omudpspoof.html b/doc/omudpspoof.html index df14bbe1..930412c8 100644 --- a/doc/omudpspoof.html +++ b/doc/omudpspoof.html @@ -7,46 +7,161 @@ <h1>UDP spoofing output module (omudpspoof)</h1> <p><b>Module Name: omstdout</b></p> -<p><b>Author: </b>David Lang <david@lang.hm> and Rainer Gerhards -<rgerhards@adiscon.com></p> -<p><b>Available Since</b>: 5.1.3</p> +<p><b>Authors: </b>Rainer Gerhards <rgerhards@adiscon.com> +and David Lang <david@lang.hm> +</p> +<p><b>Available Since</b>: 5.1.3 / v7 config since 7.2.5</p> <p><b>Description</b>:</p> <p>This module is similar to the regular UDP forwarder, but permits to spoof the sender address. Also, it enables to circle through a number of source ports. -<p><b>Configuration Directives</b>:</p> +<p><b>Important:</b> This module requires root priveleges for its low-level +socket access. As such, the <b>module will not work if rsyslog is configured to +drop privileges</b>. + +<p><b>load() Parameters</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a non-standard default template for this module.<br></li> + +</ul> +<p> </p> +<p><b>action() parameters</b>:</p> +<ul> + <li><strong>Target </strong>string<br> + Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> + + <li><strong>Port </strong>[Integer, Default 514]<br> + Name or numerical value of port to use when connecting to target. <br></li><br> + + <li><b>Template</b>[Word]<br> + Template to use as message text. + <br></li><br> + + <li><strong>SourceTemplate </strong>[Word]<br> + This is the name of the template that contains a + numerical IP address that is to be used as the source system IP address. + While it may often be a constant value, it can be generated as usual via the + property replacer, as long as it is a valid IPv4 address. If not specified, the + build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined + as follows:<br> + template(name="RSYSLOG_omudpspoofDfltSourceTpl" type="string" string="%fromhost-ip%")<br> + So in essence, the default template spoofs the address of the system the message + was received from. This is considered the most important use case. + <br></li><br> + + <li><b>SourcePortStart</b>[Word]<br> + Specifies the start value for circeling the source ports. Must be less than or + equal to the end value. Default is 32000. + <br></li><br> + + <li><b>SourcePortEnd</b>[Word]<br> + Specifies the ending value for circeling the source ports. Must be less than or + equal to the start value. Default is 42000. + <br></li><br> + + <li><b>mtu</b>[Integer, default 1500]<br> + Maximum MTU supported by the network. Default respects Ethernet and must + usually not be adjusted. Setting a too-high MTU can lead to message loss, + too low to excess message fragmentation. Change only if you really know what + you are doing. This is always given in number of bytes. + <br></li><br> +</ul> +<p><b>pre-v7 Configuration Directives</b>:</p> <ul> -<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename><br> -This is the name of the template that contains a -numerical IP address that is to be used as the source system IP address. -While it may often be a constant value, it can be generated as usual via the -property replacer, as long as it is a valid IPv4 address. If not specified, the -build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined -as follows:<br> -$template RSYSLOG_omudpspoofDfltSourceTpl,"%fromhost-ip%"<br> -So in essence, the default template spoofs the address of the system the message -was received from. This is considered the most important use case. -<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname><br> -Host that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofTargetPort</b> <port><br> -Remote port that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename><br> -This setting instructs omudpspoof to use a template different from the -default template for all of its actions that do not have a template specified -explicitely. -<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number><br> -Specifies the start value for circeling the source ports. Must be less than or -equal to the end value. Default is 32000. -<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number><br> -Specifies the ending value for circeling the source ports. Must be less than or -equal to the start value. Default is 42000. +<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename> +- equivalent to the "sourceTemplate" parameter. +<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofTargetPort</b> <port> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename> +- equivalent to the "template" load() parameter. +<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number> +- equivalent to the "SourcePortStart" parameter. +<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number> +- equivalent to the "SourcePortEnd" parameter. </ul> <b>Caveats/Known Bugs:</b> <ul> <li><b>IPv6</b> is currently not supported. If you need this capability, please let us know via the rsyslog mailing list. +<li>Versions shipped prior to rsyslog 7.2.5 do not support message sizes over 1472 bytes (more +pricesely: over the network-supported MTU). Starting with 7.2.5, those messages will be +fragmented, up to a total upper limit of 64K (induced by UDP). Message sizes over +64K will be truncated. For older versions, messages over 1472 may be totally discarded +or truncated, depending on version and environment. </ul> -<p><b>Sample:</b></p> + +<p><b>Config Samples</b></p> +<p>The following sample forwards all syslog messages in standard form to the +remote server server.example.com. The original sender's address is used. We do not +care about the source port. This example is considered the typical use case for +omudpspoof. +</p> +<textarea rows="3" cols="80">module(load="omudpspoof") +action(type="omudpspoof" target="server.example.com") +</textarea> + +<p>The following sample forwards all syslog messages in unmodified form to the +remote server server.example.com. The sender address 192.0.2.1 with fixed +source port 514 is used. +</p> +<textarea rows="7" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514) +</textarea> +<p>The following sample is exatly like the previous, but it specifies a larger size +MTU. If, for example, the envrionment supports Jumbo Ethernet frames, increasing the +MTU is useful as it reduces packet fragmentation, which most often is the source of +problems. Note that setting the MTU to a value larger than the local-attached network +supports will lead to send errors and loss of message. So use with care! +</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>Of course, the action can be combined with any type of filter, for +example a tradition PRI filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +local0.* action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>... or any complex expression-based filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>and of course it can also be combined with as many other actions +as one likes:</p> +<textarea rows="11" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then { + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") + action(type="omfile" file="/var/log/somelog") + stop # or whatever... +} +</textarea> + +<p><b>Legacy Sample (pre-v7):</b></p> <p>The following sample forwards all syslog messages in standard form to the remote server server.example.com. The original sender's address is used. We do not care about the source port. This example is considered the typical use case for diff --git a/doc/rainerscript.html b/doc/rainerscript.html index 84261bdd..c9f00386 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -51,6 +51,11 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. +<h2>Lookup Tables</h2> +<p><a href="lookup_tables.html">Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices). <h2>Functions</h2> <p>RainerScript supports a currently quite limited set of functions: <ul> @@ -80,7 +85,7 @@ if $msg contains getenv('TRIGGERVAR') then /path/to/errfile <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html index ef29c2a8..433456c1 100644 --- a/doc/rsconf1_rulesetparser.html +++ b/doc/rsconf1_rulesetparser.html @@ -9,7 +9,7 @@ <p><b>Type:</b> ruleset-specific configuration directive</p> <p><b>Parameter Values:</b> string</p> <p><b>Available since:</b> 5.3.4+</p> -<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc5425</p> +<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc3164</p> <p><b>Description:</b></p> <p> This directive permits to specify which diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 4186dac4..cb31bd9a 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -72,6 +72,7 @@ permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> <li><a href="omuxsock.html">omuxsock</a> - output module Unix domain sockets</li> <li><a href="omhdfs.html">omhdfs</a> - output module for Hadoop's HDFS file system</li> +<li><a href="ommongodb.html">ommongodb</a> - output module for MongoDB</li> </ul> <a name="pm"></a><h2>Parser Modules</h2> @@ -111,6 +112,8 @@ modules exist inside the source tree: <ul> <li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. Note that this actually is a <b>generic</b> module. +<li><a href="mmjsonparse.html">mmjsonparse</a> - used to interpret CEE/lumberjack +enabled structured log messages. <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. diff --git a/doc/v7compatibility.html b/doc/v7compatibility.html index 01faacac..da4772fe 100644 --- a/doc/v7compatibility.html +++ b/doc/v7compatibility.html @@ -82,6 +82,53 @@ errors and error messages. Starting with 7.2.1, this has been reduced to 10 successive failures. This still gives the plugin a chance to recover. In extreme cases, a plugin may now enter suspend mode where it previously did not do so. In practice, we do NOT expect that. +<h1>Notes for the 7.3/7.4 branch</h1> +<h2>"last message repeated n times" Processing</h2> +<p>This processing has been optimized and moved to the input side. This results +in usually far better performance and also de-couples different sources +from the same +processing. It is now also integrated in to the more generic rate-limiting +processing. +<h3>User-Noticable Changes</h3> +The code works almost as before, with two exceptions: +<ul> +<li>The supression amount can be different, as the new algorithm + precisely check's a single source, and while that source is being + read. The previous algorithm worked on a set of mixed messages + from multiple sources. +<li>The previous algorithm wrote a "last message repeated n times" message + at least every 60 seconds. For performance reasons, we do no longer do + this but write this message only when a new message arrives or rsyslog + is shut down. +</ul> +<p>Note that the new algorithms needs support from input modules. If old +modules which do not have the necessary support are used, duplicate +messages will most probably not be detected. Upgrading the module code is +simple, and all rsyslog-provided plugins support the new method, so this +should not be a real problem (crafting a solution would result in rather +complex code - for a case that most probably would never happen). +<h3>Performance Implications</h3> +<p>In general, the new method enables far faster output procesing. However, it +needs to be noted that the "last message repeated n" processing needs parsed +messages in order to detect duplicated. Consequently, if it is enabled the +parser step cannot be deferred to the main queue processing thread and +thus must be done during input processing. The changes workload distribution +and may have (good or bad) effect on the overall performance. If you have +a very high performance installation, it is suggested to check the performance +profile before deploying the new version. Note: for high-performance +environments it is highly recommended NOT to use "last message repeated n times" +processing but rather the other (more efficient) rate-limiting methods. These +also do NOT require the parsing step to be done during input processing. + +<h2>Stricter string-template Processing</h2> +<p>Previously, no error message for invalid string template parameters +was generated. +Rather a malformed template was generated, and error information emitted +at runtime. However, this could be quite confusing. Note that the new code +changes user experience: formerly, rsyslog and the affected +actions properly started up, but the actions did not produce proper +data. Now, there are startup error messages and the actions are NOT +executed (due to missing template due to template error). <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> |