diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/Makefile.am | 3 | ||||
| -rw-r--r-- | doc/imfile.html | 19 | ||||
| -rw-r--r-- | doc/imkmsg.html | 48 | ||||
| -rw-r--r-- | doc/imptcp.html | 16 | ||||
| -rw-r--r-- | doc/imrelp.html | 2 | ||||
| -rw-r--r-- | doc/imtcp.html | 95 | ||||
| -rw-r--r-- | doc/imudp.html | 12 | ||||
| -rw-r--r-- | doc/imuxsock.html | 55 | ||||
| -rw-r--r-- | doc/manual.html | 4 | ||||
| -rw-r--r-- | doc/rainerscript.html | 8 | ||||
| -rw-r--r-- | doc/rsyslog_conf_filter.html | 178 | ||||
| -rw-r--r-- | doc/rsyslog_conf_templates.html | 497 | ||||
| -rw-r--r-- | doc/v4compatibility.html | 2 | ||||
| -rw-r--r-- | doc/v7compatibility.html | 60 |
14 files changed, 739 insertions, 260 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index 1ae1c68d..bded9453 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -145,6 +145,9 @@ html_files = \ rsyslog_queue_pointers.jpeg \ rsyslog_queue_pointers2.jpeg \ v6compatibility.html \ + v7compatibility.html \ + rsyslog_conf_basic_structure.html \ + imkmsg.html \ src/classes.dia grfx_files = \ diff --git a/doc/imfile.html b/doc/imfile.html index 1594cdce..0997e382 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -91,7 +91,6 @@ textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for "info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is "notice".</li> <li><b>PersistStateInterval</b> [lines]</b><br> -Available in 4.7.3+, 5.6.2+<br> Specifies how often the state file shall be written when processing the input file. The default value is 0, which means a new state file is only written when the monitored files is being closed (end of rsyslogd execution). Any other @@ -101,9 +100,11 @@ to fatal errors (like power fail). Note that this setting affects imfile performance, especially when set to a low value. Frequently writing the state file is very time consuming. <li><b>ReadMode</b> [mode]</b><br> -Available in 5.7.5+ -<li><b>MaxLinesAtOnce</b> [number]</b><br> -Available in 5.9.0+ +This mode should defined when having multiline messages. The value can range from 0-2 and determines the multiline detection method. +<br>0 (default) - line based (Each line is a new message) +<br>1 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it) +<br>2 - paragraph (There is a blank line between log messages) +<li><b>MaxLinesAtOnce</b> [number]</b> <br> This is useful if multiple files need to be monitored. If set to 0, each file will be fully processed and then processing switches to the next file @@ -112,16 +113,14 @@ will be fully processed and then processing switches to the next file switched. This provides a kind of mutiplexing the load of multiple files and probably leads to a more natural distribution of events when multiple busy files are monitored. The default is 1024. -<li><b>MaxSubmitAtOnce</b> [number]</b><br> -Available in 5.9.0+ +<li><b>MaxSubmitAtOnce</b> [number]</b> <br> This is an expert option. It can be used to set the maximum input batch size that imfile can generate. The default is 1024, which is suitable for a wide range of applications. Be sure to understand rsyslog message batch processing before you modify this option. If you do not know what this doc here talks about, this is a good indication that you should NOT modify the default. -<li><b>Ruleset</b> <ruleset><br> -Available in 5.7.5+, 6.1.5+ +<li><b>Ruleset</b> <ruleset> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> @@ -181,12 +180,16 @@ directive, no file monitoring will take place.</li> seconds</span><br> equivalent to: PollingInterva</li> <li><b>$InputFilePersistStateInterval</b> [lines]</b><br> +Available in 4.7.3+, 5.6.2+<br> equivalent to: PersistStateInterval <li><b>$InputFileReadMode</b> [mode]</b><br> +Available in 5.7.5+<br> equivalent to: ReadMode <li><b>$InputFileMaxLinesAtOnce</b> [number]</b><br> +Available in 5.9.0+<br> equivalent to: MaxLinesAtOnce <li>$InputFileBindRuleset <ruleset><br> +Available in 5.7.5+, 6.1.5+<br> equivalent to: Ruleset </li> </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imkmsg.html b/doc/imkmsg.html new file mode 100644 index 00000000..61068d09 --- /dev/null +++ b/doc/imkmsg.html @@ -0,0 +1,48 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>/dev/kmsg Log Input Module (imkmsg)</title> + +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>/dev/kmsg Log Input Module</h1> +<p><b>Module Name: imkmsg</b></p> +<p><b>Authors: </b>Rainer Gerhards +<rgerhards@adiscon.com><br /> +Milan Bartos +<mbartos@redhat.com></p> +<p><b>Description</b>:</p> +<p>Reads messages from the /dev/kmsg structured kernel log and submits them to the +syslog engine.</p> +<p> +The printk log buffer constains log records. These records are exported by /dev/kmsg +device as structured data in the following format:<br /> + "level,sequnum,timestamp;<message text>\n"<br /> +There could be continuation lines starting with space that contains key/value pairs.<br /> +<br /> +Log messages are parsed as necessary into rsyslog msg_t structure. Continuation lines are parsed +as json key/value pairs and added into rsyslog's message json representation. +</p> +<p><b>Configuration Directives</b>:</p> +This module has no configuration directives. +<b>Caveats/Known Bugs:</b> +<p>This is Linux specific module and requires /dev/kmsg device with structured kernel logs. +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the /dev/kmsg log device. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling messages.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imkmsg +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imptcp.html b/doc/imptcp.html index d301b76f..7e712afa 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -24,6 +24,17 @@ specifying $InputPTCPServerRun multiple times. <p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> + +<p><b>Global Directives</b>:</p> +<ul> +<li>Threads <number><br> +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +helper threads (but not more than there are CPUs/Cores) can help improving +performance. The default value is two. +</ul> +<p><b>Action Directives</b>:</p> +<ul> <li><b>AddTLFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, @@ -78,11 +89,6 @@ name is not strictly necessary, but can be useful to apply filtering based on wh the message was received from. <li><b>Ruleset</b> <name><br> Binds specified ruleset to next server defined. -<!--<li>$InputPTCPHelperThreads <number><br> -Number of helper worker threads to process incoming messages. These -threads are utilized to pull data off the network. On a busy system, additional -helper threads (but not more than there are CPUs/Cores) can help improving -performance. The default value is two.--> <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. </ul> diff --git a/doc/imrelp.html b/doc/imrelp.html index 80ddfd53..856aff82 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -30,7 +30,7 @@ Clients send messages to the RELP server via omrelp.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><b>Ruleset</b> <name> (available in 6.3.6+)</br> +<li><b>Ruleset</b> <name></br> Binds the specified ruleset to all RELP listeners. <li><b>Port</b> <port><br> Starts a RELP server on selected port</li> diff --git a/doc/imtcp.html b/doc/imtcp.html index 649b08f8..01ea2802 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -21,9 +21,11 @@ can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> $InputTCPServerRun multiple times. This is available since version 4.3.1, earlier versions do NOT support it. </p> + <p><b>Configuration Directives</b>:</p> +<p><b>Global Directives</b>:</p> <ul> -<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +<li><b>AddtlFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -43,7 +45,7 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. -<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +<li><b>DisableLFDelimiter</b> <on/<b>off</b>><br> Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames. However, some users brought up the case that it may be useful to define a different delimiter and totally disable LF as a delimiter (the use case named were multi-line messages). This mode @@ -51,16 +53,14 @@ is non-standard and will probably come with a lot of problems. However, as there for it and it is relatively easy to support, we do so. Be sure to turn this setting to "on" only if you exactly know what you are doing. You may run into all sorts of troubles, so be prepared to wrangle with that! -<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +<li><b>NotifyOnConnectionClose</b> [on/<b>off</b>]<br> instructs imtcp to emit a message if the remote peer closes a connection.<br> <b>Important:</b> This directive is global to all listeners and must be given right after loading imtcp, otherwise it may have no effect.</li> -<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +<li><b>KeepAlive</b> <on/<b>off</b>><br> enable of disable keep-alive packets at the tcp socket layer. The default is to disable them.</li> -<li><b>$InputTCPServerRun</b> <port><br> -Starts a TCP server on selected port</li> -<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +<li><b>FlowControl</b> <<b>on</b>/off><br> This setting specifies whether some message flow control shall be exercised on the related TCP input. If set to on, messages are handled as "light delayable", which means the sender is throttled a bit when the queue becomes near-full. This is done in order @@ -69,24 +69,32 @@ may have some undesired effect in some configurations. Still, we consider this a a useful setting and thus it is the default. To turn the handling off, simply configure that explicitely. </li> -<li><b>$InputTCPMaxListeners</b> <number><br> +<li><b>MaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li><b>$InputTCPMaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> -<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +<li><b>MaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>StreamDriver.Mode</b> <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> -<li><b>$InputTCPServerInputName</b> <name><br> -Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a -name is not strictly necessary, but can be useful to apply filtering based on which input -the message was received from. -<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +<li><b>StreamDriver.AuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> -<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +<li><b>PermittedPeer</b> <id-string><br> Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected -AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> -<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeer may not be set in anonymous modes. +<br>PermittedPeer may be set either to a single peer or an array of peers either of type IP or name, depending on the tls certificate. +<br>Single peer: PermittedPeer="127.0.0.1" +<br>Array of peers: PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."]</li> +</ul> +<p><b>Action Directives</b>:</p> +<ul> +<li><b>Port</b> <port><br> +Starts a TCP server on selected port</li> +<li><b>Name</b> <name><br> +Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a +name is not strictly necessary, but can be useful to apply filtering based on which input +the message was received from. +<li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> -<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +<li><b>SupportOctetCountedFraming</b> <<b>on</b>|off><br> If set to "on", the legacy octed-counted framing (similar to RFC5425 framing) is activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing @@ -102,6 +110,55 @@ is not used and some senders emit multi-line messages into the message stream. <p><b>Example:</b></p> <p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> </p> +<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imtcp/.libs/imtcp" MaxSessions="500") # needs to be done just once +input(type="imtcp" port="514") +</textarea> +<p>Note that the global parameters (here: max sessions) need to be set when the module is loaded. Otherwise, the parameters will not apply. +</p> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +equivalent to: AddtlFrameDelimiter +<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +equivalent to: DisableLFDelimiter +<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +equivalent to: NotifyOnConnectionClose<br> +</li> +<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +equivalent to: KeepAlive</li> +<li><b>$InputTCPServerRun</b> <port><br> +equivalent to: Port</li> +<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +equivalent to: FlowControl +</li> +<li><b>$InputTCPMaxListeners</b> <number><br> +equivalent to: MaxListeners</li> +<li><b>$InputTCPMaxSessions</b> <number><br> +equivalent to: MaxSessions</li> +<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +equivalent to: StreamDriver.Mode</li> +<li><b>$InputTCPServerInputName</b> <name><br> +equivalent to: Name +<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +equivalent to: StreamDriver.AuthMode</li> +<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +equivalent to: PermittedPeer.</li> +<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +equivalent to: Ruleset</a>.</li> +<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +equivalent to: SupportOctetCountedFraming +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> +(which includes the functionality of imtcp)</li> +</ul> +<p><b>Example:</b></p> +<p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> +</p> <textarea rows="15" cols="60">$ModLoad imtcp # needs to be done just once $InputTCPMaxSessions 500 $InputTCPServerRun 514 diff --git a/doc/imudp.html b/doc/imudp.html index 3512d474..b1a3ecc9 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -33,16 +33,18 @@ the value, the less precise the timestamp. <li><b>SchedulingPolicy</b> <rr/fifo/other><br> Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time -processing under Linux (and thus reduce chance of packet loss). Available since 4.7.4+, 5.7.3+, 6.1.3+. +processing under Linux (and thus reduce chance of packet loss). <li><b>SchedulingPriority</b> <number><br> -Scheduling priority to use. Available since 4.7.4+, 5.7.3+, 6.1.3+. +Scheduling priority to use. </ul> <p><b>Action Directives</b>:</p> <ul> <li><b>Address</b> <IP><br> local IP address (or name) the UDP listens should bind to</li> <li><b>Port</b> <port><br> -default 514, start UDP server on this port</li> +default 514, start UDP server on this port. Either a single port can be specified or an array of ports. If multiple ports are specified, a listener will be automatically started for each port. Thus, no additional inputs need to be configured. +<br>Single port: Port="514" +<br>Array of ports: Port=["514","515","10514","..."]</li> <li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> @@ -70,9 +72,9 @@ equivalent to: Port </li> equivalent to: TimeRequery <li>$InputUDPServerBindRuleset <ruleset><br> equivalent to: Ruleset </li> -<li>$IMUDPSchedulingPolicy <rr/fifo/other><br> +<li>$IMUDPSchedulingPolicy <rr/fifo/other> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPolicy -<li>$IMUDPSchedulingPriority <number><br> +<li>$IMUDPSchedulingPriority <number> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPriority </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index bd207a37..a962f814 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -77,7 +77,7 @@ to the system log socket. <li><b>SysSock.UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0. +with earlier versions of rsyslog. </li> <li><b>SysSock.RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. @@ -102,7 +102,7 @@ properties for the system log socket.</li> to the next socket.</li> <li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number -of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen in 5.9.6+, +of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> @@ -112,7 +112,7 @@ burst in number of messages. Default is 200. <li><b>RateLimit.Severity</b> [numerical severity] - specifies the severity of messages that shall be rate-limited. </li> -<!--<li><b>LocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified +<!--<li><b>LocalIPIF</b> [interface name] - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. @@ -120,7 +120,7 @@ the default of "127.0.0.1" is used. <li><b>UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> +with earlier versions of rsyslog. </li> <li><b>UseSysTimeStamp</b> [<b>on</b>/off] instructs imuxsock to obtain message time from the system (via control messages) insted of using time recorded inside the message. This may be most useful in combination with systemd. Note: @@ -139,7 +139,7 @@ being reset to "off" after the Socket directive, so if you would have for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those -sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> +sockets even though their directories do not yet exist.</li> <li><b>Socket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> <li><b>HostName</b> <hostname> permits to override the hostname that shall be used inside messages taken from the <b>next</b> Socket socket. Note that @@ -195,39 +195,42 @@ SysSock.Annotate="on") <p><b>Legacy Configuration Directives</b>:</p> <ul> <li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] -<br>Please see: IgnoreTimestamp.</li> -<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - Please see: FlowControl .</li> -<li><b>$IMUXSockRateLimitInterval</b> [number] - Please see: RateLimit.Interval +<br>equivalent to: IgnoreTimestamp.</li> +<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - equivalent to: FlowControl .</li> +<li><b>$IMUXSockRateLimitInterval</b> [number] - equivalent to: RateLimit.Interval </li> -<li><b>$IMUXSockRateLimitBurst</b> [number] - Please see: RateLimit.Burst +<li><b>$IMUXSockRateLimitBurst</b> [number] - equivalent to: RateLimit.Burst </li> -<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - Please see: RateLimit.Severity +<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - equivalent to: RateLimit.Severity </li> <li><b>$IMUXSockLocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. </li> -<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - Please see: UsePIDFromSystem.</li> -<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] Please see: UseSysTimeStamp .<br> +<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: UseSysTimeStamp .<br> <li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> -Please see: SysSock.IgnoreTimestamp.</li> -<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] Please see: SysSock.Use</li> -<li><b>$SystemLogSocketName</b> <name-of-socket> Please see: SysSock.Name</li> -<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - Please see: SysSock.FlowControl.</li> -<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - Please see: SysSock.UsePIDFromSystem.</li> -<li><b>$SystemLogRateLimitInterval</b> [number] - Please see: SysSock.RateLimit.Interval. +equivalent to: SysSock.IgnoreTimestamp.</li> +<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] equivalent to: SysSock.Use</li> +<li><b>$SystemLogSocketName</b> <name-of-socket> equivalent to: SysSock.Name</li> +<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - equivalent to: SysSock.FlowControl.</li> +<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: SysSock.UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$SystemLogRateLimitInterval</b> [number] - equivalent to: SysSock.RateLimit.Interval. </li> -<li><b>$SystemLogRateLimitBurst</b> [number] - Please see: SysSock.RateLimit.Burst +<li><b>$SystemLogRateLimitBurst</b> [number] - equivalent to: SysSock.RateLimit.Burst </li> -<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - Please see: SysSock.RateLimit.Severity +<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - equivalent to: SysSock.RateLimit.Severity </li> -<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] Please see: SysSock.UseSysTimeStamp. -<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - Please see: CreatePath</li> -<li><b>$AddUnixListenSocket</b> <name-of-socket> Please see: Socket </li> -<li><b>$InputUnixListenSocketHostName</b> <hostname> Please see: HostName.</li> -<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> Please see: Annotate.</li> -<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> Please see: SysSock.Annotate.</li> +<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: SysSock.UseSysTimeStamp. +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - equivalent to: CreatePath +<br>[available since 4.7.0 and 5.3.0]</li> +<li><b>$AddUnixListenSocket</b> <name-of-socket> equivalent to: Socket </li> +<li><b>$InputUnixListenSocketHostName</b> <hostname> equivalent to: HostName.</li> +<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> equivalent to: Annotate.</li> +<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> equivalent to: SysSock.Annotate.</li> </ul> <b>Caveats/Known Bugs:</b><br> diff --git a/doc/manual.html b/doc/manual.html index 9c7c677e..185e873e 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 6.6.0 (v6-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 7.2.1 (v7-stable branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -35,6 +35,8 @@ if you upgrade from v4, read the <a href="v5compatibility.html">rsyslog v5 compatibility notes</a>, and if you upgrade from v5, read the <a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. +if you upgrade from v6, read the +<a href="v7compatibility.html">rsyslog v7 compatibility notes</a>. <p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> <p><b>Follow the links below for the</b></p> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index fcc2674d..84261bdd 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -61,6 +61,14 @@ variable, if it exists. Returns an empty string if it does not exist. <li>cstr(expr) - converts expr to a string value <li>cnum(expr) - converts expr to a number (integer) <li>re_match(expr, re) - returns 1, if expr matches re, 0 otherwise +<li>field(str, delim, matchnbr) - returns a field-based substring. str is the string +to search, delim is the numerical ascii value of the field delimiter (so that +non-printable characters can by specified) and matchnbr is the match to search +for (the first match starts at 1). This works similar as the field based +property-replacer option. +<li>prifilt(constant) - mimics a traditional PRI-based filter (like "*.*" or +"mail.info"). The traditional filter string must be given as a <b>constant string</b>. +Dynamic string evaluation is not permitted (for performance reasons). </ul> <p>The following example can be used to build a dynamic filter based on some environment variable: diff --git a/doc/rsyslog_conf_filter.html b/doc/rsyslog_conf_filter.html index fbced4a3..3efa3967 100644 --- a/doc/rsyslog_conf_filter.html +++ b/doc/rsyslog_conf_filter.html @@ -4,38 +4,95 @@ <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> <h2>Filter Conditions</h2> -<p>Rsyslog offers four different types "filter conditions":</p> +<p>Rsyslog offers three different types "filter conditions":</p> <ul> -<li>BSD-style blocks</li> +<li><a href="http://www.rainerscript.com/">RainerScript</a>-based filters</li> <li>"traditional" severity and facility based selectors</li> <li>property-based filters</li> -<li>expression-based filters</li> </ul> -<h3>Blocks</h3> -<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each -block of lines is separated from the previous block by a program or -hostname specification. A block will only log messages corresponding to -the most recent program and hostname specifications given. Thus, a -block which selects ‘ppp’ as the program, directly followed by a block -that selects messages from the hostname ‘dialhost’, then the second -block will only log messages from the ppp program on dialhost. -</p> -<p>A program specification is a line beginning with ‘!prog’ and -the following blocks will be associated with calls to syslog from that -specific program. A program specification for ‘foo’ will also match any -message logged by the kernel with the prefix ‘foo: ’. Alternatively, a -program specification ‘-foo’ causes the following blocks to be applied -to messages from any program but the one specified. A hostname -specification of the form ‘+hostname’ and the following blocks will be -applied to messages received from the specified hostname. -Alternatively, a hostname specification ‘-hostname’ causes the -following blocks to be applied to messages from any host but the one -specified. If the hostname is given as ‘@’, the local hostname will be -used. (NOT YET IMPLEMENTED) A program or hostname specification may be -reset by giving the program or hostname as ‘*’.</p> -<p>Please note that the "#!prog", "#+hostname" and "#-hostname" -syntax available in BSD syslogd is not supported by rsyslogd. By -default, no hostname or program is set.</p> +<h3>RainerScript-Based Filters</h3> +RainerScript based filters are the prime means of creating complex rsyslog configuration. +The permit filtering on arbitrary complex expressions, which can include boolean, +arithmetic and string operations. They also support full nesting of filters, just +as you know from other scripting environments. +<br> +Scripts based filters are indicated by the keyword "if", as usual. +They have this format:<br> +<br> +if expr then block else block +<br> +"If" and "then" are fixed keywords that mus be present. "expr" is a +(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for +details. +The keyword "else" and its associated block is optional. Note that a block can contain either +a single action (chain), or an arbitrary complex script enclosed in curly braces, e.g.: +<br> +<pre> +if $programname == 'prog1' then { + action(type="omfile" file="/var/log/prog1.log") + if $msg contains 'test' then + action(type="omfile" file="/var/log/prog1test.log") + else + action(type="omfile" file="/var/log/prog1notest.log") +} +</pre> +<br> +Other types of filtes can also be combined with the pure RainerScript ones. This makes +it particularly easy to migrate from early config files to RainerScript. Also, the traditional +syslog PRI-based filters are a good and easy to use addition. While they are legacy, we still +recommend there use where they are up to the job. We do NOT, however, recommend property-based +filters any longer. As an example, the following is perfectly valid: +<br> +<pre> +if $fromhost == 'host1' then { + mail.* action(type="omfile" file="/var/log/host1/mail.log") + *.err /var/log/host1/errlog # this is also still valid + # + # more "old-style rules" ... + # +} else { + mail.* action(type="omfile" file="/var/log/mail.log") + *.err /var/log/errlog + # + # more "old-style rules" ... + # +} +</pre> +<br> + +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. +If you don't like that, you can of course also use the textual property +- just be sure to use the right one. As expression support is enhanced, +this will change. For example, if you would like to filter on message +that have facility local0, start with "DEVNAME" and have either +"error1" or "error0" in their message content, you could use the +following filter:<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +Please note that the above <span style="font-weight: bold;">must +all be on one line</span>! And if you would like to store all +messages except those that contain "error1" or "error0", you just need +to add a "not":<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> +($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +If you would like to do case-insensitive comparisons, use +"contains_i" instead of "contains" and "startswith_i" instead of +"startswith".<br> +<br> +Regular expressions are supported via functions (see function list). + <h3>Selectors</h3> <p><b>Selectors are the traditional way of filtering syslog messages.</b> They have been kept in rsyslog with their original @@ -213,71 +270,6 @@ supported (except for "not" as outlined above). Please note that while it is possible to query facility and severity via property-based filters, it is far more advisable to use classic selectors (see above) for those cases.</p> -<h3>Expression-Based Filters</h3> -Expression based filters allow -filtering on arbitrary complex expressions, which can include boolean, -arithmetic and string operations. Expression filters will evolve into a -full configuration scripting language. Unfortunately, their syntax will -slightly change during that process. So if you use them now, you need -to be prepared to change your configuration files some time later. -However, we try to implement the scripting facility as soon as possible -(also in respect to stage work needed). So the window of exposure is -probably not too long.<br> -<br> -Expression based filters are indicated by the keyword "if" in column 1 -of a new line. They have this format:<br> -<br> -if expr then action-part-of-selector-line<br> -<br> -"If" and "then" are fixed keywords that mus be present. "expr" is a -(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for -details. "action-part-of-selector-line" is an action, just as you know -it (e.g. "/var/log/logfile" to write to that file).<br> -<br> -A few quick samples:<br> -<br> -<code> -*.* /var/log/file1 # the traditional way<br> -if $msg contains 'error' then /var/log/errlog # the expression-based way<br> -</code> -<br> -Right now, you need to specify numerical values if you would like to -check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. -If you don't like that, you can of course also use the textual property -- just be sure to use the right one. As expression support is enhanced, -this will change. For example, if you would like to filter on message -that have facility local0, start with "DEVNAME" and have either -"error1" or "error0" in their message content, you could use the -following filter:<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -Please note that the above <span style="font-weight: bold;">must -all be on one line</span>! And if you would like to store all -messages except those that contain "error1" or "error0", you just need -to add a "not":<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> -($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -If you would like to do case-insensitive comparisons, use -"contains_i" instead of "contains" and "startswith_i" instead of -"startswith".<br> -<br> -Note that regular expressions are currently NOT -supported in expression-based filters. These will be added later when -function support is added to the expression engine (the reason is that -regular expressions will be a separate loadable module, which requires -some more prequisites before it can be implemented).<br> - <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index b97f6609..0c189100 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -3,7 +3,7 @@ <body> <p>This is a part of the rsyslog.conf - documentation.</p> <a href="rsyslog_conf.html">back</a> -<h2>Templates</h2> +<h1>Templates</h1> <p>Templates are a key feature of rsyslog. They allow to specify any format a user might want. They are also used for dynamic file name @@ -16,67 +16,200 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> -<p>Starting with 5.5.6, there are actually two differnt types of template: +<p>Templates are specified by template() statements. They can also be specified +via $Template legacy statements. Note that these are scheduled for removal in +later versions of rsyslog, so it is probably a good idea to avoid them +for new uses. +<h2>The template() statement</h2> +<p>The template() statement is used to define templates. Note that it is a +<b>static</b> statement, that means all templates are defined when rsyslog +reads the config file. As such, templates are not affected by if-statements +or config nesting. +<p>The basic structure of the template statement is as follows: +<br><br> +<code>template(parameters)</code> +<br><br> +In addition to this simpler syntax, list templates (to be described below) +support an extended syntax: +<br><br> +<code>template(parameters) { list-descriptions }</code> +<p>Each template has a parameter <b>name</b>, which specifies the templates +name, and a parameter <b>type</b>, which specifies the template type. The name +parameter must be unique, and behaviour is unpredictable if it is not. The <b>type</b> +parameter specifies different template types. Different types simply enable +different ways to specify the template content. The template type <b>does not</b> +affect what an (output) plugin can do with it. So use the type that best fits your +needs (from a config writing point of view!). The following types are available: <ul> -<li>string based -<li>string-generator module based +<li>list +<li>subtree +<li>string +<li>plugin </ul> -<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates -have been introduced in 5.5.6. They permit a string generator, actually a C "program", -the generate a format. Obviously, it is more work required to code such a generator, -but the reward is speed improvement. If you do not need the ultimate throughput, you -can forget about string generators (so most people never need to know what they are). -You may just be interested in learning that for the most important default formats, -rsyslog already contains highly optimized string generators and these are called -without any need to configure anything. But if you have written (or purchased) a -string generator module, you need to know how to call it. Each such module has a name, -which you need to know (look it up in the module doc or ask the developer). Let's assume -that "mystrgen" is the module name. Then you can define a template for that strgen -in the following way: +The various types are described below. -<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> -(Of course, you must have first loaded the module via $ModLoad). -<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that -no string follows but a strgen module name. -<p>There are no additional parameters but the module name supported. This is because -there is no way to customize anything inside such a "template" other than by -modifying the code of the string generator. +<h3>list</h3> +<p>In this case, the template is generated by a list of constant and +variable statements. These follow the template spec in curly braces. This type is +also primarily meant for use with structure-aware outputs, like ommongodb. However, +it also works perfectly with text-based outputs. We recommend to use this mode +if more complex property substitutions needs to be done. In that case, the list-based +template syntax is much clearer than the simple string-based one. +<p>The list template contains the template header (with <b>type="list"</b>) and is followed +by <b>constant</b> and <b>property</b> statements, given in curly braces to signify +the template statement they belong to. As the name says, <b>constant</b> statements +describe constant text and <b>property</b> describes property access. There are many options +to <b>property</b>, described further below. Most of these options are used to extract +only partial property contents or to modify the text obtained (like to change its case +to upper or lower case, only). +<p>To grasp the idea, an actual sample is: +<br><pre><code>template(name="tpl1" type="list") { + constant(value="Syslog MSG is: '") + property(name="msg") + constant(value="', ") + property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + constant(value="\n") + } +</code></pre> +<br>This sample is probably primarily targeted at the usual file-based output.</p> -<p>So for most use cases, string-generator module based templates are <b>not</b> -the route to take. Usually, we use <b>string based templates</b> instead. -This is what the rest of the documentation now talks about. -<p>A template consists of a template directive, a name, the -actual template text and optional options. A sample is:</p> -<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,"\7Text -%property% some more text\n",<options></code></blockquote> -<p>The "template" (legacy: $template) is the template directive. It tells rsyslog -that this line contains a template. "MyTemplateName" is the template -name. All -other config lines refer to this name. The text within "string" is the -actual template text. The backslash is an escape character, much as it -is in C. It does all these "cool" things. For example, \7 rings the -bell (this is an ASCII value), \n is a new line. C programmers and perl -coders have the advantage of knowing this, but the set in rsyslog is a -bit restricted currently. -</p> -<p>All text in the template is used literally, except for things -within percent signs. These are properties and allow you access to the -contents of the syslog message. Properties are accessed via the -<a href="property_replacer.html">property replacer</a> -(nice name, huh) and it can do cool things, too. For -example, it can pick a substring or do date-specific formatting. More -on this is below, on some lines of the property replacer.<br> -<br> +<h4>constant statement</h4> +<p>This provides a way to specify constant text. The text is used literally. It is +primarily intended for text-based output, so that some constant text can be included. For +example, if a complex template is build for file output, one usually needs to finish it +by a newline, which can be introduced by a constant statement. Here is an actual sample +of that use case from the rsylsog testbench: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +}</code></pre> +The following escape sequences are recogniced inside the constant text: +<ul> +<li>\\ - single backslash +<li>\n - LF +<li>\ooo - (three octal digits) - represents character with this numerical value (e.g. \101 +equals "A"). Note that three +octal digits must be given (in contrast to some languagues, where between one and three are valid). +While we support octal notation, we recommend to use hex notation as this is better known. +<li>\xhh - (where h is a hex digit) - represents character with this numerical value (e.g. \x41 +equals "A"). Note that two hexadecimal digits must be given (in contrast to some languagues +where one or two are valid). +<li>... some others ... list needs to be extended +</ul> +<p>Note: if an unsupported character follows a backslash, this is treated as an error. Behaviour +is unpredictable in this case. +<p>To aid usage of the same template both for text-based outputs and structured ones, constant +text without an "outname" parameter will be ignored when creating the name/value tree +for structured outputs. So if you want to supply some constant text e.g. to mongodb, you must +include an outname, as can be seen here: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n" <b>outname="IWantThisInMyDB"</b>) +}</code></pre> + +The "constant" statement supports the following parameters: +<ul> +<li>value - the constant value to use +<li>outname - output field name (for structured outputs) +</ul> + + +<h4>property statement</h4> +<p>This statement is used to include property text. It can access all properties. Also, +options permit to specify picking only part of a property or modifying it. +It supports the following parameters: +<ul> +<li>name - the name of the property to access +<li>outname - output field name (for structured outputs) +<li>dateformat - date format to use (only for date-related properties) +<li>caseconversion - permits to convert case of the text. supported values are +"lower" and "upper" +<li>controlcharacters - specifies how to handle control characters. Supported values are +"escape", which escapes them, "space", which replaces them by a single space, and +"drop", which simply removes them from the string. +<li>securepath - used for creating pathnames suitable for use in dynafile templates +<li>format - specifiy format on a field basis. Supported values are "csv", for use when +csv-data is generated, "json", which formats proper json content (but without a field +header) and "jsonf", which formats as a complete json field. +<li>position.from - obtain substring starting from this position (1 is the first position) +<li>position.to - obtain substring up to this position +<li>field.number - obtain this field match +<li>field.delimiter - decimal value of delimiter character for field extraction +<li>regex.expression - expression to use +<li>regex.type - either ERE or BRE +<li>regex.nomatchmode - what to do if we have no match +<li>regex.match - match to use +<li>regex.submatch - submatch to use +<li>droplastlf - drop a trailing LF, if it is present +<li>mandatory - signifies a field as mandatory. If set to "on", this field will always +be present in data passed to structured outputs, even if it is empty. If "off" (the default) +empty fields will not be passed to structured outputs. This is especially useful for outputs +that support dynamic schemas (like ommongodb). +<li>spifno1stsp - expert options for RFC3164 template processing +</ul> + + +<h3>subtree</h3> +<p>Available since rsyslog 7.1.4 +<p> +In this case, the template is generated based on a complete +(CEE) subtree. This type of template is most useful for outputs that know how to +process hierarchical structure, like ommongodb. With that type, the parameter +<b>subtree</b> must be specified, which tells which subtree to use. For example +template(name="tpl1" type="subtree" subtree="$!") includes all CEE data, while +template(name="tpl2" type="subtree" subtree="$!usr!tpl2") includes only the +subtree starting at $!usr!tpl2. The core idea when using this type of template +is that the actual data is prefabricated via set and unset script statements, +and the resulting strucuture is then used inside the template. This method MUST +be used if a complete subtree needs to be placed <i>directly</i> into the +object's root. With all other template types, only subcontainers can be generated. +Note that subtree type can also be used with text-based outputs, like omfile. HOWEVER, +you do not have any capability to specify constant text, and as such cannot include +line breaks. As a consequence, using this template type for text outputs is usually +only useful for debugging or very special cases (e.g. where the text is interpreted +by a JSON parser later on). +<h4>Use case</h4> +<p>A typical use case is to first create a custom subtree and then include it into +the template, like in this small example: +<br><blockquote><code>set $!usr!tpl2!msg = $msg; +<br>set $!usr!tpl2!dataflow = field($msg, 58, 2); +<br>template(name="tpl2" type="subtree" subtree="$!usr!tpl2") +</code></blockquote> +<p>Here, we assume that $msg contains various fields, and the data from a field +is to be extracted and stored - together with the message - as field content. +<h3>string</h3> +<p>This closely resembles the legacy template statement. It +has a mandatory parameter <b>string</b>, which holds the template string to be +applied. A template string is a mix of constant text and replacement variables +(see property replacer). These variables are taken from message or other dynamic +content when the final string to be passed to a plugin is generated. String-based +templates are a great way to specify textual content, especially if no complex +manipulation to properties is necessary. Full details on how to specify template +text can be found below. +<br>Config example: +<br><blockquote><code>template(name="tpl3" type="string" string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") +</code></blockquote> +<h3>plugin</h3> +In this case, the template is generated by a plugin (which +is then called +a "strgen" or "string generator"). The format is fix as it is coded. While this +is inflexible, it provides superior performance, and is often used for that +reason (not that "regular" templates are slow - but in very demanding environments +that "last bit" can make a difference). Refer to the plugin's documentation +for further details. For this type, the paramter <b>plugin</b> must be specified and +must contain the name of the plugin as it identifies itself. Note that the +plugin must be loaded prior to being used inside a template. +<br>Config example: +<br><blockquote><code>template(name="tpl4" type="plugin" plugin="mystrgen") +</code></blockquote> + +<h3>options</h3> The <options> part is optional. It carries options -influencing the template as whole. See details below. Be sure NOT to -mistake template options with property options - the latter ones are -processed by the property replacer and apply to a SINGLE property, only -(and not the whole template).<br> +influencing the template as whole and is part of the template parameters. +See details below. Be sure NOT to mistake template options with property +options - the latter ones are processed by the property replacer and +apply to a SINGLE property, only (and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> <p><b>option.sql</b> - format the string suitable for a SQL @@ -127,50 +260,102 @@ option. Otherwise you will become vulnerable to SQL injection. <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> +template (name="TraditionalFormat" type="string" string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> <br> -Properties can be accessed by the <a href="property_replacer.html">property -replacer</a> (see there for details).</p> -<p>Templates can be used in the form of a <b>list</b> as well. This has been -introduced with <b>6.5.0</b> The list consists of two parts which are either -a <b>constant</b> or a <b>property</b>. The constants -are taking the part of "text" that you usually enter in string-based templates. -The properties stay variable, as they are a substitute for different values of a -certain type. This type of template is extremely useful for complicated cases, -as it helps you to easily keep an overview over the template. Though, it has -the disadvantage of needing more effort to create it.</p> -<br>Config example: -<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { - <br>constant(value="Test: ") - <br>property(name="msg" outname="mymessage") - <br>constant(value=" --!!!-- ") - <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") - <br>constant(value="\n") - <br>} -</code></blockquote> -<p>First, the general template option will be defined. The values of the template -itself get defined in the curly brackets. As it can be seen, we have constants -and properties in exchange. Whereas constants will be filled with a value and probably -some options, properties do direct to a property and the options that could be needed -additional format definitions.</p> -<p>We suggest to use separate lines for all constants and properties. This -helps to keep a good overview over the different parts of the template. -Though, writing it in a single line will work, it is much harder to debug -if anything goes wrong with the template. </p> +<h3>Examples</h3> +<h4>Standard Template for Writing to Files</h4> +<p><pre><code>template(name="FileFormat" type="list") { + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + property(name="msg" droplastlf="on" ) + constant(value="\n") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="FileFormat" type="string" + string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +)</code></pre> +Note that the template string itself must be on a single line. -<p><b>Please note that templates can also be -used to generate selector lines with dynamic file names.</b> For -example, if you would like to split syslog messages from different -hosts to different files (one per host), you can define the following -template:</p> -<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template -DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> -<p>This template can then be used when defining an output -selector line. It will result in something like -"/var/log/system-localhost.log"</p> +<h4>Standard Template for Forwarding to a Remote Host (RFC3164 mode)</h4> +<p><pre><code>template(name="ForwardFormat" type="list") { + constant(value="<") + property(name="PRI") + constant(value="<") + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag" position.from="1" position.to="32") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="forwardFormat" type="string" + string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +)</code></pre> +Note that the template string itself must be on a single line. + +<h4>Standard Template for write to the MySQL database</h4> +<p><pre><code>template(name="StdSQLformat" type="list" option.sql="on") { + constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") + constant(value=" values ('") + property(name="msg") + constant(value="', ") + property(name="syslogfacility") + constant(value=", '") + property(name="hostname") + constant(value="', ") + property(name="syslogpriority") + constant(value=", '") + property(name="timereported" dateFormat="mysql") + constant(value="', '") + property(name="timegenerated" dateFormat="mysql") + constant(value="', ") + property(name="iut") + constant(value=", '") + property(name="syslogtag") + constant(value="')") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="stdSQLformat" type="string" option.sql="on" + string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')" +)</code></pre> +Note that the template string itself must be on a single line. + +<h2>legacy format</h2> +<p>In pre v6-versions of rsyslog, you need to use the <code>$template</code> +statement to configure templates. They provide the equivalent to string- and +plugin-based templates. The legacy syntax continous to work in v7, however +we recommend to avoid legacy format for newly written config files. Legacy and +current config statements can coexist within the same config file. +<p>The general format is +<br><br><code>$template name,param[,options]</code></br></br> +where "name" is the template name and "param" is a single parameter +that specifies template content. The optional "options" part is used to +set template options. +<h3>string</h3> +The parameter is the same string that with the current-style format you +specify in the <b>string</b> parameter, for example: +<br><br><code>$template strtpl,"PRI: %pri%, MSG: %msg%\n"</code> +<p>Note that list templates are not available in legacy format, so you need +to use complex property replacer constructs to do complex things. + +<h3>plugin</h3> +This is equivalent to the "plugin"-type template directive. Here, the +parameter is the plugin name, with an equal sign prepended. An example +is: +<br><br><code>$template plugintpl,=myplugin</code> + +<h2>Reserved Template Names</h2> <p>Template names beginning with "RSYSLOG_" are reserved for rsyslog use. Do NOT use them if, otherwise you may receive a conflict in the future (and @@ -210,12 +395,122 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> + +<h2>The following is legacy documentation soon to be integrated.</h2> + +<!--<table> +<tr><td>param name</td><td>meaning</td></tr> +<tr><td>name</td><td>name of the template</td></tr> +</table> +--> + +<p>Starting with 5.5.6, there are actually two different types of template: +<ul> +<li>string based +<li>string-generator module based +</ul> +<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: + +<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> +(Of course, you must have first loaded the module via $ModLoad). +<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that +no string follows but a strgen module name. +<p>There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +<p>So for most use cases, string-generator module based templates are <b>not</b> +the route to take. Usually, we use <b>string based templates</b> instead. +This is what the rest of the documentation now talks about. + +<p>A template consists of a template directive, a name, the +actual template text and optional options. A sample is:</p> +<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,"\7Text +%property% some more text\n",<options></code></blockquote> +<p>The "template" (legacy: $template) is the template directive. It tells rsyslog +that this line contains a template. "MyTemplateName" is the template +name. All +other config lines refer to this name. The text within "string" is the +actual template text. The backslash is an escape character, much as it +is in C. It does all these "cool" things. For example, \7 rings the +bell (this is an ASCII value), \n is a new line. C programmers and perl +coders have the advantage of knowing this, but the set in rsyslog is a +bit restricted currently. +</p> +<p>All text in the template is used literally, except for things +within percent signs. These are properties and allow you access to the +contents of the syslog message. Properties are accessed via the +<a href="property_replacer.html">property replacer</a> +(nice name, huh) and it can do cool things, too. For +example, it can pick a substring or do date-specific formatting. More +on this is below, on some lines of the property replacer.<br> +<br> + +<br> +Properties can be accessed by the <a href="property_replacer.html">property +replacer</a> (see there for details).</p> +<p>Templates can be used in the form of a <b>list</b> as well. This has been +introduced with <b>6.5.0</b> The list consists of two parts which are either +a <b>constant</b> or a <b>property</b>. The constants +are taking the part of "text" that you usually enter in string-based templates. +The properties stay variable, as they are a substitute for different values of a +certain type. This type of template is extremely useful for complicated cases, +as it helps you to easily keep an overview over the template. Though, it has +the disadvantage of needing more effort to create it.</p> +<br>Config example: +<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { + <br>constant(value="Test: ") + <br>property(name="msg" outname="mymessage") + <br>constant(value=" --!!!-- ") + <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + <br>constant(value="\n") + <br>} +</code></blockquote> +<p>First, the general template option will be defined. The values of the template +itself get defined in the curly brackets. As it can be seen, we have constants +and properties in exchange. Whereas constants will be filled with a value and probably +some options, properties do direct to a property and the options that could be needed +additional format definitions.</p> +<p>We suggest to use separate lines for all constants and properties. This +helps to keep a good overview over the different parts of the template. +Though, writing it in a single line will work, it is much harder to debug +if anything goes wrong with the template. </p> + +<p><b>Please note that templates can also be +used to generate selector lines with dynamic file names.</b> For +example, if you would like to split syslog messages from different +hosts to different files (one per host), you can define the following +template:</p> +<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template +DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> +<p>This template can then be used when defining an output +selector line. It will result in something like +"/var/log/system-localhost.log"</p> <h3>Legacy String-based Template Samples</h3> -<p>This section provides some sample of what the default formats would -look as a text-based template. Hopefully, their description is self-explanatory. +<p>This section provides some default templates in legacy format, as used in rsyslog +previous to version 6. Note that this format is still supported, so there is no hard need +to upgrade existing configurations. However, it is strongly recommended that the legacy +constructs are not used when crafting new templates. Note that each $Template statement is on a <b>single</b> line, but probably broken accross several lines for display purposes by your browsers. Lines are separated by -empty lines. +empty lines. Keep in mind, that line breaks are important in legacy format. <p><code> $template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" <br><br> @@ -233,7 +528,7 @@ $template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, P [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> diff --git a/doc/v4compatibility.html b/doc/v4compatibility.html index 72b0f5a9..2a51adea 100644 --- a/doc/v4compatibility.html +++ b/doc/v4compatibility.html @@ -60,7 +60,7 @@ restarting rsyslogd by HUPing it. and most other deamons require that a restart command is typed in if a restart is required. <p>Rsyslog will follow this paradigm in the next versions, resulting in many benefits. In v4, we provide some support for the old-style semantics. We introduced a setting $HUPisRestart -which may be set to "on" (tradional, heavy operationg) +which may be set to "on" (tradional, heavy operation) or "off" (new, lightweight "file close only" operation). The initial versions had the default set to traditional behavior, but starting with 4.5.1 we are now using the new behavior as the default. diff --git a/doc/v7compatibility.html b/doc/v7compatibility.html new file mode 100644 index 00000000..8834cd54 --- /dev/null +++ b/doc/v7compatibility.html @@ -0,0 +1,60 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v7</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v7</h1> +This document describes things to keep in mind when moving from v6 to v7. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used v6 and want to use v7 without hassle. +<p>Version 7 builds on the new config language introduced in v6 and extends it. +Other than v6, it not just only extends the config language, but provides +considerable changes to core elements as well. The result is much more power and +ease of use as well (this time that is not contradictionary). +</p> +<h2>BSD-Style blocks</h2> +BSD style blocks are no longer supported (for good reason). See the +<a href="http://www.rsyslog.com/g/BSD">rsyslog BSD blocks info</a> +page for more information and how to upgrade your config. +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + +<h2>CEE-Properties</h2> +In rsyslog v6, CEE properties could not be used across disk-based queues. If this was +done, there content was reset. This was a missing feature in v6. In v7, this feature +has been implemented. Consequently, situations where the previous behaviour were +desired need now to be solved differently. We do not think that this will cause any +problems to anyone, especially as in v6 this was announced as a missing feature. + +<h2>omruleset and discard (~) action are deprecated</h2> +<p>Both continue to work, but have been replaced by better alternatives. +<p>The discard action (tilde character) has been replaced by the "stop" +RainerScript directive. It is considered more intuitive and offers slightly +better performance. +<p>The omruleset module has been replaced by the "call" RainerScript directive. +Call permits to execute a ruleset like a subroutine, and does so with much +higher performance than omruleset did. Note that omruleset could be run off +an async queue. This was more a side than a desired effect and is not supported +by the call statement. If that effect was needed, it can simply be simulated by +running the called rulesets actions asynchronously (what in any case is the right +way to handle this). +<p>Note that the deprecated modules emit warning messages when being used. +They tell that the construct is deprecated and which statement is to be used +as replacement. This does <b>not</b> affect operations: both modules are still +fully operational and will not be removed in the v7 timeframe. + +<h2>Retries of output plugins that do not do proper replies</h2> +<p>Some output plugins may not be able to detect if their target is capable of +accepting data again after an error (technically, they always return OK when +TryResume is called). Previously, the rsyslog core engine suspended such an action +after 1000 succesive failures. This lead to potentially a large amount of +errors and error messages. Starting with 7.2.1, this has been reduced to 10 +successive failures. This still gives the plugin a chance to recover. In extreme +cases, a plugin may now enter suspend mode where it previously did not do so. +In practice, we do NOT expect that. + +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2011-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body></html> |