diff options
Diffstat (limited to 'doc')
35 files changed, 2096 insertions, 395 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index 91d92afd..1ae1c68d 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -31,6 +31,10 @@ html_files = \ version_naming.html \ contributors.html \ dev_queue.html \ + ompipe.html \ + omfwd.html \ + omfile.html \ + omusrmsg.html \ omstdout.html \ omudpspoof.html \ omruleset.html \ @@ -110,6 +114,7 @@ html_files = \ src/tls_cert.dia \ gssapi.html \ licensing.html \ + mmnormalize.html \ ommail.html \ omuxsock.html \ omrelp.html \ @@ -125,6 +130,21 @@ html_files = \ rsyslog_conf_nomatch.html \ queues_analogy.html \ multi_ruleset.html \ + dev_oplugins.html \ + free_support.html \ + imudp.html \ + messageparser.html \ + omhdfs.html \ + omprog.html \ + queue_msg_state.jpeg \ + rsconf1_abortonuncleanconfig.html \ + rsconf1_maxopenfiles.html \ + rsconf1_omfileforcechown.html \ + rsyslog_conf_file_syntax_differences.html \ + rsyslog_conf_lines.html \ + rsyslog_queue_pointers.jpeg \ + rsyslog_queue_pointers2.jpeg \ + v6compatibility.html \ src/classes.dia grfx_files = \ diff --git a/doc/design.tex b/doc/design.tex index a3ec8f45..1def3fb7 100644 --- a/doc/design.tex +++ b/doc/design.tex @@ -811,10 +811,80 @@ b) we push the failed message back to the main queue, but with an indication that it failed in an action. This is harder to implement and most importantly harder to understand/configure, but more flexible +\section{Configuration System} +The configration system found in all versions up to v5 is based on sysklogd's +legacy. It does not have any clear distinction between config load and +activation. Starting with v6, a new config system is build. That new system +offers the necessary distinction. In the long term, the configuration language +will be enhanced towards the more flexible and easy to use RainerScript idea. + +\section{Plugin Interface} +This section describes some aspects of the plugin interface. +\subsection{Configuration Related} +To support the new v2 config system, plugins need to publish a number of entry +points that will be called by the rsyslog configuration section at various +stages of the configration load, activation and deactivation process. This list +may be extended as the configuration interface evolves. + +Plugins must not necessarily implement support for the v2 config system. If +they do, the ``beginCnfLoad'' entry point serves as a flag telling that support +is available. In that case, all other entry points need to be defined as well. +If a module does not support the v2 config system, it can still be run, but be +configured only via the legacy config system. Note that with the old system +there are also problems with droping privileges. So a legacy module may not +work correctly if privileges are dropped. + +The following entry points are available: +\begin{enumerate} + \item \emph{beginCnfLoad} -- called when a new config load begins. Only one +config load can be active at one time (no concurrent loads). + \item \emph{endCnfLoad} -- called when config load ends. This gives the module +a chance to do final changes and some cleanup. + \item \emph{checkCnf} -- called by the framework to verify a configuration. + \item \emph{activateCnfPrePrivDrop} -- called by the framework to activate a +configuration before privileges are dropped. This is an optional entry point +that shall only be implemented by plugins that need the do some processing +before rsyslog drops privileges. Processing inside this entry point should be +limited to what is absolutely necessary. The main activation work should be +done in activateCnf() as usual. + \item \emph{activateCnf} -- called by the framework to activate a +configuration. +\item \emph{freeCnf} -- called by the framework to free +(deallocate) a configuration. +\end{enumerate} + +In the current implementation, entry points are sequentially called as given +above. However, this will change. It is guaranteed that +\begin{itemize} + \item beginCnfLoad() will be followed by a matching endCnfLoad() and there +will be no new call to beginCnfLoad() before endCnfLoad() has been called. This +means no nested config load needs to be supported, + \item checkCnf() may be called at any time, even during a config load phase. +However, the config to check is a fully loaded one. + \item activateCnfPrePrivDrop(), if provided, will always be called before +activateCnf() is called. No other config-related calls will be made in between. +\end{itemize} + +\subsubsection{Output Modules} +The v1 config load system for output modules seems to provide all functionality +necessary to support the v2 system as well. As such, we currently do not +require output modules to implement the new calls to be fully supported by the +v2 system. + \section{Network Stream Subsystem} -The idea of network streams was introduced when we implemented RFC5425 (syslog over TLS) in 2008. The core idea is to encapsulate all stream-oriented network data transfer into a single transport layer and make the upper layers independent of actual transport being used. This is in line with the traditional layer approaches in communication systems. +The idea of network streams was introduced when we implemented RFC5425 (syslog +over TLS) in 2008. The core idea is to encapsulate all stream-oriented network +data transfer into a single transport layer and make the upper layers +independent of actual transport being used. This is in line with the traditional +layer approaches in communication systems. + +Under this system, the upper layer provides plugins to send and receive streams +of syslog data. Framing is provided by the upper layer. The upper layer itself +is integrated in input and output plugins, which then are used to provide +application-level syslog message objects to and from the rsyslog core. To these +upper layers, the netstream layer provides reliable and sequenced message +delivery with much of the same semantics as a usual TCP stream. -Under this system, the upper layer provides plugins to send and receive streams of syslog data. Framing is provided by the upper layer. The upper layer itself is integrated in input and output plugins, which then are used to provide application-level syslog message objects to and from the rsyslog core. To these upper layers, the netstream layer provides reliable and sequenced message delivery with much of the same semantics as a usual TCP stream. \begin{figure} \begin{center} diff --git a/doc/dev_oplugins.html b/doc/dev_oplugins.html index 63c186a3..b33b67f9 100644 --- a/doc/dev_oplugins.html +++ b/doc/dev_oplugins.html @@ -143,6 +143,11 @@ omstdout, you can see how a plugin may deal with the situation. array-passing capability not blindly be used.</b> In such cases, we can not guard the plugin from segfaulting and if the plugin (as currently always) is run within rsyslog's process space, that results in a segfault for rsyslog. So do not do this. +<p>Another possible mode is OMSR_TPL_AS_JSON, where instead of the template +a json-c memory object tree is passed to the module. The module can extract data +via json-c API calls. It MUST NOT modify the provided structure. This mode is +primarily aimed at plugins that need to process tree-like data, as found +for example in MongoDB or ElasticSearch. <h3>Batching of Messages</h3> <p>Starting with rsyslog 4.3.x, batching of output messages is supported. Previously, only a single-message interface was supported. diff --git a/doc/how2help.html b/doc/how2help.html index 4f0bd57a..7fda6949 100644 --- a/doc/how2help.html +++ b/doc/how2help.html @@ -14,6 +14,9 @@ wish list, that would be awfully helpful!</p> <li>spread word about rsyslog in forums and newsgroups</li> <li>place a link to <a href="http://www.rsyslog.com">www.rsyslog.com</a> from your home page</li> + <li>you may also want to tell others about the + <a href="http://loganalyzer.adiscon.com">log analyzer tool + created by the same folks as rsyslog</a> - at least, if you like it ;) </ul> </li> <li>let us know about rsyslog - we are eager for feedback<ul> @@ -54,4 +57,4 @@ wish list, that would be awfully helpful!</p> might do!</p> </body> -</html +</html> diff --git a/doc/imfile.html b/doc/imfile.html index 7961729b..1594cdce 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -36,17 +36,40 @@ file names in the future.</p> <p>Multiple files may be monitored by specifying $InputRunFileMonitor multiple times. </p> + <p><b>Configuration Directives</b>:</p> +<p><b>Module Directives</b></p> <ul> -<li><strong>$InputFileName /path/to/file</strong><br> +<li><span style="font-weight: bold;">PollingInterval +seconds</span><br> +This is a global setting. It specifies how often files are to be polled +for new data. The time specified is in seconds. The <span style="font-weight: bold;">default value</span> is 10 +seconds. Please note that future +releases of imfile may support per-file polling intervals, but +currently this is not the case. If multiple PollingInterval +statements are present in rsyslog.conf, only the last one is used.<br> +A short poll interval provides more rapid message forwarding, but +requires more system ressources. While it is possible, we stongly +recommend not to set the polling interval to 0 seconds. That will make +rsyslogd become a CPU hog, taking up considerable ressources. It is +supported, however, for the few very unusual situations where this +level may be needed. Even if you need quick response, 1 seconds should +be well enough. Please note that imfile keeps reading files as long as +there is any data in them. So a "polling sleep" will only happen when +nothing is left to be processed.</li> +</ul> + +<p><b>Action Directives</b></p> +<ul> +<li><strong>File /path/to/file</strong><br> The file being monitored. So far, this must be an absolute name (no macros or templates)</li> -<li><span style="font-weight: bold;">$InputFileTag +<li><span style="font-weight: bold;">Tag tag:</span><br> The tag to be used for messages that originate from this file. If you would like to see the colon after the tag, you need to specify it here (as shown above).</li> -<li><span style="font-weight: bold;">$InputFileStateFile +<li><span style="font-weight: bold;">StateFile <name-of-state-file></span><br> Rsyslog must keep track of which parts of the to be monitored file it already processed. This is done in the state file. This file always is @@ -55,40 +78,19 @@ $WorkDirectory). Be careful to use unique names for different files being monitored. If there are duplicates, all sorts of "interesting" things may happen. Rsyslog currently does not check if a name is specified multiple times.</li> -<li><span style="font-weight: bold;">$InputFileFacility +<li><span style="font-weight: bold;">Facility facility</span><br> The syslog facility to be assigned to lines read. Can be specified in textual form (e.g. "local0", "local1", ...) or as numbers (e.g. 128 for "local0"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is "local0".<span style="font-weight: bold;"></span></li> -<li><span style="font-weight: bold;">$InputFileSeverity</span><br> +<li><span style="font-weight: bold;">Severity</span><br> The syslog severity to be assigned to lines read. Can be specified in textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for "info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is "notice".</li> -<li><span style="font-weight: bold;">$InputRunFileMonitor</span><br> -This <span style="font-weight: bold;">activates</span> -the current monitor. It has no parameters. If you forget this -directive, no file monitoring will take place.</li> -<li><span style="font-weight: bold;">$InputFilePollInterval -seconds</span><br> -This is a global setting. It specifies how often files are to be polled -for new data. The time specified is in seconds. The <span style="font-weight: bold;">default value</span> is 10 -seconds. Please note that future -releases of imfile may support per-file polling intervals, but -currently this is not the case. If multiple $InputFilePollInterval -statements are present in rsyslog.conf, only the last one is used.<br> -A short poll interval provides more rapid message forwarding, but -requires more system ressources. While it is possible, we stongly -recommend not to set the polling interval to 0 seconds. That will make -rsyslogd become a CPU hog, taking up considerable ressources. It is -supported, however, for the few very unusual situations where this -level may be needed. Even if you need quick response, 1 seconds should -be well enough. Please note that imfile keeps reading files as long as -there is any data in them. So a "polling sleep" will only happen when -nothing is left to be processed.</li> -<li><b>$InputFilePersistStateInterval</b> [lines]</b><br> +<li><b>PersistStateInterval</b> [lines]</b><br> Available in 4.7.3+, 5.6.2+<br> Specifies how often the state file shall be written when processing the input file. The default value is 0, which means a new state file is only written when @@ -98,9 +100,9 @@ been processed. This setting can be used to guard against message duplication du to fatal errors (like power fail). Note that this setting affects imfile performance, especially when set to a low value. Frequently writing the state file is very time consuming. -<li><b>$InputFileReadMode</b> [mode]</b><br> +<li><b>ReadMode</b> [mode]</b><br> Available in 5.7.5+ -<li><b>$InputFileMaxLinesAtOnce</b> [number]</b><br> +<li><b>MaxLinesAtOnce</b> [number]</b><br> Available in 5.9.0+ <br> This is useful if multiple files need to be monitored. If set to 0, each file @@ -109,8 +111,16 @@ will be fully processed and then processing switches to the next file [number] lines is processed in sequence for each file, and then the file is switched. This provides a kind of mutiplexing the load of multiple files and probably leads to a more natural distribution of events when multiple busy files -are monitored. The default is 10240. -<li>$InputFileBindRuleset <ruleset><br> +are monitored. The default is 1024. +<li><b>MaxSubmitAtOnce</b> [number]</b><br> +Available in 5.9.0+ +<br> +This is an expert option. It can be used to set the maximum input batch size that +imfile can generate. The default is 1024, which is suitable for a wide range of +applications. Be sure to understand rsyslog message batch processing before you +modify this option. If you do not know what this doc here talks about, this is a +good indication that you should NOT modify the default. +<li><b>Ruleset</b> <ruleset><br> Available in 5.7.5+, 6.1.5+ Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> @@ -132,6 +142,71 @@ your distro puts rsyslog's config files). Note that only commands actually needed need to be specified. The second file uses less commands and uses defaults instead.<br> </p> +<textarea rows="15" cols="60">module(load="folder/to/rsyslog/plugins/imfile/.libs/imfile" PollingInterval="10") #needs to be done just once +# File 1 +input(type="imfile" File="/path/to/file1" +Tag="tag1" +StateFile="/var/spool/rsyslog/statefile1" +Severity="error" +Facility="local7") +# File 2 +input(type="imfile" File="/path/to/file2" +Tag="tag2" +StateFile="/var/spool/rsyslog/statefile2") +# ... and so on ... +# +</textarea> + + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><strong>$InputFileName /path/to/file</strong><br> +equivalent to: File </li> +<li><span style="font-weight: bold;">$InputFileTag +tag:</span><br> +equivalent to: Tag </li> +<li><span style="font-weight: bold;">$InputFileStateFile +<name-of-state-file></span><br> +equivalent to: StateFile </li> +<li><span style="font-weight: bold;">$InputFileFacility +facility</span><br> +equivalent to: Facility </span></li> +<li><span style="font-weight: bold;">$InputFileSeverity</span><br> +equivalent to: Severity</li> +<li><span style="font-weight: bold;">$InputRunFileMonitor</span><br> +This <span style="font-weight: bold;">activates</span> +the current monitor. It has no parameters. If you forget this +directive, no file monitoring will take place.</li> +<li><span style="font-weight: bold;">$InputFilePollInterval +seconds</span><br> +equivalent to: PollingInterva</li> +<li><b>$InputFilePersistStateInterval</b> [lines]</b><br> +equivalent to: PersistStateInterval +<li><b>$InputFileReadMode</b> [mode]</b><br> +equivalent to: ReadMode +<li><b>$InputFileMaxLinesAtOnce</b> [number]</b><br> +equivalent to: MaxLinesAtOnce +<li>$InputFileBindRuleset <ruleset><br> +equivalent to: Ruleset </li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>So far, only 100 files can be monitored. If more are needed, +the source needs to be patched. See define MAX_INPUT_FILES in imfile.c</p><p>Powertop +users may want to notice that imfile utilizes polling. Thus, it is no +good citizen when it comes to conserving system power consumption. We +are currently evaluating to move to inotify(). However, there are a +number of subtle issues, which needs to be worked out first. We will +make the change as soon as we can. If you can afford it, we recommend +using a long polling interval in the mean time. +</p> +<p><b>Sample:</b></p> +<p>The following sample monitors two files. If you need just one, +remove the second one. If you need more, add them according to the +sample ;). This code must be placed in /etc/rsyslog.conf (or wherever +your distro puts rsyslog's config files). Note that only commands +actually needed need to be specified. The second file uses less +commands and uses defaults instead.<br> +</p> <textarea rows="15" cols="60">$ModLoad imfile # needs to be done just once # File 1 diff --git a/doc/impstats.html b/doc/impstats.html index 260c1aa4..64b04a30 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -36,6 +36,10 @@ is 5 (syslog).This is useful for filtering messages.</li> <li>$PStatSeverity <numerical severity><br> The numerical syslog severity code to be used for generated messages. Default is 6 (info).This is useful for filtering messages.</li> +<li>$PStatJSON <on/<b>off</b>> (rsyslog v6.3.8+ only)<br> +If set to on, stats messages are emitted as structured cee-enhanced syslog. If +set to off, legacy format is used (which is compatible with pre v6-rsyslog). +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imptcp.html b/doc/imptcp.html index 065efae6..d301b76f 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -19,11 +19,12 @@ Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a>. <p>Multiple receivers may be configured by specifying $InputPTCPServerRun multiple times. </p> + <p><b>Configuration Directives</b>:</p> <p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> -<li>$InputPTCPServerAddtlFrameDelimiter <Delimiter><br> +<li><b>AddTLFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -43,43 +44,91 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. -<li><b>$InputPTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +<li><b>SupportOctetCountedFraming</b> <<b>on</b>|off><br> If set to "on", the legacy octed-counted framing (similar to RFC5425 framing) is activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> -<li>$InputPTCPServerNotifyOnConnectionClose [on/<b>off</b>]<br> +<li><b>ServerNotifyOnConnectionClose</b> [on/<b>off</b>]<br> instructs imptcp to emit a message if the remote peer closes a connection.<br> -<li><b>$InputPTCPServerKeepAlive</b> <on/<b>off</b>><br> +<li><b>KeepAlive</b> <on/<b>off</b>><br> enable of disable keep-alive packets at the tcp socket layer. The default is to disable them.</li> -<li><b>$InputPTCPServerKeepAlive_probes</b> <number><br> +<li><b>KeepAlive.Probes</b> <number><br> The number of unacknowledged probes to send before considering the connection dead and notifying the application layer. The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. -<li><b>$InputPTCPServerKeepAlive_intvl</b> <number><br> +<li><b>KeepAlive.Interval</b> <number><br> The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime. The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. -<li><b>$InputPTCPServerKeepAlive_time</b> <number><br> +<li><b>KeepAlive.Time</b> <number><br> The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further. The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. -<li><b>$InputPTCPServerRun</b> <port><br> -Starts a TCP server on selected port</li> -<li>$InputPTCPServerInputName <name><br> +<li><b>Port</b> <number><br> +Select a port to listen on</li> +<li><b>Name</b> <name><br> Sets a name for the inputname property. If no name is set "imptcp" is used by default. Setting a name is not strictly necessary, but can be useful to apply filtering based on which input the message was received from. -<li>$InputPTCPServerBindRuleset <name><br> +<li><b>Ruleset</b> <name><br> Binds specified ruleset to next server defined. +<!--<li>$InputPTCPHelperThreads <number><br> +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +helper threads (but not more than there are CPUs/Cores) can help improving +performance. The default value is two.--> +<li><b>Address</b> <name><br> +On multi-homed machines, specifies to which local address the listerner should be bound. +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up a TCP server on port 514:<br> +</p> +<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imptcp/.libs/imptcp") # needs to be done just once +input(type="imptcp" port="514") +</textarea> + +<p><b>Legacy Configuration Directives</b>:</p> +<p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in +their name instead of just TCP. Note that only a subset of the parameters are supported. +<ul> +<li>$InputPTCPServerAddtlFrameDelimiter <Delimiter><br> +Equivalent to: AddTLFrameDelimiter</li> +<li><b>$InputPTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +Equivalent to: SupportOctetCountedFraming +</li> +<li>$InputPTCPServerNotifyOnConnectionClose [on/<b>off</b>]<br> +Equivalent to: ServerNotifyOnConnectionClose.<br></li> +<li><b>$InputPTCPServerKeepAlive</b> <on/<b>off</b>><br> +Equivalent to: KeepAlive </li> +<li><b>$InputPTCPServerKeepAlive_probes</b> <number><br> +Equivalent to: KeepAlive.Probes</li> +<li><b>$InputPTCPServerKeepAlive_intvl</b> <number><br> +Equivalent to: KeepAlive.Interval </li> +<li><b>$InputPTCPServerKeepAlive_time</b> <number><br> +Equivalent to: KeepAlive.Time</li> +<li><b>$InputPTCPServerRun</b> <port><br> +Equivalent to: Port </li> +<li>$InputPTCPServerInputName <name><br> +Equivalent to: Name </li> +<li>$InputPTCPServerBindRuleset <name><br> +Equivalent to: Ruleset </li> +<li>$InputPTCPHelperThreads <number><br> +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +helper threads (but not more than there are CPUs/Cores) can help improving +performance. The default value is two. <li>$InputPTCPServerListenIP <name><br> -On multi-homed machines, specifies to which local address the next listerner should -be bound. +Equivalent to: Address </li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imrelp.html b/doc/imrelp.html index 2cf9c1f7..80ddfd53 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -27,9 +27,12 @@ scenarios also exists with plain tcp syslog. RELP, even with the small nits outlined above, is a much more reliable solution than plain tcp syslog and so it is highly suggested to use RELP instead of plain tcp. Clients send messages to the RELP server via omrelp.</p> + <p><b>Configuration Directives</b>:</p> <ul> -<li>InputRELPServerRun <port><br> +<li><b>Ruleset</b> <name> (available in 6.3.6+)</br> +Binds the specified ruleset to all RELP listeners. +<li><b>Port</b> <port><br> Starts a RELP server on selected port</li> </ul> <b>Caveats/Known Bugs:</b> @@ -38,6 +41,31 @@ Starts a RELP server on selected port</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> +<li>Contrary to other inputs, the ruleset can only be bound to all listeners, +not specific ones. This is due to a currently existing limitation in librelp. +</ul> +<p><b>Sample:</b></p> +<p>This sets up a RELP server on port 20514.<br> +</p> +<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imrelp/.libs/imrelp") # needs to be done just once +input(type="imrelp" port="20514") +</textarea> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li>InputRELPServerBindRuleset <name> (available in 6.3.6+)</br> +equivalent to: RuleSet +<li>InputRELPServerRun <port><br> +equivalent to: Port</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>see description</li> +<li>To obtain the remote system's IP address, you need to have at least +librelp 1.0.0 installed. Versions below it return the hostname instead +of the IP address.</li> +<li>Contrary to other inputs, the ruleset can only be bound to all listeners, +not specific ones. This is due to a currently existing limitation in librelp. </ul> <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> @@ -48,9 +76,8 @@ $InputRELPServerRun 20514 <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imudp.html b/doc/imudp.html index f0e86307..3512d474 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -2,7 +2,7 @@ <html> <head> <meta http-equiv="Content-Language" content="en"> -<title>TCP Syslog Input Module</title> +<title>UDP Syslog Input Module (imudp)</title> </head> <body> @@ -15,16 +15,13 @@ <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via UDP. <p>Multiple receivers may be configured by specifying -$UDPServerRun multiple times. +multiple input actions. </p> + <p><b>Configuration Directives</b>:</p> +<p><b>Global Directives</b>:</p> <ul> -<li>$UDPServerAddress <IP><br> -local IP address (or name) the UDP listens should bind to</li> -<li>$UDPServerRun <port><br> -former -r<port> option, default 514, start UDP server on this -port, "*" means all addresses</li> -<li>$UDPServerTimeRequery <nbr-of-times><br> +<li><b>TimeRequery</b> <nbr-of-times><br> this is a performance optimization. Getting the system time is very costly. With this setting, imudp can be instructed to obtain the precise time only once every n-times. This logic is @@ -33,7 +30,20 @@ time calls should usually be acceptable. The default value is two, because we ha seen that even without optimization the kernel often returns twice the identical time. You can set this value as high as you like, but do so at your own risk. The higher the value, the less precise the timestamp. -<li>$InputUDPServerBindRuleset <ruleset><br> +<li><b>SchedulingPolicy</b> <rr/fifo/other><br> +Can be used the set the scheduler priority, if the necessary functionality +is provided by the platform. Most useful to select "fifo" for real-time +processing under Linux (and thus reduce chance of packet loss). Available since 4.7.4+, 5.7.3+, 6.1.3+. +<li><b>SchedulingPriority</b> <number><br> +Scheduling priority to use. Available since 4.7.4+, 5.7.3+, 6.1.3+. +</ul> +<p><b>Action Directives</b>:</p> +<ul> +<li><b>Address</b> <IP><br> +local IP address (or name) the UDP listens should bind to</li> +<li><b>Port</b> <port><br> +default 514, start UDP server on this port</li> +<li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> @@ -43,6 +53,35 @@ Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> <p><b>Sample:</b></p> <p>This sets up an UPD server on port 514:<br> </p> +<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imudp/.libs/imudp") # needs to be done just once +input(type="imudp" port="514") +</textarea> + +<p><b>Legacy Configuration Directives</b>:</p> +<p>Multiple receivers may be configured by specifying +$UDPServerRun multiple times. +</p> +<ul> +<li>$UDPServerAddress <IP><br> +equivalent to: Address </li> +<li>$UDPServerRun <port><br> +equivalent to: Port </li> +<li>$UDPServerTimeRequery <nbr-of-times><br> +equivalent to: TimeRequery +<li>$InputUDPServerBindRuleset <ruleset><br> +equivalent to: Ruleset </li> +<li>$IMUDPSchedulingPolicy <rr/fifo/other><br> +equivalent to: SchedulingPolicy +<li>$IMUDPSchedulingPriority <number><br> +equivalent to: SchedulingPriority +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>currently none known</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up an UPD server on port 514:<br> +</p> <textarea rows="15" cols="60">$ModLoad imudp # needs to be done just once $UDPServerRun 514 </textarea> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 34a696d9..bd207a37 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -24,11 +24,11 @@ information). This seems to be consistent with what sysklogd did for the past four years. Alternate behaviour may be desirable if gateway-like processes send messages via the local log slot - in this case, it can be enabled via the -$InputUnixListenSocketIgnoreMsgTimestamp and $SystemLogSocketIgnoreMsgTimestamp config directives</p> +IgnoreTimestamp and SysSock.IgnoreTimestamp config directives</p> <p><b>There is input rate limiting available,</b> (since 5.7.1) to guard you against the problems of a wild running logging process. -If more than $SystemLogRateLimitInterval * $SystemLogRateLimitBurst log messages are emitted -from the same process, those messages with $SystemLogRateLimitSeverity or lower will be +If more than SysSock.RateLimit.Interval * SysSock.RateLimit.Burst log messages are emitted +from the same process, those messages with SysSock.RateLimit.Severity or lower will be dropped. It is not possible to recover anything about these messages, but imuxsock will tell you how many it has dropped one the interval has expired AND the next message is logged. Rate-limiting depends on SCM_CREDENTIALS. If the platform does not support @@ -36,7 +36,7 @@ this socket option, rate limiting is turned off. If multiple sockets are configu rate limiting works independently on each of them (that should be what you usually expect). The same functionality is available for additional log sockets, in which case the config statements just use -the prefix $IMUXSockRateLimit... but otherwise works exactly the same. +the prefix RateLimit... but otherwise works exactly the same. When working with severities, please keep in mind that higher severity numbers mean lower severity and configure things accordingly. To turn off rate limiting, set the interval to zero. @@ -46,8 +46,8 @@ the queues (which may cause exessive disk-io where it actually would not be need flow-controlling a log socket (and especially the system log socket) can lead to a very unresponsive system. As such, flow control is disabled by default. That means any log records are places as quickly as possible into the processing queues. If you would like to have -flow control, you need to enable it via the $SystemLogSocketFlowControl and -$InputUnixListenSocketFlowControl config directives. Just make sure you thought about +flow control, you need to enable it via the SysSock.FlowControl and +FlowControl config directives. Just make sure you thought about the implications. Note that for many systems, turning on flow control does not hurt. <p>Starting with rsyslog 5.9.4, <b><a href="http://www.rsyslog.com/what-are-trusted-properties/">trusted syslog properties</a> @@ -57,85 +57,177 @@ privileges are dropped (depending on how they are dropped). Note that trusted pr can be very useful, but also typically cause the message to grow rather large. Also, the format of log messages is obviously changed by adding the trusted properties at the end. For these reasons, the feature is <b>not enabled by default</b>. If you want to use it, -you must turn it on (via $SystemLogSocketAnnotate and $InputUnixListenSocketAnnotate). +you must turn it on (via SysSock.Annotate and Annotate). + <p><b>Configuration Directives</b>:</p> +<p><b>Global Parameters</b></p> <ul> -<li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] +<li><b>SysSock.IgnoreTimestamp</b> [<b>on</b>/off]<br> +Ignore timestamps included in the messages, applies to messages received via the system log socket. +</li> +<li><b>SysSock.Use</b> (imuxsock) [on/<b>off</b>] +do NOT listen for the local log socket. This is most useful if you run multiple +instances of rsyslogd where only one shall handle the system log socket. +</li> +<li><b>SysSock.Name</b> <name-of-socket> +</li> +<li><b>SysSock.FlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied +to the system log socket. +</li> +<li><b>SysSock.UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall +be obtained from the log socket itself. If so, the TAG part of the message is rewritten. +It is recommended to turn this option on, but the default is "off" to keep compatible +with earlier versions of rsyslog. This option was introduced in 5.7.0. +</li> +<li><b>SysSock.RateLimit.Interval</b> [number] - specifies the rate-limiting +interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. +</li> +<li><b>SysSock.RateLimit.Burst</b> [number] - specifies the rate-limiting +burst in number of messages. Default is 200. +</li> +<li><b>SysSock.RateLimit.Severity</b> [numerical severity] - specifies the severity of +messages that shall be rate-limited. +</li> +<li><b>SysSock.UseSysTimeStamp</b> [<b>on</b>/off] the same as $InputUnixListenSocketUseSysTimeStamp, but for the system log socket. +</li> +<li><b>SysSock.Annotate</b> <on/<b>off</b>> turn on annotation/trusted +properties for the system log socket.</li> +</ul> + +<p><b>Input Instance Parameters</b></p> +<ul> +<li><b>IgnoreTimestamp</b> [<b>on</b>/off] <br>Ignore timestamps included in the message. Applies to the next socket being added.</li> -<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied +<li><b>FlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the next socket.</li> -<li><b>$IMUXSockRateLimitInterval</b> [number] - specifies the rate-limiting +<li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen in 5.9.6+, as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> -<li><b>$IMUXSockRateLimitBurst</b> [number] - specifies the rate-limiting +<li><b>RateLimit.Burst</b> [number] - specifies the rate-limiting burst in number of messages. Default is 200. </li> -<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - specifies the severity of +<li><b>RateLimit.Severity</b> [numerical severity] - specifies the severity of messages that shall be rate-limited. </li> -<li><b>$IMUXSockLocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified +<!--<li><b>LocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. -</li> -<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall +</li>--> +<li><b>UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> -<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] instructs imuxsock +<li><b>UseSysTimeStamp</b> [<b>on</b>/off] instructs imuxsock to obtain message time from the system (via control messages) insted of using time recorded inside the message. This may be most useful in combination with systemd. Note: this option was introduced with version 5.9.1. Due to the usefulness of it, we decided to enable it by default. As such, 5.9.1 and above behave slightly different than previous versions. However, we do not see how this could negatively affect existing environments.<br> -<li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> -Ignore timestamps included in the messages, applies to messages received via the system log socket.</li> -<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] -- former -o option; -do NOT listen for the local log socket. This is most useful if you run multiple -instances of rsyslogd where only one shall handle the system log socket.</li> -<li><b>$SystemLogSocketName</b> <name-of-socket> -- former -p option</li> -<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied -to the system log socket.</li> -<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall -be obtained from the log socket itself. If so, the TAG part of the message is rewritten. -It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> -<li><b>$SystemLogRateLimitInterval</b> [number] - specifies the rate-limiting -interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. -</li> -<li><b>$SystemLogRateLimitBurst</b> [number] - specifies the rate-limiting -burst in number of messages. Default is 200. -</li> -<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - specifies the severity of -messages that shall be rate-limited. -</li> -<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] the same as $InputUnixListenSocketUseSysTimeStamp, but for the system log socket. -<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - create directories in the socket path +<li><b>CreatePath</b> [on/<b>off</b>] - create directories in the socket path if they do not already exist. They are created with 0755 permissions with the owner being the process under which rsyslogd runs. The default is not to create directories. Keep in mind, though, that rsyslogd always creates the socket itself if it does not exist (just not the directories by default). <br>Note that this statement affects the -next $AddUnixListenSocket directive that follows in sequence in the configuration file. It never works +next Socket directive that follows in sequence in the configuration file. It never works on the system log socket (where it is deemed unnecessary). Also note that it is automatically -being reset to "off" after the $AddUnixListenSocket directive, so if you would have it active +being reset to "off" after the Socket directive, so if you would have it active for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> -<li><b>$AddUnixListenSocket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> -<li><b>$InputUnixListenSocketHostName</b> <hostname> permits to override the hostname that -shall be used inside messages taken from the <b>next</b> $AddUnixListenSocket socket. Note that +<li><b>Socket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> +<li><b>HostName</b> <hostname> permits to override the hostname that +shall be used inside messages taken from the <b>next</b> Socket socket. Note that the hostname must be specified before the $AddUnixListenSocket configuration directive, and it will only affect the next one and then automatically be reset. This functionality is provided so that the local hostname can be overridden in cases where that is desired.</li> -<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> turn on annotation/trusted +<li><b>Annotate</b> <on/<b>off</b>> turn on annotation/trusted properties for the non-system log socket in question.</li> -<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> turn on annotation/trusted -properties for the system log socket.</li> +</ul> + +<b>Caveats/Known Bugs:</b><br> +<ul> +<li>There is a compile-time limit of 50 concurrent sockets. If you need more, you need to +change the array size in imuxsock.c. +<li>This documentation is sparse and incomplete. +</ul> +<p><b>Sample:</b></p> +<p>The following sample is the minimum setup required to accept syslog messages from applications running +on the local system.<br> +</p> +<textarea rows="2" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +SysSock.FlowControl="on") # enable flow control (use if needed) +</textarea> +<p>The following sample is a configuration where rsyslogd pulls logs from two +jails, and assigns different hostnames to each of the jails: </p> +<textarea rows="6" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock") # needs to be done just once + +input(type="imuxsock" HostName="jail1.example.net" Socket="/jail/1/dev/log") +input(type="imuxsock" HostName="jail2.example.net" Socket="/jail/2/dev/log") +</textarea> +<p>The following sample is a configuration where rsyslogd reads the openssh log +messages via a separate socket, but this socket is created on a temporary file +system. As rsyslogd starts up before the sshd, it needs to create the socket +directories, because it otherwise can not open the socket and thus not listen +to openssh messages. Note that it is vital not to place any other socket between +the CreatePath and the Socket.</p> +<textarea rows="6" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock") # needs to be done just once + +input(type="imuxsock" Socket="/var/run/sshd/dev/log" CreatePath="on") +</textarea> +<p>The following sample is used to turn off input rate limiting on the system log +socket. +<textarea rows="4" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +SysSock.RateLimit.Interval="0") # turn off rate limiting +</textarea> +<p>The following sample is used activate message annotation and thus trusted properties +on the system log socket. +<textarea rows="4" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +SysSock.Annotate="on") +</textarea> + + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] +<br>Please see: IgnoreTimestamp.</li> +<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - Please see: FlowControl .</li> +<li><b>$IMUXSockRateLimitInterval</b> [number] - Please see: RateLimit.Interval +</li> +<li><b>$IMUXSockRateLimitBurst</b> [number] - Please see: RateLimit.Burst +</li> +<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - Please see: RateLimit.Severity +</li> +<li><b>$IMUXSockLocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified +interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. +If this directive is not given OR the interface cannot be found (or has no IP address), +the default of "127.0.0.1" is used. +</li> +<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - Please see: UsePIDFromSystem.</li> +<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] Please see: UseSysTimeStamp .<br> +<li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> +Please see: SysSock.IgnoreTimestamp.</li> +<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] Please see: SysSock.Use</li> +<li><b>$SystemLogSocketName</b> <name-of-socket> Please see: SysSock.Name</li> +<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - Please see: SysSock.FlowControl.</li> +<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - Please see: SysSock.UsePIDFromSystem.</li> +<li><b>$SystemLogRateLimitInterval</b> [number] - Please see: SysSock.RateLimit.Interval. +</li> +<li><b>$SystemLogRateLimitBurst</b> [number] - Please see: SysSock.RateLimit.Burst +</li> +<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - Please see: SysSock.RateLimit.Severity +</li> +<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] Please see: SysSock.UseSysTimeStamp. +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - Please see: CreatePath</li> +<li><b>$AddUnixListenSocket</b> <name-of-socket> Please see: Socket </li> +<li><b>$InputUnixListenSocketHostName</b> <hostname> Please see: HostName.</li> +<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> Please see: Annotate.</li> +<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> Please see: SysSock.Annotate.</li> </ul> <b>Caveats/Known Bugs:</b><br> @@ -169,7 +261,7 @@ the $InputUnixListenSocketCreatePath and the $InputUnixListenSocketHostName.</p> <textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once $InputUnixListenSocketCreatePath on # turn on for *next* socket -$InputUnixListenSocketHostName /var/run/sshd/dev/log +$InputUnixListenSocket /var/run/sshd/dev/log </textarea> <p>The following sample is used to turn off input rate limiting on the system log socket. diff --git a/doc/manual.html b/doc/manual.html index 83b06697..9c7c677e 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 5.10.1 (stable branch) of rsyslog.</b> +<p><b>This documentation is for version 6.6.0 (v6-stable branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -30,15 +30,18 @@ between rsyslog and syslog-ng</a>.</p> <p>If you are upgrading from rsyslog v2 or stock sysklogd, <a href="v3compatibility.html">be sure to read the rsyslog v3 compatibility notes</a>, and if you are upgrading from v3, read the -<a href="v4compatibility.html">rsyslog v4 compatibility notes</a> and +<a href="v4compatibility.html">rsyslog v4 compatibility notes</a>, if you upgrade from v4, read the -<a href="v5compatibility.html">rsyslog v5 compatibility notes</a>. +<a href="v5compatibility.html">rsyslog v5 compatibility notes</a>, and +if you upgrade from v5, read the +<a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. <p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> <p><b>Follow the links below for the</b></p> <ul> <li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> -<li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a></li> +<li><a href="http://www.rsyslog.com/doc/node1.html">rsyslog.conf, new RainerScript-based format (v6+)</a></li> +<li><a href="rsyslog_conf.html">configuration file format (rsyslog.conf)</a></li> <li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> <li> <a href="property_replacer.html">property replacer, an important core component</a></li> <li><a href="bugs.html">rsyslog bug list</a></li> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html new file mode 100644 index 00000000..82f9b6a2 --- /dev/null +++ b/doc/mmnormalize.html @@ -0,0 +1,56 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Log Message Normalization Module (mmnormalize)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Log Message Normalization Module</h1> +<p><b>Module Name: mmnormalize</b></p> +<p><b>Available since: </b>6.1.2+ +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides the capability to normalize log messages via +<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to libee, unstructured text, +like usually found in log messages, can very quickly be parsed and put into +a normal form. This is done so quickly, that it usually should be possible +to normalize events in realtime. +<p>This module is implemented via the output module interface. That means that +mmnormalize should be called just like an action. After it has been called, +the normalized message properties are avaialable and can be access. These properties +are called the "CEE" properties, because liblognorm creates a format that is +inspired by the CEE approach. +<p>Note that mmnormalize should only be called once on each message. Behaviour is +undifined if multiple calls to mmnormalize happen for the same message. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$mmnormalizeRuleBase <rulebase-file><br> +Specifies which rulebase file is to use. This file is loaded. If there are +multiple mmnormalize instances, each one can use a different file. However, +a single instance can use only a single file. This parameter MUST be given, +because normalization can only happen based on a rulebase. +<li>$mmnormalizeUseRawMsg <on/off><br> +Specifies if the raw message should be used for normalization (on) or just the +MSG part of the message (off). Default is "off". +</ul> +<b>Caveats/Known Bugs:</b> +<p>None known at this time. +</ul> +<p><b>Sample:</b></p> +<p>This activates the module and applies normalization to all messages:<br> +</p> +<textarea rows="8" cols="60">$ModLoad mmnormalize +$mmnormalizeRuleBase rulebase.rb +*.* :mmnormalize: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omfile.html b/doc/omfile.html new file mode 100644 index 00000000..23ecc034 --- /dev/null +++ b/doc/omfile.html @@ -0,0 +1,167 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>File Output Module</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>File Output Module</h1> +<p><b>Module Name: omfile</b></p> +<p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> +<p> </p> + +<p><b>Global Configuration Directives</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action specific Configuration Directives</b>:</p> +<ul> + <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> + Defines a template to be used for the output. <br></li><br> + + <li><strong>ZipLevel </strong>0..9 [default 0]<br> + if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + + <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> + Defines a template to be used for the output. <br></li><br> + + <li><strong>ASyncWriting </strong>on/off [default off]<br> + if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable FlushInterval, AsyncWriting must be set to "on". Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is "on" but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. <br></li><br> + + <li><strong>FlushOnTXEnd </strong>on/off [default on]<br> + Omfile has the capability to write output using a buffered writer. Disk writes are only done when the buffer is full. So if an error happens during that write, data is potentially lost. In cases where this is unacceptable, set FlushOnTXEnd to on. Then, data is written at the end of each transaction (for pre-v5 this means after each log message) and the usual error recovery thus can handle write errors without data loss. Note that this option severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -on- is primarily an aid to preserve the traditional syslogd behaviour.<br></li><br> + + <li><strong>IOBufferSize </strong><size_nbr>, default 4k<br> + size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)<br></li><br> + + <li><strong>DirOwner </strong><br> + Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>DirGroup </strong><br> + Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>FileOwner </strong><br> + Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>FileGroup </strong><br> + Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>DirCreateMode </strong>[defaul 0700]<br> + This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> + + <li><strong>FileCreateMode </strong>[default 0644]<br> + The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> + + <li><strong>FailOnCHOwnFailure </strong>on/off [default on]<br> + This option modifies behaviour of dynaFile creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> + + <li><strong>CreateDirs </strong>on/off [default on]<br> + create directories on an as-needed basis<br></li><br> + + <li><strong>Sync </strong>on/off [default off]<br> + enables file syncing capability of omfile.<br></li><br> + + <li><strong>File </strong><br> + If the file already exists, new data is appended to it. Existing data is not truncated. If the file does not already exist, it is created. Files are kept open as long as rsyslogd is active. This conflicts with external log file rotation. In order to close a file after rotation, send rsyslogd a HUP signal after the file has been rotated away. <br></li><br> + + <li><strong>DynaFile </strong><br> + For each message, the file name is generated based on the given template. Then, this file is opened. As with the ``file'' property, data is appended if the file already exists. If the file does not exist, a new file is created. A cache of recent files is kept. Note that this cache can consume quite some memory (especially if large buffer sizes are used). Files are kept open as long as they stay inside the cache. Currently, files are only evicted from the cache when there is need to do so (due to insufficient cache size). To force-close (and evict) a dynafile from cache, send a HUP signal to rsyslogd. <br></li><br> + + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li><br> + +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command writes all syslog messages into a file.</p> +<textarea rows="5" cols="60">Module (path="builtin:omfile") +*.* action(type="omfile" +DirCreateMode="0700" +FileCreateMode="0644" +File="/var/log/messages") +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> + <li><strong>$DynaFileCacheSize </strong>(not mandatory, default will be used)<br> + Defines a template to be used for the output. <br></li><br> + + <li><strong>$OMFileZipLevel </strong>0..9 [default 0]<br> + if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + + <li><strong>$OMFileFlushInterval </strong>(not mandatory, default will be used)<br> + Defines a template to be used for the output. <br></li><br> + + <li><strong>$OMFileASyncWriting </strong>on/off [default off]<br> + if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable FlushInterval, AsyncWriting must be set to "on". Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is "on" but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. <br></li><br> + + <li><strong>$OMFileFlushOnTXEnd </strong>on/off [default on]<br> + Omfile has the capability to write output using a buffered writer. Disk writes are only done when the buffer is full. So if an error happens during that write, data is potentially lost. In cases where this is unacceptable, set FlushOnTXEnd to on. Then, data is written at the end of each transaction (for pre-v5 this means after each log message) and the usual error recovery thus can handle write errors without data loss. Note that this option severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -on- is primarily an aid to preserve the traditional syslogd behaviour.<br></li><br> + + <li><strong>$OMFileIOBufferSize </strong><size_nbr>, default 4k<br> + size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)<br></li><br> + + <li><strong>$DirOwner </strong><br> + Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>$DirGroup </strong><br> + Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>$FileOwner </strong><br> + Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>$FileGroup </strong><br> + Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>$DirCreateMode </strong>[defaul 0700]<br> + This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> + + <li><strong>$FileCreateMode </strong>[default 0644]<br> + The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> + + <li><strong>$FailOnCHOwnFailure </strong>on/off [default on]<br> + This option modifies behaviour of dynaFile creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> + + <li><strong>$F$OMFileForceCHOwn </strong><br> + force ownership change for all files<br></li><br> + + <li><strong>$CreateDirs </strong>on/off [default on]<br> + create directories on an as-needed basis<br></li><br> + + <li><strong>$ActionFileEnableSync </strong>on/off [default off]<br> + enables file syncing capability of omfile.<br></li><br> + + <li><strong>$ActionFileDefaultTemplate </strong>[templateName]<br> + sets a new default template for file actions.<br></li><br> + + <li><strong>$ResetConfigVariables </strong><br> + Resets all configuration variables to their default value. Any settings made will not be applied to configuration lines following the $ResetConfigVariables. This is a good method to make sure no side-effects exists from previous directives. This directive has no parameters.<br></li><br> + +</ul> + +<p><b>Legacy Sample:</b></p> +<p>The following command writes all syslog messages into a file.</p> +<textarea rows="5" cols="60">$ModLoad omfile +$DirCreateMode 0700 +$FileCreateMode 0644 +*.* /var/log/messages +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omfwd.html b/doc/omfwd.html new file mode 100644 index 00000000..5599ae39 --- /dev/null +++ b/doc/omfwd.html @@ -0,0 +1,118 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Forwarding Output Module</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Forwarding Output Module</h1> +<p><b>Module Name: omfwd</b></p> +<p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>The omfwd plug-in provides the core functionality of traditional message forwarding via UDP and plain TCP. It is a built-in module that does not need to be loaded. </p> +<p> </p> + +<p><b>Global Configuration Directives</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action specific Configuration Directives</b>:</p> +<ul> + <li><strong>Target </strong>string<br> + Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> + + <li><strong>Port </strong>[Default 514]<br> + Name or numerical value of port to use when connecting to target. <br></li><br> + + <li><strong>Protocol </strong>udp/tcp [default udp]<br> + Type of protocol to use for forwarding. Note that ``tcp'' means both legacy plain tcp syslog as well as RFC5425-based TCL-encrypted syslog. Which one is selected depends on the protocol drivers set before the action commend. Note that as of 6.3.6, there is no way to specify this within the action itself. <br></li><br> + + <li><strong>TCP_Framing </strong>``traditional'' or ``octet-counted'' [default traditional]<br> + Framing-Mode to be for forwarding. This affects only TCP-based protocols. It is ignored for UDP. In protocol engineering, ``framing'' means how multiple messages over the same connection are separated. Usually, this is transparent to users. Unfortunately, the early syslog protocol evolved, and so there are cases where users need to specify the framing. The traditional framing is nontransparent. With it, messages are end when a LF (aka ``line break'', ``return'') is encountered, and the next message starts immediately after the LF. If multi-line messages are received, these are essentially broken up into multiple message, usually with all but the first message segment being incorrectly formatted. The octet-counting framing solves this issue. With it, each message is prefixed with the actual message length, so that a receivers knows exactly where the message ends. Multi-line messages cause no problem here. This mode is very close to the method described in RFC5425 for TLS-enabled syslog. Unfortunately, only few syslogd implementations support octet-counted framing. As such, the traditional framing is set as default, even though it has defects. If it is known that the receiver supports octet-counted framing, it is suggested to use that framing mode. <br></li><br> + + <li><strong>ZipLevel </strong>0..9 [default 0]<br> + Compression level for messages. Rsyslog implements a proprietary capability to zip transmitted messages. Note that compression happens on a message-per-message basis. As such, there is a performance gain only for larger messages. Before compressing a message, rsyslog checks if there is some gain by compression. If so, the message is sent compressed. If not, it is sent uncompressed. As such, it is totally valid that compressed and uncompressed messages are intermixed within a conversation. <br>The compression level is specified via the usual factor of 0 to 9, with 9 being the strongest compression (taking up most processing time) and 0 being no compression at all (taking up no extra processing time). <br></li><br> + + <li><strong>RebindInterval </strong>integer<br> + Permits to specify an interval at which the current connection is broken and re-established. This setting is primarily an aid to load balancers. After the configured number of messages has been transmitted, the current connection is terminated and a new one started. Note that this setting applies to both TCP and UDP traffic. For UDP, the new ``connection'' uses a different source port (ports are cycled and not reused too frequently). This usually is perceived as a ``new connection'' by load balancers, which in turn forward messages to another physical target system. <br></li><br> + + <li><strong>StreamDriver </strong>string<br> + Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + + <li><strong>StreamDriverMode </strong>integer [default 0]<br> + mode to use with the stream driver (driver-specific)<br></li><br> + + <li><strong>StreamDriverAuthMode </strong>string<br> + authentication mode to use with the stream driver. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific).<br></li><br> + + <li><strong>StreamDriverPermittedPeers </strong>string<br> + accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific)<br></li><br> + + <li><strong>ResendLastMSGOnReconnect </strong>on/off<br> + Permits to resend the last message when a connection is reconnected. This setting affects TCP-based syslog, only. It is most useful for traditional, plain TCP syslog. Using this protocol, it is not always possible to know which messages were successfully transmitted to the receiver when a connection breaks. In many cases, the last message sent is lost. By switching this setting to "yes", rsyslog will always retransmit the last message when a connection is reestablished. This reduces potential message loss, but comes at the price that some messages may be duplicated (what usually is more acceptable). <br></li><br> + +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> +<textarea rows="5" cols="60">Module (path="builtin:omfwd") +*.* action(type="omfwd" +Target="192.168.2.11" +Port="10514" +Protocol="tcp" +) +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> + <li><strong>$ActionForwardDefaultTemplateName </strong>string [templatename]<br> + sets a new default template for UDP and plain TCP forwarding action<br></li><br> + + <li><strong>$ActionSendTCPRebindInterval </strong>integer<br> + instructs the TCP send action to close and re-open the connection to the remote host every nbr of messages sent. Zero, the default, means that no such processing is done. This directive is useful for use with load-balancers. Note that there is some performance overhead associated with it, so it is advisable to not too often "rebind" the connection (what "too often" actually means depends on your configuration, a rule of thumb is that it should be not be much more often than once per second).<br></li><br> + + <li><strong>$ActionSendUDPRebindInterval </strong>integer<br> + instructs the UDP send action to rebind the send socket every nbr of messages sent. Zero, the default, means that no rebind is done. This directive is useful for use with load-balancers.<br></li><br> + + <li><strong>$ActionSendStreamDriver </strong><driver basename><br> + just like $DefaultNetstreamDriver, but for the specific action <br></li><br> + + <li><strong>$ActionSendStreamDriverMode </strong><mode> [default 0]<br> + mode to use with the stream driver (driver-specific)<br></li><br> + + <li><strong>$ActionSendStreamDriverAuthMode </strong><mode><br> + authentication mode to use with the stream driver. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific))<br></li><br> + + <li><strong>$ActionSendStreamDriverPermittedPeers </strong><ID><br> + accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific) <br></li><br> + + <li><strong>$ActionSendResendLastMsgOnReconnect </strong>on/off [default off]<br> + specifies if the last message is to be resend when a connecition breaks and has been reconnected. May increase reliability, but comes at the risk of message duplication. <br></li><br> + + <li><strong>$ResetConfigVariables </strong><br> + Resets all configuration variables to their default value. Any settings made will not be applied to configuration lines following the $ResetConfigVariables. This is a good method to make sure no side-effects exists from previous directives. This directive has no parameters.<br></li><br> + +</ul> + +<p><b>Legacy Sample:</b></p> +<p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> +<textarea rows="5" cols="60">$ModLoad omfwd +*.* @@192.168.2.11:10514 +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html index ec1d01b6..008dcb81 100644 --- a/doc/omlibdbi.html +++ b/doc/omlibdbi.html @@ -54,32 +54,38 @@ dlopen()ed plugin (as omlibdbi is). So in short, you probably save you a lot of headache if you make sure you have at least libdbi version 0.8.3 on your system. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> -<li><span style="font-weight: bold;">$ActionLibdbiDriverDirectory /path/to/dbd/drivers</span><br>This -is a global setting. It points libdbi to its driver directory. Usually, -you do not need to set it. If you installed libdbi-driver's at a -non-standard location, you may need to specify the directory here. If -you are unsure, do <span style="font-weight: bold;">not</span> use this configuration directive. Usually, everything works just fine.<strong></strong></li><li><strong>$ActionLibdbiDriver drivername</strong><br> +<li><b>server</b><br>Name or address of the MySQL server +<li><b>db</b><br>Database to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +<li><b>driver</b><br> Name of the dbidriver to use, see libdbi-drivers documentation. As a quick excerpt, at least those were available at the time of this writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", "freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use ompgsql instead).</li> -<li><span style="font-weight: bold;">$ActionLibdbiHost -hostname</span><br> +<li><b>driverdirectory</b><br> +Path to the libdbi drivers. Usually, +you do not need to set it. If you installed libdbi-drivers at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <b>not</b> use this configuration directive. +Usually, everything works just fine.</li> +</ul> +<p><b>Legacy (pre-v6) Configuration Directives</b>:</p> +<ul> +<li><b>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</b> +- like the driverdirectory action parameter. +<li><strong>$ActionLibdbiDriver drivername</strong><br> - like the drivername action parameter. +<li><span style="font-weight: bold;">$ActionLibdbiHost hostname</span> - like the server action parameter The host to connect to.</li> -<li><span style="font-weight: bold;">$ActionLibdbiUserName -user</span><br> -The user used to connect to the database.</li> -<li><span style="font-weight: bold;">$ActionlibdbiPassword</span><br> -That user's password.</li> -<li><span style="font-weight: bold;">$ActionlibdbiDBName -db</span><br> -The database that shall be written to.</li> -<li><span style="font-weight: bold;">selector -line: :omlibdbi:<span style="font-style: italic;">;template</span></span><br> +<li><b>$ActionLibdbiUserName user</b> - like the uid action parameter +<li><b>$ActionlibdbiPassword</b> - like the pwd action parameter +<li><b>$ActionlibdbiDBName db</b> - like the db action parameter +<li><b>selector line: :omlibdbi:<i>;template</i></b><br> executes the recently configured omlibdbi action. The ;template part is optional. If no template is provided, a default template is used (which is currently optimized for MySQL - sorry, folks...)</li> @@ -108,7 +114,14 @@ database "syslog_db" on mysqlsever.example.com. The server is MySQL and being accessed under the account of "user" with password "pwd" (if you have empty passwords, just remove the $ActionLibdbiPassword line).<br> </p> -<textarea rows="15" cols="60">$ModLoad omlibdbi +<textarea rows="5" cols="60">$ModLoad omlibdbi +*.* action(type="omlibdbi" driver="mysql" + server="mysqlserver.example.com" db="syslog_db" + uid="user" pwd="pwd" +</textarea> +<p><b>Sample:</b></p> +<p>The same as above, but in legacy config format (pre rsyslog-v6): +<textarea rows="10" cols="60">$ModLoad omlibdbi $ActionLibdbiDriver mysql $ActionLibdbiHost mysqlserver.example.com $ActionLibdbiUserName user @@ -121,8 +134,7 @@ $ActionLibdbiDBName syslog_db <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. -Released under the GNU GPL version 3 or higher.</font></p> +Released under the ASL 2.0.</font></p> </body></html> diff --git a/doc/ommysql.html b/doc/ommysql.html index daef9cab..7769fb86 100644 --- a/doc/ommysql.html +++ b/doc/ommysql.html @@ -15,28 +15,37 @@ <p>This module provides native support for logging to MySQL databases. It offers superior performance over the more generic <a href="omlibdbi.html">omlibdbi</a> module. </p> -<p><b>Configuration Directives</b>:</p> -<p>ommysql mostly uses the "old style" configuration, with almost everything on the -action line itself. A few newer features are being migrated to the new style-config -directive configuration system. +<p><b>Action Parameters</b>:</p> <ul> -<li><b>$ActionOmmysqlServerPort <port></b><br>Permits to select +<li><b>server</b><br>Name or address of the MySQL server +<li><b>serverport</b><br>Permits to select a non-standard port for the MySQL server. The default is 0, which means the -system default port is used. There is no need to specify this directive unless +system default port is used. There is no need to specify this parameter unless you know the server is running on a non-standard listen port. -<li><b>$OmMySQLConfigFile <file name></b><br>Permits the selection +<li><b>db</b><br>Database to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +<li><b>mysqlconfig.file</b><br>Permits the selection of an optional MySQL Client Library configuration file (my.cnf) for extended configuration functionality. The use of this configuration directive is necessary only if you have a non-standard environment or if fine-grained control over the database connection is desired.</li> -<li><b>$OmMySQLConfigSection <string></b><br>Permits the selection of the -section within the configuration file specified by the <b>$OmMySQLConfigFile</b> directive. +<li><b>mysqlconfig.section</b><br>Permits the selection of the +section within the configuration file specified by the <b>myselconfig.file</b> parameter. <br>This will likely only be used where the database administrator provides a single configuration file with multiple profiles. -<br>This configuration directive is ignored unless <b>$OmMySQLConfigFile</b> is also used -in the rsyslog configration file. +<br>This configuration parameter is ignored unless <b>mysqlconfig.file</b> is also used. <br>If omitted, the MySQL Client Library default of "client" will be used.</li> -<li>Action parameters: +</ul> +<p><b>Legacy (pre-v6) Configuration Directives</b>:</p> +<p>ommysql mostly uses the "very old style" (v0) configuration, with almost everything on the +action line itself. +<ul> +<li><b>$ActionOmmysqlServerPort <port></b> - like the "serverport" action parameter. +<li><b>$OmMySQLConfigFile <file name></b> - like the "mysqlconfig.file" action parameter. +<li><b>$OmMySQLConfigSection <string></b> - like the "mysqlconfig.file" action parameter. +<li>Action line: <br><b>:ommysql:database-server,database-name,database-userid,database-password</b> <br>All parameters should be filled in for a successful connect. </ul> @@ -57,15 +66,20 @@ database "syslog_db" on mysqlsever.example.com. The server is being accessed under the account of "user" with password "pwd". </p> <textarea rows="5" cols="80">$ModLoad ommysql +*.* action(type="ommysql" server="mysqlserver.example.com" serverport="1234" + db="syslog_db" uid="user" pwd="pwd") +</textarea> +<p><b>Legacy Sample:</b></p> +<p>The same as above, but in legacy config format (pre rsyslog-v6): +<textarea rows="5" cols="80">$ModLoad ommysql $ActionOmmysqlServerPort 1234 # use non-standard port *.* :ommysql:mysqlserver.example.com,syslog_db,user,pwd </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. -Released under the GNU GPL version 3 or higher.</font></p> +Released under the ASL 2.0.</font></p> </body></html> diff --git a/doc/ompipe.html b/doc/ompipe.html new file mode 100644 index 00000000..49915b78 --- /dev/null +++ b/doc/ompipe.html @@ -0,0 +1,62 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Pipe Output Module</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Pipe Output Module</h1> +<p><b>Module Name: omfwd</b></p> +<p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>The ompipe plug-in provides the core functionality for logging output to named pipes (fifos). It is a built-in module that does not need to be loaded. </p> +<p> </p> + +<p><b>Global Configuration Directives</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action specific Configuration Directives</b>:</p> +<ul> + <li><strong>Pipe </strong>string<br> + A fifo or named pipe can be used as a destination for log messages.<br></li><br> + + + +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> +<textarea rows="5" cols="60">Module (path="builtin:ompipe") +*.* action(type="ompipe" +Pipe="NameofPipe" +) +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> +<p>rsyslog has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a destination for log messages by prepending a pipe symbol ("|'') to the name of the file. This is handy for debugging. Note that the fifo must be created with the mkfifo(1) command before rsyslogd is started. + +</p> + +<p><b>Legacy Sample:</b></p> +<p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> +<textarea rows="5" cols="60">$ModLoad ompipe +*.* |/var/log/pipe +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omsnmp.html b/doc/omsnmp.html index b38a594f..202bb5bb 100644 --- a/doc/omsnmp.html +++ b/doc/omsnmp.html @@ -22,6 +22,153 @@ developer (headers) package installed. </p> <p> </p> <p><b>Configuration Directives</b>:</p> <ul> + <li><strong>transport </strong>(This parameter is optional, the + default value is "udp")<br> + <br> + Defines the transport type you wish to use. Technically we can support all + transport types which are supported by NET-SNMP. <br> + To name a few possible values: <br> + <br> + udp, tcp, udp6, tcp6, icmp, icmp6 ...<br> + <br> + Example: <strong>transport udp<br> + </strong></li> + <li><strong>server</strong><br> + <br> + This can be a hostname or ip address, and is our snmp target host. This + parameter is required, if the snmptarget is not defined, nothing will be + send. <br> + <br> + Example: <strong>server server.domain.xxx</strong><br> + </li> + <li><strong>port </strong>(This parameter is optional, the + default value is "162")<br> + <br> + The port which will be used, common values are port 162 or 161. <br> + <br> + Example: <strong>port 162</strong><br> + </li> + <li><strong>version </strong>(This parameter is optional, the + default value is "1")<br> + <br> + There can only be two choices for this parameter for now. <br> + 0 means SNMPv1 will be used.<br> + 1 means SNMPv2c will be used. <br> + Any other value will default to 1. <br> + <br> + Example: <strong>version 1</strong><br> + </li> + <li><strong>community </strong>(This parameter is optional, the + default value is "public")<br> + <br> + This sets the used SNMP Community.<br> + <br> + Example:<strong> community public<br> + </strong><br> + </li> + <li><strong>trapoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogtrap")<br> + This configuration parameter is used for <strong>SNMPv2</strong> only.<br> + <br> + This is the OID which defines the trap-type, or notifcation-type rsyslog + uses to send the trap. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. Downloads of these mib files + can be found here: <br> + <a href="http://www.adiscon.org/download/ADISCON-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MIB.txt</a><br> + <a href="http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt</a><br> + <br> + Thanks to the net-snmp + mailinglist for the help and the recommendations ;).<br> + <br> + Example: <strong>trapoid 1.3.6.1.4.1.19406.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>trapoid ADISCON-MONITORWARE-MIB::syslogtrap<br> + </strong> + </li> + <li><strong>messageoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogMsg")<br> + <br> + This OID will be used as a variable, type "OCTET STRING". This variable will + contain up to 255 characters of the original syslog message including syslog header. It is recommend to + use the default OID. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. + To download these custom mibs, see the description of <strong>$actionsnmptrapoid. + </strong><br> + <br> + Example: <strong>messageoid 1.3.6.1.4.1.19406.1.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>messageoid + ADISCON-MONITORWARE-MIB::syslogMsg<br> + </strong><br> + </li> + <li><strong>enterpriseoid </strong>(This parameter is optional, + the default value is "1.3.6.1.4.1.3.1.1" which means "enterprises.cmu.1.1")<br> + <br> + Customize this value if needed. I recommend to use the default value unless + you require to use a different OID. <br> + This configuration parameter is used for <strong>SNMPv1</strong> only. It + has no effect if <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>enterpriseoid 1.3.6.1.4.1.3.1.1 <br> + </strong><br> + </li> + <li><strong>specifictype </strong>(This parameter is optional, + the default value is "0")<strong> </strong><br> + <br> + This is the specific trap number. This configuration parameter is used for + <strong>SNMPv1</strong> only. It has no effect if <strong>SNMPv2</strong> is + used. <br> + <br> + Example: <strong>specifictype 0<br> + </strong><br> + </li> + <li><strong>traptype</strong> (This parameter is optional, the + default value is "6" which means SNMP_TRAP_ENTERPRISESPECIFIC) <br> + <br> + There are only 7 Possible trap types defined which can be used here. These + trap types are: <br> + 0 = SNMP_TRAP_COLDSTART<br> + 1 = SNMP_TRAP_WARMSTART<br> + 2 = SNMP_TRAP_LINKDOWN<br> + 3 = SNMP_TRAP_LINKUP<br> + 4 = SNMP_TRAP_AUTHFAIL<br> + 5 = SNMP_TRAP_EGPNEIGHBORLOSS<br> + 6 = SNMP_TRAP_ENTERPRISESPECIFIC<br> + <br> + Any other value will default to 6 automatically. This configuration + parameter is used for <strong>SNMPv1</strong> only. It has no effect if + <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>traptype 6</strong><br> + </li> + <li><strong>template </strong>[templateName]<strong> </strong><br> + <br> + sets a new default template for file actions. + </li> +</ul> +<p> </p> +<p><b>Caveats/Known Bugs:</b></p><ul><li>In order to decode the custom OIDs, you + will need to have the adiscon mibs installed. </li></ul> +<p><b>Sample:</b></p> +<p>The following commands send every message as a snmp trap.</p> +<textarea rows="10" cols="60">Module (path="omsnmp") +*.* action( type="omsnmp" +transport="udp" +target="localhost" +targetport="162" +version="1" +community="public") + +</textarea> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> <li><strong>$actionsnmptransport </strong>(This parameter is optional, the default value is "udp")<br> <br> @@ -164,6 +311,7 @@ $actionsnmpcommunity public *.* :omsnmp: </textarea> + <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/omusrmsg.html b/doc/omusrmsg.html new file mode 100644 index 00000000..eccfef2d --- /dev/null +++ b/doc/omusrmsg.html @@ -0,0 +1,64 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>User Message Output Module</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>User Message Output Module</h1> +<p><b>Module Name: omusrmsg</b></p> +<p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>The omusrmsg plug-in provides the core functionality for logging output to a logged on user. It is a built-in module that does not need to be loaded. </p> +<p> </p> + +<p><b>Global Configuration Directives</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action specific Configuration Directives</b>:</p> +<ul> + <li><strong>Users </strong>string<br> + Must be a valid user name or root.<br></li><br> + + + +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command sends all critical syslog messages to a user and to root.</p> +<textarea rows="5" cols="60">Module (path="builtin:omusrmsg") +*.=crit action(type="omusrmsg" +Users="ExampleUser" +Users="root" +) +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> +<p> + No specific configuration directives available. See configuration sample below for details on using the plugin. +</p> + +<p><b>Legacy Sample:</b></p> +<p>The following command sends all critical syslog messages to a user and to root.</p> +<textarea rows="5" cols="60">$ModLoad omusrmsg +*.=crit :omusrmsg:exampleuser +& root +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omuxsock.html b/doc/omuxsock.html index 5fa569eb..a1c09228 100644 --- a/doc/omuxsock.html +++ b/doc/omuxsock.html @@ -29,7 +29,7 @@ actions and each of them should use the same template.</li> <p><b>Sample:</b></p> <p>The following sample writes all messages to the "/tmp/socksample" socket. </p> -<textarea rows="4" cols="80">$ModLoad omucsock +<textarea rows="4" cols="80">$ModLoad omuxsock $OMUxSockSocket /tmp/socksample *.* :omuxsock: </textarea> diff --git a/doc/pmlastmsg.html b/doc/pmlastmsg.html index 2abeac6a..fd26dbd5 100644 --- a/doc/pmlastmsg.html +++ b/doc/pmlastmsg.html @@ -27,6 +27,13 @@ parser chain</a>. It processes all those messages that contain a PRI, then none some spaces and then the exact text (case-insensitive) "last message repeated n times" where n must be an integer. All other messages are left untouched. +<p><b>Please note:</b> this parser module makes it possible that these messages +are properly detected. It does <b>not</b> drop them. If you intend to drop those +messages, you need to use the usual filter logic in combination with the discard +action. As a side-note, please keep on your mind that the sender discarded messages +when the "last message repeated n times" message is emited. You want to consider if +that really is what you intend to happen. If not, go change the sender. + <p><b>Configuration Directives</b>:</p> <p>There do not currently exist any configuration directives for this module. <p><b>Examples:</b></p> diff --git a/doc/property_replacer.html b/doc/property_replacer.html index 5dbdc4c6..dc09d33c 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -13,7 +13,7 @@ the value, e.g. by converting all characters to lower case.</p> <p>Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by the property replacer. The full syntax is as follows:</p> -<blockquote><b><code>%propname:fromChar:toChar:options%</code></b></blockquote> +<blockquote><b><code>%propname:fromChar:toChar:options:fieldname%</code></b></blockquote> <h2>Available Properties</h2> <p><b><code>propname</code></b> is the name of the property to access. It is case-insensitive (prior to 3.17.0, they were case-senstive). @@ -65,8 +65,7 @@ BSD syslogd. For example, when TAG is "named[12345]", programname is </tr> <tr> <td><b>pri-text</b></td> -<td>the PRI part of the message in a textual form with the numerical PRI appended in -brackes (e.g. "local0.err<133>")</td> +<td>the PRI part of the message in textual form (e.g. "syslog.info")</td> </tr> <tr> <td><b>iut</b></td> @@ -139,6 +138,25 @@ draft-ietf-syslog-protocol</td> <td>The contents of the MSGID field from IETF draft draft-ietf-syslog-protocol</td> </tr> +<tr> +<td><b>parsesuccess</b></td> +<td>This returns the status of the <b>last</b> called higher level parser, +like mmjsonparse. A higher level parser parses the actual message for additional +structured data and maintains an extra property table while doing so (this is +often referred to as "cee data" because the idea was originally rooted in the +cee effort, only (but has been extended since then). Note that higher level +parsers must explicitely support (and set) this property. So, depending on the +parser, it may not be set correctly. +<br>If the parser properly supports it, the value "OK" means that parsing was +successfull, while "FAIL" means the parser could not successfully obtain any data. +Failure state is not necessarily an error. For example, it may simple indicate +that the cee-enhanced syslog parser (mmjsonparse) did not detect cee-enhanced format, +what can be totally valid. Using this property, further processing of the message +can be directed based on this parsing outcome. If no parser has been called at the +time this property is accessed, it will contain "FAIL". +<br><b>This property is available since version 6.3.8.</b> +</td> +</tr> <td><b>inputname</b></td> <td>The name of the input module that generated the message (e.g. "imuxsock", "imudp"). Note that not all modules @@ -161,6 +179,10 @@ than messages generated somewhere. in templates for RFC5424 support, when the character set is know to be Unicode.</td> </tr> +<td><b>$uptime</b></td> +<td>system-uptime in seconds (as reported by operating system). +</td> +</tr> <tr> <td><b>$now</b></td> <td>The current date stamp in the format YYYY-MM-DD</td> @@ -201,12 +223,20 @@ range from 0 to 3 (for the four quater hours that are in each hour)</td> <td>The name of the current host as it knows itself (probably useful for filtering in a generic way)</td> </tr> +<tr> +<td><b>$!<name></b></td> +<td>This is the "bridge" to syslog message normalization (via +<a href="mmnormalize.html">mmnormalize</a>): name is a name defined +inside the normalization rule. It has the value selected by the rule +or none if no rule with this field did match. +</td> +</tr> </tbody> </table> <p>Properties starting with a $-sign are so-called system properties. These do NOT stem from the message but are rather internally-generated.</p> -<h2>Character Positions</h2> +<h2>Legacy Character Positions</h2> <p><b><code>FromChar</code></b> and <b><code>toChar</code></b> are used to build substrings. They specify the offset within the string that should be copied. Offset counting starts at 1, so if you need to @@ -274,15 +304,6 @@ fields in the property is requested. The field number must be placed in the "ToChar" parameter. An example where the 3rd field (delimited by TAB) from the msg property is extracted is as follows: "%msg:F:3%". The same example with semicolon as delimiter is "%msg:F,59:3%".</p> -<p>The use of fields does not permit to select substrings, what is rather -unfortunate. To solve this issue, starting with 6.3.9, fromPos and toPos -can be specified for strings as well. However, the syntax is quite ugly, but -it was the only way to integrate this functonality into the already-existing -system. To do so, use ",fromPos" and ",toPos" during field extraction. -Let's assume you want to extract the substring from position 5 to 9 in the previous -example. Then, the syntax is as follows: "%msg:F,59,5:3,9%". As you can see, -"F,59" means field-mode, with semicolon delimiter and ",5" means starting -at position 5. Then "3,9" means field 3 and string extraction to position 9. <p>Please note that the special characters "F" and "R" are case-sensitive. Only upper case works, lower case will return an error. There are no white spaces permitted inside the sequence (that will lead @@ -312,6 +333,243 @@ It is modeled after perl compatible regular expressions. <h2>Property Options</h2> <b><code>property options</code></b> are +case-insensitive. They are available as of version 6.5.0. +Currently, the following options are defined: +<p></p> +<table> +<tbody> +<tr> +<td><b>Name</b></td> +<td>New format. Name of the template / property / constant.</td> +</tr> +<tr> +<td><b>Outname</b></td> +<td>This field permits to specify a field name for structured-data emitting property replacer options. +It is most useful to set, for example, the name for JSON-based fields (like used in ommngodb). For +text-based modules, it is simply ignored. +If not specified, the original property name is used, with the exception of properties starting with +"$!", where that prefix is removed. Note that unnamaned constants are NOT forwarded to output modules +that expect structure (like ommnogodb). To pass constants, an outname must be set. +</tr> +<tr> +<td><b>CaseConversion</b></td> +<td>New format. Additional values below.</td> +</tr> +<tr> +<td>upper</td> +<td>convert property to lowercase only</td> +</tr> +<tr> +<td>lower</td> +<td>convert property text to uppercase only</td> +</tr> +<tr> +<td><b>DateFormat</b></td> +<td>New format, additional parameter is needed. See below.</td> +</tr> +<tr> +<td>mysql</td> +<td>format as mysql date</td> +</tr> +<tr> +<td>pgsql</td> +<td>format as pgsql date</td> +</tr> +<tr> +<td>rfc3164</td> +<td>format as RFC 3164 date</td> +</tr> +<tr> +<tr> +<td valign="top">rfc3164-buggyday</td> +<td>similar to date-rfc3164, but emulates a common coding error: RFC 3164 demands +that a space is written for single-digit days. With this option, a zero is +written instead. This format seems to be used by syslog-ng and the +date-rfc3164-buggyday option can be used in migration scenarios where otherwise +lots of scripts would need to be adjusted. It is recommended <i>not</i> to use this +option when forwarding to remote hosts - they may treat the date as invalid +(especially when parsing strictly according to RFC 3164).</td> +<br><i>This feature was introduced in rsyslog 4.6.2 and v4 versions above and +5.5.3 and all versions above.</i> +</tr> +<tr> +<td>rfc3339</td> +<td>format as RFC 3339 date</td> +</tr> +<tr> +<td>unixtimestamp</td> +<td>format as unix timestamp (seconds since epoch)</td> +</tr> +<tr> +<td>subseconds</td> +<td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> +</tr> +<tr> +<td><b>ControlCharacters</b></td> +<td>Option values for how to process control characters</td> +</tr> +<tr> +<td valign="top">escape</td> +<td>replace control characters (ASCII value 127 and values +less then 32) with an escape sequence. The sequnce is +"#<charval>" where charval is the 3-digit decimal value +of the control character. For example, a tabulator would be replaced by +"#009".<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top">space</td> +<td>replace control characters by spaces<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top">drop</td> +<td>drop control characters - the resulting string will +neither contain control characters, escape sequences nor any other +replacement character like space.<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td><b>SecurePath</b></td> +<td>Option values for securing path templates.</td> +</tr> +<tr> +<td valign="top">drop</td> +<td>Drops slashes inside the field (e.g. "a/b" becomes "ab"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +<tr> +<td valign="top">replace</td> +<td>Replace slashes inside the field by an underscore. (e.g. "a/b" becomes "a_b"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +<tr> +<td><b>Format</b></td> +<td>Option values for the general output format.</td> +</tr> +<tr> +<td>json</td> +<td>encode the value so that it can be used inside a JSON field. This means +that several characters (according to the JSON spec) are being escaped, for +example US-ASCII LF is replaced by "\n". +The json option cannot be used together with either jsonf or csv options. +</td> +</tr> +<tr> +<td>jsonf</td> +<td><i>(available in 6.3.9+)</i> +This signifies that the property should be expressed as a json <b>f</b>ield. +That means not only the property is written, but rather a complete json field in +the format<br> +"fieldname"="value"</b> +where "filedname" is the assigend field name (or the property name if none was assigned) +and value is the end result of property replacer operation. Note that value supports +all property replacer options, like substrings, case converson and the like. +Values are properly json-escaped. However, field names are (currently) not. It is +expected that proper field names are configured. +The jsonf option cannot be used together with either json or csv options. +</td> +</tr> +<tr> +<td valign="top">csv</td> +<td>formats the resulting field (after all modifications) in CSV format +as specified in <a href="http://www.ietf.org/rfc/rfc4180.txt">RFC 4180</a>. +Rsyslog will always use double quotes. Note that in order to have full CSV-formatted +text, you need to define a proper template. An example is this one: +<br>$template csvline,"%syslogtag:::csv%,%msg:::csv%" +<br>Most importantly, you need to provide the commas between the fields +inside the template. +The csv option cannot be used together with either json or jsonf options. +<br><i>This feature was introduced in rsyslog 4.1.6.</i> +</td> +</tr> +<tr> +<td><b>droplastlf</b></td> +<td>The last LF in the message (if any), is dropped. +Especially useful for PIX.</td> +</tr> +<tr> +<td valign="top"><b>spifno1stsp</b></td> +<td>This option looks scary and should probably not be used by a user. For any field +given, it returns either a single space character or no character at all. Field content +is never returned. A space is returned if (and only if) the first character of the +field's content is NOT a space. This option is kind of a hack to solve a problem rooted +in RFC 3164: 3164 specifies no delimiter between the syslog tag sequence and the actual +message text. Almost all implementation in fact delemit the two by a space. As of +RFC 3164, this space is part of the message text itself. This leads to a problem when +building the message (e.g. when writing to disk or forwarding). Should a delimiting +space be included if the message does not start with one? If not, the tag is immediately +followed by another non-space character, which can lead some log parsers to misinterpret +what is the tag and what the message. The problem finally surfaced when the klog module +was restructured and the tag correctly written. It exists with other message sources, +too. The solution was the introduction of this special property replacer option. Now, +the default template can contain a conditional space, which exists only if the +message does not start with one. While this does not solve all issues, it should +work good enough in the far majority of all cases. If you read this text and have +no idea of what it is talking about - relax: this is a good indication you will never +need this option. Simply forget about it ;) +</td> +</tr> +<tr> +<td></td> +<td></td> +</tr> +<tr> +<td><b>New character position</b></td> +<td>In addition to the above mentioned Character Positions in the legacy format, +positions can be determined by specifying the correct options for the properties. +Again, this is mostly for using the list format.</td> +</tr> +<tr> +<td>position.From</td> +<td>Character position in the property to start from.</td> +</tr> +<tr> +<td>position.To</td> +<td>Character position that determines the end for extraction. If the value is "$" +then the end of the string will be used.</td> +</tr> +<tr> +<td>field.Number</td> +<td>The number of the field, which should be used for the search operation with Regex.</td> +</tr> +<tr> +<td>field.Delimiter</td> +<td>The Character that should delimit a field. Example: ",". Everything in a +property until this character is considered a field.</td> +</tr> +<tr> +<td>regex.Expression</td> +<td>Value to be compared to property.</td> +</tr> +<tr> +<td>regex.Type</td> +<td>Values BRE or ERE</td> +</tr> +<tr> +<td>regex.NoMatchMode</td> +<td>DFLT, BLANK, ZERO, FIELD</td> +</tr> +<tr> +<td>regex.Match</td> +<td>Match to use.</td> +</tr> +<tr> +<td>regex.Submatch</td> +<td>Submatch to use. Values 0-9 whereas 0 = All</td> +</tr> +</tbody> +</table> + + + +<h2>Legacy Property Options</h2> +<b><code>property options</code></b> are case-insensitive. Currently, the following options are defined: <p></p> <table> @@ -325,6 +583,29 @@ case-insensitive. Currently, the following options are defined: <td>convert property text to uppercase only</td> </tr> <tr> +<td><b>json</b></td> +<td>encode the value so that it can be used inside a JSON field. This means +that several characters (according to the JSON spec) are being escaped, for +example US-ASCII LF is replaced by "\n". +The json option cannot be used together with either jsonf or csv options. +</td> +</tr> +<tr> +<td><b>jsonf</b></td> +<td><i>(available in 6.3.9+)</i> +This signifies that the property should be expressed as a json <b>f</b>ield. +That means not only the property is written, but rather a complete json field in +the format<br> +"fieldname"="value"</b> +where "filedname" is the assigend field name (or the property name if none was assigned) +and value is the end result of property replacer operation. Note that value supports +all property replacer options, like substrings, case converson and the like. +Values are properly json-escaped. However, field names are (currently) not. It is +expected that proper field names are configured. +The jsonf option cannot be used together with either json or csv options. +</td> +</tr> +<tr> <td valign="top"><b>csv</b></td> <td>formats the resulting field (after all modifications) in CSV format as specified in <a href="http://www.ietf.org/rfc/rfc4180.txt">RFC 4180</a>. @@ -333,6 +614,7 @@ text, you need to define a proper template. An example is this one: <br>$template csvline,"%syslogtag:::csv%,%msg:::csv%" <br>Most importantly, you need to provide the commas between the fields inside the template. +The csv option cannot be used together with either json or jsonf options. <br><i>This feature was introduced in rsyslog 4.1.6.</i> </td> </tr> @@ -367,6 +649,10 @@ option when forwarding to remote hosts - they may treat the date as invalid <td>format as RFC 3339 date</td> </tr> <tr> +<td><b>date-unixtimestamp</b></td> +<td>format as unix timestamp (seconds since epoch)</td> +</tr> +<tr> <td><b>date-subseconds</b></td> <td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> </tr> @@ -428,19 +714,31 @@ Useful for secure pathname generation (with dynafiles). Useful for secure pathname generation (with dynafiles). </td> </tr> +<tr> +<td><b>mandatory-field</b></td> +<td>In templates that are used for building field lists (in particular, ommongodb), include +this field, even if it is empty (or NULL). If not set, the field will be removed from +the output field set if empty. The latter is the default case. +</tr> </tbody> </table> <p>To use multiple options, simply place them one after each other with a comma delmimiting them. For example "escape-cc,sp-if-no-1st-sp". If you use conflicting options together, the last one will override the previous one. For example, using "escape-cc,drop-cc" will use drop-cc and "drop-cc,escape-cc" will use escape-cc mode. +<h2>Fieldname</h2> +<p><i>(available in 6.3.9+)</i> +<p>This field permits to specify a field name for structured-data emitting property replacer +options. It was initially introduced to support the "jsonf" option, for which it provides +the capability to set an alternative field name. If it is not specified, it defaults to +the property name. <h2>Further Links</h2> <ul> <li>Article on "<a href="rsyslog_recording_pri.html">Recording the Priority of Syslog Messages</a>" (describes use of templates to record severity and facility of a message)</li> <li><a href="rsyslog_conf.html">Configuration file -syntax</a>, this is where you actually use the property replacer.</li> +format</a>, this is where you actually use the property replacer.</li> </ul> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] diff --git a/doc/rainerscript.html b/doc/rainerscript.html index 63a79040..fcc2674d 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -58,6 +58,9 @@ recommended. variable, if it exists. Returns an empty string if it does not exist. <li>strlen(str) - returns the length of the provided string <li>tolower(str) - converts the provided string into lowercase +<li>cstr(expr) - converts expr to a string value +<li>cnum(expr) - converts expr to a number (integer) +<li>re_match(expr, re) - returns 1, if expr matches re, 0 otherwise </ul> <p>The following example can be used to build a dynamic filter based on some environment variable: @@ -69,7 +72,7 @@ if $msg contains getenv('TRIGGERVAR') then /path/to/errfile <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/rscript_abnf.html b/doc/rscript_abnf.html index d60edb5c..9172d945 100644 --- a/doc/rscript_abnf.html +++ b/doc/rscript_abnf.html @@ -21,7 +21,58 @@ and many other languages).</p> <p>Below is the formal language definitionin ABNF (RFC 2234) format: <br> </p> -<pre>; <span style="font-weight: bold;">all of this is a working document and may change!</span> -- rgerhards, 2008-02-24<br><br>script := *stmt<br>stmt := (if_stmt / block / vardef / run_s / load_s)<br>vardef := "var" ["scope" = ("global" / "event")] <br>block := "begin" stmt "end"<br>load_s := "load" constraint ("module") modpath params ; load mod only if expr is true<br>run_s := "run" constraint ("input") name<br>constraint:= "if" expr ; constrains some one-time commands<br>modpath := expr<br>params := ["params" *1param *("," param) "endparams"]<br>param := paramname) "=" expr<br>paramname := [*(obqualifier ".") name]<br>modpath:= ; path to module<br>?line? := cfsysline / cfli<br>cfsysline:= BOL "$" *char EOL ; how to handle the first line? (no EOL in front!)<br>BOL := ; Begin of Line - implicitely set on file beginning and after each EOL<br>EOL := 0x0a ;LF<br>if_stmt := "if" expr "then"<br>old_filter:= BOL facility "." severity ; no whitespace allowed between BOL and facility!<br>facility := "*" / "auth" / "authpriv" / "cron" / "daemon" / "kern" / "lpr" / <br> "mail" / "mark" / "news" / "security" / "syslog" / "user" / "uucp" / <br> "local0" .. "local7" / "mark"<br> ; The keyword security should not be used anymore<br> ; mark is just internal<br>severity := TBD ; not really relevant in this context<br><br>; and now the actual expression<br>expr := e_and *("or" e_and)<br>e_and := e_cmp *("and" e_cmp)<br>e_cmp := val 0*1(cmp_op val)<br>val := term *(("+" / "-" / "&") term)<br>term := factor *(("*" / "/" / "%") factor)<br>factor := ["not"] ["-"] terminal<br>terminal := var / constant / function / ( "(" expr ")" )<br>function := name "(" *("," expr) ")"<br>var := "$" varname<br>varname := msgvar / sysvar<br>msgvar := name<br>sysvar := "$" name<br>name := alpha *(alnum)<br>constant := string / number<br>string := simpstr / tplstr ; tplstr will be implemented in next phase<br>simpstr := "'" *char "'" ; use your imagination for char ;)<br>tplstr := '"' template '"' ; not initially implemented<br>number := ["-"] 1*digit ; 0nn = octal, 0xnn = hex, nn = decimal<br>cmp_op := "==" / "!=" / "<>" / "<" / ">" / "<=" / ">=" / "contains" / "contains_i" / "startswith" / "startswith_i"<br>digit := %x30-39<br>alpha := "a" ... "z" # all letters<br>alnum :* alpha / digit / "_" /"-" # "-" necessary to cover currently-existing message properties<br></pre> +<pre>; <span style="font-weight: bold;">all of this is a working document and may change!</span> -- rgerhards, 2008-02-24<br> +<br> +script := *stmt<br> +stmt := (if_stmt / block / vardef / run_s / load_s)<br> +vardef := "var" ["scope" = ("global" / "event")] <br> +block := "begin" stmt "end"<br> +load_s := "load" constraint ("module") modpath params ; load mod only if expr is true<br> +run_s := "run" constraint ("input") name<br> +constraint:= "if" expr ; constrains some one-time commands<br> +modpath := expr<br> +params := ["params" *1param *("," param) "endparams"]<br> +param := paramname) "=" expr<br> +paramname := [*(obqualifier ".") name]<br> +modpath:= ; path to module<br> +?line? := cfsysline / cfli<br> +cfsysline:= BOL "$" *char EOL ; how to handle the first line? (no EOL in front!)<br> +BOL := ; Begin of Line - implicitely set on file beginning and after each EOL<br> +EOL := 0x0a ;LF<br> +if_stmt := "if" expr "then"<br> +old_filter:= BOL facility "." severity ; no whitespace allowed between BOL and facility!<br> +facility := "*" / "auth" / "authpriv" / "cron" / "daemon" / "kern" / "lpr" / <br> +"mail" / "mark" / "news" / "security" / "syslog" / "user" / "uucp" / <br> +"local0" .. "local7" / "mark"<br> +; The keyword security should not be used anymore<br> +; mark is just internal<br> +severity := TBD ; not really relevant in this context<br> +<br> +; and now the actual expression<br> +expr := e_and *("or" e_and)<br> +e_and := e_cmp *("and" e_cmp)<br> +e_cmp := val 0*1(cmp_op val)<br> +val := term *(("+" / "-" / "&") term)<br> +term := factor *(("*" / "/" / "%") factor)<br> +factor := ["not"] ["-"] terminal<br> +terminal := var / constant / function / ( "(" expr ")" )<br> +function := name "(" *("," expr) ")"<br> +var := "$" varname<br> +varname := msgvar / sysvar / ceevar<br> +msgvar := name<br> +ceevar := "!" name<br> +sysvar := "$" name<br> +name := alpha *(alnum)<br> +constant := string / number<br> +string := simpstr / tplstr ; tplstr will be implemented in next phase<br> +simpstr := "'" *char "'" ; use your imagination for char ;)<br> +tplstr := '"' template '"' ; not initially implemented<br> +number := ["-"] 1*digit ; 0nn = octal, 0xnn = hex, nn = decimal<br> +cmp_op := "==" / "!=" / "<>" / "<" / ">" / "<=" / ">=" / "contains" / "contains_i" / "startswith" / "startswith_i"<br> +digit := %x30-39<br> +alpha := "a" ... "z" # all letters<br> +alnum :* alpha / digit / "_" /"-" # "-" necessary to cover currently-existing message properties<br> +</pre> <h2>Samples</h2> <p>Some samples of RainerScript:</p><p>define function IsLinux<br>begin<br> if $environ contains "linux" then return true else return false<br>end</p><p>load if IsLinux() 'imklog.so' params name='klog' endparams /* load klog under linux only */<br>run if IsLinux() input 'klog'<br>load 'ommysql.so'</p><p>if $message contains "error" then<br> action<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> [?action.template='templatename'?] or [?action.sql='insert into diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 703e7a6e..6aa2e460 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -1,10 +1,10 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>rsyslog.conf file</title></head> +<html><head><title>rsyslog.conf configuration file</title></head> <body> <h1>rsyslog.conf configuration file</h1> -<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, +<p><b>Rsyslog is configured via the rsyslog.conf file</b>, typically found in /etc. By default, rsyslogd reads the file -/etc/rsyslog.conf. This may be changed by a command line option.</p> +/etc/rsyslog.conf. This may be changed by command line option "-f".</p> <p><a href="http://wiki.rsyslog.com/index.php/Configuration_Samples"> Configuration file examples can be found in the rsyslog wiki</a>. Also keep the @@ -12,65 +12,26 @@ keep the on your mind. These are ready-to-use real building blocks for rsyslog configuration. </p> -<p>There is also one sample file provided together with the -documentation set. If you do not like to read, be sure to have at least -a quick look at -<a href="rsyslog-example.conf">rsyslog-example.conf</a>. -</p> <p>While rsyslogd contains enhancements over standard syslogd, efforts have been made to keep the configuration file as compatible as possible. While, for obvious reasons, <a href="features.html">enhanced features</a> require a different config file syntax, rsyslogd should be able to work with a standard syslog.conf file. This is especially useful while you are migrating from syslogd to rsyslogd.</p> -<h2><a href="rsyslog_conf_modules.html">Modules</a></h2> -<h2>Lines</h2> -Lines can be continued by specifying a backslash ("\") as the last -character of the line. There is a hard-coded maximum line length of 4K. -If you need lines larger than that, you need to change compile-time -settings inside rsyslog and recompile. -<h2><a href="rsyslog_conf_global.html">Configuration Directives</a></h2> -<h2>Basic Structure</h2> -<p>Rsyslog supports standard sysklogd's configuration file format -and extends it. So in general, you can take a "normal" syslog.conf and -use it together with rsyslogd. It will understand everything. However, -to use most of rsyslogd's unique features, you need to add extended -configuration directives.</p> -<p>Rsyslogd supports the classical, selector-based rule lines. -They are still at the heart of it and all actions are initiated via -rule lines. A rule lines is any line not starting with a $ or the -comment sign (#). Lines starting with $ carry rsyslog-specific -directives.</p> -<p>Every rule line consists of two fields, a selector field and -an action field. These two fields are separated by one or more spaces -or tabs. The selector field specifies a pattern of facilities and -priorities belonging to the specified action.<br> -<br> -Lines starting with a hash mark ("#'') and empty lines are ignored. -</p> -<h2><a href="rsyslog_conf_templates.html">Templates</a></h2> -<h2><a href="rsyslog_conf_output.html">Output Channels</a></h2> -<h2><a href="rsyslog_conf_filter.html">Filter Conditions</a></h2> -<h2><a href="rsyslog_conf_actions.html">Actions</a></h2> -<h2><a href="rsyslog_conf_examples.html">Examples</a></h2> -<p>Here you will find examples for templates and selector lines. I hope -they are self-explanatory. If not, please see -www.monitorware.com/rsyslog/ for advise.</p> -<h2>Configuration File Syntax Differences</h2> -<p>Rsyslogd uses a slightly different syntax for its -configuration file than the original BSD sources. Originally all -messages of a specific priority and above were forwarded to the log -file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd -more flexible and to use it in a more intuitive manner.<br> -<br> -The original BSD syslogd doesn't understand spaces as separators -between the selector and the action field.<br> -<br> -When compared to syslogd from sysklogd package, rsyslogd offers -additional -<a href="features.html">features</a> (like template -and database support). For obvious reasons, the syntax for defining -such features is available in rsyslogd, only.</p> + +<p><b>Follow the links below to learn more about specific topics:</b></p> +<ul> +<li><a href="rsyslog_conf_modules.html">Modules</a></li> +<li><a href="rsyslog_conf_lines.html">Lines</a></li> +<li><a href="rsyslog_conf_global.html">Configuration Directives</a></li> +<li><a href="rsyslog_conf_basic_structure.html">Basic Structure</a></li> +<li><a href="rsyslog_conf_templates.html">Templates</a></li> +<li><a href="rsyslog_conf_output.html">Output Channels</a></li> +<li><a href="rsyslog_conf_filter.html">Filter Conditions</a></li> +<li><a href="rsyslog_conf_actions.html">Actions</a></li> +<li><a href="rsyslog_conf_file_syntax_differences.html">Configuration File Syntax Differences</a></li> +<li><a href="rsyslog_conf_examples.html">Examples</a></li> +</ul> <p>[<a href="rsyslog_conf.html">back to top</a>] [<a href="manual.html">manual index</a>] @@ -82,4 +43,4 @@ Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer Ge version 3 or higher.</font></p> </body> </html> -> + diff --git a/doc/rsyslog_conf_basic_structure.html b/doc/rsyslog_conf_basic_structure.html new file mode 100644 index 00000000..4ce78de0 --- /dev/null +++ b/doc/rsyslog_conf_basic_structure.html @@ -0,0 +1,35 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Basic Structure - rsyslog.conf</title></head> +<body> +<p>This is a part of the rsyslog.conf documentation.</p> +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<h1>Basic Structure</h1> +<p>Rsyslog supports standard sysklogd's configuration file format +and extends it. So in general, you can take a "normal" syslog.conf and +use it together with rsyslogd. It will understand everything. However, +to use most of rsyslogd's unique features, you need to add extended +configuration directives.</p> +<p>Rsyslogd supports the classical, selector-based rule lines. +They are still at the heart of it and all actions are initiated via +rule lines. A rule lines is any line not starting with a $ or the +comment sign (#). Lines starting with $ carry rsyslog-specific +directives.</p> +<p>Every rule line consists of two fields, a selector field and +an action field. These two fields are separated by one or more spaces +or tabs. The selector field specifies a pattern of facilities and +priorities belonging to the specified action.<br> +<br> +Lines starting with a hash mark ("#'') and empty lines are ignored. +</p> + +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> + diff --git a/doc/rsyslog_conf_file_syntax_differences.html b/doc/rsyslog_conf_file_syntax_differences.html new file mode 100644 index 00000000..bfac8926 --- /dev/null +++ b/doc/rsyslog_conf_file_syntax_differences.html @@ -0,0 +1,32 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Configuration File Syntax Differences - rsyslog.conf</title></head> +<body> +<p>This is a part of the rsyslog.conf documentation.</p> +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<h1>Configuration File Syntax Differences</h1> +<p>Rsyslogd uses a slightly different syntax for its +configuration file than the original BSD sources. Originally all +messages of a specific priority and above were forwarded to the log +file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd +more flexible and to use it in a more intuitive manner.<br> +<br> +The original BSD syslogd doesn't understand spaces as separators +between the selector and the action field.<br> +<br> +When compared to syslogd from sysklogd package, rsyslogd offers +additional +<a href="features.html">features</a> (like template +and database support). For obvious reasons, the syntax for defining +such features is available in rsyslogd, only.</p> + +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> + diff --git a/doc/rsyslog_conf_filter.html b/doc/rsyslog_conf_filter.html index 34839616..fbced4a3 100644 --- a/doc/rsyslog_conf_filter.html +++ b/doc/rsyslog_conf_filter.html @@ -117,6 +117,13 @@ currently supported:</p> the property. There must be an exact match, wildcards are not supported.</td> </tr> <tr> +<td>isempty</td> +<td>Checks if the property is empty. The value is discarded. This is +especially useful when working with normalized data, where some fields +may be populated based on normalization result. +Available since 6.6.2. +</tr> +<tr> <td>isequal</td> <td>Compares the "value" string provided and the property contents. These two values must be exactly equal to match. The diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 6c20f4c2..a4d760eb 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -52,10 +52,10 @@ This directive will timeout previous messages seen if they are older than 20 minutes. In the example above, the count would now be always 1 and consequently no rule would ever be triggered. -<li>$ActionFileDefaultTemplate [templateName] - sets a new default template for file actions</li> -<li>$ActionFileEnableSync [on/<span style="font-weight: bold;">off</span>] - enables file +<li><a href="omfile.html">$ActionFileDefaultTemplate</a> [templateName] - sets a new default template for file actions</li> +<li><a href="omfile.html">$ActionFileEnableSync</a> [on/<span style="font-weight: bold;">off</span>] - enables file syncing capability of omfile</li> -<li>$ActionForwardDefaultTemplate [templateName] - sets a new +<li><a href="omfwd.html">$ActionForwardDefaultTemplate</a> [templateName] - sets a new default template for UDP and plain TCP forwarding action</li> <li>$ActionGSSForwardDefaultTemplate [templateName] - sets a new default template for GSS-API forwarding action</li> @@ -93,23 +93,23 @@ default 60000 (1 minute)]</li> <li>$ActionQueueWorkerThreadMinumumMessages <number>, default 100</li> <li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li> <li>$ActionResumeRetryCount <number> [default 0, -1 means eternal]</li> -<li>$ActionSendResendLastMsgOnReconnect <[on/<b>off</b>]> specifies if the last message is to be resend when a connecition breaks and has been reconnected. May increase reliability, but comes at the risk of message duplication. -<li>$ActionSendStreamDriver <driver basename> just like $DefaultNetstreamDriver, but for the specific action</li> -<li>$ActionSendStreamDriverMode <mode>, default 0, mode to use with the stream driver (driver-specific)</li> -<li>$ActionSendStreamDriverAuthMode <mode>, authentication mode to use with the stream driver. Note that this directive requires TLS +<li><a href="omfwd.html">$ActionSendResendLastMsgOnReconnect</a> <[on/<b>off</b>]> specifies if the last message is to be resend when a connecition breaks and has been reconnected. May increase reliability, but comes at the risk of message duplication. +<li><a href="omfwd.html">$ActionSendStreamDriver</a> <driver basename> just like $DefaultNetstreamDriver, but for the specific action</li> +<li><a href="omfwd.html">$ActionSendStreamDriverMode</a> <mode>, default 0, mode to use with the stream driver (driver-specific)</li> +<li><a href="omfwd.html">$ActionSendStreamDriverAuthMode</a> <mode>, authentication mode to use with the stream driver. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific)</li> -<li>$ActionSendStreamDriverPermittedPeer <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS +<li><a href="omfwd.html">$ActionSendStreamDriverPermittedPeer</a> <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific) -<span style="font-weight: bold;"> directive may go away</span>!</li> -<li><b>$ActionSendTCPRebindInterval</b> nbr</a>- [available since 4.5.1] - instructs the TCP send +<li><a href="omfwd.html"><b>$ActionSendTCPRebindInterval</b> nbr</a>- [available since 4.5.1] - instructs the TCP send action to close and re-open the connection to the remote host every nbr of messages sent. Zero, the default, means that no such processing is done. This directive is useful for use with load-balancers. Note that there is some performance overhead associated with it, so it is advisable to not too often "rebind" the connection (what "too often" actually means depends on your configuration, a rule of thumb is that it should be not be much more often than once per second).</li> -<li><b>$ActionSendUDPRebindInterval</b> nbr</a>- [available since 4.3.2] - instructs the UDP send +<li><a href="omfwd.html"><b>$ActionSendUDPRebindInterval</b> nbr</a>- [available since 4.3.2] - instructs the UDP send action to rebind the send socket every nbr of messages sent. Zero, the default, means that no rebind is done. This directive is useful for use with load-balancers.</li> <li><b>$ActionWriteAllMarkMessages</b> [on/<b>off</b>]- [available since 5.1.5] - normally, mark messages @@ -133,22 +133,22 @@ functionality. the provided <i>name</i> (the default default ruleset is named "RSYSLOG_DefaultRuleset"). It is advised to also read our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> -<li><b>$CreateDirs</b> [<b>on</b>/off] - create directories on an as-needed basis</li> -<li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> -<li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> -<li><a href="rsconf1_dirowner.html">$DirOwner</a></li> +<li><a href="omfile.html"><b>$CreateDirs</b></a> [<b>on</b>/off] - create directories on an as-needed basis</li> +<li><a href="omfile.html">$DirCreateMode</a></li> +<li><a href="omfile.html">$DirGroup</a></li> +<li><a href="omfile.html">$DirOwner</a></li> <li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> <li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> -<li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> +<li><a href="omfile.html">$DynaFileCacheSize</a></li> <li><a href="rsconf1_escape8bitcharsonreceive.html">$Escape8BitCharactersOnReceive</a></li> <li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> <li><b>$EscapeControlCharactersOnReceive</b> [<b>on</b>|off] - escape USASCII HT character</li> <li>$SpaceLFOnReceive [on/<b>off</b>] - instructs rsyslogd to replace LF with spaces during message reception (sysklogd compatibility aid)</li> <li>$ErrorMessagesToStderr [<b>on</b>|off] - direct rsyslogd error message to stderr (in addition to other targets)</li> -<li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> -<li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> -<li><a href="rsconf1_filegroup.html">$FileGroup</a></li> -<li><a href="rsconf1_fileowner.html">$FileOwner</a></li> +<li><a href="omfile.html">$FailOnChownFailure</a></li> +<li><a href="omfile.html">$FileCreateMode</a></li> +<li><a href="omfile.html">$FileGroup</a></li> +<li><a href="omfile.html">$FileOwner</a></li> <li><a href="rsconf1_generateconfiggraph.html">$GenerateConfigGraph</a></li> <li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> <li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> @@ -224,7 +224,7 @@ supported in order to be compliant to the upcoming new syslog RFC series. <li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li> <li><a href="rsconf1_moddir.html">$ModDir</a></li> <li><a href="rsconf1_modload.html">$ModLoad</a></li> -<li><b>$OMFileAsyncWriting</b> [on/<b>off</b>], if turned on, the files will be written +<li><a href="omfile.html"><b>$OMFileAsyncWriting</b></a> [on/<b>off</b>], if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable $OMFileFlushInterval, $OMFileAsyncWriting must be set to "on". Otherwise, the flush @@ -232,11 +232,11 @@ interval will be ignored. Also note that when $OMFileFlushOnTXEnd is "on" but $OMFileAsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. -<li><b>$OMFileZipLevel</b> 0..9 [default 0] - if greater 0, turns on gzip compression +<li><a href="omfile.html"><b>$OMFileZipLevel</b></a> 0..9 [default 0] - if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.</li> -<li><b>$OMFileIOBufferSize</b> <size_nbr>, default 4k, size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)</li> -<li><b>$OMFileFlushOnTXEnd</b> <[<b>on</b>/off]>, default on. Omfile has the +<li><a href="omfile.html"><b>$OMFileIOBufferSize</b></a> <size_nbr>, default 4k, size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)</li> +<li><a href="omfile.html"><b>$OMFileFlushOnTXEnd</b></a> <[<b>on</b>/off]>, default on. Omfile has the capability to write output using a buffered writer. Disk writes are only done when the buffer is full. So if an error happens during that write, data is potentially lost. In cases where @@ -246,7 +246,7 @@ error recovery thus can handle write errors without data loss. Note that this op severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -on- is primarily an aid to preserve the traditional syslogd behaviour.</li> -<li><a href="rsconf1_omfileforcechown.html">$omfileForceChown</a> - force ownership change for all files</li> +<li><a href="omfile.html">$omfileForceChown</a> - force ownership change for all files</li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most diff --git a/doc/rsyslog_conf_lines.html b/doc/rsyslog_conf_lines.html new file mode 100644 index 00000000..0e6cc0d3 --- /dev/null +++ b/doc/rsyslog_conf_lines.html @@ -0,0 +1,23 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Lines - rsyslog.conf</title></head> +<body> +<p>This is a part of the rsyslog.conf documentation.</p> +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<h1>Lines</h1> +<p>Lines can be continued by specifying a backslash ("\") as the last +character of the line. There is a hard-coded maximum line length of 4K.<br> +If you need lines larger than that, you need to change compile-time +settings inside rsyslog and recompile. +</p> + +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> + diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 9fa35ccd..cbd60faf 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -51,6 +51,10 @@ to message generators. <p>Output modules process messages. With them, message formats can be transformed and messages be transmitted to various different targets. <ul> +<li><a href="omfile.html">omfile</a> - file output module</li> +<li><a href="omfwd.html">omfwd</a> - syslog forwarding output module</li> +<li><a href="ompipe.html">ompipe</a> - named pipe output module</li> +<li><a href="omusrmsg.html">omusrmsg</a> - user message output module</li> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> <li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li> <li><a href="omrelp.html">omrelp</a> - RELP output module</li> @@ -94,7 +98,7 @@ repated n times" messages emitted by some syslogds. They can be implemented using either the output module or the parser module interface. From the rsyslog core's point of view, they actually are output or parser modules, it is their implementation that makes them special. -<p>Currently, there do not exist any such modules, but could be written with +<p>Currently, there exists only a limited set of such modules, but new ones could be written with the methods the engine provides. They could be used, for example, to: <ul> <li>anonymize message content @@ -103,8 +107,10 @@ the methods the engine provides. They could be used, for example, to: <p>Message modification modules are usually written for one specific task and thus usually are not generic enough to be reused. However, existing module's code is probably an excellent starting base for writing a new module. Currently, the following -modules existin inside the source tree +modules exist inside the source tree: <ul> +<li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. +Note that this actually is a <b>generic</b> module. <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index bd0b3253..b97f6609 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -33,26 +33,31 @@ string generator module, you need to know how to call it. Each such module has a which you need to know (look it up in the module doc or ask the developer). Let's assume that "mystrgen" is the module name. Then you can define a template for that strgen in the following way: + +<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> +<p>Legacy example:</p> <blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> (Of course, you must have first loaded the module via $ModLoad). -<p>The important part is the equal sign: it tells the rsyslog config parser that +<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that no string follows but a strgen module name. <p>There are no additional parameters but the module name supported. This is because there is no way to customize anything inside such a "template" other than by modifying the code of the string generator. <p>So for most use cases, string-generator module based templates are <b>not</b> -the route to take. Usually, us use <b>string based templates</b> instead. +the route to take. Usually, we use <b>string based templates</b> instead. This is what the rest of the documentation now talks about. <p>A template consists of a template directive, a name, the actual template text and optional options. A sample is:</p> +<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> +<p>Legacy example:</p> <blockquote><code>$template MyTemplateName,"\7Text %property% some more text\n",<options></code></blockquote> -<p>The "$template" is the template directive. It tells rsyslog +<p>The "template" (legacy: $template) is the template directive. It tells rsyslog that this line contains a template. "MyTemplateName" is the template name. All -other config lines refer to this name. The text within quotes is the +other config lines refer to this name. The text within "string" is the actual template text. The backslash is an escape character, much as it is in C. It does all these "cool" things. For example, \7 rings the bell (this is an ASCII value), \n is a new line. C programmers and perl @@ -69,24 +74,30 @@ on this is below, on some lines of the property replacer.<br> <br> The <options> part is optional. It carries options influencing the template as whole. See details below. Be sure NOT to -mistake template options with property options - the later ones are +mistake template options with property options - the latter ones are processed by the property replacer and apply to a SINGLE property, only (and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> -<p><b>sql</b> - format the string suitable for a SQL +<p><b>option.sql</b> - format the string suitable for a SQL statement in MySQL format. This will replace single quotes ("'") and the backslash character by their backslash-escaped counterpart ("\'" and "\\") inside each field. Please note that in MySQL configuration, the <code class="literal">NO_BACKSLASH_ESCAPES</code> mode must be turned off for this format to work (this is the default).</p> -<p><b>stdsql</b> - format the string suitable for a +<p><b>option.stdsql</b> - format the string suitable for a SQL statement that is to be sent to a standards-compliant sql server. This will replace single quotes ("'") by two single quotes ("''") inside each field. You must use stdsql together with MySQL if in MySQL configuration the <code class="literal">NO_BACKSLASH_ESCAPES</code> is turned on.</p> +<p><b>option.json</b> - format the string suitable for a +json statement. +This will replace single quotes ("'") by two single quotes ("''") +inside each field.</p> +<p>At no time, multiple template option should be used. This can cause +unpredictable behaviour and is against all logic.</p> <p>Either the <b>sql</b> or <b>stdsql</b> option <b>must</b> be specified when a template is used for writing to a database, otherwise injection might occur. Please note @@ -120,11 +131,41 @@ $template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> <br> Properties can be accessed by the <a href="property_replacer.html">property replacer</a> (see there for details).</p> -<p><b>Please note that templates can also by +<p>Templates can be used in the form of a <b>list</b> as well. This has been +introduced with <b>6.5.0</b> The list consists of two parts which are either +a <b>constant</b> or a <b>property</b>. The constants +are taking the part of "text" that you usually enter in string-based templates. +The properties stay variable, as they are a substitute for different values of a +certain type. This type of template is extremely useful for complicated cases, +as it helps you to easily keep an overview over the template. Though, it has +the disadvantage of needing more effort to create it.</p> +<br>Config example: +<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { + <br>constant(value="Test: ") + <br>property(name="msg" outname="mymessage") + <br>constant(value=" --!!!-- ") + <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + <br>constant(value="\n") + <br>} +</code></blockquote> +<p>First, the general template option will be defined. The values of the template +itself get defined in the curly brackets. As it can be seen, we have constants +and properties in exchange. Whereas constants will be filled with a value and probably +some options, properties do direct to a property and the options that could be needed +additional format definitions.</p> +<p>We suggest to use separate lines for all constants and properties. This +helps to keep a good overview over the different parts of the template. +Though, writing it in a single line will work, it is much harder to debug +if anything goes wrong with the template. </p> + + +<p><b>Please note that templates can also be used to generate selector lines with dynamic file names.</b> For example, if you would like to split syslog messages from different hosts to different files (one per host), you can define the following template:</p> +<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> +<p>Legacy example:</p> <blockquote><code>$template DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> <p>This template can then be used when defining an output @@ -169,7 +210,7 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> -<h3>String-based Template Samples</h3> +<h3>Legacy String-based Template Samples</h3> <p>This section provides some sample of what the default formats would look as a text-based template. Hopefully, their description is self-explanatory. Note that each $Template statement is on a <b>single</b> line, but probably broken diff --git a/doc/rsyslog_ng_comparison.html b/doc/rsyslog_ng_comparison.html index 7d12a4a7..44c895f7 100644 --- a/doc/rsyslog_ng_comparison.html +++ b/doc/rsyslog_ng_comparison.html @@ -4,24 +4,45 @@ <a href="features.html">back</a> <h1>rsyslog vs. syslog-ng</h1> <p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> -(2008-05-06)</i></small></p> -<p><i>Warning</i>: this comparison is a little outdated, take it with a grain -of salt and be sure to check the links at the bottom (both syslog-ng as well as -rsyslog features are missing, but our priority is on creating great software not -continously updating this comparison ;)). -<p>We have often been asked about a comparison sheet between -rsyslog and syslog-ng. Unfortunately, I do not know much about -syslog-ng, I did not even use it once. Also, there seems to be no -comprehensive feature sheet available for syslog-ng (that recently -changed, see below). So I started this -comparison, but it probably is not complete. For sure, I miss some -syslog-ng features. This is not an attempt to let rsyslog shine more -than it should. I just used the <a href="features.html">rsyslog -feature sheet</a> as a starting point, simply because it was -available. If you would like to add anything to the chart, or correct -it, please simply <a href="mailto:rgerhards@adiscon.com">drop -me a line</a>. I would love to see a real honest and up-to-date -comparison sheet, so please don't be shy ;)</p> +(2008-05-06), slightly updated 2012-01-09</i></small></p> +<p><b>This comparison page is rooted nearly 5 years in the past and has become severely +outdated since then.</b> It was unmaintained for several years and contained false +information on both syslog-ng and rsyslog as technology had advanced so much. +<p>This page was initially written because so many people asked about a comparison when +rsyslog was in its infancy. So I tried to create one, but it was hard to maintain as both +projects grew and added feature after feature. I have to admit we did not try hard to keep +it current -- there were many other priorities. I even had forgetten about this page, when I +saw that Peter Czanik blogged about its +<a href="http://blogs.balabit.com/2012/01/05/rsyslog-vs-syslog-ng/">incorrectness</a> (it must be noted +that Peter is wrong on RELP -- it is well alive). I now remember +that he asked me some time ago about this page, what I somehow lost... I guess he must have been +rather grumpy about that :-( +<p>Visiting this page after so many years is interesting, because it shows how much has changed since then. +Obviously, one of my main goals in regard to syslog-ng is reached: in 2007, I blogged that +<a href="http://blog.gerhards.net/2007/08/why-does-world-need-another-syslogd.html">the +world needs another syslogd</a> in order to have healthy competition and a greate feature +set in the free editions. In my opinion, the timeline clearly tells that rsyslog's competition +has driven more syslog-ng features from the commercial to the free edition. Also, I found +it interesting to see that syslog-ng has adapted rsyslog's licensing scheme, modular design and +multi-threadedness. On the other hand, the Balabit folks have obviously done a quicker and +better move on log normalization with what they call patterndb (it is very roughly equivalent +to what rsyslog has just recently introduced with the help of liblognorm). + +<p>To that account, I think the projects are closer together than 5 years ago. I should now +go ahead and create a new feature comparison. Given previous experience, I think this does not +work out. In the future, we will probably focus on some top features, as Balabit does. However, +that requires some time and I have to admit I do not like to drop this page that has a lot of +inbound links. So I think I do the useful thing by providing these notes and removing the +syslog-ng information. So it can't be wrong on syslog-ng any more. Note that it still contains +some incorrect information about rsyslog (it's the state it had 5 years ago!). The core idea is +to start with updating the <a href="features.html">rsyslog feature sheet</a> and from there +on work to a complete comparision. Of course, feel free to read on if you like to get some sense +of history (and inspiration on what you can still do -- but more ;)). +<br><br> +Thanks,<br> +Rainer Gerhards +<p> + <table border="1"> <tbody> <tr> @@ -37,50 +58,50 @@ comparison sheet, so please don't be shy ;)</p> <tr> <td valign="top">UNIX domain socket</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">UDP</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">TCP</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top"><a href="http://www.librelp.com">RELP</a></td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">RFC 3195/BEEP</td> <td valign="top">yes (via <a href="im3195.html">im3195</a>)</td> -<td valign="top">no</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">kernel log</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">file</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">mark message generator as an optional input</td> <td valign="top">yes</td> -<td valign="top">no (?)</td> +<td valign="top"></td> <td></td> </tr> <tr> @@ -89,8 +110,7 @@ optional input</td> <a href="http://www.eventreporter.com">EventReporter</a> or <a href="http://www.mwagent.com">MonitorWare Agent</a> (both commercial software, both fund rsyslog development)</td> -<td valign="top">via separate Windows agent, paid -edition only</td> +<td valign="top"></td> </tr> <tr> <td colspan="3" valign="top"><b><br> @@ -100,83 +120,82 @@ Network (Protocol) Support</b><br> <tr> <td valign="top">support for (plain) tcp based syslog</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for GSS-API</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to limit the allowed network senders (syslog ACLs)</td> <td valign="top">yes</td> -<td valign="top">yes (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for syslog-transport-tls based framing on syslog/tcp connections</td> <td valign="top">yes</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">udp syslog</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">syslog over RELP<br> truly reliable message delivery (<a href="http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html">Why is plain tcp syslog not reliable?</a>)</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">on the wire (zlib) message compression</td> <td valign="top">yes</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for receiving messages via reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a> delivery</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for <a href="rsyslog_tls.html">TLS/SSL-protected syslog</a> </td> <td valign="top"><a href="rsyslog_tls.html">natively</a> (since 3.19.0)<br><a href="rsyslog_stunnel.html">via stunnel</a></td> -<td valign="top">via stunnel<br> -paid edition natively</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for IETF's new syslog-protocol draft</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for IETF's new syslog-transport-tls draft</td> <td valign="top">yes<br>(since 3.19.0 - world's first implementation)</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for IPv6</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">native ability to send SNMP traps</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to preserve the original hostname in NAT environments and relay chains</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td colspan="3" valign="top"><br> @@ -187,81 +206,81 @@ hostname in NAT environments and relay chains</td> <td valign="top">Filtering for syslog facility and priority</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">Filtering for hostname</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">Filtering for application</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">Filtering for message contents</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">Filtering for sending IP address</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">ability to filter on any other message field not mentioned above (including substrings and the like)</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td>support for complex filters, using full boolean algebra with and/or/not operators and parenthesis</td> <td>yes</td> -<td>yes</td> +<td></td> </tr> <tr> <td>Support for reusable filters: specify a filter once and use it in multiple selector lines</td> <td>no</td> -<td>yes</td> +<td></td> </tr> <tr> <td>support for arbritrary complex arithmetic and string expressions inside filters</td> <td>yes</td> -<td>no</td> +<td></td> </tr> <tr> <td valign="top">ability to use regular expressions in filters</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for discarding messages based on filters</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">ability to filter out messages based on sequence of appearing</td> <td valign="top">yes (starting with 3.21.3)</td> -<td valign="top">no</td> +<td valign="top"></td> <td></td> </tr> <tr> <td valign="top">powerful BSD-style hostname and program name blocks for easy multi-host support</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td></td> @@ -277,47 +296,47 @@ program name blocks for easy multi-host support</td> <td valign="top">MySQL</td> <td valign="top"><a href="rsyslog_mysql.html">yes</a> (native ommysql, <a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">yes (via libdibi)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">PostgreSQL</td> <td valign="top">yes (native ompgsql, <a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">yes (via libdibi)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Oracle</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">yes (via libdibi)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">SQLite</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">yes (via libdibi)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Microsoft SQL (Open TDS)</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Sybase (Open TDS)</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Firebird/Interbase</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Ingres</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">mSQL</td> <td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td colspan="3" valign="top"><br> @@ -328,26 +347,26 @@ program name blocks for easy multi-host support</td> <td valign="top">support for on-demand on-disk spooling of messages</td> <td valign="top">yes</td> -<td valign="top">paid edition only</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to limit disk space used by spool files</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">each action can use its own, independant set of spool files</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">different sets of spool files can be placed on different disk</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to process spooled @@ -356,18 +375,18 @@ during off-peak hours, during peak hours they are enqueued only)</td> <td valign="top"><a href="http://wiki.rsyslog.com/index.php/OffPeakHours">yes</a><br> (can independently be configured for the main queue and each action queue)</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to configure backup syslog/database servers </td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td>Professional Support</td> <td><a href="professional_support.html">yes</a></td> -<td>yes</td> +<td></td> </tr> <tr> <td colspan="3" valign="top"><br> @@ -378,20 +397,20 @@ syslog/database servers </td> <td valign="top">config file format</td> <td valign="top">compatible to legacy syslogd but ugly</td> -<td valign="top">clean but not backwards compatible</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to include config file from within other config files</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td height="25" valign="top">ability to include all config files existing in a specific directory</td> <td height="25" valign="top">yes</td> -<td height="25" valign="top">no</td> +<td height="25" valign="top"></td> </tr> <tr> <td colspan="3" valign="top"><br> @@ -403,13 +422,13 @@ existing in a specific directory</td> loadable modules</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td valign="top">Support for third-party input plugins</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> </tr> @@ -417,7 +436,7 @@ plugins</td> <td valign="top">Support for third-party output plugins</td> <td valign="top">yes</td> -<td valign="top">no</td> +<td valign="top"></td> </tr> <tr> <td colspan="3" valign="top"><br> @@ -430,79 +449,78 @@ plugins</td> <td valign="top">ability to generate file names and directories (log targets) dynamically</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">control of log output format, including ability to present channel and priority as visible log data</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr><td valign="top">native ability to send mail messages</td> <td valign="top">yes (<a href="ommail.html">ommail</a>, introduced in 3.17.0)</td> -<td valign="top">no (only via piped external process)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">good timestamp format control; at a minimum, ISO 8601/RFC 3339 second-resolution UTC zone</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to reformat message contents and work with substrings</td> <td valign="top">yes</td> -<td valign="top">I think yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for log files larger than 2gb</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for log file size limitation and automatic rollover command execution</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">support for running multiple syslogd instances on a single machine</td> <td valign="top">yes</td> -<td valign="top">? (but I think yes)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to execute shell scripts on received messages</td> -<td valign="top">yes</td> +<td valign="top"></td> <td valign="top">yes</td> </tr> <tr> <td valign="top">ability to pipe messages to a continously running program</td> -<td valign="top">no</td> -<td valign="top">yes</td> +<td valign="top"></td> +<td valign="top"></td> </tr> <tr> <td valign="top">massively multi-threaded for tomorrow's multi-core machines</td> <td valign="top">yes</td> -<td valign="top">no (only multithreaded with -database destinations)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to control repeated line reduction ("last message repeated n times") on a per selector-line basis</td> <td valign="top">yes</td> -<td valign="top">yes (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">supports multiple actions per selector/filter condition</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> <td></td> </tr> <tr> @@ -510,24 +528,23 @@ selector/filter condition</td> <td valign="top"><a href="http://www.phplogcon.org">phpLogCon</a><br> [also works with <a href="http://freshmeat.net/projects/php-syslog-ng/"> php-syslog-ng</a>]</td> -<td valign="top"><a href="http://freshmeat.net/projects/php-syslog-ng/"> -php-syslog-ng</a></td> +<td valign="top"></td> </tr> <tr> <td valign="top">using text files as input source</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">rate-limiting output actions</td> <td valign="top">yes</td> -<td valign="top">yes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">discard low-priority messages under system stress</td> <td valign="top">yes</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td height="43" valign="top">flow control @@ -535,40 +552,39 @@ system stress</td> <td height="43" valign="top">yes (advanced, with multiple ways to slow down inputs depending on individual input capabilities, based on watermarks)</td> -<td height="43" valign="top">yes (limited? -"stops accepting messages")</td> +<td height="43" valign="top"></td> </tr> <tr> <td valign="top">rewriting messages</td> <td valign="top">yes</td> -<td valign="top">yes (at least I think so...)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">output data into various formats</td> <td valign="top">yes</td> -<td valign="top">yes (looks somewhat limited to me)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">ability to control "message repeated n times" generation</td> <td valign="top">yes</td> -<td valign="top">no (?)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">license</td> <td valign="top">GPLv3 (GPLv2 for v2 branch)</td> -<td valign="top">GPL (paid edition is closed source)</td> +<td valign="top"></td> </tr> <tr> <td valign="top">supported platforms</td> <td valign="top">Linux, BSD, anecdotical seen on Solaris; compilation and basic testing done on HP UX</td> -<td valign="top">many popular *nixes</td> +<td valign="top"></td> </tr> <tr> <td valign="top">DNS cache</td> -<td valign="top">no</td> -<td valign="top">yes</td> +<td valign="top"></td> +<td valign="top"></td> </tr> </tbody> </table> @@ -585,11 +601,6 @@ that vast experience and sometimes even on the code.</p> argument why it is good to have another strong syslogd besides syslog-ng</b>. You may want to read it at my blog at "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">Why does the world need another syslogd?</a>".</p> -<p>Balabit, the vendor of syslog-ng, has just recently done a -feature sheet. I have not yet been able to fully work through it. In -the mean time, you may want to read it in parallel. It is available at -<a href="http://www.balabit.com/network-security/syslog-ng/features/detailed/">Balabit's -site</a>.</p> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/rsyslog_recording_pri.html b/doc/rsyslog_recording_pri.html index a092980c..abcadf2a 100644 --- a/doc/rsyslog_recording_pri.html +++ b/doc/rsyslog_recording_pri.html @@ -45,7 +45,7 @@ percent signs is literal text, which is simply written as specified.</p> <p>Thankfully, rsyslog provides message properties for the priority. These are called "PRI", "syslogfacility" and "syslogpriority" (case is important!). They are numerical values. Starting with rsyslog 1.13.4, there is also a property "pri-text", which -contains the priority in friendly text format (e.g. "local0.err<133>"). For the rest +contains the priority in friendly text format (e.g. "syslog.info"). For the rest of this article, I assume that you run version 1.13.4 or higher.</p> <p>Recording the priority is now a simple matter of adding the respective field to the template. It now looks like this:</p> @@ -83,29 +83,29 @@ A little bit of configuration is required.</p> <p>Below is some sample data created with the template specified above. Note the priority recording at the start of each line.</p> <p> -<code>kern.info<6>: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 00:04.0<br> -kern.info<6>: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 01:00.0<br> -kern.warn<4>: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11<br> -kern.warn<4>: Jun 15 18:10:38 host kernel: Socket status: 30000006<br> -kern.warn<4>: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11<br> -kern.warn<4>: Jun 15 18:10:38 host kernel: Socket status: 30000010<br> -kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0c00-0x0cff: clean.<br> -kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0100-0x04ff: excluding 0x100-0x107 0x378-0x37f 0x4d0-0x4d7<br> -kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0a00-0x0aff: clean.<br> -local7.notice<189>: Jun 15 18:17:24 host dd: 1+0 records out<br> -local7.notice<189>: Jun 15 18:17:24 host random: Saving random seed: succeeded<br> -local7.notice<189>: Jun 15 18:17:25 host portmap: portmap shutdown succeeded<br> -local7.notice<189>: Jun 15 18:17:25 host network: Shutting down interface eth1: succeeded<br> -local7.notice<189>: Jun 15 18:17:25 host network: Shutting down loopback interface: succeeded<br> -local7.notice<189>: Jun 15 18:17:25 host pcmcia: Shutting down PCMCIA services: cardmgr<br> -user.notice<13>: Jun 15 18:17:25 host /etc/hotplug/net.agent: NET unregister event not supported<br> -local7.notice<189>: Jun 15 18:17:27 host pcmcia: modules.<br> -local7.notice<189>: Jun 15 18:17:29 host rc: Stopping pcmcia: succeeded<br> -local7.notice<189>: Jun 15 18:17:30 host rc: Starting killall: succeeded<br> -syslog.info<46>: Jun 15 18:17:33 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2464"] exiting on signal 15.<br> -syslog.info<46>: Jun 18 10:55:47 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"][x-configInfo udpReception="Yes" udpPort="514" tcpReception="Yes" tcpPort="1470"] restart<br> -user.notice<13>: Jun 18 10:55:50 host rger: test<br> -syslog.info<46>: Jun 18 10:55:52 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"] exiting on signal 2.</code></p> +<code>kern.info: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 00:04.0<br> +kern.info: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 01:00.0<br> +kern.warn: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11<br> +kern.warn: Jun 15 18:10:38 host kernel: Socket status: 30000006<br> +kern.warn: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11<br> +kern.warn: Jun 15 18:10:38 host kernel: Socket status: 30000010<br> +kern.info: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0c00-0x0cff: clean.<br> +kern.info: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0100-0x04ff: excluding 0x100-0x107 0x378-0x37f 0x4d0-0x4d7<br> +kern.info: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0a00-0x0aff: clean.<br> +local7.notice: Jun 15 18:17:24 host dd: 1+0 records out<br> +local7.notice: Jun 15 18:17:24 host random: Saving random seed: succeeded<br> +local7.notice: Jun 15 18:17:25 host portmap: portmap shutdown succeeded<br> +local7.notice: Jun 15 18:17:25 host network: Shutting down interface eth1: succeeded<br> +local7.notice: Jun 15 18:17:25 host network: Shutting down loopback interface: succeeded<br> +local7.notice: Jun 15 18:17:25 host pcmcia: Shutting down PCMCIA services: cardmgr<br> +user.notice: Jun 15 18:17:25 host /etc/hotplug/net.agent: NET unregister event not supported<br> +local7.notice: Jun 15 18:17:27 host pcmcia: modules.<br> +local7.notice: Jun 15 18:17:29 host rc: Stopping pcmcia: succeeded<br> +local7.notice: Jun 15 18:17:30 host rc: Starting killall: succeeded<br> +syslog.info: Jun 15 18:17:33 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2464"] exiting on signal 15.<br> +syslog.info: Jun 18 10:55:47 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"][x-configInfo udpReception="Yes" udpPort="514" tcpReception="Yes" tcpPort="1470"] restart<br> +user.notice: Jun 18 10:55:50 host rger: test<br> +syslog.info: Jun 18 10:55:52 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"] exiting on signal 2.</code></p> <h2>Feedback Requested</h2> <P>I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please diff --git a/doc/v6compatibility.html b/doc/v6compatibility.html new file mode 100644 index 00000000..7ce8c001 --- /dev/null +++ b/doc/v6compatibility.html @@ -0,0 +1,198 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v6</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v6</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2011-10-27)</i></small></p> +<p> +This document describes things to keep in mind when moving from v5 to v6. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used a previous version and want to use the +current one without hassle. +<p>Version 6 offers a better config language and some other improvements. +As the config system has many ties into the rsyslog engine AND all plugins, +the changes are somewhat intrusive. Note, however, that core processing has +not been changed much in v6 and will not. So once the configuration is loaded, +the stability of v6 is quite comparable to v5. +</p> +<h2>Property "pri-text"</h2> +<p>Traditionally, this property did not only return the textual form +of the pri ("local0.err"), but also appended the numerical value to it +("local0.err<133>"). This sounds odd and was left unnoticed for some years. +In October 2011, this odd behaviour was brought up on the rsyslog mailing list +by Gregory K. Ruiz-Ade. Code review showed that the behaviour was intentional, +but no trace of what the intention was when it was introduced could be found. +The documentation was also unclear, it said no numerical value was present, +but the samples had it. We agreed that the additional numerical value is +of disadvantage. We also guessed that this property is very rarely being used, +otherwise the problem should have been raised much earlier. However, we +didn't want to change behaviour in older builds. So v6 was set to clean up +the situation. In v6, text-pri will always return the textual part only +("local0.err") and the numerical value will not be contained any longer inside +the string. If you actually need that value, it can fairly easily be added +via the template system. +<p><b>If you have used this property previously and relied on the numerical +part, you need to update your rsyslog configuration files.</b> +<h2>Plugin ABI</h2> +<p>The plugin interface has considerably been changed to support the new +config language. All plugins need to be upgraded. This usually does not require +much coding. However, if the new config language shall be supported, more +changes must be made to plugin code. All project-supported plugins have been +upgraded, so this compatibility issue is only of interest for you if you have +custom plugins or use some user-contributed plugins from the rsyslog project +that are not maintained by the project itself (omoracle is an example). Please +expect some further plugin instablity during the initial v6 releases. +<h2>RainerScript based rsyslog.conf</h2> +<p>A better config format was the main release target for rsyslog v6. It comes in the +flavor of so-called RainerScript +(<a href="http://blog.gerhards.net/2008/02/introducing-rainerscript-and-some.html">why the +name RainerScript?</a>). RainerScript supports legacy syslog.conf format, much as you know it +from other syslogd's (like sysklogd or the BSD syslogd's) as well as previous versions +of rsyslog. Initial work on RainerScript began in v4, and the if-construct was already +supported in v4 and v5. Version 6 has now taken this further. After long discussions we +decided to use the legacy format as a basis, and lightly extend it by native RainerScript +constructs. The main goal was to make sure that previous knowledge and config systems +could still be used while offering a much more intuitive and powerful way of configuring +rsyslog. +<p>RainerScript has been implemented from scratch and with new tools (flex/bison, for those in the +know). Starting with 6.3.3, this new config file processor replaces the legacy one. Note that +the new processor handles all formats, extended RainerScript as well as legacy syslog.conf format. +There are some legacy construct that were especially hard to translate. You'll read about them in +other parts of this document (especially outchannels, which require a format change). + +<p>In v6, all legacy formats are supported. In the long term, we may remove some of the ugly +rsyslog-specific constructs. Good candidates are all configuration commands starting with +a dollar sign, like "$ActionFileDefaultTemplate"). However, this will not be the case before +rsyslog v7 or (much more likely) v8/9. Right now, you also need to use these commands, because +not all have already been converted to the new RainerScript format. + +<p>In 6.3.3, the new parser is used, but almost none of the extended RainerScript capabilities +are available. They will incrementally be introduced with the following releases. Note that for +some features (most importantly if-then-else nested blocks), the v6 core engine is not +capable enough. It is our aim to provide a much better config language to as many rsyslog +users as quickly as possible. As such, we refrain from doing big engine changes in v6. This +in turn means we cannot introduce some features into RainerScript that we really want to see. +These features will come up with rsyslog v7, which will have even better flow control +capabilities inside the core engine. Note that v7 will fully support v6 RainerScript. +Let us also say that the v6 version is not a low-end quick hack: it offers full-fledged +syslog message processing control, capable of doing the best you can find inside the +industry. We just say that v7 will come up with even more advanced capabilites. +<p>Please note that we tried hard to make the RainerScript parser compatible with +all legacy config files. However, we may have failed in one case or another. So if you +experience problems during config processing, chances are there may be a problem +on the rsyslog side. In that case, please let us know. + +<p>Please see the +<a href="http://blog.gerhards.net/2011/07/rsyslog-633-config-format-improvements.html">blog +post about rsyslog 6.3.3 config format</a> for details of what is currently supported. + +<h2>compatibility mode</h2> +<p>Compatibility mode (specified via -c option) has been removed. This was a migration aid from +sysklogd and very early versions of rsyslog. As all major distros now have rsyslog as their +default, and thus ship rsyslog-compliant config files, there is no longer a need for +compatibility mode. Removing it provides easier to maintain code. Also, practice has shown +that many users were confused by compatibility mode (and even some package maintainers got +it wrong). So this not only cleans up the code but rather removes a frequent source of +error. +<p>It must be noted, though, that this means rsyslog is no longer a 100% drop-in replacement +for sysklogd. If you convert an extremely old system, you need to checks its config and +probably need to apply some very mild changes to the config file. +<h2>abort on config errors</h2> +<p>Previous versions accepted some malformedness inside the config file without aborting. This +could lead to some uncertainty about which configuration was actually running. In v6 there +are some situations where config file errors can not be ignored. In these cases rsyslog +emits error messages to stderr, and then exists with a non-zero exit code. It is important +to check for those cases as this means log data is potentially lost. +Please note that +the root problem is the same for earlier versions as well. With them, it was just harder +to spot why things went wrong (and if at all). +<h2>Default Batch Sizes</h2> +<p>Due to their positive effect on performance and comparatively low overhead, +default batch sizes have been increased. Starting with 6.3.4, the action queues +have a default batch size of 128 messages. +<h2>Default action queue enqueue timeout</h2> +<p>This timeout previously was 2seconds, and has been reduced to 50ms (starting with 6.5.0). This change +was made as a long timeout will caused delays in the associated main queue, something +that was quite unexpected to users. Now, this can still happen, but the effect is much +less harsh (but still considerable on a busy system). Also, 50ms should be fairly enough +for most output sources, except when they are really broken (like network disconnect). If +they are really broken, even a 2second timeout does not help, so we hopefully get the best +of both worlds with the new timeout. A specific timeout can of course still be configured, +it is just the timeout that changed. +<h2>outchannels</h2> +<p>Outchannels are a to-be-removed feature of rsyslog, at least as far as the config +syntax is concerned. Nevertheless, v6 still supports it, but a new syntax is required +for the action. Let's assume your outchannel is named "channel". The previous syntax was +<blockquote><code> +*.* $channel +</code> </blockquote> +This was deprecated in v5 and no longer works in v6. Instead, you need to specify +<blockquote><code> +*.* :omfile:$channel +</code></blockquote> +Note that this syntax is available starting with rsyslog v4. It is important to keep on your +mind that future versions of rsyslog will require different syntax and/or drop outchannel support +completely. So if at all possible, avoid using this feature. If you must use it, be prepared for +future changes and watch announcements very carefully. +<h2>ompipe default template</h2> +<p>Starting with 6.5.0, ompipe does no longer use the omfile default template. +Instead, the default template must be set via the module load statement. +An example is +<blockquote><code> +module(load="builtin:ompipe" template="myDefaultTemplate") +</code> </blockquote> +<p>For obvious reasons, the default template must be defined somewhere in +the config file, otherwise errors will happen during the config load +phase. +<h2>omusrmsg</h2> +<p>The omusrmsg module is used to send messages to users. In legacy-legacy +config format (that is the very old sysklogd style), it was suffucient to use +just the user name to call this action, like in this example: +<blockquote><code> +*.* rgerhards +</code> </blockquote> +This format is very ambigious and causes headache (see +<a href="http://blog.gerhards.net/2011/07/why-omusrmsg-is-evil-and-how-it-is.html">blog post +on omusrmsg</a> for details). Thus the format has been superseded by this syntax +(which is legacy format ;-)): +<blockquote><code> +*.* :omusrmsg:rgerhards +</code> </blockquote> +That syntax is supported since later subversions of version 4. +<p>Rsyslog v6 still supports the legacy-legacy format, but in a very strict +sense. For example, if multiple users or templates are given, no spaces +must be included in the action line. For example, this works up to v5, but no +longer in v6: +<blockquote><code> +*.* rgerhards, bgerhards +</code> </blockquote> +To fix it in a way that is compatible with pre-v4, use (note the removed space!): +<blockquote><code> +*.* rgerhards,bgerhards +</code> </blockquote> +Of course, it probably is better to understand in native v6 format: +<blockquote><code> +*.* action(type="omusrmsg" users="rgerhards, bgerhards") +</code> </blockquote> +As you see, here you may include spaces between user names. +<p>In the long term, legacy-legacy format will most probably totally disappear, +so it is a wise decision to change config files at least to the legacy +format (with ":omusrmsg:" in front of the name). + +<h2>Escape Sequences in Script-Based Filters</h2> +<p>In v5, escape sequences were very simplistic. Inside a string, "\x" meant +"x" with x being any character. This has been changed so that the usual set of +escapes is supported, must importantly "\n", "\t", "\xhh" (with hh being hex digits) +and "\ooo" with (o being octal digits). So if one of these sequences was used +previously, results are obviously different. However, that should not create any +real problems, because it is hard to envision why someone should have done that +(why write "\n" when you can also write "n"?). +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2011 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body></html> |