diff options
Diffstat (limited to 'doc/property_replacer.html')
| -rw-r--r-- | doc/property_replacer.html | 317 |
1 files changed, 312 insertions, 5 deletions
diff --git a/doc/property_replacer.html b/doc/property_replacer.html index f0153f2a..dc09d33c 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -13,7 +13,7 @@ the value, e.g. by converting all characters to lower case.</p> <p>Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by the property replacer. The full syntax is as follows:</p> -<blockquote><b><code>%propname:fromChar:toChar:options%</code></b></blockquote> +<blockquote><b><code>%propname:fromChar:toChar:options:fieldname%</code></b></blockquote> <h2>Available Properties</h2> <p><b><code>propname</code></b> is the name of the property to access. It is case-insensitive (prior to 3.17.0, they were case-senstive). @@ -65,8 +65,7 @@ BSD syslogd. For example, when TAG is "named[12345]", programname is </tr> <tr> <td><b>pri-text</b></td> -<td>the PRI part of the message in a textual form with the numerical PRI appended in -brackes (e.g. "local0.err<133>")</td> +<td>the PRI part of the message in textual form (e.g. "syslog.info")</td> </tr> <tr> <td><b>iut</b></td> @@ -139,6 +138,25 @@ draft-ietf-syslog-protocol</td> <td>The contents of the MSGID field from IETF draft draft-ietf-syslog-protocol</td> </tr> +<tr> +<td><b>parsesuccess</b></td> +<td>This returns the status of the <b>last</b> called higher level parser, +like mmjsonparse. A higher level parser parses the actual message for additional +structured data and maintains an extra property table while doing so (this is +often referred to as "cee data" because the idea was originally rooted in the +cee effort, only (but has been extended since then). Note that higher level +parsers must explicitely support (and set) this property. So, depending on the +parser, it may not be set correctly. +<br>If the parser properly supports it, the value "OK" means that parsing was +successfull, while "FAIL" means the parser could not successfully obtain any data. +Failure state is not necessarily an error. For example, it may simple indicate +that the cee-enhanced syslog parser (mmjsonparse) did not detect cee-enhanced format, +what can be totally valid. Using this property, further processing of the message +can be directed based on this parsing outcome. If no parser has been called at the +time this property is accessed, it will contain "FAIL". +<br><b>This property is available since version 6.3.8.</b> +</td> +</tr> <td><b>inputname</b></td> <td>The name of the input module that generated the message (e.g. "imuxsock", "imudp"). Note that not all modules @@ -161,6 +179,10 @@ than messages generated somewhere. in templates for RFC5424 support, when the character set is know to be Unicode.</td> </tr> +<td><b>$uptime</b></td> +<td>system-uptime in seconds (as reported by operating system). +</td> +</tr> <tr> <td><b>$now</b></td> <td>The current date stamp in the format YYYY-MM-DD</td> @@ -201,12 +223,20 @@ range from 0 to 3 (for the four quater hours that are in each hour)</td> <td>The name of the current host as it knows itself (probably useful for filtering in a generic way)</td> </tr> +<tr> +<td><b>$!<name></b></td> +<td>This is the "bridge" to syslog message normalization (via +<a href="mmnormalize.html">mmnormalize</a>): name is a name defined +inside the normalization rule. It has the value selected by the rule +or none if no rule with this field did match. +</td> +</tr> </tbody> </table> <p>Properties starting with a $-sign are so-called system properties. These do NOT stem from the message but are rather internally-generated.</p> -<h2>Character Positions</h2> +<h2>Legacy Character Positions</h2> <p><b><code>FromChar</code></b> and <b><code>toChar</code></b> are used to build substrings. They specify the offset within the string that should be copied. Offset counting starts at 1, so if you need to @@ -303,6 +333,243 @@ It is modeled after perl compatible regular expressions. <h2>Property Options</h2> <b><code>property options</code></b> are +case-insensitive. They are available as of version 6.5.0. +Currently, the following options are defined: +<p></p> +<table> +<tbody> +<tr> +<td><b>Name</b></td> +<td>New format. Name of the template / property / constant.</td> +</tr> +<tr> +<td><b>Outname</b></td> +<td>This field permits to specify a field name for structured-data emitting property replacer options. +It is most useful to set, for example, the name for JSON-based fields (like used in ommngodb). For +text-based modules, it is simply ignored. +If not specified, the original property name is used, with the exception of properties starting with +"$!", where that prefix is removed. Note that unnamaned constants are NOT forwarded to output modules +that expect structure (like ommnogodb). To pass constants, an outname must be set. +</tr> +<tr> +<td><b>CaseConversion</b></td> +<td>New format. Additional values below.</td> +</tr> +<tr> +<td>upper</td> +<td>convert property to lowercase only</td> +</tr> +<tr> +<td>lower</td> +<td>convert property text to uppercase only</td> +</tr> +<tr> +<td><b>DateFormat</b></td> +<td>New format, additional parameter is needed. See below.</td> +</tr> +<tr> +<td>mysql</td> +<td>format as mysql date</td> +</tr> +<tr> +<td>pgsql</td> +<td>format as pgsql date</td> +</tr> +<tr> +<td>rfc3164</td> +<td>format as RFC 3164 date</td> +</tr> +<tr> +<tr> +<td valign="top">rfc3164-buggyday</td> +<td>similar to date-rfc3164, but emulates a common coding error: RFC 3164 demands +that a space is written for single-digit days. With this option, a zero is +written instead. This format seems to be used by syslog-ng and the +date-rfc3164-buggyday option can be used in migration scenarios where otherwise +lots of scripts would need to be adjusted. It is recommended <i>not</i> to use this +option when forwarding to remote hosts - they may treat the date as invalid +(especially when parsing strictly according to RFC 3164).</td> +<br><i>This feature was introduced in rsyslog 4.6.2 and v4 versions above and +5.5.3 and all versions above.</i> +</tr> +<tr> +<td>rfc3339</td> +<td>format as RFC 3339 date</td> +</tr> +<tr> +<td>unixtimestamp</td> +<td>format as unix timestamp (seconds since epoch)</td> +</tr> +<tr> +<td>subseconds</td> +<td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> +</tr> +<tr> +<td><b>ControlCharacters</b></td> +<td>Option values for how to process control characters</td> +</tr> +<tr> +<td valign="top">escape</td> +<td>replace control characters (ASCII value 127 and values +less then 32) with an escape sequence. The sequnce is +"#<charval>" where charval is the 3-digit decimal value +of the control character. For example, a tabulator would be replaced by +"#009".<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top">space</td> +<td>replace control characters by spaces<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top">drop</td> +<td>drop control characters - the resulting string will +neither contain control characters, escape sequences nor any other +replacement character like space.<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td><b>SecurePath</b></td> +<td>Option values for securing path templates.</td> +</tr> +<tr> +<td valign="top">drop</td> +<td>Drops slashes inside the field (e.g. "a/b" becomes "ab"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +<tr> +<td valign="top">replace</td> +<td>Replace slashes inside the field by an underscore. (e.g. "a/b" becomes "a_b"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +<tr> +<td><b>Format</b></td> +<td>Option values for the general output format.</td> +</tr> +<tr> +<td>json</td> +<td>encode the value so that it can be used inside a JSON field. This means +that several characters (according to the JSON spec) are being escaped, for +example US-ASCII LF is replaced by "\n". +The json option cannot be used together with either jsonf or csv options. +</td> +</tr> +<tr> +<td>jsonf</td> +<td><i>(available in 6.3.9+)</i> +This signifies that the property should be expressed as a json <b>f</b>ield. +That means not only the property is written, but rather a complete json field in +the format<br> +"fieldname"="value"</b> +where "filedname" is the assigend field name (or the property name if none was assigned) +and value is the end result of property replacer operation. Note that value supports +all property replacer options, like substrings, case converson and the like. +Values are properly json-escaped. However, field names are (currently) not. It is +expected that proper field names are configured. +The jsonf option cannot be used together with either json or csv options. +</td> +</tr> +<tr> +<td valign="top">csv</td> +<td>formats the resulting field (after all modifications) in CSV format +as specified in <a href="http://www.ietf.org/rfc/rfc4180.txt">RFC 4180</a>. +Rsyslog will always use double quotes. Note that in order to have full CSV-formatted +text, you need to define a proper template. An example is this one: +<br>$template csvline,"%syslogtag:::csv%,%msg:::csv%" +<br>Most importantly, you need to provide the commas between the fields +inside the template. +The csv option cannot be used together with either json or jsonf options. +<br><i>This feature was introduced in rsyslog 4.1.6.</i> +</td> +</tr> +<tr> +<td><b>droplastlf</b></td> +<td>The last LF in the message (if any), is dropped. +Especially useful for PIX.</td> +</tr> +<tr> +<td valign="top"><b>spifno1stsp</b></td> +<td>This option looks scary and should probably not be used by a user. For any field +given, it returns either a single space character or no character at all. Field content +is never returned. A space is returned if (and only if) the first character of the +field's content is NOT a space. This option is kind of a hack to solve a problem rooted +in RFC 3164: 3164 specifies no delimiter between the syslog tag sequence and the actual +message text. Almost all implementation in fact delemit the two by a space. As of +RFC 3164, this space is part of the message text itself. This leads to a problem when +building the message (e.g. when writing to disk or forwarding). Should a delimiting +space be included if the message does not start with one? If not, the tag is immediately +followed by another non-space character, which can lead some log parsers to misinterpret +what is the tag and what the message. The problem finally surfaced when the klog module +was restructured and the tag correctly written. It exists with other message sources, +too. The solution was the introduction of this special property replacer option. Now, +the default template can contain a conditional space, which exists only if the +message does not start with one. While this does not solve all issues, it should +work good enough in the far majority of all cases. If you read this text and have +no idea of what it is talking about - relax: this is a good indication you will never +need this option. Simply forget about it ;) +</td> +</tr> +<tr> +<td></td> +<td></td> +</tr> +<tr> +<td><b>New character position</b></td> +<td>In addition to the above mentioned Character Positions in the legacy format, +positions can be determined by specifying the correct options for the properties. +Again, this is mostly for using the list format.</td> +</tr> +<tr> +<td>position.From</td> +<td>Character position in the property to start from.</td> +</tr> +<tr> +<td>position.To</td> +<td>Character position that determines the end for extraction. If the value is "$" +then the end of the string will be used.</td> +</tr> +<tr> +<td>field.Number</td> +<td>The number of the field, which should be used for the search operation with Regex.</td> +</tr> +<tr> +<td>field.Delimiter</td> +<td>The Character that should delimit a field. Example: ",". Everything in a +property until this character is considered a field.</td> +</tr> +<tr> +<td>regex.Expression</td> +<td>Value to be compared to property.</td> +</tr> +<tr> +<td>regex.Type</td> +<td>Values BRE or ERE</td> +</tr> +<tr> +<td>regex.NoMatchMode</td> +<td>DFLT, BLANK, ZERO, FIELD</td> +</tr> +<tr> +<td>regex.Match</td> +<td>Match to use.</td> +</tr> +<tr> +<td>regex.Submatch</td> +<td>Submatch to use. Values 0-9 whereas 0 = All</td> +</tr> +</tbody> +</table> + + + +<h2>Legacy Property Options</h2> +<b><code>property options</code></b> are case-insensitive. Currently, the following options are defined: <p></p> <table> @@ -316,6 +583,29 @@ case-insensitive. Currently, the following options are defined: <td>convert property text to uppercase only</td> </tr> <tr> +<td><b>json</b></td> +<td>encode the value so that it can be used inside a JSON field. This means +that several characters (according to the JSON spec) are being escaped, for +example US-ASCII LF is replaced by "\n". +The json option cannot be used together with either jsonf or csv options. +</td> +</tr> +<tr> +<td><b>jsonf</b></td> +<td><i>(available in 6.3.9+)</i> +This signifies that the property should be expressed as a json <b>f</b>ield. +That means not only the property is written, but rather a complete json field in +the format<br> +"fieldname"="value"</b> +where "filedname" is the assigend field name (or the property name if none was assigned) +and value is the end result of property replacer operation. Note that value supports +all property replacer options, like substrings, case converson and the like. +Values are properly json-escaped. However, field names are (currently) not. It is +expected that proper field names are configured. +The jsonf option cannot be used together with either json or csv options. +</td> +</tr> +<tr> <td valign="top"><b>csv</b></td> <td>formats the resulting field (after all modifications) in CSV format as specified in <a href="http://www.ietf.org/rfc/rfc4180.txt">RFC 4180</a>. @@ -324,6 +614,7 @@ text, you need to define a proper template. An example is this one: <br>$template csvline,"%syslogtag:::csv%,%msg:::csv%" <br>Most importantly, you need to provide the commas between the fields inside the template. +The csv option cannot be used together with either json or jsonf options. <br><i>This feature was introduced in rsyslog 4.1.6.</i> </td> </tr> @@ -358,6 +649,10 @@ option when forwarding to remote hosts - they may treat the date as invalid <td>format as RFC 3339 date</td> </tr> <tr> +<td><b>date-unixtimestamp</b></td> +<td>format as unix timestamp (seconds since epoch)</td> +</tr> +<tr> <td><b>date-subseconds</b></td> <td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> </tr> @@ -419,19 +714,31 @@ Useful for secure pathname generation (with dynafiles). Useful for secure pathname generation (with dynafiles). </td> </tr> +<tr> +<td><b>mandatory-field</b></td> +<td>In templates that are used for building field lists (in particular, ommongodb), include +this field, even if it is empty (or NULL). If not set, the field will be removed from +the output field set if empty. The latter is the default case. +</tr> </tbody> </table> <p>To use multiple options, simply place them one after each other with a comma delmimiting them. For example "escape-cc,sp-if-no-1st-sp". If you use conflicting options together, the last one will override the previous one. For example, using "escape-cc,drop-cc" will use drop-cc and "drop-cc,escape-cc" will use escape-cc mode. +<h2>Fieldname</h2> +<p><i>(available in 6.3.9+)</i> +<p>This field permits to specify a field name for structured-data emitting property replacer +options. It was initially introduced to support the "jsonf" option, for which it provides +the capability to set an alternative field name. If it is not specified, it defaults to +the property name. <h2>Further Links</h2> <ul> <li>Article on "<a href="rsyslog_recording_pri.html">Recording the Priority of Syslog Messages</a>" (describes use of templates to record severity and facility of a message)</li> <li><a href="rsyslog_conf.html">Configuration file -syntax</a>, this is where you actually use the property replacer.</li> +format</a>, this is where you actually use the property replacer.</li> </ul> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] |