diff options
Diffstat (limited to 'doc/lookup_tables.html')
| -rw-r--r-- | doc/lookup_tables.html | 209 |
1 files changed, 209 insertions, 0 deletions
diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html new file mode 100644 index 00000000..f8fdbbf6 --- /dev/null +++ b/doc/lookup_tables.html @@ -0,0 +1,209 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Lookup Tables</title> +</head> + +<body> +<h1>Lookup Tables</h1> + +<p><b><font color="red">NOTE: this is</font> proposed functionality, which is +<font color="red">NOT YET IMPLEMENTED</font>!</b> + +<p><b>Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices).</b> +<p>The base idea is to use a message variable as an index into a table which then +returns another value. For example, $fromhost-ip could be used as an index, with +the table value representing the type of server or the department or remote office +it is located in. A main point with lookup tables is that the lookup is very fast. +So while lookup tables can be emulated with if-elseif constructs, they are generally +much faster. Also, it is possible to reload lookup tables during rsyslog runtime without +the need for a full restart. +<p>The lookup tables itself exists in a separate configuration file (one per table). This +file is loaded on rsyslog startup and when a reload is requested. Note that there are +two classes of lookup tables: "static" and "dynamic", where "static" means that the +table cannot by reloaded during runtime. For "dynamic" tables, this restriction +obviously does not exist. Note that dynamic tables require some type of locking to +coordinate the reloads (we use pthread reader/writer locks). This incurs some overhead +which static tables do not have. As such, it is advisable to use static tables if the +table either never changes or changes very infrequently and a full rsyslog restart +is acceptable in order to reload it. +<p>Lookup tables can be access via the lookup() built-in function. The core idea is to +set a local variable to the lookup result and later on use that local variable in templates. +<p>More details on usage now follow. +<h2>Lookup Table File Format</h2> +<p>Lookup table files consist of a header line and data lines, all inside a plain +text file. +<h3>Header</h3> +<p>The header line is required to be +<pre> +LookupTable version="1" nomatch="string" +</pre> +<p>Note that case is important, so "lookuptable" would be invalid. +This is intentional to protect against false positives. The version parameter +must be given and must always be one for this version of rsyslog. The nomatch +parameter is optional. If specified, it contains the value to be used if lookup() +is provided an index value for which no entry exists. The default for +"nomatch" is the empty string. Note that for version 1, +parameters must be provided in the order as specified above. +<h3>Data Lines</h3> +Each data line must follow this format: +<pre> +"index"="value" # comment +</pre> +Where "index" and "value" are the actual values to be used. Standard escapes can be used, +most importantly '\"' represents the double quote characters. Whitespace is ignored, as +is everything after the comment character. +<h3>Comment Lines</h3> +<p>Lines starting with "#" are comments and will be ignored. Leading whitespace before the +comment character is permitted. +<h3>Example</h3> +<p>This is a sample of how an ip-to-office mapping may look like: +<pre> +LookupTable version="1" nomatch="unk" +# IP 10.0.1.0/1/2 belong to office A +"10.0.1.0"="A" +"10.0.1.1"="A" +"10.0.1.2"="A" +# IP 10.0.2.0/1/2 belong to office B +"10.0.2.0"="B" +"10.0.2.1"="B" +"10.0.2.2"="B" +# if a different IP comes in, the value "unk" +# is returend thanks to the nomatch parameter in +# the first line. +</pre> + +<h2>RainerScript Statements</h2> +<h3>lookup_table() Object</h3> +<p>This statement defines and intially loads a lookup table. Its format is +as follows: +<pre> +lookup_table(name="name" file="/path/to/file" type="static|dynamic" + reloadOnHUP="on|off") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (mandatory)<br> + Defines the name of lookup table for further reference + inside the configuration. Names must be unique. Note that + it is possible, though not advisible, to have different + names for the same file. + <li><b>file</b> (mandatory)<br> + Specifies the full path for the lookup table file. This file + must be readable for the user rsyslog is run under (important + when dropping privileges). It must point to a valid lookup + table file as described above. + <li><b>type</b> (optional, default "dynamic")<br> + Set the type of this lookup table. Only dynamic tables + can be reloaded during runtime, but perform a bit + less well for the same reason (for details see + intro text). + <li><b>reloadOnHUP</b> (optional, default "on")<br> + Specifies if the table shall automatically be reloaded + as part of HUP processing. For static tables, the + default is "off" and specifying "on" triggers an + error message. Note that the default of "on" may be + somewhat suboptimal performance-wise, but probably + is what the user intuitively expects. Turn it off + if you know that you do not need the automatic + reload capability. +</ul> + +<h3>lookup() Function</h3> +<p>This function is used to actually do the table lookup. Format: +<pre> +lookup_table("name", indexvalue) +</pre> +<h4>Parameters</h4> +<ul> + <li><b>return value</b><br> + The function returns the string that is associated with the + given indexvalue. If the indexvalue is not present inside the + lookup table, the "nomatch" string is returned (or an empty string + if it is not defined). + <li><b>name</b> (constant string)<br> + The lookup table to be used. Note that this must be specificed as a + constant. In theory, variable table names could be made possible, but + their runtime behaviour is not as good as for static names, and we do + not (yet) see good use cases where dynamic table names could be useful. + <li><b>indexvalue</b> (expression)<br> + The value to be looked up. While this is an arbitrary RainerScript expression, + it's final value is always converted to a string in order to conduct + the lookup. For example, "lookup(table, 3+4)" would be exactly the same + as "lookup(table, "7")". In most cases, indexvalue will probably be + a single variable, but it could also be the result of all RainerScript-supported + expression types (like string concatenation or substring extraction). + Valid samples are "lookup(name, $fromhost-ip & $hostname)" or + "lookup(name, substr($fromhost-ip, 0, 5))" as well as of course the + usual "lookup(table, $fromhost-ip)". +</ul> + + +<h3>load_lookup_table Statement</h3> + +<p><b>Note: in the final implementation, this MAY be implemented as an action. +This is a low-level decesion that must be made during the detail development +process. Parameters and semantics will remain the same of this happens.</b> + +<p>This statement is used to reload a dynamic lookup table. It will fail if +the table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight performance +hit during the load phase. It is assume that always a triggering condition +is used to load the table. +<pre> +load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (string)<br> + The lookup table to be used. + <li><b>errOnFail</b> (boolean, default "on")<br> + Specifies whether or not an error message is to be emitted if + there are any problems reloading the lookup table. + <li><b>valueOnFail</b> (optional, string)<br> + This parameter affects processing if the lookup table cannot + be loaded for some reason: If the parameter is not present, + the previous table will be kept in use. If the parameter is + given, the previous table will no longer be used, and instead + an empty table be with nomath=valueOnFail be generated. In short, + that means when the parameter is set and the reload fails, + all matches will always return what is specified in valueOnFail. +</ul> + +<h3>Usage example</h3> +<p>For clarity, we show only those parts of rsyslog.conf that affect +lookup tables. We use the remote office example that an example lookup +table file is given above for. +<pre> +lookup_table(name="ip2office" file="/path/to/ipoffice.lu" + type="dynamic" reloadOnHUP="off") + + +template(name="depfile" type="string" + string="/var/log/%$usr.dep%/messages") + +set $usr.dep = lookup("ip2office", $fromhost-ip); +action(type="omfile" dynfile="depfile") + +# support for reload "commands" +if $fromhost-ip == "10.0.1.123" + and $msg contains "reload office lookup table" + then + load_lookup_table(name="ip2office" errOnFail="on") +</pre> + +<p>Note: for performance reasons, it makes sense to put the reload command into +a dedicated ruleset, bound to a specific listener - which than should also +be sufficiently secured, e.g. via TLS mutual auth. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> |