From ad9d197649d53cbd11566cd2b8091e1eca9e2f62 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Thu, 17 Jan 2013 13:02:03 +0100 Subject: design: draft lookup table functionality --- doc/Makefile.am | 1 + doc/lookup_tables.html | 209 +++++++++++++++++++++++++++++++++++++++++++++++++ doc/omfile.html | 4 +- doc/rainerscript.html | 7 +- 4 files changed, 218 insertions(+), 3 deletions(-) create mode 100644 doc/lookup_tables.html diff --git a/doc/Makefile.am b/doc/Makefile.am index a05edba5..d81750a2 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -70,6 +70,7 @@ html_files = \ tls_cert_client.html \ tls_cert_scenario.html \ rainerscript.html \ + lookup_tables.html \ rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html new file mode 100644 index 00000000..f8fdbbf6 --- /dev/null +++ b/doc/lookup_tables.html @@ -0,0 +1,209 @@ + + +Lookup Tables + + + +

Lookup Tables

+ +

NOTE: this is proposed functionality, which is +NOT YET IMPLEMENTED! + +

Lookup tables are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices). +

The base idea is to use a message variable as an index into a table which then +returns another value. For example, $fromhost-ip could be used as an index, with +the table value representing the type of server or the department or remote office +it is located in. A main point with lookup tables is that the lookup is very fast. +So while lookup tables can be emulated with if-elseif constructs, they are generally +much faster. Also, it is possible to reload lookup tables during rsyslog runtime without +the need for a full restart. +

The lookup tables itself exists in a separate configuration file (one per table). This +file is loaded on rsyslog startup and when a reload is requested. Note that there are +two classes of lookup tables: "static" and "dynamic", where "static" means that the +table cannot by reloaded during runtime. For "dynamic" tables, this restriction +obviously does not exist. Note that dynamic tables require some type of locking to +coordinate the reloads (we use pthread reader/writer locks). This incurs some overhead +which static tables do not have. As such, it is advisable to use static tables if the +table either never changes or changes very infrequently and a full rsyslog restart +is acceptable in order to reload it. +

Lookup tables can be access via the lookup() built-in function. The core idea is to +set a local variable to the lookup result and later on use that local variable in templates. +

More details on usage now follow. +

Lookup Table File Format

+

Lookup table files consist of a header line and data lines, all inside a plain +text file. +

Header

+

The header line is required to be +

+LookupTable version="1" nomatch="string"
+
+

Note that case is important, so "lookuptable" would be invalid. +This is intentional to protect against false positives. The version parameter +must be given and must always be one for this version of rsyslog. The nomatch +parameter is optional. If specified, it contains the value to be used if lookup() +is provided an index value for which no entry exists. The default for +"nomatch" is the empty string. Note that for version 1, +parameters must be provided in the order as specified above. +

Data Lines

+Each data line must follow this format: +
+"index"="value" # comment
+
+Where "index" and "value" are the actual values to be used. Standard escapes can be used, +most importantly '\"' represents the double quote characters. Whitespace is ignored, as +is everything after the comment character. +

Comment Lines

+

Lines starting with "#" are comments and will be ignored. Leading whitespace before the +comment character is permitted. +

Example

+

This is a sample of how an ip-to-office mapping may look like: +

+LookupTable version="1" nomatch="unk"
+# IP 10.0.1.0/1/2 belong to office A
+"10.0.1.0"="A"
+"10.0.1.1"="A"
+"10.0.1.2"="A"
+# IP 10.0.2.0/1/2 belong to office B
+"10.0.2.0"="B"
+"10.0.2.1"="B"
+"10.0.2.2"="B"
+# if a different IP comes in, the value "unk"
+# is returend thanks to the nomatch parameter in
+# the first line.
+
+ +

RainerScript Statements

+

lookup_table() Object

+

This statement defines and intially loads a lookup table. Its format is +as follows: +

+lookup_table(name="name" file="/path/to/file" type="static|dynamic"
+             reloadOnHUP="on|off")
+
+

Parameters

+ + +

lookup() Function

+

This function is used to actually do the table lookup. Format: +

+lookup_table("name", indexvalue)
+
+

Parameters

+ + + +

load_lookup_table Statement

+ +

Note: in the final implementation, this MAY be implemented as an action. +This is a low-level decesion that must be made during the detail development +process. Parameters and semantics will remain the same of this happens. + +

This statement is used to reload a dynamic lookup table. It will fail if +the table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight performance +hit during the load phase. It is assume that always a triggering condition +is used to load the table. +

+load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value")
+
+

Parameters

+ + +

Usage example

+

For clarity, we show only those parts of rsyslog.conf that affect +lookup tables. We use the remote office example that an example lookup +table file is given above for. +

+lookup_table(name="ip2office" file="/path/to/ipoffice.lu"
+             type="dynamic" reloadOnHUP="off")
+
+
+template(name="depfile" type="string"
+         string="/var/log/%$usr.dep%/messages")
+
+set $usr.dep = lookup("ip2office", $fromhost-ip);
+action(type="omfile" dynfile="depfile")
+
+# support for reload "commands"
+if $fromhost-ip == "10.0.1.123"
+   and $msg contains "reload office lookup table"
+   then
+   load_lookup_table(name="ip2office" errOnFail="on")
+
+ +

Note: for performance reasons, it makes sense to put the reload command into +a dedicated ruleset, bound to a specific listener - which than should also +be sufficiently secured, e.g. via TLS mutual auth. + +

[rsyslog.conf overview] +[manual index] [rsyslog site]

+

This documentation is part of the +rsyslog project.
+Copyright © 2008-2013 by Rainer Gerhards and +Adiscon. +Released under the GNU GPL version 3 or higher.

+ + diff --git a/doc/omfile.html b/doc/omfile.html index 5376af59..75ac5f49 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -13,7 +13,7 @@

The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded.

 

-

Module Confguration Parameters:

+

Module Configuration Parameters: