From d9905a864c1adc4fe8a519ed8d65189645390e59 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Thu, 14 Feb 2008 09:19:38 +0000 Subject: - enhanced file monitor doc - implemented $InputFilePollInterval config directive --- ChangeLog | 2 + doc/imfile.html | 119 ++++++++++++++++++++++++++++++++++-------------- plugins/imfile/imfile.c | 20 +++++--- 3 files changed, 99 insertions(+), 42 deletions(-) diff --git a/ChangeLog b/ChangeLog index e637c580..d945fe44 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,7 @@ --------------------------------------------------------------------------- Version 3.11.2 (rgerhards), 2008-02-?? +- added the capability to monitor text files and process their content + as syslog messages (including forwarding) - removed some left-over unnecessary dbgprintf's (cluttered screen, cosmetic) - doc bugfix: html documentation for omsnmp was missing diff --git a/doc/imfile.html b/doc/imfile.html index 6942b366..15ace7c1 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -13,46 +13,95 @@

Provides the ability to convert any standard text file into a syslog message. A standard text file is a file consisting of printable characters with lines -being delimited by LF.

The -file is read line-by-line and any line read is passed to rsyslog's rule -engine. The rule engine applies filter conditons and selects which -actions needs to be carried out.

As new lines are written they -are taken from the file and processed. Please note that this happens -based on a polling interval and not immediately. The file monitor -support file rotation. To fully work, rsyslogd must run while the file -is rotated. Then, any remaining lines from the old file are read and -processed and when done with that, the new file is being processed from -the beginning. If rsyslogd is stopped during rotation, the new file is -read, but any not-yet-reported lines from the previous file can no -longer be obtained.

When rsyslogd is stopped while monitoring a -text file, it records the last processed location and continues to work -from there upon restart. So no data is lost during a restart (except, -as noted above, if the file is rotated just in this very moment).

Currently, -the file must have a fixed name and location (directory). It is planned -to add support for dynamically generating file names in the future.

Multiple files may be monitored by specifying $InputRunFileMonitor multiple times. +being delimited by LF.

+

The file is read line-by-line and any line read is passed to +rsyslog's rule engine. The rule engine applies filter conditons and +selects which actions needs to be carried out.

+

As new lines are written they are taken from the file and +processed. Please note that this happens based on a polling interval +and not immediately. The file monitor support file rotation. To fully +work, rsyslogd must run while the file is rotated. Then, any remaining +lines from the old file are read and processed and when done with that, +the new file is being processed from the beginning. If rsyslogd is +stopped during rotation, the new file is read, but any not-yet-reported +lines from the previous file can no longer be obtained.

+

When rsyslogd is stopped while monitoring a text file, it +records the last processed location and continues to work from there +upon restart. So no data is lost during a restart (except, as noted +above, if the file is rotated just in this very moment).

+

Currently, the file must have a fixed name and location +(directory). It is planned to add support for dynamically generating +file names in the future.

+

Multiple files may be monitored by specifying +$InputRunFileMonitor multiple times.

Configuration Directives:

Caveats/Known Bugs: +The file being monitored. So far, this must be an absolute name (no +macros or templates) +
  • $InputFileTag +tag:
    +The tag to be used for messages that originate from this file. If you +would like to see the colon after the tag, you need to specify it here +(as shown above).
    • +
  • $InputFileStateFile +<name-of-state-file>
    +Rsyslog must keep track of which parts of the to be monitored file it +already processed. This is done in the state file. This file always is +created in the rsyslog working directory (configurable via +$WorkDirectory). Be careful to use unique names for different files +being monitored. If there are duplicates, all sorts of "interesting" +things may happen. Rsyslog currently does not check if a name is +specified multiple times.
    • +
  • $InputRunFileMonitor
    +This activates +the current monitor. It has no parameters. If you forget this +directive, no file monitoring will take place.
    • +
  • $InputFilePollInterval +seconds
    +This is a global setting. It specifies how often files are to be polled +for new data. The time specified is in seconds. The default value is 10 seconds. Please note that future +releases of imfile may support per-file polling intervals, but +currently this is not the case. If multiple $InputFilePollInterval +statements are present in rsyslog.conf, only the last one is used.
    +A short poll interval provides more rapid message forwarding, but +requires more system ressources. While it is possible, we stongly +recommend not to set the polling interval to 0 seconds. That will make +rsyslogd become a CPU hog, taking up considerable ressources. It is +supported, however, for the few very unusual situations where this +level may be needed. Even if you need quick response, 1 seconds should +be well enough. Please note that imfile keeps reading files as long as +there is any data in them. So a "polling sleep" will only happen when +nothing is left to be processed.
    • + +Caveats/Known Bugs: +

    So far, only 100 files can be monitored. If more are needed, +the source needs to be patched. See define MAX_INPUT_FILES in imfile.c +

    Sample:

    -

    The following sample monitors a single file.

    - +

    The following sample monitors two files. If you need just one, +remove the second one. If you need more, add them according to the +sample ;). This code must be placed in /etc/rsyslog.conf (or wherever +your distro puts rsyslog's config files).
    +

    +

    [rsyslog.conf overview] [manual index] [rsyslog site]

    This documentation is part of the diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index ce7134d2..b8198a4c 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -68,6 +68,7 @@ typedef struct fileInfo_s { static uchar *pszFileName = NULL; static uchar *pszFileTag = NULL; static uchar *pszStateFile = NULL; +static int iPollInterval = 10; /* number of seconds to sleep when there was no file activity */ static int iFacility; static int iSeverity; @@ -127,10 +128,10 @@ openFile(fileInfo_t *pThis) /* check if the file exists */ if(stat((char*) pszSFNam, &stat_buf) == -1) { if(errno == ENOENT) { - dbgoprint((obj_t*) pThis, "clean startup, no .si file found\n"); + /* currently no object! dbgoprint((obj_t*) pThis, "clean startup, no .si file found\n"); */ ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); } else { - dbgoprint((obj_t*) pThis, "error %d trying to access .si file\n", errno); + /* currently no object! dbgoprint((obj_t*) pThis, "error %d trying to access .si file\n", errno); */ ABORT_FINALIZE(RS_RET_IO_ERROR); } } @@ -158,7 +159,6 @@ finalize_it: strmDestruct(&psSF); if(iRet != RS_RET_OK) { - dbgoprint((obj_t*) pThis, "error %d reading .si file - can not read persisted info (not necessarily an error)\n", iRet); CHKiRet(strmConstruct(&pThis->pStrm)); CHKiRet(strmSettOperationsMode(pThis->pStrm, STREAMMODE_READ)); CHKiRet(strmSetsType(pThis->pStrm, STREAMTYPE_FILE_MONITOR)); @@ -252,7 +252,12 @@ CODESTARTrunInput pollFile(&files[i]); } - srSleep(3, 0); +RUNLOG_VAR("%d", iPollInterval); + /* Note: the 10ns additional wait is vitally important. It guards rsyslog against totally + * hogging the CPU if the users selects a polling interval of 0 seconds. It doesn't hurt any + * other valid scenario. So do not remove. -- rgerhards, 2008-02-14 + */ + srSleep(iPollInterval, 10); /* ------------------------------------------------------------------------------------------ * * DO NOT TOUCH the following code - it will soon be part of the module generation macros! */ @@ -398,6 +403,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a /* set defaults... */ + iPollInterval = 10; iFacility = 12; /* see RFC 3164 for values */ iSeverity = 4; @@ -411,7 +417,6 @@ static rsRetVal addMonitor(void __attribute__((unused)) *pVal, uchar __attribute DEFiRet; fileInfo_t *pThis; -RUNLOG_VAR("%d", iFilPtr); if(iFilPtr < MAX_INPUT_FILES) { pThis = &files[iFilPtr]; ++iFilPtr; @@ -453,9 +458,10 @@ CODEmodInit_QueryRegCFSLineHdlr /* use numerical values as of RFC 3164 for the time being... */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputfileseverity", 0, eCmdHdlrInt, NULL, &iSeverity, STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputfilesfacility", 0, eCmdHdlrInt, + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputfilefacility", 0, eCmdHdlrInt, NULL, &iFacility, STD_LOADABLE_MODULE_ID)); - /* things missing, e.g. polling intervall... */ + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputfilepollinterval", 0, eCmdHdlrInt, + NULL, &iPollInterval, STD_LOADABLE_MODULE_ID)); /* that command ads a new file! */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputrunfilemonitor", 0, eCmdHdlrGetWord, addMonitor, NULL, STD_LOADABLE_MODULE_ID)); -- cgit v1.2.3