diff options
223 files changed, 13009 insertions, 4427 deletions
@@ -1,4 +1,463 @@ --------------------------------------------------------------------------- +Version 7.3.7 [devel] 2013-02-?? +- add support for anonymizing IPv4 addresses +- add support for writing to the Linux Journal (omjournal) +- imuxsock: add capability to ignore messages from ourselfes + This helps prevent message routing loops, and is vital to have + if omjournal is used together with traditional syslog. +- field() function now supports a string as field delimiter +- added ability to configure debug system via rsyslog.conf +- bugfix: imuxsock segfault when system log socket was used +- bugfix: mmjsonparse segfault if new-style config was used +- bugfix: script == comparison did not work properly on JSON objects +- bugfix: field() function did never return "***FIELD NOT FOUND***" + instead it returned "***ERROR in field() FUNCTION***" in that case +--------------------------------------------------------------------------- +Version 7.3.6 [devel] 2013-01-28 +- greatly improved speed of large-array [N]EQ RainerScript comparisons + Thanks to David Lang for a related discussion that inspired the idea + to do this with a much simpler (yet sufficient) approach than orignally + planned for. +- greatly improved speed of DNS cache for large cache sizes +- general performance improvements +- omfile: added stats counters for dynafile caches +- omfile: improved async writing, finally enabled full async write + also fixed a couple of smaller issues along that way +- impstats: added ability to write stats records to local file + and avoid going through the syslog log stream. syslog logging can now + also be turned off (see doc for details). +- bugfix: imklog issued wrong facility in error messages + ...what could lead to problems in other parts of the code +- fix compile problem in imklog +- added capability to output thread-id-to-function debug info + This is a useful debug aid, but nothing of concern for regular users. +--------------------------------------------------------------------------- +Version 7.3.5 [devel] 2012-12-19 +- ommysql: addded batching/transaction support +- enhanced script optimizer to optimize common PRI-based comparisons + These constructs are especially used in SUSE default config files, + but also by many users (as they are more readable than the equivalent + PRI-based filter). +- omudpspoof: add support for new config system +- omudpspoof: add support for packets larger than 1472 bytes + On Ethernet, they need to be transmitted in multiple fragments. While + it is known that fragmentation can cause issues, it is the best choice + to be made in that case. Also improved debug output. +- bugfix: omudpspoof failed depending on the execution environment + The v7 engine closes fds, and closed some of libnet's fds as well, what + lead to problems (unfortunately, at least some libnet versions do not + report a proper error state but still "success"...). The order of libnet + calls has been adjusted to by in sync with what the core engine does. +- bugfix: segfault on imuxsock startup if system log socket is used + and no ratelimiting supported. Happens only during initial config + read phase, once this is over, everything works stable. +- bugfix: mmnormalize build problems +- bugfix: mmnormalize could abort rsyslog if config parameter was in error +- bugfix: no error message for invalid string template parameters + rather a malformed template was generated, and error information emitted + at runtime. However, this could be quite confusing. Note that with this + "bugfix" user experience changes: formerly, rsyslog and the affected + actions properly started up, but the actions did not produce proper + data. Now, there are startup error messages and the actions are NOT + executed (due to missing template due to template error). +- bugfix[minor]: invalid error code when mmnormalize could not access + rulebase +- bugfix(kind of): script optimizer did not work for complex boolean + expressions +- doc bugfix: corrections and improvements in mmnormalize html doc page +- bugfix: some message properties could be garbled due to race condition + This happened only on very high volume systems, if the same message was + being processed by two different actions. This was a regression caused + by the new config processor, which did no longer properly enable msg + locking in multithreaded cases. The bugfix is actually a refactoring of + the msg locking code - we no longer do unlocked operations, as the use + case for it has mostly gone away. It is potentially possible only at + very low-end systems, and there the small additional overhead of doing + the locking does not really hurt. Instead, the removal of that + capability can actually slightly improve performance in common cases, + as the code path is smaller and requires slightly less memory writes. + That probably outperforms the extra locking overhead (which in the + low-end case always happens in user space, without need for kernel + support as we can always directly aquire the lock - there is no + contention at all). +- build system cleanup (thanks to Michael Biebl for this!) +- bugfix: omelasticsearch did not properly compile on some platforms + due to missing libmath. Thanks to Michael Biebl for the fix +--------------------------------------------------------------------------- +Version 7.3.4 [devel] 2012-11-23 +- further (and rather drastically) improved disk queue performance + we now save one third of the IO calls +- imklog: added ParseKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- imklog: added KeepKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- bugfix: improper handling of backslash in string-type template()s +- bugfix: leading quote (") in string-type template() lead to thight loop + on startup +- bugfix: no error msg on invalid field option in legacy/string template +- bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds + ... actually, they are microseconds. So the fractional part of the + timestamp was not properly formatted. (import from 5.10.2) + Thanks to Marius Tomaschewski for the bug report and the patch idea. +--------------------------------------------------------------------------- +Version 7.3.3 [devel] 2012-11-07 +- improved disk queue performance +- bugfix: dynafile zip files could be corrupted + This could happen if a dynafile was destructed before the first write. + In practice, this could happen if few lines were written to a file and + it then became evicted from the dynafile cache. This would probably + look very random, because it depended on the timing in regard to + message volume and dynafile cache size. +--------------------------------------------------------------------------- +Version 7.3.2 [devel] 2012-10-30 +- mmnormalize: support for v6+ config interface added +- mmjsonparse: support for v6+ config interface added +--------------------------------------------------------------------------- +Version 7.3.2 [devel] 2012-10-30 +- totally reworked ratelimiting and "last message repeated n times" + all over rsyslog code. Each of the supported inputs now supports + linux-like ratelimiting (formerly only imuxsock did). Also, the + "last message repeated n times" is now processed at the input side + and no longer at the output side of rsyslog processing. This + provides the basis for new future additions as well as usually more + performance and a much simpler output part (which can be even further + refactored). +- imtcp: support for Linux-Type ratelimiting added +- imptcp: support for Linux-Type ratelimiting added +- imudp enhancements: + * support for input batching added (performance improvement) + * support for Linux-Type ratelimiting added +- permited action-like statements (stop, call, ...) in action lists +- bugfix: segfault on startup when modules using MSG_PASSING mode are used +- omelasticsearch: support for writing data errors to local file added +- omelasticsearch: fix check for bulk processing status response +--------------------------------------------------------------------------- +Version 7.3.1 [devel] 2012-10-19 +- optimized template processing performance, especially for $NOW family + of properties +- change lumberjack cookie to "@cee:" from "@cee: " + CEE originally specified the cookie with SP, whereas other lumberjack + tools used it without space. In order to keep interop with lumberjack, + we now use the cookie without space as well. I hope this can be changed + in CEE as well when it is released at a later time. + Thanks to Miloslav Trmač for pointing this out and a similiar v7 patch. +- bugfix: imuxsock and imklog truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski, Florian Piekert and Milan Bartos for their help in + solving this issue. +- bugfix: imuxsock did not properly honor $LocalHostIPIF +--------------------------------------------------------------------------- +Version 7.3.0 [devel] 2012-10-09 +- omlibdbi improvements, added + * support for config load phases & module() parameters + * support for default templates + * driverdirectory is now cleanly a global parameter, but can no longer + be specified as an action paramter. Note that in previous versions + this parameter was ignored in all but the first action definition +- improved omfile zip writer to increase compression + This was achieved by somewhat reducing the robustness of the zip archive. + This is controlled by the new action parameter "VeryReliableZip". +---------------------------------------------------------------------------- +Version 7.2.6 [v7-stable] 2013-01-?? +- slightly improved config parser error messages when invalid escapes happen +- bugfix: omelasticsearch failed when authentication data was provided + ... at least in most cases it emitted an error message: + "snprintf failed when trying to build auth string" + Thanks to Joerg Heinemann for alerting us. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=404 +- bugfix: some property-based filter were incorrectly parsed + This usually lead to a syntax error on startup and rsyslogd not actually + starting up. The problem was the regex, which did not care for double + quote characters to follow in the action part - unfortunately something + that can frequently happen with v6+ format. An example: + :programname, isequal, "as" {action(type="omfile" ...) } + Here, the part + :programname, isequal, "as" {action(type="omfile" + was treated as the property filter, and the rest as action part. + Obviously, this did not work out. Unfortunately, such situations usually + resulted in very hard to understand error messages. +---------------------------------------------------------------------------- +Version 7.2.5 [v7-stable] 2013-01-08 +- build system cleanup (thanks to Michael Biebl for this!) +- bugfix: omelasticsearch did not properly compile on some platforms + due to missing libmath. Thanks to Michael Biebl for the fix +- bugfix: invalid DST handling under Solaris + Thanks to Scott Severtson for the patch. +- bugfix: on termination, actions were incorrectly called + The problem was that incomplete fiter evaluation was done *during the + shutdown phase*. This affected only the LAST batches being processed. No + problem existed during the regular run. Could usually only happen on + very busy systems, which were still busy during shutdown. +- bugfix: very large memory consumption (and probably out of memory) when + FromPos was specified in template, but ToPos not. + Thanks to Radu Gheorghe for alerting us of this bug. +- bugfix: timeval2syslogTime cause problems on some platforms + due to invalid assumption on structure data types. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=394 + Thanks to David Hill for the patch [under ASL2.0 as per email conversation + 2013-01-03]. +- bugfix: compile errors in im3195 + Thanks to Martin Körper for the patch +- bugfix: doGetFileCreateMode() had invalid validity check ;) + Thanks to Chandler Latour for the patch. +- bugfix: mmjsonparse errornously returned action error when no CEE cookie + was present. +---------------------------------------------------------------------------- +Version 7.2.4 [v7-stable] 2012-12-07 +- enhance: permit RFC3339 timestamp in local log socket messages + Thanks to Sebastien Ponce for the patch. +- imklog: added ParseKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- fix missing functionality: ruleset(){} could not specify ruleset queue + The "queue.xxx" parameter set was not supported, and legacy ruleset + config statements did not work (by intention). The fix introduces the + "queue.xxx" parameter set. It has some regression potential, but only + for the new functionality. Note that using that interface it is possible + to specify duplicate queue file names, which will cause trouble. This + will be solved in v7.3, because there is a too-large regression + potential for the v7.2 stable branch. +- imklog: added KeepKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds + ... actually, they are microseconds. So the fractional part of the + timestamp was not properly formatted. (import from 5.10.2) + Thanks to Marius Tomaschewski for the bug report and the patch idea. +- bugfix: supportoctetcountedframing parameter did not work in imptcp +- bugfix: modules not (yet) supporting new conf format were not properly + registered. This lead to a "module not found" error message instead of + the to-be-expected "module does not support new style" error message. + That invalid error message could be quite misleading and actually stop + people from addressing the real problem (aka "go nuts" ;)) +- bugfix: template "type" parameter is mandatory (but was not) +- bugfix: some message properties could be garbled due to race condition + This happened only on very high volume systems, if the same message was + being processed by two different actions. This was a regression caused + by the new config processor, which did no longer properly enable msg + locking in multithreaded cases. The bugfix is actually a refactoring of + the msg locking code - we no longer do unlocked operations, as the use + case for it has mostly gone away. It is potentially possible only at + very low-end systems, and there the small additional overhead of doing + the locking does not really hurt. Instead, the removal of that + capability can actually slightly improve performance in common cases, + as the code path is smaller and requires slightly less memory writes. + That probably outperforms the extra locking overhead (which in the + low-end case always happens in user space, without need for kernel + support as we can always directly aquire the lock - there is no + contention at all). +---------------------------------------------------------------------------- +Version 7.2.3 [v7-stable] 2012-10-21 +- regression fix: rsyslogd terminated when wild-card $IncludeConfig did not + find actual include files. For example, if this directive is present: + $IncludeConfig /etc/rsyslog.d/*.conf + and there are no *.conf files in /etc/rsyslog.d (but rsyslog.d exists), + rsyslogd will emit an error message and terminate. Previous (and expected) + behaviour is that an empty file set is no problem. HOWEVER, if the + directory itself does not exist, this is flagged as an error and will + load to termination (no startup). + Unfortunately, this is often the case by default in many distros, so this + actually prevents rsyslog startup. +---------------------------------------------------------------------------- +Version 7.2.2 [v7-stable] 2012-10-16 +- doc improvements +- enabled to build without libuuid, at loss of uuid functionality + this enables smoother builds on older systems that do not support + libuuid. Loss of functionality should usually not matter too much as + uuid support has only recently been added and is very seldom used. +- bugfix: omfwd did not properly support "template" parameter +- bugfix: potential segfault when re_match() function was used + Thanks to oxpa for the patch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=371 +- bugfix: potential abort of imtcp on rsyslogd shutdown +- bugfix: imzmq3 segfault with PULL subscription + Thanks to Martin Nilsson for the patch. +- bugfix: improper handling of backslash in string-type template()s +- bugfix: leading quote (") in string-type template() lead to thight loop + on startup +- bugfix: no error msg on invalid field option in legacy/string template +- bugfix: potential segfault due to invalid param handling in comparisons + This could happen in RainerScript comparisons (like contains); in some + cases an unitialized variable was accessed, which could lead to an + invalid free and in turn to a segfault. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=372 + Thanks to Georgi Georgiev for reporting this bug and his great help + in solving it. +- bugfix: no error msg on unreadable $IncludeConfig path +- bugfix: $IncludeConfig did not correctly process directories + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=376 + The testbench was also enhanced to check for these cases. + Thanks to Georgi Georgiev for the bug report. +- bugfix: make rsyslog compile on kfreebsd again + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=380 + Thanks to Guillem Jover for the patch. +- bugfix: garbled message if field name was used with jsonf property option + The length for the field name was invalidly computed, resulting in either + truncated field names or including extra random data. If the random data + contained NULs, the rest of the message became unreadable. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=374 +- bugfix: potential segfault at startup with property-based filter + If the property name was followed by a space before the comma, rsyslogd + aborted on startup. Note that no segfault could happen if the initial + startup went well (this was a problem with the config parser). + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=381 +- bugfix: imfile discarded some file parts + File lines that were incomplete (LF missing) *at the time imfile polled + the file* were partially discarded. That part of the line that was read + without the LF was discarded, and the rest of the line was submitted in + the next polling cycle. This is now changed so that the partial content + is saved until the complete line is read. Note that the patch affects + only read mode 0. + Thanks to Milan Bartos for providing the base idea for the solution. +---------------------------------------------------------------------------- +Version 7.2.1 [v7-stable] 2012-10-29 +- bugfix: ruleset()-object did only support a single statement +- added -D rsyslogd option to enable config parser debug mode +- improved syntax error messages by outputting the error token +- the rsyslog core now suspeneds actions after 10 failures in a row + This was former the case after 1,000 failures and could cause rsyslog + to be spammed/ressources misused. See the v6 compatibility doc for more + details. +- ommongodb rate-limits error messages to prevent spamming the syslog + closes (for v7.2): http://bugzilla.adiscon.com/show_bug.cgi?id=366 +---------------------------------------------------------------------------- +Version 7.2.0 [v7-stable] 2012-10-22 +This starts a new stable branch based on 7.1.12 plus the following changes: +- bugfix: imuxsock did not properly honor $LocalHostIPIF +- omruleset/omdiscard do no longer issue "deprecated" warings, as 7.1 + grammar does not permit to use the replacements under all circumstances +---------------------------------------------------------------------------- +Version 7.1.12 [beta] 2012-10-18 +- minor updates to better support newer systemd developments + Thanks to Michael Biebl for the patches. +- build system cleanup + Thanks to Michael Biebl for the patch series. +- cleanup: removed remains of -c option (compatibility mode) + both from code & doc and emitted warning message if still used + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=361 + Thanks to Michael Biebl for reporting & suggestions +- bugfix: imklog truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski and Florian Piekert for their help in solving this issue. +---------------------------------------------------------------------------- +Version 7.1.11 [beta] 2012-10-16 +- bugfix: imuxsock truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski, Florian Piekert and Milan Bartos for their help in + solving this issue. +- bugfix: do not crash if set statement is used with date field + Thanks to Miloslav Trmač for the patch. +- change lumberjack cookie to "@cee:" from "@cee: " + CEE originally specified the cookie with SP, whereas other lumberjack + tools used it without space. In order to keep interop with lumberjack, + we now use the cookie without space as well. I hope this can be changed + in CEE as well when it is released at a later time. + Thanks to Miloslav Trmač for pointing this out and a similiar v7 patch. +- added deprecated note to omruleset (plus clue to use "call") +- added deprecated note to discard action (plus clue to use "stop") +--------------------------------------------------------------------------- +Version 7.1.10 [beta] 2012-10-11 + - bugfix: m4 directory was not present in release tarball + - bugfix: small memory leak with string-type templates + - bugfix: small memory leak when template was specified in omfile + - bugfix: some config processing warning messages were treated as errors + - bugfix: small memory leak when processing action() statements + - bugfix: unknown action() parameters were not reported +--------------------------------------------------------------------------- +Version 7.1.9 [beta] 2012-10-09 +- bugfix: comments inside objects (e.g. action()) were not properly handled +- bugfix: in (non)equal comparisons the position of arrays influenced result + This behaviour is OK for "contains"-type of comparisons (which have quite + different semantics), but not for == and <>, which shall be commutative. + This has been fixed now, so there is no difference any longer if the + constant string array is the left or right hand operand. We solved this + via the optimizer, as it keeps the actual script execution code small. +--------------------------------------------------------------------------- +Version 7.1.8 [beta] 2012-10-02 +- bugfix: ruleset(){} directive errornously changed default ruleset + much like the $ruleset legacy conf statement. This potentially lead + to statements being assigned to the wrong ruleset. +- improved module doc +- added "parser" parameter to ruleset(), so that parser chain can be + configured +- implemented "continue" RainerScript statement +--------------------------------------------------------------------------- +Version 7.1.7 [devel] 2012-10-01 +- implemented RainerScript "call" statement +- implemented RainerScript array-based string comparison operations +- implemented imtcp "permittedPeers" module-global parameter +- imudp: support for specifying multiple ports via array added +--------------------------------------------------------------------------- +Version 7.1.6 [devel] 2012-09-28 +- implemented RainerScript input() statement, including support for it + in major input plugins +- implemented RainerScript ruleset() statement +--------------------------------------------------------------------------- +Version 7.1.5 [devel] 2012-09-25 +- implemented RainerScript prifield() function +- implemented RainerScript field() function +- added new module imkmsg to process structured kernel log + Thanks to Milan Bartos for contributing this module +- implemented basic RainerScript optimizer, which will speed up script + operations +- bugfix: invalid free if function re_match() was incorrectly used + if the config file parser detected that param 2 was not constant, some + data fields were not initialized. The destructor did not care about that. + This bug happened only if rsyslog startup was unclean. +--------------------------------------------------------------------------- +Version 7.1.4 [devel] 2012-09-19 +- implemented ability for CEE-based properties to be stored in disk queues +- implemented string concatenation in expressions via &-operator +- implemented json subtree copy in variable assignment +- implemented full JSON support for variable manipulation +- introduced "subtree"-type templates +- bugfix: omfile action did not respect "template" parameter + ... and used default template in all cases +- bugfix: MsgDup() did not copy CEE structure + This function was called at various places, most importantly during + "last messages repeated n times" processing and omruleset. If CEE(JSON) + data was present, it was lost as part of the copy process. +- bugfix: debug output indicated improper queue type +--------------------------------------------------------------------------- +Version 7.1.3 [devel] 2012-09-17 +- introduced "set" and "unset" config statements +- bugfix: missing support for escape sequences in RainerScript + only \' was supported. Now the usual set is supported. Note that v5 + used \x as escape where x was any character (e.g. "\n" meant "n" and NOT + LF). This also means there is some incompatibility to v5 for well-know + sequences. Better break it now than later. +- bugfix: invalid property name in property-filter could cause abort + if action chaining (& operator) was used + http://bugzilla.adiscon.com/show_bug.cgi?id=355 + Thanks to pilou@gmx.com for the bug report +--------------------------------------------------------------------------- +Version 7.1.2 [devel] 2012-09-12 +- bugfix: messages were duplicated, sometimes massively + regression from new code in 7.1.1 and reason for early release +- bugfix: remove invalid socket option call from imuxsock + Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom +- bugfix: abort when invalid property name was configured + in property-based filter +- bugfix: multiple rulesets did no longer work correctly (7.1.1 regression) +--------------------------------------------------------------------------- +Version 7.1.1 [devel] 2012-09-11 +- MAJOR NEW FEATURE: rulengine now fully supports nesting + including if ... then ... else ... constructs. This is a big change + and it obviously has a lot of bug potential. +- BSD-style (filter) blocks are no longer supported + see http://www.rsyslog.com/g/BSD for details and solution +- imuxsock now stores trusted properties by default in the CEE root + This was done in order to keep compatible with other implementations of + the lumberjack schema + Thanks to Miloslav Trmač for pointing to this. +- bugfix: string-generating templates caused abort if CEE field could not + be found +--------------------------------------------------------------------------- +Version 7.1.0 [devel] 2012-09-06 +- added support for hierarchical properties (CEE/lumberjack) +- added pure JSON output plugin parameter passing mode +- ommongodb now supports templates +- bugfix: imtcp could abort on exit due to invalid free() +- imported bugfixes from 6.4.1 +--------------------------------------------------------------------------- Version 6.6.1 [v6-stable] 2012-10-?? - bugfix: build problems on some platforms - bugfix: misaddressing of $mmnormalizeuserawmsg parameter @@ -66,13 +525,9 @@ Version 6.5.1 [beta] 2012-10-11 - bugfix: imtcp could abort on exit due to invalid free() - bugfix: remove invalid socket option call from imuxsock Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom -- bugfix: missing support for escape sequences in RainerScript - only \' was supported. Now the usual set is supported. Note that v5 - used \x as escape where x was any character (e.g. "\n" meant "n" and NOT - LF). This also means there is some incompatibility to v5 for well-know - sequences. Better break it now than later. -- bugfix: remove invalid socket option call from imuxsock - Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom +- added pure JSON output plugin parameter passing mode +- ommongodb now supports templates +- bugfix: imtcp could abort on exit due to invalid free() - bugfix: missing support for escape sequences in RainerScript only \' was supported. Now the usual set is supported. Note that v5 used \x as escape where x was any character (e.g. "\n" meant "n" and NOT @@ -734,7 +1189,7 @@ Version 5.10.2 [V5-STABLE], 201?-??-?? Thanks to Tomas Heinrich for the patch. - bugfix[kind of]: omudpspoof discarded messages >1472 bytes (MTU size) it now truncates these message, but ensures they are sent. Note that - 7.2.5+ will switch to fragmented UDP messages instead (up to 64K) + 7.3.5+ will switch to fragmented UDP messages instead (up to 64K) --------------------------------------------------------------------------- Version 5.10.1 [V5-STABLE], 2012-10-17 - bugfix: imuxsock and imklog truncated head of received message diff --git a/Makefile.am b/Makefile.am index 98708c80..f99fa9c7 100644 --- a/Makefile.am +++ b/Makefile.am @@ -58,10 +58,11 @@ CLEANFILES = \ endif EXTRA_DIST = \ - freebsd/rsyslogd \ - slackware/rc.rsyslogd \ + platform/README \ + platform/freebsd/rsyslogd \ + platform/slackware/rc.rsyslogd \ + platform/redhat/rsyslog.conf \ contrib/README \ - rsyslog.conf \ COPYING \ COPYING.LESSER \ COPYING.ASL20 \ @@ -70,7 +71,7 @@ EXTRA_DIST = \ contrib/gnutls/key.pem \ rsyslog.service.in -SUBDIRS = doc runtime grammar . plugins/immark plugins/imuxsock plugins/imtcp plugins/imudp plugins/omtesting +SUBDIRS = doc runtime grammar compat . plugins/immark plugins/imuxsock plugins/imtcp plugins/imudp plugins/omtesting if ENABLE_RSYSLOGD SUBDIRS += tools @@ -80,6 +81,10 @@ if ENABLE_IMKLOG SUBDIRS += plugins/imklog endif +if ENABLE_IMKMSG +SUBDIRS += plugins/imkmsg +endif + if ENABLE_IMPSTATS SUBDIRS += plugins/impstats endif @@ -172,6 +177,10 @@ if ENABLE_OMHDFS SUBDIRS += plugins/omhdfs endif +if ENABLE_OMJOURNAL +SUBDIRS += plugins/omjournal +endif + if ENABLE_ELASTICSEARCH SUBDIRS += plugins/omelasticsearch endif @@ -220,6 +229,10 @@ if ENABLE_MMAUDIT SUBDIRS += plugins/mmaudit endif +if ENABLE_MMANON +SUBDIRS += plugins/mmanon +endif + if ENABLE_ORACLE SUBDIRS += plugins/omoracle endif @@ -240,7 +253,6 @@ SUBDIRS += tests # in a make distcheck is so that we detect code that accidently was not updated # when some global update happened. DISTCHECK_CONFIGURE_FLAGS= --enable-gssapi_krb5 \ - --enable-gnutls \ --enable-imfile \ --enable-snmp \ --enable-libdbi \ @@ -12,11 +12,11 @@ * necessary to triple-check that everything works well in *all* modes. * The different modes (and calling sequence) are: * - * if set iExecEveryNthOccur > 1 || f_ReduceRepeated || iSecsExecOnceInterval + * if set iExecEveryNthOccur > 1 || iSecsExecOnceInterval * - doSubmitToActionQComplexBatch * - helperSubmitToActionQComplexBatch * - doActionCallAction - * handles duplicate message processing, but in essence calls + * handles mark message reduction, but in essence calls * - actionWriteToAction * - qqueueEnqObj * (now queue engine processing) @@ -289,6 +289,11 @@ rsRetVal actionDestruct(action_t *pThis) DEFiRet; ASSERT(pThis != NULL); + if(!strcmp((char*)modGetName(pThis->pMod), "builtin:omdiscard")) { + /* discard actions will be optimized out */ + FINALIZE; + } + if(pThis->pQueue != NULL) { qqueueDestruct(&pThis->pQueue); } @@ -302,16 +307,13 @@ rsRetVal actionDestruct(action_t *pThis) if(pThis->pMod != NULL) pThis->pMod->freeInstance(pThis->pModData); - if(pThis->f_pMsg != NULL) - msgDestruct(&pThis->f_pMsg); - pthread_mutex_destroy(&pThis->mutAction); pthread_mutex_destroy(&pThis->mutActExec); d_free(pThis->pszName); d_free(pThis->ppTpl); +finalize_it: d_free(pThis); - RETiRet; } @@ -362,6 +364,10 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) ASSERT(pThis != NULL); + if(!strcmp((char*)modGetName(pThis->pMod), "builtin:omdiscard")) { + /* discard actions will be optimized out */ + FINALIZE; + } /* generate a friendly name for us action stats */ if(pThis->pszName == NULL) { snprintf((char*) pszAName, sizeof(pszAName)/sizeof(uchar), "action %d", iActionNbr); @@ -401,16 +407,11 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) * mode is much faster processing (and simpler code) -- rgerhards, 2010-06-08 */ if( pThis->iExecEveryNthOccur > 1 - || pThis->f_ReduceRepeated || pThis->iSecsExecOnceInterval ) { DBGPRINTF("info: firehose mode disabled for action because " - "iExecEveryNthOccur=%d, " - "ReduceRepeated=%d, " - "iSecsExecOnceInterval=%d\n", - pThis->iExecEveryNthOccur, pThis->f_ReduceRepeated, - pThis->iSecsExecOnceInterval - ); + "iExecEveryNthOccur=%d, iSecsExecOnceInterval=%d\n", + pThis->iExecEveryNthOccur, pThis->iSecsExecOnceInterval); pThis->submitToActQ = doSubmitToActionQComplexBatch; } else if(pThis->bWriteAllMarkMsgs == RSFALSE) { /* nearly full-speed submission mode, default case */ @@ -429,7 +430,7 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) CHKiRet(qqueueConstruct(&pThis->pQueue, cs.ActionQueType, 1, cs.iActionQueueSize, (rsRetVal (*)(void*, batch_t*, int*))processBatchMain)); obj.SetName((obj_t*) pThis->pQueue, pszAName); - qqueueSetpUsr(pThis->pQueue, pThis); + qqueueSetpAction(pThis->pQueue, pThis); if(queueParams == NULL) { /* use legacy params? */ /* ... set some properties ... */ @@ -642,7 +643,7 @@ actionDoRetry(action_t *pThis, time_t ttNow, int *pbShutdownImmediate) iRetries = 0; while((*pbShutdownImmediate == 0) && pThis->eState == ACT_STATE_RTRY) { iRet = pThis->pMod->tryResume(pThis->pModData); - if((pThis->iResumeOKinRow > 999) && (pThis->iResumeOKinRow % 1000 == 0)) { + if((pThis->iResumeOKinRow > 9) && (pThis->iResumeOKinRow % 10 == 0)) { bTreatOKasSusp = 1; pThis->iResumeOKinRow = 0; } else { @@ -765,7 +766,6 @@ rsRetVal actionDbgPrint(action_t *pThis) pThis->pMod->dbgPrintInstInfo(pThis->pModData); dbgprintf("\n"); dbgprintf("\tInstance data: 0x%lx\n", (unsigned long) pThis->pModData); - dbgprintf("\tRepeatedMsgReduction: %d\n", pThis->f_ReduceRepeated); dbgprintf("\tResume Interval: %d\n", pThis->iResumeInterval); if(pThis->eState == ACT_STATE_SUSP) { dbgprintf("\tresume next retry: %u, number retries: %d", @@ -792,7 +792,8 @@ rsRetVal actionDbgPrint(action_t *pThis) /* prepare the calling parameters for doAction() * rgerhards, 2009-05-07 */ -static rsRetVal prepareDoActionParams(action_t *pAction, batch_obj_t *pElem) +static rsRetVal +prepareDoActionParams(action_t *pAction, batch_obj_t *pElem, struct syslogTime *ttNow) { int i; msg_t *pMsg; @@ -802,23 +803,23 @@ static rsRetVal prepareDoActionParams(action_t *pAction, batch_obj_t *pElem) ASSERT(pAction != NULL); ASSERT(pElem != NULL); - pMsg = (msg_t*) pElem->pUsrp; + pMsg = pElem->pMsg; /* here we must loop to process all requested strings */ for(i = 0 ; i < pAction->iNumTpls ; ++i) { switch(pAction->eParamPassing) { case ACT_STRING_PASSING: CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(pElem->staticActStrings[i]), - &pElem->staticLenStrings[i])); + &pElem->staticLenStrings[i], ttNow)); pElem->staticActParams[i] = pElem->staticActStrings[i]; break; case ACT_ARRAY_PASSING: - CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pElem->staticActParams[i]))); + CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pElem->staticActParams[i]), ttNow)); break; case ACT_MSG_PASSING: pElem->staticActParams[i] = (void*) pMsg; break; case ACT_JSON_PASSING: - CHKiRet(tplToJSON(pAction->ppTpl[i], pMsg, &json)); + CHKiRet(tplToJSON(pAction->ppTpl[i], pMsg, &json, ttNow)); pElem->staticActParams[i] = (void*) json; break; default:dbgprintf("software bug/error: unknown pAction->eParamPassing %d in prepareDoActionParams\n", @@ -851,9 +852,12 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) ASSERT(pAction != NULL); + if(pAction->eParamPassing == ACT_STRING_PASSING || pAction->eParamPassing == ACT_MSG_PASSING) + goto done; /* we need to do nothing with these types! */ + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { pElem = &(pBatch->pElem[i]); - if(pElem->bFilterOK && pElem->state != BATCH_STATE_DISC) { + if(batchIsValidElem(pBatch, i)) { switch(pAction->eParamPassing) { case ACT_ARRAY_PASSING: ppMsgs = (uchar***) pElem->staticActParams; @@ -870,19 +874,6 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) } } break; - case ACT_STRING_PASSING: - case ACT_MSG_PASSING: - /* nothing to do in that case */ - /* TODO ... and yet we do something ;) This is considered not - * really needed, but I was not bold enough to remove that while - * fixing the stable. It should be removed in a devel version - * soon (I really don't see a reason why we would need it). - * rgerhards, 2010-12-16 - */ - for(j = 0 ; j < pAction->iNumTpls ; ++j) { - ((uchar**)pElem->staticActParams)[j] = NULL; - } - break; case ACT_JSON_PASSING: for(j = 0 ; j < pAction->iNumTpls ; ++j) { json_object_put((struct json_object*) @@ -890,11 +881,15 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) pElem->staticActParams[j] = NULL; } break; + case ACT_STRING_PASSING: + case ACT_MSG_PASSING: + /* can never happen, just to keep compiler happy! */ + break; } } } - RETiRet; +done: RETiRet; } @@ -1051,9 +1046,8 @@ tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) /* NOTE: do NOT extend the filter below! Anything else must be done on the * enq side of the queue (see file header comment)! -- rgerhards, 2011-06-15 */ - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC) { - pMsg = (msg_t*) pBatch->pElem[i].pUsrp; + if(batchIsValidElem(pBatch, i)) { + pMsg = pBatch->pElem[i].pMsg; localRet = actionProcessMessage(pAction, pMsg, pBatch->pElem[i].staticActParams, pBatch->pbShutdownImmediate); DBGPRINTF("action %p call returned %d\n", pAction, localRet); @@ -1076,11 +1070,11 @@ tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) ++iCommittedUpTo; //pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; } - pBatch->pElem[i].state = BATCH_STATE_SUB; + pBatch->eltState[i] = BATCH_STATE_SUB; } else if(localRet == RS_RET_DEFER_COMMIT) { - pBatch->pElem[i].state = BATCH_STATE_SUB; + pBatch->eltState[i] = BATCH_STATE_SUB; } else if(localRet == RS_RET_DISCARDMSG) { - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } else { dbgprintf("tryDoAction: unexpected error code %d[nElem %d, Commited UpTo %d], finalizing\n", localRet, *pnElem, iCommittedUpTo); @@ -1142,9 +1136,9 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) } else if(localRet == RS_RET_ACTION_FAILED) { /* in this case, everything not yet committed is BAD */ for(i = pBatch->iDoneUpTo ; i < wasDoneTo + nElem ; ++i) { - if( pBatch->pElem[i].state != BATCH_STATE_DISC - && pBatch->pElem[i].state != BATCH_STATE_COMM ) { - pBatch->pElem[i].state = BATCH_STATE_BAD; + if( pBatch->eltState[i] != BATCH_STATE_DISC + && pBatch->eltState[i] != BATCH_STATE_COMM ) { + pBatch->eltState[i] = BATCH_STATE_BAD; pBatch->pElem[i].bPrevWasSuspended = 1; STATSCOUNTER_INC(pAction->ctrFail, pAction->mutCtrFail); } @@ -1173,6 +1167,29 @@ finalize_it: } +/* copy "active" array of batch, as we need to modify it. The caller + * must make sure the new array is freed and the orginal batch + * pointer is restored (thus the caller must save it). If active + * is currently NULL, this is properly handled. + * Note: the batches active pointer is modified, so it must be + * saved BEFORE calling this function! + * rgerhards, 2012-09-12 + */ +static rsRetVal +copyActive(batch_t *pBatch) +{ + sbool *active; + DEFiRet; + + CHKmalloc(active = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); + if(pBatch->active == NULL) + memset(active, 1, batchNumMsgs(pBatch)); + else + memcpy(active, pBatch->active, batchNumMsgs(pBatch)); + pBatch->active = active; +finalize_it: + RETiRet; +} /* The following function prepares a batch for processing, that it is * reinitializes batch states, generates strings and does everything else @@ -1183,19 +1200,29 @@ finalize_it: * rgerhards, 2010-06-14 */ static inline rsRetVal -prepareBatch(action_t *pAction, batch_t *pBatch) +prepareBatch(action_t *pAction, batch_t *pBatch, sbool **activeSave, int *bMustRestoreActivePtr) { int i; batch_obj_t *pElem; + struct syslogTime ttNow; DEFiRet; + /* indicate we have not yet read the date */ + ttNow.year = 0; + pBatch->iDoneUpTo = 0; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { pElem = &(pBatch->pElem[i]); - if(pElem->bFilterOK && pElem->state != BATCH_STATE_DISC) { - pElem->state = BATCH_STATE_RDY; - if(prepareDoActionParams(pAction, pElem) != RS_RET_OK) - pElem->bFilterOK = RSFALSE; + if(batchIsValidElem(pBatch, i)) { + pBatch->eltState[i] = BATCH_STATE_RDY; + if(prepareDoActionParams(pAction, pElem, &ttNow) != RS_RET_OK) { + /* make sure we have our copy of "active" array */ + if(!*bMustRestoreActivePtr) { + *activeSave = pBatch->active; + copyActive(pBatch); + } + pBatch->active[i] = RSFALSE; + } } } RETiRet; @@ -1228,6 +1255,8 @@ static rsRetVal processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) { int *pbShutdownImmdtSave; + sbool *activeSave; + int bMustRestoreActivePtr = 0; rsRetVal localRet; DEFiRet; @@ -1235,7 +1264,7 @@ processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) pbShutdownImmdtSave = pBatch->pbShutdownImmediate; pBatch->pbShutdownImmediate = pbShutdownImmediate; - CHKiRet(prepareBatch(pAction, pBatch)); + CHKiRet(prepareBatch(pAction, pBatch, &activeSave, &bMustRestoreActivePtr)); /* We now must guard the output module against execution by multiple threads. The * plugin interface specifies that output modules must not be thread-safe (except @@ -1258,6 +1287,11 @@ processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) if(iRet == RS_RET_OK) iRet = localRet; + + if(bMustRestoreActivePtr) { + free(pBatch->active); + pBatch->active = activeSave; + } finalize_it: pBatch->pbShutdownImmediate = pbShutdownImmdtSave; @@ -1343,9 +1377,9 @@ doSubmitToActionQ(action_t *pAction, msg_t *pMsg) STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); if(pAction->pQueue->qType == QUEUETYPE_DIRECT) - iRet = qqueueEnqObjDirect(pAction->pQueue, (void*) MsgAddRef(pMsg)); + iRet = qqueueEnqMsgDirect(pAction->pQueue, MsgAddRef(pMsg)); else - iRet = qqueueEnqObj(pAction->pQueue, eFLOWCTL_NO_DELAY, (void*) MsgAddRef(pMsg)); + iRet = qqueueEnqMsg(pAction->pQueue, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); finalize_it: RETiRet; @@ -1360,14 +1394,10 @@ finalize_it: * be filtered out before calling us (what is done currently!). */ rsRetVal -actionWriteToAction(action_t *pAction) +actionWriteToAction(action_t *pAction, msg_t *pMsg) { - msg_t *pMsgSave; /* to save current message pointer, necessary to restore - it in case it needs to be updated (e.g. repeated msgs) */ DEFiRet; - pMsgSave = NULL; /* indicate message poiner not saved */ - /* first, we check if the action should actually be called. The action-specific * $ActionExecOnlyEveryNthTime permits us to execute an action only every Nth * time. So we need to check if we need to drop the (otherwise perfectly executable) @@ -1394,43 +1424,6 @@ actionWriteToAction(action_t *pAction) } } - /* then check if this is a regular message or the repeation of - * a previous message. If so, we need to change the message text - * to "last message repeated n times" and then go ahead and write - * it. Please note that we can not modify the message object, because - * that would update it in other selectors as well. As such, we first - * need to create a local copy of the message, which we than can update. - * rgerhards, 2007-07-10 - */ - if(pAction->f_prevcount > 1) { - msg_t *pMsg; - size_t lenRepMsg; - uchar szRepMsg[1024]; - - if((pMsg = MsgDup(pAction->f_pMsg)) == NULL) { - /* it failed - nothing we can do against it... */ - DBGPRINTF("Message duplication failed, dropping repeat message.\n"); - ABORT_FINALIZE(RS_RET_ERR); - } - - if(pAction->bRepMsgHasMsg == 0) { /* old format repeat message? */ - lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " last message repeated %d times", - pAction->f_prevcount); - } else { - lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " message repeated %d times: [%.800s]", - pAction->f_prevcount, getMSG(pAction->f_pMsg)); - } - - /* We now need to update the other message properties. Please note that digital - * signatures inside the message are also invalidated. - */ - datetime.getCurrTime(&(pMsg->tRcvdAt), &(pMsg->ttGenTime)); - memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); - MsgReplaceMSG(pMsg, szRepMsg, lenRepMsg); - pMsgSave = pAction->f_pMsg; /* save message pointer for later restoration */ - pAction->f_pMsg = pMsg; /* use the new msg (pointer will be restored below) */ - } - DBGPRINTF("Called action(complex case), logging to %s\n", module.GetStateName(pAction->pMod)); /* now check if we need to drop the message because otherwise the action would be too @@ -1451,31 +1444,14 @@ actionWriteToAction(action_t *pAction) /* we use reception time, not dequeue time - this is considered more appropriate and also faster ;) * rgerhards, 2008-09-17 */ pAction->tLastExec = getActNow(pAction); /* re-init time flags */ - pAction->f_time = pAction->f_pMsg->ttGenTime; + pAction->f_time = pMsg->ttGenTime; /* When we reach this point, we have a valid, non-disabled action. * So let's enqueue our message for execution. -- rgerhards, 2007-07-24 */ - iRet = doSubmitToActionQ(pAction, pAction->f_pMsg); - - if(iRet == RS_RET_OK) - pAction->f_prevcount = 0; /* message processed, so we start a new cycle */ + iRet = doSubmitToActionQ(pAction, pMsg); finalize_it: - if(pMsgSave != NULL) { - /* we had saved the original message pointer. That was - * done because we needed to create a temporary one - * (most often for "message repeated n time" handling). If so, - * we need to restore the original one now, so that procesing - * can continue as normal. We also need to discard the temporary - * one, as we do not like memory leaks ;) Please note that the original - * message object will be discarded by our callers, so this is nothing - * of our business. rgerhards, 2007-07-10 - */ - msgDestruct(&pAction->f_pMsg); - pAction->f_pMsg = pMsgSave; /* restore it */ - } - RETiRet; } @@ -1489,7 +1465,7 @@ doActionCallAction(action_t *pAction, batch_t *pBatch, int idxBtch) msg_t *pMsg; DEFiRet; - pMsg = (msg_t*)(pBatch->pElem[idxBtch].pUsrp); + pMsg = pBatch->pElem[idxBtch].pMsg; pAction->tActNow = -1; /* we do not yet know our current time (clear prev. value) */ /* don't output marks to recently written outputs */ @@ -1498,43 +1474,8 @@ doActionCallAction(action_t *pAction, batch_t *pBatch, int idxBtch) ABORT_FINALIZE(RS_RET_OK); } - /* suppress duplicate messages */ - if ((pAction->f_ReduceRepeated == 1) && pAction->f_pMsg != NULL && - (pMsg->msgFlags & MARK) == 0 && getMSGLen(pMsg) == getMSGLen(pAction->f_pMsg) && - !ustrcmp(getMSG(pMsg), getMSG(pAction->f_pMsg)) && - !strcmp(getHOSTNAME(pMsg), getHOSTNAME(pAction->f_pMsg)) && - !strcmp(getPROCID(pMsg, LOCK_MUTEX), getPROCID(pAction->f_pMsg, LOCK_MUTEX)) && - !strcmp(getAPPNAME(pMsg, LOCK_MUTEX), getAPPNAME(pAction->f_pMsg, LOCK_MUTEX))) { - pAction->f_prevcount++; - DBGPRINTF("msg repeated %d times, %ld sec of %d.\n", - pAction->f_prevcount, (long) getActNow(pAction) - pAction->f_time, - repeatinterval[pAction->f_repeatcount]); - /* use current message, so we have the new timestamp (means we need to discard previous one) */ - msgDestruct(&pAction->f_pMsg); - pAction->f_pMsg = MsgAddRef(pMsg); - /* If domark would have logged this by now, flush it now (so we don't hold - * isolated messages), but back off so we'll flush less often in the future. - */ - if(getActNow(pAction) > REPEATTIME(pAction)) { - iRet = actionWriteToAction(pAction); - BACKOFF(pAction); - } - } else {/* new message, save it */ - /* first check if we have a previous message stored - * if so, emit and then discard it first - */ - if(pAction->f_pMsg != NULL) { - if(pAction->f_prevcount > 0) - actionWriteToAction(pAction); - /* we do not care about iRet above - I think it's right but if we have - * some troubles, you know where to look at ;) -- rgerhards, 2007-08-01 - */ - msgDestruct(&pAction->f_pMsg); - } - pAction->f_pMsg = MsgAddRef(pMsg); - /* call the output driver */ - iRet = actionWriteToAction(pAction); - } + /* call the output driver */ + iRet = actionWriteToAction(pAction, pMsg); finalize_it: /* we need to update the batch to handle failover processing correctly */ @@ -1564,7 +1505,8 @@ DEFFUNC_llExecFunc(doActivateActions) } actionDisable(pThis); } - DBGPRINTF("Action %p: queue %p started\n", pThis, pThis->pQueue); + DBGPRINTF("Action %s[%p]: queue %p started\n", modGetName(pThis->pMod), + pThis, pThis->pQueue); ENDfunc return RS_RET_OK; /* we ignore errors, we can not do anything either way */ } @@ -1598,22 +1540,15 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) time_t now = 0; time_t lastAct; int i; - int bModifiedFilter; - sbool FilterSave[1024]; - sbool *pFilterSave; + sbool *activeSave; DEFiRet; - if(batchNumMsgs(pBatch) <= (int) (sizeof(FilterSave)/sizeof(sbool))) { - pFilterSave = FilterSave; - } else { - CHKmalloc(pFilterSave = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); - } + activeSave = pBatch->active; + copyActive(pBatch); - bModifiedFilter = 0; for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(!pBatch->pElem[i].bFilterOK) + if((pBatch->eltState[i] == BATCH_STATE_DISC) || !pBatch->active[i]) continue; - pFilterSave[i] = pBatch->pElem[i].bFilterOK; if(now == 0) { now = datetime.GetTime(NULL); /* good time call - the only one done */ } @@ -1622,17 +1557,17 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) * also faster ;) -- rgerhards, 2008-09-17 */ do { lastAct = pAction->f_time; - if(((msg_t*)(pBatch->pElem[i].pUsrp))->msgFlags & MARK) { + if(pBatch->pElem[i].pMsg->msgFlags & MARK) { if((now - lastAct) < MarkInterval / 2) { - pBatch->pElem[i].bFilterOK = 0; - bModifiedFilter = 1; - DBGPRINTF("action was recently called, ignoring mark message\n"); + pBatch->active[i] = 0; + DBGPRINTF("batch item %d: action was recently called, ignoring " + "mark message\n", i); break; /* do not update timestamp for non-written mark messages */ } } } while(ATOMIC_CAS_time_t(&pAction->f_time, lastAct, - ((msg_t*)(pBatch->pElem[i].pUsrp))->ttGenTime, &pAction->mutCAS) == 0); - if(pBatch->pElem[i].bFilterOK) { + pBatch->pElem[i].pMsg->ttGenTime, &pAction->mutCAS) == 0); + if(pBatch->active[i]) { DBGPRINTF("Called action(NotAllMark), processing batch[%d] via '%s'\n", i, module.GetStateName(pAction->pMod)); } @@ -1640,17 +1575,8 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) iRet = doSubmitToActionQBatch(pAction, pBatch); - if(bModifiedFilter) { - /* in this case, we need to restore previous state */ - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - /* note: clang static code analyzer reports a false positive below */ - pBatch->pElem[i].bFilterOK = pFilterSave[i]; - } - } - -finalize_it: - if(pFilterSave != FilterSave) - free(pFilterSave); + free(pBatch->active); + pBatch->active = activeSave; RETiRet; } @@ -1660,8 +1586,7 @@ countStatsBatchEnq(action_t *pAction, batch_t *pBatch) { int i; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC) { + if( batchIsValidElem(pBatch, i)) { STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); } } @@ -1675,18 +1600,13 @@ countStatsBatchEnq(action_t *pAction, batch_t *pBatch) static inline rsRetVal doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) { - sbool FilterSave[1024]; - sbool *pFilterSave; sbool bNeedSubmit; - sbool bModifiedFilter; + sbool *activeSave; int i; DEFiRet; - if(batchNumMsgs(pBatch) <= (int) (sizeof(FilterSave)/sizeof(sbool))) { - pFilterSave = FilterSave; - } else { - CHKmalloc(pFilterSave = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); - } + activeSave = pBatch->active; + copyActive(pBatch); /* note: for direct mode, we need to adjust the filter property. For non-direct * this is not necessary, because in that case we enqueue only what actually needs @@ -1694,37 +1614,25 @@ doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) */ if(pAction->bExecWhenPrevSusp) { bNeedSubmit = 0; - bModifiedFilter = 0; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - pFilterSave[i] = pBatch->pElem[i].bFilterOK; if(!pBatch->pElem[i].bPrevWasSuspended) { - DBGPRINTF("action enq stage: change bFilterOK to 0 due to " + DBGPRINTF("action enq stage: change active to 0 due to " "failover case in elem %d\n", i); - pBatch->pElem[i].bFilterOK = 0; - bModifiedFilter = 1; + pBatch->active[i] = 0; } - if(pBatch->pElem[i].bFilterOK && pBatch->pElem[i].state != BATCH_STATE_DISC) { + if(batchIsValidElem(pBatch, i)) { STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); bNeedSubmit = 1; } - DBGPRINTF("action %p[%d]: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, i, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p[%d]: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, i, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); } if(bNeedSubmit) { /* note: stats were already computed above */ iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); } else { - DBGPRINTF("no need to submit batch, all bFilterOK==0 or discarded\n"); - } - if(bModifiedFilter) { - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, - pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - /* note: clang static code analyzer reports a false positive below */ - pBatch->pElem[i].bFilterOK = pFilterSave[i]; - } + DBGPRINTF("no need to submit batch, all invalid\n"); } } else { if(GatherStats) @@ -1732,7 +1640,8 @@ doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); } -finalize_it: + free(pBatch->active); + pBatch->active = activeSave; RETiRet; } @@ -1755,13 +1664,12 @@ doSubmitToActionQBatch(action_t *pAction, batch_t *pBatch) * TODO: optimize this, we may do at least a multi-submit! */ for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC + if( batchIsValidElem(pBatch, i) && (pAction->bExecWhenPrevSusp == 0 || pBatch->pElem[i].bPrevWasSuspended == 1)) { - doSubmitToActionQ(pAction, (msg_t*)(pBatch->pElem[i].pUsrp)); + doSubmitToActionQ(pAction, pBatch->pElem[i].pMsg); } } } @@ -1784,11 +1692,10 @@ helperSubmitToActionQComplexBatch(action_t *pAction, batch_t *pBatch) DBGPRINTF("Called action %p (complex case), logging to %s\n", pAction, module.GetStateName(pAction->pMod)); for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC + if( batchIsValidElem(pBatch, i) && ((pAction->bExecWhenPrevSusp == 0) || pBatch->pElem[i].bPrevWasSuspended) ) { doActionCallAction(pAction, pBatch, i); } @@ -1858,7 +1765,6 @@ actionApplyCnfParam(action_t *pAction, struct cnfparamvals *pvals) } - /* add an Action to the current selector * The pOMSR is freed, as it is not needed after this function. * Note: this function pulls global data that specifies action config state. @@ -1923,7 +1829,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, && (pAction->ppTpl[i] = tplFind(ourConf, (char*)pTplName, strlen((char*)pTplName))) == NULL) { snprintf(errMsg, sizeof(errMsg) / sizeof(char), - " Could not find template '%s' - action disabled\n", + " Could not find template '%s' - action disabled", pTplName); errno = 0; errmsg.LogError(0, RS_RET_NOT_FOUND, "%s", errMsg); @@ -1954,13 +1860,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, pAction->pMod = pMod; pAction->pModData = pModData; - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) { - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - } else { - DBGPRINTF("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* check if the module is compatible with select features (currently no such features exist) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ if(bSuspended) @@ -2060,13 +1960,8 @@ actionNewInst(struct nvlst *lst, action_t **ppAction) if((iRet = addAction(&pAction, pMod, pModData, pOMSR, paramvals, queueParams, (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - else { - DBGPRINTF("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* check if the module is compatible with select features + * (currently no such features exist) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ loadConf->actions.nbrActions++; /* one more active action! */ } @@ -2078,33 +1973,6 @@ finalize_it: RETiRet; } - -/* Process a rsyslog v6 action config object (the now-primary config method). - * rgerhards, 2011-07-19 - */ -rsRetVal -actionProcessCnf(struct cnfobj __attribute__((unused)) *o) -{ - DEFiRet; -#if 0 /* we need to check if we actually need this functionality -- later! */ -// This is for STAND-ALONE actions at the conf file TOP level - struct cnfparamvals *paramvals; - - paramvals = nvlstGetParams(o->nvlst, &pblk, NULL); - if(paramvals == NULL) { - iRet = RS_RET_ERR; - goto finalize_it; - } - DBGPRINTF("action param blk after actionProcessCnf:\n"); - cnfparamsPrint(&pblk, paramvals); - - /* now find module to activate */ -finalize_it: -#endif - RETiRet; -} - - /* TODO: we are not yet a real object, the ClassInit here just looks like it is.. */ rsRetVal actionClassInit(void) @@ -46,9 +46,8 @@ typedef enum { /* the following struct defines the action object data structure */ -typedef struct action_s action_t; struct action_s { - time_t f_time; /* used for "message repeated n times" - be careful, old, old code */ + time_t f_time; /* used for "max. n messages in m seconds" processing */ time_t tActNow; /* the current time for an action execution. Initially set to -1 and populated on an as-needed basis. This is a performance optimization. */ time_t tLastExec; /* time this action was last executed */ @@ -69,9 +68,6 @@ struct action_s { struct modInfo_s *pMod;/* pointer to output module handling this selector */ void *pModData; /* pointer to module data - content is module-specific */ sbool bRepMsgHasMsg; /* "message repeated..." has msg fragment in it (0-no, 1-yes) */ - short f_ReduceRepeated;/* reduce repeated lines 0 - no, 1 - yes */ - int f_prevcount; /* repetition cnt of prevline */ - int f_repeatcount; /* number of "repeated" msgs */ rsRetVal (*submitToActQ)(action_t *, batch_t *);/* function submit message to action queue */ rsRetVal (*qConstruct)(struct queue_s *pThis); enum { ACT_STRING_PASSING = 0, ACT_ARRAY_PASSING = 1, ACT_MSG_PASSING = 2, @@ -80,10 +76,6 @@ struct action_s { int iNumTpls; /* number of array entries for template element below */ struct template **ppTpl;/* array of template to use - strings must be passed to doAction * in this order. */ - msg_t *f_pMsg; /* pointer to the message (this will replace the other vars with msg - * content later). This is preserved after the message has been - * processed - it is also used to detect duplicates. - */ qqueue_t *pQueue; /* action queue */ pthread_mutex_t mutAction; /* primary action mutex */ pthread_mutex_t mutActExec; /* mutex to guard actual execution of doAction for single-threaded modules */ @@ -104,7 +96,7 @@ rsRetVal actionDestruct(action_t *pThis); rsRetVal actionDbgPrint(action_t *pThis); rsRetVal actionSetGlobalResumeInterval(int iNewVal); rsRetVal actionDoAction(action_t *pAction); -rsRetVal actionWriteToAction(action_t *pAction); +rsRetVal actionWriteToAction(action_t *pAction, msg_t *pMsg); rsRetVal actionCallHUPHdlr(action_t *pAction); rsRetVal actionClassInit(void); rsRetVal addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringRequest_t *pOMSR, struct cnfparamvals *actParams, struct cnfparamvals *queueParams, int bSuspended); diff --git a/compat/Makefile.am b/compat/Makefile.am new file mode 100644 index 00000000..635ad280 --- /dev/null +++ b/compat/Makefile.am @@ -0,0 +1,6 @@ +pkglib_LTLIBRARIES = compat.la + +compat_la_SOURCES = getifaddrs.c +compat_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +compat_la_LDFLAGS = -module -avoid-version +compat_la_LIBADD = $(IMUDP_LIBS) diff --git a/compat/getifaddrs.c b/compat/getifaddrs.c new file mode 100755 index 00000000..8acd4840 --- /dev/null +++ b/compat/getifaddrs.c @@ -0,0 +1,275 @@ +#include "config.h" +#ifndef HAVE_GETIFADDRS +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ + +/* + * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. + */ + +#include <netdb.h> +#include <nss_dbdefs.h> +#include <netinet/in.h> +#include <sys/socket.h> +#include <string.h> +#include <stdio.h> +#include <sys/sockio.h> +#include <sys/types.h> +#include <stdlib.h> +#include <net/if.h> +#include <ifaddrs.h> +#include <libsocket_priv.h> + +/* + * Create a linked list of `struct ifaddrs' structures, one for each + * address that is UP. If successful, store the list in *ifap and + * return 0. On errors, return -1 and set `errno'. + * + * The storage returned in *ifap is allocated dynamically and can + * only be properly freed by passing it to `freeifaddrs'. + */ +int +getifaddrs(struct ifaddrs **ifap) +{ + int err; + char *cp; + struct ifaddrs *curr; + + if (ifap == NULL) { + errno = EINVAL; + return (-1); + } + *ifap = NULL; + err = getallifaddrs(AF_UNSPEC, ifap, LIFC_ENABLED); + if (err == 0) { + for (curr = *ifap; curr != NULL; curr = curr->ifa_next) { + if ((cp = strchr(curr->ifa_name, ':')) != NULL) + *cp = '\0'; + } + } + return (err); +} + +void +freeifaddrs(struct ifaddrs *ifa) +{ + struct ifaddrs *curr; + + while (ifa != NULL) { + curr = ifa; + ifa = ifa->ifa_next; + free(curr->ifa_name); + free(curr->ifa_addr); + free(curr->ifa_netmask); + free(curr->ifa_dstaddr); + free(curr); + } +} + +/* + * Returns all addresses configured on the system. If flags contain + * LIFC_ENABLED, only the addresses that are UP are returned. + * Address list that is returned by this function must be freed + * using freeifaddrs(). + */ +int +getallifaddrs(sa_family_t af, struct ifaddrs **ifap, int64_t flags) +{ + struct lifreq *buf = NULL; + struct lifreq *lifrp; + struct lifreq lifrl; + int ret; + int s, n, numifs; + struct ifaddrs *curr, *prev; + sa_family_t lifr_af; + int sock4; + int sock6; + int err; + + if ((sock4 = socket(AF_INET, SOCK_DGRAM, 0)) < 0) + return (-1); + if ((sock6 = socket(AF_INET6, SOCK_DGRAM, 0)) < 0) { + err = errno; + close(sock4); + errno = err; + return (-1); + } + +retry: + /* Get all interfaces from SIOCGLIFCONF */ + ret = getallifs(sock4, af, &buf, &numifs, (flags & ~LIFC_ENABLED)); + if (ret != 0) + goto fail; + + /* + * Loop through the interfaces obtained from SIOCGLIFCOMF + * and retrieve the addresses, netmask and flags. + */ + prev = NULL; + lifrp = buf; + *ifap = NULL; + for (n = 0; n < numifs; n++, lifrp++) { + + /* Prepare for the ioctl call */ + (void) strncpy(lifrl.lifr_name, lifrp->lifr_name, + sizeof (lifrl.lifr_name)); + lifr_af = lifrp->lifr_addr.ss_family; + if (af != AF_UNSPEC && lifr_af != af) + continue; + + s = (lifr_af == AF_INET ? sock4 : sock6); + + if (ioctl(s, SIOCGLIFFLAGS, (caddr_t)&lifrl) < 0) + goto fail; + if ((flags & LIFC_ENABLED) && !(lifrl.lifr_flags & IFF_UP)) + continue; + + /* + * Allocate the current list node. Each node contains data + * for one ifaddrs structure. + */ + curr = calloc(1, sizeof (struct ifaddrs)); + if (curr == NULL) + goto fail; + + if (prev != NULL) { + prev->ifa_next = curr; + } else { + /* First node in the linked list */ + *ifap = curr; + } + prev = curr; + + curr->ifa_flags = lifrl.lifr_flags; + if ((curr->ifa_name = strdup(lifrp->lifr_name)) == NULL) + goto fail; + + curr->ifa_addr = malloc(sizeof (struct sockaddr_storage)); + if (curr->ifa_addr == NULL) + goto fail; + (void) memcpy(curr->ifa_addr, &lifrp->lifr_addr, + sizeof (struct sockaddr_storage)); + + /* Get the netmask */ + if (ioctl(s, SIOCGLIFNETMASK, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_netmask = malloc(sizeof (struct sockaddr_storage)); + if (curr->ifa_netmask == NULL) + goto fail; + (void) memcpy(curr->ifa_netmask, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + + /* Get the destination for a pt-pt interface */ + if (curr->ifa_flags & IFF_POINTOPOINT) { + if (ioctl(s, SIOCGLIFDSTADDR, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_dstaddr = malloc( + sizeof (struct sockaddr_storage)); + if (curr->ifa_dstaddr == NULL) + goto fail; + (void) memcpy(curr->ifa_dstaddr, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + } else if (curr->ifa_flags & IFF_BROADCAST) { + if (ioctl(s, SIOCGLIFBRDADDR, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_broadaddr = malloc( + sizeof (struct sockaddr_storage)); + if (curr->ifa_broadaddr == NULL) + goto fail; + (void) memcpy(curr->ifa_broadaddr, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + } + + } + free(buf); + close(sock4); + close(sock6); + return (0); +fail: + err = errno; + free(buf); + freeifaddrs(*ifap); + *ifap = NULL; + if (err == ENXIO) + goto retry; + close(sock4); + close(sock6); + errno = err; + return (-1); +} + +/* + * Do a SIOCGLIFCONF and store all the interfaces in `buf'. + */ +int +getallifs(int s, sa_family_t af, struct lifreq **lifr, int *numifs, + int64_t lifc_flags) +{ + struct lifnum lifn; + struct lifconf lifc; + size_t bufsize; + char *tmp; + caddr_t *buf = (caddr_t *)lifr; + + lifn.lifn_family = af; + lifn.lifn_flags = lifc_flags; + + *buf = NULL; +retry: + if (ioctl(s, SIOCGLIFNUM, &lifn) < 0) + goto fail; + + /* + * When calculating the buffer size needed, add a small number + * of interfaces to those we counted. We do this to capture + * the interface status of potential interfaces which may have + * been plumbed between the SIOCGLIFNUM and the SIOCGLIFCONF. + */ + bufsize = (lifn.lifn_count + 4) * sizeof (struct lifreq); + + if ((tmp = realloc(*buf, bufsize)) == NULL) + goto fail; + + *buf = tmp; + lifc.lifc_family = af; + lifc.lifc_flags = lifc_flags; + lifc.lifc_len = bufsize; + lifc.lifc_buf = *buf; + if (ioctl(s, SIOCGLIFCONF, (char *)&lifc) < 0) + goto fail; + + *numifs = lifc.lifc_len / sizeof (struct lifreq); + if (*numifs >= (lifn.lifn_count + 4)) { + /* + * If every entry was filled, there are probably + * more interfaces than (lifn.lifn_count + 4). + * Redo the ioctls SIOCGLIFNUM and SIOCGLIFCONF to + * get all the interfaces. + */ + goto retry; + } + return (0); +fail: + free(*buf); + *buf = NULL; + return (-1); +} +#endif /* HAVE_GETIFADDRS */ diff --git a/compat/ifaddrs.h b/compat/ifaddrs.h new file mode 100755 index 00000000..ec359c13 --- /dev/null +++ b/compat/ifaddrs.h @@ -0,0 +1,89 @@ +#include "config.h" +#ifndef HAVE_GETIFADDRS +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ +/* + * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. + */ +#ifndef _IFADDRS_H +#define _IFADDRS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <sys/types.h> + +/* + * The `getifaddrs' function generates a linked list of these structures. + * Each element of the list describes one network interface. + */ +#if defined(_INT64_TYPE) +struct ifaddrs { + struct ifaddrs *ifa_next; /* Pointer to the next structure. */ + char *ifa_name; /* Name of this network interface. */ + uint64_t ifa_flags; /* Flags as from SIOCGLIFFLAGS ioctl. */ + struct sockaddr *ifa_addr; /* Network address of this interface. */ + struct sockaddr *ifa_netmask; /* Netmask of this interface. */ + union { + /* + * At most one of the following two is valid. If the + * IFF_BROADCAST bit is set in `ifa_flags', then + * `ifa_broadaddr' is valid. If the IFF_POINTOPOINT bit is + * set, then `ifa_dstaddr' is valid. It is never the case that + * both these bits are set at once. + */ + struct sockaddr *ifu_broadaddr; + struct sockaddr *ifu_dstaddr; + } ifa_ifu; + void *ifa_data; /* Address-specific data (may be unused). */ +/* + * This may have been defined in <net/if.h>. + */ +#ifndef ifa_broadaddr +#define ifa_broadaddr ifa_ifu.ifu_broadaddr /* broadcast address */ +#endif +#ifndef ifa_dstaddr +#define ifa_dstaddr ifa_ifu.ifu_dstaddr /* other end of p-to-p link */ +#endif +}; +#endif + +/* + * Create a linked list of `struct ifaddrs' structures, one for each + * network interface on the host machine. If successful, store the + * list in *ifap and return 0. On errors, return -1 and set `errno'. + * + * The storage returned in *ifap is allocated dynamically and can + * only be properly freed by passing it to `freeifaddrs'. + */ +extern int getifaddrs(struct ifaddrs **); + +/* Reclaim the storage allocated by a previous `getifaddrs' call. */ +extern void freeifaddrs(struct ifaddrs *); + + +#ifdef __cplusplus +} +#endif + +#endif /* _IFADDRS_H */ +#endif /* HAVE_GETIFADDRS */ diff --git a/configure.ac b/configure.ac index 7664cf39..a0c063c2 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[6.6.0],[rsyslog@lists.adiscon.com]) +AC_INIT([rsyslog],[7.3.6],[rsyslog@lists.adiscon.com]) AM_INIT_AUTOMAKE m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) @@ -35,8 +35,6 @@ PKG_PROG_PKG_CONFIG PKG_CHECK_MODULES(LIBESTR, libestr >= 0.1.2) PKG_CHECK_MODULES(LIBEE, libee >= 0.4.0) PKG_CHECK_MODULES([JSON_C], [json]) -AC_SUBST([JSON_CFLAGS]) -AC_SUBST([JSON_LIBS]) case "${host}" in *-*-linux*) @@ -116,7 +114,7 @@ AC_TYPE_SIGNAL AC_FUNC_STAT AC_FUNC_STRERROR_R AC_FUNC_VPRINTF -AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync lseek64]) +AC_CHECK_FUNCS([flock basename alarm clock_gettime getifaddrs gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64]) # the check below is probably ugly. If someone knows how to do it in a better way, please # let me know! -- rgerhards, 2010-10-06 @@ -124,6 +122,7 @@ AC_CHECK_DECL([SCM_CREDENTIALS], [AC_DEFINE(HAVE_SCM_CREDENTIALS, [1], [set defi #include <sys/socket.h>]) AC_CHECK_DECL([SO_TIMESTAMP], [AC_DEFINE(HAVE_SO_TIMESTAMP, [1], [set define])], [], [#include <sys/types.h> #include <sys/socket.h>]) +AC_CHECK_MEMBER([struct sysinfo.uptime], [AC_DEFINE(HAVE_SYSINFO_UPTIME, [1], [set define])], [], [#include <sys/sysinfo.h>]) # Check for MAXHOSTNAMELEN AC_MSG_CHECKING(for MAXHOSTNAMELEN) @@ -147,7 +146,7 @@ RS_ATOMIC_OPERATIONS RS_ATOMIC_OPERATIONS_64BIT # fall back to POSIX sems for atomic operations (cpu expensive) -AC_CHECK_HEADERS([semaphore.h]) +AC_CHECK_HEADERS([semaphore.h sys/syscall.h]) # Additional module directories @@ -245,22 +244,7 @@ AM_CONDITIONAL(ENABLE_GSSAPI, test x$enable_gssapi_krb5 = xyes) # multithreading via pthreads -AC_ARG_ENABLE(pthreads, - [AS_HELP_STRING([--enable-pthreads],[Enable multithreading via pthreads @<:@default=yes@:>@])], - [case "${enableval}" in - yes) enable_pthreads="yes" ;; - no) enable_pthreads="no" ;; - *) AC_MSG_ERROR(bad value ${enableval} for --enable-pthreads) ;; - esac], - [enable_pthreads=yes] -) - -if test "x$enable_pthreads" = "xno"; then - AC_MSG_ERROR(rsyslog v3+ does no longer support single threading mode -- use a previous version for that); -fi - -if test "x$enable_pthreads" != "xno"; then - AC_CHECK_HEADERS( +AC_CHECK_HEADERS( [pthread.h], [ AC_CHECK_LIB( @@ -281,8 +265,7 @@ if test "x$enable_pthreads" != "xno"; then ) ], [AC_MSG_FAILURE([pthread is missing])] - ) -fi +) AC_CHECK_FUNCS( [pthread_setschedparam], @@ -330,26 +313,17 @@ AM_CONDITIONAL(ENABLE_IMKLOG_BSD, test x$os_type = xbsd) AM_CONDITIONAL(ENABLE_IMKLOG_LINUX, test x$os_type = xlinux) AM_CONDITIONAL(ENABLE_IMKLOG_SOLARIS, test x$os_type = xsolaris) - -# -# SYSLOG_UNIXAF -# -AC_MSG_CHECKING(for SYSLOG_UNIXAF support) -AC_ARG_ENABLE([unix], - [AS_HELP_STRING([--disable-unix], [Disable support for unix])], - [ - if test "x${enableval}" = "xyes"; then - AC_MSG_RESULT([yes]) - AC_DEFINE([SYSLOG_UNIXAF], [1], [Description]) - else - AC_MSG_RESULT([no]) - fi - ], - [ - # enabled by default - AC_MSG_RESULT([yes]) - AC_DEFINE([SYSLOG_UNIXAF], [1], [Description]) - ]) +# kmsg +AC_ARG_ENABLE(kmsg, + [AS_HELP_STRING([--enable-kmsg],[Kmsg structured kernel logs functionality @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_kmsg="yes" ;; + no) enable_kmsg="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-kmsg) ;; + esac], + [enable_kmsg="no"] +) +AM_CONDITIONAL(ENABLE_IMKMSG, test x$enable_kmsg = xyes) # inet @@ -689,22 +663,6 @@ AM_CONDITIONAL(ENABLE_OMLIBDBI, test x$enable_libdbi = xyes) AC_SUBST(LIBDBI_CFLAGS) AC_SUBST(LIBDBI_LIBS) -# libuuid support -AC_CHECK_HEADERS( - [uuid/uuid.h],, - [AC_MSG_FAILURE([libuuid is missing])] -) -AC_CHECK_LIB( - [uuid], - [uuid_generate], - [LIBUUID_CFLAGS="" - LIBUUID_LIBS="-luuid" - ], - [AC_MSG_FAILURE([libuuid library is missing])] -) -AC_SUBST(LIBUUID_CFLAGS) -AC_SUBST(LIBUUID_LIBS) - # SNMP support AC_ARG_ENABLE(snmp, [AS_HELP_STRING([--enable-snmp],[Enable SNMP support @<:@default=no@:>@])], @@ -734,6 +692,23 @@ AC_SUBST(SNMP_CFLAGS) AC_SUBST(SNMP_LIBS) +# uuid support +AC_ARG_ENABLE(uuid, + [AS_HELP_STRING([--enable-uuid],[Enable support for uuid generation @<:@default=yes@:>@])], + [case "${enableval}" in + yes) enable_uuid="yes" ;; + no) enable_uuid="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-uuid) ;; + esac], + [enable_uuid=yes] +) +if test "x$enable_uuid" = "xyes"; then + PKG_CHECK_MODULES([LIBUUID], [uuid]) + AC_DEFINE(USE_LIBUUID, 1, [Define if you want to enable libuuid support]) +fi +AM_CONDITIONAL(ENABLE_UUID, test x$enable_uuid = xyes) + + # elasticsearch support AC_ARG_ENABLE(elasticsearch, [AS_HELP_STRING([--enable-elasticsearch],[Enable elasticsearch output module @<:@default=no@:>@])], @@ -745,27 +720,10 @@ AC_ARG_ENABLE(elasticsearch, [enable_elasticsearch=no] ) if test "x$enable_elasticsearch" = "xyes"; then - AC_CHECK_PROG( - [HAVE_CURL_CONFIG], - [curl-config], - [yes],,, - ) - if test "x${HAVE_CURL_CONFIG}" != "xyes"; then - AC_MSG_FAILURE([curl-config not found in PATH]) - fi - AC_CHECK_LIB( - [curl], - [curl_global_init], - [CURL_CFLAGS="`curl-config --cflags`" - CURL_LIBS="`curl-config --libs`" - ], - [AC_MSG_FAILURE([curl library is missing])], - [`curl-config --libs --cflags`] - ) + PKG_CHECK_MODULES([CURL], [libcurl]) + LT_LIB_M fi AM_CONDITIONAL(ENABLE_ELASTICSEARCH, test x$enable_elasticsearch = xyes) -AC_SUBST(CURL_CFLAGS) -AC_SUBST(CURL_LIBS) # GnuTLS support @@ -783,8 +741,6 @@ if test "x$enable_gnutls" = "xyes"; then AC_DEFINE([ENABLE_GNUTLS], [1], [Indicator that GnuTLS is present]) fi AM_CONDITIONAL(ENABLE_GNUTLS, test x$enable_gnutls = xyes) -AC_SUBST(GNUTLS_CFLAGS) -AC_SUBST(GNUTLS_LIBS) # support for building the rsyslogd runtime @@ -800,7 +756,6 @@ AC_ARG_ENABLE(rsyslogrt, if test "x$enable_rsyslogrt" = "xyes"; then RSRT_CFLAGS1="-I\$(top_srcdir)/runtime -I\$(top_srcdir) -I\$(top_srcdir)/grammar" RSRT_LIBS1="\$(top_builddir)/runtime/librsyslog.la" - #??CNF_LIBS="\$(top_builddir)/grammar/libgrammar.la" fi AM_CONDITIONAL(ENABLE_RSYSLOGRT, test x$enable_rsyslogrt = xyes) RSRT_CFLAGS="\$(RSRT_CFLAGS1) \$(LIBESTR_CFLAGS) \$(JSON_C_FLAGS)" @@ -902,8 +857,6 @@ if test "x$enable_mmnormalize" = "xyes"; then PKG_CHECK_MODULES(LIBLOGNORM, lognorm >= 0.3.1) fi AM_CONDITIONAL(ENABLE_MMNORMALIZE, test x$enable_mmnormalize = xyes) -AC_SUBST(LOGNORM_CFLAGS) -AC_SUBST(LOGNORM_LIBS) # mmnjsonparse @@ -922,7 +875,6 @@ fi AM_CONDITIONAL(ENABLE_MMJSONPARSE, test x$enable_mmjsonparse = xyes) - # mmaudit AC_ARG_ENABLE(mmaudit, [AS_HELP_STRING([--enable-mmaudit],[Enable building mmaudit support @<:@default=no@:>@])], @@ -939,6 +891,19 @@ fi AM_CONDITIONAL(ENABLE_MMAUDIT, test x$enable_mmaudit = xyes) +# mmanon +AC_ARG_ENABLE(mmanon, + [AS_HELP_STRING([--enable-mmanon],[Enable building mmanon support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmanon="yes" ;; + no) enable_mmanon="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmanon) ;; + esac], + [enable_mmanon=no] +) +AM_CONDITIONAL(ENABLE_MMANON, test x$enable_mmanon = xyes) + + # RELP support AC_ARG_ENABLE(relp, [AS_HELP_STRING([--enable-relp],[Enable RELP support @<:@default=no@:>@])], @@ -953,8 +918,6 @@ if test "x$enable_relp" = "xyes"; then PKG_CHECK_MODULES(RELP, relp >= 1.0.1) fi AM_CONDITIONAL(ENABLE_RELP, test x$enable_relp = xyes) -AC_SUBST(RELP_CFLAGS) -AC_SUBST(RELP_LIBS) # RFC 3195 support @@ -971,8 +934,6 @@ if test "x$enable_rfc3195" = "xyes"; then PKG_CHECK_MODULES(LIBLOGGING, liblogging >= 0.7.1) fi AM_CONDITIONAL(ENABLE_RFC3195, test x$enable_rfc3195 = xyes) -AC_SUBST(LIBLOGGING_CFLAGS) -AC_SUBST(LIBLOGGING_LIBS) # enable/disable the testbench (e.g. because some important parts @@ -1108,6 +1069,21 @@ AC_ARG_ENABLE(omstdout, ) AM_CONDITIONAL(ENABLE_OMSTDOUT, test x$enable_omstdout = xyes) +# settings for omjournal +AC_ARG_ENABLE(omjournal, + [AS_HELP_STRING([--enable-omjournal],[Compiles omjournal @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_omjournal="yes" ;; + no) enable_omjournal="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-omjournal) ;; + esac], + [enable_omjournal=no] +) +if test "x$enable_omjournal" = "xyes"; then + PKG_CHECK_MODULES([LIBSYSTEMD_JOURNAL], [libsystemd-journal >= 197]) +fi +AM_CONDITIONAL(ENABLE_OMJOURNAL, test x$enable_omjournal = xyes) + # settings for pmlastmsg AC_ARG_ENABLE(pmlastmsg, @@ -1176,7 +1152,7 @@ AM_CONDITIONAL(ENABLE_PMRFC3164SD, test x$enable_pmrfc3164sd = xyes) # settings for omruleset AC_ARG_ENABLE(omruleset, - [AS_HELP_STRING([--enable-omruleset],[Compiles ruleset forwarding module @<:@default=yes@:>@])], + [AS_HELP_STRING([--enable-omruleset],[Compiles ruleset forwarding module @<:@default=no@:>@])], [case "${enableval}" in yes) enable_omruleset="yes" ;; no) enable_omruleset="no" ;; @@ -1205,9 +1181,6 @@ fi AM_CONDITIONAL(ENABLE_GUI, test x$enable_gui = xyes) -AC_SUBST(RELP_CFLAGS) -AC_SUBST(RELP_LIBS) - # settings for omuxsock AC_ARG_ENABLE(omuxsock, [AS_HELP_STRING([--enable-omuxsock],[Compiles omuxsock module @<:@default=no@:>@])], @@ -1278,8 +1251,6 @@ AC_ARG_ENABLE(ommongodb, ) if test "x$enable_ommongodb" = "xyes"; then PKG_CHECK_MODULES(LIBMONGO_CLIENT, libmongo-client >= 0.1.4) - AC_SUBST(LIBMONGO_CLIENT_CFLAGS) - AC_SUBST(LIBMONGO_CLIENT_LIBS) fi AM_CONDITIONAL(ENABLE_OMMONGODB, test x$enable_ommongodb = xyes) # end of mongodb code @@ -1296,8 +1267,6 @@ AC_ARG_ENABLE(imzmq3, ) if test "x$enable_imzmq3" = "xyes"; then PKG_CHECK_MODULES(CZMQ, libczmq >= 1.1.0) - AC_SUBST(CZMQ_CFLAGS) - AC_SUBST(CZMQ_LIBS) fi AM_CONDITIONAL(ENABLE_IMZMQ3, test x$enable_imzmq3 = xyes) @@ -1315,8 +1284,6 @@ AC_ARG_ENABLE(omzmq3, ) if test "x$enable_omzmq3" = "xyes"; then PKG_CHECK_MODULES(CZMQ, libczmq >= 1.1.0) - AC_SUBST(CZMQ_CFLAGS) - AC_SUBST(CZMQ_LIBS) fi AM_CONDITIONAL(ENABLE_OMZMQ3, test x$enable_omzmq3 = xyes) @@ -1336,8 +1303,6 @@ AC_ARG_ENABLE(omhiredis, # if test "x$enable_omhiredis" = "xyes"; then PKG_CHECK_MODULES(HIREDIS, hiredis >= 0.10.1) - AC_SUBST(HIREDIS_CFLAGS) - AC_SUBST(HIREDIS_LIBS) fi AM_CONDITIONAL(ENABLE_OMHIREDIS, test x$enable_omhiredis = xyes) @@ -1345,6 +1310,7 @@ AM_CONDITIONAL(ENABLE_OMHIREDIS, test x$enable_omhiredis = xyes) AC_CONFIG_FILES([Makefile \ runtime/Makefile \ + compat/Makefile \ grammar/Makefile \ tools/Makefile \ doc/Makefile \ @@ -1355,9 +1321,11 @@ AC_CONFIG_FILES([Makefile \ plugins/imuxsock/Makefile \ plugins/immark/Makefile \ plugins/imklog/Makefile \ + plugins/imkmsg/Makefile \ plugins/omhdfs/Makefile \ plugins/omprog/Makefile \ plugins/omstdout/Makefile \ + plugins/omjournal/Makefile \ plugins/pmrfc3164sd/Makefile \ plugins/pmlastmsg/Makefile \ plugins/pmcisconames/Makefile \ @@ -1372,7 +1340,7 @@ AC_CONFIG_FILES([Makefile \ plugins/impstats/Makefile \ plugins/imrelp/Makefile \ plugins/imdiag/Makefile \ - plugins/imzmq3/Makefile \ + plugins/imzmq3/Makefile \ plugins/omtesting/Makefile \ plugins/omgssapi/Makefile \ plugins/ommysql/Makefile \ @@ -1384,11 +1352,12 @@ AC_CONFIG_FILES([Makefile \ plugins/omoracle/Makefile \ plugins/omudpspoof/Makefile \ plugins/ommongodb/Makefile \ - plugins/omhiredis/Makefile \ - plugins/omzmq3/Makefile \ + plugins/omhiredis/Makefile \ + plugins/omzmq3/Makefile \ plugins/mmnormalize/Makefile \ plugins/mmjsonparse/Makefile \ plugins/mmaudit/Makefile \ + plugins/mmanon/Makefile \ plugins/omelasticsearch/Makefile \ plugins/sm_cust_bindcdr/Makefile \ plugins/mmsnmptrapd/Makefile \ @@ -1407,9 +1376,12 @@ echo " rsyslog runtime will be built: $enable_rsyslogrt" echo " rsyslogd will be built: $enable_rsyslogd" echo " GUI components will be built: $enable_gui" echo " Unlimited select() support enabled: $enable_unlimited_select" +echo " uuid support enabled: $enable_uuid" +echo " anonymization support enabled: $enable_mmanon" echo echo "---{ input plugins }---" echo " Klog functionality enabled: $enable_klog ($os_type)" +echo " /dev/kmsg functionality enabled: $enable_kmsg" echo " plain tcp input module enabled: $enable_imptcp" echo " threaded plain tcp input module enabled: $enable_imttcp" echo " imdiag enabled: $enable_imdiag" @@ -1422,6 +1394,7 @@ echo "---{ output plugins }---" echo " Mail support enabled: $enable_mail" echo " omprog module will be compiled: $enable_omprog" echo " omstdout module will be compiled: $enable_omstdout" +echo " omjournal module will be compiled: $enable_omjournal" echo " omhdfs module will be compiled: $enable_omhdfs" echo " omelasticsearch module will be compiled: $enable_elasticsearch" echo " omruleset module will be compiled: $enable_omruleset" @@ -27,26 +27,18 @@ #ifndef DIRTY_H_INCLUDED #define DIRTY_H_INCLUDED 1 -rsRetVal multiSubmitMsg(multi_submit_t *pMultiSub); -rsRetVal submitMsg(msg_t *pMsg); +rsRetVal __attribute__((deprecated)) multiSubmitMsg(multi_submit_t *pMultiSub); +rsRetVal multiSubmitMsg2(multi_submit_t *pMultiSub); /* friends only! */ +rsRetVal submitMsg2(msg_t *pMsg); +rsRetVal __attribute__((deprecated)) submitMsg(msg_t *pMsg); +rsRetVal multiSubmitFlush(multi_submit_t *pMultiSub); rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags); -rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime, ruleset_t *pRuleset); +rsRetVal __attribute__((deprecated)) parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime, ruleset_t *pRuleset); rsRetVal diagGetMainMsgQSize(int *piSize); /* for imdiag */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName); +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams); -/* Intervals at which we flush out "message repeated" messages, - * in seconds after previous message is logged. After each flush, - * we move to the next interval until we reach the largest. - * TODO: move this to action object! Only action.c and syslogd.c use it. - */ extern int MarkInterval; -extern int repeatinterval[2]; extern qqueue_t *pMsgQueue; /* the main message queue */ extern int iConfigVerify; /* is this just a config verify run? */ extern int bHaveMainQueue; -#define MAXREPEAT ((int)((sizeof(repeatinterval) / sizeof(repeatinterval[0])) - 1)) -#define REPEATTIME(f) ((f)->f_time + repeatinterval[(f)->f_repeatcount]) -#define BACKOFF(f) { if (++(f)->f_repeatcount > MAXREPEAT) \ - (f)->f_repeatcount = MAXREPEAT; \ - } #endif /* #ifndef DIRTY_H_INCLUDED */ diff --git a/doc/Makefile.am b/doc/Makefile.am index 1ae1c68d..de2e1df5 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -34,6 +34,8 @@ html_files = \ ompipe.html \ omfwd.html \ omfile.html \ + omjournal.html \ + mmanon.html \ omusrmsg.html \ omstdout.html \ omudpspoof.html \ @@ -70,6 +72,7 @@ html_files = \ tls_cert_client.html \ tls_cert_scenario.html \ rainerscript.html \ + lookup_tables.html \ rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ @@ -115,13 +118,13 @@ html_files = \ gssapi.html \ licensing.html \ mmnormalize.html \ + mmjsonparse.html \ ommail.html \ omuxsock.html \ omrelp.html \ syslog_parsing.html \ troubleshoot.html \ rsyslog_conf_actions.html \ - rsyslog_conf_examples.html \ rsyslog_conf_filter.html \ rsyslog_conf_global.html \ rsyslog_conf_modules.html \ @@ -130,6 +133,7 @@ html_files = \ rsyslog_conf_nomatch.html \ queues_analogy.html \ multi_ruleset.html \ + multi_ruleset_legacy_format.html \ dev_oplugins.html \ free_support.html \ imudp.html \ @@ -140,11 +144,13 @@ html_files = \ rsconf1_abortonuncleanconfig.html \ rsconf1_maxopenfiles.html \ rsconf1_omfileforcechown.html \ - rsyslog_conf_file_syntax_differences.html \ - rsyslog_conf_lines.html \ rsyslog_queue_pointers.jpeg \ rsyslog_queue_pointers2.jpeg \ v6compatibility.html \ + v7compatibility.html \ + rsyslog_conf_basic_structure.html \ + rsyslog_conf_sysklogd_compatibility.html \ + imkmsg.html \ src/classes.dia grfx_files = \ diff --git a/doc/confsamples/normalization.conf b/doc/confsamples/normalization.conf new file mode 100644 index 00000000..7cfd92ef --- /dev/null +++ b/doc/confsamples/normalization.conf @@ -0,0 +1,187 @@ +# this is a config sample for log normalization, but can +# be used as a more complex general sample. +# It is based on a plain standard rsyslog.conf for Red Hat systems. +# +# NOTE: Absolute path names for modules are used in this config +# so that we can run a different rsyslog version alongside the +# regular system-installed rsyslogd. Remove these path names +# for production environment. + +#### MODULES #### + +# we do not run imuxsock as we don't want to mess with the main system logger +#module(load="/home/rger/proj/rsyslog/plugins/imuxsock/.libs/imuxsock") # provides support for local system logging (e.g. via logger command) +#module(load="imklog") # provides kernel logging support (previously done by rklogd) +module(load="/home/rger/proj/rsyslog/plugins/imudp/.libs/imudp") # Provides UDP syslog reception +module(load="/home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp") +module(load="/home/rger/proj/rsyslog/plugins/mmjsonparse/.libs/mmjsonparse") +module(load="/home/rger/proj/rsyslog/plugins/mmnormalize/.libs/mmnormalize") + +/* We assume to have all TCP logging (for simplicity) + * Note that we use different ports to point different sources + * to the right rule sets for normalization. While there are + * other methods (e.g. based on tag or source), using multiple + * ports is both the easiest as well as the fastest. + */ +input(type="imtcp" port="13514" Ruleset="WindowsRsyslog") +input(type="imtcp" port="13515" Ruleset="LinuxPlainText") +input(type="imtcp" port="13516" Ruleset="WindowsSnare") + +#debug: +action(type="omfile" file="/home/rger/proj/rsyslog/logfile") + +/* This ruleset handles structured logging. + * It is the only one ever called for remote machines + * but executed in addition to the standard action for + * the local machine. The ultimate goal is to forward + * to some Vendor's analysis tool (which digests a + * structured log format, here we use Lumberjack). + */ +template(name="lumberjack" type="string" string="%$!all-json%\n") + + +/* the rsyslog Windows Agent uses native Lumberjack format + * (better said: is configured to use it) + */ +ruleset(name="WindowsRsyslog") { + action(type="mmjsonparse") + if $parsesuccess == "OK" then { + if $!id == 4634 then + set $!usr!type = "logoff"; + else if $!id == 4624 then + set $!usr!type = "logon"; + set $!usr!rcvdfrom = $!source; + set $!usr!rcvdat = $timereported; + set $!usr!user = $!TargetDomainName & "\\" & $!TargetUserName; + call outwriter + } +} + +/* This handles clumsy snare format. Note that "#011" are + * the escape sequences for tab chars used by snare. + */ +ruleset(name="WindowsSnare") { + set $!usr!type = field($rawmsg, "#011", 6); + if $!usr!type == 4634 then { + set $!usr!type = "logoff"; + set $!doProces = 1; + } else if $!usr!type == 4624 then { + set $!usr!type = "logon"; + set $!doProces = 1; + } else + set $!doProces = 0; + if $!doProces == 1 then { + set $!usr!rcvdfrom = field($rawmsg, 32, 4); + set $!usr!rcvdat = field($rawmsg, "#011", 5); + /* we need to fix up the snare date */ + set $!usr!rcvdat = field($!usr!rcvdat, 32, 2) & " " & + field($!usr!rcvdat, 32, 3) & " " & + field($!usr!rcvdat, 32, 4); + set $!usr!user = field($rawmsg, "#011", 8); + call outwriter + } +} + +/* plain Linux log messages (here: ssh and sudo) need to be + * parsed - we use mmnormalize for fast and efficient parsing + * here. + */ +ruleset(name="LinuxPlainText") { + action(type="mmnormalize" + rulebase="/home/rger/proj/rsyslog/linux.rb" userawmsg="on") + if $parsesuccess == "OK" and $!user != "" then { + if $!type == "opened" then + set $!usr!type = "logon"; + else if $!type == "closed" then + set $!usr!type = "logoff"; + set $!usr!rcvdfrom = $!rcvdfrom; + set $!usr!rcvdat = $!rcvdat; + set $!usr!user = $!user; + call outwriter + } +} + +/* with CSV, we the reader must receive information on the + * field names via some other method (e.g. tool configuration, + * prepending of a header to the written CSV-file). All of + * this is highly dependant on the actual CSV dialect needed. + * Below, we cover the basics. + */ +template(name="csv" type="list") { + property(name="$!usr!rcvdat" format="csv") + constant(value=",") + property(name="$!usr!rcvdfrom" format="csv") + constant(value=",") + property(name="$!usr!user" format="csv") + constant(value=",") + property(name="$!usr!type" format="csv") + constant(value="\n") +} + +/* template for Lumberjack-style logging. Note that the extra + * LF at the end is just for wrinting it to file - it MUST NOT + * be included for messages intended to be sent to a remote system. + * For the latter use case, the syslog header must also be prepended, + * something we have also not done for simplicity (as we write to files). + * Note that we use a JSON-shortcut: If a tree name is specified, JSON + * for its whole subtree is generated. Thus, we only need to specify the + * $!usr top node to get everytihing we need. + */ +template(name="cee" type="string" string="@cee: %$!usr%\n") + + +/* this ruleset simulates forwarding to the final destination */ +ruleset(name="outwriter"){ + action(type="omfile" + file="/home/rger/proj/rsyslog/logfile.csv" template="csv") + action(type="omfile" + file="/home/rger/proj/rsyslog/logfile.cee" template="cee") +} + + +/* below is just the usual "uninteresting" stuff... + * Note that this goes into the default rule set. So + * local logging is handled "as usual" without the need + * for any extra effort. + */ + + +#### GLOBAL DIRECTIVES #### + +# Use default timestamp format +$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + +# Include all config files in /etc/rsyslog.d/ +# commented out not to interfere with the system rsyslogd +# (just for this test configuration!) +#$IncludeConfig /etc/rsyslog.d/*.conf + + +#### RULES #### + +# Log all kernel messages to the console. +# Logging much else clutters up the screen. +#kern.* /dev/console + +# Log anything (except mail) of level info or higher. +# Don't log private authentication messages! +*.info;mail.none;authpriv.none;cron.none /var/log/messages + +# The authpriv file has restricted access. +authpriv.* /var/log/secure + +# Log all the mail messages in one place. +mail.* /var/log/maillog + + +# Log cron stuff +cron.* /var/log/cron + +# Everybody gets emergency messages +*.emerg :omusrmsg:* + +# Save news errors of level crit and higher in a special file. +uucp,news.crit /var/log/spooler + +# Save boot messages also to boot.log +local7.* /var/log/boot.log diff --git a/doc/debug.html b/doc/debug.html index 6aeb7975..996bf5c8 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -49,6 +49,11 @@ FileTrace=vm.c FileTrace=expr.c"</li> <li><b>Debug</b> - if present, turns on the debug system and enables debug output <li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>OutputTidToStderr</b> - if present, makes rsyslog output information about +the thread id (tid) of newly create processesto stderr. Note that not necessarily +all new threads are reported (depends on the code, e.g. of plugins). This is +only available under Linux. This usually does NOT work when privileges have +been dropped (that's not a bug, but the way it is). <li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> </ul> </ul> @@ -70,6 +75,26 @@ rsyslog core, we get a number of data structures wrong. <p>For these reasons, we utilize environment variables to initialize and configure the debugging system. We understand this may be somewhat painful, but now you know there are at least some good reasons for doing so. +<p>HOWEVER, if you have a too hard time to set debug instructions using the environment +variables, there is a cure, described in the next paragraph. + +<h2>Enabling Debug via rsyslog.conf</h2> +<p>As described in the previous paragraph, enabling debug via rsyslog.conf +may not be perfect for some debugging needs, but basic debug output will work - and +that is what most often is requried. There are limited options available, but these +cover the most important use cases. +<p>Debug processing is done via legacy config statements. There currently +is no plan to move these over to the v6+ config system. Availabe settings are +<ul> +<li>$DebugFile <filename> - sets the debug file name +<li>$DebugLevel <0|1|2> - sets the respective debug level, where +0 means debug off, 1 is debug on demand activated (but debug mode off) +and 2 is full debug mode. +</ul> +<p>Note that in theory it is forbidden to specify these parameters more +than once. However, we do not enforce that and if it happens results +are undefined. + <h2>Getting debug information from a running Instance</h2> <p>It is possible to obtain debugging information from a running instance, but this requires some setup. We assume that the instance runs in the background, so debug output to @@ -138,7 +163,7 @@ instance of rsyslogd can be aborted by pressing ctl-c. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/imfile.html b/doc/imfile.html index 1594cdce..0997e382 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -91,7 +91,6 @@ textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for "info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is "notice".</li> <li><b>PersistStateInterval</b> [lines]</b><br> -Available in 4.7.3+, 5.6.2+<br> Specifies how often the state file shall be written when processing the input file. The default value is 0, which means a new state file is only written when the monitored files is being closed (end of rsyslogd execution). Any other @@ -101,9 +100,11 @@ to fatal errors (like power fail). Note that this setting affects imfile performance, especially when set to a low value. Frequently writing the state file is very time consuming. <li><b>ReadMode</b> [mode]</b><br> -Available in 5.7.5+ -<li><b>MaxLinesAtOnce</b> [number]</b><br> -Available in 5.9.0+ +This mode should defined when having multiline messages. The value can range from 0-2 and determines the multiline detection method. +<br>0 (default) - line based (Each line is a new message) +<br>1 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it) +<br>2 - paragraph (There is a blank line between log messages) +<li><b>MaxLinesAtOnce</b> [number]</b> <br> This is useful if multiple files need to be monitored. If set to 0, each file will be fully processed and then processing switches to the next file @@ -112,16 +113,14 @@ will be fully processed and then processing switches to the next file switched. This provides a kind of mutiplexing the load of multiple files and probably leads to a more natural distribution of events when multiple busy files are monitored. The default is 1024. -<li><b>MaxSubmitAtOnce</b> [number]</b><br> -Available in 5.9.0+ +<li><b>MaxSubmitAtOnce</b> [number]</b> <br> This is an expert option. It can be used to set the maximum input batch size that imfile can generate. The default is 1024, which is suitable for a wide range of applications. Be sure to understand rsyslog message batch processing before you modify this option. If you do not know what this doc here talks about, this is a good indication that you should NOT modify the default. -<li><b>Ruleset</b> <ruleset><br> -Available in 5.7.5+, 6.1.5+ +<li><b>Ruleset</b> <ruleset> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> @@ -181,12 +180,16 @@ directive, no file monitoring will take place.</li> seconds</span><br> equivalent to: PollingInterva</li> <li><b>$InputFilePersistStateInterval</b> [lines]</b><br> +Available in 4.7.3+, 5.6.2+<br> equivalent to: PersistStateInterval <li><b>$InputFileReadMode</b> [mode]</b><br> +Available in 5.7.5+<br> equivalent to: ReadMode <li><b>$InputFileMaxLinesAtOnce</b> [number]</b><br> +Available in 5.9.0+<br> equivalent to: MaxLinesAtOnce <li>$InputFileBindRuleset <ruleset><br> +Available in 5.7.5+, 6.1.5+<br> equivalent to: Ruleset </li> </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imklog.html b/doc/imklog.html index 05292ddf..1f195b16 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -15,7 +15,10 @@ syslog engine.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><strong>$KLogInternalMsgFacility +<li><strong>LogPath</strong><br> +The path to the Kernel log. This value should only be changed if you really know what +you are doing.</li> +<li><strong>InternalMsgFacility <facility></strong><br> The facility which messages internally generated by imklog will have. imklog generates some messages of itself (e.g. on problems, startup and @@ -26,13 +29,54 @@ need to specify this configuratin directive - it is included primarily for few limited cases where it is needed for good reason. Bottom line: if you don't have a good idea why you should use this setting, do not touch it.</li> -<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility -[on/<span style="font-style: italic;">off</span>]<br> -</span>At least under BSD the kernel log may contain entries +<li><b>PermitNonKernelFacility [on/<i>off</i>]</b><br> +At least under BSD the kernel log may contain entries with non-kernel facilities. This setting controls how those are handled. The default is "off", in which case these messages are ignored. Switch it to on to submit non-kernel messages to rsyslog -processing.<span style="font-weight: bold;"></span></li> +processing.</li> +<li><b>ParseKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled and the kernel creates a timestamp for its log messages, this timestamp will be +parsed and converted into regular message time instead to use the receive time of the kernel +message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the +clock used by the kernel to create the timestamps is not supposed to be as accurate as the +monotonic clock required to convert it. Depending on the hardware and kernel, it can result +in message time differences between kernel and system messages which occurred at same time. +<li><b>KeepKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin +of in each message rather than to remove it, when it could be parsed and converted into +local time for use as regular message time. Only used when <b>ParseKernelTimestamp</b> is on. +<li><b>ConsoleLogLevel</b> [<i>number</i>] +(former klogd -c option) -- sets the console log level. If specified, only messages with +up to the specified level are printed to the console. The default is -1, which means that +the current settings are not modified. To get this behavior, do not specify +ConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time +it is changed, the previous definition is re-set. The one activate will be that one that is +active when imklog actually starts processing. In short words: do not specify this +directive more than once! +</ul> +<b>Caveats/Known Bugs:</b> +<p>This is obviously platform specific and requires platform +drivers. +Currently, imklog functionality is available on Linux and BSD.</p> +<p>This module is <b>not supported on Solaris</b> and not needed there. +For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the kernel log. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling kernel messages.<br> +</p> +<textarea rows="4" cols="60">module(load="imklog") +</textarea> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><strong>$KLogInternalMsgFacility +<facility></strong><br> +equivalent to: InternalMsgFacility</li> +<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility +[on/<span style="font-style: italic;">off</span>]<br> +equivalent to: PermitNonKernelFacility</li> <li><span style="font-weight: bold;"></span>$DebugPrintKernelSymbols [on/<b>off</b>]<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -50,14 +94,7 @@ it except if you have a very good reason. If you have one, let us know because otherwise new versions will no longer support it.<br> Linux only, ignored on other platforms (but may be specified)</li> <li><b>$klogConsoleLogLevel</b> [<i>number</i>] -(former klogd -c option) -- sets the console log level. If specified, only messages with -up to the specified level are printed to the console. The default is -1, which means that -the current settings are not modified. To get this behavior, do not specify -$klogConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time -it is changed, the previous definition is re-set. The one activate will be that one that is -active when imklog actually starts processing. In short words: do not specify this -directive more than once! -<br><b>Linux only</b>, ignored on other platforms (but may be specified)</li> +<br>equivalent to: ConsoleLogLevel</li> <li><b>$klogUseSyscallInterface</b> [on/<b>off</b>] -- former klogd -s option<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -65,40 +102,17 @@ Linux only, ignored on other platforms (but may be specified)</li> former klogd -2 option<br> Linux only, ignored on other platforms (but may be specified)<br style="font-weight: bold;"> </li> -<li><b>$klogParseKernelTimestamp</b> [on/<b>off</b>] -If enabled and the kernel creates a timestamp for its log messages, this timestamp will be -parsed and converted into regular message time instead to use the receive time of the kernel -message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the -clock used by the kernel to create the timestamps is not supposed to be as accurate as the -monotonic clock required to convert it. Depending on the hardware and kernel, it can result -in message time differences between kernel and system messages which occurred at same time. -</li> -<li><b>$klogKeepKernelTimestamp</b> [on/<b>off</b>] -If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin -of in each message rather than to remove it, when it could be parsed and converted into -local time for use as regular message time. Only used, when $klogParseKernelTimestamp is on. -</li> +<li><b>$klogParseKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: ParseKernelTimeStamp</li> +<li><b>$klogKeepKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: KeepKernelTimeStamp</li> </ul> -<b>Caveats/Known Bugs:</b> -<p>This is obviously platform specific and requires platform -drivers. -Currently, imklog functionality is available on Linux and BSD.</p> -<p>This module is <b>not supported on Solaris</b> and not needed there. -For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> -<p><b>Sample:</b></p> -<p>The following sample pulls messages from the kernel log. All -parameters are left by default, which is usually a good idea. Please -note that loading the plugin is sufficient to activate it. No directive -is needed to start pulling kernel messages.<br> -</p> -<textarea rows="15" cols="60">$ModLoad imklog -</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imkmsg.html b/doc/imkmsg.html new file mode 100644 index 00000000..23b96147 --- /dev/null +++ b/doc/imkmsg.html @@ -0,0 +1,50 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>/dev/kmsg Log Input Module (imkmsg)</title> + +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>/dev/kmsg Log Input Module</h1> +<p><b>Module Name: imkmsg</b></p> +<p><b>Authors: </b>Rainer Gerhards +<rgerhards@adiscon.com><br /> +Milan Bartos +<mbartos@redhat.com></p> +<p><b>Description</b>:</p> +<p>Reads messages from the /dev/kmsg structured kernel log and submits them to the +syslog engine.</p> +<p> +The printk log buffer constains log records. These records are exported by /dev/kmsg +device as structured data in the following format:<br /> + "level,sequnum,timestamp;<message text>\n"<br /> +There could be continuation lines starting with space that contains key/value pairs.<br /> +<br /> +Log messages are parsed as necessary into rsyslog msg_t structure. Continuation lines are parsed +as json key/value pairs and added into rsyslog's message json representation. +</p> +<p><b>Configuration Directives</b>:</p> +<p>This module has no configuration directives.</p> +<b>Caveats/Known Bugs:</b> +<p>This module can't be used together with imklog module. When using one of them, make sure the other +one is not enabled.</p> +<p>This is Linux specific module and requires /dev/kmsg device with structured kernel logs.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the /dev/kmsg log device. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling messages.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imkmsg +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/impstats.html b/doc/impstats.html index 64b04a30..8db9c6f6 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -16,26 +16,66 @@ availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version). <p>The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). -Be sure that your configuration records the counter messages (default is syslog.info). +Be sure that your configuration records the counter messages (default is syslog.=info). +Besides logging to the regular syslog stream, the module can also be configured to +write statistics data into a (local) file. <p>Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be noticable (for high-load environments). <p>The rsyslog website has an updated overview of available <a href="http://rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a>. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Module Confguration Parameters</b>:</p> +<p>This module supports module parameters, only. <ul> -<li>$PStatInterval <Seconds><br> -Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the -actual interval may be a bit longer. We do not try to be precise and so the interval is -actually a sleep period which is entered after generating all messages. So the actual -interval is what is configured here plus the actual time required to generate messages. -In general, the difference should not really matter. -<li>$PStatFacility <numerical facility><br> -The numerical syslog facility code to be used for generated messages. Default -is 5 (syslog).This is useful for filtering messages.</li> -<li>$PStatSeverity <numerical severity><br> -The numerical syslog severity code to be used for generated messages. Default -is 6 (info).This is useful for filtering messages.</li> + <li><strong>interval </strong>[seconds] (default 300 [5minutes])<br> + Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the + actual interval may be a bit longer. We do not try to be precise and so the interval is + actually a sleep period which is entered after generating all messages. So the actual + interval is what is configured here plus the actual time required to generate messages. + In general, the difference should not really matter. + <br></li> + <li><strong>facility </strong>[templateName]<br> + The numerical syslog facility code to be used for generated messages. Default + is 5 (syslog). This is useful for filtering messages. + <br></li> + <li><strong>severity </strong>[templateName]<br> + The numerical syslog severity code to be used for generated messages. Default + is 6 (info).This is useful for filtering messages. + <br></li> + <li><strong>format </strong>[json/cee/<b>legacy</b>](rsyslog v6.3.8+ only)<br> + Specifies the format of emitted stats messages. The default of "legacy" is + compatible with pre v6-rsyslog. The other options provide support for + structured formats (note the "cee" is actually "project lumberack" logging). + <br></li> + <li><strong>log.syslog </strong>[<b>on</b>/off] - available since 7.3.6<br> + This is a boolean setting specifying if data should be sent + to the usual syslog stream. This is useful if custom formatting + or more elaborate processing is desired. However, output is placed + under the same restrictions as regular syslog data, especially in + regard to the queue position (stats data may sit for an extended + period of time in queues if they are full).<br></li> + <li><strong>log.file </strong>[file name] - available since 7.3.6<br> + If specified, statistics data is written the specified file. For + robustness, this should be a local file. The file format cannot be + customized, it consists of a date header, followed by a colon, + followed by the actual statistics record, all on one line. Only + very limited error handling is done, so if things go wrong stats + records will probably be lost. Logging to file an be a useful + alternative if for some reasons (e.g. full queues) the regular + syslog stream method shall not be used solely. Note that turning + on file logging does NOT turn of syslog logging. If that is desired + log.syslog="off" must be explicitely set. + <br></li> + +</ul> +<p><b>Legacx Configuration Directives</b>:</p> +A limited set of parameters can also be set via the legacy configuration +syntax. Note that this is intended as an upward compatibilit layer, so +newer features are intentionally <b>not</b> available via legacy directives. +<ul> +<li>$PStatInterval <Seconds> - same as the "interval" parameter. +<li>$PStatFacility <numerical facility> - same as the "facility" parameter. +<li>$PStatSeverity <numerical severity> - same as the "severity" parameter. <li>$PStatJSON <on/<b>off</b>> (rsyslog v6.3.8+ only)<br> If set to on, stats messages are emitted as structured cee-enhanced syslog. If set to off, legacy format is used (which is compatible with pre v6-rsyslog). @@ -45,23 +85,45 @@ set to off, legacy format is used (which is compatible with pre v6-rsyslog). <ul> <li>This module MUST be loaded right at the top of rsyslog.conf, otherwise stats may not get turned on in all places.</li> -<li>experimental code</li> </ul> -<p><b>Sample:</b></p> +<p><b>Samples:</b></p> <p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:<br> </p> -<textarea rows="8" cols="60">$ModLoad impstats +<textarea rows="5" cols="60">module(load="impstats" interval="600" severity="7") + +# to actually gather the data: +syslog.=debug /var/log/rsyslog-stats +</textarea> +<p><b>Legacy Sample:</b></p> +<p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:</p> +<textarea rows="6" cols="60">$ModLoad impstats $PStatInterval 600 $PStatSeverity 7 -syslog.debug /var/log/rsyslog-stats +syslog.=debug /var/log/rsyslog-stats </textarea> +<p>In the next sample, the default interval of 5 minutes is used. However, this time +stats data is NOT emitted to the syslog stream but to a local file instead. +<p> +<textarea rows="3" cols="70">module(load="impstats" interval="600" severity="7" + log.syslog="off" /* need to turn log stream logging off! */ + log.file="/path/to/local/stats.log") +</textarea> +<p>And finally, we log to both the regular syslog log stream as well as a file. +Within the log stream, we forward the data records to another server: +<p> +<textarea rows="4" cols="70">module(load="impstats" interval="600" severity="7" + log.file="/path/to/local/stats.log") + +syslog.=debug @central.example.net +</textarea> + <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imptcp.html b/doc/imptcp.html index d301b76f..b5bd0970 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -13,18 +13,31 @@ <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via plain TCP syslog. This is a specialised input plugin tailored for high performance on Linux. It will -probably not run on any other platform. Also, it does no provide TLS services. +probably not run on any other platform. Also, it does not provide TLS services. Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a>. <p>This module has no limit on the number of listeners and sessions that can be used. -<p>Multiple receivers may be configured by -specifying $InputPTCPServerRun multiple times. </p> <p><b>Configuration Directives</b>:</p> <p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> -<li><b>AddTLFrameDelimiter</b> <Delimiter><br> + +<p><b>Module Parameters</b>:</p> +<p>These paramters can be used with the "module()" statement. They apply +globaly to all inputs defined by the module. +<ul> +<li>Threads <number><br> +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +helper threads (but not more than there are CPUs/Cores) can help improving +performance. The default value is two. +</ul> +<p><b>Input Parameters</b>:</p> +<p>These parameters can be used with the "input()" statement. They apply to the +input they are specified with. +<ul> +<li><b>AddtlFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -78,13 +91,15 @@ name is not strictly necessary, but can be useful to apply filtering based on wh the message was received from. <li><b>Ruleset</b> <name><br> Binds specified ruleset to next server defined. -<!--<li>$InputPTCPHelperThreads <number><br> -Number of helper worker threads to process incoming messages. These -threads are utilized to pull data off the network. On a busy system, additional -helper threads (but not more than there are CPUs/Cores) can help improving -performance. The default value is two.--> <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> @@ -93,13 +108,11 @@ On multi-homed machines, specifies to which local address the listerner should b <p><b>Sample:</b></p> <p>This sets up a TCP server on port 514:<br> </p> -<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imptcp/.libs/imptcp") # needs to be done just once +<textarea rows="4" cols="60">module(load="/folder/to/rsyslog/plugins/imptcp/.libs/imptcp") # needs to be done just once input(type="imptcp" port="514") </textarea> <p><b>Legacy Configuration Directives</b>:</p> -<p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in -their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> <li>$InputPTCPServerAddtlFrameDelimiter <Delimiter><br> Equivalent to: AddTLFrameDelimiter</li> @@ -137,7 +150,7 @@ Equivalent to: Address </li> <p><b>Sample:</b></p> <p>This sets up a TCP server on port 514:<br> </p> -<textarea rows="15" cols="60">$ModLoad imptcp # +<textarea rows="3" cols="60">$ModLoad imptcp # needs to be done just once $InputPTCPServerRun 514 </textarea> @@ -146,7 +159,7 @@ $InputPTCPServerRun 514 <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2010-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imrelp.html b/doc/imrelp.html index 80ddfd53..856aff82 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -30,7 +30,7 @@ Clients send messages to the RELP server via omrelp.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><b>Ruleset</b> <name> (available in 6.3.6+)</br> +<li><b>Ruleset</b> <name></br> Binds the specified ruleset to all RELP listeners. <li><b>Port</b> <port><br> Starts a RELP server on selected port</li> diff --git a/doc/imtcp.html b/doc/imtcp.html index 649b08f8..b9f0b056 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -17,13 +17,11 @@ Encryption is natively provided by selecting the approprioate network stream driver and can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> (an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> -<p>Multiple receivers may be configured by specifying -$InputTCPServerRun multiple times. This is available since version 4.3.1, earlier -versions do NOT support it. -</p> + <p><b>Configuration Directives</b>:</p> +<p><b>Global Directives</b>:</p> <ul> -<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +<li><b>AddtlFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -43,7 +41,7 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. -<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +<li><b>DisableLFDelimiter</b> <on/<b>off</b>><br> Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames. However, some users brought up the case that it may be useful to define a different delimiter and totally disable LF as a delimiter (the use case named were multi-line messages). This mode @@ -51,16 +49,14 @@ is non-standard and will probably come with a lot of problems. However, as there for it and it is relatively easy to support, we do so. Be sure to turn this setting to "on" only if you exactly know what you are doing. You may run into all sorts of troubles, so be prepared to wrangle with that! -<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +<li><b>NotifyOnConnectionClose</b> [on/<b>off</b>]<br> instructs imtcp to emit a message if the remote peer closes a connection.<br> <b>Important:</b> This directive is global to all listeners and must be given right after loading imtcp, otherwise it may have no effect.</li> -<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +<li><b>KeepAlive</b> <on/<b>off</b>><br> enable of disable keep-alive packets at the tcp socket layer. The default is to disable them.</li> -<li><b>$InputTCPServerRun</b> <port><br> -Starts a TCP server on selected port</li> -<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +<li><b>FlowControl</b> <<b>on</b>/off><br> This setting specifies whether some message flow control shall be exercised on the related TCP input. If set to on, messages are handled as "light delayable", which means the sender is throttled a bit when the queue becomes near-full. This is done in order @@ -69,29 +65,93 @@ may have some undesired effect in some configurations. Still, we consider this a a useful setting and thus it is the default. To turn the handling off, simply configure that explicitely. </li> -<li><b>$InputTCPMaxListeners</b> <number><br> +<li><b>MaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li><b>$InputTCPMaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> -<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +<li><b>MaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>StreamDriver.Mode</b> <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> -<li><b>$InputTCPServerInputName</b> <name><br> -Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a -name is not strictly necessary, but can be useful to apply filtering based on which input -the message was received from. -<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +<li><b>StreamDriver.AuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> -<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +<li><b>PermittedPeer</b> <id-string><br> Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected -AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> -<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeer may not be set in anonymous modes. +<br>PermittedPeer may be set either to a single peer or an array of peers either of type IP or name, depending on the tls certificate. +<br>Single peer: PermittedPeer="127.0.0.1" +<br>Array of peers: PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."]</li> +</ul> +<p><b>Action Directives</b>:</p> +<ul> +<li><b>Port</b> <port><br> +Starts a TCP server on selected port</li> +<li><b>Name</b> <name><br> +Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a +name is not strictly necessary, but can be useful to apply filtering based on which input +the message was received from. +<li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> -<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +<li><b>SupportOctetCountedFraming</b> <<b>on</b>|off><br> If set to "on", the legacy octed-counted framing (similar to RFC5425 framing) is activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> +(which includes the functionality of imtcp)</li> +</ul> +<p><b>Example:</b></p> +<p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> +</p> +<textarea rows="15" cols="60">module(load="imtcp" MaxSessions="500") +input(type="imtcp" port="514") +</textarea> +<p>Note that the global parameters (here: max sessions) need to be set when the module is loaded. Otherwise, the parameters will not apply. +</p> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +equivalent to: AddtlFrameDelimiter +<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +equivalent to: DisableLFDelimiter +<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +equivalent to: NotifyOnConnectionClose<br> +</li> +<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +equivalent to: KeepAlive</li> +<li><b>$InputTCPServerRun</b> <port><br> +equivalent to: Port</li> +<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +equivalent to: FlowControl +</li> +<li><b>$InputTCPMaxListeners</b> <number><br> +equivalent to: MaxListeners</li> +<li><b>$InputTCPMaxSessions</b> <number><br> +equivalent to: MaxSessions</li> +<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +equivalent to: StreamDriver.Mode</li> +<li><b>$InputTCPServerInputName</b> <name><br> +equivalent to: Name +<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +equivalent to: StreamDriver.AuthMode</li> +<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +equivalent to: PermittedPeer.</li> +<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +equivalent to: Ruleset</a>.</li> +<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +equivalent to: SupportOctetCountedFraming +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imudp.html b/doc/imudp.html index 3512d474..f2d04d65 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -33,18 +33,27 @@ the value, the less precise the timestamp. <li><b>SchedulingPolicy</b> <rr/fifo/other><br> Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time -processing under Linux (and thus reduce chance of packet loss). Available since 4.7.4+, 5.7.3+, 6.1.3+. +processing under Linux (and thus reduce chance of packet loss). <li><b>SchedulingPriority</b> <number><br> -Scheduling priority to use. Available since 4.7.4+, 5.7.3+, 6.1.3+. +Scheduling priority to use. </ul> <p><b>Action Directives</b>:</p> <ul> <li><b>Address</b> <IP><br> local IP address (or name) the UDP listens should bind to</li> <li><b>Port</b> <port><br> -default 514, start UDP server on this port</li> +default 514, start UDP server on this port. Either a single port can be specified or an array of ports. If multiple ports are specified, a listener will be automatically started for each port. Thus, no additional inputs need to be configured. +<br>Single port: Port="514" +<br>Array of ports: Port=["514","515","10514","..."]</li> <li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> @@ -53,7 +62,7 @@ Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> <p><b>Sample:</b></p> <p>This sets up an UPD server on port 514:<br> </p> -<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imudp/.libs/imudp") # needs to be done just once +<textarea rows="15" cols="60">module(load="imudp") # needs to be done just once input(type="imudp" port="514") </textarea> @@ -70,9 +79,9 @@ equivalent to: Port </li> equivalent to: TimeRequery <li>$InputUDPServerBindRuleset <ruleset><br> equivalent to: Ruleset </li> -<li>$IMUDPSchedulingPolicy <rr/fifo/other><br> +<li>$IMUDPSchedulingPolicy <rr/fifo/other> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPolicy -<li>$IMUDPSchedulingPriority <number><br> +<li>$IMUDPSchedulingPriority <number> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPriority </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index bd207a37..ee9e2a05 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -65,6 +65,12 @@ you must turn it on (via SysSock.Annotate and Annotate). <li><b>SysSock.IgnoreTimestamp</b> [<b>on</b>/off]<br> Ignore timestamps included in the messages, applies to messages received via the system log socket. </li> +<li><b>SysSock.IgnoreOwnMessages</b> [<b>on</b>/off] (available since 7.3.7)<br> +Ignores messages that originated from the same instance of rsyslogd. There usually +is no reason to receive messages from ourselfs. This setting is vital +when writing messages to the Linux journal. See <a href="omjournal.html">omjournal</a> +module documentation for a more in-depth description. +</li> <li><b>SysSock.Use</b> (imuxsock) [on/<b>off</b>] do NOT listen for the local log socket. This is most useful if you run multiple instances of rsyslogd where only one shall handle the system log socket. @@ -77,7 +83,7 @@ to the system log socket. <li><b>SysSock.UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0. +with earlier versions of rsyslog. </li> <li><b>SysSock.RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. @@ -98,11 +104,17 @@ properties for the system log socket.</li> <ul> <li><b>IgnoreTimestamp</b> [<b>on</b>/off] <br>Ignore timestamps included in the message. Applies to the next socket being added.</li> +<li><b>IgnoreOwnMessages</b> [<b>on</b>/off] (available since 7.3.7)<br> +Ignore messages that originated from the same instance of rsyslogd. There usually +is no reason to receive messages from ourselfs. This setting is vital +when writing messages to the Linux journal. See <a href="omjournal.html">omjournal</a> +module documentation for a more in-depth description. +</li> <li><b>FlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the next socket.</li> <li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number -of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen in 5.9.6+, +of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> @@ -112,7 +124,7 @@ burst in number of messages. Default is 200. <li><b>RateLimit.Severity</b> [numerical severity] - specifies the severity of messages that shall be rate-limited. </li> -<!--<li><b>LocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified +<!--<li><b>LocalIPIF</b> [interface name] - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. @@ -120,7 +132,7 @@ the default of "127.0.0.1" is used. <li><b>UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> +with earlier versions of rsyslog. </li> <li><b>UseSysTimeStamp</b> [<b>on</b>/off] instructs imuxsock to obtain message time from the system (via control messages) insted of using time recorded inside the message. This may be most useful in combination with systemd. Note: @@ -139,7 +151,7 @@ being reset to "off" after the Socket directive, so if you would have for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those -sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> +sockets even though their directories do not yet exist.</li> <li><b>Socket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> <li><b>HostName</b> <hostname> permits to override the hostname that shall be used inside messages taken from the <b>next</b> Socket socket. Note that @@ -195,39 +207,42 @@ SysSock.Annotate="on") <p><b>Legacy Configuration Directives</b>:</p> <ul> <li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] -<br>Please see: IgnoreTimestamp.</li> -<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - Please see: FlowControl .</li> -<li><b>$IMUXSockRateLimitInterval</b> [number] - Please see: RateLimit.Interval +<br>equivalent to: IgnoreTimestamp.</li> +<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - equivalent to: FlowControl .</li> +<li><b>$IMUXSockRateLimitInterval</b> [number] - equivalent to: RateLimit.Interval </li> -<li><b>$IMUXSockRateLimitBurst</b> [number] - Please see: RateLimit.Burst +<li><b>$IMUXSockRateLimitBurst</b> [number] - equivalent to: RateLimit.Burst </li> -<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - Please see: RateLimit.Severity +<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - equivalent to: RateLimit.Severity </li> <li><b>$IMUXSockLocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. </li> -<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - Please see: UsePIDFromSystem.</li> -<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] Please see: UseSysTimeStamp .<br> +<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: UseSysTimeStamp .<br> <li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> -Please see: SysSock.IgnoreTimestamp.</li> -<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] Please see: SysSock.Use</li> -<li><b>$SystemLogSocketName</b> <name-of-socket> Please see: SysSock.Name</li> -<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - Please see: SysSock.FlowControl.</li> -<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - Please see: SysSock.UsePIDFromSystem.</li> -<li><b>$SystemLogRateLimitInterval</b> [number] - Please see: SysSock.RateLimit.Interval. +equivalent to: SysSock.IgnoreTimestamp.</li> +<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] equivalent to: SysSock.Use</li> +<li><b>$SystemLogSocketName</b> <name-of-socket> equivalent to: SysSock.Name</li> +<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - equivalent to: SysSock.FlowControl.</li> +<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: SysSock.UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$SystemLogRateLimitInterval</b> [number] - equivalent to: SysSock.RateLimit.Interval. </li> -<li><b>$SystemLogRateLimitBurst</b> [number] - Please see: SysSock.RateLimit.Burst +<li><b>$SystemLogRateLimitBurst</b> [number] - equivalent to: SysSock.RateLimit.Burst </li> -<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - Please see: SysSock.RateLimit.Severity +<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - equivalent to: SysSock.RateLimit.Severity </li> -<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] Please see: SysSock.UseSysTimeStamp. -<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - Please see: CreatePath</li> -<li><b>$AddUnixListenSocket</b> <name-of-socket> Please see: Socket </li> -<li><b>$InputUnixListenSocketHostName</b> <hostname> Please see: HostName.</li> -<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> Please see: Annotate.</li> -<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> Please see: SysSock.Annotate.</li> +<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: SysSock.UseSysTimeStamp. +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - equivalent to: CreatePath +<br>[available since 4.7.0 and 5.3.0]</li> +<li><b>$AddUnixListenSocket</b> <name-of-socket> equivalent to: Socket </li> +<li><b>$InputUnixListenSocketHostName</b> <hostname> equivalent to: HostName.</li> +<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> equivalent to: Annotate.</li> +<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> equivalent to: SysSock.Annotate.</li> </ul> <b>Caveats/Known Bugs:</b><br> diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html new file mode 100644 index 00000000..d72810f1 --- /dev/null +++ b/doc/lookup_tables.html @@ -0,0 +1,205 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Lookup Tables</title> +</head> + +<body> +<h1>Lookup Tables</h1> + +<p><b><font color="red">NOTE: this is</font> proposed functionality, which is +<font color="red">NOT YET IMPLEMENTED</font>!</b> + +<p><b>Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices).</b> +<p>The base idea is to use a message variable as an index into a table which then +returns another value. For example, $fromhost-ip could be used as an index, with +the table value representing the type of server or the department or remote office +it is located in. A main point with lookup tables is that the lookup is very fast. +So while lookup tables can be emulated with if-elseif constructs, they are generally +much faster. Also, it is possible to reload lookup tables during rsyslog runtime without +the need for a full restart. +<p>The lookup tables itself exists in a separate configuration file (one per table). This +file is loaded on rsyslog startup and when a reload is requested. +<p>There are different types of lookup tables: +<ul> +<li><b>string</b> - the value to be looked up is an arbitrary string. Only exact +some strings match. +<li><b>array</b> - the value to be looked up is an integer number from a consequtive set. +The set does not need to start at zero or one, but there must be no number missing. So, for example +5,6,7,8,9 would be a valid set of index values, while 1,2,4,5 would not be (due to missing +2). +A match happens if the requested number is present. +<li><b>sparseArray</b> - the value to be looked up is an integer value, but there may +be gaps inside the set of values (usually there are large gaps). A typical use case would +be the matching of IPv4 address information. A match happens on the first value that is +less than or equal to the requested value. +</ul> +<p>Note that index integer numbers are represented by unsigned 32 bits. +<p>Lookup tables can be access via the lookup() built-in function. The core idea is to +set a local variable to the lookup result and later on use that local variable in templates. +<p>More details on usage now follow. +<h2>Lookup Table File Format</h2> +<p>Lookup table files contain a single JSON object. This object contains of a header and a +table part. +<h3>Header</h3> +<p>The header is the top-level json. It has paramters "version", "nomatch", and "type". +The version parameter +must be given and must always be one for this version of rsyslog. The nomatch +parameter is optional. If specified, it contains the value to be used if lookup() +is provided an index value for which no entry exists. The default for +"nomatch" is the empty string. Type specifies the type of lookup to be done. +<h3>Table</h3> +This must be an array of elements, even if only a single value exists (for obvious +reasons, we do not expect this to occur often). Each array element must contain two +fields "index" and "value". +<h3>Example</h3> +<p>This is a sample of how an ip-to-office mapping may look like: +<pre> +{ "version":1, "nomatch":"unk", "type":"string", + "table":[ {"index":"10.0.1.1", "value":"A" }, + {"index":"10.0.1.2", "value":"A" }, + {"index":"10.0.1.3", "value":"A" }, + {"index":"10.0.2.1", "value":"B" }, + {"index":"10.0.2.2", "value":"B" }, + {"index":"10.0.2.3", "value":"B" } + ] +} +</pre> +Note: if a different IP comes in, the value "unk" +is returend thanks to the nomatch parameter in +the first line. +<p> +<h2>RainerScript Statements</h2> +<h3>lookup_table() Object</h3> +<p>This statement defines and intially loads a lookup table. Its format is +as follows: +<pre> +lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (mandatory)<br> + Defines the name of lookup table for further reference + inside the configuration. Names must be unique. Note that + it is possible, though not advisible, to have different + names for the same file. + <li><b>file</b> (mandatory)<br> + Specifies the full path for the lookup table file. This file + must be readable for the user rsyslog is run under (important + when dropping privileges). It must point to a valid lookup + table file as described above. + <li><b>reloadOnHUP</b> (optional, default "on")<br> + Specifies if the table shall automatically be reloaded + as part of HUP processing. For static tables, the + default is "off" and specifying "on" triggers an + error message. Note that the default of "on" may be + somewhat suboptimal performance-wise, but probably + is what the user intuitively expects. Turn it off + if you know that you do not need the automatic + reload capability. +</ul> + +<h3>lookup() Function</h3> +<p>This function is used to actually do the table lookup. Format: +<pre> +lookup_table("name", indexvalue) +</pre> +<h4>Parameters</h4> +<ul> + <li><b>return value</b><br> + The function returns the string that is associated with the + given indexvalue. If the indexvalue is not present inside the + lookup table, the "nomatch" string is returned (or an empty string + if it is not defined). + <li><b>name</b> (constant string)<br> + The lookup table to be used. Note that this must be specificed as a + constant. In theory, variable table names could be made possible, but + their runtime behaviour is not as good as for static names, and we do + not (yet) see good use cases where dynamic table names could be useful. + <li><b>indexvalue</b> (expression)<br> + The value to be looked up. While this is an arbitrary RainerScript expression, + it's final value is always converted to a string in order to conduct + the lookup. For example, "lookup(table, 3+4)" would be exactly the same + as "lookup(table, "7")". In most cases, indexvalue will probably be + a single variable, but it could also be the result of all RainerScript-supported + expression types (like string concatenation or substring extraction). + Valid samples are "lookup(name, $fromhost-ip & $hostname)" or + "lookup(name, substr($fromhost-ip, 0, 5))" as well as of course the + usual "lookup(table, $fromhost-ip)". +</ul> + + +<h3>load_lookup_table Statement</h3> + +<p><b>Note: in the final implementation, this MAY be implemented as an action. +This is a low-level decesion that must be made during the detail development +process. Parameters and semantics will remain the same of this happens.</b> + +<p>This statement is used to reload a lookup table. It will fail if +the table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight performance +hit during the load phase. It is assume that always a triggering condition +is used to load the table. +<pre> +load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (string)<br> + The lookup table to be used. + <li><b>errOnFail</b> (boolean, default "on")<br> + Specifies whether or not an error message is to be emitted if + there are any problems reloading the lookup table. + <li><b>valueOnFail</b> (optional, string)<br> + This parameter affects processing if the lookup table cannot + be loaded for some reason: If the parameter is not present, + the previous table will be kept in use. If the parameter is + given, the previous table will no longer be used, and instead + an empty table be with nomath=valueOnFail be generated. In short, + that means when the parameter is set and the reload fails, + all matches will always return what is specified in valueOnFail. +</ul> + +<h3>Usage example</h3> +<p>For clarity, we show only those parts of rsyslog.conf that affect +lookup tables. We use the remote office example that an example lookup +table file is given above for. +<pre> +lookup_table(name="ip2office" file="/path/to/ipoffice.lu" + reloadOnHUP="off") + + +template(name="depfile" type="string" + string="/var/log/%$usr.dep%/messages") + +set $usr.dep = lookup("ip2office", $fromhost-ip); +action(type="omfile" dynfile="depfile") + +# support for reload "commands" +if $fromhost-ip == "10.0.1.123" + and $msg contains "reload office lookup table" + then + load_lookup_table(name="ip2office" errOnFail="on") +</pre> + +<p>Note: for performance reasons, it makes sense to put the reload command into +a dedicated ruleset, bound to a specific listener - which than should also +be sufficiently secured, e.g. via TLS mutual auth. + +<h2>Implementation Details</h2> +<p>The lookup table functionality is implemented via highly efficient algorithms. +The string lookup is based on a parse tree and has O(1) time complexity. The array +lookup is also O(1). In case of sparseArray, we have O(log n). +<p>To preserve space and, more important, increase cache hit performance, equal +data values are only stored once, no matter how often a lookup index points to them. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/manual.html b/doc/manual.html index 9c7c677e..05a03e29 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -13,13 +13,13 @@ It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its <a href="features.html"> advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_tls.html">encryption protected syslog</a> relay chains while at the same time being very easy to setup for the -novice user. And as we know what enterprise users really need, there is -also <a href="http://www.rsyslog.com/professional-services">professional -rsyslog support</a> available directly from the source!</p> +novice user. And as we know what enterprise users really need, there are +also <a href="http://www.rsyslog.com/professional-services"> rsyslog +professional services</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 6.6.0 (v6-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 7.3.6 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -35,12 +35,13 @@ if you upgrade from v4, read the <a href="v5compatibility.html">rsyslog v5 compatibility notes</a>, and if you upgrade from v5, read the <a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. +if you upgrade from v6, read the +<a href="v7compatibility.html">rsyslog v7 compatibility notes</a>. <p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> <p><b>Follow the links below for the</b></p> <ul> <li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> -<li><a href="http://www.rsyslog.com/doc/node1.html">rsyslog.conf, new RainerScript-based format (v6+)</a></li> <li><a href="rsyslog_conf.html">configuration file format (rsyslog.conf)</a></li> <li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> <li> <a href="property_replacer.html">property replacer, an important core component</a></li> diff --git a/doc/mmanon.html b/doc/mmanon.html new file mode 100644 index 00000000..16065a1f --- /dev/null +++ b/doc/mmanon.html @@ -0,0 +1,119 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>IP Address Anonimization Module (mmanon)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>IP Address Anonimization Module (mmanon)</h1> +<p><b>Module Name: mmanon</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.3.7</p> +<p><b>Description</b>:</p> +<p>The mmanon module permits to anonymize IP addresses. It is a message +modification module that actually changes the IP address inside the message, +so after calling mmanon, the original message can no longer be obtained. +Note that anonymization will break digital signatures on the message, if +they exist. +<p><i>How are IP-Addresses defined?</i> +<p>We assume that an IP address consists of four octets in dotted notation, +where each of the octets has a value between 0 and 255, inclusively. After +the last octet, there must be either a space or a colon. So, for example, +"1.2.3.4 Test" and "1.2.3.4:514 Test" are detected as containing valid IP +addresses, whereas this is not the case for "1.2.300.4 Test" or +"1.2.3.4-Test". The message text may contain multiple addresses. If so, +each of them is anonimized (according to the same rules). +<b>Important:</b> We may change the set of acceptable characters after +the last octet in the future, if there are good reasons to do so. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>mode</b> - default "rewrite"<br> +There exists the "simple" and "rewrite" mode. In simple mode, only octets +as whole can be anonymized and the length of the message is never changed. +This means that when the last three octets of the address 10.1.12.123 are +anonymized, the result will be 10.0.00.000. This means that the length of the +original octets is still visible and may be used to draw some privacy-evasive +conclusions. This mode is slightly faster than "overwrite" mode, and this +may matter in high throughput environments.<br> +The default "rewrite" mode will do full anonymization of any number of bits +and it will also normlize the address, so that no information about the +original IP address is available. So in the above example, 10.1.12.123 would +be anonymized to 10.0.0.0. +<li><b>ipv4.bits</b> - default 16<br> +This set the number of bits that should be anonymized (bits are from the +right, so lower bits are anonymized first). This setting permits to save +network information while still anonymizing user-specific data. The more +bits you discard, the better the anonymization obviously is. The default +of 16 bits reflects what German data privacy rules consider as being +sufficinetly anonymized. We assume, this can also be used as a rough +but conservative guideline for other countries.<br> +Note: when in simple mode, only bits on a byte boundary can be specified. +As such, any value other than 8, 16, 24 or 32 is invalid. If an invalid +value is given, it is rounded to the next byte boundary (so we favor stronger +anonymization in that case). For example, a bit value of 12 will become 16 in +simple mode (an error message is also emitted). +<li><b>replacementChar</b> - default "x"<br> +In simple mode, this sets the character +that the to-be-anonymized part of the IP address is to be overwritten +with. In rewrite mode, this parameter is <b>not permitted</b>, as in +this case we need not necessarily rewrite full octets. As such, the anonymized +part is always zero-filled and replacementChar is of no use. If it is +specified, an error message is emitted and the parameter ignored. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li><b>only IPv4</b> is supported +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we write one file without anonymization and another one +with the message anonymized. Note that once mmanon has run, access to the +original message is no longer possible (execept if stored in user +variables before anonymization). +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>This next snippet is almost identical to the first one, but +here we anonymize the full IPv4 address. Note that by +modifying the number of bits, you can anonymize different parts +of the address. Keep in mind that in simple mode (used here), the bit values +must match IP address bytes, so for IPv4 only the values 8, 16, 24 and +32 are valid. Also, in this example the replacement is done +via asterisks instead of lower-case "x"-letters. Also keep in mind that +"replacementChar" can only be set in simple mode. +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon" ipv4.bits="32" mode="simple" replacementChar="*") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>The next snippet is also based on the first one, but anonimzes an +"odd" number of bits, 12. The value of 12 is used by some folks as a +compromise between keeping privacy and still permiting to gain some +more in-depth insight from log files. Note that anonymizing 12 bits +may be insufficient to fulfill legal requirements (if such exist). +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon" ipv4.bits="12") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmjsonparse.html b/doc/mmjsonparse.html new file mode 100644 index 00000000..c2c862d7 --- /dev/null +++ b/doc/mmjsonparse.html @@ -0,0 +1,45 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>CEE/lumberjack JSON support Module (mmjsonparse)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Log Message Normalization Module</h1> +<p><b>Module Name: mmjsonparse</b></p> +<p><b>Available since: </b>6.6.0+ +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides support for parsing structured log messages +that follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked +and, if present, the JSON-encoded structured message content is parsed. +The properties are than available as original message properties. +</p> +<p><b>Action specific Configuration Directives</b>:</p> +<p>currently none +<ul> +<p><b>Legacy Configuration Directives</b>:</p> +<p>none +<b>Caveats/Known Bugs:</b> +<p>None known at this time. +</ul> +<p><b>Sample:</b></p> +<p>This activates the module and applies normalization to all messages:<br> +</p> +<textarea rows="2" cols="60">module(load="mmjsonparse") +action(type="mmjsonparse") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="2" cols="60">$ModLoad mmjsonparse +*.* :mmjsonparse: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2012 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html index 82f9b6a2..787bd957 100644 --- a/doc/mmnormalize.html +++ b/doc/mmnormalize.html @@ -11,37 +11,53 @@ <p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> <p><b>Description</b>:</p> <p>This module provides the capability to normalize log messages via -<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to libee, unstructured text, +<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to liblognorm, unstructured text, like usually found in log messages, can very quickly be parsed and put into -a normal form. This is done so quickly, that it usually should be possible +a normal form. This is done so quickly, that it should be possible to normalize events in realtime. -<p>This module is implemented via the output module interface. That means that +<p>This module is implemented via the output module interface. This means that mmnormalize should be called just like an action. After it has been called, -the normalized message properties are avaialable and can be access. These properties -are called the "CEE" properties, because liblognorm creates a format that is -inspired by the CEE approach. +the normalized message properties are avaialable and can be accessed. These properties +are called the "CEE/lumberjack" properties, because liblognorm creates a format that is +inspired by the CEE/lumberjack approach. +<p><b>Please note:</b> CEE/lumberjack properties are different from regular properties. +They have always "$!" prepended to the property name given in the rulebase. Such a +property needs to be called with <b>%$!propertyname%</b>. <p>Note that mmnormalize should only be called once on each message. Behaviour is -undifined if multiple calls to mmnormalize happen for the same message. +undefined if multiple calls to mmnormalize happen for the same message. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> -<li>$mmnormalizeRuleBase <rulebase-file><br> -Specifies which rulebase file is to use. This file is loaded. If there are +<li><b>ruleBase</b> [word]<br> +Specifies which rulebase file is to use. If there are multiple mmnormalize instances, each one can use a different file. However, a single instance can use only a single file. This parameter MUST be given, -because normalization can only happen based on a rulebase. -<li>$mmnormalizeUseRawMsg <on/off><br> +because normalization can only happen based on a rulebase. It is recommended +that an absolute path name is given. Information on how to create the rulebase +can be found in the <a href="http://www.liblognorm.com/files/manual/index.html">liblognorm manual</a>. +<li><b>useRawMsg</b> [boolean]<br> Specifies if the raw message should be used for normalization (on) or just the MSG part of the message (off). Default is "off". </ul> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li>$mmnormalizeRuleBase <rulebase-file> - equivalent to the "ruleBase" +parameter. +<li>$mmnormalizeUseRawMsg <on/off> - equivalent to the "useRawMsg" +parameter. +</ul> <b>Caveats/Known Bugs:</b> <p>None known at this time. </ul> <p><b>Sample:</b></p> <p>This activates the module and applies normalization to all messages:<br> </p> -<textarea rows="8" cols="60">$ModLoad mmnormalize -$mmnormalizeRuleBase rulebase.rb +<textarea rows="2" cols="60">module(load="mmnormalize") +action(type="mmnormalize" ruleBase="/path/to/rulebase.rb") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="3" cols="60">$ModLoad mmnormalize +$mmnormalizeRuleBase /path/to/rulebase.rb *.* :mmnormalize: </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] @@ -49,7 +65,7 @@ $mmnormalizeRuleBase rulebase.rb <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2010-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index da65b4ba..37c54065 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -31,7 +31,7 @@ You can think of a traditional config file just as a single default rule set, wh automatically bound to each of the inputs. This is even what actually happens. When rsyslog.conf is processed, the config file parser looks for the directive -<pre>$RuleSet <name> +<pre>ruleset(name="rulesetname"); </pre> <p>Where name is any name the user likes (but must not start with "RSYSLOG_", which @@ -63,7 +63,7 @@ to seperate the messages by any other method. <p>Binding to rulesets is input-specifc. For imtcp, this is done via the -<pre>$InputTCPServerBindRuleset <name> +<pre>input(type="imptcp" port="514" ruleset="rulesetname"); </pre> directive. Note that "name" must be the name of a ruleset that is already defined @@ -116,8 +116,12 @@ filters on the message, processes it and then discards it: <pre> # ... module loading ... # process remote messages -:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile -& ~ +if $fromhost-ip == '192.168.152.137' then { + action(type="omfile" file="/var/log/remotefile02") + stop + } + + # only messages not from 192.0.21 make it past this point # The authpriv file has restricted access. @@ -131,7 +135,7 @@ cron.* /var/log/cron ... more ... </pre> -<p>Note the tilde character, which is the discard action!. Also note that we assume that +<p>Note that "stop" is the discard action!. Also note that we assume that 192.0.2.1 is the sole remote sender (to keep it simple). <p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception @@ -141,66 +145,15 @@ case and bind it to the receiver. This may be written as follows: # ... module loading ... # process remote messages # define new ruleset and add rules to it: -$RuleSet remote -*.* /var/log/remotefile +ruleset(name="remote"){ + action(type="omfile" file="/var/log/remotefile") +} # only messages not from 192.0.21 make it past this point -# bind ruleset to tcp listener -$InputTCPServerBindRuleset remote -# and activate it: -$InputTCPServerRun 10514 - -# switch back to the default ruleset: -$RuleSet RSYSLOG_DefaultRuleset -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... +# bind ruleset to tcp listener and activate it: +input(type="imptcp" port="10514" ruleset="remote") </pre> -<p>Here, we need to switch back to the default ruleset after we have defined our custom -one. This is why I recommend a different ordering, which I find more intuitive. The sample -below has it, and it leads to the same results: - -<pre> -# ... module loading ... -# at first, this is a copy of the unmodified rsyslog.conf -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... -# end of the "regular" rsyslog.conf. Now come the new definitions: - -# process remote messages -# define new ruleset and add rules to it: -$RuleSet remote -*.* /var/log/remotefile - -# bind ruleset to tcp listener -$InputTCPServerBindRuleset remote -# and activate it: -$InputTCPServerRun 10514 -</pre> - -<p>Here, we do not switch back to the default ruleset, because this is not needed as it is -completely defined when we begin the "remote" ruleset. - -<p>Now look at the examples and compare them to the single-ruleset solution. You will notice -that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use -"*.*" as all messages now means all messages that are being processed by this -rule set and all of them come in via the TCP receiver! This is what makes using multiple -rulesets so much easier. - <h3>Split local and remote logging for three different ports</h3> <p>This example is almost like the first one, but it extends it a little bit. While it is very similar, I hope it is different enough to provide a useful example why you may want @@ -217,47 +170,34 @@ written to 10516's general log file. <pre> # ... module loading ... -# at first, this is a copy of the unmodified rsyslog.conf -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... -# end of the "regular" rsyslog.conf. Now come the new definitions: - # process remote messages -#define rulesets first -$RuleSet remote10514 -*.* /var/log/remote10514 - -$RuleSet remote10515 -*.* /var/log/remote10515 +ruleset(name="remote10514"){ + action(type="omfile" file="/var/log/remote10514") +} -$RuleSet remote10516 -mail.* /var/log/mail10516 -& ~ -# note that the discard-action will prevent this messag from -# being written to the remote10516 file - as usual... -*.* /var/log/remote10516 +ruleset(name="remote10515"){ + action(type="omfile" file="/var/log/remote10515") +} -# and now define listners bound to the relevant ruleset -$InputTCPServerBindRuleset remote10514 -$InputTCPServerRun 10514 +ruleset(name="test1"){ + if prifilt("mail.*") then { + /var/log/mail10516 + stop + # note that the stop-command will prevent this message from + # being written to the remote10516 file - as usual... + } + /var/log/remote10516 +} -$InputTCPServerBindRuleset remote10515 -$InputTCPServerRun 10515 -$InputTCPServerBindRuleset remote10516 -$InputTCPServerRun 10516 +# and now define listners bound to the relevant ruleset +input(type="imptcp" port="10514" ruleset="remote10514") +input(type="imptcp" port="10515" ruleset="remote10515") +input(type="imptcp" port="10516" ruleset="remote10516") </pre> -<p>Note that the "mail.*" rule inside the "remote10516" ruleset does -not affect processing inside any other rule set, including the default rule set. + <h2>Performance</h2> @@ -289,10 +229,6 @@ dedicated queue for each of the inputs. <p>By default, rulesets do <b>not</b> have their own queue. It must be activated via the <a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> directive. -<h3>Future Enhancements</h3> -<p>In the long term, multiple rule sets will probably lay the foundation for even better -optimizations. So it is not a bad idea to get aquainted with them. - <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> diff --git a/doc/multi_ruleset_legacy_format.html b/doc/multi_ruleset_legacy_format.html new file mode 100644 index 00000000..5a9e7a4a --- /dev/null +++ b/doc/multi_ruleset_legacy_format.html @@ -0,0 +1,192 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Multiple Rulesets in legacy format</title></head> +<body> +<h1>Multiple Rulesets in rsyslog</h1> +<p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports +multiple rulesets within a single configuration. +This is especially useful for routing the recpetion of remote messages to a set of specific rules. +Note that the input module must support binding to non-standard rulesets, so the functionality +may not be available with all inputs.<p> +<b>Attention: this guide is shortened and only contains the samples in legacy format.</b> +Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html<a> + + +<h2>Examples</h2> +<h3>Split local and remote logging</h3> +<p>Let's say you have a pretty standard system that logs its local messages to the usual +bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf +might look like this: + +<pre> +# ... module loading ... +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Now, you want to add receive messages from a remote system and log these to +a special file, but you do not want to have these messages written to the files +specified above. The traditional approach is to add a rule in front of all others that +filters on the message, processes it and then discards it: + +<pre> +# ... module loading ... +# process remote messages +:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile +& ~ +# only messages not from 192.0.21 make it past this point + +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Note the tilde character, which is the discard action!. Also note that we assume that +192.0.2.1 is the sole remote sender (to keep it simple). + +<p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception +case and bind it to the receiver. This may be written as follows: + +<pre> +# ... module loading ... +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile +# only messages not from 192.0.21 make it past this point + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 + +# switch back to the default ruleset: +$RuleSet RSYSLOG_DefaultRuleset +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Here, we need to switch back to the default ruleset after we have defined our custom +one. This is why I recommend a different ordering, which I find more intuitive. The sample +below has it, and it leads to the same results: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 +</pre> + +<p>Here, we do not switch back to the default ruleset, because this is not needed as it is +completely defined when we begin the "remote" ruleset. + +<p>Now look at the examples and compare them to the single-ruleset solution. You will notice +that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use +"*.*" as all messages now means all messages that are being processed by this +rule set and all of them come in via the TCP receiver! This is what makes using multiple +rulesets so much easier. + +<h3>Split local and remote logging for three different ports</h3> +<p>This example is almost like the first one, but it extends it a little bit. While it is +very similar, I hope it is different enough to provide a useful example why you may want +to have more than two rulesets. + +<p>Again, we would like to use the "regular" log files for local logging, only. But +this time we set up three syslog/tcp listeners, each one listening to a different +port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into +different files. Also, logs received from 10516 (and only from that port!) with +"mail.*" priority, shall be written into a specif file and <b>not</b> be +written to 10516's general log file. + +<p>This is the config: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages + +#define rulesets first +$RuleSet remote10514 +*.* /var/log/remote10514 + +$RuleSet remote10515 +*.* /var/log/remote10515 + +$RuleSet remote10516 +mail.* /var/log/mail10516 +& ~ +# note that the discard-action will prevent this messag from +# being written to the remote10516 file - as usual... +*.* /var/log/remote10516 + +# and now define listners bound to the relevant ruleset +$InputTCPServerBindRuleset remote10514 +$InputTCPServerRun 10514 + +$InputTCPServerBindRuleset remote10515 +$InputTCPServerRun 10515 + +$InputTCPServerBindRuleset remote10516 +$InputTCPServerRun 10516 +</pre> + +<p>Note that the "mail.*" rule inside the "remote10516" ruleset does +not affect processing inside any other rule set, including the default rule set. + + +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omfile.html b/doc/omfile.html index 23ecc034..2c5ab97a 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -13,14 +13,14 @@ <p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> <p> </p> -<p><b>Global Configuration Directives</b>:</p> +<p><b>Module Configuration Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> sets a new default template for file actions.<br></li> </ul> <p> </p> -<p><b>Action specific Configuration Directives</b>:</p> +<p><b>Action Confguration Parameters</b>:</p> <ul> <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -28,6 +28,16 @@ <li><strong>ZipLevel </strong>0..9 [default 0]<br> if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + <li><b>VeryReliableZip</b> [<b>on</b>/off] (v7.3.0+) - if ZipLevel is greater 0, + then this setting controls if extra headers are written to make the resulting file + extra hardened against malfunction. If set to off, data appended to previously unclean + closed files may not be accessible without extra tools. + Note that this risk is usually expected to be bearable, and thus "off" is the default mode. + The extra headers considerably + degrade compression, files with this option set to "on" may be four to five times as + large as files processed in "off" mode. + </li><br> + <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -80,7 +90,7 @@ <p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> <p><b>Sample:</b></p> <p>The following command writes all syslog messages into a file.</p> -<textarea rows="5" cols="60">Module (path="builtin:omfile") +<textarea rows="5" cols="60">Module (load="builtin:omfile") *.* action(type="omfile" DirCreateMode="0700" FileCreateMode="0644" @@ -160,7 +170,7 @@ $FileCreateMode 0644 index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omfwd.html b/doc/omfwd.html index 5599ae39..fb9145e1 100644 --- a/doc/omfwd.html +++ b/doc/omfwd.html @@ -16,7 +16,7 @@ <p><b>Global Configuration Directives</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li> + sets a non-standard default template for this module.<br></li> </ul> <p> </p> diff --git a/doc/omjournal.html b/doc/omjournal.html new file mode 100644 index 00000000..c42d9841 --- /dev/null +++ b/doc/omjournal.html @@ -0,0 +1,83 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Linux Journal Output Module (omjournal)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Linux Journal Output Module (omjournal)</h1> +<p><b>Module Name: omjournal</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.3.7</p> +<p><b>Description</b>:</p> +<p>The omjournal output module provides an interface to the Linux journal. +It is meant to be used in those cases where the Linux journal is being used +as the sole system log database. With omjournal, messages from various +sources (e.g. files and remote devices) can also be written to the journal +and processed by its tools. +<p>A typical use case we had on our mind is a SOHO environment, where the +user wants to include syslog data obtained from the local router to be +part of the journal data. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<p>Currently none. + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>One needs to be careful that no message routing loop is created. The +systemd journal forwards messages it receives to the traditional syslog +system (if present). That means rsyslog will receive the same message that +it just wrote as new input on imuxsock. If not handled specially and assuming +all messages be written to the journal, the message would be emitted to the +journal again and a deadly loop is started. +<p>To prevent that, imuxsock by default does not accept messages originating +from its own process ID, aka it ignores messages from the current instance of +rsyslogd. However, this setting can be changed, and if so the problem may occur. +</ul> + +<p><b>Sample:</b></p> +<p>We assume we have a DSL router inside the network and would like to +receive its syslog message into the journal. Note that this configuration can be +used without havoing any other syslog functionality at all (most importantly, there +is no need to write any file to /var/log!). We assume syslog over UDP, as this +is the most probable choice for the SOHO environment that this use case reflects. +To log to syslog data to the journal, add the following snippet to rsyslog.conf: +<textarea rows="20" cols="60">/* first, we make sure all necessary + * modules are present: + */ +module(load="imudp") # input module for UDP syslog +module(load="omjournal") # output module for journal + +/* then, define the actual server that listens to the + * router. Note that 514 is the default port for UDP + * syslog and that we use a dedicated ruleset to + * avoid mixing messages with the local log stream + * (if there is any). + */ +input(type="imudp" port="514" ruleset="writeToJournal") + +/* inside that ruleset, we just write data to the journal: */ +ruleset(name="writeToJournal") { + action(type="omjournal") +} +</textarea> +<p>Note that this can be your sole rsyslog.conf if you do not use rsyslog +for anything else than receving the router syslog messages. +<p>If you do not receive messages, <b>you probably need to enable inbound UDP +syslog traffic in your firewall</b>. + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html index 008dcb81..e47c7f57 100644 --- a/doc/omlibdbi.html +++ b/doc/omlibdbi.html @@ -54,7 +54,23 @@ dlopen()ed plugin (as omlibdbi is). So in short, you probably save you a lot of headache if you make sure you have at least libdbi version 0.8.3 on your system. </p> -<p><b>Action Parameters</b>:</p> +<p><b>Module Parameters</b></p> +<ul> +<li><b>template</b><br> +The default template to use. This template is used when no template is +explicitely specified in the action() statement. +<li><b>driverdirectory</b><br> +Path to the libdbi drivers. Usually, +you do not need to set it. If you installed libdbi-drivers at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <b>not</b> use this configuration directive. +Usually, everything works just fine. +Note that this was an action() paramter in rsyslog versions below 7.3.0. +However, only the first action's driverdirectory parameter was actually used. +This has been cleaned up in 7.3.0, where this now is a module paramter. +</li> +</ul> +<p><b>Action Parameters</b></p> <ul> <li><b>server</b><br>Name or address of the MySQL server <li><b>db</b><br>Database to use @@ -68,24 +84,18 @@ writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", "freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use ompgsql instead).</li> -<li><b>driverdirectory</b><br> -Path to the libdbi drivers. Usually, -you do not need to set it. If you installed libdbi-drivers at a -non-standard location, you may need to specify the directory here. If -you are unsure, do <b>not</b> use this configuration directive. -Usually, everything works just fine.</li> </ul> <p><b>Legacy (pre-v6) Configuration Directives</b>:</p> +<p>It is strongly recommended NOT to use legacy format. <ul> -<li><b>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</b> +<li><i>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</i> - like the driverdirectory action parameter. -<li><strong>$ActionLibdbiDriver drivername</strong><br> - like the drivername action parameter. -<li><span style="font-weight: bold;">$ActionLibdbiHost hostname</span> - like the server action parameter -The host to connect to.</li> -<li><b>$ActionLibdbiUserName user</b> - like the uid action parameter -<li><b>$ActionlibdbiPassword</b> - like the pwd action parameter -<li><b>$ActionlibdbiDBName db</b> - like the db action parameter -<li><b>selector line: :omlibdbi:<i>;template</i></b><br> +<li><i>$ActionLibdbiDriver drivername</i> - like the drivername action parameter +<li><i>$ActionLibdbiHost hostname</i> - like the server action parameter +<li><i>$ActionLibdbiUserName user</i> - like the uid action parameter +<li><i>$ActionlibdbiPassword</i> - like the pwd action parameter +<li><i>$ActionlibdbiDBName db</i> - like the db action parameter +<li><i>selector line: :omlibdbi:<code>;template</code></i><br> executes the recently configured omlibdbi action. The ;template part is optional. If no template is provided, a default template is used (which is currently optimized for MySQL - sorry, folks...)</li> @@ -114,14 +124,14 @@ database "syslog_db" on mysqlsever.example.com. The server is MySQL and being accessed under the account of "user" with password "pwd" (if you have empty passwords, just remove the $ActionLibdbiPassword line).<br> </p> -<textarea rows="5" cols="60">$ModLoad omlibdbi +<textarea rows="5" cols="60">module(load="omlibdbi") *.* action(type="omlibdbi" driver="mysql" server="mysqlserver.example.com" db="syslog_db" uid="user" pwd="pwd" </textarea> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>The same as above, but in legacy config format (pre rsyslog-v6): -<textarea rows="10" cols="60">$ModLoad omlibdbi +<textarea rows="8" cols="60">$ModLoad omlibdbi $ActionLibdbiDriver mysql $ActionLibdbiHost mysqlserver.example.com $ActionLibdbiUserName user diff --git a/doc/ommongodb.html b/doc/ommongodb.html new file mode 100644 index 00000000..a6112642 --- /dev/null +++ b/doc/ommongodb.html @@ -0,0 +1,60 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>MongoDB Output Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>MongoDB Output Module</h1> +<p><b>Module Name: ommongodb</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides native support for logging to MongoDB. +</p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>server</b><br>Name or address of the MongoDB server +<li><b>serverport</b><br>Permits to select +a non-standard port for the MongoDB server. The default is 0, which means the +system default port is used. There is no need to specify this parameter unless +you know the server is running on a non-standard listen port. +<li><b>db</b><br>Database to use +<li><b>collection</b><br>Collection to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +</ul> +<p>Note rsyslog contains a canned default template to write to the MongoDB. It +will be used automatically if no other template is specified to be used. This template is: +<p> +<textarea rows="5" cols="80">template(name="BSON" type="string" string="\"sys\" : \"%hostname%\", \"time\" : \"%timereported:::rfc3339%\", \"time_rcvd\" : \"%timegenerated:::rfc3339%\", \"msg\" : \"%msg%\", \"syslog_fac\" : \"%syslogfacility%\", \"syslog_sever\" : \"%syslogseverity%\", \"syslog_tag\" : \"%syslogtag%\", \"procid\" : \"%programname%\", \"pid\" : \"%procid%\", \"level\" : \"%syslogpriority-text%\"") +</textarea> +<p>This creates the BSON document needed for MongoDB if no template is specified. The default +schema is aligned to CEE and project lumberjack. As such, the field names are standard +lumberjack field names, and <b>not</b> +<a href="property_replacer.html">rsyslog property names</a>. When specifying templates, be sure +to use rsyslog property names as given in the table. If you would like to use lumberjack-based +field names inside MongoDB (which probably is useful depending on the use case), you need to +select fields names based on the lumberjack schema. +If you just want to use a subset of the fields, but with lumberjack names, you can look up the +mapping in the default template. For example, the lumberjack field "level" contains the rsyslog +property "syslogpriority-text". +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog" and into the collection "log" on mongosever.example.com. The server is +being accessed under the account of "user" with password "pwd". +</p> +<textarea rows="5" cols="80">module(load="ommongodb") +*.* action(type="ommongodb" server="mongoserver.example.com" db="syslog" collection="log" uid="user" pwd="pwd") +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the ASL 2.0.</font></p> +</body></html> diff --git a/doc/omudpspoof.html b/doc/omudpspoof.html index df14bbe1..930412c8 100644 --- a/doc/omudpspoof.html +++ b/doc/omudpspoof.html @@ -7,46 +7,161 @@ <h1>UDP spoofing output module (omudpspoof)</h1> <p><b>Module Name: omstdout</b></p> -<p><b>Author: </b>David Lang <david@lang.hm> and Rainer Gerhards -<rgerhards@adiscon.com></p> -<p><b>Available Since</b>: 5.1.3</p> +<p><b>Authors: </b>Rainer Gerhards <rgerhards@adiscon.com> +and David Lang <david@lang.hm> +</p> +<p><b>Available Since</b>: 5.1.3 / v7 config since 7.2.5</p> <p><b>Description</b>:</p> <p>This module is similar to the regular UDP forwarder, but permits to spoof the sender address. Also, it enables to circle through a number of source ports. -<p><b>Configuration Directives</b>:</p> +<p><b>Important:</b> This module requires root priveleges for its low-level +socket access. As such, the <b>module will not work if rsyslog is configured to +drop privileges</b>. + +<p><b>load() Parameters</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a non-standard default template for this module.<br></li> + +</ul> +<p> </p> +<p><b>action() parameters</b>:</p> +<ul> + <li><strong>Target </strong>string<br> + Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> + + <li><strong>Port </strong>[Integer, Default 514]<br> + Name or numerical value of port to use when connecting to target. <br></li><br> + + <li><b>Template</b>[Word]<br> + Template to use as message text. + <br></li><br> + + <li><strong>SourceTemplate </strong>[Word]<br> + This is the name of the template that contains a + numerical IP address that is to be used as the source system IP address. + While it may often be a constant value, it can be generated as usual via the + property replacer, as long as it is a valid IPv4 address. If not specified, the + build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined + as follows:<br> + template(name="RSYSLOG_omudpspoofDfltSourceTpl" type="string" string="%fromhost-ip%")<br> + So in essence, the default template spoofs the address of the system the message + was received from. This is considered the most important use case. + <br></li><br> + + <li><b>SourcePortStart</b>[Word]<br> + Specifies the start value for circeling the source ports. Must be less than or + equal to the end value. Default is 32000. + <br></li><br> + + <li><b>SourcePortEnd</b>[Word]<br> + Specifies the ending value for circeling the source ports. Must be less than or + equal to the start value. Default is 42000. + <br></li><br> + + <li><b>mtu</b>[Integer, default 1500]<br> + Maximum MTU supported by the network. Default respects Ethernet and must + usually not be adjusted. Setting a too-high MTU can lead to message loss, + too low to excess message fragmentation. Change only if you really know what + you are doing. This is always given in number of bytes. + <br></li><br> +</ul> +<p><b>pre-v7 Configuration Directives</b>:</p> <ul> -<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename><br> -This is the name of the template that contains a -numerical IP address that is to be used as the source system IP address. -While it may often be a constant value, it can be generated as usual via the -property replacer, as long as it is a valid IPv4 address. If not specified, the -build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined -as follows:<br> -$template RSYSLOG_omudpspoofDfltSourceTpl,"%fromhost-ip%"<br> -So in essence, the default template spoofs the address of the system the message -was received from. This is considered the most important use case. -<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname><br> -Host that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofTargetPort</b> <port><br> -Remote port that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename><br> -This setting instructs omudpspoof to use a template different from the -default template for all of its actions that do not have a template specified -explicitely. -<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number><br> -Specifies the start value for circeling the source ports. Must be less than or -equal to the end value. Default is 32000. -<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number><br> -Specifies the ending value for circeling the source ports. Must be less than or -equal to the start value. Default is 42000. +<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename> +- equivalent to the "sourceTemplate" parameter. +<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofTargetPort</b> <port> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename> +- equivalent to the "template" load() parameter. +<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number> +- equivalent to the "SourcePortStart" parameter. +<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number> +- equivalent to the "SourcePortEnd" parameter. </ul> <b>Caveats/Known Bugs:</b> <ul> <li><b>IPv6</b> is currently not supported. If you need this capability, please let us know via the rsyslog mailing list. +<li>Versions shipped prior to rsyslog 7.2.5 do not support message sizes over 1472 bytes (more +pricesely: over the network-supported MTU). Starting with 7.2.5, those messages will be +fragmented, up to a total upper limit of 64K (induced by UDP). Message sizes over +64K will be truncated. For older versions, messages over 1472 may be totally discarded +or truncated, depending on version and environment. </ul> -<p><b>Sample:</b></p> + +<p><b>Config Samples</b></p> +<p>The following sample forwards all syslog messages in standard form to the +remote server server.example.com. The original sender's address is used. We do not +care about the source port. This example is considered the typical use case for +omudpspoof. +</p> +<textarea rows="3" cols="80">module(load="omudpspoof") +action(type="omudpspoof" target="server.example.com") +</textarea> + +<p>The following sample forwards all syslog messages in unmodified form to the +remote server server.example.com. The sender address 192.0.2.1 with fixed +source port 514 is used. +</p> +<textarea rows="7" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514) +</textarea> +<p>The following sample is exatly like the previous, but it specifies a larger size +MTU. If, for example, the envrionment supports Jumbo Ethernet frames, increasing the +MTU is useful as it reduces packet fragmentation, which most often is the source of +problems. Note that setting the MTU to a value larger than the local-attached network +supports will lead to send errors and loss of message. So use with care! +</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>Of course, the action can be combined with any type of filter, for +example a tradition PRI filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +local0.* action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>... or any complex expression-based filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>and of course it can also be combined with as many other actions +as one likes:</p> +<textarea rows="11" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then { + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") + action(type="omfile" file="/var/log/somelog") + stop # or whatever... +} +</textarea> + +<p><b>Legacy Sample (pre-v7):</b></p> <p>The following sample forwards all syslog messages in standard form to the remote server server.example.com. The original sender's address is used. We do not care about the source port. This example is considered the typical use case for diff --git a/doc/property_replacer.html b/doc/property_replacer.html index dc09d33c..c7624b2d 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -228,7 +228,15 @@ for filtering in a generic way)</td> <td>This is the "bridge" to syslog message normalization (via <a href="mmnormalize.html">mmnormalize</a>): name is a name defined inside the normalization rule. It has the value selected by the rule -or none if no rule with this field did match. +or none if no rule with this field did match. You can also use these +properties to specify JSON fields from the CEE-enhanced syslog +message, once you parse it with <a href="mmjsonparse.html">mmjsonparse</a> +</td> +</tr> +<tr> +<td><b>$!all-json</b></td> +<td>This is the JSON part of the CEE-enhanced syslog message, which +can be parsed with <a href="mmjsonparse.html">mmjsonparse</a> </td> </tr> </tbody> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index fcc2674d..d76316ed 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -51,6 +51,11 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. +<h2>Lookup Tables</h2> +<p><a href="lookup_tables.html">Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices). <h2>Functions</h2> <p>RainerScript supports a currently quite limited set of functions: <ul> @@ -61,6 +66,24 @@ variable, if it exists. Returns an empty string if it does not exist. <li>cstr(expr) - converts expr to a string value <li>cnum(expr) - converts expr to a number (integer) <li>re_match(expr, re) - returns 1, if expr matches re, 0 otherwise +<li>field(str, delim, matchnbr) - returns a field-based substring. str is the string +to search, delim is the delimiter and matchnbr is the match to search +for (the first match starts at 1). This works similar as the field based +property-replacer option. +Versions prior to 7.3.7 only support a single character as delimiter character. +Starting with version 7.3.7, a full string can be used as delimiter. If a single +character is being used as delimiter, delim is the numerical ascii value of the +field delimiter character (so that non-printable characters can by specified). If a +string is used as delmiter, a multi-character string (e.g. "#011") is to be +specified. Samples:<br> +set $!usr!field = field($msg, 32, 3); -- the third field, delimited by space<br> +set $!usr!field = field($msg, "#011", 3); -- the third field, delmited by "#011"<br> +Note that when a single character is specified as string [field($msg, ",", 3)] a +string-based extraction is done, which is more performance intense than the +equivalent single-character [field($msg, 44 ,3)] extraction. +<li>prifilt(constant) - mimics a traditional PRI-based filter (like "*.*" or +"mail.info"). The traditional filter string must be given as a <b>constant string</b>. +Dynamic string evaluation is not permitted (for performance reasons). </ul> <p>The following example can be used to build a dynamic filter based on some environment variable: @@ -72,7 +95,7 @@ if $msg contains getenv('TRIGGERVAR') then /path/to/errfile <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html index ef29c2a8..433456c1 100644 --- a/doc/rsconf1_rulesetparser.html +++ b/doc/rsconf1_rulesetparser.html @@ -9,7 +9,7 @@ <p><b>Type:</b> ruleset-specific configuration directive</p> <p><b>Parameter Values:</b> string</p> <p><b>Available since:</b> 5.3.4+</p> -<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc5425</p> +<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc3164</p> <p><b>Description:</b></p> <p> This directive permits to specify which diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 6aa2e460..c5f4d2e3 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -21,16 +21,15 @@ especially useful while you are migrating from syslogd to rsyslogd.</p> <p><b>Follow the links below to learn more about specific topics:</b></p> <ul> -<li><a href="rsyslog_conf_modules.html">Modules</a></li> -<li><a href="rsyslog_conf_lines.html">Lines</a></li> -<li><a href="rsyslog_conf_global.html">Configuration Directives</a></li> <li><a href="rsyslog_conf_basic_structure.html">Basic Structure</a></li> +<li><a href="rsyslog_conf_modules.html">Modules</a></li> <li><a href="rsyslog_conf_templates.html">Templates</a></li> -<li><a href="rsyslog_conf_output.html">Output Channels</a></li> <li><a href="rsyslog_conf_filter.html">Filter Conditions</a></li> -<li><a href="rsyslog_conf_actions.html">Actions</a></li> -<li><a href="rsyslog_conf_file_syntax_differences.html">Configuration File Syntax Differences</a></li> -<li><a href="rsyslog_conf_examples.html">Examples</a></li> +<li><a href="rsyslog_conf_actions.html">Actions (legacy format)</a></li> +<li><a href="rsyslog_conf_output.html">Output Channels</a></li> +<!--<li><a href="rsyslog_conf_examples.html">Examples</a></li>--> +<li><a href="rsyslog_conf_global.html">Legacy Configuration Directives</a></li> +<li><a href="rsyslog_conf_sysklogd_compatibility.html">sysklogd compatibility</a></li> </ul> <p>[<a href="rsyslog_conf.html">back to top</a>] @@ -38,7 +37,7 @@ especially useful while you are migrating from syslogd to rsyslogd.</p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 2e2293ce..0c7705f8 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -3,7 +3,7 @@ <body> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> -<h2>Actions</h2> +<h2>Actions (legacy format)</h2> <p>The action field of a rule describes what to do with the message. In general, message content is written to a kind of "logfile". But also other actions might be done, like writing to a database table diff --git a/doc/rsyslog_conf_basic_structure.html b/doc/rsyslog_conf_basic_structure.html index 4ce78de0..fad1b110 100644 --- a/doc/rsyslog_conf_basic_structure.html +++ b/doc/rsyslog_conf_basic_structure.html @@ -1,33 +1,92 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html><head><title>Basic Structure - rsyslog.conf</title></head> <body> +<h1>Basic rsyslog.conf Structure</h1> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Basic Structure</h1> -<p>Rsyslog supports standard sysklogd's configuration file format -and extends it. So in general, you can take a "normal" syslog.conf and -use it together with rsyslogd. It will understand everything. However, -to use most of rsyslogd's unique features, you need to add extended -configuration directives.</p> -<p>Rsyslogd supports the classical, selector-based rule lines. -They are still at the heart of it and all actions are initiated via -rule lines. A rule lines is any line not starting with a $ or the -comment sign (#). Lines starting with $ carry rsyslog-specific -directives.</p> -<p>Every rule line consists of two fields, a selector field and -an action field. These two fields are separated by one or more spaces -or tabs. The selector field specifies a pattern of facilities and -priorities belonging to the specified action.<br> -<br> -Lines starting with a hash mark ("#'') and empty lines are ignored. -</p> +<p>Rsyslog supports three different types of configuration statements +concurrently: +<ul> +<li><b>sysklogd</b> - this is the plain old format, thaught everywhere +and still pretty useful for simple use cases. Note that some very +few constructs are no longer supported because they are incompatible +with newer features. These are mentioned in the compatibility docs. +<li><b>legacy rsyslog</b> - these are statements that begin with a dollar +sign. They set some configuration parameters and modify e.g. the way +actions operate. This is the only format supported in pre-v6 versions of +rsyslog. It is still fully supported in v6 and above. Note that some +plugins and features may still only be available through legacy format +(because plugins need to be explicitely upgraded to use the new style +format, and this hasn't happened to all plugins). +<li><b>RainerScript</b> - the new style format. This is the best and most +precise format to be used for more complex cases. The rest of this page +assumes RainerScript based rsyslog.conf. +</ul> +<p>The rsyslog.conf files consists of statements. For old style (sysklogd & legacy +rsyslog), lines do matter. For new style (RainerScript) line spacing is irrelevant. +Most importantly, this means with new style actions and all other objects can split +across lines as users want to. +<h2>Comments</h2> +<p>There are two types of comments: +<ul> +<li><b>#-Comments</b> - start with a hash sign (#) and run to the end of the line +<li><b>C-style Comments</b> - start with /* and end with */, just like in the C +programming language. They can be used to comment out multiple lines at one. Comment +nesting is not supported, but #-Comments can be contained inside a C-style comment. +</ul> + +<h2>Processing Order</h2> +<p>Directives are processed from the top of rsyslog.conf to the bottom. Sequence +matters. For example, if you stop processing of a message, obviously all statements +after the stop statement are never evaluated. + +<h3>Flow Control Statements</h3> +<ul> +<li><b>if expr then ... else ...</b> - conditional execution +<li><b>stop</b> - stops processing the current message +<li><b>call</b> - calls a ruleset (just like a subroutine call) +<li><b>continue</b> - a NOP, useful e.g. inside the then part of an if +</ul> + +<h3>Data Manipulation Statements</h3> +<ul> +<li><b>set</b> - sets a user variable +<li><b>unset</b> - deletes a previously set user variable +</ul> + +<h2>Inputs</h2> +<p>Every input requires an input module to be loaded and a listener defined for it. +Full details can be found inside the <a href="rsyslog_conf_modules.html">rsyslog +modules</a> documentation. Once loaded, inputs are defined via the +<b>input()</b> object. + +<h2>Outputs</h2> +<p>Outputs are also called "actions". A small set of actions is pre-loaded (like +the output file writer, which is used in almost every rsyslog.conf), others must +be loaded just like inputs. +<p>An action is invoked via the <b>action(type="type" ...)</b> object. Type is +mandatory and must contain the name of the plugin to be called (e.g. "omfile" or +"ommongodb"). Other paramters may be present. Their type and use depends on +the output plugin in question. + +<h2>Rulesets and Rules</h2> +<p>Rulesets and rules form the basis of rsyslog processing. In short, a rule +is a way how rsyslog shall process a specific message. Usually, there is a type +of filter (if-statement) in front of the rule. Complex nesting of rules is possible, +much like in a programming language. +<p>Rulesets are containers for rules. A single ruleset can contain many rules. In +the programming language analogy, one may think of a ruleset like being a program. +A ruleset can be "bound" (assigned) to a specific input. In the analogy, this means that when +a message comes in via that input, the "program" (ruleset) bound to it will be executed +(but not any other!). +<p>There is detail documentation available for +<a href="multi_ruleset">rsyslog rulesets</a>. <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_examples.html b/doc/rsyslog_conf_examples.html deleted file mode 100644 index b46460e5..00000000 --- a/doc/rsyslog_conf_examples.html +++ /dev/null @@ -1,209 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Examples - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">back</a> -<h2>Examples</h2> -<p>Below are example for templates and selector lines. I hope -they are self-explanatory. If not, please see -www.monitorware.com/rsyslog/ for advise.</p> -<h3>TEMPLATES</h3> -<p>Please note that the samples are split across multiple lines. -A template MUST NOT actually be split across multiple lines.<br> -<br> -A template that resembles traditional syslogd file output:<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> -%syslogtag%%msg:::drop-last-lf%\n"<br> -<br> -A template that tells you a little more about the message:<br> -$template -precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> -%syslogtag%,%msg%\n"<br> -<br> -A template for RFC 3164 format:<br> -$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% -%syslogtag%%msg%"<br> -<br> -A template for the format traditonally used for user messages:<br> -$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> -<br> -And a template with the traditonal wall-message format:<br> -$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at -%timegenerated%<br> -<br> -A template that can be used for the database write (please note the SQL<br> -template option)<br> -$template MySQLInsert,"insert iut, message, receivedat values<br> -('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')<br> -into systemevents\r\n", SQL<br> -<br> -The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> -format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> -format, you do not feel bad if you don't know it ;)). It's interesting -to see how it takes different parts out of the date stamps. What -happens is that the date stamp is split into the actual date and time -and the these two are combined with just a comma in between them.<br> -<br> -$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> -%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,<br> -%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,<br> -%syslogtag%%msg%\n"</p> -<h3>SELECTOR LINES</h3> -<p># Store critical stuff in critical<br> -#<br> -*.=crit;kern.none /var/adm/critical<br> -<br> -This will store all messages with the priority crit in the file -/var/adm/critical, except for any kernel message.<br> -<br> -<br> -# Kernel messages are first, stored in the kernel<br> -# file, critical messages and higher ones also go<br> -# to another host and to the console. Messages to<br> -# the host finlandia are forwarded in RFC 3164<br> -# format (using the template defined above).<br> -#<br> -kern.* /var/adm/kernel<br> -kern.crit @finlandia;RFC3164fmt<br> -kern.crit /dev/console<br> -kern.info;kern.!err /var/adm/kernel-info<br> -<br> -The first rule direct any message that has the kernel facility to the -file /var/adm/kernel.<br> -<br> -The second statement directs all kernel messages of the priority crit -and higher to the remote host finlandia. This is useful, because if the -host crashes and the disks get irreparable errors you might not be able -to read the stored messages. If they're on a remote host, too, you -still can try to find out the reason for the crash.<br> -<br> -The third rule directs these messages to the actual console, so the -person who works on the machine will get them, too.<br> -<br> -The fourth line tells rsyslogd to save all kernel messages that come -with priorities from info up to warning in the file -/var/adm/kernel-info. Everything from err and higher is excluded.<br> -<br> -<br> -# The tcp wrapper loggs with mail.info, we display<br> -# all the connections on tty12<br> -#<br> -mail.=info /dev/tty12<br> -<br> -This directs all messages that uses mail.info (in source LOG_MAIL | -LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper -tcpd(8) uses this as it's default.<br> -<br> -<br> -# Store all mail concerning stuff in a file<br> -#<br> -mail.*;mail.!=info /var/adm/mail<br> -<br> -This pattern matches all messages that come with the mail facility, -except for the info priority. These will be stored in the file -/var/adm/mail.<br> -<br> -<br> -# Log all mail.info and news.info messages to info<br> -#<br> -mail,news.=info /var/adm/info<br> -<br> -This will extract all messages that come either with mail.info or with -news.info and store them in the file /var/adm/info.<br> -<br> -<br> -# Log info and notice messages to messages file<br> -#<br> -*.=info;*.=notice;\<br> -mail.none /var/log/messages<br> -<br> -This lets rsyslogd log all messages that come with either the info or -the notice facility into the file /var/log/messages, except for all<br> -messages that use the mail facility.<br> -<br> -<br> -# Log info messages to messages file<br> -#<br> -*.=info;\<br> -mail,news.none /var/log/messages<br> -<br> -This statement causes rsyslogd to log all messages that come with the -info priority to the file /var/log/messages. But any message coming -either with the mail or the news facility will not be stored.<br> -<br> -<br> -# Emergency messages will be displayed using wall<br> -#<br> -*.=emerg *<br> -<br> -This rule tells rsyslogd to write all emergency messages to all -currently logged in users. This is the wall action.<br> -<br> -<br> -# Messages of the priority alert will be directed<br> -# to the operator<br> -#<br> -*.alert root,rgerhards<br> -<br> -This rule directs all messages with a priority of alert or higher to -the terminals of the operator, i.e. of the users "root'' and -"rgerhards'' if they're logged in.<br> -<br> -<br> -*.* @finlandia<br> -<br> -This rule would redirect all messages to a remote host called -finlandia. This is useful especially in a cluster of machines where all -syslog messages will be stored on only one machine.<br> -<br> -In the format shown above, UDP is used for transmitting the message. -The destination port is set to the default auf 514. Rsyslog is also -capable of using much more secure and reliable TCP sessions for message -forwarding. Also, the destination port can be specified. To select TCP, -simply add one additional @ in front of the host name (that is, @host -is UPD, @@host is TCP). For example:<br> -<br> -<br> -*.* @@finlandia<br> -<br> -To specify the destination port on the remote machine, use a colon -followed by the port number after the machine name. The following -forwards to port 1514 on finlandia:<br> -<br> -<br> -*.* @@finlandia:1514<br> -<br> -This syntax works both with TCP and UDP based syslog. However, you will -probably primarily need it for TCP, as there is no well-accepted port -for this transport (it is non-standard). For UDP, you can usually stick -with the default auf 514, but might want to modify it for security rea-<br> -sons. If you would like to do that, it's quite easy:<br> -<br> -<br> -*.* @finlandia:1514<br> -<br> -<br> -<br> -*.* >dbhost,dbname,dbuser,dbpassword;dbtemplate<br> -<br> -This rule writes all message to the database "dbname" hosted on -"dbhost". The login is done with user "dbuser" and password -"dbpassword". The actual table that is updated is specified within the -template (which contains the insert statement). The template is called -"dbtemplate" in this case.</p> -<p>:msg,contains,"error" @errorServer</p> -<p>This rule forwards all messages that contain the word "error" -in the msg part to the server "errorServer". Forwarding is via UDP. -Please note the colon in fron</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_file_syntax_differences.html b/doc/rsyslog_conf_file_syntax_differences.html deleted file mode 100644 index bfac8926..00000000 --- a/doc/rsyslog_conf_file_syntax_differences.html +++ /dev/null @@ -1,32 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Configuration File Syntax Differences - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Configuration File Syntax Differences</h1> -<p>Rsyslogd uses a slightly different syntax for its -configuration file than the original BSD sources. Originally all -messages of a specific priority and above were forwarded to the log -file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd -more flexible and to use it in a more intuitive manner.<br> -<br> -The original BSD syslogd doesn't understand spaces as separators -between the selector and the action field.<br> -<br> -When compared to syslogd from sysklogd package, rsyslogd offers -additional -<a href="features.html">features</a> (like template -and database support). For obvious reasons, the syntax for defining -such features is available in rsyslogd, only.</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 3 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_filter.html b/doc/rsyslog_conf_filter.html index fbced4a3..a795193f 100644 --- a/doc/rsyslog_conf_filter.html +++ b/doc/rsyslog_conf_filter.html @@ -4,38 +4,95 @@ <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> <h2>Filter Conditions</h2> -<p>Rsyslog offers four different types "filter conditions":</p> +<p>Rsyslog offers three different types "filter conditions":</p> <ul> -<li>BSD-style blocks</li> +<li><a href="http://www.rainerscript.com/">RainerScript</a>-based filters</li> <li>"traditional" severity and facility based selectors</li> <li>property-based filters</li> -<li>expression-based filters</li> </ul> -<h3>Blocks</h3> -<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each -block of lines is separated from the previous block by a program or -hostname specification. A block will only log messages corresponding to -the most recent program and hostname specifications given. Thus, a -block which selects ‘ppp’ as the program, directly followed by a block -that selects messages from the hostname ‘dialhost’, then the second -block will only log messages from the ppp program on dialhost. -</p> -<p>A program specification is a line beginning with ‘!prog’ and -the following blocks will be associated with calls to syslog from that -specific program. A program specification for ‘foo’ will also match any -message logged by the kernel with the prefix ‘foo: ’. Alternatively, a -program specification ‘-foo’ causes the following blocks to be applied -to messages from any program but the one specified. A hostname -specification of the form ‘+hostname’ and the following blocks will be -applied to messages received from the specified hostname. -Alternatively, a hostname specification ‘-hostname’ causes the -following blocks to be applied to messages from any host but the one -specified. If the hostname is given as ‘@’, the local hostname will be -used. (NOT YET IMPLEMENTED) A program or hostname specification may be -reset by giving the program or hostname as ‘*’.</p> -<p>Please note that the "#!prog", "#+hostname" and "#-hostname" -syntax available in BSD syslogd is not supported by rsyslogd. By -default, no hostname or program is set.</p> +<h3>RainerScript-Based Filters</h3> +RainerScript based filters are the prime means of creating complex rsyslog configuration. +The permit filtering on arbitrary complex expressions, which can include boolean, +arithmetic and string operations. They also support full nesting of filters, just +as you know from other scripting environments. +<br> +Scripts based filters are indicated by the keyword "if", as usual. +They have this format:<br> +<br> +if expr then block else block +<br> +"If" and "then" are fixed keywords that mus be present. "expr" is a +(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for +details. +The keyword "else" and its associated block is optional. Note that a block can contain either +a single action (chain), or an arbitrary complex script enclosed in curly braces, e.g.: +<br> +<pre> +if $programname == 'prog1' then { + action(type="omfile" file="/var/log/prog1.log") + if $msg contains 'test' then + action(type="omfile" file="/var/log/prog1test.log") + else + action(type="omfile" file="/var/log/prog1notest.log") +} +</pre> +<br> +Other types of filtes can also be combined with the pure RainerScript ones. This makes +it particularly easy to migrate from early config files to RainerScript. Also, the traditional +syslog PRI-based filters are a good and easy to use addition. While they are legacy, we still +recommend there use where they are up to the job. We do NOT, however, recommend property-based +filters any longer. As an example, the following is perfectly valid: +<br> +<pre> +if $fromhost == 'host1' then { + mail.* action(type="omfile" file="/var/log/host1/mail.log") + *.err /var/log/host1/errlog # this is also still valid + # + # more "old-style rules" ... + # +} else { + mail.* action(type="omfile" file="/var/log/mail.log") + *.err /var/log/errlog + # + # more "old-style rules" ... + # +} +</pre> +<br> + +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. +If you don't like that, you can of course also use the textual property +- just be sure to use the right one. As expression support is enhanced, +this will change. For example, if you would like to filter on message +that have facility local0, start with "DEVNAME" and have either +"error1" or "error0" in their message content, you could use the +following filter:<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +Please note that the above <span style="font-weight: bold;">must +all be on one line</span>! And if you would like to store all +messages except those that contain "error1" or "error0", you just need +to add a "not":<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> +($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +If you would like to do case-insensitive comparisons, use +"contains_i" instead of "contains" and "startswith_i" instead of +"startswith".<br> +<br> +Regular expressions are supported via functions (see function list). + <h3>Selectors</h3> <p><b>Selectors are the traditional way of filtering syslog messages.</b> They have been kept in rsyslog with their original @@ -140,9 +197,14 @@ of the property value. For example, if you search for "val" with <p>it will be a match if msg contains "values are in this message" but it won't match if the msg contains "There are values in this message" (in the later case, contains would match). Please note -that "startswith" is by far faster than regular expressions. So even -once they are implemented, it can make very much sense -(performance-wise) to use "startswith".</p> +that "startswith" is by far faster than regular expressions. So +it makes very much sense (performance-wise) to use "startswith".</p> +<p>Note: when processing syslog messages, please note that $msg usually +starts with a space. The reason for this is RFC3164. Please read the +<a href="http://www.rsyslog.com/log-normalization-and-the-leading-space/">detail +description</a> of what that means to you. In short, you need to make sure +that you include the first space if you use "startswith", otherwise you will +not get matches. </td> </tr> <tr> @@ -213,71 +275,6 @@ supported (except for "not" as outlined above). Please note that while it is possible to query facility and severity via property-based filters, it is far more advisable to use classic selectors (see above) for those cases.</p> -<h3>Expression-Based Filters</h3> -Expression based filters allow -filtering on arbitrary complex expressions, which can include boolean, -arithmetic and string operations. Expression filters will evolve into a -full configuration scripting language. Unfortunately, their syntax will -slightly change during that process. So if you use them now, you need -to be prepared to change your configuration files some time later. -However, we try to implement the scripting facility as soon as possible -(also in respect to stage work needed). So the window of exposure is -probably not too long.<br> -<br> -Expression based filters are indicated by the keyword "if" in column 1 -of a new line. They have this format:<br> -<br> -if expr then action-part-of-selector-line<br> -<br> -"If" and "then" are fixed keywords that mus be present. "expr" is a -(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for -details. "action-part-of-selector-line" is an action, just as you know -it (e.g. "/var/log/logfile" to write to that file).<br> -<br> -A few quick samples:<br> -<br> -<code> -*.* /var/log/file1 # the traditional way<br> -if $msg contains 'error' then /var/log/errlog # the expression-based way<br> -</code> -<br> -Right now, you need to specify numerical values if you would like to -check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. -If you don't like that, you can of course also use the textual property -- just be sure to use the right one. As expression support is enhanced, -this will change. For example, if you would like to filter on message -that have facility local0, start with "DEVNAME" and have either -"error1" or "error0" in their message content, you could use the -following filter:<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -Please note that the above <span style="font-weight: bold;">must -all be on one line</span>! And if you would like to store all -messages except those that contain "error1" or "error0", you just need -to add a "not":<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> -($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -If you would like to do case-insensitive comparisons, use -"contains_i" instead of "contains" and "startswith_i" instead of -"startswith".<br> -<br> -Note that regular expressions are currently NOT -supported in expression-based filters. These will be added later when -function support is added to the expression engine (the reason is that -regular expressions will be a separate loadable module, which requires -some more prequisites before it can be implemented).<br> - <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index a4d760eb..651808f6 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -67,7 +67,7 @@ default 0 (no delay). Simple rate-limiting!]</li> <li>$ActionQueueDiscardMark <number> [default 9750]</li> <li>$ActionQueueDiscardSeverity <number> -[*numerical* severity! default 4 (warning)]</li> +[*numerical* severity! default 8 (nothing discarded)]</li> <li>$ActionQueueFileName <name></li> <li>$ActionQueueHighWaterMark <number> [default 8000]</li> diff --git a/doc/rsyslog_conf_lines.html b/doc/rsyslog_conf_lines.html deleted file mode 100644 index 0e6cc0d3..00000000 --- a/doc/rsyslog_conf_lines.html +++ /dev/null @@ -1,23 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Lines - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Lines</h1> -<p>Lines can be continued by specifying a backslash ("\") as the last -character of the line. There is a hard-coded maximum line length of 4K.<br> -If you need lines larger than that, you need to change compile-time -settings inside rsyslog and recompile. -</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 3 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index cbd60faf..554b20f4 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -35,7 +35,7 @@ to message generators. <ul> <li><a href="imfile.html">imfile</a> - input module for text files</li> <li><a href="imrelp.html">imrelp</a> - RELP input module</li> -<li>imudp - udp syslog message input</li> +<li><a href="imudp.html">imudp</a> - udp syslog message input</li> <li><a href="imtcp.html">imtcp</a> - input plugin for tcp syslog</li> <li><a href="imptcp.html">imptcp</a> - input plugin for plain tcp syslog (no TLS but faster)</li> <li><a href="imgssapi.html">imgssapi</a> - input plugin for plain tcp and GSS-enabled syslog</li> @@ -53,6 +53,7 @@ and messages be transmitted to various different targets. <ul> <li><a href="omfile.html">omfile</a> - file output module</li> <li><a href="omfwd.html">omfwd</a> - syslog forwarding output module</li> +<li><a href="omjournal.html">omjournal</a> - Linux journal output module</li> <li><a href="ompipe.html">ompipe</a> - named pipe output module</li> <li><a href="omusrmsg.html">omusrmsg</a> - user message output module</li> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> @@ -72,6 +73,7 @@ permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> <li><a href="omuxsock.html">omuxsock</a> - output module Unix domain sockets</li> <li><a href="omhdfs.html">omhdfs</a> - output module for Hadoop's HDFS file system</li> +<li><a href="ommongodb.html">ommongodb</a> - output module for MongoDB</li> </ul> <a name="pm"></a><h2>Parser Modules</h2> @@ -99,18 +101,18 @@ They can be implemented using either the output module or the parser module inte From the rsyslog core's point of view, they actually are output or parser modules, it is their implementation that makes them special. <p>Currently, there exists only a limited set of such modules, but new ones could be written with -the methods the engine provides. They could be used, for example, to: -<ul> -<li>anonymize message content -<li>add dynamically computed content to message (fields) -</ul> +the methods the engine provides. They could be used, for example, to +add dynamically computed content to message (fields). <p>Message modification modules are usually written for one specific task and thus usually are not generic enough to be reused. However, existing module's code is probably an excellent starting base for writing a new module. Currently, the following modules exist inside the source tree: <ul> +<li><a href="mmanon.html">mmanon</a> - used to anonymize log messages. <li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. Note that this actually is a <b>generic</b> module. +<li><a href="mmjsonparse.html">mmjsonparse</a> - used to interpret CEE/lumberjack +enabled structured log messages. <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. @@ -179,7 +181,7 @@ filter settings. This graphic above is a high-level message flow diagram. [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_sysklogd_compatibility.html b/doc/rsyslog_conf_sysklogd_compatibility.html new file mode 100644 index 00000000..c95d6fda --- /dev/null +++ b/doc/rsyslog_conf_sysklogd_compatibility.html @@ -0,0 +1,31 @@ +<html><head><title>sysklogdcompatibility - rsyslog.conf</title></head> +<body> +<h1>sysklogd compatibility</h1> +<p>This is a part of the rsyslog.conf documentation.</p> +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<p>Rsyslog supports standard sysklogd's configuration file format +and extends it. So in general, you can take a "normal" syslog.conf and +use it together with rsyslogd. It will understand everything. However, +to use most of rsyslogd's unique features, you need to add extended +configuration directives.</p> +<p>Rsyslogd supports the classical, selector-based rule lines. +They are still at the heart of it and all actions are initiated via +rule lines. +However, there are ample new directives, either in rsyslog traditional +format (starting with a dollar sign) or in RainerScript format. These +work together with sysklogd statements. A few select statements are +no longer supported and may generate error messages. They are mentioned +in the compatibility notes. +</p> + +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> + diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index b97f6609..0c189100 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -3,7 +3,7 @@ <body> <p>This is a part of the rsyslog.conf - documentation.</p> <a href="rsyslog_conf.html">back</a> -<h2>Templates</h2> +<h1>Templates</h1> <p>Templates are a key feature of rsyslog. They allow to specify any format a user might want. They are also used for dynamic file name @@ -16,67 +16,200 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> -<p>Starting with 5.5.6, there are actually two differnt types of template: +<p>Templates are specified by template() statements. They can also be specified +via $Template legacy statements. Note that these are scheduled for removal in +later versions of rsyslog, so it is probably a good idea to avoid them +for new uses. +<h2>The template() statement</h2> +<p>The template() statement is used to define templates. Note that it is a +<b>static</b> statement, that means all templates are defined when rsyslog +reads the config file. As such, templates are not affected by if-statements +or config nesting. +<p>The basic structure of the template statement is as follows: +<br><br> +<code>template(parameters)</code> +<br><br> +In addition to this simpler syntax, list templates (to be described below) +support an extended syntax: +<br><br> +<code>template(parameters) { list-descriptions }</code> +<p>Each template has a parameter <b>name</b>, which specifies the templates +name, and a parameter <b>type</b>, which specifies the template type. The name +parameter must be unique, and behaviour is unpredictable if it is not. The <b>type</b> +parameter specifies different template types. Different types simply enable +different ways to specify the template content. The template type <b>does not</b> +affect what an (output) plugin can do with it. So use the type that best fits your +needs (from a config writing point of view!). The following types are available: <ul> -<li>string based -<li>string-generator module based +<li>list +<li>subtree +<li>string +<li>plugin </ul> -<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates -have been introduced in 5.5.6. They permit a string generator, actually a C "program", -the generate a format. Obviously, it is more work required to code such a generator, -but the reward is speed improvement. If you do not need the ultimate throughput, you -can forget about string generators (so most people never need to know what they are). -You may just be interested in learning that for the most important default formats, -rsyslog already contains highly optimized string generators and these are called -without any need to configure anything. But if you have written (or purchased) a -string generator module, you need to know how to call it. Each such module has a name, -which you need to know (look it up in the module doc or ask the developer). Let's assume -that "mystrgen" is the module name. Then you can define a template for that strgen -in the following way: +The various types are described below. -<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> -(Of course, you must have first loaded the module via $ModLoad). -<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that -no string follows but a strgen module name. -<p>There are no additional parameters but the module name supported. This is because -there is no way to customize anything inside such a "template" other than by -modifying the code of the string generator. +<h3>list</h3> +<p>In this case, the template is generated by a list of constant and +variable statements. These follow the template spec in curly braces. This type is +also primarily meant for use with structure-aware outputs, like ommongodb. However, +it also works perfectly with text-based outputs. We recommend to use this mode +if more complex property substitutions needs to be done. In that case, the list-based +template syntax is much clearer than the simple string-based one. +<p>The list template contains the template header (with <b>type="list"</b>) and is followed +by <b>constant</b> and <b>property</b> statements, given in curly braces to signify +the template statement they belong to. As the name says, <b>constant</b> statements +describe constant text and <b>property</b> describes property access. There are many options +to <b>property</b>, described further below. Most of these options are used to extract +only partial property contents or to modify the text obtained (like to change its case +to upper or lower case, only). +<p>To grasp the idea, an actual sample is: +<br><pre><code>template(name="tpl1" type="list") { + constant(value="Syslog MSG is: '") + property(name="msg") + constant(value="', ") + property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + constant(value="\n") + } +</code></pre> +<br>This sample is probably primarily targeted at the usual file-based output.</p> -<p>So for most use cases, string-generator module based templates are <b>not</b> -the route to take. Usually, we use <b>string based templates</b> instead. -This is what the rest of the documentation now talks about. -<p>A template consists of a template directive, a name, the -actual template text and optional options. A sample is:</p> -<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,"\7Text -%property% some more text\n",<options></code></blockquote> -<p>The "template" (legacy: $template) is the template directive. It tells rsyslog -that this line contains a template. "MyTemplateName" is the template -name. All -other config lines refer to this name. The text within "string" is the -actual template text. The backslash is an escape character, much as it -is in C. It does all these "cool" things. For example, \7 rings the -bell (this is an ASCII value), \n is a new line. C programmers and perl -coders have the advantage of knowing this, but the set in rsyslog is a -bit restricted currently. -</p> -<p>All text in the template is used literally, except for things -within percent signs. These are properties and allow you access to the -contents of the syslog message. Properties are accessed via the -<a href="property_replacer.html">property replacer</a> -(nice name, huh) and it can do cool things, too. For -example, it can pick a substring or do date-specific formatting. More -on this is below, on some lines of the property replacer.<br> -<br> +<h4>constant statement</h4> +<p>This provides a way to specify constant text. The text is used literally. It is +primarily intended for text-based output, so that some constant text can be included. For +example, if a complex template is build for file output, one usually needs to finish it +by a newline, which can be introduced by a constant statement. Here is an actual sample +of that use case from the rsylsog testbench: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +}</code></pre> +The following escape sequences are recogniced inside the constant text: +<ul> +<li>\\ - single backslash +<li>\n - LF +<li>\ooo - (three octal digits) - represents character with this numerical value (e.g. \101 +equals "A"). Note that three +octal digits must be given (in contrast to some languagues, where between one and three are valid). +While we support octal notation, we recommend to use hex notation as this is better known. +<li>\xhh - (where h is a hex digit) - represents character with this numerical value (e.g. \x41 +equals "A"). Note that two hexadecimal digits must be given (in contrast to some languagues +where one or two are valid). +<li>... some others ... list needs to be extended +</ul> +<p>Note: if an unsupported character follows a backslash, this is treated as an error. Behaviour +is unpredictable in this case. +<p>To aid usage of the same template both for text-based outputs and structured ones, constant +text without an "outname" parameter will be ignored when creating the name/value tree +for structured outputs. So if you want to supply some constant text e.g. to mongodb, you must +include an outname, as can be seen here: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n" <b>outname="IWantThisInMyDB"</b>) +}</code></pre> + +The "constant" statement supports the following parameters: +<ul> +<li>value - the constant value to use +<li>outname - output field name (for structured outputs) +</ul> + + +<h4>property statement</h4> +<p>This statement is used to include property text. It can access all properties. Also, +options permit to specify picking only part of a property or modifying it. +It supports the following parameters: +<ul> +<li>name - the name of the property to access +<li>outname - output field name (for structured outputs) +<li>dateformat - date format to use (only for date-related properties) +<li>caseconversion - permits to convert case of the text. supported values are +"lower" and "upper" +<li>controlcharacters - specifies how to handle control characters. Supported values are +"escape", which escapes them, "space", which replaces them by a single space, and +"drop", which simply removes them from the string. +<li>securepath - used for creating pathnames suitable for use in dynafile templates +<li>format - specifiy format on a field basis. Supported values are "csv", for use when +csv-data is generated, "json", which formats proper json content (but without a field +header) and "jsonf", which formats as a complete json field. +<li>position.from - obtain substring starting from this position (1 is the first position) +<li>position.to - obtain substring up to this position +<li>field.number - obtain this field match +<li>field.delimiter - decimal value of delimiter character for field extraction +<li>regex.expression - expression to use +<li>regex.type - either ERE or BRE +<li>regex.nomatchmode - what to do if we have no match +<li>regex.match - match to use +<li>regex.submatch - submatch to use +<li>droplastlf - drop a trailing LF, if it is present +<li>mandatory - signifies a field as mandatory. If set to "on", this field will always +be present in data passed to structured outputs, even if it is empty. If "off" (the default) +empty fields will not be passed to structured outputs. This is especially useful for outputs +that support dynamic schemas (like ommongodb). +<li>spifno1stsp - expert options for RFC3164 template processing +</ul> + + +<h3>subtree</h3> +<p>Available since rsyslog 7.1.4 +<p> +In this case, the template is generated based on a complete +(CEE) subtree. This type of template is most useful for outputs that know how to +process hierarchical structure, like ommongodb. With that type, the parameter +<b>subtree</b> must be specified, which tells which subtree to use. For example +template(name="tpl1" type="subtree" subtree="$!") includes all CEE data, while +template(name="tpl2" type="subtree" subtree="$!usr!tpl2") includes only the +subtree starting at $!usr!tpl2. The core idea when using this type of template +is that the actual data is prefabricated via set and unset script statements, +and the resulting strucuture is then used inside the template. This method MUST +be used if a complete subtree needs to be placed <i>directly</i> into the +object's root. With all other template types, only subcontainers can be generated. +Note that subtree type can also be used with text-based outputs, like omfile. HOWEVER, +you do not have any capability to specify constant text, and as such cannot include +line breaks. As a consequence, using this template type for text outputs is usually +only useful for debugging or very special cases (e.g. where the text is interpreted +by a JSON parser later on). +<h4>Use case</h4> +<p>A typical use case is to first create a custom subtree and then include it into +the template, like in this small example: +<br><blockquote><code>set $!usr!tpl2!msg = $msg; +<br>set $!usr!tpl2!dataflow = field($msg, 58, 2); +<br>template(name="tpl2" type="subtree" subtree="$!usr!tpl2") +</code></blockquote> +<p>Here, we assume that $msg contains various fields, and the data from a field +is to be extracted and stored - together with the message - as field content. +<h3>string</h3> +<p>This closely resembles the legacy template statement. It +has a mandatory parameter <b>string</b>, which holds the template string to be +applied. A template string is a mix of constant text and replacement variables +(see property replacer). These variables are taken from message or other dynamic +content when the final string to be passed to a plugin is generated. String-based +templates are a great way to specify textual content, especially if no complex +manipulation to properties is necessary. Full details on how to specify template +text can be found below. +<br>Config example: +<br><blockquote><code>template(name="tpl3" type="string" string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") +</code></blockquote> +<h3>plugin</h3> +In this case, the template is generated by a plugin (which +is then called +a "strgen" or "string generator"). The format is fix as it is coded. While this +is inflexible, it provides superior performance, and is often used for that +reason (not that "regular" templates are slow - but in very demanding environments +that "last bit" can make a difference). Refer to the plugin's documentation +for further details. For this type, the paramter <b>plugin</b> must be specified and +must contain the name of the plugin as it identifies itself. Note that the +plugin must be loaded prior to being used inside a template. +<br>Config example: +<br><blockquote><code>template(name="tpl4" type="plugin" plugin="mystrgen") +</code></blockquote> + +<h3>options</h3> The <options> part is optional. It carries options -influencing the template as whole. See details below. Be sure NOT to -mistake template options with property options - the latter ones are -processed by the property replacer and apply to a SINGLE property, only -(and not the whole template).<br> +influencing the template as whole and is part of the template parameters. +See details below. Be sure NOT to mistake template options with property +options - the latter ones are processed by the property replacer and +apply to a SINGLE property, only (and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> <p><b>option.sql</b> - format the string suitable for a SQL @@ -127,50 +260,102 @@ option. Otherwise you will become vulnerable to SQL injection. <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> +template (name="TraditionalFormat" type="string" string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> <br> -Properties can be accessed by the <a href="property_replacer.html">property -replacer</a> (see there for details).</p> -<p>Templates can be used in the form of a <b>list</b> as well. This has been -introduced with <b>6.5.0</b> The list consists of two parts which are either -a <b>constant</b> or a <b>property</b>. The constants -are taking the part of "text" that you usually enter in string-based templates. -The properties stay variable, as they are a substitute for different values of a -certain type. This type of template is extremely useful for complicated cases, -as it helps you to easily keep an overview over the template. Though, it has -the disadvantage of needing more effort to create it.</p> -<br>Config example: -<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { - <br>constant(value="Test: ") - <br>property(name="msg" outname="mymessage") - <br>constant(value=" --!!!-- ") - <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") - <br>constant(value="\n") - <br>} -</code></blockquote> -<p>First, the general template option will be defined. The values of the template -itself get defined in the curly brackets. As it can be seen, we have constants -and properties in exchange. Whereas constants will be filled with a value and probably -some options, properties do direct to a property and the options that could be needed -additional format definitions.</p> -<p>We suggest to use separate lines for all constants and properties. This -helps to keep a good overview over the different parts of the template. -Though, writing it in a single line will work, it is much harder to debug -if anything goes wrong with the template. </p> +<h3>Examples</h3> +<h4>Standard Template for Writing to Files</h4> +<p><pre><code>template(name="FileFormat" type="list") { + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + property(name="msg" droplastlf="on" ) + constant(value="\n") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="FileFormat" type="string" + string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +)</code></pre> +Note that the template string itself must be on a single line. -<p><b>Please note that templates can also be -used to generate selector lines with dynamic file names.</b> For -example, if you would like to split syslog messages from different -hosts to different files (one per host), you can define the following -template:</p> -<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template -DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> -<p>This template can then be used when defining an output -selector line. It will result in something like -"/var/log/system-localhost.log"</p> +<h4>Standard Template for Forwarding to a Remote Host (RFC3164 mode)</h4> +<p><pre><code>template(name="ForwardFormat" type="list") { + constant(value="<") + property(name="PRI") + constant(value="<") + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag" position.from="1" position.to="32") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="forwardFormat" type="string" + string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +)</code></pre> +Note that the template string itself must be on a single line. + +<h4>Standard Template for write to the MySQL database</h4> +<p><pre><code>template(name="StdSQLformat" type="list" option.sql="on") { + constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") + constant(value=" values ('") + property(name="msg") + constant(value="', ") + property(name="syslogfacility") + constant(value=", '") + property(name="hostname") + constant(value="', ") + property(name="syslogpriority") + constant(value=", '") + property(name="timereported" dateFormat="mysql") + constant(value="', '") + property(name="timegenerated" dateFormat="mysql") + constant(value="', ") + property(name="iut") + constant(value=", '") + property(name="syslogtag") + constant(value="')") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="stdSQLformat" type="string" option.sql="on" + string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')" +)</code></pre> +Note that the template string itself must be on a single line. + +<h2>legacy format</h2> +<p>In pre v6-versions of rsyslog, you need to use the <code>$template</code> +statement to configure templates. They provide the equivalent to string- and +plugin-based templates. The legacy syntax continous to work in v7, however +we recommend to avoid legacy format for newly written config files. Legacy and +current config statements can coexist within the same config file. +<p>The general format is +<br><br><code>$template name,param[,options]</code></br></br> +where "name" is the template name and "param" is a single parameter +that specifies template content. The optional "options" part is used to +set template options. +<h3>string</h3> +The parameter is the same string that with the current-style format you +specify in the <b>string</b> parameter, for example: +<br><br><code>$template strtpl,"PRI: %pri%, MSG: %msg%\n"</code> +<p>Note that list templates are not available in legacy format, so you need +to use complex property replacer constructs to do complex things. + +<h3>plugin</h3> +This is equivalent to the "plugin"-type template directive. Here, the +parameter is the plugin name, with an equal sign prepended. An example +is: +<br><br><code>$template plugintpl,=myplugin</code> + +<h2>Reserved Template Names</h2> <p>Template names beginning with "RSYSLOG_" are reserved for rsyslog use. Do NOT use them if, otherwise you may receive a conflict in the future (and @@ -210,12 +395,122 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> + +<h2>The following is legacy documentation soon to be integrated.</h2> + +<!--<table> +<tr><td>param name</td><td>meaning</td></tr> +<tr><td>name</td><td>name of the template</td></tr> +</table> +--> + +<p>Starting with 5.5.6, there are actually two different types of template: +<ul> +<li>string based +<li>string-generator module based +</ul> +<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: + +<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> +(Of course, you must have first loaded the module via $ModLoad). +<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that +no string follows but a strgen module name. +<p>There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +<p>So for most use cases, string-generator module based templates are <b>not</b> +the route to take. Usually, we use <b>string based templates</b> instead. +This is what the rest of the documentation now talks about. + +<p>A template consists of a template directive, a name, the +actual template text and optional options. A sample is:</p> +<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,"\7Text +%property% some more text\n",<options></code></blockquote> +<p>The "template" (legacy: $template) is the template directive. It tells rsyslog +that this line contains a template. "MyTemplateName" is the template +name. All +other config lines refer to this name. The text within "string" is the +actual template text. The backslash is an escape character, much as it +is in C. It does all these "cool" things. For example, \7 rings the +bell (this is an ASCII value), \n is a new line. C programmers and perl +coders have the advantage of knowing this, but the set in rsyslog is a +bit restricted currently. +</p> +<p>All text in the template is used literally, except for things +within percent signs. These are properties and allow you access to the +contents of the syslog message. Properties are accessed via the +<a href="property_replacer.html">property replacer</a> +(nice name, huh) and it can do cool things, too. For +example, it can pick a substring or do date-specific formatting. More +on this is below, on some lines of the property replacer.<br> +<br> + +<br> +Properties can be accessed by the <a href="property_replacer.html">property +replacer</a> (see there for details).</p> +<p>Templates can be used in the form of a <b>list</b> as well. This has been +introduced with <b>6.5.0</b> The list consists of two parts which are either +a <b>constant</b> or a <b>property</b>. The constants +are taking the part of "text" that you usually enter in string-based templates. +The properties stay variable, as they are a substitute for different values of a +certain type. This type of template is extremely useful for complicated cases, +as it helps you to easily keep an overview over the template. Though, it has +the disadvantage of needing more effort to create it.</p> +<br>Config example: +<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { + <br>constant(value="Test: ") + <br>property(name="msg" outname="mymessage") + <br>constant(value=" --!!!-- ") + <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + <br>constant(value="\n") + <br>} +</code></blockquote> +<p>First, the general template option will be defined. The values of the template +itself get defined in the curly brackets. As it can be seen, we have constants +and properties in exchange. Whereas constants will be filled with a value and probably +some options, properties do direct to a property and the options that could be needed +additional format definitions.</p> +<p>We suggest to use separate lines for all constants and properties. This +helps to keep a good overview over the different parts of the template. +Though, writing it in a single line will work, it is much harder to debug +if anything goes wrong with the template. </p> + +<p><b>Please note that templates can also be +used to generate selector lines with dynamic file names.</b> For +example, if you would like to split syslog messages from different +hosts to different files (one per host), you can define the following +template:</p> +<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template +DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> +<p>This template can then be used when defining an output +selector line. It will result in something like +"/var/log/system-localhost.log"</p> <h3>Legacy String-based Template Samples</h3> -<p>This section provides some sample of what the default formats would -look as a text-based template. Hopefully, their description is self-explanatory. +<p>This section provides some default templates in legacy format, as used in rsyslog +previous to version 6. Note that this format is still supported, so there is no hard need +to upgrade existing configurations. However, it is strongly recommended that the legacy +constructs are not used when crafting new templates. Note that each $Template statement is on a <b>single</b> line, but probably broken accross several lines for display purposes by your browsers. Lines are separated by -empty lines. +empty lines. Keep in mind, that line breaks are important in legacy format. <p><code> $template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" <br><br> @@ -233,7 +528,7 @@ $template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, P [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> diff --git a/doc/v4compatibility.html b/doc/v4compatibility.html index 72b0f5a9..2a51adea 100644 --- a/doc/v4compatibility.html +++ b/doc/v4compatibility.html @@ -60,7 +60,7 @@ restarting rsyslogd by HUPing it. and most other deamons require that a restart command is typed in if a restart is required. <p>Rsyslog will follow this paradigm in the next versions, resulting in many benefits. In v4, we provide some support for the old-style semantics. We introduced a setting $HUPisRestart -which may be set to "on" (tradional, heavy operationg) +which may be set to "on" (tradional, heavy operation) or "off" (new, lightweight "file close only" operation). The initial versions had the default set to traditional behavior, but starting with 4.5.1 we are now using the new behavior as the default. diff --git a/doc/v7compatibility.html b/doc/v7compatibility.html new file mode 100644 index 00000000..da4772fe --- /dev/null +++ b/doc/v7compatibility.html @@ -0,0 +1,138 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v7</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v7</h1> +This document describes things to keep in mind when moving from v6 to v7. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used v6 and want to use v7 without hassle. +<p>Version 7 builds on the new config language introduced in v6 and extends it. +Other than v6, it not just only extends the config language, but provides +considerable changes to core elements as well. The result is much more power and +ease of use as well (this time that is not contradictionary). +</p> +<h2>BSD-Style blocks</h2> +BSD style blocks are no longer supported (for good reason). See the +<a href="http://www.rsyslog.com/g/BSD">rsyslog BSD blocks info</a> +page for more information and how to upgrade your config. +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + +<h2>CEE-Properties</h2> +In rsyslog v6, CEE properties could not be used across disk-based queues. If this was +done, there content was reset. This was a missing feature in v6. In v7, this feature +has been implemented. Consequently, situations where the previous behaviour were +desired need now to be solved differently. We do not think that this will cause any +problems to anyone, especially as in v6 this was announced as a missing feature. + +<h2>omusrmsg: using just a username or "*" is deprecated</h2> +<p>In legacy config format, the asterisk denotes writing the message to all users. +This is usually used for emergency messages and configured like this: +<pre> +*.emerg * +</pre> +<p>Unfortunately, the use of this single character conflicts with other uses, for +example with the multiplication operator. While rsyslog up to versions v7.4 preserves the meaning of +asterisk as an action, it is deprecated and will probably be removed in future versions. +Consequently, a warning message is emitted. To make this warning go away, the action must +be explicitly given, as follows: +<pre> +*.emerg :omusrmsg:* +</pre> +<p>The same holds true for user names. For example +<pre> +*.emerg john +</pre> +<p>at a minimum should be rewritten as +<pre> +*.emerg :omusrmsg:john +</pre> +<p>Of course, for even more clarity the new RainerScript style of action can +also be used: +<pre> +*.emerg action(type="omusrmsg" users="john") +</pre> +<p>In Rainer's blog, there is more +<a href="http://blog.gerhards.net/2011/07/why-omusrmsg-is-evil-and-how-it-is.html">background +information on why omusrmsg needed to be changed</a> available. + +<h2>omruleset and discard (~) action are deprecated</h2> +<p>Both continue to work, but have been replaced by better alternatives. +<p>The discard action (tilde character) has been replaced by the "stop" +RainerScript directive. It is considered more intuitive and offers slightly +better performance. +<p>The omruleset module has been replaced by the "call" RainerScript directive. +Call permits to execute a ruleset like a subroutine, and does so with much +higher performance than omruleset did. Note that omruleset could be run off +an async queue. This was more a side than a desired effect and is not supported +by the call statement. If that effect was needed, it can simply be simulated by +running the called rulesets actions asynchronously (what in any case is the right +way to handle this). +<p>Note that the deprecated modules emit warning messages when being used. +They tell that the construct is deprecated and which statement is to be used +as replacement. This does <b>not</b> affect operations: both modules are still +fully operational and will not be removed in the v7 timeframe. + +<h2>Retries of output plugins that do not do proper replies</h2> +<p>Some output plugins may not be able to detect if their target is capable of +accepting data again after an error (technically, they always return OK when +TryResume is called). Previously, the rsyslog core engine suspended such an action +after 1000 succesive failures. This lead to potentially a large amount of +errors and error messages. Starting with 7.2.1, this has been reduced to 10 +successive failures. This still gives the plugin a chance to recover. In extreme +cases, a plugin may now enter suspend mode where it previously did not do so. +In practice, we do NOT expect that. +<h1>Notes for the 7.3/7.4 branch</h1> +<h2>"last message repeated n times" Processing</h2> +<p>This processing has been optimized and moved to the input side. This results +in usually far better performance and also de-couples different sources +from the same +processing. It is now also integrated in to the more generic rate-limiting +processing. +<h3>User-Noticable Changes</h3> +The code works almost as before, with two exceptions: +<ul> +<li>The supression amount can be different, as the new algorithm + precisely check's a single source, and while that source is being + read. The previous algorithm worked on a set of mixed messages + from multiple sources. +<li>The previous algorithm wrote a "last message repeated n times" message + at least every 60 seconds. For performance reasons, we do no longer do + this but write this message only when a new message arrives or rsyslog + is shut down. +</ul> +<p>Note that the new algorithms needs support from input modules. If old +modules which do not have the necessary support are used, duplicate +messages will most probably not be detected. Upgrading the module code is +simple, and all rsyslog-provided plugins support the new method, so this +should not be a real problem (crafting a solution would result in rather +complex code - for a case that most probably would never happen). +<h3>Performance Implications</h3> +<p>In general, the new method enables far faster output procesing. However, it +needs to be noted that the "last message repeated n" processing needs parsed +messages in order to detect duplicated. Consequently, if it is enabled the +parser step cannot be deferred to the main queue processing thread and +thus must be done during input processing. The changes workload distribution +and may have (good or bad) effect on the overall performance. If you have +a very high performance installation, it is suggested to check the performance +profile before deploying the new version. Note: for high-performance +environments it is highly recommended NOT to use "last message repeated n times" +processing but rather the other (more efficient) rate-limiting methods. These +also do NOT require the parsing step to be done during input processing. + +<h2>Stricter string-template Processing</h2> +<p>Previously, no error message for invalid string template parameters +was generated. +Rather a malformed template was generated, and error information emitted +at runtime. However, this could be quite confusing. Note that the new code +changes user experience: formerly, rsyslog and the affected +actions properly started up, but the actions did not produce proper +data. Now, there are startup error messages and the actions are NOT +executed (due to missing template due to template error). + +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2011-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body></html> diff --git a/grammar/grammar.y b/grammar/grammar.y index cdb19c3d..c5bad689 100644 --- a/grammar/grammar.y +++ b/grammar/grammar.y @@ -6,10 +6,9 @@ * of course, encouraged to use new constructs only. But it needs to be noted * that some of the legacy constructs (specifically the in-front-of-action * PRI filter) are very hard to beat in ease of use, at least for simpler - * cases. So while we hope that cfsysline support can be dropped some time in - * the future, we will probably keep these useful constructs. + * cases. * - * Copyright 2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -37,7 +36,7 @@ #define YYDEBUG 1 extern int yylineno; -/* keep compile rule cleam of errors */ +/* keep compile rule clean of errors */ extern int yylex(void); extern int yyerror(char*); %} @@ -48,26 +47,29 @@ extern int yyerror(char*); es_str_t *estr; enum cnfobjType objType; struct cnfobj *obj; + struct cnfstmt *stmt; struct nvlst *nvlst; struct objlst *objlst; - struct cnfactlst *actlst; struct cnfexpr *expr; - struct cnfrule *rule; + struct cnfarray *arr; struct cnffunc *func; struct cnffparamlst *fparams; } %token <estr> NAME -%token <estr> VALUE %token <estr> FUNC %token <objType> BEGINOBJ %token ENDOBJ -%token <s> CFSYSLINE %token BEGIN_ACTION %token BEGIN_PROPERTY %token BEGIN_CONSTANT %token BEGIN_TPL +%token BEGIN_RULESET %token STOP +%token SET +%token UNSET +%token CONTINUE +%token <cnfstmt> CALL %token <s> LEGACY_ACTION %token <s> LEGACY_RULESET %token <s> PRIFILT @@ -76,6 +78,7 @@ extern int yyerror(char*); %token <s> BSD_HOST_SELECTOR %token IF %token THEN +%token ELSE %token OR %token AND %token NOT @@ -93,33 +96,23 @@ extern int yyerror(char*); %token CMP_STARTSWITH %token CMP_STARTSWITHI -%type <nvlst> nv nvlst +%type <nvlst> nv nvlst value %type <obj> obj property constant %type <objlst> propconst -%type <actlst> actlst -%type <actlst> act -%type <s> cfsysline -%type <actlst> block %type <expr> expr -%type <rule> rule -%type <rule> scriptfilt +%type <stmt> stmt s_act actlst block script %type <fparams> fparams +%type <arr> array arrayelt %left AND OR %left CMP_EQ CMP_NE CMP_LE CMP_GE CMP_LT CMP_GT CMP_CONTAINS CMP_CONTAINSI CMP_STARTSWITH CMP_STARTSWITHI -%left '+' '-' +%left '+' '-' '&' %left '*' '/' '%' %nonassoc UMINUS NOT -%expect 3 -/* these shift/reduce conflicts are created by the CFSYSLINE construct, which we - * unfortunately can not avoid. The problem is that CFSYSLINE can occur both in - * global context as well as within an action. It's not permitted somewhere else, - * but this is suficient for conflicts. The "dangling else" built-in resolution - * works well to solve this issue, so we accept it (it's a wonder that our - * old style grammar doesn't work at all, so we better do not complain...). - * Use "bison -v rscript.y" if more conflicts arise and check rscript.out for - * were exactly these conflicts exits. +%expect 1 /* dangling else */ +/* If more erors show up, Use "bison -v grammar.y" if more conflicts arise and + * check grammar.output for were exactly these conflicts exits. */ %% /* note: we use left recursion below, because that saves stack space AND @@ -128,41 +121,55 @@ extern int yyerror(char*); */ conf: /* empty (to end recursion) */ | conf obj { cnfDoObj($2); } - | conf rule { cnfDoRule($2); } - | conf cfsysline { cnfDoCfsysline($2); } + | conf stmt { cnfDoScript($2); } | conf LEGACY_RULESET { cnfDoCfsysline($2); } | conf BSD_TAG_SELECTOR { cnfDoBSDTag($2); } | conf BSD_HOST_SELECTOR { cnfDoBSDHost($2); } obj: BEGINOBJ nvlst ENDOBJ { $$ = cnfobjNew($1, $2); } - | BEGIN_ACTION nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_ACTION, $2); } | BEGIN_TPL nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_TPL, $2); } | BEGIN_TPL nvlst ENDOBJ '{' propconst '}' { $$ = cnfobjNew(CNFOBJ_TPL, $2); $$->subobjs = $5; } + | BEGIN_RULESET nvlst ENDOBJ '{' script '}' + { $$ = cnfobjNew(CNFOBJ_RULESET, $2); + $$->script = $5; + } propconst: { $$ = NULL; } | propconst property { $$ = objlstAdd($1, $2); } | propconst constant { $$ = objlstAdd($1, $2); } property: BEGIN_PROPERTY nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_PROPERTY, $2); } constant: BEGIN_CONSTANT nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_CONSTANT, $2); } -cfsysline: CFSYSLINE { $$ = $1; } nvlst: { $$ = NULL; } | nvlst nv { $2->next = $1; $$ = $2; } -nv: NAME '=' VALUE { $$ = nvlstNew($1, $3); } -rule: PRIFILT actlst { $$ = cnfruleNew(CNFFILT_PRI, $2); $$->filt.s = $1; } - | PROPFILT actlst { $$ = cnfruleNew(CNFFILT_PROP, $2); $$->filt.s = $1; } - | scriptfilt { $$ = $1; } - -scriptfilt: IF expr THEN actlst { $$ = cnfruleNew(CNFFILT_SCRIPT, $4); - $$->filt.expr = $2; } -block: actlst { $$ = $1; } - | block actlst { $2->next = $1; $$ = $2; } -actlst: act { $$=$1; } - | actlst '&' act { $3->next = $1; $$ = $3; } - | actlst cfsysline { $$ = cnfactlstAddSysline($1, $2); } - | '{' block '}' { $$ = $2; } -act: BEGIN_ACTION nvlst ENDOBJ { $$ = cnfactlstNew(CNFACT_V2, $2, NULL); } - | LEGACY_ACTION { $$ = cnfactlstNew(CNFACT_LEGACY, NULL, $1); } +nv: NAME '=' value { $$ = nvlstSetName($3, $1); } +value: STRING { $$ = nvlstNewStr($1); } + | array { $$ = nvlstNewArray($1); } +script: stmt { $$ = $1; } + | script stmt { $$ = scriptAddStmt($1, $2); } +stmt: actlst { $$ = $1; } + | IF expr THEN block { $$ = cnfstmtNew(S_IF); + $$->d.s_if.expr = $2; + $$->d.s_if.t_then = $4; + $$->d.s_if.t_else = NULL; } + | IF expr THEN block ELSE block { $$ = cnfstmtNew(S_IF); + $$->d.s_if.expr = $2; + $$->d.s_if.t_then = $4; + $$->d.s_if.t_else = $6; } + | SET VAR '=' expr ';' { $$ = cnfstmtNewSet($2, $4); } + | UNSET VAR ';' { $$ = cnfstmtNewUnset($2); } + | PRIFILT block { $$ = cnfstmtNewPRIFILT($1, $2); } + | PROPFILT block { $$ = cnfstmtNewPROPFILT($1, $2); } +block: stmt { $$ = $1; } + | '{' script '}' { $$ = $2; } +actlst: s_act { $$ = $1; } + | actlst '&' s_act { $$ = scriptAddStmt($1, $3); } +/* s_act are actions and action-like statements */ +s_act: BEGIN_ACTION nvlst ENDOBJ { $$ = cnfstmtNewAct($2); } + | LEGACY_ACTION { $$ = cnfstmtNewLegaAct($1); } + | STOP { $$ = cnfstmtNew(S_STOP); } + | CALL NAME { $$ = cnfstmtNewCall($2); } + | CONTINUE { $$ = cnfstmtNewContinue(); } expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | expr OR expr { $$ = cnfexprNew(OR, $1, $3); } | NOT expr { $$ = cnfexprNew(NOT, NULL, $2); } @@ -176,6 +183,7 @@ expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | expr CMP_CONTAINSI expr { $$ = cnfexprNew(CMP_CONTAINSI, $1, $3); } | expr CMP_STARTSWITH expr { $$ = cnfexprNew(CMP_STARTSWITH, $1, $3); } | expr CMP_STARTSWITHI expr { $$ = cnfexprNew(CMP_STARTSWITHI, $1, $3); } + | expr '&' expr { $$ = cnfexprNew('&', $1, $3); } | expr '+' expr { $$ = cnfexprNew('+', $1, $3); } | expr '-' expr { $$ = cnfexprNew('-', $1, $3); } | expr '*' expr { $$ = cnfexprNew('*', $1, $3); } @@ -188,8 +196,12 @@ expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | NUMBER { $$ = (struct cnfexpr*) cnfnumvalNew($1); } | STRING { $$ = (struct cnfexpr*) cnfstringvalNew($1); } | VAR { $$ = (struct cnfexpr*) cnfvarNew($1); } + | array { $$ = (struct cnfexpr*) $1; } fparams: expr { $$ = cnffparamlstNew($1, NULL); } | expr ',' fparams { $$ = cnffparamlstNew($1, $3); } +array: '[' arrayelt ']' { $$ = $2; } +arrayelt: STRING { $$ = cnfarrayNew($1); } + | arrayelt ',' STRING { $$ = cnfarrayAdd($1, $3); } %% /* diff --git a/grammar/lexer.l b/grammar/lexer.l index 86fd97b4..e1f5a9c3 100644 --- a/grammar/lexer.l +++ b/grammar/lexer.l @@ -45,6 +45,8 @@ /* INCL is in $IncludeConfig processing (skip to include file) */ %x LINENO /* LINENO: support for setting the linenumber */ +%x INCALL + /* INCALL: support for the call statement */ %x EXPR /* EXPR is a bit ugly, but we need it to support pre v6-syntax. The problem * is that cfsysline statement start with $..., the same like variables in @@ -86,7 +88,6 @@ extern int yydebug; /* somehow, I need these prototype even though the headers are * included. I guess that's some autotools magic I don't understand... */ -//char *strdup(char*); int fileno(FILE *stream); %} @@ -96,15 +97,20 @@ int fileno(FILE *stream); /* keywords */ "if" { BEGIN EXPR; return IF; } <EXPR>"then" { BEGIN INITIAL; return THEN; } +<EXPR>";" { BEGIN INITIAL; return ';'; } <EXPR>"or" { return OR; } <EXPR>"and" { return AND; } <EXPR>"not" { return NOT; } +<EXPR>"=" | <EXPR>"," | <EXPR>"*" | <EXPR>"/" | <EXPR>"%" | <EXPR>"+" | +<EXPR>"&" | <EXPR>"-" | +<EXPR>"[" | +<EXPR>"]" | <EXPR>"(" | <EXPR>")" { return yytext[0]; } <EXPR>"==" { return CMP_EQ; } @@ -121,7 +127,7 @@ int fileno(FILE *stream); <EXPR>0[0-7]+ | /* octal number */ <EXPR>0x[0-7a-f] | /* hex number, following rule is dec; strtoll handles all! */ <EXPR>([1-9][0-9]*|0) { yylval.n = strtoll(yytext, NULL, 0); return NUMBER; } -<EXPR>\$[$!]{0,1}[a-z][a-z0-9\-_\.]* { yylval.s = strdup(yytext); return VAR; } +<EXPR>\$[$!]{0,1}[a-z][!a-z0-9\-_\.]* { yylval.s = strdup(yytext); return VAR; } <EXPR>\'([^'\\]|\\['"\\$bntr]|\\x[0-9a-f][0-9a-f]|\\[0-7][0-7][0-7])*\' { yytext[yyleng-1] = '\0'; unescapeStr((uchar*)yytext+1, yyleng-2); @@ -135,15 +141,28 @@ int fileno(FILE *stream); <EXPR>[ \t\n] <EXPR>[a-z][a-z0-9_]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); return FUNC; } -<EXPR>. { dbgprintf("invalid char in expr: %s\n", yytext); } +<EXPR>. { parser_errmsg("invalid character '%s' in expression " + "- is there an invalid escape sequence somewhere?", + yytext); } +<INCALL>[ \t\n] +<INCALL>. { parser_errmsg("invalid character '%s' in 'call' statement" + "- is there an invalid escape sequence somewhere?", + yytext); } +<INCALL>[a-zA-Z][a-zA-Z0-9_\.]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); + BEGIN INITIAL; + return NAME; } "&" { return '&'; } "{" { return '{'; } "}" { return '}'; } -"ruleset" { dbgprintf("RULESET\n"); } +"stop" { return STOP; } +"else" { return ELSE; } +"call" { BEGIN INCALL; return CALL; } +"set" { BEGIN EXPR; return SET; } +"unset" { BEGIN EXPR; return UNSET; } +"continue" { return CONTINUE; } /* line number support because the "preprocessor" combines lines and so needs * to tell us the real source line. */ -"stop" { dbgprintf("STOP\n"); return STOP; } "preprocfilelinenumber(" { BEGIN LINENO; } <LINENO>[0-9]+ { yylineno = atoi(yytext) - 1; } <LINENO>")" { BEGIN INITIAL; } @@ -159,6 +178,8 @@ int fileno(FILE *stream); BEGIN INOBJ; return BEGINOBJ; } "template"[ \n\t]*"(" { yylval.objType = CNFOBJ_TPL; BEGIN INOBJ; return BEGIN_TPL; } +"ruleset"[ \n\t]*"(" { yylval.objType = CNFOBJ_RULESET; + BEGIN INOBJ; return BEGIN_RULESET; } "property"[ \n\t]*"(" { yylval.objType = CNFOBJ_PROPERTY; BEGIN INOBJ; return BEGIN_PROPERTY; } "constant"[ \n\t]*"(" { yylval.objType = CNFOBJ_CONSTANT; @@ -168,25 +189,33 @@ int fileno(FILE *stream); "module"[ \n\t]*"(" { yylval.objType = CNFOBJ_MODULE; BEGIN INOBJ; return BEGINOBJ; } "action"[ \n\t]*"(" { BEGIN INOBJ; return BEGIN_ACTION; } -^[ \t]*:\$?[a-z\-]+[ ]*,[ ]*!?[a-z]+[ ]*,[ ]*\".*\" { - yylval.s = strdup(yytext); return PROPFILT; } -^[ \t]*[\*a-z][\*a-z]*[0-7]*[\.,][,!=;\.\*a-z0-7]+ { yylval.s = strdup(yytext); return PRIFILT; } +^[ \t]*:\$?[a-z\-]+[ ]*,[ ]*!?[a-z]+[ ]*,[ ]*\"(\\\"|[^\"])*\" { + yylval.s = strdup(rmLeadingSpace(yytext)); + dbgprintf("lexer: propfilt is '%s'\n", yylval.s); + return PROPFILT; + } +^[ \t]*[\*a-z][\*a-z]*[0-7]*[\.,][,!=;\.\*a-z0-7]+ { yylval.s = strdup(rmLeadingSpace(yytext)); return PRIFILT; } "~" | "*" | \-\/[^*][^\n]* | \/[^*][^\n]* | :[a-z0-9]+:[^\n]* | [\|\.\-\@\^?~>][^\n]+ | -[a-z0-9_][a-z0-9_\-\+,;]* { yylval.s = strdup(yytext); - dbgprintf("lex: LEGA ACT: '%s'\n", yytext); - return LEGACY_ACTION; } +[a-z0-9_][a-z0-9_\-\+,;]* { yylval.s = yytext; return LEGACY_ACTION; } <INOBJ>")" { BEGIN INITIAL; return ENDOBJ; } <INOBJ>[a-z][a-z0-9_\.]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); return NAME; } +<INOBJ>"," | +<INOBJ>"[" | +<INOBJ>"]" | <INOBJ>"=" { return(yytext[0]); } <INOBJ>\"([^"\\]|\\['"?\\abfnrtv]|\\[0-7]{1,3})*\" { - yylval.estr = es_newStrFromBuf(yytext+1, yyleng-2); - return VALUE; } + yytext[yyleng-1] = '\0'; + unescapeStr((uchar*)yytext+1, yyleng-2); + yylval.estr = es_newStrFromBuf(yytext+1, strlen(yytext)-1); + return STRING; } + /*yylval.estr = es_newStrFromBuf(yytext+1, yyleng-2); + return VALUE; }*/ "/*" { preCommentState = YY_START; BEGIN COMMENT; } <INOBJ>"/*" { preCommentState = YY_START; BEGIN COMMENT; } <EXPR>"/*" { preCommentState = YY_START; BEGIN COMMENT; } @@ -194,7 +223,9 @@ int fileno(FILE *stream); <COMMENT>([^*]|\n)+|. <INOBJ>#.*$ /* skip comments in input */ <INOBJ>[ \n\t] -<INOBJ>. { dbgprintf("INOBJ: invalid char '%s'\n", yytext); } +<INOBJ>. { parser_errmsg("invalid character '%s' in object definition " + "- is there an invalid escape sequence somewhere?", + yytext); } \$[a-z]+.*$ { /* see comment on $IncludeConfig above */ if(!strncasecmp(yytext, "$includeconfig ", 14)) { yyless(14); @@ -203,8 +234,7 @@ int fileno(FILE *stream); yylval.s = strdup(yytext); return LEGACY_RULESET; } else { - yylval.s = strdup(yytext); - return CFSYSLINE; + cnfDoCfsysline(strdup(yytext)); } } ![^ \t\n]+[ \t]*$ { yylval.s = strdup(yytext); return BSD_TAG_SELECTOR; } @@ -213,8 +243,9 @@ int fileno(FILE *stream); ^[ \t]*[+-][a-z0-9.:-]+[ \t]*$ { yylval.s = strdup(yytext); return BSD_HOST_SELECTOR; } \#.*\n /* skip comments in input */ [\n\t ] /* drop whitespace */ -. { dbgprintf("invalid char: %s\n", yytext); - } +. { parser_errmsg("invalid character '%s' " + "- is there an invalid escape sequence somewhere?", + yytext); } <<EOF>> { if(popfile() != 0) yyterminate(); } %% diff --git a/grammar/parserif.h b/grammar/parserif.h index 597cfe40..aa271ec4 100644 --- a/grammar/parserif.h +++ b/grammar/parserif.h @@ -15,7 +15,7 @@ extern int yylineno; * these functions. */ void cnfDoObj(struct cnfobj *o); -void cnfDoRule(struct cnfrule *rule); +void cnfDoScript(struct cnfstmt *script); void cnfDoCfsysline(char *ln); void cnfDoBSDTag(char *ln); void cnfDoBSDHost(char *ln); diff --git a/grammar/rainerscript.c b/grammar/rainerscript.c index 072bdd5d..6dbdad63 100644 --- a/grammar/rainerscript.c +++ b/grammar/rainerscript.c @@ -2,7 +2,7 @@ * * Module begun 2011-07-01 by Rainer Gerhards * - * Copyright 2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -22,7 +22,6 @@ * A copy of the GPL can be found in the file "COPYING" in this distribution. * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. */ - #include "config.h" #include <stdio.h> #include <stdlib.h> @@ -37,16 +36,206 @@ #include <libestr.h> #include "rsyslog.h" #include "rainerscript.h" +#include "conf.h" #include "parserif.h" +#include "rsconf.h" #include "grammar.h" #include "queue.h" #include "srUtils.h" #include "regexp.h" #include "obj.h" +#include "modules.h" +#include "ruleset.h" DEFobjCurrIf(obj) DEFobjCurrIf(regexp) +struct cnfexpr* cnfexprOptimize(struct cnfexpr *expr); +static void cnfstmtOptimizePRIFilt(struct cnfstmt *stmt); +static void cnfarrayPrint(struct cnfarray *ar, int indent); +struct cnffunc * cnffuncNew_prifilt(int fac); + +/* debug support: convert token to a human-readable string. Note that + * this function only supports a single thread due to a static buffer. + * This is deemed a solid solution, as it is intended to be used during + * startup, only. + * NOTE: This function MUST be updated if new tokens are defined in the + * grammar. + */ +char * +tokenToString(int token) +{ + char *tokstr; + static char tokbuf[512]; + + switch(token) { + case NAME: tokstr = "NAME"; break; + case FUNC: tokstr = "FUNC"; break; + case BEGINOBJ: tokstr ="BEGINOBJ"; break; + case ENDOBJ: tokstr ="ENDOBJ"; break; + case BEGIN_ACTION: tokstr ="BEGIN_ACTION"; break; + case BEGIN_PROPERTY: tokstr ="BEGIN_PROPERTY"; break; + case BEGIN_CONSTANT: tokstr ="BEGIN_CONSTANT"; break; + case BEGIN_TPL: tokstr ="BEGIN_TPL"; break; + case BEGIN_RULESET: tokstr ="BEGIN_RULESET"; break; + case STOP: tokstr ="STOP"; break; + case SET: tokstr ="SET"; break; + case UNSET: tokstr ="UNSET"; break; + case CONTINUE: tokstr ="CONTINUE"; break; + case CALL: tokstr ="CALL"; break; + case LEGACY_ACTION: tokstr ="LEGACY_ACTION"; break; + case LEGACY_RULESET: tokstr ="LEGACY_RULESET"; break; + case PRIFILT: tokstr ="PRIFILT"; break; + case PROPFILT: tokstr ="PROPFILT"; break; + case IF: tokstr ="IF"; break; + case THEN: tokstr ="THEN"; break; + case ELSE: tokstr ="ELSE"; break; + case OR: tokstr ="OR"; break; + case AND: tokstr ="AND"; break; + case NOT: tokstr ="NOT"; break; + case VAR: tokstr ="VAR"; break; + case STRING: tokstr ="STRING"; break; + case NUMBER: tokstr ="NUMBER"; break; + case CMP_EQ: tokstr ="CMP_EQ"; break; + case CMP_NE: tokstr ="CMP_NE"; break; + case CMP_LE: tokstr ="CMP_LE"; break; + case CMP_GE: tokstr ="CMP_GE"; break; + case CMP_LT: tokstr ="CMP_LT"; break; + case CMP_GT: tokstr ="CMP_GT"; break; + case CMP_CONTAINS: tokstr ="CMP_CONTAINS"; break; + case CMP_CONTAINSI: tokstr ="CMP_CONTAINSI"; break; + case CMP_STARTSWITH: tokstr ="CMP_STARTSWITH"; break; + case CMP_STARTSWITHI: tokstr ="CMP_STARTSWITHI"; break; + case UMINUS: tokstr ="UMINUS"; break; + default: snprintf(tokbuf, sizeof(tokbuf), "%c[%d]", token, token); + tokstr = tokbuf; break; + } + return tokstr; +} + + +char* +getFIOPName(unsigned iFIOP) +{ + char *pRet; + switch(iFIOP) { + case FIOP_CONTAINS: + pRet = "contains"; + break; + case FIOP_ISEQUAL: + pRet = "isequal"; + break; + case FIOP_STARTSWITH: + pRet = "startswith"; + break; + case FIOP_REGEX: + pRet = "regex"; + break; + case FIOP_EREREGEX: + pRet = "ereregex"; + break; + case FIOP_ISEMPTY: + pRet = "isempty"; + break; + default: + pRet = "NOP"; + break; + } + return pRet; +} + +static void +prifiltInvert(struct funcData_prifilt *prifilt) +{ + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + prifilt->pmask[i] = ~prifilt->pmask[i]; + } +} + +/* set prifilt so that it matches for some severities, sev is its numerical + * value. Mode is one of the compop tokens CMP_EQ, CMP_LT, CMP_LE, CMP_GT, + * CMP_GE, CMP_NE. + */ +static void +prifiltSetSeverity(struct funcData_prifilt *prifilt, int sev, int mode) +{ + static int lessthanmasks[] = { 0x00, 0x01, 0x03, 0x07, 0x0f, 0x1f, 0x3f, 0x7f, 0xff }; + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + if(mode == CMP_EQ || mode == CMP_NE) + prifilt->pmask[i] = 1 << sev; + else if(mode == CMP_LT) + prifilt->pmask[i] = lessthanmasks[sev]; + else if(mode == CMP_LE) + prifilt->pmask[i] = lessthanmasks[sev+1]; + else if(mode == CMP_GT) + prifilt->pmask[i] = ~lessthanmasks[sev+1]; + else if(mode == CMP_GE) + prifilt->pmask[i] = ~lessthanmasks[sev]; + else + DBGPRINTF("prifiltSetSeverity: program error, invalid mode %s\n", + tokenToString(mode)); + } + if(mode == CMP_NE) + prifiltInvert(prifilt); +} + +/* set prifilt so that it matches for some facilities, fac is its numerical + * value. Mode is one of the compop tokens CMP_EQ, CMP_LT, CMP_LE, CMP_GT, + * CMP_GE, CMP_NE. For the given facilities, all severities are enabled. + * NOTE: fac MUST be in the range 0..24 (not multiplied by 8)! + */ +static void +prifiltSetFacility(struct funcData_prifilt *prifilt, int fac, int mode) +{ + int i; + + memset(prifilt->pmask, 0, sizeof(prifilt->pmask)); + switch(mode) { + case CMP_EQ: + prifilt->pmask[fac] = TABLE_ALLPRI; + break; + case CMP_NE: + prifilt->pmask[fac] = TABLE_ALLPRI; + prifiltInvert(prifilt); + break; + case CMP_LT: + for(i = 0 ; i < fac ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_LE: + for(i = 0 ; i < fac+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_GE: + for(i = fac ; i < LOG_NFACILITIES+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_GT: + for(i = fac+1 ; i < LOG_NFACILITIES+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + default:break; + } +} + +/* combine a prifilt with AND/OR (the respective token values are + * used to keep things simple). + */ +static void +prifiltCombine(struct funcData_prifilt *prifilt, struct funcData_prifilt *prifilt2, int mode) +{ + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + if(mode == AND) + prifilt->pmask[i] = prifilt->pmask[i] & prifilt2->pmask[i]; + else + prifilt->pmask[i] = prifilt->pmask[i] | prifilt2->pmask[i]; + } +} + + void readConfFile(FILE *fp, es_str_t **str) { @@ -98,6 +287,14 @@ readConfFile(FILE *fp, es_str_t **str) es_addChar(str, '\0'); } +/* comparison function for qsort() and bsearch() string array compare */ +static int +qs_arrcmp(const void *s1, const void *s2) +{ + return es_strcmp(*((es_str_t**)s1), *((es_str_t**)s2)); +} + + struct objlst* objlstNew(struct cnfobj *o) { @@ -107,7 +304,6 @@ objlstNew(struct cnfobj *o) lst->next = NULL; lst->obj = o; } -dbgprintf("AAAA: creating new objlst\n"); cnfobjPrint(o); return lst; @@ -131,6 +327,22 @@ objlstAdd(struct objlst *root, struct cnfobj *o) return root; } +/* add stmt to current script, always return root stmt pointer */ +struct cnfstmt* +scriptAddStmt(struct cnfstmt *root, struct cnfstmt *s) +{ + struct cnfstmt *l; + + if(root == NULL) { + root = s; + } else { /* find last, linear search ok, as only during config phase */ + for(l = root ; l->next != NULL ; l = l->next) + ; + l->next = s; + } + return root; +} + void objlstDestruct(struct objlst *lst) { @@ -155,13 +367,12 @@ objlstPrint(struct objlst *lst) } struct nvlst* -nvlstNew(es_str_t *name, es_str_t *value) +nvlstNewStr(es_str_t *value) { struct nvlst *lst; if((lst = malloc(sizeof(struct nvlst))) != NULL) { lst->next = NULL; - lst->name = name; lst->val.datatype = 'S'; lst->val.d.estr = value; lst->bUsed = 0; @@ -170,6 +381,28 @@ nvlstNew(es_str_t *name, es_str_t *value) return lst; } +struct nvlst* +nvlstNewArray(struct cnfarray *ar) +{ + struct nvlst *lst; + + if((lst = malloc(sizeof(struct nvlst))) != NULL) { + lst->next = NULL; + lst->val.datatype = 'A'; + lst->val.d.ar = ar; + lst->bUsed = 0; + } + + return lst; +} + +struct nvlst* +nvlstSetName(struct nvlst *lst, es_str_t *name) +{ + lst->name = name; + return lst; +} + void nvlstDestruct(struct nvlst *lst) { @@ -179,8 +412,7 @@ nvlstDestruct(struct nvlst *lst) toDel = lst; lst = lst->next; es_deleteStr(toDel->name); - if(toDel->val.datatype == 'S') - es_deleteStr(toDel->val.d.estr); + varDelete(&toDel->val); free(toDel); } } @@ -192,11 +424,21 @@ nvlstPrint(struct nvlst *lst) dbgprintf("nvlst %p:\n", lst); while(lst != NULL) { name = es_str2cstr(lst->name, NULL); - // TODO: support for non-string types - value = es_str2cstr(lst->val.d.estr, NULL); - dbgprintf("\tname: '%s', value '%s'\n", name, value); + switch(lst->val.datatype) { + case 'A': + dbgprintf("\tname: '%s':\n", name); + cnfarrayPrint(lst->val.d.ar, 5); + break; + case 'S': + value = es_str2cstr(lst->val.d.estr, NULL); + dbgprintf("\tname: '%s', value '%s'\n", name, value); + free(value); + break; + default:dbgprintf("nvlstPrint: unknown type '%s'\n", + tokenToString(lst->val.datatype)); + break; + } free(name); - free(value); lst = lst->next; } } @@ -380,17 +622,17 @@ doGetFileCreateMode(struct nvlst *valnode, struct cnfparamdescr *param, if(es_strlen(valnode->val.d.estr) == 4) { c = es_getBufAddr(valnode->val.d.estr); - if(!( (c[0] == '0') - && (c[1] >= '0' && c[1] <= '7') - && (c[2] >= '0' && c[2] <= '7') - && (c[3] >= '0' && c[3] <= '7') ) ) { + if( (c[0] == '0') + && (c[1] >= '0' && c[1] <= '7') + && (c[2] >= '0' && c[2] <= '7') + && (c[3] >= '0' && c[3] <= '7') ) { fmtOK = 1; } } if(fmtOK) { val->val.datatype = 'N'; - val->val.d.n = (c[1]-'0') * 64 + (c[2]-'0') * 8 + (c[3]-'0');; + val->val.d.n = (c[1]-'0') * 64 + (c[2]-'0') * 8 + (c[3]-'0'); } else { cstr = es_str2cstr(valnode->val.d.estr, NULL); parser_errmsg("file modes need to be specified as " @@ -515,6 +757,7 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, es_size_t i; int r = 1; unsigned char *c; + val->val.datatype = 'S'; val->val.d.estr = es_newStr(32); c = es_getBufAddr(valnode->val.d.estr); @@ -523,7 +766,7 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, } if(i != es_strlen(valnode->val.d.estr)) { parser_errmsg("parameter '%s' contains whitespace, which is not " - "permitted - data after first whitespace ignored", + "permitted", param->name); r = 0; } @@ -531,6 +774,30 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, } static inline int +doGetArray(struct nvlst *valnode, struct cnfparamdescr *param, + struct cnfparamvals *val) +{ + int r = 1; + + switch(valnode->val.datatype) { + case 'S': + /* a constant string is assumed to be a single-element array */ + val->val.datatype = 'A'; + val->val.d.ar = cnfarrayNew(es_strdup(valnode->val.d.estr)); + break; + case 'A': + val->val.datatype = 'A'; + val->val.d.ar = cnfarrayDup(valnode->val.d.ar); + break; + default:parser_errmsg("parameter '%s' must be an array, but is a " + "different datatype", param->name); + r = 0; + break; + } + return r; +} + +static inline int doGetChar(struct nvlst *valnode, struct cnfparamdescr *param, struct cnfparamvals *val) { @@ -556,8 +823,15 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, uchar *cstr; int r; - dbgprintf("XXXX: in nvlstGetParam, name '%s', type %d, valnode->bUsed %d\n", + DBGPRINTF("nvlstGetParam: name '%s', type %d, valnode->bUsed %d\n", param->name, (int) param->type, valnode->bUsed); + if(valnode->val.datatype != 'S' && param->type != eCmdHdlrArray) { + parser_errmsg("parameter '%s' is not a string, which is not " + "permitted", + param->name); + r = 0; + goto done; + } valnode->bUsed = 1; val->bUsed = 1; switch(param->type) { @@ -613,6 +887,9 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, val->val.d.estr = es_strdup(valnode->val.d.estr); r = 1; break; + case eCmdHdlrArray: + r = doGetArray(valnode, param, val); + break; case eCmdHdlrGoneAway: parser_errmsg("parameter '%s' is no longer supported", param->name); @@ -623,7 +900,7 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, r = 0; break; } - return r; +done: return r; } @@ -684,6 +961,30 @@ nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, } +/* check if at least one cnfparamval is actually set + * returns 1 if so, 0 otherwise + */ +int +cnfparamvalsIsSet(struct cnfparamblk *params, struct cnfparamvals *vals) +{ + int i; + + if(vals == NULL) + return 0; + if(params->version != CNFPARAMBLK_VERSION) { + dbgprintf("nvlstGetParams: invalid param block version " + "%d, expected %d\n", + params->version, CNFPARAMBLK_VERSION); + return 0; + } + for(i = 0 ; i < params->nParams ; ++i) { + if(vals[i].bUsed) + return 1; + } + return 0; +} + + void cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals) { @@ -700,6 +1001,9 @@ cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals) dbgprintf(" '%s'", cstr); free(cstr); break; + case 'A': + cnfarrayPrint(vals[i].val.d.ar, 0); + break; case 'N': dbgprintf("%lld", vals[i].val.d.n); break; @@ -724,6 +1028,7 @@ cnfobjNew(enum cnfobjType objType, struct nvlst *lst) o->objType = objType; o->nvlst = lst; o->subobjs = NULL; + o->script = NULL; } return o; @@ -747,116 +1052,6 @@ cnfobjPrint(struct cnfobj *o) } -struct cnfactlst* -cnfactlstNew(enum cnfactType actType, struct nvlst *lst, char *actLine) -{ - struct cnfactlst *actlst; - - if((actlst = malloc(sizeof(struct cnfactlst))) != NULL) { - actlst->next = NULL; - actlst->syslines = NULL; - actlst->actType = actType; - actlst->lineno = yylineno; - actlst->cnfFile = strdup(cnfcurrfn); - if(actType == CNFACT_V2) - actlst->data.lst = lst; - else - actlst->data.legActLine = actLine; - } - return actlst; -} - -struct cnfactlst* -cnfactlstAddSysline(struct cnfactlst* actlst, char *line) -{ - struct cnfcfsyslinelst *cflst; - - if((cflst = malloc(sizeof(struct cnfcfsyslinelst))) != NULL) { - cflst->line = line; - if(actlst->syslines == NULL) { - cflst->next = NULL; - } else { - cflst->next = actlst->syslines; - } - actlst->syslines = cflst; - } - return actlst; -} - - -void -cnfactlstDestruct(struct cnfactlst *actlst) -{ - struct cnfactlst *toDel; - - while(actlst != NULL) { - toDel = actlst; - actlst = actlst->next; - free(toDel->cnfFile); - cnfcfsyslinelstDestruct(toDel->syslines); - if(toDel->actType == CNFACT_V2) - nvlstDestruct(toDel->data.lst); - else - free(toDel->data.legActLine); - free(toDel); - } - -} - -static inline struct cnfcfsyslinelst* -cnfcfsyslinelstReverse(struct cnfcfsyslinelst *lst) -{ - struct cnfcfsyslinelst *curr, *prev; - if(lst == NULL) - return NULL; - prev = NULL; - while(lst != NULL) { - curr = lst; - lst = lst->next; - curr->next = prev; - prev = curr; - } - return prev; -} - -struct cnfactlst* -cnfactlstReverse(struct cnfactlst *actlst) -{ - struct cnfactlst *curr, *prev; - - prev = NULL; - while(actlst != NULL) { - curr = actlst; - actlst = actlst->next; - curr->syslines = cnfcfsyslinelstReverse(curr->syslines); - curr->next = prev; - prev = curr; - } - return prev; -} - -void -cnfactlstPrint(struct cnfactlst *actlst) -{ - struct cnfcfsyslinelst *cflst; - - while(actlst != NULL) { - dbgprintf("aclst %p: ", actlst); - if(actlst->actType == CNFACT_V2) { - dbgprintf("V2 action type: "); - nvlstPrint(actlst->data.lst); - } else { - dbgprintf("legacy action line: '%s'\n", - actlst->data.legActLine); - } - for( cflst = actlst->syslines - ; cflst != NULL ; cflst = cflst->next) { - dbgprintf("action:cfsysline: '%s'\n", cflst->line); - } - actlst = actlst->next; - } -} - struct cnfexpr* cnfexprNew(unsigned nodetype, struct cnfexpr *l, struct cnfexpr *r) { @@ -883,15 +1078,19 @@ done: * try to convert it to one. The semantics from es_str2num() * are used (bSuccess tells if the conversion went well or not). */ -static inline long long +static long long var2Number(struct var *r, int *bSuccess) { long long n; if(r->datatype == 'S') { n = es_str2num(r->d.estr, bSuccess); } else { - n = r->d.n; - if(bSuccess) + if(r->datatype == 'J') { + n = (r->d.json == NULL) ? 0 : json_object_get_int(r->d.json); + } else { + n = r->d.n; + } + if(bSuccess != NULL) *bSuccess = 1; } return n; @@ -902,12 +1101,131 @@ var2Number(struct var *r, int *bSuccess) static inline es_str_t * var2String(struct var *r, int *bMustFree) { + es_str_t *estr; + char *cstr; + rs_size_t lenstr; if(r->datatype == 'N') { *bMustFree = 1; - return es_newStrFromNumber(r->d.n); + estr = es_newStrFromNumber(r->d.n); + } else if(r->datatype == 'J') { + *bMustFree = 1; + if(r->d.json == NULL) { + cstr = "", + lenstr = 0; + } else { + cstr = (char*)json_object_get_string(r->d.json); + lenstr = strlen(cstr); + } + estr = es_newStrFromCStr(cstr, lenstr); + } else { + *bMustFree = 0; + estr = r->d.estr; } - *bMustFree = 0; - return r->d.estr; + return estr; +} + +static uchar* +var2CString(struct var *r, int *bMustFree) +{ + uchar *cstr; + es_str_t *estr; + estr = var2String(r, bMustFree); + cstr = (uchar*) es_str2cstr(estr, NULL); + if(*bMustFree) + es_deleteStr(estr); + *bMustFree = 1; + return cstr; +} + +static rsRetVal +doExtractFieldByChar(uchar *str, uchar delim, int matchnbr, uchar **resstr) +{ + int iCurrFld; + int iLen; + uchar *pBuf; + uchar *pFld; + uchar *pFldEnd; + DEFiRet; + + /* first, skip to the field in question */ + iCurrFld = 1; + pFld = str; + while(*pFld && iCurrFld < matchnbr) { + /* skip fields until the requested field or end of string is found */ + while(*pFld && (uchar) *pFld != delim) + ++pFld; /* skip to field terminator */ + if(*pFld == delim) { + ++pFld; /* eat it */ + ++iCurrFld; + } + } + dbgprintf("field() field requested %d, field found %d\n", matchnbr, iCurrFld); + + if(iCurrFld == matchnbr) { + /* field found, now extract it */ + /* first of all, we need to find the end */ + pFldEnd = pFld; + while(*pFldEnd && *pFldEnd != delim) + ++pFldEnd; + --pFldEnd; /* we are already at the delimiter - so we need to + * step back a little not to copy it as part of the field. */ + /* we got our end pointer, now do the copy */ + iLen = pFldEnd - pFld + 1; /* the +1 is for an actual char, NOT \0! */ + CHKmalloc(pBuf = MALLOC((iLen + 1) * sizeof(char))); + /* now copy */ + memcpy(pBuf, pFld, iLen); + pBuf[iLen] = '\0'; /* terminate it */ + *resstr = pBuf; + } else { + ABORT_FINALIZE(RS_RET_FIELD_NOT_FOUND); + } +finalize_it: + RETiRet; +} + + +static rsRetVal +doExtractFieldByStr(uchar *str, char *delim, rs_size_t lenDelim, int matchnbr, uchar **resstr) +{ + int iCurrFld; + int iLen; + uchar *pBuf; + uchar *pFld; + uchar *pFldEnd; + DEFiRet; + + /* first, skip to the field in question */ + iCurrFld = 1; + pFld = str; + while(pFld != NULL && iCurrFld < matchnbr) { + if((pFld = (uchar*) strstr((char*)pFld, delim)) != NULL) { + pFld += lenDelim; + ++iCurrFld; + } + } + dbgprintf("field() field requested %d, field found %d\n", matchnbr, iCurrFld); + + if(iCurrFld == matchnbr) { + /* field found, now extract it */ + /* first of all, we need to find the end */ + pFldEnd = (uchar*) strstr((char*)pFld, delim); + if(pFldEnd == NULL) { + iLen = strlen((char*) pFld); + } else { /* found delmiter! Note that pFldEnd *is* already on + * the first delmi char, we don't need that. */ + iLen = pFldEnd - pFld; + } + /* we got our end pointer, now do the copy */ + CHKmalloc(pBuf = MALLOC((iLen + 1) * sizeof(char))); + /* now copy */ + memcpy(pBuf, pFld, iLen); + pBuf[iLen] = '\0'; /* terminate it */ + *resstr = pBuf; + } else { + ABORT_FINALIZE(RS_RET_FIELD_NOT_FOUND); + } +finalize_it: + RETiRet; } /* Perform a function call. This has been moved out of cnfExprEval in order @@ -921,8 +1239,13 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) int bMustFree; es_str_t *estr; char *str; + uchar *resStr; int retval; struct var r[CNFFUNC_MAX_ARGS]; + int delim; + int matchnbr; + struct funcData_prifilt *pPrifilt; + rsRetVal localRet; dbgprintf("rainerscript: executing function id %d\n", func->fID); switch(func->fID) { @@ -990,8 +1313,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) break; case CNFFUNC_RE_MATCH: cnfexprEval(func->expr[0], &r[0], usrptr); - estr = var2String(&r[0], &bMustFree); - str = es_str2cstr(estr, NULL); + str = (char*) var2CString(&r[0], &bMustFree); retval = regexp.regexec(func->funcdata, str, 0, NULL, 0); if(retval == 0) ret->d.n = 1; @@ -1002,9 +1324,50 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } } ret->datatype = 'N'; - if(bMustFree) es_deleteStr(estr); - free(str); + if(bMustFree) free(str); + if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + break; + case CNFFUNC_FIELD: + cnfexprEval(func->expr[0], &r[0], usrptr); + cnfexprEval(func->expr[1], &r[1], usrptr); + cnfexprEval(func->expr[2], &r[2], usrptr); + str = (char*) var2CString(&r[0], &bMustFree); + matchnbr = var2Number(&r[2], NULL); + if(r[1].datatype == 'S') { + char *delimstr; + delimstr = (char*) es_str2cstr(r[1].d.estr, NULL); + localRet = doExtractFieldByStr((uchar*)str, delimstr, es_strlen(r[1].d.estr), + matchnbr, &resStr); + free(delimstr); + } else { + delim = var2Number(&r[1], NULL); + localRet = doExtractFieldByChar((uchar*)str, (char) delim, matchnbr, &resStr); + } + if(localRet == RS_RET_OK) { + ret->d.estr = es_newStrFromCStr((char*)resStr, strlen((char*)resStr)); + free(resStr); + } else if(localRet == RS_RET_FIELD_NOT_FOUND) { + ret->d.estr = es_newStrFromCStr("***FIELD NOT FOUND***", + sizeof("***FIELD NOT FOUND***")-1); + } else { + ret->d.estr = es_newStrFromCStr("***ERROR in field() FUNCTION***", + sizeof("***ERROR in field() FUNCTION***")-1); + } + ret->datatype = 'S'; + if(bMustFree) free(str); if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + if(r[1].datatype == 'S') es_deleteStr(r[1].d.estr); + if(r[2].datatype == 'S') es_deleteStr(r[2].d.estr); + break; + case CNFFUNC_PRIFILT: + pPrifilt = (struct funcData_prifilt*) func->funcdata; + if( (pPrifilt->pmask[((msg_t*)usrptr)->iFacility] == TABLE_NOPRI) || + ((pPrifilt->pmask[((msg_t*)usrptr)->iFacility] + & (1<<((msg_t*)usrptr)->iSeverity)) == 0) ) + ret->d.n = 0; + else + ret->d.n = 1; + ret->datatype = 'N'; break; default: if(Debug) { @@ -1018,6 +1381,67 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } } +static inline void +evalVar(struct cnfvar *var, void *usrptr, struct var *ret) +{ + rsRetVal localRet; + es_str_t *estr; + struct json_object *json; + + if(var->name[0] == '$' && var->name[1] == '!') { + /* TODO: unify string libs */ + estr = es_newStrFromBuf(var->name+1, strlen(var->name)-1); + localRet = msgGetCEEPropJSON((msg_t*)usrptr, estr, &json); + es_deleteStr(estr); + ret->datatype = 'J'; + ret->d.json = (localRet == RS_RET_OK) ? json : NULL; + } else { + ret->datatype = 'S'; + ret->d.estr = cnfGetVar(var->name, usrptr); + } + +} + +/* perform a string comparision operation against a while array. Semantic is + * that one one comparison is true, the whole construct is true. + * TODO: we can obviously optimize this process. One idea is to + * compile a regex, which should work faster than serial comparison. + * Note: compiling a regex does NOT work at all. I experimented with that + * and it was generally 5 to 10 times SLOWER than what we do here... + */ +static int +evalStrArrayCmp(es_str_t *estr_l, struct cnfarray* ar, int cmpop) +{ + int i; + int r = 0; + es_str_t **res; + if(cmpop == CMP_EQ) { + res = bsearch(&estr_l, ar->arr, ar->nmemb, sizeof(es_str_t*), qs_arrcmp); + r = res != NULL; + } else if(cmpop == CMP_NE) { + res = bsearch(&estr_l, ar->arr, ar->nmemb, sizeof(es_str_t*), qs_arrcmp); + r = res == NULL; + } else { + for(i = 0 ; (r == 0) && (i < ar->nmemb) ; ++i) { + switch(cmpop) { + case CMP_STARTSWITH: + r = es_strncmp(estr_l, ar->arr[i], es_strlen(ar->arr[i])) == 0; + break; + case CMP_STARTSWITHI: + r = es_strncasecmp(estr_l, ar->arr[i], es_strlen(ar->arr[i])) == 0; + break; + case CMP_CONTAINS: + r = es_strContains(estr_l, ar->arr[i]) != -1; + break; + case CMP_CONTAINSI: + r = es_strCaseContains(estr_l, ar->arr[i]) != -1; + break; + } + } + } + return r; +} + #define FREE_BOTH_RET \ if(r.datatype == 'S') es_deleteStr(r.d.estr); \ if(l.datatype == 'S') es_deleteStr(l.d.estr) @@ -1029,16 +1453,27 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) ret->d.n = var2Number(&l, &convok_l) x var2Number(&r, &convok_r); \ FREE_BOTH_RET +/* NOTE: array as right-hand argument MUST be handled by user */ #define PREP_TWO_STRINGS \ cnfexprEval(expr->l, &l, usrptr); \ estr_l = var2String(&l, &bMustFree2); \ - cnfexprEval(expr->r, &r, usrptr); \ - estr_r = var2String(&r, &bMustFree) + if(expr->r->nodetype == 'S') { \ + estr_r = ((struct cnfstringval*)expr->r)->estr;\ + bMustFree = 0; \ + } else if(expr->r->nodetype != 'A') { \ + cnfexprEval(expr->r, &r, usrptr); \ + estr_r = var2String(&r, &bMustFree); \ + } else { \ + /* Note: this is not really necessary, but if we do not */ \ + /* do it, we get a very irritating compiler warning... */ \ + estr_r = NULL; \ + } #define FREE_TWO_STRINGS \ - if(bMustFree) es_deleteStr(estr_r); \ - if(bMustFree2) es_deleteStr(estr_l); \ - FREE_BOTH_RET + if(bMustFree) es_deleteStr(estr_r); \ + if(expr->r->nodetype != 'S' && expr->r->nodetype != 'A' && r.datatype == 'S') es_deleteStr(r.d.estr); \ + if(bMustFree2) es_deleteStr(estr_l); \ + if(l.datatype == 'S') es_deleteStr(l.d.estr) /* evaluate an expression. * Note that we try to avoid malloc whenever possible (because of @@ -1059,29 +1494,63 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) int bMustFree, bMustFree2; long long n_r, n_l; - dbgprintf("eval expr %p, type '%c'(%u)\n", expr, expr->nodetype, expr->nodetype); + dbgprintf("eval expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); switch(expr->nodetype) { /* note: comparison operations are extremely similar. The code can be copyied, only * places flagged with "CMP" need to be changed. */ case CMP_EQ: + /* this is optimized in regard to right param as a PoC for all compOps + * So this is a NOT yet the copy template! + */ cnfexprEval(expr->l, &l, usrptr); - cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; if(l.datatype == 'S') { - if(r.datatype == 'S') { - ret->d.n = !es_strcmp(l.d.estr, r.d.estr); /*CMP*/ + if(expr->r->nodetype == 'S') { + ret->d.n = !es_strcmp(l.d.estr, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(l.d.estr, (struct cnfarray*) expr->r, CMP_EQ); } else { - n_l = var2Number(&l, &convok_l); - if(convok_l) { - ret->d.n = (n_l == r.d.n); /*CMP*/ + cnfexprEval(expr->r, &r, usrptr); + if(r.datatype == 'S') { + ret->d.n = !es_strcmp(l.d.estr, r.d.estr); /*CMP*/ } else { - estr_r = var2String(&r, &bMustFree); - ret->d.n = !es_strcmp(l.d.estr, estr_r); /*CMP*/ - if(bMustFree) es_deleteStr(estr_r); + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l == r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = !es_strcmp(l.d.estr, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } + } + if(r.datatype == 'S') es_deleteStr(r.d.estr); + } + } else if(l.datatype == 'J') { + estr_l = var2String(&l, &bMustFree); + if(expr->r->nodetype == 'S') { + ret->d.n = !es_strcmp(estr_l, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_EQ); + } else { + cnfexprEval(expr->r, &r, usrptr); + if(r.datatype == 'S') { + ret->d.n = !es_strcmp(estr_l, r.d.estr); /*CMP*/ + } else { + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l == r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = !es_strcmp(estr_l, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } } + if(r.datatype == 'S') es_deleteStr(r.d.estr); } + if(bMustFree) es_deleteStr(estr_l); } else { + cnfexprEval(expr->r, &r, usrptr); if(r.datatype == 'S') { n_r = var2Number(&r, &convok_r); if(convok_r) { @@ -1094,24 +1563,31 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) } else { ret->d.n = (l.d.n == r.d.n); /*CMP*/ } + if(r.datatype == 'S') es_deleteStr(r.d.estr); } - FREE_BOTH_RET; + if(l.datatype == 'S') es_deleteStr(l.d.estr); break; case CMP_NE: cnfexprEval(expr->l, &l, usrptr); cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; if(l.datatype == 'S') { - if(r.datatype == 'S') { - ret->d.n = es_strcmp(l.d.estr, r.d.estr); /*CMP*/ + if(expr->r->nodetype == 'S') { + ret->d.n = es_strcmp(l.d.estr, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(l.d.estr, (struct cnfarray*) expr->r, CMP_NE); } else { - n_l = var2Number(&l, &convok_l); - if(convok_l) { - ret->d.n = (n_l != r.d.n); /*CMP*/ + if(r.datatype == 'S') { + ret->d.n = es_strcmp(l.d.estr, r.d.estr); /*CMP*/ } else { - estr_r = var2String(&r, &bMustFree); - ret->d.n = es_strcmp(l.d.estr, estr_r); /*CMP*/ - if(bMustFree) es_deleteStr(estr_r); + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l != r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = es_strcmp(l.d.estr, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } } } } else { @@ -1265,25 +1741,45 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) case CMP_STARTSWITH: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strncmp(estr_l, estr_r, estr_r->lenStr) == 0; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_STARTSWITH); + bMustFree = 0; + } else { + ret->d.n = es_strncmp(estr_l, estr_r, estr_r->lenStr) == 0; + } FREE_TWO_STRINGS; break; case CMP_STARTSWITHI: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strncasecmp(estr_l, estr_r, estr_r->lenStr) == 0; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_STARTSWITHI); + bMustFree = 0; + } else { + ret->d.n = es_strncasecmp(estr_l, estr_r, estr_r->lenStr) == 0; + } FREE_TWO_STRINGS; break; case CMP_CONTAINS: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strContains(estr_l, estr_r) != -1; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_CONTAINS); + bMustFree = 0; + } else { + ret->d.n = es_strContains(estr_l, estr_r) != -1; + } FREE_TWO_STRINGS; break; case CMP_CONTAINSI: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strCaseContains(estr_l, estr_r) != -1; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_CONTAINSI); + bMustFree = 0; + } else { + ret->d.n = es_strCaseContains(estr_l, estr_r) != -1; + } FREE_TWO_STRINGS; break; case OR: @@ -1330,9 +1826,27 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) ret->datatype = 'S'; ret->d.estr = es_strdup(((struct cnfstringval*)expr)->estr); break; + case 'A': + /* if an array is used with "normal" operations, it just evaluates + * to its first element. + */ + ret->datatype = 'S'; + ret->d.estr = es_strdup(((struct cnfarray*)expr)->arr[0]); + break; case 'V': + evalVar((struct cnfvar*)expr, usrptr, ret); + break; + case '&': + /* TODO: think about optimization, should be possible ;) */ + PREP_TWO_STRINGS; + if(expr->r->nodetype == 'A') { + estr_r = ((struct cnfarray*)expr->r)->arr[0]; + bMustFree = 0; + } ret->datatype = 'S'; - ret->d.estr = cnfGetVar(((struct cnfvar*)expr)->name, usrptr); + ret->d.estr = es_strdup(estr_l); + es_addStr(&ret->d.estr, estr_r); + FREE_TWO_STRINGS; break; case '+': COMP_NUM_BINOP(+); @@ -1369,6 +1883,16 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) //--------------------------------------------------------- +void +cnfarrayContentDestruct(struct cnfarray *ar) +{ + unsigned short i; + for(i = 0 ; i < ar->nmemb ; ++i) { + es_deleteStr(ar->arr[i]); + } + free(ar->arr); +} + static inline void cnffuncDestruct(struct cnffunc *func) { @@ -1380,12 +1904,13 @@ cnffuncDestruct(struct cnffunc *func) /* some functions require special destruction */ switch(func->fID) { case CNFFUNC_RE_MATCH: - regexp.regfree(func->funcdata); - free(func->funcdata); - free(func->fname); + if(func->funcdata != NULL) + regexp.regfree(func->funcdata); break; default:break; } + free(func->funcdata); + free(func->fname); } /* Destruct an expression and all sub-expressions contained in it. @@ -1394,7 +1919,13 @@ void cnfexprDestruct(struct cnfexpr *expr) { - dbgprintf("cnfexprDestruct expr %p, type '%c'(%u)\n", expr, expr->nodetype, expr->nodetype); + if(expr == NULL) { + /* this is valid and can happen during optimizer run! */ + DBGPRINTF("cnfexprDestruct got NULL ptr - valid, so doing nothing\n"); + return; + } + + DBGPRINTF("cnfexprDestruct expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); switch(expr->nodetype) { case CMP_NE: case CMP_EQ: @@ -1408,6 +1939,7 @@ cnfexprDestruct(struct cnfexpr *expr) case CMP_CONTAINSI: case OR: case AND: + case '&': case '+': case '-': case '*': @@ -1431,6 +1963,9 @@ cnfexprDestruct(struct cnfexpr *expr) case 'F': cnffuncDestruct((struct cnffunc*)expr); break; + case 'A': + cnfarrayContentDestruct((struct cnfarray*)expr); + break; default:break; } free(expr); @@ -1459,13 +1994,39 @@ doIndent(int indent) for(i = 0 ; i < indent ; ++i) dbgprintf(" "); } + +static void +pmaskPrint(uchar *pmask, int indent) +{ + int i; + doIndent(indent); + dbgprintf("pmask: "); + for (i = 0; i <= LOG_NFACILITIES; i++) + if (pmask[i] == TABLE_NOPRI) + dbgprintf(" X "); + else + dbgprintf("%2X ", pmask[i]); + dbgprintf("\n"); +} + +static void +cnfarrayPrint(struct cnfarray *ar, int indent) +{ + int i; + doIndent(indent); dbgprintf("ARRAY:\n"); + for(i = 0 ; i < ar->nmemb ; ++i) { + doIndent(indent+1); + cstrPrint("string '", ar->arr[i]); + dbgprintf("'\n"); + } +} + void cnfexprPrint(struct cnfexpr *expr, int indent) { struct cnffunc *func; int i; - //dbgprintf("expr %p, indent %d, type '%c'\n", expr, indent, expr->nodetype); switch(expr->nodetype) { case CMP_EQ: cnfexprPrint(expr->l, indent+1); @@ -1549,6 +2110,9 @@ cnfexprPrint(struct cnfexpr *expr, int indent) cstrPrint("string '", ((struct cnfstringval*)expr)->estr); dbgprintf("'\n"); break; + case 'A': + cnfarrayPrint((struct cnfarray*)expr, indent); + break; case 'N': doIndent(indent); dbgprintf("%lld\n", ((struct cnfnumval*)expr)->val); @@ -1562,10 +2126,16 @@ cnfexprPrint(struct cnfexpr *expr, int indent) func = (struct cnffunc*) expr; cstrPrint("function '", func->fname); dbgprintf("' (id:%d, params:%hu)\n", func->fID, func->nParams); + if(func->fID == CNFFUNC_PRIFILT) { + struct funcData_prifilt *pD; + pD = (struct funcData_prifilt*) func->funcdata; + pmaskPrint(pD->pmask, indent+1); + } for(i = 0 ; i < func->nParams ; ++i) { cnfexprPrint(func->expr[i], indent+1); } break; + case '&': case '+': case '-': case '*': @@ -1579,11 +2149,93 @@ cnfexprPrint(struct cnfexpr *expr, int indent) cnfexprPrint(expr->r, indent+1); break; default: - dbgprintf("error: unknown nodetype %u\n", - (unsigned) expr->nodetype); + dbgprintf("error: unknown nodetype %u['%c']\n", + (unsigned) expr->nodetype, (char) expr->nodetype); break; } } +void +cnfstmtPrint(struct cnfstmt *root, int indent) +{ + struct cnfstmt *stmt; + char *cstr; + //dbgprintf("stmt %p, indent %d, type '%c'\n", expr, indent, expr->nodetype); + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + switch(stmt->nodetype) { + case S_NOP: + doIndent(indent); dbgprintf("NOP\n"); + break; + case S_STOP: + doIndent(indent); dbgprintf("STOP\n"); + break; + case S_CALL: + cstr = es_str2cstr(stmt->d.s_call.name, NULL); + doIndent(indent); dbgprintf("CALL [%s]\n", cstr); + free(cstr); + break; + case S_ACT: + doIndent(indent); dbgprintf("ACTION %p [%s]\n", stmt->d.act, stmt->printable); + break; + case S_IF: + doIndent(indent); dbgprintf("IF\n"); + cnfexprPrint(stmt->d.s_if.expr, indent+1); + doIndent(indent); dbgprintf("THEN\n"); + cnfstmtPrint(stmt->d.s_if.t_then, indent+1); + if(stmt->d.s_if.t_else != NULL) { + doIndent(indent); dbgprintf("ELSE\n"); + cnfstmtPrint(stmt->d.s_if.t_else, indent+1); + } + doIndent(indent); dbgprintf("END IF\n"); + break; + case S_SET: + doIndent(indent); dbgprintf("SET %s =\n", + stmt->d.s_set.varname); + cnfexprPrint(stmt->d.s_set.expr, indent+1); + doIndent(indent); dbgprintf("END SET\n"); + break; + case S_UNSET: + doIndent(indent); dbgprintf("UNSET %s\n", + stmt->d.s_unset.varname); + break; + case S_PRIFILT: + doIndent(indent); dbgprintf("PRIFILT '%s'\n", stmt->printable); + pmaskPrint(stmt->d.s_prifilt.pmask, indent); + cnfstmtPrint(stmt->d.s_prifilt.t_then, indent+1); + if(stmt->d.s_prifilt.t_else != NULL) { + doIndent(indent); dbgprintf("ELSE\n"); + cnfstmtPrint(stmt->d.s_prifilt.t_else, indent+1); + } + doIndent(indent); dbgprintf("END PRIFILT\n"); + break; + case S_PROPFILT: + doIndent(indent); dbgprintf("PROPFILT\n"); + doIndent(indent); dbgprintf("\tProperty.: '%s'\n", + propIDToName(stmt->d.s_propfilt.propID)); + if(stmt->d.s_propfilt.propName != NULL) { + cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + doIndent(indent); + dbgprintf("\tCEE-Prop.: '%s'\n", cstr); + free(cstr); + } + doIndent(indent); dbgprintf("\tOperation: "); + if(stmt->d.s_propfilt.isNegated) + dbgprintf("NOT "); + dbgprintf("'%s'\n", getFIOPName(stmt->d.s_propfilt.operation)); + if(stmt->d.s_propfilt.pCSCompValue != NULL) { + doIndent(indent); dbgprintf("\tValue....: '%s'\n", + rsCStrGetSzStrNoNULL(stmt->d.s_propfilt.pCSCompValue)); + } + doIndent(indent); dbgprintf("THEN\n"); + cnfstmtPrint(stmt->d.s_propfilt.t_then, indent+1); + doIndent(indent); dbgprintf("END PROPFILT\n"); + break; + default: + dbgprintf("error: unknown stmt type %u\n", + (unsigned) stmt->nodetype); + break; + } + } +} struct cnfnumval* cnfnumvalNew(long long val) @@ -1607,6 +2259,52 @@ cnfstringvalNew(es_str_t *estr) return strval; } +/* creates array AND adds first element to it */ +struct cnfarray* +cnfarrayNew(es_str_t *val) +{ + struct cnfarray *ar; + if((ar = malloc(sizeof(struct cnfarray))) != NULL) { + ar->nodetype = 'A'; + ar->nmemb = 1; + if((ar->arr = malloc(sizeof(es_str_t*))) == NULL) { + free(ar); + ar = NULL; + goto done; + } + ar->arr[0] = val; + } +done: return ar; +} + +struct cnfarray* +cnfarrayAdd(struct cnfarray *ar, es_str_t *val) +{ + es_str_t **newptr; + if((newptr = realloc(ar->arr, (ar->nmemb+1)*sizeof(es_str_t*))) == NULL) { + DBGPRINTF("cnfarrayAdd: realloc failed, item ignored, ar->arr=%p\n", ar->arr); + goto done; + } else { + ar->arr = newptr; + ar->arr[ar->nmemb] = val; + ar->nmemb++; + } +done: return ar; +} + +/* duplicate an array (deep copy) */ +struct cnfarray* +cnfarrayDup(struct cnfarray *old) +{ + int i; + struct cnfarray *ar; + ar = cnfarrayNew(es_strdup(old->arr[0])); + for(i = 1 ; i < old->nmemb ; ++i) { + cnfarrayAdd(ar, es_strdup(old->arr[i])); + } + return ar; +} + struct cnfvar* cnfvarNew(char *name) { @@ -1618,62 +2316,713 @@ cnfvarNew(char *name) return var; } -struct cnfrule * -cnfruleNew(enum cnfFiltType filttype, struct cnfactlst *actlst) +struct cnfstmt * +cnfstmtNew(unsigned s_type) { - struct cnfrule* cnfrule; - if((cnfrule = malloc(sizeof(struct cnfrule))) != NULL) { - cnfrule->nodetype = 'R'; - cnfrule->filttype = filttype; - cnfrule->actlst = cnfactlstReverse(actlst); + struct cnfstmt* cnfstmt; + if((cnfstmt = malloc(sizeof(struct cnfstmt))) != NULL) { + cnfstmt->nodetype = s_type; + cnfstmt->printable = NULL; + cnfstmt->next = NULL; } - return cnfrule; + return cnfstmt; } void -cnfrulePrint(struct cnfrule *rule) +cnfstmtDestruct(struct cnfstmt *root) +{ + struct cnfstmt *stmt, *todel; + for(stmt = root ; stmt != NULL ; ) { + switch(stmt->nodetype) { + case S_NOP: + case S_STOP: + break; + case S_CALL: + es_deleteStr(stmt->d.s_call.name); + break; + case S_ACT: + actionDestruct(stmt->d.act); + break; + case S_IF: + cnfexprDestruct(stmt->d.s_if.expr); + if(stmt->d.s_if.t_then != NULL) { + cnfstmtDestruct(stmt->d.s_if.t_then); + } + if(stmt->d.s_if.t_else != NULL) { + cnfstmtDestruct(stmt->d.s_if.t_else); + } + break; + case S_SET: + free(stmt->d.s_set.varname); + cnfexprDestruct(stmt->d.s_set.expr); + break; + case S_UNSET: + free(stmt->d.s_set.varname); + break; + case S_PRIFILT: + cnfstmtDestruct(stmt->d.s_prifilt.t_then); + cnfstmtDestruct(stmt->d.s_prifilt.t_else); + break; + case S_PROPFILT: + if(stmt->d.s_propfilt.propName != NULL) + es_deleteStr(stmt->d.s_propfilt.propName); + if(stmt->d.s_propfilt.regex_cache != NULL) + rsCStrRegexDestruct(&stmt->d.s_propfilt.regex_cache); + if(stmt->d.s_propfilt.pCSCompValue != NULL) + cstrDestruct(&stmt->d.s_propfilt.pCSCompValue); + cnfstmtDestruct(stmt->d.s_propfilt.t_then); + break; + default: + dbgprintf("error: unknown stmt type during destruct %u\n", + (unsigned) stmt->nodetype); + break; + } + free(stmt->printable); + todel = stmt; + stmt = stmt->next; + free(todel); + } +} + +struct cnfstmt * +cnfstmtNewSet(char *var, struct cnfexpr *expr) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_SET)) != NULL) { + cnfstmt->d.s_set.varname = (uchar*) var; + cnfstmt->d.s_set.expr = expr; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewCall(es_str_t *name) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_CALL)) != NULL) { + cnfstmt->d.s_call.name = name; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewUnset(char *var) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_UNSET)) != NULL) { + cnfstmt->d.s_unset.varname = (uchar*) var; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewContinue(void) +{ + return cnfstmtNew(S_NOP); +} + +struct cnfstmt * +cnfstmtNewPRIFILT(char *prifilt, struct cnfstmt *t_then) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_PRIFILT)) != NULL) { + cnfstmt->printable = (uchar*)prifilt; + cnfstmt->d.s_prifilt.t_then = t_then; + cnfstmt->d.s_prifilt.t_else = NULL; + DecodePRIFilter((uchar*)prifilt, cnfstmt->d.s_prifilt.pmask); + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewPROPFILT(char *propfilt, struct cnfstmt *t_then) +{ + struct cnfstmt* cnfstmt; + rsRetVal lRet; + if((cnfstmt = cnfstmtNew(S_PROPFILT)) != NULL) { + cnfstmt->printable = (uchar*)propfilt; + cnfstmt->d.s_propfilt.t_then = t_then; + cnfstmt->d.s_propfilt.propName = NULL; + cnfstmt->d.s_propfilt.regex_cache = NULL; + cnfstmt->d.s_propfilt.pCSCompValue = NULL; + lRet = DecodePropFilter((uchar*)propfilt, cnfstmt); + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewAct(struct nvlst *lst) +{ + struct cnfstmt* cnfstmt; + char namebuf[256]; + rsRetVal localRet; + if((cnfstmt = cnfstmtNew(S_ACT)) == NULL) + goto done; + localRet = actionNewInst(lst, &cnfstmt->d.act); + if(localRet == RS_RET_OK_WARN) { + parser_errmsg("warnings occured in file '%s' around line %d", + cnfcurrfn, yylineno); + } else if(localRet != RS_RET_OK) { + parser_errmsg("errors occured in file '%s' around line %d", + cnfcurrfn, yylineno); + cnfstmt->nodetype = S_NOP; /* disable action! */ + goto done; + } + snprintf(namebuf, sizeof(namebuf)-1, "action(type=\"%s\" ...)", + modGetName(cnfstmt->d.act->pMod)); + namebuf[255] = '\0'; /* be on safe side */ + cnfstmt->printable = (uchar*)strdup(namebuf); + nvlstChkUnused(lst); + nvlstDestruct(lst); +done: return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewLegaAct(char *actline) +{ + struct cnfstmt* cnfstmt; + rsRetVal localRet; + if((cnfstmt = cnfstmtNew(S_ACT)) == NULL) + goto done; + cnfstmt->printable = (uchar*)strdup((char*)actline); + localRet = cflineDoAction(loadConf, (uchar**)&actline, &cnfstmt->d.act); + if(localRet != RS_RET_OK && localRet != RS_RET_OK_WARN) { + parser_errmsg("%s occured in file '%s' around line %d", + (localRet == RS_RET_OK_WARN) ? "warnings" : "errors", + cnfcurrfn, yylineno); + if(localRet != RS_RET_OK_WARN) { + cnfstmt->nodetype = S_NOP; /* disable action! */ + goto done; + } + } +done: return cnfstmt; +} + + +/* returns 1 if the two expressions are constants, 0 otherwise + * if both are constants, the expression subtrees are destructed + * (this is an aid for constant folding optimizing) + */ +static int +getConstNumber(struct cnfexpr *expr, long long *l, long long *r) +{ + int ret = 0; + cnfexprOptimize(expr->l); + cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'N') { + if(expr->r->nodetype == 'N') { + ret = 1; + *l = ((struct cnfnumval*)expr->l)->val; + *r = ((struct cnfnumval*)expr->r)->val; + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } else if(expr->r->nodetype == 'S') { + ret = 1; + *l = ((struct cnfnumval*)expr->l)->val; + *r = es_str2num(((struct cnfstringval*)expr->r)->estr, NULL); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } + } else if(expr->l->nodetype == 'S') { + if(expr->r->nodetype == 'N') { + ret = 1; + *l = es_str2num(((struct cnfstringval*)expr->l)->estr, NULL); + *r = ((struct cnfnumval*)expr->r)->val; + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } else if(expr->r->nodetype == 'S') { + ret = 1; + *l = es_str2num(((struct cnfstringval*)expr->l)->estr, NULL); + *r = es_str2num(((struct cnfstringval*)expr->r)->estr, NULL); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } + } + return ret; +} + + +/* constant folding for string concatenation */ +static inline void +constFoldConcat(struct cnfexpr *expr) +{ + es_str_t *estr; + cnfexprOptimize(expr->l); + cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'S') { + if(expr->r->nodetype == 'S') { + estr = ((struct cnfstringval*)expr->l)->estr; + ((struct cnfstringval*)expr->l)->estr = NULL; + es_addStr(&estr, ((struct cnfstringval*)expr->r)->estr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } else if(expr->r->nodetype == 'N') { + es_str_t *numstr; + estr = ((struct cnfstringval*)expr->l)->estr; + ((struct cnfstringval*)expr->l)->estr = NULL; + numstr = es_newStrFromNumber(((struct cnfnumval*)expr->r)->val); + es_addStr(&estr, numstr); + es_deleteStr(numstr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } + } else if(expr->l->nodetype == 'N') { + if(expr->r->nodetype == 'S') { + estr = es_newStrFromNumber(((struct cnfnumval*)expr->l)->val); + es_addStr(&estr, ((struct cnfstringval*)expr->r)->estr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } else if(expr->r->nodetype == 'S') { + es_str_t *numstr; + estr = es_newStrFromNumber(((struct cnfnumval*)expr->l)->val); + numstr = es_newStrFromNumber(((struct cnfnumval*)expr->r)->val); + es_addStr(&estr, numstr); + es_deleteStr(numstr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } + } +} + + +/* optimize comparisons with syslog severity/facility. This is a special + * handler as the numerical values also support GT, LT, etc ops. + */ +static inline struct cnfexpr* +cnfexprOptimize_CMP_severity_facility(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(!strcmp("$syslogseverity", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'N') { + int sev = (int) ((struct cnfnumval*)expr->r)->val; + if(sev >= 0 && sev <= 7) { + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); /* fac is irrelevant, set below... */ + prifiltSetSeverity(func->funcdata, sev, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } else { + parser_errmsg("invalid syslogseverity %d, expression will always " + "evaluate to FALSE", sev); + } + } + } else if(!strcmp("$syslogfacility", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'N') { + int fac = (int) ((struct cnfnumval*)expr->r)->val; + if(fac >= 0 && fac <= 24) { + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); /* fac is irrelevant, set below... */ + prifiltSetFacility(func->funcdata, fac, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } else { + parser_errmsg("invalid syslogfacility %d, expression will always " + "evaluate to FALSE", fac); + } + } + } + return expr; +} + +/* optimize a comparison with a variable as left-hand operand + * NOTE: Currently support CMP_EQ, CMP_NE only and code NEEDS + * TO BE CHANGED for other comparisons! + */ +static inline struct cnfexpr* +cnfexprOptimize_CMP_var(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(!strcmp("$syslogfacility-text", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'S') { + char *cstr = es_str2cstr(((struct cnfstringval*)expr->r)->estr, NULL); + int fac = decodeSyslogName((uchar*)cstr, syslogFacNames); + if(fac == -1) { + parser_errmsg("invalid facility '%s', expression will always " + "evaluate to FALSE", cstr); + } else { + /* we can acutally optimize! */ + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(fac); + if(expr->nodetype == CMP_NE) + prifiltInvert(func->funcdata); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } + free(cstr); + } + } else if(!strcmp("$syslogseverity-text", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'S') { + char *cstr = es_str2cstr(((struct cnfstringval*)expr->r)->estr, NULL); + int sev = decodeSyslogName((uchar*)cstr, syslogPriNames); + if(sev == -1) { + parser_errmsg("invalid syslogseverity '%s', expression will always " + "evaluate to FALSE", cstr); + } else { + /* we can acutally optimize! */ + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); + prifiltSetSeverity(func->funcdata, sev, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } + free(cstr); + } + } else { + expr = cnfexprOptimize_CMP_severity_facility(expr); + } + return expr; +} + +static inline struct cnfexpr* +cnfexprOptimize_NOT(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(expr->r->nodetype == 'F') { + func = (struct cnffunc *)expr->r; + if(func->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimize NOT prifilt() to inverted prifilt()\n"); + expr->r = NULL; + cnfexprDestruct(expr); + prifiltInvert(func->funcdata); + expr = (struct cnfexpr*) func; + } + } + return expr; +} + +static inline struct cnfexpr* +cnfexprOptimize_AND_OR(struct cnfexpr *expr) +{ + struct cnffunc *funcl, *funcr; + + if(expr->l->nodetype == 'F') { + if(expr->r->nodetype == 'F') { + funcl = (struct cnffunc *)expr->l; + funcr = (struct cnffunc *)expr->r; + if(funcl->fID == CNFFUNC_PRIFILT && funcr->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimize combine AND/OR prifilt()\n"); + expr->l = NULL; + prifiltCombine(funcl->funcdata, funcr->funcdata, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) funcl; + } + } + } + return expr; +} + + +/* optimize array for EQ/NEQ comparisons. We sort the array in + * this case so that we can apply binary search later on. + */ +static inline void +cnfexprOptimize_CMPEQ_arr(struct cnfarray *arr) +{ + DBGPRINTF("optimizer: sorting array for CMP_EQ/NEQ comparison\n"); + qsort(arr->arr, arr->nmemb, sizeof(es_str_t*), qs_arrcmp); +} + + +/* (recursively) optimize an expression */ +struct cnfexpr* +cnfexprOptimize(struct cnfexpr *expr) { - dbgprintf("------ start rule %p:\n", rule); - dbgprintf("%s: ", cnfFiltType2str(rule->filttype)); - switch(rule->filttype) { - case CNFFILT_NONE: + long long ln, rn; + struct cnfexpr *exprswap; + + dbgprintf("optimize expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); + switch(expr->nodetype) { + case '&': + constFoldConcat(expr); break; - case CNFFILT_PRI: - case CNFFILT_PROP: - dbgprintf("%s\n", rule->filt.s); + case '+': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln + rn; + } break; - case CNFFILT_SCRIPT: - dbgprintf("\n"); - cnfexprPrint(rule->filt.expr, 0); + case '-': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln - rn; + } + break; + case '*': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln * rn; + } + break; + case '/': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln / rn; + } + break; + case '%': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln % rn; + } + break; + case CMP_NE: + case CMP_EQ: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'A') { + if(expr->r->nodetype == 'A') { + parser_errmsg("warning: '==' or '<>' " + "comparison of two constant string " + "arrays makes no sense"); + } else { /* swap for simpler execution step */ + exprswap = expr->l; + expr->l = expr->r; + expr->r = exprswap; + } + } else if(expr->l->nodetype == 'V') { + expr = cnfexprOptimize_CMP_var(expr); + } + if(expr->r->nodetype == 'A') { + cnfexprOptimize_CMPEQ_arr((struct cnfarray *)expr->r); + } + break; + case CMP_LE: + case CMP_GE: + case CMP_LT: + case CMP_GT: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_CMP_severity_facility(expr); + break; + case CMP_CONTAINS: + case CMP_CONTAINSI: + case CMP_STARTSWITH: + case CMP_STARTSWITHI: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + break; + case AND: + case OR: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_AND_OR(expr); + break; + case NOT: + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_NOT(expr); + break; + default:/* nodetypes we cannot optimize */ break; } - cnfactlstPrint(rule->actlst); - dbgprintf("------ end rule %p\n", rule); + return expr; } -/* note: the sysline itself was already freed during processing - * and as such MUST NOT be freed again! +/* removes NOPs from a statement list and returns the + * first non-NOP entry. */ -void -cnfcfsyslinelstDestruct(struct cnfcfsyslinelst *cfslst) +static inline struct cnfstmt * +removeNOPs(struct cnfstmt *root) { - struct cnfcfsyslinelst *toDel; - while(cfslst != NULL) { - toDel = cfslst; - cfslst = cfslst->next; - free(toDel); + struct cnfstmt *stmt, *toDel, *prevstmt = NULL; + struct cnfstmt *newRoot = NULL; + + if(root == NULL) goto done; + stmt = root; + while(stmt != NULL) { + if(stmt->nodetype == S_NOP) { + if(prevstmt != NULL) + /* end chain, is rebuild if more non-NOPs follow */ + prevstmt->next = NULL; + toDel = stmt; + stmt = stmt->next; + cnfstmtDestruct(toDel); + } else { + if(newRoot == NULL) + newRoot = stmt; + if(prevstmt != NULL) + prevstmt->next = stmt; + prevstmt = stmt; + stmt = stmt->next; + } } +done: return newRoot; } -void -cnfruleDestruct(struct cnfrule *rule) + +static inline void +cnfstmtOptimizeIf(struct cnfstmt *stmt) +{ + struct cnfstmt *t_then, *t_else; + struct cnfexpr *expr; + struct cnffunc *func; + struct funcData_prifilt *prifilt; + + expr = stmt->d.s_if.expr = cnfexprOptimize(stmt->d.s_if.expr); + stmt->d.s_if.t_then = removeNOPs(stmt->d.s_if.t_then); + stmt->d.s_if.t_else = removeNOPs(stmt->d.s_if.t_else); + cnfstmtOptimize(stmt->d.s_if.t_then); + cnfstmtOptimize(stmt->d.s_if.t_else); + + if(stmt->d.s_if.expr->nodetype == 'F') { + func = (struct cnffunc*)expr; + if(func->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimizer: change IF to PRIFILT\n"); + t_then = stmt->d.s_if.t_then; + t_else = stmt->d.s_if.t_else; + stmt->nodetype = S_PRIFILT; + prifilt = (struct funcData_prifilt*) func->funcdata; + memcpy(stmt->d.s_prifilt.pmask, prifilt->pmask, + sizeof(prifilt->pmask)); + stmt->d.s_prifilt.t_then = t_then; + stmt->d.s_prifilt.t_else = t_else; + if(func->nParams == 0) + stmt->printable = (uchar*)strdup("[Optimizer Result]"); + else + stmt->printable = (uchar*) + es_str2cstr(((struct cnfstringval*)func->expr[0])->estr, NULL); + cnfexprDestruct(expr); + cnfstmtOptimizePRIFilt(stmt); + } + } +} + +static inline void +cnfstmtOptimizeAct(struct cnfstmt *stmt) +{ + action_t *pAct; + + pAct = stmt->d.act; + if(!strcmp((char*)modGetName(pAct->pMod), "builtin:omdiscard")) { + DBGPRINTF("optimizer: replacing omdiscard by STOP\n"); + actionDestruct(stmt->d.act); + stmt->nodetype = S_STOP; + } +} + +static void +cnfstmtOptimizePRIFilt(struct cnfstmt *stmt) +{ + int i; + int isAlways = 1; + struct cnfstmt *subroot, *last; + + stmt->d.s_prifilt.t_then = removeNOPs(stmt->d.s_prifilt.t_then); + cnfstmtOptimize(stmt->d.s_prifilt.t_then); + + for(i = 0; i <= LOG_NFACILITIES; i++) + if(stmt->d.s_prifilt.pmask[i] != 0xff) { + isAlways = 0; + break; + } + if(!isAlways) + goto done; + + DBGPRINTF("optimizer: removing always-true PRIFILT %p\n", stmt); + if(stmt->d.s_prifilt.t_else != NULL) { + parser_errmsg("error: always-true PRI filter has else part!\n"); + cnfstmtDestruct(stmt->d.s_prifilt.t_else); + } + free(stmt->printable); + stmt->printable = NULL; + subroot = stmt->d.s_prifilt.t_then; + if(subroot == NULL) { + /* very strange, we set it to NOP, best we can do + * This case is NOT expected in practice + */ + stmt->nodetype = S_NOP; + goto done; + } + for(last = subroot ; last->next != NULL ; last = last->next) + /* find last node in subtree */; + last->next = stmt->next; + memcpy(stmt, subroot, sizeof(struct cnfstmt)); + free(subroot); + +done: return; +} + +/* we abuse "optimize" a bit. Actually, we obtain a ruleset pointer, as + * all rulesets are only known later in the process (now!). + */ +static void +cnfstmtOptimizeCall(struct cnfstmt *stmt) { - if( rule->filttype == CNFFILT_PRI - || rule->filttype == CNFFILT_PROP) - free(rule->filt.s); - cnfactlstDestruct(rule->actlst); - free(rule); + ruleset_t *pRuleset; + rsRetVal localRet; + uchar *rsName; + + rsName = (uchar*) es_str2cstr(stmt->d.s_call.name, NULL); + localRet = rulesetGetRuleset(loadConf, &pRuleset, rsName); + if(localRet != RS_RET_OK) { + /* in that case, we accept that a NOP will "survive" */ + parser_errmsg("ruleset '%s' cannot be found\n", rsName); + es_deleteStr(stmt->d.s_call.name); + stmt->nodetype = S_NOP; + goto done; + } + DBGPRINTF("CALL obtained ruleset ptr %p for ruleset %s\n", pRuleset, rsName); + stmt->d.s_call.stmt = pRuleset->root; +done: + free(rsName); + return; } +/* (recursively) optimize a statement */ +void +cnfstmtOptimize(struct cnfstmt *root) +{ + struct cnfstmt *stmt; + if(root == NULL) goto done; + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { +dbgprintf("RRRR: stmtOptimize: stmt %p, nodetype %u\n", stmt, stmt->nodetype); + switch(stmt->nodetype) { + case S_IF: + cnfstmtOptimizeIf(stmt); + break; + case S_PRIFILT: + cnfstmtOptimizePRIFilt(stmt); + break; + case S_PROPFILT: + stmt->d.s_propfilt.t_then = removeNOPs(stmt->d.s_propfilt.t_then); + cnfstmtOptimize(stmt->d.s_propfilt.t_then); + break; + case S_SET: + stmt->d.s_set.expr = cnfexprOptimize(stmt->d.s_set.expr); + break; + case S_ACT: + cnfstmtOptimizeAct(stmt); + break; + case S_CALL: + cnfstmtOptimizeCall(stmt); + break; + case S_STOP: + if(stmt->next != NULL) + parser_errmsg("STOP is followed by unreachable statements!\n"); + break; + case S_UNSET: /* nothing to do */ + break; + case S_NOP: + DBGPRINTF("optimizer error: we see a NOP, how come?\n"); + break; + default: + dbgprintf("error: unknown stmt type %u during optimizer run\n", + (unsigned) stmt->nodetype); + break; + } + } +done: return; +} + struct cnffparamlst * cnffparamlstNew(struct cnfexpr *expr, struct cnffparamlst *next) @@ -1735,6 +3084,20 @@ funcName2ID(es_str_t *fname, unsigned short nParams) return CNFFUNC_INVALID; } return CNFFUNC_RE_MATCH; + } else if(!es_strbufcmp(fname, (unsigned char*)"field", sizeof("field") - 1)) { + if(nParams != 3) { + parser_errmsg("number of parameters for field() must be three " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_FIELD; + } else if(!es_strbufcmp(fname, (unsigned char*)"prifilt", sizeof("prifilt") - 1)) { + if(nParams != 1) { + parser_errmsg("number of parameters for prifilt() must be one " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_PRIFILT; } else { return CNFFUNC_INVALID; } @@ -1775,6 +3138,30 @@ finalize_it: RETiRet; } + +static inline rsRetVal +initFunc_prifilt(struct cnffunc *func) +{ + struct funcData_prifilt *pData; + uchar *cstr; + DEFiRet; + + func->funcdata = NULL; + if(func->expr[0]->nodetype != 'S') { + parser_errmsg("param 1 of prifilt() must be a constant string"); + FINALIZE; + } + + CHKmalloc(pData = calloc(1, sizeof(struct funcData_prifilt))); + func->funcdata = pData; + cstr = (uchar*)es_str2cstr(((struct cnfstringval*) func->expr[0])->estr, NULL); + CHKiRet(DecodePRIFilter(cstr, pData->pmask)); + free(cstr); +finalize_it: + RETiRet; +} + + struct cnffunc * cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) { @@ -1792,6 +3179,7 @@ cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) func->nodetype = 'F'; func->fname = fname; func->nParams = nParams; + func->funcdata = NULL; func->fID = funcName2ID(fname, nParams); /* shuffle params over to array (access speed!) */ param = paramlst; @@ -1807,12 +3195,39 @@ cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) /* need to compile the regexp in param 2, so this MUST be a constant */ initFunc_re_match(func); break; + case CNFFUNC_PRIFILT: + initFunc_prifilt(func); + break; default:break; } } return func; } + +/* A special function to create a prifilt() expression during optimization + * phase. + */ +struct cnffunc * +cnffuncNew_prifilt(int fac) +{ + struct cnffunc* func; + + if((func = malloc(sizeof(struct cnffunc))) != NULL) { + func->nodetype = 'F'; + func->fname = es_newStrFromCStr("prifilt", sizeof("prifilt")-1); + func->nParams = 0; + func->fID = CNFFUNC_PRIFILT; + func->funcdata = calloc(1, sizeof(struct funcData_prifilt)); + ((struct funcData_prifilt *)func->funcdata)->pmask[fac >> 3] = TABLE_ALLPRI; + } + return func; +} + + +/* returns 0 if everything is OK and config parsing shall continue, + * and 1 if things are so wrong that config parsing shall be aborted. + */ int cnfDoInclude(char *name) { @@ -1823,6 +3238,7 @@ cnfDoInclude(char *name) glob_t cfgFiles; struct stat fileInfo; char nameBuf[MAXFNAME+1]; + char cwdBuf[MAXFNAME+1]; finalName = name; if(stat(name, &fileInfo) == 0) { @@ -1833,21 +3249,37 @@ cnfDoInclude(char *name) finalName = nameBuf; } } + /* Use GLOB_MARK to append a trailing slash for directories. */ - result = glob(finalName, GLOB_MARK, NULL, &cfgFiles); - if(result == GLOB_NOSPACE || result == GLOB_ABORTED || cfgFiles.gl_pathc == 0) { + /* Use GLOB_NOMAGIC to detect wildcards that match nothing. */ + result = glob(finalName, GLOB_MARK | GLOB_NOMAGIC, NULL, &cfgFiles); + + /* Silently ignore wildcards that match nothing */ + if(result == GLOB_NOMATCH) { + return 0; + } + + if(result == GLOB_NOSPACE || result == GLOB_ABORTED) { char errStr[1024]; rs_strerror_r(errno, errStr, sizeof(errStr)); - parser_errmsg("error accessing config file or directory '%s': %s", - finalName, errStr); + if(getcwd(cwdBuf, sizeof(cwdBuf)) == NULL) + strcpy(cwdBuf, "??getcwd() failed??"); + parser_errmsg("error accessing config file or directory '%s' [cwd:%s]: %s", + finalName, cwdBuf, errStr); return 1; } for(i = 0; i < cfgFiles.gl_pathc; i++) { cfgFile = cfgFiles.gl_pathv[i]; - - if(stat(cfgFile, &fileInfo) != 0) - continue; /* continue with the next file if we can't stat() the file */ + if(stat(cfgFile, &fileInfo) != 0) { + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + if(getcwd(cwdBuf, sizeof(cwdBuf)) == NULL) + strcpy(cwdBuf, "??getcwd() failed??"); + parser_errmsg("error accessing config file or directory '%s' " + "[cwd: %s]: %s", cfgFile, cwdBuf, errStr); + return 1; + } if(S_ISREG(fileInfo.st_mode)) { /* config file */ dbgprintf("requested to include config file '%s'\n", cfgFile); @@ -1867,8 +3299,16 @@ cnfDoInclude(char *name) void varDelete(struct var *v) { - if(v->datatype == 'S') + switch(v->datatype) { + case 'S': es_deleteStr(v->d.estr); + break; + case 'A': + cnfarrayContentDestruct(v->d.ar); + free(v->d.ar); + break; + default:break; + } } void @@ -1876,7 +3316,9 @@ cnfparamvalsDestruct(struct cnfparamvals *paramvals, struct cnfparamblk *blk) { int i; for(i = 0 ; i < blk->nParams ; ++i) { - varDelete(¶mvals[i].val); + if(paramvals[i].bUsed) { + varDelete(¶mvals[i].val); + } } free(paramvals); } @@ -1909,6 +3351,15 @@ cstrPrint(char *text, es_str_t *estr) free(str); } +char * +rmLeadingSpace(char *s) +{ + char *p; + for(p = s ; *p && isspace(*p) ; ++p) + ; + return(p); +} + /* init must be called once before any parsing of the script files start */ rsRetVal initRainerscript(void) diff --git a/grammar/rainerscript.h b/grammar/rainerscript.h index f5627597..59ce53f3 100644 --- a/grammar/rainerscript.h +++ b/grammar/rainerscript.h @@ -3,7 +3,11 @@ #include <stdio.h> #include <libestr.h> #include <typedefs.h> +#include <sys/types.h> +#include <regex.h> + +#define LOG_NFACILITIES 24 /* current number of syslog facilities */ #define CNFFUNC_MAX_ARGS 32 /**< maximum number of arguments that any function can have (among * others, this is used to size data structures). @@ -13,6 +17,7 @@ extern int Debug; /* 1 if in debug mode, 0 otherwise -- to be enhanced */ enum cnfobjType { CNFOBJ_ACTION, + CNFOBJ_RULESET, CNFOBJ_GLOBAL, CNFOBJ_INPUT, CNFOBJ_MODULE, @@ -29,6 +34,9 @@ cnfobjType2str(enum cnfobjType ot) case CNFOBJ_ACTION: return "action"; break; + case CNFOBJ_RULESET: + return "ruleset"; + break; case CNFOBJ_GLOBAL: return "global"; break; @@ -63,16 +71,20 @@ enum cnfactType { CNFACT_V2, CNFACT_LEGACY }; struct var { union { es_str_t *estr; - struct cnfexpr *expr; + struct cnfarray *ar; long long n; + struct json_object *json; } d; - char datatype; /* 'N' number, 'S' string, 'E' expression */ + char datatype; /* 'N' number, 'S' string, 'J' JSON, 'A' array + * Note: 'A' is only supported during config phase + */ }; struct cnfobj { enum cnfobjType objType; struct nvlst *nvlst; struct objlst *subobjs; + struct cnfstmt *script; }; struct objlst { @@ -91,23 +103,6 @@ struct nvlst { */ }; -struct cnfcfsyslinelst { - struct cnfcfsyslinelst *next; - char *line; -}; - -struct cnfactlst { - struct cnfactlst *next; - struct cnfcfsyslinelst *syslines; - enum cnfactType actType; - union { - struct nvlst *lst; - char *legActLine; - } data; - char *cnfFile; - int lineno; -}; - /* the following structures support expressions, and may (very much later * be the sole foundation for the AST. * @@ -118,7 +113,19 @@ struct cnfactlst { * R - rule * S - string * V - var + * A - (string) array + * ... plus the S_* #define's below: */ +#define S_STOP 4000 +#define S_PRIFILT 4001 +#define S_PROPFILT 4002 +#define S_IF 4003 +#define S_ACT 4004 +#define S_NOP 4005 /* usually used to disable some statement */ +#define S_SET 4006 +#define S_UNSET 4007 +#define S_CALL 4008 + enum cnfFiltType { CNFFILT_NONE, CNFFILT_PRI, CNFFILT_PROP, CNFFILT_SCRIPT }; static inline char* cnfFiltType2str(enum cnfFiltType filttype) @@ -137,14 +144,44 @@ cnfFiltType2str(enum cnfFiltType filttype) } -struct cnfrule { +struct cnfstmt { unsigned nodetype; - enum cnfFiltType filttype; + struct cnfstmt *next; + uchar *printable; /* printable text for debugging */ union { - char *s; - struct cnfexpr *expr; - } filt; - struct cnfactlst *actlst; + struct { + struct cnfexpr *expr; + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_if; + struct { + uchar *varname; + struct cnfexpr *expr; + } s_set; + struct { + uchar *varname; + } s_unset; + struct { + es_str_t *name; + struct cnfstmt *stmt; + } s_call; + struct { + uchar pmask[LOG_NFACILITIES+1]; /* priority mask */ + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_prifilt; + struct { + fiop_t operation; + regex_t *regex_cache;/* cache for compiled REs, if used */ + struct cstr_s *pCSCompValue;/* value to "compare" against */ + sbool isNegated; + uintTiny propID;/* ID of the requested property */ + es_str_t *propName;/* name of property for CEE-based filters */ + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_propfilt; + struct action_s *act; + } d; }; struct cnfexpr { @@ -168,6 +205,12 @@ struct cnfvar { char *name; }; +struct cnfarray { + unsigned nodetype; + int nmemb; + es_str_t **arr; +}; + struct cnffparamlst { unsigned nodetype; /* P */ struct cnffparamlst *next; @@ -182,7 +225,9 @@ enum cnffuncid { CNFFUNC_TOLOWER, CNFFUNC_CSTR, CNFFUNC_CNUM, - CNFFUNC_RE_MATCH + CNFFUNC_RE_MATCH, + CNFFUNC_FIELD, + CNFFUNC_PRIFILT }; struct cnffunc { @@ -233,13 +278,19 @@ struct cnfparamvals { /* the values we obtained for param descr. */ unsigned char bUsed; }; +struct funcData_prifilt { + uchar pmask[LOG_NFACILITIES+1]; /* priority mask */ +}; + int cnfParseBuffer(char *buf, unsigned lenBuf); void readConfFile(FILE *fp, es_str_t **str); struct objlst* objlstNew(struct cnfobj *obj); void objlstDestruct(struct objlst *lst); void objlstPrint(struct objlst *lst); -struct nvlst* nvlstNew(es_str_t *name, es_str_t *value); +struct nvlst* nvlstNewArray(struct cnfarray *ar); +struct nvlst* nvlstNewStr(es_str_t *value); +struct nvlst* nvlstSetName(struct nvlst *lst, es_str_t *name); void nvlstDestruct(struct nvlst *lst); void nvlstPrint(struct nvlst *lst); void nvlstChkUnused(struct nvlst *lst); @@ -247,11 +298,6 @@ struct nvlst* nvlstFindName(struct nvlst *lst, es_str_t *name); struct cnfobj* cnfobjNew(enum cnfobjType objType, struct nvlst *lst); void cnfobjDestruct(struct cnfobj *o); void cnfobjPrint(struct cnfobj *o); -struct cnfactlst* cnfactlstNew(enum cnfactType actType, struct nvlst *lst, char *actLine); -void cnfactlstDestruct(struct cnfactlst *actlst); -void cnfactlstPrint(struct cnfactlst *actlst); -struct cnfactlst* cnfactlstAddSysline(struct cnfactlst* actlst, char *line); -struct cnfactlst* cnfactlstReverse(struct cnfactlst *actlst); struct cnfexpr* cnfexprNew(unsigned nodetype, struct cnfexpr *l, struct cnfexpr *r); void cnfexprPrint(struct cnfexpr *expr, int indent); void cnfexprEval(struct cnfexpr *expr, struct var *ret, void *pusr); @@ -259,9 +305,6 @@ int cnfexprEvalBool(struct cnfexpr *expr, void *usrptr); void cnfexprDestruct(struct cnfexpr *expr); struct cnfnumval* cnfnumvalNew(long long val); struct cnfstringval* cnfstringvalNew(es_str_t *estr); -struct cnfrule * cnfruleNew(enum cnfFiltType filttype, struct cnfactlst *actlst); -void cnfruleDestruct(struct cnfrule *rule); -void cnfrulePrint(struct cnfrule *rule); struct cnfvar* cnfvarNew(char *name); struct cnffunc * cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst); struct cnffparamlst * cnffparamlstNew(struct cnfexpr *expr, struct cnffparamlst *next); @@ -270,9 +313,29 @@ int cnfparamGetIdx(struct cnfparamblk *params, char *name); struct cnfparamvals* nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, struct cnfparamvals *vals); void cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals); +int cnfparamvalsIsSet(struct cnfparamblk *params, struct cnfparamvals *vals); void varDelete(struct var *v); void cnfparamvalsDestruct(struct cnfparamvals *paramvals, struct cnfparamblk *blk); -void cnfcfsyslinelstDestruct(struct cnfcfsyslinelst *cfslst); +struct cnfstmt * cnfstmtNew(unsigned s_type); +void cnfstmtPrint(struct cnfstmt *stmt, int indent); +struct cnfstmt* scriptAddStmt(struct cnfstmt *root, struct cnfstmt *s); +struct objlst* objlstAdd(struct objlst *root, struct cnfobj *o); +char *rmLeadingSpace(char *s); +struct cnfstmt * cnfstmtNewPRIFILT(char *prifilt, struct cnfstmt *t_then); +struct cnfstmt * cnfstmtNewPROPFILT(char *propfilt, struct cnfstmt *t_then); +struct cnfstmt * cnfstmtNewAct(struct nvlst *lst); +struct cnfstmt * cnfstmtNewLegaAct(char *actline); +struct cnfstmt * cnfstmtNewSet(char *var, struct cnfexpr *expr); +struct cnfstmt * cnfstmtNewUnset(char *var); +struct cnfstmt * cnfstmtNewCall(es_str_t *name); +struct cnfstmt * cnfstmtNewContinue(void); +void cnfstmtDestruct(struct cnfstmt *root); +void cnfstmtOptimize(struct cnfstmt *root); +struct cnfarray* cnfarrayNew(es_str_t *val); +struct cnfarray* cnfarrayDup(struct cnfarray *old); +struct cnfarray* cnfarrayAdd(struct cnfarray *ar, es_str_t *val); +void cnfarrayContentDestruct(struct cnfarray *ar); +char* getFIOPName(unsigned iFIOP); rsRetVal initRainerscript(void); void unescapeStr(uchar *s, int len); @@ -262,7 +262,7 @@ rsRetVal parsDelimCStr(rsParsObj *pThis, cstr_t **ppCStr, char cDelim, int bTrim pC = rsCStrGetBufBeg(pThis->pCStr) + pThis->iCurrPos; - while(pThis->iCurrPos < rsCStrLen(pThis->pCStr) && *pC != cDelim && *pC != ' ') { + while(pThis->iCurrPos < rsCStrLen(pThis->pCStr) && *pC != cDelim) { CHKiRet(cstrAppendChar(pCStr, bConvLower ? tolower(*pC) : *pC)); ++pThis->iCurrPos; ++pC; diff --git a/platform/README b/platform/README new file mode 100644 index 00000000..adbc3013 --- /dev/null +++ b/platform/README @@ -0,0 +1,4 @@ +This subdirectory contains platform-specific files. They are maintained +based on a best effort basis, and are not necessarily the same like the +specific platform ships them. Some files are changed in the way the +rsyslog projects would recommend them; some may even be outdated. diff --git a/freebsd/rsyslogd b/platform/freebsd/rsyslogd index 1cbcb6fe..1cbcb6fe 100755 --- a/freebsd/rsyslogd +++ b/platform/freebsd/rsyslogd diff --git a/platform/redhat/rsyslog.conf b/platform/redhat/rsyslog.conf new file mode 100644 index 00000000..340874d1 --- /dev/null +++ b/platform/redhat/rsyslog.conf @@ -0,0 +1,88 @@ +/* rsyslog configuration file (for Red Hat-based systems) + * note that most of this config file uses old-style format, + * because it is well-known AND quite suitable for simple cases + * like we have with the default config. For more advanced + * things, RainerScript configuration is suggested. + * + * For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html + * or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html + * If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html + */ + +#### MODULES #### + +module(load="imuxsock") # provides support for local system logging (e.g. via logger command) +module(load="imklog") # provides kernel logging support (previously done by rklogd) +#module(load"immark") # provides --MARK-- message capability + +# Provides UDP syslog reception +# for parameters see http://www.rsyslog.com/doc/imudp.html +#module(load="imudp") # needs to be done just once +#input(type="imudp" port="514") + +# Provides TCP syslog reception +# for parameters see http://www.rsyslog.com/doc/imtcp.html +#module(load="imtcp") # needs to be done just once +#input(type="imtcp" port="514") + + +#### GLOBAL DIRECTIVES #### + +# Use default timestamp format +$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + +# File syncing capability is disabled by default. This feature is usually not required, +# not useful and an extreme performance hit +#$ActionFileEnableSync on + +# Include all config files in /etc/rsyslog.d/ +$IncludeConfig /etc/rsyslog.d/*.conf + + +#### RULES #### + +# Log all kernel messages to the console. +# Logging much else clutters up the screen. +#kern.* /dev/console + +# Log anything (except mail) of level info or higher. +# Don't log private authentication messages! +*.info;mail.none;authpriv.none;cron.none /var/log/messages + +# The authpriv file has restricted access. +authpriv.* /var/log/secure + +# Log all the mail messages in one place. +mail.* /var/log/maillog + + +# Log cron stuff +cron.* /var/log/cron + +# Everybody gets emergency messages +*.emerg :omusrmsg:* + +# Save news errors of level crit and higher in a special file. +uucp,news.crit /var/log/spooler + +# Save boot messages also to boot.log +local7.* /var/log/boot.log + + +# ### begin forwarding rule ### +# The statement between the begin ... end define a SINGLE forwarding +# rule. They belong together, do NOT split them. If you create multiple +# forwarding rules, duplicate the whole block! +# Remote Logging (we use TCP for reliable delivery) +# +# An on-disk queue is created for this action. If the remote host is +# down, messages are spooled to disk and sent when it is up again. +#$WorkDirectory /var/lib/rsyslog # where to place spool files +#$ActionQueueFileName fwdRule1 # unique name prefix for spool files +#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) +#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown +#$ActionQueueType LinkedList # run asynchronously +#$ActionResumeRetryCount -1 # infinite retries if host is down +# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional +#*.* @@remote-host:514 +# ### end of the forwarding rule ### diff --git a/slackware/rc.rsyslogd b/platform/slackware/rc.rsyslogd index f5f8f458..f5f8f458 100755 --- a/slackware/rc.rsyslogd +++ b/platform/slackware/rc.rsyslogd diff --git a/plugins/cust1/cust1.c b/plugins/cust1/cust1.c deleted file mode 100644 index e69de29b..00000000 --- a/plugins/cust1/cust1.c +++ /dev/null diff --git a/plugins/im3195/im3195.c b/plugins/im3195/im3195.c index c75e0e34..b8a4a140 100644 --- a/plugins/im3195/im3195.c +++ b/plugins/im3195/im3195.c @@ -48,6 +48,7 @@ #include "cfsysline.h" #include "msg.h" #include "errmsg.h" +#include "unicode-helper.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -56,6 +57,7 @@ MODULE_CNFNAME("im3195") /* Module static data */ DEF_IMOD_STATIC_DATA DEFobjCurrIf(errmsg) +DEFobjCurrIf(prop) /* configuration settings */ @@ -71,6 +73,8 @@ static int listenPort = 601; */ static srAPIObj* pAPI; +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ + /* This method is called when a message has been fully received. * It passes the received message to the rsyslog main message @@ -90,7 +94,7 @@ void OnReceive(srAPIObj __attribute__((unused)) *pMyAPI, srSLMGObj* pSLMG) srSLMGGetRawMSG(pSLMG, &pszRawMsg); parseAndSubmitMessage(fromHost, fromHostIP, pszRawMsg, strlen((char*)pszRawMsg), - PARSE_HOSTNAME, eFLOWCTL_FULL_DELAY, (uchar*)"im3195", NULL, 0, NULL); + PARSE_HOSTNAME, eFLOWCTL_FULL_DELAY, pInputName, NULL, 0, NULL); } @@ -171,8 +175,12 @@ ENDafterRun BEGINmodExit CODESTARTmodExit srAPIExitLib(pAPI); /* terminate liblogging */ + /* global variable cleanup */ + if(pInputName != NULL) + prop.Destruct(&pInputName); /* release objects we used */ objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDmodExit @@ -193,9 +201,15 @@ CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"input3195listenport", 0, eCmdHdlrInt, NULL, &listenPort, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); + + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("im3195"), sizeof("im3195") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + ENDmodInit /* vim:set ai: */ diff --git a/plugins/imdiag/imdiag.c b/plugins/imdiag/imdiag.c index 09742537..5fdc6ef1 100644 --- a/plugins/imdiag/imdiag.c +++ b/plugins/imdiag/imdiag.c @@ -53,11 +53,11 @@ #include "srUtils.h" #include "msg.h" #include "datetime.h" +#include "ratelimit.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP -MODULE_CNFNAME("imdiag") /* static data */ DEF_IMOD_STATIC_DATA @@ -200,7 +200,7 @@ finalize_it: /* actually submit a message to the rsyslog core */ static rsRetVal -doInjectMsg(int iNum) +doInjectMsg(int iNum, ratelimit_t *ratelimiter) { uchar szMsg[1024]; msg_t *pMsg; @@ -220,7 +220,7 @@ doInjectMsg(int iNum) pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; MsgSetRcvFrom(pMsg, pRcvDummy); CHKiRet(MsgSetRcvFromIP(pMsg, pRcvIPDummy)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(ratelimitAddMsg(ratelimiter, NULL, pMsg)); finalize_it: RETiRet; @@ -238,6 +238,7 @@ injectMsg(uchar *pszCmd, tcps_sess_t *pSess) int iFrom; int nMsgs; int i; + ratelimit_t *ratelimit; DEFiRet; /* we do not check errors here! */ @@ -245,13 +246,15 @@ injectMsg(uchar *pszCmd, tcps_sess_t *pSess) iFrom = atoi((char*)wordBuf); getFirstWord(&pszCmd, wordBuf, sizeof(wordBuf)/sizeof(uchar), TO_LOWERCASE); nMsgs = atoi((char*)wordBuf); + ratelimitNew(&ratelimit, "imdiag", "injectmsg"); for(i = 0 ; i < nMsgs ; ++i) { - doInjectMsg(i + iFrom); + doInjectMsg(i + iFrom, ratelimit); } CHKiRet(sendResponse(pSess, "%d messages injected\n", nMsgs)); DBGPRINTF("imdiag: %d messages injected\n", nMsgs); + ratelimitDestruct(ratelimit); finalize_it: RETiRet; diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 453b6b05..0f155c10 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -48,6 +48,7 @@ #include "prop.h" #include "stringbuf.h" #include "ruleset.h" +#include "ratelimit.h" MODULE_TYPE_INPUT /* must be present for input modules, do not remove */ MODULE_TYPE_NOKEEP @@ -82,6 +83,7 @@ typedef struct fileInfo_s { strm_t *pStrm; /* its stream (NULL if not assigned) */ int readMode; /* which mode to use in ReadMulteLine call? */ ruleset_t *pRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + ratelimit_t *ratelimiter; multi_submit_t multiSub; } fileInfo_t; @@ -189,9 +191,7 @@ static rsRetVal enqLine(fileInfo_t *pInfo, cstr_t *cstrLine) pMsg->iFacility = LOG_FAC(pInfo->iFacility); pMsg->iSeverity = LOG_PRI(pInfo->iSeverity); MsgSetRuleset(pMsg, pInfo->pRuleset); - pInfo->multiSub.ppMsgs[pInfo->multiSub.nElem++] = pMsg; - if(pInfo->multiSub.nElem == pInfo->multiSub.maxElem) - CHKiRet(multiSubmitMsg(&pInfo->multiSub)); + ratelimitAddMsg(pInfo->ratelimiter, &pInfo->multiSub, pMsg); finalize_it: RETiRet; } @@ -246,6 +246,8 @@ finalize_it: strm.Destruct(&psSF); if(iRet != RS_RET_OK) { + if(pThis->pStrm != NULL) + strm.Destruct(&pThis->pStrm); CHKiRet(strm.Construct(&pThis->pStrm)); CHKiRet(strm.SettOperationsMode(pThis->pStrm, STREAMMODE_READ)); CHKiRet(strm.SetsType(pThis->pStrm, STREAMTYPE_FILE_MONITOR)); @@ -304,18 +306,7 @@ static rsRetVal pollFile(fileInfo_t *pThis, int *pbHadFileData) } finalize_it: - if(pThis->multiSub.nElem > 0) { - /* submit everything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&pThis->multiSub)); - } - ; /*EMPTY STATEMENT - needed to keep compiler happy - see below! */ - /* Note: the problem above is that pthread:cleanup_pop() is a macro which - * evaluates to something like "} while(0);". So the code would become - * "finalize_it: }", that is a label without a statement. The C standard does - * not permit this. So we add an empty statement "finalize_it: ; }" and - * everybody is happy. Note that without the ;, an error is reported only - * on some platforms/compiler versions. -- rgerhards, 2008-08-15 - */ + multiSubmitFlush(&pThis->multiSub); pthread_cleanup_pop(0); if(pCStr != NULL) { @@ -423,6 +414,7 @@ addListner(instanceConf_t *inst) pThis->lenTag = ustrlen(pThis->pszTag); pThis->pszStateFile = (uchar*) strdup((char*) inst->pszStateFile); + CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(msg_t*))); pThis->multiSub.maxElem = inst->nMultiSub; pThis->multiSub.nElem = 0; @@ -773,6 +765,8 @@ CODESTARTafterRun persistStrmState(&files[i]); strm.Destruct(&(files[i].pStrm)); } + ratelimitDestruct(files[i].ratelimiter); + free(files[i].multiSub.ppMsgs); free(files[i].pszFileName); free(files[i].pszTag); free(files[i].pszStateFile); diff --git a/plugins/imklog/bsd.c b/plugins/imklog/bsd.c index cddc6737..9c2eebb2 100644 --- a/plugins/imklog/bsd.c +++ b/plugins/imklog/bsd.c @@ -58,9 +58,6 @@ static int fklog = -1; /* kernel log fd */ #ifdef OS_LINUX /* submit a message to imklog Syslog() API. In this function, we check if * a kernel timestamp is present and, if so, extract and strip it. - * Note: this is an extra processing step. We should revisit the whole - * idea in v6 and remove all that old stuff that we do not longer need - * (like symbol resolution). <-- TODO * Note that this is heavily Linux specific and thus is not compiled or * used for BSD. * Special thanks to Lennart Poettering for suggesting on how to convert @@ -175,7 +172,7 @@ klogWillRun(modConfData_t *pModConf) fklog = open((char*)GetPath(pModConf), O_RDONLY, 0); if (fklog < 0) { - imklogLogIntMsg(RS_RET_ERR_OPEN_KLOG, "imklog: cannot open kernel log(%s): %s.", + imklogLogIntMsg(LOG_ERR, "imklog: cannot open kernel log(%s): %s.", GetPath(pModConf), rs_strerror_r(errno, errmsg, sizeof(errmsg))); ABORT_FINALIZE(RS_RET_ERR_OPEN_KLOG); } diff --git a/plugins/imklog/imklog.c b/plugins/imklog/imklog.c index a24fc63b..810ac264 100644 --- a/plugins/imklog/imklog.c +++ b/plugins/imklog/imklog.c @@ -95,7 +95,7 @@ static struct cnfparamdescr modpdescr[] = { { "permitnonkernelfacility", eCmdHdlrBinary, 0 }, { "consoleloglevel", eCmdHdlrInt, 0 }, { "parsekerneltimestamp", eCmdHdlrBinary, 0 }, - { "keepkerneltimestamp", eCmdHdlrBinary, 0 }, + { "keepkerneltimestamp", eCmdHdlrBinary, 0 }, { "internalmsgfacility", eCmdHdlrFacility, 0 } }; static struct cnfparamblk modpblk = @@ -105,7 +105,7 @@ static struct cnfparamblk modpblk = }; static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ -static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ +static prop_t *pLocalHostIP = NULL; static inline void initConfigSettings(void) @@ -150,7 +150,8 @@ enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity, struct timeval * MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); pMsg->iFacility = iFacility; pMsg->iSeverity = iSeverity; - CHKiRet(submitMsg(pMsg)); + /* note: we do NOT use rate-limiting, as the kernel itself does rate-limiting */ + CHKiRet(submitMsg2(pMsg)); finalize_it: RETiRet; @@ -294,6 +295,7 @@ CODESTARTbeginCnfLoad pModConf->bParseKernelStamp = 0; pModConf->bKeepKernelStamp = 0; pModConf->console_log_level = -1; + pModConf->bKeepKernelStamp = 0; pModConf->iFacilIntMsg = klogFacilIntMsg(); loadModConf->configSetViaV2Method = 0; bLegacyCnfModGlobalsPermitted = 1; diff --git a/plugins/imklog/imklog.h b/plugins/imklog/imklog.h index fa517ccc..1cf9b05a 100644 --- a/plugins/imklog/imklog.h +++ b/plugins/imklog/imklog.h @@ -35,9 +35,9 @@ struct modConfData_s { int iFacilIntMsg; uchar *pszPath; int console_log_level; - sbool bPermitNonKernel; sbool bParseKernelStamp; sbool bKeepKernelStamp; + sbool bPermitNonKernel; sbool configSetViaV2Method; }; diff --git a/plugins/imkmsg/Makefile.am b/plugins/imkmsg/Makefile.am new file mode 100644 index 00000000..87c177d2 --- /dev/null +++ b/plugins/imkmsg/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = imkmsg.la +imkmsg_la_SOURCES = imkmsg.c imkmsg.h + +imkmsg_la_SOURCES += kmsg.c + +imkmsg_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +imkmsg_la_LDFLAGS = -module -avoid-version +imkmsg_la_LIBADD = diff --git a/plugins/imkmsg/imkmsg.c b/plugins/imkmsg/imkmsg.c new file mode 100644 index 00000000..2a97f82d --- /dev/null +++ b/plugins/imkmsg/imkmsg.c @@ -0,0 +1,295 @@ +/* The kernel log module. + * + * This is rsyslog Linux only module for reading structured kernel logs. + * Module is based on imklog module so it retains its structure + * and other part is currently in kmsg.c file instead of this (imkmsg.c) + * For more information see that file. + * + * To test under Linux: + * echo test1 > /dev/kmsg + * + * Copyright (C) 2008-2012 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <assert.h> +#include <string.h> +#include <stdarg.h> +#include <ctype.h> +#include <stdlib.h> +#include <sys/socket.h> + +#include "dirty.h" +#include "cfsysline.h" +#include "obj.h" +#include "msg.h" +#include "module-template.h" +#include "datetime.h" +#include "imkmsg.h" +#include "net.h" +#include "glbl.h" +#include "prop.h" +#include "errmsg.h" +#include "unicode-helper.h" + +MODULE_TYPE_INPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("imkmsg") + +/* Module static data */ +DEF_IMOD_STATIC_DATA +DEFobjCurrIf(datetime) +DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) +DEFobjCurrIf(net) +DEFobjCurrIf(errmsg) + +/* config settings */ +typedef struct configSettings_s { + int iFacilIntMsg; /* the facility to use for internal messages (set by driver) */ +} configSettings_t; +static configSettings_t cs; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ +static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ + +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ +static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ + +static inline void +initConfigSettings(void) +{ + cs.iFacilIntMsg = klogFacilIntMsg(); +} + + +/* enqueue the the kernel message into the message queue. + * The provided msg string is not freed - thus must be done + * by the caller. + * rgerhards, 2008-04-12 + */ +static rsRetVal +enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity, struct timeval *tp, struct json_object *json) +{ + struct syslogTime st; + msg_t *pMsg; + DEFiRet; + + assert(msg != NULL); + assert(pszTag != NULL); + + if(tp == NULL) { + CHKiRet(msgConstruct(&pMsg)); + } else { + datetime.timeval2syslogTime(tp, &st); + CHKiRet(msgConstructWithTime(&pMsg, &st, tp->tv_sec)); + } + MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsgWOSize(pMsg, (char*)msg); + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, pLocalHostIP); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); + pMsg->iFacility = iFacility; + pMsg->iSeverity = iSeverity; + pMsg->json = json; + CHKiRet(submitMsg(pMsg)); + +finalize_it: + RETiRet; +} + + +/* log an imkmsg-internal message + * rgerhards, 2008-04-14 + */ +rsRetVal imkmsgLogIntMsg(int priority, char *fmt, ...) +{ + DEFiRet; + va_list ap; + uchar msgBuf[2048]; /* we use the same size as sysklogd to remain compatible */ + + va_start(ap, fmt); + vsnprintf((char*)msgBuf, sizeof(msgBuf) / sizeof(char), fmt, ap); + va_end(ap); + + logmsgInternal(NO_ERRCODE ,priority, msgBuf, 0); + + RETiRet; +} + + +/* log a message from /dev/kmsg + */ +rsRetVal Syslog(int priority, uchar *pMsg, struct timeval *tp, struct json_object *json) +{ + DEFiRet; + iRet = enqMsg((uchar*)pMsg, (uchar*) "kernel:", LOG_FAC(priority), LOG_PRI(priority), tp, json); + RETiRet; +} + + +/* helper for some klog drivers which need to know the MaxLine global setting. They can + * not obtain it themselfs, because they are no modules and can not query the object hander. + * It would probably be a good idea to extend the interface to support it, but so far + * we create a (sufficiently valid) work-around. -- rgerhards, 2008-11-24 + */ +int klog_getMaxLine(void) +{ + return glbl.GetMaxLine(); +} + + +BEGINrunInput +CODESTARTrunInput + /* this is an endless loop - it is terminated when the thread is + * signalled to do so. This, however, is handled by the framework, + * right into the sleep below. + */ + while(!pThrd->bShallStop) { + /* klogLogKMsg() waits for the next kernel message, obtains it + * and then submits it to the rsyslog main queue. + * rgerhards, 2008-04-09 + */ + CHKiRet(klogLogKMsg(runModConf)); + } +finalize_it: +ENDrunInput + + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; + /* init our settings */ + pModConf->iFacilIntMsg = klogFacilIntMsg(); + loadModConf->configSetViaV2Method = 0; + bLegacyCnfModGlobalsPermitted = 1; + /* init legacy config vars */ + initConfigSettings(); +ENDbeginCnfLoad + + +BEGINendCnfLoad +CODESTARTendCnfLoad + if(!loadModConf->configSetViaV2Method) { + /* persist module-specific settings from legacy config system */ + loadModConf->iFacilIntMsg = cs.iFacilIntMsg; + } + + loadModConf = NULL; /* done loading */ +ENDendCnfLoad + + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + + +BEGINactivateCnfPrePrivDrop +CODESTARTactivateCnfPrePrivDrop + runModConf = pModConf; + iRet = klogWillRun(runModConf); +ENDactivateCnfPrePrivDrop + + +BEGINactivateCnf +CODESTARTactivateCnf +ENDactivateCnf + + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINwillRun +CODESTARTwillRun +ENDwillRun + + +BEGINafterRun +CODESTARTafterRun + iRet = klogAfterRun(runModConf); +ENDafterRun + + +BEGINmodExit +CODESTARTmodExit + if(pInputName != NULL) + prop.Destruct(&pInputName); + if(pLocalHostIP != NULL) + prop.Destruct(&pLocalHostIP); + + /* release objects we used */ + objRelease(glbl, CORE_COMPONENT); + objRelease(net, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_IMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES +ENDqueryEtryPt + +static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + cs.iFacilIntMsg = klogFacilIntMsg(); + return RS_RET_OK; +} + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(net, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.CreateStringProp(&pInputName, UCHAR_CONSTANT("imkmsg"), sizeof("imkmsg") - 1)); + CHKiRet(prop.CreateStringProp(&pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + + /* init legacy config settings */ + initConfigSettings(); + + CHKiRet(omsdRegCFSLineHdlr((uchar *)"debugprintkernelsymbols", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbollookup", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbolstwice", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogusesyscallinterface", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, + resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); +ENDmodInit +/* vim:set ai: + */ diff --git a/plugins/imkmsg/imkmsg.h b/plugins/imkmsg/imkmsg.h new file mode 100644 index 00000000..220a1634 --- /dev/null +++ b/plugins/imkmsg/imkmsg.h @@ -0,0 +1,64 @@ +/* imkmsg.h + * These are the definitions for the kmsg message generation module. + * + * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef IMKLOG_H_INCLUDED +#define IMKLOG_H_INCLUDED 1 + +#include "rsyslog.h" +#include "dirty.h" + +/* we need to have the modConf type present in all submodules */ +struct modConfData_s { + rsconf_t *pConf; + int iFacilIntMsg; + uchar *pszPath; + int console_log_level; + sbool bPermitNonKernel; + sbool configSetViaV2Method; +}; + +/* interface to "drivers" + * the platform specific drivers must implement these entry points. Only one + * driver may be active at any given time, thus we simply rely on the linker + * to resolve the addresses. + * rgerhards, 2008-04-09 + */ +rsRetVal klogLogKMsg(modConfData_t *pModConf); +rsRetVal klogWillRun(modConfData_t *pModConf); +rsRetVal klogAfterRun(modConfData_t *pModConf); +int klogFacilIntMsg(); + +/* the functions below may be called by the drivers */ +rsRetVal imkmsgLogIntMsg(int priority, char *fmt, ...) __attribute__((format(printf,2, 3))); +rsRetVal Syslog(int priority, uchar *msg, struct timeval *tp, struct json_object *json); + +/* prototypes */ +extern int klog_getMaxLine(void); /* work-around for klog drivers to get configured max line size */ +extern int InitKsyms(modConfData_t*); +extern void DeinitKsyms(void); +extern int InitMsyms(void); +extern void DeinitMsyms(void); +extern char * ExpandKadds(char *, char *); +extern void SetParanoiaLevel(int); + +#endif /* #ifndef IMKLOG_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/plugins/imkmsg/kmsg.c b/plugins/imkmsg/kmsg.c new file mode 100644 index 00000000..b771d68a --- /dev/null +++ b/plugins/imkmsg/kmsg.c @@ -0,0 +1,241 @@ +/* imkmsg driver for Linux /dev/kmsg structured logging + * + * This contains Linux-specific functionality to read /dev/kmsg + * For a general overview, see head comment in imkmsg.c. + * This is heavily based on imklog bsd.c file. + * + * Copyright 2008-2012 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#include <stdlib.h> +#include <time.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <string.h> +#include <ctype.h> +#ifdef OS_LINUX +#include <sys/klog.h> +#endif +#include <json/json.h> + +#include "rsyslog.h" +#include "srUtils.h" +#include "debug.h" +#include "imkmsg.h" + +/* globals */ +static int fklog = -1; /* kernel log fd */ + +#ifndef _PATH_KLOG +# define _PATH_KLOG "/dev/kmsg" +#endif + +/* submit a message to imkmsg Syslog() API. In this function, we parse + * necessary information from kernel log line, and make json string + * from the rest. + */ +static void +submitSyslog(uchar *buf) +{ + long offs = 0; + struct timeval tv; + long int timestamp = 0; + struct timespec monotonic; + struct timespec realtime; + char name[1024]; + char value[1024]; + char msg[1024]; + int priority = 0; + long int sequnum = 0; + struct json_object *json = NULL, *jval; + + /* create new json object */ + json = json_object_new_object(); + + /* get priority */ + for (; isdigit(*buf); buf++) { + priority += (priority * 10) + (*buf - '0'); + } + buf++; + + /* get messages sequence number and add it to json */ + for (; isdigit(*buf); buf++) { + sequnum = (sequnum * 10) + (*buf - '0'); + } + buf++; /* skip , */ + jval = json_object_new_int(sequnum); + json_object_object_add(json, "sequnum", jval); + + /* get timestamp */ + for (; isdigit(*buf); buf++) { + timestamp += (timestamp * 10) + (*buf - '0'); + } + buf++; /* skip ; */ + + /* get message */ + offs = 0; + for (; *buf != '\n' && *buf != '\0'; buf++, offs++) { + msg[offs] = *buf; + } + msg[offs] = '\0'; + jval = json_object_new_string((char*)msg); + json_object_object_add(json, "msg", jval); + + if (*buf != '\0') /* message has appended properties, skip \n */ + buf++; + + while (strlen((char *)buf)) { + /* get name of the property */ + buf++; /* skip ' ' */ + offs = 0; + for (; *buf != '=' && *buf != ' '; buf++, offs++) { + name[offs] = *buf; + } + name[offs] = '\0'; + buf++; /* skip = or ' ' */; + + offs = 0; + for (; *buf != '\n' && *buf != '\0'; buf++, offs++) { + value[offs] = *buf; + } + value[offs] = '\0'; + if (*buf != '\0') { + buf++; /* another property, skip \n */ + } + + jval = json_object_new_string((char*)value); + json_object_object_add(json, name, jval); + } + + /* calculate timestamp */ + clock_gettime(CLOCK_MONOTONIC, &monotonic); + clock_gettime(CLOCK_REALTIME, &realtime); + tv.tv_sec = realtime.tv_sec + ((timestamp / 1000000l) - monotonic.tv_sec); + tv.tv_usec = (realtime.tv_nsec + ((timestamp / 1000000000l) - monotonic.tv_nsec)) / 1000; + + Syslog(priority, (uchar *)msg, &tv, json); +} + + +/* open the kernel log - will be called inside the willRun() imkmsg entry point + */ +rsRetVal +klogWillRun(modConfData_t *pModConf) +{ + char errmsg[2048]; + int r; + DEFiRet; + + fklog = open(_PATH_KLOG, O_RDONLY, 0); + if (fklog < 0) { + imkmsgLogIntMsg(RS_RET_ERR_OPEN_KLOG, "imkmsg: cannot open kernel log(%s): %s.", + _PATH_KLOG, rs_strerror_r(errno, errmsg, sizeof(errmsg))); + ABORT_FINALIZE(RS_RET_ERR_OPEN_KLOG); + } + + /* Set level of kernel console messaging.. */ + if(pModConf->console_log_level != -1) { + r = klogctl(8, NULL, pModConf->console_log_level); + if(r != 0) { + imkmsgLogIntMsg(LOG_WARNING, "imkmsg: cannot set console log level: %s", + rs_strerror_r(errno, errmsg, sizeof(errmsg))); + /* make sure we do not try to re-set! */ + pModConf->console_log_level = -1; + } + } + +finalize_it: + RETiRet; +} + +/* Read kernel log while data are available, each read() reads one + * record of printk buffer. + */ +static void +readkmsg(void) +{ + int i; + uchar pRcv[8096+1]; + char errmsg[2048]; + + for (;;) { + dbgprintf("imkmsg waiting for kernel log line\n"); + + /* every read() from the opened device node receives one record of the printk buffer */ + i = read(fklog, pRcv, 8096); + + if (i > 0) { + /* successful read of message of nonzero length */ + pRcv[i] = '\0'; + } else { + /* something went wrong - error or zero length message */ + if (i < 0 && errno != EINTR && errno != EAGAIN) { + /* error occured */ + imkmsgLogIntMsg(LOG_ERR, + "imkmsg: error reading kernel log - shutting down: %s", + rs_strerror_r(errno, errmsg, sizeof(errmsg))); + fklog = -1; + } + break; + } + + submitSyslog(pRcv); + } +} + + +/* to be called in the module's AfterRun entry point + * rgerhards, 2008-04-09 + */ +rsRetVal klogAfterRun(modConfData_t *pModConf) +{ + DEFiRet; + if(fklog != -1) + close(fklog); + /* Turn on logging of messages to console, but only if a log level was speficied */ + if(pModConf->console_log_level != -1) + klogctl(7, NULL, 0); + RETiRet; +} + + +/* to be called in the module's WillRun entry point, this is the main + * "message pull" mechanism. + * rgerhards, 2008-04-09 + */ +rsRetVal klogLogKMsg(modConfData_t __attribute__((unused)) *pModConf) +{ + DEFiRet; + readkmsg(); + RETiRet; +} + + +/* provide the (system-specific) default facility for internal messages + * rgerhards, 2008-04-14 + */ +int +klogFacilIntMsg(void) +{ + return LOG_SYSLOG; +} + diff --git a/plugins/impstats/impstats.c b/plugins/impstats/impstats.c index 62599969..cdd205fd 100644 --- a/plugins/impstats/impstats.c +++ b/plugins/impstats/impstats.c @@ -27,6 +27,8 @@ #include <signal.h> #include <string.h> #include <pthread.h> +#include <fcntl.h> +#include <sys/uio.h> #include "dirty.h" #include "cfsysline.h" #include "module-template.h" @@ -63,11 +65,14 @@ typedef struct configSettings_s { } configSettings_t; struct modConfData_s { - rsconf_t *pConf; /* our overall config object */ + rsconf_t *pConf; /* our overall config object */ int iStatsInterval; int iFacility; int iSeverity; + int logfd; /* fd if logging to file, or -1 if closed */ statsFmtType_t statsFmt; + sbool bLogToSyslog; + char *logfile; sbool configSetViaV2Method; }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ @@ -82,6 +87,8 @@ static struct cnfparamdescr modpdescr[] = { { "interval", eCmdHdlrInt, 0 }, { "facility", eCmdHdlrInt, 0 }, { "severity", eCmdHdlrInt, 0 }, + { "log.syslog", eCmdHdlrBinary, 0 }, + { "log.file", eCmdHdlrGetWord, 0 }, { "format", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk modpblk = @@ -120,7 +127,7 @@ initConfigSettings(void) /* actually submit a message to the rsyslog core */ -static inline rsRetVal +static inline void doSubmitMsg(uchar *line) { msg_t *pMsg; @@ -138,11 +145,55 @@ doSubmitMsg(uchar *line) pMsg->iSeverity = runModConf->iSeverity; pMsg->msgFlags = 0; - submitMsg(pMsg); + /* we do not use rate-limiting, as the stats message always need to be emitted */ + submitMsg2(pMsg); + DBGPRINTF("impstats: submit [%d,%d] msg '%s'\n", runModConf->iFacility, + runModConf->iSeverity, line); finalize_it: - RETiRet; + return; +} + + +/* log stats message to file; limited error handling done */ +static inline void +doLogToFile(cstr_t *cstr) +{ + struct iovec iov[4]; + ssize_t nwritten; + ssize_t nexpect; + time_t t; + char timebuf[32]; + + if(cstrLen(cstr) == 0) + goto done; + if(runModConf->logfd == -1) { + runModConf->logfd = open(runModConf->logfile, O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, S_IRUSR|S_IWUSR); + if(runModConf->logfd == -1) { + dbgprintf("error opening stats file %s\n", runModConf->logfile); + goto done; + } + } + time(&t); + iov[0].iov_base = ctime_r(&t, timebuf); + iov[0].iov_len = nexpect = strlen(iov[0].iov_base) - 1; /* -1: strip \n */ + iov[1].iov_base = ": "; + iov[1].iov_len = 2; + nexpect += 2; + iov[2].iov_base = rsCStrGetSzStrNoNULL(cstr); + iov[2].iov_len = (size_t) cstrLen(cstr); + nexpect += cstrLen(cstr); + iov[3].iov_base = "\n"; + iov[3].iov_len = 1; + nexpect++; + nwritten = writev(runModConf->logfd, iov, 4); + + if(nwritten != nexpect) { + dbgprintf("error writing stats file %s, nwritten %lld, expected %lld\n", + runModConf->logfile, (long long) nwritten, (long long) nexpect); + } +done: return; } @@ -153,7 +204,10 @@ static rsRetVal doStatsLine(void __attribute__((unused)) *usrptr, cstr_t *cstr) { DEFiRet; - doSubmitMsg(rsCStrGetSzStrNoNULL(cstr)); + if(runModConf->bLogToSyslog) + doSubmitMsg(rsCStrGetSzStrNoNULL(cstr)); + if(runModConf->logfile != NULL) + doLogToFile(cstr); RETiRet; } @@ -178,6 +232,9 @@ CODESTARTbeginCnfLoad loadModConf->iFacility = DEFAULT_FACILITY; loadModConf->iSeverity = DEFAULT_SEVERITY; loadModConf->statsFmt = statsFmt_Legacy; + loadModConf->logfd = -1; + loadModConf->logfile = NULL; + loadModConf->bLogToSyslog = 1; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config vars */ initConfigSettings(); @@ -210,6 +267,10 @@ CODESTARTsetModCnf loadModConf->iFacility = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "severity")) { loadModConf->iSeverity = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "log.syslog")) { + loadModConf->bLogToSyslog = (sbool) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "log.file")) { + loadModConf->logfile = es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "format")) { mode = es_str2cstr(pvals[i].val.d.estr, NULL); if(!strcasecmp(mode, "json")) { @@ -270,7 +331,9 @@ BEGINactivateCnf rsRetVal localRet; CODESTARTactivateCnf runModConf = pModConf; - DBGPRINTF("impstats: stats interval %d seconds\n", runModConf->iStatsInterval); + DBGPRINTF("impstats: stats interval %d seconds, logToSyslog %d, logFile %s\n", + runModConf->iStatsInterval, runModConf->bLogToSyslog, + runModConf->logfile == NULL ? "deactivated" : (char*)runModConf->logfile); localRet = statsobj.EnableStats(); if(localRet != RS_RET_OK) { errmsg.LogError(0, localRet, "impstats: error enabling statistics gathering"); @@ -282,6 +345,9 @@ ENDactivateCnf BEGINfreeCnf CODESTARTfreeCnf + if(runModConf->logfd != -1) + close(runModConf->logfd); + free(runModConf->logfile); ENDfreeCnf @@ -297,6 +363,7 @@ CODESTARTrunInput if(glbl.GetGlobalInputTermState() == 1) break; /* terminate input! */ + DBGPRINTF("impstats: woke up, generating messages\n"); generateStatsMsgs(); } ENDrunInput diff --git a/plugins/imptcp/imptcp.c b/plugins/imptcp/imptcp.c index a13fd990..5c8bb67a 100644 --- a/plugins/imptcp/imptcp.c +++ b/plugins/imptcp/imptcp.c @@ -67,6 +67,7 @@ #include "ruleset.h" #include "msg.h" #include "statsobj.h" +#include "ratelimit.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ /* the define is from tcpsrv.h, we need to find a new (but easier!!!) abstraction layer some time ... */ @@ -121,6 +122,8 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + int ratelimitInterval; + int ratelimitBurst; struct instanceConf_s *next; }; @@ -158,6 +161,8 @@ static struct cnfparamdescr inppdescr[] = { { "keepalive.time", eCmdHdlrInt, 0 }, { "keepalive.interval", eCmdHdlrInt, 0 }, { "addtlframedelimiter", eCmdHdlrInt, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -195,6 +200,7 @@ struct ptcpsrv_s { sbool bKeepAlive; /* support keep-alive packets */ sbool bEmitMsgOnClose; sbool bSuppOctetFram; + ratelimit_t *ratelimiter; }; /* the ptcp session object. Describes a single active session. @@ -295,6 +301,7 @@ destructSess(ptcpsess_t *pSess) static void destructSrv(ptcpsrv_t *pSrv) { + ratelimitDestruct(pSrv->ratelimiter); prop.Destruct(&pSrv->pInputName); pthread_mutex_destroy(&pSrv->mutSessLst); free(pSrv->pszInputName); @@ -414,7 +421,9 @@ startupSrv(ptcpsrv_t *pSrv) #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - DBGPRINTF("error %d while binding tcp socket\n", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; @@ -677,14 +686,7 @@ doSubmitMsg(ptcpsess_t *pThis, struct syslogTime *stTime, time_t ttGenTime, mult MsgSetRuleset(pMsg, pSrv->pRuleset); STATSCOUNTER_INC(pThis->pLstn->ctrSubmit, pThis->pLstn->mutCtrSubmit); - if(pMultiSub == NULL) { - CHKiRet(submitMsg(pMsg)); - } else { - pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; - if(pMultiSub->nElem == pMultiSub->maxElem) - CHKiRet(multiSubmitMsg(pMultiSub)); - } - + ratelimitAddMsg(pSrv->ratelimiter, pMultiSub, pMsg); finalize_it: /* reset status variables */ @@ -803,12 +805,11 @@ processDataRcvd(ptcpsess_t *pThis, char c, struct syslogTime *stTime, time_t ttG * we have just received a bunch of data! -- rgerhards, 2009-06-16 * EXTRACT from tcps_sess.c */ -#define NUM_MULTISUB 1024 static rsRetVal DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) { multi_submit_t multiSub; - msg_t *pMsgs[NUM_MULTISUB]; + msg_t *pMsgs[CONF_NUM_MULTISUB]; struct syslogTime stTime; time_t ttGenTime; char *pEnd; @@ -819,7 +820,7 @@ DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) datetime.getCurrTime(&stTime, &ttGenTime); multiSub.ppMsgs = pMsgs; - multiSub.maxElem = NUM_MULTISUB; + multiSub.maxElem = CONF_NUM_MULTISUB; multiSub.nElem = 0; /* We now copy the message to the session buffer. */ @@ -829,15 +830,11 @@ DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); } - if(multiSub.nElem > 0) { - /* submit anything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&multiSub)); - } + iRet = multiSubmitFlush(&multiSub); finalize_it: RETiRet; } -#undef NUM_MULTISUB /****************************************** --END-- TCP SUPPORT FUNCTIONS ***********************************/ @@ -1049,6 +1046,8 @@ createInstance(instanceConf_t **pinst) inst->bEmitMsgOnClose = 0; inst->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; inst->pBindRuleset = NULL; + inst->ratelimitBurst = 10000; /* arbitrary high limit */ + inst->ratelimitInterval = 0; /* off */ /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -1128,6 +1127,9 @@ addListner(modConfData_t __attribute__((unused)) *modConf, instanceConf_t *inst) pSrv->iKeepAliveProbes = inst->iKeepAliveProbes; pSrv->iKeepAliveTime = inst->iKeepAliveTime; pSrv->bEmitMsgOnClose = inst->bEmitMsgOnClose; + CHKiRet(ratelimitNew(&pSrv->ratelimiter, "imtcp", (char*)inst->pszBindPort)); + ratelimitSetLinuxLike(pSrv->ratelimiter, inst->ratelimitInterval, inst->ratelimitBurst); + ratelimitSetThreadSafe(pSrv->ratelimiter); CHKmalloc(pSrv->port = ustrdup(inst->pszBindPort)); pSrv->iAddtlFrameDelim = inst->iAddtlFrameDelim; if(inst->pszBindAddr == NULL) @@ -1442,7 +1444,7 @@ CODESTARTnewInpInst inst->pszInputName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(inppblk.descr[i].name, "supportOctetCountedFraming")) { + } else if(!strcmp(inppblk.descr[i].name, "supportoctetcountedframing")) { inst->bSuppOctetFram = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "keepalive")) { inst->bKeepAlive = (int) pvals[i].val.d.n; @@ -1456,6 +1458,10 @@ CODESTARTnewInpInst inst->iAddtlFrameDelim = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "notifyonconnectionclose")) { inst->bEmitMsgOnClose = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; } else { dbgprintf("imptcp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); diff --git a/plugins/imrelp/imrelp.c b/plugins/imrelp/imrelp.c index fe987a50..dc67f4fe 100644 --- a/plugins/imrelp/imrelp.c +++ b/plugins/imrelp/imrelp.c @@ -113,11 +113,29 @@ static struct cnfparamblk inppblk = * we will only see the hostname (twice). -- rgerhards, 2009-10-14 */ static relpRetVal -onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *pMsg, size_t lenMsg) +onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *msg, size_t lenMsg) { + prop_t *pProp = NULL; + msg_t *pMsg; DEFiRet; - parseAndSubmitMessage(pHostname, pIP, pMsg, lenMsg, PARSE_HOSTNAME, - eFLOWCTL_LIGHT_DELAY, pInputName, NULL, 0, runModConf->pBindRuleset); + + CHKiRet(msgConstruct(&pMsg)); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsg(pMsg, (char*)msg, lenMsg); + MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + MsgSetRuleset(pMsg, runModConf->pBindRuleset); + pMsg->msgFlags = PARSE_HOSTNAME | NEEDS_PARSING; + + /* TODO: optimize this, we can store it inside the session, requires + * changes to librelp --> next librelp iteration?. rgerhards, 2012-10-29 + */ + MsgSetRcvFromStr(pMsg, pHostname, ustrlen(pHostname), &pProp); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(MsgSetRcvFromIPStr(pMsg, pIP, ustrlen(pIP), &pProp)); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(submitMsg2(pMsg)); + +finalize_it: RETiRet; } @@ -303,7 +321,14 @@ ENDactivateCnf BEGINfreeCnf + instanceConf_t *inst, *del; CODESTARTfreeCnf + for(inst = pModConf->root ; inst != NULL ; ) { + free(inst->pszBindPort); + del = inst; + inst = inst->next; + free(del); + } ENDfreeCnf /* This is used to terminate the plugin. Note that the signal handler blocks diff --git a/plugins/imtcp/imtcp.c b/plugins/imtcp/imtcp.c index 3ad03615..fc22d452 100644 --- a/plugins/imtcp/imtcp.c +++ b/plugins/imtcp/imtcp.c @@ -36,7 +36,6 @@ * * rgerhards, 2008-05-19 */ - #include "config.h" #include <stdlib.h> #include <assert.h> @@ -62,6 +61,7 @@ #include "errmsg.h" #include "tcpsrv.h" #include "ruleset.h" +#include "rainerscript.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ MODULE_TYPE_INPUT @@ -105,6 +105,8 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ + int ratelimitInterval; + int ratelimitBurst; int bSuppOctetFram; struct instanceConf_s *next; }; @@ -123,6 +125,7 @@ struct modConfData_s { sbool bKeepAlive; sbool bEmitMsgOnClose; /* emit an informational message on close by remote peer */ uchar *pszStrmDrvrAuthMode; /* authentication mode to use */ + struct cnfarray *permittedPeers; sbool configSetViaV2Method; }; @@ -140,6 +143,7 @@ static struct cnfparamdescr modpdescr[] = { { "maxlistners", eCmdHdlrPositiveInt, 0 }, { "streamdriver.mode", eCmdHdlrPositiveInt, 0 }, { "streamdriver.authmode", eCmdHdlrString, 0 }, + { "permittedpeer", eCmdHdlrArray, 0 }, { "keepalive", eCmdHdlrBinary, 0 } }; static struct cnfparamblk modpblk = @@ -153,7 +157,9 @@ static struct cnfparamdescr inppdescr[] = { { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ { "name", eCmdHdlrString, 0 }, { "ruleset", eCmdHdlrString, 0 }, - { "supportOctetCountedFraming", eCmdHdlrBinary, 0 } + { "supportOctetCountedFraming", eCmdHdlrBinary, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -249,6 +255,8 @@ createInstance(instanceConf_t **pinst) inst->pszBindRuleset = NULL; inst->pszInputName = NULL; inst->bSuppOctetFram = 1; + inst->ratelimitInterval = 0; + inst->ratelimitBurst = 10000; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -332,6 +340,7 @@ addListner(modConfData_t *modConf, instanceConf_t *inst) CHKiRet(tcpsrv.SetRuleset(pOurTcpsrv, inst->pBindRuleset)); CHKiRet(tcpsrv.SetInputName(pOurTcpsrv, inst->pszInputName == NULL ? UCHAR_CONSTANT("imtcp") : inst->pszInputName)); + CHKiRet(tcpsrv.SetLinuxLikeRatelimiters(pOurTcpsrv, inst->ratelimitInterval, inst->ratelimitBurst)); tcpsrv.configureTCPListen(pOurTcpsrv, inst->pszBindPort, inst->bSuppOctetFram); finalize_it: @@ -374,6 +383,10 @@ CODESTARTnewInpInst inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "supportOctetCountedFraming")) { inst->bSuppOctetFram = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; } else { dbgprintf("imtcp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); @@ -400,6 +413,7 @@ CODESTARTbeginCnfLoad loadModConf->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; loadModConf->bDisableLFDelim = 0; loadModConf->pszStrmDrvrAuthMode = NULL; + loadModConf->permittedPeers = NULL; loadModConf->configSetViaV2Method = 0; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config variables */ @@ -445,8 +459,10 @@ CODESTARTsetModCnf loadModConf->bKeepAlive = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "streamdriver.mode")) { loadModConf->iStrmDrvrMode = (int) pvals[i].val.d.n; - } else if(!strcmp(modpblk.descr[i].name, "streamdriver.mode")) { + } else if(!strcmp(modpblk.descr[i].name, "streamdriver.authmode")) { loadModConf->pszStrmDrvrAuthMode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(modpblk.descr[i].name, "permittedpeer")) { + loadModConf->permittedPeers = cnfarrayDup(pvals[i].val.d.ar); } else { dbgprintf("imtcp: program error, non-handled " "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); @@ -482,10 +498,10 @@ CODESTARTendCnfLoad loadModConf->pszStrmDrvrAuthMode = NULL; } else { loadModConf->pszStrmDrvrAuthMode = cs.pszStrmDrvrAuthMode; + cs.pszStrmDrvrAuthMode = NULL; } } - if((cs.pszStrmDrvrAuthMode == NULL) || (cs.pszStrmDrvrAuthMode[0] == '\0')) - free(cs.pszStrmDrvrAuthMode); + free(cs.pszStrmDrvrAuthMode); cs.pszStrmDrvrAuthMode = NULL; loadModConf = NULL; /* done loading */ @@ -517,8 +533,15 @@ ENDcheckCnf BEGINactivateCnfPrePrivDrop instanceConf_t *inst; + int i; CODESTARTactivateCnfPrePrivDrop runModConf = pModConf; + if(runModConf->permittedPeers != NULL) { + for(i = 0 ; i < runModConf->permittedPeers->nmemb ; ++i) { + setPermittedPeer(NULL, (uchar*) + es_str2cstr(runModConf->permittedPeers->arr[i], NULL)); + } + } for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { addListner(pModConf, inst); } @@ -538,6 +561,11 @@ ENDactivateCnf BEGINfreeCnf instanceConf_t *inst, *del; CODESTARTfreeCnf + free(pModConf->pszStrmDrvrAuthMode); + if(pModConf->permittedPeers != NULL) { + cnfarrayContentDestruct(pModConf->permittedPeers); + free(pModConf->permittedPeers); + } for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindPort); free(inst->pszInputName); @@ -564,7 +592,9 @@ ENDwillRun BEGINafterRun CODESTARTafterRun - /* do cleanup here */ + if(pOurTcpsrv != NULL) + iRet = tcpsrv.Destruct(&pOurTcpsrv); + net.clearAllowedSenders(UCHAR_CONSTANT("TCP")); ENDafterRun @@ -578,9 +608,6 @@ ENDisCompatibleWithFeature BEGINmodExit CODESTARTmodExit - if(pOurTcpsrv != NULL) - iRet = tcpsrv.Destruct(&pOurTcpsrv); - if(pPermPeersRoot != NULL) { net.DestructPermittedPeers(&pPermPeersRoot); } @@ -643,8 +670,6 @@ CODEmodInit_QueryRegCFSLineHdlr /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverrun"), 0, eCmdHdlrGetWord, addInstance, NULL, STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverstreamdriverpermittedpeer"), 0, eCmdHdlrGetWord, - setPermittedPeer, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverinputname"), 0, eCmdHdlrGetWord, NULL, &cs.pszInputName, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverbindruleset"), 0, eCmdHdlrGetWord, @@ -652,6 +677,8 @@ CODEmodInit_QueryRegCFSLineHdlr /* module-global config params - will be disabled in configs that are loaded * via module(...). */ + CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverstreamdriverpermittedpeer"), 0, eCmdHdlrGetWord, + setPermittedPeer, NULL, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverstreamdriverauthmode"), 0, eCmdHdlrGetWord, NULL, &cs.pszStrmDrvrAuthMode, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverkeepalive"), 0, eCmdHdlrBinary, diff --git a/plugins/imttcp/imttcp.c b/plugins/imttcp/imttcp.c index c72886b3..9bd11f77 100644 --- a/plugins/imttcp/imttcp.c +++ b/plugins/imttcp/imttcp.c @@ -365,7 +365,9 @@ createSrv(ttcpsrv_t *pSrv) #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - DBGPRINTF("error %d while binding tcp socket", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index ea0a8282..dde8f105 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -4,8 +4,6 @@ * NOTE: read comments in module-template.h to understand how this file * works! * - * File begun on 2007-12-21 by RGerhards (extracted from syslogd.c) - * * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. @@ -53,6 +51,7 @@ #include "prop.h" #include "ruleset.h" #include "statsobj.h" +#include "ratelimit.h" #include "unicode-helper.h" MODULE_TYPE_INPUT @@ -77,6 +76,7 @@ static struct lstn_s { int sock; /* socket */ ruleset_t *pRuleset; /* bound ruleset */ statsobj_t *stats; /* listener stats */ + ratelimit_t *ratelimiter; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) } *lcnfRoot = NULL, *lcnfLast = NULL; @@ -109,6 +109,8 @@ struct instanceConf_s { uchar *pszBindPort; /* Port to bind socket to */ uchar *pszBindRuleset; /* name of ruleset to bind to */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + int ratelimitInterval; + int ratelimitBurst; struct instanceConf_s *next; }; @@ -138,9 +140,11 @@ static struct cnfparamblk modpblk = /* input instance parameters */ static struct cnfparamdescr inppdescr[] = { - { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ + { "port", eCmdHdlrArray, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ { "address", eCmdHdlrString, 0 }, - { "ruleset", eCmdHdlrString, 0 } + { "ruleset", eCmdHdlrString, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -165,6 +169,8 @@ createInstance(instanceConf_t **pinst) inst->pszBindPort = NULL; inst->pszBindAddr = NULL; inst->pszBindRuleset = NULL; + inst->ratelimitBurst = 10000; /* arbitrary high limit */ + inst->ratelimitInterval = 0; /* off */ /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -223,7 +229,7 @@ addListner(instanceConf_t *inst) struct lstn_s *newlcnfinfo; uchar *bindName; uchar *port; - uchar statname[64]; + uchar dispname[64]; /* check which address to bind to. We could do this more compact, but have not * done so in order to make the code more readable. -- rgerhards, 2007-12-27 @@ -248,11 +254,14 @@ addListner(instanceConf_t *inst) newlcnfinfo->next = NULL; newlcnfinfo->sock = newSocks[iSrc]; newlcnfinfo->pRuleset = inst->pBindRuleset; + snprintf((char*)dispname, sizeof(dispname), "imudp(%s:%s)", bindName, port); + dispname[sizeof(dispname)-1] = '\0'; /* just to be on the save side... */ + CHKiRet(ratelimitNew(&newlcnfinfo->ratelimiter, (char*)dispname, NULL)); + ratelimitSetLinuxLike(newlcnfinfo->ratelimiter, inst->ratelimitInterval, + inst->ratelimitBurst); /* support statistics gathering */ CHKiRet(statsobj.Construct(&(newlcnfinfo->stats))); - snprintf((char*)statname, sizeof(statname), "imudp(%s:%s)", bindName, port); - statname[sizeof(statname)-1] = '\0'; /* just to be on the save side... */ - CHKiRet(statsobj.SetName(newlcnfinfo->stats, statname)); + CHKiRet(statsobj.SetName(newlcnfinfo->stats, dispname)); STATSCOUNTER_INIT(newlcnfinfo->ctrSubmit, newlcnfinfo->mutCtrSubmit); CHKiRet(statsobj.AddCounter(newlcnfinfo->stats, UCHAR_CONSTANT("submitted"), ctrType_IntCtr, &(newlcnfinfo->ctrSubmit))); @@ -304,7 +313,6 @@ std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, insta static inline rsRetVal processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted) { - DEFiRet; int iNbrTimeUsed; time_t ttGenTime; struct syslogTime stTime; @@ -314,9 +322,15 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f msg_t *pMsg; prop_t *propFromHost = NULL; prop_t *propFromHostIP = NULL; + multi_submit_t multiSub; + msg_t *pMsgs[CONF_NUM_MULTISUB]; char errStr[1024]; + DEFiRet; assert(pThrd != NULL); + multiSub.ppMsgs = pMsgs; + multiSub.maxElem = CONF_NUM_MULTISUB; + multiSub.nElem = 0; iNbrTimeUsed = 0; while(1) { /* loop is terminated if we have a bad receive, done below in the body */ if(pThrd->bShallStop == RSTRUE) @@ -367,7 +381,7 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f *pbIsPermitted = 1; /* no check -> everything permitted */ } - DBGPRINTF("recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, pRcvBuf); + DBGPRINTF("imudp:recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, pRcvBuf); if(*pbIsPermitted != 0) { if((runModConf->iTimeRequery == 0) || (iNbrTimeUsed++ % runModConf->iTimeRequery) == 0) { @@ -383,12 +397,15 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f if(*pbIsPermitted == 2) pMsg->msgFlags |= NEEDS_ACLCHK_U; /* request ACL check after resolution */ CHKiRet(msgSetFromSockinfo(pMsg, &frominet)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(ratelimitAddMsg(lstn->ratelimiter, &multiSub, pMsg)); STATSCOUNTER_INC(lstn->ctrSubmit, lstn->mutCtrSubmit); } } + finalize_it: + multiSubmitFlush(&multiSub); + if(propFromHost != NULL) prop.Destruct(&propFromHost); if(propFromHostIP != NULL) @@ -664,10 +681,42 @@ rsRetVal rcvMainLoop(thrdInfo_t *pThrd) #endif /* #if HAVE_EPOLL_CREATE1 */ +static inline rsRetVal +createListner(es_str_t *port, struct cnfparamvals *pvals) +{ + instanceConf_t *inst; + int i; + DEFiRet; + + CHKiRet(createInstance(&inst)); + inst->pszBindPort = (uchar*)es_str2cstr(port, NULL); + for(i = 0 ; i < inppblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(inppblk.descr[i].name, "port")) { + continue; /* array, handled by caller */ + } else if(!strcmp(inppblk.descr[i].name, "address")) { + inst->pszBindAddr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { + inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; + } else { + dbgprintf("imudp: program error, non-handled " + "param '%s'\n", inppblk.descr[i].name); + } + } +finalize_it: + RETiRet; +} + + BEGINnewInpInst struct cnfparamvals *pvals; - instanceConf_t *inst; int i; + int portIdx; CODESTARTnewInpInst DBGPRINTF("newInpInst (imudp)\n"); @@ -677,28 +726,17 @@ CODESTARTnewInpInst "imudp: required parameter are missing\n"); ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); } - if(Debug) { dbgprintf("input param blk in imudp:\n"); cnfparamsPrint(&inppblk, pvals); } - CHKiRet(createInstance(&inst)); - - for(i = 0 ; i < inppblk.nParams ; ++i) { - if(!pvals[i].bUsed) - continue; - if(!strcmp(inppblk.descr[i].name, "port")) { - inst->pszBindPort = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(inppblk.descr[i].name, "address")) { - inst->pszBindAddr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { - inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else { - dbgprintf("imudp: program error, non-handled " - "param '%s'\n", inppblk.descr[i].name); - } + portIdx = cnfparamGetIdx(&inppblk, "port"); + assert(portIdx != -1); + for(i = 0 ; i < pvals[portIdx].val.d.ar->nmemb ; ++i) { + createListner(pvals[portIdx].val.d.ar->arr[i], pvals); } + finalize_it: CODE_STD_FINALIZERnewInpInst cnfparamvalsDestruct(pvals, &inppblk); @@ -867,6 +905,7 @@ CODESTARTafterRun net.clearAllowedSenders((uchar*)"UDP"); for(lstn = lcnfRoot ; lstn != NULL ; ) { statsobj.Destruct(&(lstn->stats)); + ratelimitDestruct(lstn->ratelimiter); close(lstn->sock); lstnDel = lstn; lstn = lstn->next; diff --git a/plugins/imuxsock/Makefile.am b/plugins/imuxsock/Makefile.am index 34a0ad9a..28f9f9e3 100644 --- a/plugins/imuxsock/Makefile.am +++ b/plugins/imuxsock/Makefile.am @@ -1,6 +1,6 @@ pkglib_LTLIBRARIES = imuxsock.la imuxsock_la_SOURCES = imuxsock.c -imuxsock_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -I../../runtime/hashtable -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +imuxsock_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) imuxsock_la_LDFLAGS = -module -avoid-version imuxsock_la_LIBADD = $(RSRT_LIBS) diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index 79c6b734..a5360bee 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -6,7 +6,7 @@ * * File begun on 2007-12-20 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -55,6 +55,7 @@ #include "statsobj.h" #include "datetime.h" #include "hashtable.h" +#include "ratelimit.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -105,15 +106,6 @@ STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) STATSCOUNTER_DEF(ctrLostRatelimit, mutCtrLostRatelimit) STATSCOUNTER_DEF(ctrNumRatelimiters, mutCtrNumRatelimiters) -struct rs_ratelimit_state { - unsigned short interval; - unsigned short burst; - unsigned done; - unsigned missed; - time_t begin; -}; -typedef struct rs_ratelimit_state rs_ratelimit_state_t; - /* a very simple "hash function" for process IDs - we simply use the * pid itself: it is quite expected that all pids may log some time, but @@ -143,6 +135,7 @@ typedef struct lstn_s { int flowCtl; /* flow control settings for this socket */ int ratelimitInterval; int ratelimitBurst; + ratelimit_t *dflt_ratelimiter;/*ratelimiter to apply if none else is to be used */ intTiny ratelimitSev; /* severity level (and below) for which rate-limiting shall apply */ struct hashtable *ht; /* our hashtable for rate-limiting */ sbool bParseHost; /* should parser parse host name? read-only after startup */ @@ -151,6 +144,7 @@ typedef struct lstn_s { sbool bAnnotate; /* annotate events with trusted properties */ sbool bParseTrusted; /* parse trusted properties */ sbool bWritePid; /* write original PID into tag */ + sbool bDiscardOwnMsgs; /* discard messages that originated from ourselves */ sbool bUseSysTimeStamp; /* use timestamp from system (instead of from message) */ } lstn_t; static lstn_t listeners[MAXFUNIX]; @@ -164,8 +158,6 @@ static int startIndexUxLocalSockets; /* process fd from that index on (used to static int nfd = 1; /* number of Unix sockets open / read-only after startup */ static int sd_fds = 0; /* number of systemd activated sockets */ -static ee_ctx ctxee = NULL; /* library context */ - /* config vars for legacy config system */ #define DFLT_bCreatePath 0 #define DFLT_ratelimitInterval 0 @@ -208,6 +200,7 @@ struct instanceConf_s { int ratelimitSeverity; int bAnnotate; /* annotate trusted properties */ int bParseTrusted; /* parse trusted properties */ + sbool bDiscardOwnMsgs; /* discard messages that originated from our own pid? */ struct instanceConf_s *next; }; @@ -225,6 +218,7 @@ struct modConfData_s { sbool bOmitLocalLogging; sbool bWritePidSysSock; sbool bUseSysTimeStamp; + sbool bDiscardOwnMsgs; sbool configSetViaV2Method; }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ @@ -235,6 +229,7 @@ static struct cnfparamdescr modpdescr[] = { { "syssock.use", eCmdHdlrBinary, 0 }, { "syssock.name", eCmdHdlrGetWord, 0 }, { "syssock.ignoretimestamp", eCmdHdlrBinary, 0 }, + { "syssock.ignoreownmessages", eCmdHdlrBinary, 0 }, { "syssock.flowcontrol", eCmdHdlrBinary, 0 }, { "syssock.usesystimestamp", eCmdHdlrBinary, 0 }, { "syssock.annotate", eCmdHdlrBinary, 0 }, @@ -254,6 +249,7 @@ static struct cnfparamdescr inppdescr[] = { { "socket", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: addunixlistensocket */ { "createpath", eCmdHdlrBinary, 0 }, { "parsetrusted", eCmdHdlrBinary, 0 }, + { "ignoreownmessages", eCmdHdlrBinary, 0 }, { "hostname", eCmdHdlrString, 0 }, { "ignoretimestamp", eCmdHdlrBinary, 0 }, { "flowcontrol", eCmdHdlrBinary, 0 }, @@ -273,74 +269,9 @@ static struct cnfparamblk inppblk = /* we do not use this, because we do not bind to a ruleset so far * enable when this is changed: #include "im-helper.h" */ /* must be included AFTER the type definitions! */ -static void -initRatelimitState(struct rs_ratelimit_state *rs, unsigned short interval, unsigned short burst) -{ - rs->interval = interval; - rs->burst = burst; - rs->done = 0; - rs->missed = 0; - rs->begin = 0; -} - static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ -/* ratelimiting support, modelled after the linux kernel - * returns 1 if message is within rate limit and shall be - * processed, 0 otherwise. - * This implementation is NOT THREAD-SAFE and must not - * be called concurrently. - */ -static inline int -withinRatelimit(struct rs_ratelimit_state *rs, time_t tt, pid_t pid) -{ - int ret; - uchar msgbuf[1024]; - - if(rs->interval == 0) { - ret = 1; - goto finalize_it; - } - - assert(rs->burst != 0); - - if(rs->begin == 0) - rs->begin = tt; - - /* resume if we go out of out time window */ - if(tt > rs->begin + rs->interval) { - if(rs->missed) { - snprintf((char*)msgbuf, sizeof(msgbuf), - "imuxsock lost %u messages from pid %lu due to rate-limiting", - rs->missed, (unsigned long) pid); - logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); - rs->missed = 0; - } - rs->begin = 0; - rs->done = 0; - } - - /* do actual limit check */ - if(rs->burst > rs->done) { - rs->done++; - ret = 1; - } else { - if(rs->missed == 0) { - snprintf((char*)msgbuf, sizeof(msgbuf), - "imuxsock begins to drop messages from pid %lu due to rate-limiting", - (unsigned long) pid); - logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); - } - rs->missed++; - ret = 0; - } - -finalize_it: - return ret; -} - - /* create input instance, set default paramters, and * add it to the list of instances. */ @@ -362,6 +293,7 @@ createInstance(instanceConf_t **pinst) inst->bWritePid = 0; inst->bAnnotate = 0; inst->bParseTrusted = 0; + inst->bDiscardOwnMsgs = 1; inst->next = NULL; /* node created, let's add to config */ @@ -447,7 +379,8 @@ addListner(instanceConf_t *inst) CHKiRet(prop.ConstructFinalize(listeners[nfd].hostName)); } if(inst->ratelimitInterval > 0) { - if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, NULL)) == NULL) { + if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))ratelimitDestruct)) == NULL) { /* in this case, we simply turn off rate-limiting */ DBGPRINTF("imuxsock: turning off rate limiting because we could not " "create hash table\n"); @@ -461,11 +394,16 @@ addListner(instanceConf_t *inst) listeners[nfd].flags = inst->bIgnoreTimestamp ? IGNDATE : NOFLAG; listeners[nfd].bCreatePath = inst->bCreatePath; listeners[nfd].sockName = ustrdup(inst->sockName); - listeners[nfd].bUseCreds = (inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate) ? 1 : 0; + listeners[nfd].bUseCreds = (inst->bDiscardOwnMsgs || inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate) ? 1 : 0; listeners[nfd].bAnnotate = inst->bAnnotate; listeners[nfd].bParseTrusted = inst->bParseTrusted; + listeners[nfd].bDiscardOwnMsgs = inst->bDiscardOwnMsgs; listeners[nfd].bWritePid = inst->bWritePid; listeners[nfd].bUseSysTimeStamp = inst->bUseSysTimeStamp; + CHKiRet(ratelimitNew(&listeners[nfd].dflt_ratelimiter, "imuxsock", NULL)); + ratelimitSetLinuxLike(listeners[nfd].dflt_ratelimiter, + listeners[nfd].ratelimitInterval, + listeners[nfd].ratelimitBurst); nfd++; } else { errmsg.LogError(0, NO_ERRCODE, "Out of unix socket name descriptors, ignoring %s\n", @@ -477,7 +415,7 @@ finalize_it: } -/* discard all log sockets except for "socket" 0. Data for it comes from +/* discard/Destruct all log sockets except for "socket" 0. Data for it comes from * the constant memory pool - and if not, it is freeed via some other pointer. */ static rsRetVal discardLogSockets(void) @@ -495,6 +433,7 @@ static rsRetVal discardLogSockets(void) if(listeners[i].ht != NULL) { hashtable_destroy(listeners[i].ht, 1); /* 1 => free all values automatically */ } + ratelimitDestruct(listeners[i].dflt_ratelimiter); } return RS_RET_OK; @@ -606,19 +545,22 @@ finalize_it: * listener (the latter being a performance enhancement). */ static inline rsRetVal -findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) +findRatelimiter(lstn_t *pLstn, struct ucred *cred, ratelimit_t **prl) { - rs_ratelimit_state_t *rl; + ratelimit_t *rl; int r; pid_t *keybuf; + char pidbuf[256]; DEFiRet; if(cred == NULL) FINALIZE; +#if 0 // TODO: check deactivated? if(pLstn->ratelimitInterval == 0) { *prl = NULL; FINALIZE; } +#endif rl = hashtable_search(pLstn->ht, &cred->pid); if(rl == NULL) { @@ -626,10 +568,13 @@ findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) DBGPRINTF("imuxsock: no ratelimiter for pid %lu, creating one\n", (unsigned long) cred->pid); STATSCOUNTER_INC(ctrNumRatelimiters, mutCtrNumRatelimiters); - CHKmalloc(rl = malloc(sizeof(rs_ratelimit_state_t))); + snprintf(pidbuf, sizeof(pidbuf), "pid %lu", + (unsigned long) cred->pid); + pidbuf[sizeof(pidbuf)-1] = '\0'; /* to be on safe side */ + CHKiRet(ratelimitNew(&rl, "imuxsock", pidbuf)); + ratelimitSetLinuxLike(rl, pLstn->ratelimitInterval, pLstn->ratelimitBurst); CHKmalloc(keybuf = malloc(sizeof(pid_t))); *keybuf = cred->pid; - initRatelimitState(rl, pLstn->ratelimitInterval, pLstn->ratelimitBurst); r = hashtable_insert(pLstn->ht, keybuf, rl); if(r == 0) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); @@ -638,6 +583,8 @@ findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) *prl = rl; finalize_it: + if(*prl == NULL) + *prl = pLstn->dflt_ratelimiter; RETiRet; } @@ -690,14 +637,12 @@ getTrustedProp(struct ucred *cred, char *propName, uchar *buf, size_t lenBuf, in if((fd = open(namebuf, O_RDONLY)) == -1) { DBGPRINTF("error reading '%s'\n", namebuf); - *lenProp = 0; - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } if((lenRead = read(fd, buf, lenBuf - 1)) == -1) { DBGPRINTF("error reading file data for '%s'\n", namebuf); - *lenProp = 0; close(fd); - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } /* we strip after the first \n */ @@ -733,8 +678,7 @@ getTrustedExe(struct ucred *cred, uchar *buf, size_t lenBuf, int* lenProp) if((lenRead = readlink(namebuf, (char*)buf, lenBuf - 1)) == -1) { DBGPRINTF("error reading link '%s'\n", namebuf); - *lenProp = 0; - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } buf[lenRead] = '\0'; @@ -767,26 +711,6 @@ copyescaped(uchar *dstbuf, uchar *inbuf, int inlen) } -/* Creates new field to be added to event - * used for SystemLogParseTrusted parsing - */ -struct ee_field * -createNewField(char *fieldname, char *value, int lenValue) { - es_str_t *newStr; - struct ee_value *newVal; - struct ee_field *newField; - - newStr = es_newStrFromBuf(value, (es_size_t) lenValue); - - newVal = ee_newValue(ctxee); - ee_setStrValue(newVal, newStr); - - newField = ee_newFieldFromNV(ctxee, fieldname, newVal); - - return newField; -} - - /* submit received message to the queue engine * We now parse the message according to expected format so that we * can also mangle it if necessary. @@ -805,16 +729,21 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim uchar bufParseTAG[CONF_TAG_MAXSIZE]; struct syslogTime st; time_t tt; - rs_ratelimit_state_t *ratelimiter = NULL; int lenProp; + ratelimit_t *ratelimiter = NULL; uchar propBuf[1024]; uchar msgbuf[8192]; uchar *pmsgbuf; int toffs; /* offset for trusted properties */ struct syslogTime dummyTS; - struct ee_event *event = NULL; + struct json_object *json = NULL, *jval; DEFiRet; + if(pLstn->bDiscardOwnMsgs && cred != NULL && cred->pid == glblGetOurPid()) { + DBGPRINTF("imuxsock: discarding message from our own pid\n"); + FINALIZE; + } + /* TODO: handle format errors?? */ /* we need to parse the pri first, because we need the severity for * rate-limiting as well. @@ -845,10 +774,12 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim tt = ts->tv_sec; } +#if 0 // TODO: think about stats counters (or wait for request...?) if(ratelimiter != NULL && !withinRatelimit(ratelimiter, tt, cred->pid)) { STATSCOUNTER_INC(ctrLostRatelimit, mutCtrLostRatelimit); FINALIZE; } +#endif /* created trusted properties */ if(cred != NULL && pLstn->bAnnotate) { @@ -859,45 +790,27 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim } if (pLstn->bParseTrusted) { - struct ee_field *newField; - - if(ctxee == NULL) { - if((ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } + json = json_object_new_object(); + /* create value string, create field, and add it */ + jval = json_object_new_int(cred->pid); + json_object_object_add(json, "pid", jval); + jval = json_object_new_int(cred->uid); + json_object_object_add(json, "uid", jval); + jval = json_object_new_int(cred->gid); + json_object_object_add(json, "gid", jval); + if(getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "appname", jval); + } + if(getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "exe", jval); + } + if(getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "cmd", jval); } - - event = ee_newEvent(ctxee); - - /* create value string, create field, and add it to event */ - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->pid); - newField = createNewField("pid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->uid); - newField = createNewField("uid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->gid); - newField = createNewField("gid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("appname", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("exe", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("cmd", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - } else { - memcpy(pmsgbuf, pRcv, lenRcv); memcpy(pmsgbuf+lenRcv, " @[", 3); toffs = lenRcv + 3; /* next free location */ @@ -907,23 +820,20 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim memcpy(pmsgbuf+toffs, propBuf, lenProp); toffs = toffs + lenProp; - getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { + if(getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { memcpy(pmsgbuf+toffs, " _COMM=", 7); memcpy(pmsgbuf+toffs+7, propBuf, lenProp); toffs = toffs + 7 + lenProp; } - getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { + if(getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { memcpy(pmsgbuf+toffs, " _EXE=", 6); memcpy(pmsgbuf+toffs+6, propBuf, lenProp); toffs = toffs + 6 + lenProp; } - getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { - memcpy(pmsgbuf+toffs, " _CMDLINE=", 9); - toffs = toffs + 9 + - copyescaped(pmsgbuf+toffs+9, propBuf, lenProp); + if(getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + memcpy(pmsgbuf+toffs, " _CMDLINE=", 10); + toffs = toffs + 10 + + copyescaped(pmsgbuf+toffs+10, propBuf, lenProp); } /* finalize string */ @@ -949,12 +859,11 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim parse++; lenMsg--; /* '>' */ - /* event is saved to pMsg */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - if (event != NULL) { - pMsg->event = event; + if(json != NULL) { + /* as per lumberjack spec, these properties need to go into + * the CEE root. + */ + msgAddJSON(pMsg, (uchar*)"!", json); } if(ts == NULL) { @@ -962,15 +871,19 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim /* in this case, we still need to find out if we have a valid * datestamp or not .. and advance the parse pointer accordingly. */ - datetime.ParseTIMESTAMP3164(&dummyTS, &parse, &lenMsg); + if (datetime.ParseTIMESTAMP3339(&dummyTS, &parse, &lenMsg) != RS_RET_OK) { + datetime.ParseTIMESTAMP3164(&dummyTS, &parse, &lenMsg); + } } else { - if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK) { + if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK && + datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK) { DBGPRINTF("we have a problem, invalid timestamp in msg!\n"); } } } else { /* if we pulled the time from the system, we need to update the message text */ uchar *tmpParse = parse; /* just to check correctness of TS */ - if(datetime.ParseTIMESTAMP3164(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK) { + if(datetime.ParseTIMESTAMP3339(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK || + datetime.ParseTIMESTAMP3164(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK) { /* We modify the message only if it contained a valid timestamp, * otherwise we do not touch it at all. */ datetime.formatTimestamp3164(&st, (char*)parse, 0); @@ -1001,8 +914,7 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim MsgSetRcvFrom(pMsg, pLstn->hostName == NULL ? glbl.GetLocalHostNameProp() : pLstn->hostName); CHKiRet(MsgSetRcvFromIP(pMsg, pLocalHostIP)); - CHKiRet(submitMsg(pMsg)); - + ratelimitAddMsg(ratelimiter, NULL, pMsg); STATSCOUNTER_INC(ctrSubmit, mutCtrSubmit); finalize_it: RETiRet; @@ -1137,13 +1049,18 @@ activateListeners() listeners[0].ratelimitInterval = runModConf->ratelimitIntervalSysSock; listeners[0].ratelimitBurst = runModConf->ratelimitBurstSysSock; listeners[0].ratelimitSev = runModConf->ratelimitSeveritySysSock; - listeners[0].bUseCreds = (runModConf->bWritePidSysSock || runModConf->ratelimitIntervalSysSock || runModConf->bAnnotateSysSock) ? 1 : 0; + listeners[0].bUseCreds = (runModConf->bWritePidSysSock || runModConf->ratelimitIntervalSysSock || runModConf->bAnnotateSysSock || runModConf->bDiscardOwnMsgs) ? 1 : 0; listeners[0].bWritePid = runModConf->bWritePidSysSock; listeners[0].bAnnotate = runModConf->bAnnotateSysSock; listeners[0].bParseTrusted = runModConf->bParseTrusted; + listeners[0].bDiscardOwnMsgs = runModConf->bDiscardOwnMsgs; listeners[0].bUseSysTimeStamp = runModConf->bUseSysTimeStamp; listeners[0].flags = runModConf->bIgnoreTimestamp ? IGNDATE : NOFLAG; listeners[0].flowCtl = runModConf->bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; + CHKiRet(ratelimitNew(&listeners[0].dflt_ratelimiter, "imuxsock", NULL)); + ratelimitSetLinuxLike(listeners[0].dflt_ratelimiter, + listeners[0].ratelimitInterval, + listeners[0].ratelimitBurst); sd_fds = sd_listen_fds(0); if(sd_fds < 0) { @@ -1185,6 +1102,7 @@ CODESTARTbeginCnfLoad pModConf->bWritePidSysSock = 0; pModConf->bAnnotateSysSock = 0; pModConf->bParseTrusted = 0; + pModConf->bDiscardOwnMsgs = 1; pModConf->ratelimitIntervalSysSock = DFLT_ratelimitInterval; pModConf->ratelimitBurstSysSock = DFLT_ratelimitBurst; pModConf->ratelimitSeveritySysSock = DFLT_ratelimitSeverity; @@ -1219,6 +1137,8 @@ CODESTARTsetModCnf loadModConf->pLogSockName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "syssock.ignoretimestamp")) { loadModConf->bIgnoreTimestamp = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "syssock.ignoreownmessages")) { + loadModConf->bDiscardOwnMsgs = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.flowcontrol")) { loadModConf->bUseFlowCtl = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.usesystimestamp")) { @@ -1279,6 +1199,8 @@ CODESTARTnewInpInst inst->bCreatePath = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "parsetrusted")) { inst->bParseTrusted = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "ignoreownmessages")) { + inst->bDiscardOwnMsgs = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "hostname")) { inst->pLogHostName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "ignoretimestamp")) { @@ -1461,10 +1383,6 @@ CODESTARTafterRun discardLogSockets(); nfd = 1; - if(ctxee != NULL) { - ee_exitCtx(ctxee); - ctxee = NULL; - } ENDafterRun @@ -1572,8 +1490,16 @@ CODEmodInit_QueryRegCFSLineHdlr listeners[0].bUseCreds = 0; listeners[0].bAnnotate = 0; listeners[0].bParseTrusted = 0; + listeners[0].bDiscardOwnMsgs = 1; listeners[0].bCreatePath = 0; listeners[0].bUseSysTimeStamp = 1; + if((listeners[0].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))ratelimitDestruct)) == NULL) { + /* in this case, we simply turn off rate-limiting */ + DBGPRINTF("imuxsock: turning off rate limiting for system socket " + "because we could not create hash table\n"); + listeners[0].ratelimitInterval = 0; + } /* initialize socket names */ for(i = 1 ; i < MAXFUNIX ; ++i) { diff --git a/plugins/imzmq3/imzmq3.c b/plugins/imzmq3/imzmq3.c index dc1d64d3..52c12a53 100644 --- a/plugins/imzmq3/imzmq3.c +++ b/plugins/imzmq3/imzmq3.c @@ -375,8 +375,10 @@ static rsRetVal createSocket(socket_info* info, void** sock) { zsocket_set_rcvhwm(*sock, info->rcvHWM); /* Set subscriptions.*/ - for (ii = 0; ii < sizeof(info->subscriptions)/sizeof(char*); ++ii) - zsocket_set_subscribe(*sock, info->subscriptions[ii]); + if (info->type == ZMQ_SUB) { + for (ii = 0; ii < sizeof(info->subscriptions)/sizeof(char*); ++ii) + zsocket_set_subscribe(*sock, info->subscriptions[ii]); + } diff --git a/plugins/mmanon/Makefile.am b/plugins/mmanon/Makefile.am new file mode 100644 index 00000000..98f0da24 --- /dev/null +++ b/plugins/mmanon/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmanon.la + +mmanon_la_SOURCES = mmanon.c +mmanon_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmanon_la_LDFLAGS = -module -avoid-version +mmanon_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmanon/mmanon.c b/plugins/mmanon/mmanon.c new file mode 100644 index 00000000..fc0c8a03 --- /dev/null +++ b/plugins/mmanon/mmanon.c @@ -0,0 +1,400 @@ +/* mmanon.c + * anonnymize IP addresses inside the syslog message part + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmanon") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +/* precomputed table of IPv4 anonymization masks */ +static const uint32_t ipv4masks[33] = { + 0xffffffff, 0xfffffffe, 0xfffffffc, 0xfffffff8, + 0xfffffff0, 0xffffffe0, 0xffffffc0, 0xffffff80, + 0xffffff00, 0xfffffe00, 0xfffffc00, 0xfffff800, + 0xfffff000, 0xffffe000, 0xffffc000, 0xffff8000, + 0xffff0000, 0xfffe0000, 0xfffc0000, 0xfff80000, + 0xfff00000, 0xffe00000, 0xffc00000, 0xff800000, + 0xff000000, 0xfe000000, 0xfc000000, 0xf8000000, + 0xf0000000, 0xe0000000, 0xc0000000, 0x80000000, + 0x00000000 + }; + +/* define operation modes we have */ +#define SIMPLE_MODE 0 /* just overwrite */ +#define REWRITE_MODE 1 /* rewrite IP address, canoninized */ +typedef struct _instanceData { + char replChar; + int8_t mode; + struct { + int8_t bits; + } ipv4; +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "mode", eCmdHdlrGetWord, 0 }, + { "replacementchar", eCmdHdlrGetChar, 0 }, + { "ipv4.bits", eCmdHdlrInt, 0 }, +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->mode = REWRITE_MODE; + pData->replChar = 'x'; + pData->ipv4.bits = 16; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; + sbool bHadBitsErr; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmanon)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "mode")) { + if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"simple", + sizeof("simple")-1)) { + pData->mode = SIMPLE_MODE; + } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"rewrite", + sizeof("rewrite")-1)) { + pData->mode = REWRITE_MODE; + } else { + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_INVLD_MODE, + "mmanon: invalid anonymization mode '%s' - ignored", + cstr); + free(cstr); + } + pData->replChar = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else if(!strcmp(actpblk.descr[i].name, "replacementchar")) { + pData->replChar = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else if(!strcmp(actpblk.descr[i].name, "ipv4.bits")) { + pData->ipv4.bits = (int8_t) pvals[i].val.d.n; + } else { + dbgprintf("mmanon: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + + if(pData->mode == SIMPLE_MODE) { + bHadBitsErr = 0; + if(pData->ipv4.bits < 8) { + pData->ipv4.bits = 8; + bHadBitsErr = 1; + } else if(pData->ipv4.bits < 16) { + pData->ipv4.bits = 16; + bHadBitsErr = 1; + } else if(pData->ipv4.bits < 24) { + pData->ipv4.bits = 24; + bHadBitsErr = 1; + } else if(pData->ipv4.bits != 32) { + pData->ipv4.bits = 32; + bHadBitsErr = 1; + } + if(bHadBitsErr) + errmsg.LogError(0, RS_RET_INVLD_ANON_BITS, + "mmanon: invalid number of ipv4 bits " + "in simple mode, corrected to %d", + pData->ipv4.bits); + } else { /* REWRITE_MODE */ + if(pData->ipv4.bits < 1 || pData->ipv4.bits > 32) { + pData->ipv4.bits = 32; + errmsg.LogError(0, RS_RET_INVLD_ANON_BITS, + "mmanon: invalid number of ipv4 bits " + "in rewrite mode, corrected to %d", + pData->ipv4.bits); + } + if(pData->replChar != 'x') { + errmsg.LogError(0, RS_RET_REPLCHAR_IGNORED, + "mmanon: replacementChar parameter is ignored " + "in rewrite mode"); + } + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +static int +getnum(uchar *msg, int lenMsg, int *idx) +{ + int num = 0; + int i = *idx; + + while(i < lenMsg && msg[i] >= '0' && msg[i] <= '9') { + num = num * 10 + msg[i] - '0'; + ++i; + } + + *idx = i; + return num; +} + + +/* write an IP address octet to the output position */ +static int +writeOctet(uchar *msg, int idx, int *nxtidx, uint8_t octet) +{ + if(octet > 99) { + msg[idx++] = '0' + octet / 100; + octet = octet % 100; + } + if(octet > 9) { + msg[idx++] = '0' + octet / 10; + octet = octet % 10; + } + msg[idx++] = '0' + octet; + + if(nxtidx != NULL) { + if(idx + 1 != *nxtidx) { + /* we got shorter, fix it! */ + msg[idx] = '.'; + *nxtidx = idx + 1; + } + } + return idx; +} + +/* currently works for IPv4 only! */ +void +anonip(instanceData *pData, uchar *msg, int *pLenMsg, int *idx) +{ + int i = *idx; + int octet; + uint32_t ipv4addr; + int ipstart[4]; + int j; + int endpos; + int lenMsg = *pLenMsg; + + while(i < lenMsg && (msg[i] <= '0' || msg[i] >= '9')) { + ++i; /* skip to first number */ + } + if(i >= lenMsg) + goto done; + + /* got digit, let's see if ip */ + ipstart[0] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr = octet << 24; + ++i; + ipstart[1] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr |= octet << 16; + ++i; + ipstart[2] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr |= octet << 8; + ++i; + ipstart[3] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || !(msg[i] == ' ' || msg[i] == ':')) goto done; + ipv4addr |= octet; + + /* OK, we now found an ip address */ + if(pData->mode == SIMPLE_MODE) { + if(pData->ipv4.bits == 8) + j = ipstart[3]; + else if(pData->ipv4.bits == 16) + j = ipstart[2]; + else if(pData->ipv4.bits == 24) + j = ipstart[1]; + else /* due to our checks, this *must* be 32 */ + j = ipstart[0]; + while(j < i) { + if(msg[j] != '.') + msg[j] = pData->replChar; + ++j; + } + } else { /* REWRITE_MODE */ + ipv4addr &= ipv4masks[pData->ipv4.bits]; + if(pData->ipv4.bits > 24) + writeOctet(msg, ipstart[0], &(ipstart[1]), ipv4addr >> 24); + if(pData->ipv4.bits > 16) + writeOctet(msg, ipstart[1], &(ipstart[2]), (ipv4addr >> 16) & 0xff); + if(pData->ipv4.bits > 8) + writeOctet(msg, ipstart[2], &(ipstart[3]), (ipv4addr >> 8) & 0xff); + endpos = writeOctet(msg, ipstart[3], NULL, ipv4addr & 0xff); + /* if we had truncation, we need to shrink the msg */ + dbgprintf("existing i %d, endpos %d\n", i, endpos); + if(i - endpos > 0) { + *pLenMsg = lenMsg - (i - endpos); + memmove(msg+endpos, msg+i, lenMsg - i + 1); + } + } + +done: *idx = i; + return; +} + + +BEGINdoAction + msg_t *pMsg; + uchar *msg; + int lenMsg; + int i; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + lenMsg = getMSGLen(pMsg); + msg = getMSG(pMsg); + for(i = 0 ; i < lenMsg ; ++i) { + anonip(pData, msg, &lenMsg, &i); + } + if(lenMsg != getMSGLen(pMsg)) + setMSGLen(pMsg, lenMsg); +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmanon:", sizeof(":mmanon:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmanon supports only v6+ config format, use: " + "action(type=\"mmanon\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmanon: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmaudit/mmaudit.c b/plugins/mmaudit/mmaudit.c index fcefd013..018e1771 100644 --- a/plugins/mmaudit/mmaudit.c +++ b/plugins/mmaudit/mmaudit.c @@ -67,13 +67,8 @@ DEFobjCurrIf(errmsg); DEF_OMOD_STATIC_DATA typedef struct _instanceData { - ee_ctx ctxee; /**< context to be used for libee */ -} instanceData; - -typedef struct configSettings_s { int dummy; /* remove when the first real parameter is needed */ -} configSettings_t; -static configSettings_t cs; +} instanceData; BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars @@ -93,7 +88,6 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - ee_exitCtx(pData->ctxee); ENDfreeInstance @@ -169,17 +163,20 @@ finalize_it: /* parse the audit record and create libee structure */ static rsRetVal -audit_parse(instanceData *pData, uchar *buf, struct ee_event **event) +audit_parse(uchar *buf, struct json_object **jsonRoot) { - es_str_t *estr; + struct json_object *json; + struct json_object *jval; char name[1024]; char val[1024]; DEFiRet; - *event = ee_newEvent(pData->ctxee); - if(event == NULL) { + *jsonRoot = json_object_new_object(); + if(*jsonRoot == NULL) { ABORT_FINALIZE(RS_RET_ERR); } + json = json_object_new_object(); + json_object_object_add(*jsonRoot, "data", json); while(*buf) { //dbgprintf("audit_parse, buf: '%s'\n", buf); @@ -189,10 +186,8 @@ audit_parse(instanceData *pData, uchar *buf, struct ee_event **event) } ++buf; CHKiRet(parseValue(&buf, val, sizeof(val))); - - estr = es_newStrFromCStr(val, strlen(val)); - ee_addStrFieldToEvent(*event, name, estr); - es_deleteStr(estr); + jval = json_object_new_string(val); + json_object_object_add(json, name, jval); dbgprintf("mmaudit: parsed %s=%s\n", name, val); } @@ -206,9 +201,10 @@ BEGINdoAction msg_t *pMsg; uchar *buf; int typeID; - struct ee_event *event; + struct json_object *jsonRoot; + struct json_object *json; + struct json_object *jval; int i; - es_str_t *estr; char auditID[1024]; int bSuccess = 0; CODESTARTdoAction @@ -252,48 +248,24 @@ dbgprintf("mmaudit: msg is '%s'\n", buf); } buf += 2; -dbgprintf("mmaudit: cookie found, type %d, auditID '%s', rest of message: '%s'\n", typeID, auditID, buf); - audit_parse(pData, buf, &event); - if(event == NULL) { + audit_parse(buf, &jsonRoot); + if(jsonRoot == NULL) { DBGPRINTF("mmaudit: audit parse error, assuming no " "audit message: '%s'\n", buf); FINALIZE; } /* we now need to shuffle the "outer" properties into that stream */ - estr = es_newStrFromCStr(auditID, strlen(auditID)); - ee_addStrFieldToEvent(event, "audithdr.auditid", estr); - es_deleteStr(estr); - - /* we abuse auditID a bit to save space... (TODO: change!) */ - snprintf(auditID, sizeof(auditID), "%d", typeID); - estr = es_newStrFromCStr(auditID, strlen(auditID)); - ee_addStrFieldToEvent(event, "audithdr.type", estr); - es_deleteStr(estr); - - /* TODO: in the long term, we need to think about merging & different - name spaces (probably best to add the newly-obtained event as a child to - the existing event...) - */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - pMsg->event = event; + json = json_object_new_object(); + json_object_object_add(jsonRoot, "hdr", json); + jval = json_object_new_string(auditID); + json_object_object_add(json, "auditid", jval); + jval = json_object_new_int(typeID); + json_object_object_add(json, "type", jval); + + msgAddJSON(pMsg, (uchar*)"!audit", jsonRoot); bSuccess = 1; -#if 1 - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - es_str_t *str; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmaudit generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ -#endif finalize_it: MsgSetParseSuccess(pMsg, bSuccess); ENDdoAction @@ -318,13 +290,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct diff --git a/plugins/mmjsonparse/mmjsonparse.c b/plugins/mmjsonparse/mmjsonparse.c index 111ecc2f..c47aceb6 100644 --- a/plugins/mmjsonparse/mmjsonparse.c +++ b/plugins/mmjsonparse/mmjsonparse.c @@ -36,7 +36,7 @@ #include <unistd.h> #include <ctype.h> #include <libestr.h> -#include <libee/libee.h> +#include <json/json.h> #include "conf.h" #include "syslogd-types.h" #include "template.h" @@ -59,22 +59,49 @@ DEFobjCurrIf(errmsg); DEF_OMOD_STATIC_DATA typedef struct _instanceData { - ee_ctx ctxee; /**< context to be used for libee */ + struct json_tokener *tokener; } instanceData; -typedef struct configSettings_s { - int dummy; /* remove when the first real parameter is needed */ -} configSettings_t; -static configSettings_t cs; +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ -BEGINinitConfVars /* (re)set config variables to default values */ -CODESTARTinitConfVars - resetConfigVariables(NULL, NULL); -ENDinitConfVars + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance + pData->tokener = json_tokener_new(); + if(pData->tokener == NULL) { + errmsg.LogError(0, RS_RET_ERR, "error: could not create json " + "tokener, cannot activate action"); + ABORT_FINALIZE(RS_RET_ERR); + } +finalize_it: ENDcreateInstance @@ -85,13 +112,14 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - ee_exitCtx(pData->ctxee); + if(pData->tokener != NULL) + json_tokener_free(pData->tokener); ENDfreeInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo - dbgprintf("mmjsonparse\n"); + DBGPRINTF("mmjsonparse\n"); ENDdbgPrintInstInfo @@ -99,13 +127,57 @@ BEGINtryResume CODESTARTtryResume ENDtryResume + +static rsRetVal +processJSON(instanceData *pData, msg_t *pMsg, char *buf, size_t lenBuf) +{ + struct json_object *json; + const char *errMsg; + DEFiRet; + + assert(pData->tokener != NULL); + DBGPRINTF("mmjsonparse: toParse: '%s'\n", buf); + json_tokener_reset(pData->tokener); + + json = json_tokener_parse_ex(pData->tokener, buf, lenBuf); + if(Debug) { + errMsg = NULL; + if(json == NULL) { + enum json_tokener_error err; + + err = pData->tokener->err; + if(err != json_tokener_continue) + errMsg = json_tokener_errors[err]; + else + errMsg = "Unterminated input"; + } else if((size_t)pData->tokener->char_offset < lenBuf) + errMsg = "Extra characters after JSON object"; + else if(!json_object_is_type(json, json_type_object)) + errMsg = "JSON value is not an object"; + if(errMsg != NULL) { + DBGPRINTF("mmjsonparse: Error parsing JSON '%s': %s\n", + buf, errMsg); + } + } + if(json == NULL + || ((size_t)pData->tokener->char_offset < lenBuf) + || (!json_object_is_type(json, json_type_object))) { + ABORT_FINALIZE(RS_RET_NO_CEE_MSG); + } + + msgAddJSON(pMsg, (uchar*)"!", json); +finalize_it: + RETiRet; +} + #define COOKIE "@cee:" #define LEN_COOKIE (sizeof(COOKIE)-1) BEGINdoAction msg_t *pMsg; uchar *buf; - struct ee_event *event; int bSuccess = 0; + struct json_object *jval; + struct json_object *json; CODESTARTdoAction pMsg = (msg_t*) ppString[0]; /* note that we can performance-optimize the interface, but this also @@ -114,50 +186,45 @@ CODESTARTdoAction */ buf = getMSG(pMsg); -dbgprintf("mmjsonparse: msg is '%s'\n", buf); while(*buf && isspace(*buf)) { ++buf; } if(*buf == '\0' || strncmp((char*)buf, COOKIE, LEN_COOKIE)) { DBGPRINTF("mmjsonparse: no JSON cookie: '%s'\n", buf); - FINALIZE; + ABORT_FINALIZE(RS_RET_NO_CEE_MSG); } buf += LEN_COOKIE; -dbgprintf("mmjsonparse: cookie found, rest of message: '%s'\n", buf); - event = ee_newEventFromJSON(pData->ctxee, (char*)buf); - if(event == NULL) { - DBGPRINTF("mmjsonparse: JSON parse error, assuming no " - "JSON-enhanced message: '%s'\n", buf); - FINALIZE; - } - /* TODO: in the long term, we need to think about merging & different - name spaces (probably best to add the newly-obtained event as a child to - the existing event...) - */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - pMsg->event = event; + CHKiRet(processJSON(pData, pMsg, (char*) buf, strlen((char*)buf))); bSuccess = 1; - -#if 1 - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - es_str_t *str; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmjsonparse generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ -#endif finalize_it: + if(iRet == RS_RET_NO_CEE_MSG) { + /* add buf as msg */ + json = json_object_new_object(); + jval = json_object_new_string((char*)buf); + json_object_object_add(json, "msg", jval); + msgAddJSON(pMsg, (uchar*)"!", json); + iRet = RS_RET_OK; + } MsgSetParseSuccess(pMsg, bSuccess); ENDdoAction +BEGINnewActInst +CODESTARTnewActInst + /* Note: we currently do not have any parameters, so we do not need + * the lst ptr. However, we will most probably need params in the + * future. + */ + DBGPRINTF("newActInst (mmjsonparse)\n"); + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + /*setInstParamDefaults(pData);*/ + +CODE_STD_FINALIZERnewActInst +/* cnfparamvalsDestruct(pvals, &actpblk);*/ +ENDnewActInst BEGINparseSelectorAct CODESTARTparseSelectorAct @@ -178,13 +245,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -198,6 +258,8 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES ENDqueryEtryPt @@ -217,10 +279,10 @@ BEGINmodInit() unsigned long opts; int bMsgPassingSupported; CODESTARTmodInit -INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmjsonparse: module compiled with rsyslog version %s.\n", VERSION); /* check if the rsyslog core supports parameter passing code */ bMsgPassingSupported = 0; localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", diff --git a/plugins/mmnormalize/mmnormalize.c b/plugins/mmnormalize/mmnormalize.c index 2dacb80b..fd2004a3 100644 --- a/plugins/mmnormalize/mmnormalize.c +++ b/plugins/mmnormalize/mmnormalize.c @@ -4,9 +4,12 @@ * * NOTE: read comments in module-template.h for details on the calling interface! * + * TODO: check if we can replace libee via JSON system - currently that part + * is pretty inefficient... rgerhards, 2012-08-27 + * * File begun on 2010-01-01 by RGerhards * - * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * Copyright 2010-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -37,6 +40,7 @@ #include <unistd.h> #include <libestr.h> #include <libee/libee.h> +#include <json/json.h> #include <liblognorm.h> #include "conf.h" #include "syslogd-types.h" @@ -61,6 +65,7 @@ DEF_OMOD_STATIC_DATA typedef struct _instanceData { sbool bUseRawMsg; /**< use %rawmsg% instead of %msg% */ + uchar *rulebase; /**< name of rulebase to use */ ln_ctx ctxln; /**< context to be used for liblognorm */ ee_ctx ctxee; /**< context to be used for libee */ } instanceData; @@ -71,6 +76,58 @@ typedef struct configSettings_s { } configSettings_t; static configSettings_t cs; +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "rulebase", eCmdHdlrGetWord, 1 }, + { "userawmsg", eCmdHdlrBinary, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* to be called to build the libee part of the instance ONCE ALL PARAMETERS ARE CORRECT + * (and set within pData!). + */ +static rsRetVal +buildInstance(instanceData *pData) +{ + DEFiRet; + if((pData->ctxee = ee_initCtx()) == NULL) { + errmsg.LogError(0, RS_RET_ERR_LIBEE_INIT, "error: could not initialize libee " + "ctx, cannot activate action"); + ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); + } + + if((pData->ctxln = ln_initCtx()) == NULL) { + errmsg.LogError(0, RS_RET_ERR_LIBLOGNORM_INIT, "error: could not initialize " + "liblognorm ctx, cannot activate action"); + ee_exitCtx(pData->ctxee); + ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_INIT); + } + ln_setEECtx(pData->ctxln, pData->ctxee); + if(ln_loadSamples(pData->ctxln, (char*) pData->rulebase) != 0) { + errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rulebase '%s' " + "could not be loaded cannot activate action", cs.rulebase); + ee_exitCtx(pData->ctxee); + ln_exitCtx(pData->ctxln); + ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); + } +finalize_it: + RETiRet; +} + + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars resetConfigVariables(NULL, NULL); @@ -82,6 +139,35 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + + +BEGINendCnfLoad +CODESTARTendCnfLoad + loadModConf = NULL; /* done loading */ + /* free legacy config vars */ + free(cs.rulebase); + cs.rulebase = NULL; +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature ENDisCompatibleWithFeature @@ -89,6 +175,7 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance + free(pData->rulebase); ee_exitCtx(pData->ctxee); ln_exitCtx(pData->ctxln); ENDfreeInstance @@ -108,8 +195,12 @@ BEGINdoAction msg_t *pMsg; es_str_t *str; uchar *buf; + char *cstrJSON; int len; int r; + struct ee_event *event = NULL; + struct json_tokener *tokener; + struct json_object *json; CODESTARTdoAction pMsg = (msg_t*) ppString[0]; /* note that we can performance-optimize the interface, but this also @@ -123,7 +214,7 @@ CODESTARTdoAction len = getMSGLen(pMsg); } str = es_newStrFromCStr((char*)buf, len); - r = ln_normalize(pData->ctxln, str, &pMsg->event); + r = ln_normalize(pData->ctxln, str, &event); if(r != 0) { DBGPRINTF("error %d during ln_normalize\n", r); MsgSetParseSuccess(pMsg, 0); @@ -131,19 +222,76 @@ CODESTARTdoAction MsgSetParseSuccess(pMsg, 1); } es_deleteStr(str); - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmnormalize generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ + + /* reformat to our json data struct */ + /* TODO: this is all extremly ineffcient! */ + ee_fmtEventToJSON(event, &str); + cstrJSON = es_str2cstr(str, NULL); + dbgprintf("mmnormalize generated: %s\n", cstrJSON); + + tokener = json_tokener_new(); + json = json_tokener_parse_ex(tokener, cstrJSON, strlen((char*)cstrJSON)); + json_tokener_free(tokener); + msgAddJSON(pMsg, (uchar*)"!", json); + + free(cstrJSON); + es_deleteStr(str); ENDdoAction +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->rulebase = NULL; + pData->bUseRawMsg = 0; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; + int bDestructPValsOnExit; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmnormalize)\n"); + + bDestructPValsOnExit = 0; + pvals = nvlstGetParams(lst, &actpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "mmnormalize: error reading " + "config parameters"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + bDestructPValsOnExit = 1; + + if(Debug) { + dbgprintf("action param blk in mmnormalize:\n"); + cnfparamsPrint(&actpblk, pvals); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "rulebase")) { + pData->rulebase = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "userawmsg")) { + pData->bUseRawMsg = (int) pvals[i].val.d.n; + } else { + DBGPRINTF("mmnormalize: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + + iRet = buildInstance(pData); +CODE_STD_FINALIZERnewActInst + if(bDestructPValsOnExit) + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + BEGINparseSelectorAct CODESTARTparseSelectorAct CODE_STD_STRING_REQUESTparseSelectorAct(1) @@ -153,15 +301,21 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) } if(cs.rulebase == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: no normalization rulebase was specified, use " + errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: no normalization rulebase was specified, use " "$MMNormalizeSampleDB directive first!"); - ABORT_FINALIZE(RS_RET_NO_RULESET); + ABORT_FINALIZE(RS_RET_NO_RULEBASE); } /* ok, if we reach this point, we have something for us */ p += sizeof(":mmnormalize:") - 1; /* eat indicator sequence (-1 because of '\0'!) */ CHKiRet(createInstance(&pData)); + pData->rulebase = cs.rulebase; + pData->bUseRawMsg = cs.bUseRawMsg; + /* all config vars auto-reset! */ + cs.bUseRawMsg = 0; + cs.rulebase = NULL; /* we used it up! */ + /* check if a non-standard template is to be applied */ if(*(p-1) == ';') --p; @@ -169,34 +323,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } - - if((pData->ctxln = ln_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize liblognorm ctx, cannot " - "activate action"); - ee_exitCtx(pData->ctxee); - ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_INIT); - } - ln_setEECtx(pData->ctxln, pData->ctxee); - if(ln_loadSamples(pData->ctxln, (char*) cs.rulebase) != 0) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: normalization rulebase '%s' could not be loaded " - "cannot activate action", cs.rulebase); - ee_exitCtx(pData->ctxee); - ln_exitCtx(pData->ctxln); - ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); - } - pData->bUseRawMsg = cs.bUseRawMsg; - - /* all config vars auto-reset! */ - cs.bUseRawMsg = 0; - free(cs.rulebase); - cs.rulebase = NULL; + CHKiRet(buildInstance(pData)); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -210,6 +337,8 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt @@ -244,6 +373,7 @@ INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmnormalize: module compiled with rsyslog version %s.\n", VERSION); /* check if the rsyslog core supports parameter passing code */ bMsgPassingSupported = 0; localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", diff --git a/plugins/mmsnmptrapd/mmsnmptrapd.c b/plugins/mmsnmptrapd/mmsnmptrapd.c index b1ac2f64..b79a311b 100644 --- a/plugins/mmsnmptrapd/mmsnmptrapd.c +++ b/plugins/mmsnmptrapd/mmsnmptrapd.c @@ -362,6 +362,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omelasticsearch/Makefile.am b/plugins/omelasticsearch/Makefile.am index a574c72f..ba85a896 100644 --- a/plugins/omelasticsearch/Makefile.am +++ b/plugins/omelasticsearch/Makefile.am @@ -1,8 +1,9 @@ pkglib_LTLIBRARIES = omelasticsearch.la -omelasticsearch_la_SOURCES = omelasticsearch.c +# TODO: replace cJSON +omelasticsearch_la_SOURCES = omelasticsearch.c cJSON/cjson.c cJSON/cjson.h omelasticsearch_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) omelasticsearch_la_LDFLAGS = -module -avoid-version -omelasticsearch_la_LIBADD = $(CURL_LIBS) +omelasticsearch_la_LIBADD = $(CURL_LIBS) $(LIBM) EXTRA_DIST = diff --git a/plugins/omelasticsearch/README b/plugins/omelasticsearch/README new file mode 100644 index 00000000..9021bc0e --- /dev/null +++ b/plugins/omelasticsearch/README @@ -0,0 +1,17 @@ +How to produce an error: +======================== +It's quite easy to get 400, if you put a wrong mapping to your +index. That would be easy to reproduce in "normal" omelasticsearch usage +conditions, by only altering the ES configuration: + +1. Make your index first. Let's call it "testindex": +$ curl -XPUT localhost:9200/testindex/ + +2. Put your mapping for a search type called "mytype", where you specify +that date property should be an integer: +$ curl -XPUT localhost:9200/testindex/mytype/_mapping -d '{"mytype":{"properties": {"timegenerated":{"type":"integer"}}}}' + +3. Now try to insert something where date is not an integer: +$ curl -XPOST localhost:9200/testindex/mytype/ -d '{"timegenerated":"bla"}' +{"error":"MapperParsingException[Failed to parse [date]]; nested: NumberFormatException[For input string: \"bla\"]; ","status":400} + diff --git a/plugins/omelasticsearch/cJSON/README b/plugins/omelasticsearch/cJSON/README new file mode 100644 index 00000000..7531c049 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/README @@ -0,0 +1,247 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +Welcome to cJSON. + +cJSON aims to be the dumbest possible parser that you can get your job done with. +It's a single file of C, and a single header file. + +JSON is described best here: http://www.json.org/ +It's like XML, but fat-free. You use it to move data around, store things, or just +generally represent your program's state. + + +First up, how do I build? +Add cJSON.c to your project, and put cJSON.h somewhere in the header search path. +For example, to build the test app: + +gcc cJSON.c test.c -o test -lm +./test + + +As a library, cJSON exists to take away as much legwork as it can, but not get in your way. +As a point of pragmatism (i.e. ignoring the truth), I'm going to say that you can use it +in one of two modes: Auto and Manual. Let's have a quick run-through. + + +I lifted some JSON from this page: http://www.json.org/fatfree.html +That page inspired me to write cJSON, which is a parser that tries to share the same +philosophy as JSON itself. Simple, dumb, out of the way. + +Some JSON: +{ + "name": "Jack (\"Bee\") Nimble", + "format": { + "type": "rect", + "width": 1920, + "height": 1080, + "interlace": false, + "frame rate": 24 + } +} + +Assume that you got this from a file, a webserver, or magic JSON elves, whatever, +you have a char * to it. Everything is a cJSON struct. +Get it parsed: + cJSON *root = cJSON_Parse(my_json_string); + +This is an object. We're in C. We don't have objects. But we do have structs. +What's the framerate? + + cJSON *format = cJSON_GetObjectItem(root,"format"); + int framerate = cJSON_GetObjectItem(format,"frame rate")->valueint; + + +Want to change the framerate? + cJSON_GetObjectItem(format,"frame rate")->valueint=25; + +Back to disk? + char *rendered=cJSON_Print(root); + +Finished? Delete the root (this takes care of everything else). + cJSON_Delete(root); + +That's AUTO mode. If you're going to use Auto mode, you really ought to check pointers +before you dereference them. If you want to see how you'd build this struct in code? + cJSON *root,*fmt; + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "name", cJSON_CreateString("Jack (\"Bee\") Nimble")); + cJSON_AddItemToObject(root, "format", fmt=cJSON_CreateObject()); + cJSON_AddStringToObject(fmt,"type", "rect"); + cJSON_AddNumberToObject(fmt,"width", 1920); + cJSON_AddNumberToObject(fmt,"height", 1080); + cJSON_AddFalseToObject (fmt,"interlace"); + cJSON_AddNumberToObject(fmt,"frame rate", 24); + +Hopefully we can agree that's not a lot of code? There's no overhead, no unnecessary setup. +Look at test.c for a bunch of nice examples, mostly all ripped off the json.org site, and +a few from elsewhere. + +What about manual mode? First up you need some detail. +Let's cover how the cJSON objects represent the JSON data. +cJSON doesn't distinguish arrays from objects in handling; just type. +Each cJSON has, potentially, a child, siblings, value, a name. + +The root object has: Object Type and a Child +The Child has name "name", with value "Jack ("Bee") Nimble", and a sibling: +Sibling has type Object, name "format", and a child. +That child has type String, name "type", value "rect", and a sibling: +Sibling has type Number, name "width", value 1920, and a sibling: +Sibling has type Number, name "height", value 1080, and a sibling: +Sibling hs type False, name "interlace", and a sibling: +Sibling has type Number, name "frame rate", value 24 + +Here's the structure: +typedef struct cJSON { + struct cJSON *next,*prev; + struct cJSON *child; + + int type; + + char *valuestring; + int valueint; + double valuedouble; + + char *string; +} cJSON; + +By default all values are 0 unless set by virtue of being meaningful. + +next/prev is a doubly linked list of siblings. next takes you to your sibling, +prev takes you back from your sibling to you. +Only objects and arrays have a "child", and it's the head of the doubly linked list. +A "child" entry will have prev==0, but next potentially points on. The last sibling has next=0. +The type expresses Null/True/False/Number/String/Array/Object, all of which are #defined in +cJSON.h + +A Number has valueint and valuedouble. If you're expecting an int, read valueint, if not read +valuedouble. + +Any entry which is in the linked list which is the child of an object will have a "string" +which is the "name" of the entry. When I said "name" in the above example, that's "string". +"string" is the JSON name for the 'variable name' if you will. + +Now you can trivially walk the lists, recursively, and parse as you please. +You can invoke cJSON_Parse to get cJSON to parse for you, and then you can take +the root object, and traverse the structure (which is, formally, an N-tree), +and tokenise as you please. If you wanted to build a callback style parser, this is how +you'd do it (just an example, since these things are very specific): + +void parse_and_callback(cJSON *item,const char *prefix) +{ + while (item) + { + char *newprefix=malloc(strlen(prefix)+strlen(item->name)+2); + sprintf(newprefix,"%s/%s",prefix,item->name); + int dorecurse=callback(newprefix, item->type, item); + if (item->child && dorecurse) parse_and_callback(item->child,newprefix); + item=item->next; + free(newprefix); + } +} + +The prefix process will build you a separated list, to simplify your callback handling. +The 'dorecurse' flag would let the callback decide to handle sub-arrays on it's own, or +let you invoke it per-item. For the item above, your callback might look like this: + +int callback(const char *name,int type,cJSON *item) +{ + if (!strcmp(name,"name")) { /* populate name */ } + else if (!strcmp(name,"format/type") { /* handle "rect" */ } + else if (!strcmp(name,"format/width") { /* 800 */ } + else if (!strcmp(name,"format/height") { /* 600 */ } + else if (!strcmp(name,"format/interlace") { /* false */ } + else if (!strcmp(name,"format/frame rate") { /* 24 */ } + return 1; +} + +Alternatively, you might like to parse iteratively. +You'd use: + +void parse_object(cJSON *item) +{ + int i; for (i=0;i<cJSON_GetArraySize(item);i++) + { + cJSON *subitem=cJSON_GetArrayItem(item,i); + // handle subitem. + } +} + +Or, for PROPER manual mode: + +void parse_object(cJSON *item) +{ + cJSON *subitem=item->child; + while (subitem) + { + // handle subitem + if (subitem->child) parse_object(subitem->child); + + subitem=subitem->next; + } +} + +Of course, this should look familiar, since this is just a stripped-down version +of the callback-parser. + +This should cover most uses you'll find for parsing. The rest should be possible +to infer.. and if in doubt, read the source! There's not a lot of it! ;) + + +In terms of constructing JSON data, the example code above is the right way to do it. +You can, of course, hand your sub-objects to other functions to populate. +Also, if you find a use for it, you can manually build the objects. +For instance, suppose you wanted to build an array of objects? + +cJSON *objects[24]; + +cJSON *Create_array_of_anything(cJSON **items,int num) +{ + int i;cJSON *prev, *root=cJSON_CreateArray(); + for (i=0;i<24;i++) + { + if (!i) root->child=objects[i]; + else prev->next=objects[i], objects[i]->prev=prev; + prev=objects[i]; + } + return root; +} + +and simply: Create_array_of_anything(objects,24); + +cJSON doesn't make any assumptions about what order you create things in. +You can attach the objects, as above, and later add children to each +of those objects. + +As soon as you call cJSON_Print, it renders the structure to text. + + + +The test.c code shows how to handle a bunch of typical cases. If you uncomment +the code, it'll load, parse and print a bunch of test files, also from json.org, +which are more complex than I'd care to try and stash into a const char array[]. + + +Enjoy cJSON! + + +- Dave Gamble, Aug 2009 diff --git a/plugins/omelasticsearch/cJSON/cjson.c b/plugins/omelasticsearch/cJSON/cjson.c new file mode 100644 index 00000000..99a831e9 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/cjson.c @@ -0,0 +1,514 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +/* cJSON */ +/* JSON parser in C. */ + +#include <string.h> +#include <stdio.h> +#include <math.h> +#include <stdlib.h> +#include <float.h> +#include <limits.h> +#include <ctype.h> +#include "cjson.h" + +static const char *ep; + +const char *cJSON_GetErrorPtr() {return ep;} + +static int cJSON_strcasecmp(const char *s1,const char *s2) +{ + if (!s1) return (s1==s2)?0:1;if (!s2) return 1; + for(; tolower(*s1) == tolower(*s2); ++s1, ++s2) if(*s1 == 0) return 0; + return tolower(*(const unsigned char *)s1) - tolower(*(const unsigned char *)s2); +} + +static void *(*cJSON_malloc)(size_t sz) = malloc; +static void (*cJSON_free)(void *ptr) = free; + +static char* cJSON_strdup(const char* str) +{ + size_t len; + char* copy; + + len = strlen(str) + 1; + if (!(copy = (char*)cJSON_malloc(len))) return 0; + memcpy(copy,str,len); + return copy; +} + +void cJSON_InitHooks(cJSON_Hooks* hooks) +{ + if (!hooks) { /* Reset hooks */ + cJSON_malloc = malloc; + cJSON_free = free; + return; + } + + cJSON_malloc = (hooks->malloc_fn)?hooks->malloc_fn:malloc; + cJSON_free = (hooks->free_fn)?hooks->free_fn:free; +} + +/* Internal constructor. */ +static cJSON *cJSON_New_Item() +{ + cJSON* node = (cJSON*)cJSON_malloc(sizeof(cJSON)); + if (node) memset(node,0,sizeof(cJSON)); + return node; +} + +/* Delete a cJSON structure. */ +void cJSON_Delete(cJSON *c) +{ + cJSON *next; + while (c) + { + next=c->next; + if (!(c->type&cJSON_IsReference) && c->child) cJSON_Delete(c->child); + if (!(c->type&cJSON_IsReference) && c->valuestring) cJSON_free(c->valuestring); + if (c->string) cJSON_free(c->string); + cJSON_free(c); + c=next; + } +} + +/* Parse the input text to generate a number, and populate the result into item. */ +static const char *parse_number(cJSON *item,const char *num) +{ + double n=0,sign=1,scale=0;int subscale=0,signsubscale=1; + + /* Could use sscanf for this? */ + if (*num=='-') sign=-1,num++; /* Has sign? */ + if (*num=='0') num++; /* is zero */ + if (*num>='1' && *num<='9') do n=(n*10.0)+(*num++ -'0'); while (*num>='0' && *num<='9'); /* Number? */ + if (*num=='.' && num[1]>='0' && num[1]<='9') {num++; do n=(n*10.0)+(*num++ -'0'),scale--; while (*num>='0' && *num<='9');} /* Fractional part? */ + if (*num=='e' || *num=='E') /* Exponent? */ + { num++;if (*num=='+') num++; else if (*num=='-') signsubscale=-1,num++; /* With sign? */ + while (*num>='0' && *num<='9') subscale=(subscale*10)+(*num++ - '0'); /* Number? */ + } + + n=sign*n*pow(10.0,(scale+subscale*signsubscale)); /* number = +/- number.fraction * 10^+/- exponent */ + + item->valuedouble=n; + item->valueint=(int)n; + item->type=cJSON_Number; + return num; +} + +/* Render the number nicely from the given item into a string. */ +char *cJSON_print_number(cJSON *item) +{ + char *str; + double d=item->valuedouble; + if (fabs(((double)item->valueint)-d)<=DBL_EPSILON && d<=INT_MAX && d>=INT_MIN) + { + str=(char*)cJSON_malloc(21); /* 2^64+1 can be represented in 21 chars. */ + if (str) sprintf(str,"%d",item->valueint); + } + else + { + str=(char*)cJSON_malloc(64); /* This is a nice tradeoff. */ + if (str) + { + if (fabs(floor(d)-d)<=DBL_EPSILON) sprintf(str,"%.0f",d); + else if (fabs(d)<1.0e-6 || fabs(d)>1.0e9) sprintf(str,"%e",d); + else sprintf(str,"%f",d); + } + } + return str; +} + +/* Parse the input text into an unescaped cstring, and populate item. */ +static const unsigned char firstByteMark[7] = { 0x00, 0x00, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC }; +static const char *parse_string(cJSON *item,const char *str) +{ + const char *ptr=str+1;char *ptr2;char *out;int len=0;unsigned uc,uc2; + if (*str!='\"') {ep=str;return 0;} /* not a string! */ + + while (*ptr!='\"' && *ptr && ++len) if (*ptr++ == '\\') ptr++; /* Skip escaped quotes. */ + + out=(char*)cJSON_malloc(len+1); /* This is how long we need for the string, roughly. */ + if (!out) return 0; + + ptr=str+1;ptr2=out; + while (*ptr!='\"' && *ptr) + { + if (*ptr!='\\') *ptr2++=*ptr++; + else + { + ptr++; + switch (*ptr) + { + case 'b': *ptr2++='\b'; break; + case 'f': *ptr2++='\f'; break; + case 'n': *ptr2++='\n'; break; + case 'r': *ptr2++='\r'; break; + case 't': *ptr2++='\t'; break; + case 'u': /* transcode utf16 to utf8. */ + sscanf(ptr+1,"%4x",&uc);ptr+=4; /* get the unicode char. */ + + if ((uc>=0xDC00 && uc<=0xDFFF) || uc==0) break; // check for invalid. + + if (uc>=0xD800 && uc<=0xDBFF) // UTF16 surrogate pairs. + { + if (ptr[1]!='\\' || ptr[2]!='u') break; // missing second-half of surrogate. + sscanf(ptr+3,"%4x",&uc2);ptr+=6; + if (uc2<0xDC00 || uc2>0xDFFF) break; // invalid second-half of surrogate. + uc=0x10000 | ((uc&0x3FF)<<10) | (uc2&0x3FF); + } + + len=4;if (uc<0x80) len=1;else if (uc<0x800) len=2;else if (uc<0x10000) len=3; ptr2+=len; + + switch (len) { + case 4: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 3: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 2: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 1: *--ptr2 =(uc | firstByteMark[len]); + } + ptr2+=len; + break; + default: *ptr2++=*ptr; break; + } + ptr++; + } + } + *ptr2=0; + if (*ptr=='\"') ptr++; + item->valuestring=out; + item->type=cJSON_String; + return ptr; +} + +/* Render the cstring provided to an escaped version that can be printed. */ +static char *print_string_ptr(const char *str) +{ + const char *ptr;char *ptr2,*out;int len=0;unsigned char token; + + if (!str) return cJSON_strdup(""); + ptr=str;while ((token=*ptr) && ++len) {if (strchr("\"\\\b\f\n\r\t",token)) len++; else if (token<32) len+=5;ptr++;} + + out=(char*)cJSON_malloc(len+3); + if (!out) return 0; + + ptr2=out;ptr=str; + *ptr2++='\"'; + while (*ptr) + { + if ((unsigned char)*ptr>31 && *ptr!='\"' && *ptr!='\\') *ptr2++=*ptr++; + else + { + *ptr2++='\\'; + switch (token=*ptr++) + { + case '\\': *ptr2++='\\'; break; + case '\"': *ptr2++='\"'; break; + case '\b': *ptr2++='b'; break; + case '\f': *ptr2++='f'; break; + case '\n': *ptr2++='n'; break; + case '\r': *ptr2++='r'; break; + case '\t': *ptr2++='t'; break; + default: sprintf(ptr2,"u%04x",token);ptr2+=5; break; /* escape and print */ + } + } + } + *ptr2++='\"';*ptr2++=0; + return out; +} +/* Invote print_string_ptr (which is useful) on an item. */ +static char *print_string(cJSON *item) {return print_string_ptr(item->valuestring);} + +/* Predeclare these prototypes. */ +static const char *parse_value(cJSON *item,const char *value); +static char *print_value(cJSON *item,int depth,int fmt); +static const char *parse_array(cJSON *item,const char *value); +static char *print_array(cJSON *item,int depth,int fmt); +static const char *parse_object(cJSON *item,const char *value); +static char *print_object(cJSON *item,int depth,int fmt); + +/* Utility to jump whitespace and cr/lf */ +static const char *skip(const char *in) {while (in && *in && (unsigned char)*in<=32) in++; return in;} + +/* Parse an object - create a new root, and populate. */ +cJSON *cJSON_Parse(const char *value) +{ + cJSON *c=cJSON_New_Item(); + ep=0; + if (!c) return 0; /* memory fail */ + + if (!parse_value(c,skip(value))) {cJSON_Delete(c);return 0;} + return c; +} + +/* Render a cJSON item/entity/structure to text. */ +char *cJSON_Print(cJSON *item) {return print_value(item,0,1);} +char *cJSON_PrintUnformatted(cJSON *item) {return print_value(item,0,0);} + +/* Parser core - when encountering text, process appropriately. */ +static const char *parse_value(cJSON *item,const char *value) +{ + if (!value) return 0; /* Fail on null. */ + if (!strncmp(value,"null",4)) { item->type=cJSON_NULL; return value+4; } + if (!strncmp(value,"false",5)) { item->type=cJSON_False; return value+5; } + if (!strncmp(value,"true",4)) { item->type=cJSON_True; item->valueint=1; return value+4; } + if (*value=='\"') { return parse_string(item,value); } + if (*value=='-' || (*value>='0' && *value<='9')) { return parse_number(item,value); } + if (*value=='[') { return parse_array(item,value); } + if (*value=='{') { return parse_object(item,value); } + + ep=value;return 0; /* failure. */ +} + +/* Render a value to text. */ +static char *print_value(cJSON *item,int depth,int fmt) +{ + char *out=0; + if (!item) return 0; + switch ((item->type)&255) + { + case cJSON_NULL: out=cJSON_strdup("null"); break; + case cJSON_False: out=cJSON_strdup("false");break; + case cJSON_True: out=cJSON_strdup("true"); break; + case cJSON_Number: out=cJSON_print_number(item);break; + case cJSON_String: out=print_string(item);break; + case cJSON_Array: out=print_array(item,depth,fmt);break; + case cJSON_Object: out=print_object(item,depth,fmt);break; + } + return out; +} + +/* Build an array from input text. */ +static const char *parse_array(cJSON *item,const char *value) +{ + cJSON *child; + if (*value!='[') {ep=value;return 0;} /* not an array! */ + + item->type=cJSON_Array; + value=skip(value+1); + if (*value==']') return value+1; /* empty array. */ + + item->child=child=cJSON_New_Item(); + if (!item->child) return 0; /* memory fail */ + value=skip(parse_value(child,skip(value))); /* skip any spacing, get the value. */ + if (!value) return 0; + + while (*value==',') + { + cJSON *new_item; + if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ + child->next=new_item;new_item->prev=child;child=new_item; + value=skip(parse_value(child,skip(value+1))); + if (!value) return 0; /* memory fail */ + } + + if (*value==']') return value+1; /* end of array */ + ep=value;return 0; /* malformed. */ +} + +/* Render an array to text */ +static char *print_array(cJSON *item,int depth,int fmt) +{ + char **entries; + char *out=0,*ptr,*ret;int len=5; + cJSON *child=item->child; + int numentries=0,i=0,fail=0; + + /* How many entries in the array? */ + while (child) numentries++,child=child->next; + /* Allocate an array to hold the values for each */ + entries=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!entries) return 0; + memset(entries,0,numentries*sizeof(char*)); + /* Retrieve all the results: */ + child=item->child; + while (child && !fail) + { + ret=print_value(child,depth+1,fmt); + entries[i++]=ret; + if (ret) len+=strlen(ret)+2+(fmt?1:0); else fail=1; + child=child->next; + } + + /* If we didn't fail, try to malloc the output string */ + if (!fail) out=(char*)cJSON_malloc(len); + /* If that fails, we fail. */ + if (!out) fail=1; + + /* Handle failure. */ + if (fail) + { + for (i=0;i<numentries;i++) if (entries[i]) cJSON_free(entries[i]); + cJSON_free(entries); + return 0; + } + + /* Compose the output array. */ + *out='['; + ptr=out+1;*ptr=0; + for (i=0;i<numentries;i++) + { + strcpy(ptr,entries[i]);ptr+=strlen(entries[i]); + if (i!=numentries-1) {*ptr++=',';if(fmt)*ptr++=' ';*ptr=0;} + cJSON_free(entries[i]); + } + cJSON_free(entries); + *ptr++=']';*ptr++=0; + return out; +} + +/* Build an object from the text. */ +static const char *parse_object(cJSON *item,const char *value) +{ + cJSON *child; + if (*value!='{') {ep=value;return 0;} /* not an object! */ + + item->type=cJSON_Object; + value=skip(value+1); + if (*value=='}') return value+1; /* empty array. */ + + item->child=child=cJSON_New_Item(); + if (!item->child) return 0; + value=skip(parse_string(child,skip(value))); + if (!value) return 0; + child->string=child->valuestring;child->valuestring=0; + if (*value!=':') {ep=value;return 0;} /* fail! */ + value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ + if (!value) return 0; + + while (*value==',') + { + cJSON *new_item; + if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ + child->next=new_item;new_item->prev=child;child=new_item; + value=skip(parse_string(child,skip(value+1))); + if (!value) return 0; + child->string=child->valuestring;child->valuestring=0; + if (*value!=':') {ep=value;return 0;} /* fail! */ + value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ + if (!value) return 0; + } + + if (*value=='}') return value+1; /* end of array */ + ep=value;return 0; /* malformed. */ +} + +/* Render an object to text. */ +static char *print_object(cJSON *item,int depth,int fmt) +{ + char **entries=0,**names=0; + char *out=0,*ptr,*ret,*str;int len=7,i=0,j; + cJSON *child=item->child; + int numentries=0,fail=0; + /* Count the number of entries. */ + while (child) numentries++,child=child->next; + /* Allocate space for the names and the objects */ + entries=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!entries) return 0; + names=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!names) {cJSON_free(entries);return 0;} + memset(entries,0,sizeof(char*)*numentries); + memset(names,0,sizeof(char*)*numentries); + + /* Collect all the results into our arrays: */ + child=item->child;depth++;if (fmt) len+=depth; + while (child) + { + names[i]=str=print_string_ptr(child->string); + entries[i++]=ret=print_value(child,depth,fmt); + if (str && ret) len+=strlen(ret)+strlen(str)+2+(fmt?2+depth:0); else fail=1; + child=child->next; + } + + /* Try to allocate the output string */ + if (!fail) out=(char*)cJSON_malloc(len); + if (!out) fail=1; + + /* Handle failure */ + if (fail) + { + for (i=0;i<numentries;i++) {if (names[i]) cJSON_free(names[i]);if (entries[i]) cJSON_free(entries[i]);} + cJSON_free(names);cJSON_free(entries); + return 0; + } + + /* Compose the output: */ + *out='{';ptr=out+1;if (fmt)*ptr++='\n';*ptr=0; + for (i=0;i<numentries;i++) + { + if (fmt) for (j=0;j<depth;j++) *ptr++='\t'; + strcpy(ptr,names[i]);ptr+=strlen(names[i]); + *ptr++=':';if (fmt) *ptr++='\t'; + strcpy(ptr,entries[i]);ptr+=strlen(entries[i]); + if (i!=numentries-1) *ptr++=','; + if (fmt) *ptr++='\n';*ptr=0; + cJSON_free(names[i]);cJSON_free(entries[i]); + } + + cJSON_free(names);cJSON_free(entries); + if (fmt) for (i=0;i<depth-1;i++) *ptr++='\t'; + *ptr++='}';*ptr++=0; + return out; +} + +/* Get Array size/item / object item. */ +int cJSON_GetArraySize(cJSON *array) {cJSON *c=array->child;int i=0;while(c)i++,c=c->next;return i;} +cJSON *cJSON_GetArrayItem(cJSON *array,int item) {cJSON *c=array->child; while (c && item>0) item--,c=c->next; return c;} +cJSON *cJSON_GetObjectItem(cJSON *object,const char *string) {cJSON *c=object->child; while (c && cJSON_strcasecmp(c->string,string)) c=c->next; return c;} + +/* Utility for array list handling. */ +static void suffix_object(cJSON *prev,cJSON *item) {prev->next=item;item->prev=prev;} +/* Utility for handling references. */ +static cJSON *create_reference(cJSON *item) {cJSON *ref=cJSON_New_Item();if (!ref) return 0;memcpy(ref,item,sizeof(cJSON));ref->string=0;ref->type|=cJSON_IsReference;ref->next=ref->prev=0;return ref;} + +/* Add item to array/object. */ +void cJSON_AddItemToArray(cJSON *array, cJSON *item) {cJSON *c=array->child;if (!item) return; if (!c) {array->child=item;} else {while (c && c->next) c=c->next; suffix_object(c,item);}} +void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item) {if (!item) return; if (item->string) cJSON_free(item->string);item->string=cJSON_strdup(string);cJSON_AddItemToArray(object,item);} +void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item) {cJSON_AddItemToArray(array,create_reference(item));} +void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item) {cJSON_AddItemToObject(object,string,create_reference(item));} + +cJSON *cJSON_DetachItemFromArray(cJSON *array,int which) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return 0; + if (c->prev) c->prev->next=c->next;if (c->next) c->next->prev=c->prev;if (c==array->child) array->child=c->next;c->prev=c->next=0;return c;} +void cJSON_DeleteItemFromArray(cJSON *array,int which) {cJSON_Delete(cJSON_DetachItemFromArray(array,which));} +cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string) {int i=0;cJSON *c=object->child;while (c && cJSON_strcasecmp(c->string,string)) i++,c=c->next;if (c) return cJSON_DetachItemFromArray(object,i);return 0;} +void cJSON_DeleteItemFromObject(cJSON *object,const char *string) {cJSON_Delete(cJSON_DetachItemFromObject(object,string));} + +/* Replace array/object items with new ones. */ +void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return; + newitem->next=c->next;newitem->prev=c->prev;if (newitem->next) newitem->next->prev=newitem; + if (c==array->child) array->child=newitem; else newitem->prev->next=newitem;c->next=c->prev=0;cJSON_Delete(c);} +void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem){int i=0;cJSON *c=object->child;while(c && cJSON_strcasecmp(c->string,string))i++,c=c->next;if(c){newitem->string=cJSON_strdup(string);cJSON_ReplaceItemInArray(object,i,newitem);}} + +/* Create basic types: */ +cJSON *cJSON_CreateNull() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_NULL;return item;} +cJSON *cJSON_CreateTrue() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_True;return item;} +cJSON *cJSON_CreateFalse() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_False;return item;} +cJSON *cJSON_CreateBool(int b) {cJSON *item=cJSON_New_Item();if(item)item->type=b?cJSON_True:cJSON_False;return item;} +cJSON *cJSON_CreateNumber(double num) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_Number;item->valuedouble=num;item->valueint=(int)num;}return item;} +cJSON *cJSON_CreateString(const char *string) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_String;item->valuestring=cJSON_strdup(string);}return item;} +cJSON *cJSON_CreateArray() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Array;return item;} +cJSON *cJSON_CreateObject() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Object;return item;} + +/* Create Arrays: */ +cJSON *cJSON_CreateIntArray(int *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateFloatArray(float *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateDoubleArray(double *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateStringArray(const char **strings,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateString(strings[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} diff --git a/plugins/omelasticsearch/cJSON/cjson.h b/plugins/omelasticsearch/cJSON/cjson.h new file mode 100644 index 00000000..a621720c --- /dev/null +++ b/plugins/omelasticsearch/cJSON/cjson.h @@ -0,0 +1,130 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +#ifndef cJSON__h +#define cJSON__h + +#ifdef __cplusplus +extern "C" +{ +#endif + +/* cJSON Types: */ +#define cJSON_False 0 +#define cJSON_True 1 +#define cJSON_NULL 2 +#define cJSON_Number 3 +#define cJSON_String 4 +#define cJSON_Array 5 +#define cJSON_Object 6 + +#define cJSON_IsReference 256 + +/* The cJSON structure: */ +typedef struct cJSON { + struct cJSON *next,*prev; /* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */ + struct cJSON *child; /* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */ + + int type; /* The type of the item, as above. */ + + char *valuestring; /* The item's string, if type==cJSON_String */ + int valueint; /* The item's number, if type==cJSON_Number */ + double valuedouble; /* The item's number, if type==cJSON_Number */ + + char *string; /* The item's name string, if this item is the child of, or is in the list of subitems of an object. */ +} cJSON; + +typedef struct cJSON_Hooks { + void *(*malloc_fn)(size_t sz); + void (*free_fn)(void *ptr); +} cJSON_Hooks; + +/* Supply malloc, realloc and free functions to cJSON */ +extern void cJSON_InitHooks(cJSON_Hooks* hooks); + + +/* Supply a block of JSON, and this returns a cJSON object you can interrogate. Call cJSON_Delete when finished. */ +extern cJSON *cJSON_Parse(const char *value); +/* Render a cJSON entity to text for transfer/storage. Free the char* when finished. */ +extern char *cJSON_Print(cJSON *item); +/* Render a cJSON entity to text for transfer/storage without any formatting. Free the char* when finished. */ +extern char *cJSON_PrintUnformatted(cJSON *item); +/* Delete a cJSON entity and all subentities. */ +extern void cJSON_Delete(cJSON *c); + +/* Returns the number of items in an array (or object). */ +extern int cJSON_GetArraySize(cJSON *array); +/* Retrieve item number "item" from array "array". Returns NULL if unsuccessful. */ +extern cJSON *cJSON_GetArrayItem(cJSON *array,int item); +/* Get item "string" from object. Case insensitive. */ +extern cJSON *cJSON_GetObjectItem(cJSON *object,const char *string); + +/* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */ +extern const char *cJSON_GetErrorPtr(); + +/* These calls create a cJSON item of the appropriate type. */ +extern cJSON *cJSON_CreateNull(); +extern cJSON *cJSON_CreateTrue(); +extern cJSON *cJSON_CreateFalse(); +extern cJSON *cJSON_CreateBool(int b); +extern cJSON *cJSON_CreateNumber(double num); +extern cJSON *cJSON_CreateString(const char *string); +extern cJSON *cJSON_CreateArray(); +extern cJSON *cJSON_CreateObject(); + +/* These utilities create an Array of count items. */ +extern cJSON *cJSON_CreateIntArray(int *numbers,int count); +extern cJSON *cJSON_CreateFloatArray(float *numbers,int count); +extern cJSON *cJSON_CreateDoubleArray(double *numbers,int count); +extern cJSON *cJSON_CreateStringArray(const char **strings,int count); + +/* Append item to the specified array/object. */ +extern void cJSON_AddItemToArray(cJSON *array, cJSON *item); +extern void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item); +/* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */ +extern void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item); +extern void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item); + +/* Remove/Detatch items from Arrays/Objects. */ +extern cJSON *cJSON_DetachItemFromArray(cJSON *array,int which); +extern void cJSON_DeleteItemFromArray(cJSON *array,int which); +extern cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string); +extern void cJSON_DeleteItemFromObject(cJSON *object,const char *string); + +/* Update array items. */ +extern void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem); +extern void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem); + +/* rger: added helpers */ + +char *cJSON_print_number(cJSON *item); +#define cJSON_AddNullToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateNull()) +#define cJSON_AddTrueToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateTrue()) +#define cJSON_AddFalseToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateFalse()) +#define cJSON_AddNumberToObject(object,name,n) cJSON_AddItemToObject(object, name, cJSON_CreateNumber(n)) +#define cJSON_AddStringToObject(object,name,s) cJSON_AddItemToObject(object, name, cJSON_CreateString(s)) + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/plugins/omelasticsearch/cJSON/test.c b/plugins/omelasticsearch/cJSON/test.c new file mode 100644 index 00000000..2cab632a --- /dev/null +++ b/plugins/omelasticsearch/cJSON/test.c @@ -0,0 +1,156 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +#include <stdio.h> +#include <stdlib.h> +#include "cJSON.h" + +/* Parse text to JSON, then render back to text, and print! */ +void doit(char *text) +{ + char *out;cJSON *json; + + json=cJSON_Parse(text); + if (!json) {printf("Error before: [%s]\n",cJSON_GetErrorPtr());} + else + { + out=cJSON_Print(json); + cJSON_Delete(json); + printf("%s\n",out); + free(out); + } +} + +/* Read a file, parse, render back, etc. */ +void dofile(char *filename) +{ + FILE *f=fopen(filename,"rb");fseek(f,0,SEEK_END);long len=ftell(f);fseek(f,0,SEEK_SET); + char *data=malloc(len+1);fread(data,1,len,f);fclose(f); + doit(data); + free(data); +} + +/* Used by some code below as an example datatype. */ +struct record {const char *precision;double lat,lon;const char *address,*city,*state,*zip,*country; }; + +/* Create a bunch of objects as demonstration. */ +void create_objects() +{ + cJSON *root,*fmt,*img,*thm,*fld;char *out;int i; /* declare a few. */ + + /* Here we construct some JSON standards, from the JSON site. */ + + /* Our "Video" datatype: */ + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "name", cJSON_CreateString("Jack (\"Bee\") Nimble")); + cJSON_AddItemToObject(root, "format", fmt=cJSON_CreateObject()); + cJSON_AddStringToObject(fmt,"type", "rect"); + cJSON_AddNumberToObject(fmt,"width", 1920); + cJSON_AddNumberToObject(fmt,"height", 1080); + cJSON_AddFalseToObject (fmt,"interlace"); + cJSON_AddNumberToObject(fmt,"frame rate", 24); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); /* Print to text, Delete the cJSON, print it, release the string. + + /* Our "days of the week" array: */ + const char *strings[7]={"Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"}; + root=cJSON_CreateStringArray(strings,7); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + /* Our matrix: */ + int numbers[3][3]={{0,-1,0},{1,0,0},{0,0,1}}; + root=cJSON_CreateArray(); + for (i=0;i<3;i++) cJSON_AddItemToArray(root,cJSON_CreateIntArray(numbers[i],3)); + +/* cJSON_ReplaceItemInArray(root,1,cJSON_CreateString("Replacement")); */ + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + + /* Our "gallery" item: */ + int ids[4]={116,943,234,38793}; + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "Image", img=cJSON_CreateObject()); + cJSON_AddNumberToObject(img,"Width",800); + cJSON_AddNumberToObject(img,"Height",600); + cJSON_AddStringToObject(img,"Title","View from 15th Floor"); + cJSON_AddItemToObject(img, "Thumbnail", thm=cJSON_CreateObject()); + cJSON_AddStringToObject(thm, "Url", "http:/*www.example.com/image/481989943"); + cJSON_AddNumberToObject(thm,"Height",125); + cJSON_AddStringToObject(thm,"Width","100"); + cJSON_AddItemToObject(img,"IDs", cJSON_CreateIntArray(ids,4)); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + /* Our array of "records": */ + struct record fields[2]={ + {"zip",37.7668,-1.223959e+2,"","SAN FRANCISCO","CA","94107","US"}, + {"zip",37.371991,-1.22026e+2,"","SUNNYVALE","CA","94085","US"}}; + + root=cJSON_CreateArray(); + for (i=0;i<2;i++) + { + cJSON_AddItemToArray(root,fld=cJSON_CreateObject()); + cJSON_AddStringToObject(fld, "precision", fields[i].precision); + cJSON_AddNumberToObject(fld, "Latitude", fields[i].lat); + cJSON_AddNumberToObject(fld, "Longitude", fields[i].lon); + cJSON_AddStringToObject(fld, "Address", fields[i].address); + cJSON_AddStringToObject(fld, "City", fields[i].city); + cJSON_AddStringToObject(fld, "State", fields[i].state); + cJSON_AddStringToObject(fld, "Zip", fields[i].zip); + cJSON_AddStringToObject(fld, "Country", fields[i].country); + } + +/* cJSON_ReplaceItemInObject(cJSON_GetArrayItem(root,1),"City",cJSON_CreateIntArray(ids,4)); */ + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + +} + +int main (int argc, const char * argv[]) { + /* a bunch of json: */ + char text1[]="{\n\"name\": \"Jack (\\\"Bee\\\") Nimble\", \n\"format\": {\"type\": \"rect\", \n\"width\": 1920, \n\"height\": 1080, \n\"interlace\": false,\"frame rate\": 24\n}\n}"; + char text2[]="[\"Sunday\", \"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\", \"Friday\", \"Saturday\"]"; + char text3[]="[\n [0, -1, 0],\n [1, 0, 0],\n [0, 0, 1]\n ]\n"; + char text4[]="{\n \"Image\": {\n \"Width\": 800,\n \"Height\": 600,\n \"Title\": \"View from 15th Floor\",\n \"Thumbnail\": {\n \"Url\": \"http:/*www.example.com/image/481989943\",\n \"Height\": 125,\n \"Width\": \"100\"\n },\n \"IDs\": [116, 943, 234, 38793]\n }\n }"; + char text5[]="[\n {\n \"precision\": \"zip\",\n \"Latitude\": 37.7668,\n \"Longitude\": -122.3959,\n \"Address\": \"\",\n \"City\": \"SAN FRANCISCO\",\n \"State\": \"CA\",\n \"Zip\": \"94107\",\n \"Country\": \"US\"\n },\n {\n \"precision\": \"zip\",\n \"Latitude\": 37.371991,\n \"Longitude\": -122.026020,\n \"Address\": \"\",\n \"City\": \"SUNNYVALE\",\n \"State\": \"CA\",\n \"Zip\": \"94085\",\n \"Country\": \"US\"\n }\n ]"; + + /* Process each json textblock by parsing, then rebuilding: */ + doit(text1); + doit(text2); + doit(text3); + doit(text4); + doit(text5); + + /* Parse standard testfiles: +/* dofile("../../tests/test1"); */ +/* dofile("../../tests/test2"); */ +/* dofile("../../tests/test3"); */ +/* dofile("../../tests/test4"); */ +/* dofile("../../tests/test5"); */ + + /* Now some samplecode for building objects concisely: */ + create_objects(); + + return 0; +} diff --git a/plugins/omelasticsearch/cJSON/tests/test1 b/plugins/omelasticsearch/cJSON/tests/test1 new file mode 100644 index 00000000..eacfbf5e --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test1 @@ -0,0 +1,22 @@ +{ + "glossary": { + "title": "example glossary", + "GlossDiv": { + "title": "S", + "GlossList": { + "GlossEntry": { + "ID": "SGML", + "SortAs": "SGML", + "GlossTerm": "Standard Generalized Markup Language", + "Acronym": "SGML", + "Abbrev": "ISO 8879:1986", + "GlossDef": { + "para": "A meta-markup language, used to create markup languages such as DocBook.", + "GlossSeeAlso": ["GML", "XML"] + }, + "GlossSee": "markup" + } + } + } + } +} diff --git a/plugins/omelasticsearch/cJSON/tests/test2 b/plugins/omelasticsearch/cJSON/tests/test2 new file mode 100644 index 00000000..5600991a --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test2 @@ -0,0 +1,11 @@ +{"menu": { + "id": "file", + "value": "File", + "popup": { + "menuitem": [ + {"value": "New", "onclick": "CreateNewDoc()"}, + {"value": "Open", "onclick": "OpenDoc()"}, + {"value": "Close", "onclick": "CloseDoc()"} + ] + } +}} diff --git a/plugins/omelasticsearch/cJSON/tests/test3 b/plugins/omelasticsearch/cJSON/tests/test3 new file mode 100644 index 00000000..5662b377 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test3 @@ -0,0 +1,26 @@ +{"widget": { + "debug": "on", + "window": { + "title": "Sample Konfabulator Widget", + "name": "main_window", + "width": 500, + "height": 500 + }, + "image": { + "src": "Images/Sun.png", + "name": "sun1", + "hOffset": 250, + "vOffset": 250, + "alignment": "center" + }, + "text": { + "data": "Click Here", + "size": 36, + "style": "bold", + "name": "text1", + "hOffset": 250, + "vOffset": 100, + "alignment": "center", + "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;" + } +}}
\ No newline at end of file diff --git a/plugins/omelasticsearch/cJSON/tests/test4 b/plugins/omelasticsearch/cJSON/tests/test4 new file mode 100644 index 00000000..d540b57f --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test4 @@ -0,0 +1,88 @@ +{"web-app": { + "servlet": [ + { + "servlet-name": "cofaxCDS", + "servlet-class": "org.cofax.cds.CDSServlet", + "init-param": { + "configGlossary:installationAt": "Philadelphia, PA", + "configGlossary:adminEmail": "ksm@pobox.com", + "configGlossary:poweredBy": "Cofax", + "configGlossary:poweredByIcon": "/images/cofax.gif", + "configGlossary:staticPath": "/content/static", + "templateProcessorClass": "org.cofax.WysiwygTemplate", + "templateLoaderClass": "org.cofax.FilesTemplateLoader", + "templatePath": "templates", + "templateOverridePath": "", + "defaultListTemplate": "listTemplate.htm", + "defaultFileTemplate": "articleTemplate.htm", + "useJSP": false, + "jspListTemplate": "listTemplate.jsp", + "jspFileTemplate": "articleTemplate.jsp", + "cachePackageTagsTrack": 200, + "cachePackageTagsStore": 200, + "cachePackageTagsRefresh": 60, + "cacheTemplatesTrack": 100, + "cacheTemplatesStore": 50, + "cacheTemplatesRefresh": 15, + "cachePagesTrack": 200, + "cachePagesStore": 100, + "cachePagesRefresh": 10, + "cachePagesDirtyRead": 10, + "searchEngineListTemplate": "forSearchEnginesList.htm", + "searchEngineFileTemplate": "forSearchEngines.htm", + "searchEngineRobotsDb": "WEB-INF/robots.db", + "useDataStore": true, + "dataStoreClass": "org.cofax.SqlDataStore", + "redirectionClass": "org.cofax.SqlRedirection", + "dataStoreName": "cofax", + "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver", + "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon", + "dataStoreUser": "sa", + "dataStorePassword": "dataStoreTestQuery", + "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';", + "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log", + "dataStoreInitConns": 10, + "dataStoreMaxConns": 100, + "dataStoreConnUsageLimit": 100, + "dataStoreLogLevel": "debug", + "maxUrlLength": 500}}, + { + "servlet-name": "cofaxEmail", + "servlet-class": "org.cofax.cds.EmailServlet", + "init-param": { + "mailHost": "mail1", + "mailHostOverride": "mail2"}}, + { + "servlet-name": "cofaxAdmin", + "servlet-class": "org.cofax.cds.AdminServlet"}, + + { + "servlet-name": "fileServlet", + "servlet-class": "org.cofax.cds.FileServlet"}, + { + "servlet-name": "cofaxTools", + "servlet-class": "org.cofax.cms.CofaxToolsServlet", + "init-param": { + "templatePath": "toolstemplates/", + "log": 1, + "logLocation": "/usr/local/tomcat/logs/CofaxTools.log", + "logMaxSize": "", + "dataLog": 1, + "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log", + "dataLogMaxSize": "", + "removePageCache": "/content/admin/remove?cache=pages&id=", + "removeTemplateCache": "/content/admin/remove?cache=templates&id=", + "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder", + "lookInContext": 1, + "adminGroupID": 4, + "betaServer": true}}], + "servlet-mapping": { + "cofaxCDS": "/", + "cofaxEmail": "/cofaxutil/aemail/*", + "cofaxAdmin": "/admin/*", + "fileServlet": "/static/*", + "cofaxTools": "/tools/*"}, + + "taglib": { + "taglib-uri": "cofax.tld", + "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}
\ No newline at end of file diff --git a/plugins/omelasticsearch/cJSON/tests/test5 b/plugins/omelasticsearch/cJSON/tests/test5 new file mode 100644 index 00000000..49980ca2 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test5 @@ -0,0 +1,27 @@ +{"menu": { + "header": "SVG Viewer", + "items": [ + {"id": "Open"}, + {"id": "OpenNew", "label": "Open New"}, + null, + {"id": "ZoomIn", "label": "Zoom In"}, + {"id": "ZoomOut", "label": "Zoom Out"}, + {"id": "OriginalView", "label": "Original View"}, + null, + {"id": "Quality"}, + {"id": "Pause"}, + {"id": "Mute"}, + null, + {"id": "Find", "label": "Find..."}, + {"id": "FindAgain", "label": "Find Again"}, + {"id": "Copy"}, + {"id": "CopyAgain", "label": "Copy Again"}, + {"id": "CopySVG", "label": "Copy SVG"}, + {"id": "ViewSVG", "label": "View SVG"}, + {"id": "ViewSource", "label": "View Source"}, + {"id": "SaveAs", "label": "Save As"}, + null, + {"id": "Help"}, + {"id": "About", "label": "About Adobe CVG Viewer..."} + ] +}} diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c index 50acdf11..f27fe62b 100644 --- a/plugins/omelasticsearch/omelasticsearch.c +++ b/plugins/omelasticsearch/omelasticsearch.c @@ -34,6 +34,10 @@ #include <signal.h> #include <errno.h> #include <time.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include "cJSON/cjson.h" #include "conf.h" #include "syslogd-types.h" #include "srUtils.h" @@ -64,8 +68,10 @@ STATSCOUNTER_DEF(indexSuccess, mutIndexSuccess) */ typedef struct curl_slist HEADER; typedef struct _instanceData { - uchar *server; int port; + int replyLen; + int fdErrFile; /* error file fd or -1 if not open */ + uchar *server; uchar *uid; uchar *pwd; uchar *searchIndex; @@ -73,6 +79,9 @@ typedef struct _instanceData { uchar *parent; uchar *tplName; uchar *timeout; + uchar *restURL; /* last used URL for error reporting */ + uchar *errorFile; + char *reply; sbool dynSrchIdx; sbool dynSrchType; sbool dynParent; @@ -104,6 +113,7 @@ static struct cnfparamdescr actpdescr[] = { { "bulkmode", eCmdHdlrBinary, 0 }, { "asyncrepl", eCmdHdlrBinary, 0 }, { "timeout", eCmdHdlrGetWord, 0 }, + { "errorfile", eCmdHdlrGetWord, 0 }, { "template", eCmdHdlrGetWord, 1 } }; static struct cnfparamblk actpblk = @@ -114,6 +124,8 @@ static struct cnfparamblk actpblk = BEGINcreateInstance CODESTARTcreateInstance + pData->restURL = NULL; + pData->fdErrFile = -1; ENDcreateInstance BEGINisCompatibleWithFeature @@ -132,6 +144,8 @@ CODESTARTfreeInstance curl_easy_cleanup(pData->curlHandle); pData->curlHandle = NULL; } + if(pData->fdErrFile != -1) + close(pData->fdErrFile); free(pData->server); free(pData->uid); free(pData->pwd); @@ -139,6 +153,9 @@ CODESTARTfreeInstance free(pData->searchType); free(pData->parent); free(pData->tplName); + free(pData->timeout); + free(pData->restURL); + free(pData->errorFile); ENDfreeInstance BEGINdbgPrintInstInfo @@ -158,6 +175,8 @@ CODESTARTdbgPrintInstInfo dbgprintf("\tdynamic parent=%d\n", pData->dynParent); dbgprintf("\tasync replication=%d\n", pData->asyncRepl); dbgprintf("\tbulkmode=%d\n", pData->bulkmode); + dbgprintf("\terrorfile='%s'\n", pData->errorFile == NULL ? + (uchar*)"(not configured)" : pData->errorFile); ENDdbgPrintInstInfo @@ -202,12 +221,16 @@ checkConn(instanceData *pData) curl_easy_setopt(curl, CURLOPT_URL, cstr); free(cstr); + pData->reply = NULL; + pData->replyLen = 0; + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); res = curl_easy_perform(curl); if(res != CURLE_OK) { DBGPRINTF("omelasticsearch: checkConn() curl_easy_perform() " "failed: %s\n", curl_easy_strerror(res)); ABORT_FINALIZE(RS_RET_SUSPENDED); } + free(pData->reply); DBGPRINTF("omelasticsearch: checkConn() completed with success\n"); finalize_it: @@ -271,7 +294,6 @@ static rsRetVal setCurlURL(instanceData *pData, uchar **tpls) { char authBuf[1024]; - char *restURL; uchar *searchIndex; uchar *searchType; uchar *parent; @@ -305,16 +327,17 @@ setCurlURL(instanceData *pData, uchar **tpls) if(r == 0) r = es_addBuf(&url, "parent=", sizeof("parent=")-1); if(r == 0) r = es_addBuf(&url, (char*)parent, ustrlen(parent)); } - restURL = es_str2cstr(url, NULL); - curl_easy_setopt(pData->curlHandle, CURLOPT_URL, restURL); + + free(pData->restURL); + pData->restURL = (uchar*)es_str2cstr(url, NULL); + curl_easy_setopt(pData->curlHandle, CURLOPT_URL, pData->restURL); es_deleteStr(url); - DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", restURL); - free(restURL); + DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", pData->restURL); if(pData->uid != NULL) { rLocal = snprintf(authBuf, sizeof(authBuf), "%s:%s", pData->uid, (pData->pwd == NULL) ? "" : (char*)pData->pwd); - if(rLocal != (int) es_strlen(url)) { + if(rLocal < 1) { errmsg.LogError(0, RS_RET_ERR, "omelasticsearch: snprintf failed " "when trying to build auth string (return %d)\n", rLocal); @@ -347,9 +370,6 @@ buildBatch(instanceData *pData, uchar *message, uchar **tpls) # define META_END "\"}}\n" getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent); -dbgprintf("AAA: searchIndex: '%s'\n", searchIndex); -dbgprintf("AAA: searchType: '%s'\n", searchType); -dbgprintf("AAA: parent: '%s'\n", parent); r = es_addBuf(&pData->batch.data, META_STRT, sizeof(META_STRT)-1); if(r == 0) r = es_addBuf(&pData->batch.data, (char*)searchIndex, ustrlen(searchIndex)); @@ -373,22 +393,168 @@ finalize_it: RETiRet; } + +/* write data error request/replies to separate error file + * Note: we open the file but never close it before exit. If it + * needs to be closed, HUP must be sent. + */ +static inline rsRetVal +writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) +{ + char *rendered = NULL; + cJSON *errRoot; + cJSON *req; + cJSON *replyRoot = *pReplyRoot; + size_t toWrite; + ssize_t wrRet; + char errStr[1024]; + DEFiRet; + + if(pData->errorFile == NULL) { + DBGPRINTF("omelasticsearch: no local error logger defined - " + "ignoring ES error information\n"); + FINALIZE; + } + + if(pData->fdErrFile == -1) { + pData->fdErrFile = open((char*)pData->errorFile, + O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, + S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP); + if(pData->fdErrFile == -1) { + rs_strerror_r(errno, errStr, sizeof(errStr)); + DBGPRINTF("omelasticsearch: error opening error file: %s\n", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + } + if((req=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); + cJSON_AddItemToObject(req, "url", cJSON_CreateString((char*)pData->restURL)); + cJSON_AddItemToObject(req, "postdata", cJSON_CreateString((char*)reqmsg)); + + if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); + cJSON_AddItemToObject(errRoot, "request", req); + cJSON_AddItemToObject(errRoot, "reply", replyRoot); + rendered = cJSON_Print(errRoot); + /* we do not do real error-handling on the err file, as this finally complicates + * things way to much. + */ + DBGPRINTF("omelasticsearch: error record: '%s'\n", rendered); + toWrite = strlen(rendered); + wrRet = write(pData->fdErrFile, rendered, toWrite); + if(wrRet != (ssize_t) toWrite) { + DBGPRINTF("omelasticsearch: error %d writing error file, write returns %lld\n", + errno, (long long) wrRet); + } + free(rendered); + cJSON_Delete(errRoot); + *pReplyRoot = NULL; /* tell caller not to delete once again! */ + +finalize_it: + if(rendered != NULL) + free(rendered); + RETiRet; +} + + +static inline rsRetVal +checkResultBulkmode(instanceData *pData, cJSON *root) +{ + int i; + int numitems; + cJSON *items; + cJSON *item; + cJSON *create; + cJSON *ok; + DEFiRet; + + items = cJSON_GetObjectItem(root, "items"); + if(items == NULL || items->type != cJSON_Array) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "bulkmode insert does not return array, reply is: %s\n", + pData->reply); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + numitems = cJSON_GetArraySize(items); +DBGPRINTF("omelasticsearch: %d items in reply\n", numitems); + for(i = 0 ; i < numitems ; ++i) { + item = cJSON_GetArrayItem(items, i); + if(item == NULL) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain reply array item %d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + create = cJSON_GetObjectItem(item, "create"); + if(create == NULL || create->type != cJSON_Object) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain 'create' item for #%d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + ok = cJSON_GetObjectItem(create, "ok"); + if(ok == NULL || ok->type != cJSON_True) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "item %d, prop ok (%p) not ok\n", i, ok); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + } + +finalize_it: + RETiRet; +} + + +static inline rsRetVal +checkResult(instanceData *pData, uchar *reqmsg) +{ + cJSON *root; + cJSON *ok; + DEFiRet; + + root = cJSON_Parse(pData->reply); + if(root == NULL) { + DBGPRINTF("omelasticsearch: could not parse JSON result \n"); + ABORT_FINALIZE(RS_RET_ERR); + } + + if(pData->bulkmode) { + iRet = checkResultBulkmode(pData, root); + } else { + ok = cJSON_GetObjectItem(root, "ok"); + if(ok == NULL || ok->type != cJSON_True) { + iRet = RS_RET_DATAFAIL; + } + } + + /* Note: we ignore errors writing the error file, as we cannot handle + * these in any case. + */ + if(iRet == RS_RET_DATAFAIL) { + writeDataError(pData, &root, reqmsg); + iRet = RS_RET_OK; /* we have handled the problem! */ + } + +finalize_it: + if(root != NULL) + cJSON_Delete(root); + RETiRet; +} + + static rsRetVal -curlPost(instanceData *instance, uchar *message, int msglen, uchar **tpls) +curlPost(instanceData *pData, uchar *message, int msglen, uchar **tpls) { CURLcode code; - CURL *curl = instance->curlHandle; + CURL *curl = pData->curlHandle; DEFiRet; - if(instance->dynSrchIdx || instance->dynSrchType || instance->dynParent) - CHKiRet(setCurlURL(instance, tpls)); + pData->reply = NULL; + pData->replyLen = 0; + + if(pData->dynSrchIdx || pData->dynSrchType || pData->dynParent) + CHKiRet(setCurlURL(pData, tpls)); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, (char *)message); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, (char *)message); curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, msglen); -dbgprintf("omelasticsearch: do curl_easy_perform()\n"); code = curl_easy_perform(curl); -DBGPRINTF("omelasticsearch: curl_easy_perform() returned %lld\n", (long long) code); switch (code) { case CURLE_COULDNT_RESOLVE_HOST: case CURLE_COULDNT_RESOLVE_PROXY: @@ -398,12 +564,18 @@ DBGPRINTF("omelasticsearch: curl_easy_perform() returned %lld\n", (long long) co DBGPRINTF("omelasticsearch: we are suspending ourselfs due " "to failure %lld of curl_easy_perform()\n", (long long) code); - return RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); default: STATSCOUNTER_INC(indexSubmit, mutIndexSubmit); - return RS_RET_OK; + break; } + + pData->reply[pData->replyLen] = '\0'; /* byte has been reserved in malloc */ + DBGPRINTF("omelasticsearch: es reply: '%s'\n", pData->reply); + + CHKiRet(checkResult(pData, message)); finalize_it: + free(pData->reply); RETiRet; } @@ -424,7 +596,6 @@ CODESTARTdoAction if(pData->bulkmode) { CHKiRet(buildBatch(pData, ppString[0], ppString)); } else { -dbgprintf("omelasticsearch: doAction calling curlPost\n"); CHKiRet(curlPost(pData, ppString[0], strlen((char*)ppString[0]), ppString)); } @@ -449,35 +620,20 @@ ENDendTransaction size_t curlResult(void *ptr, size_t size, size_t nmemb, void *userdata) { - unsigned int i; char *p = (char *)ptr; - char *jsonData = (char *)userdata; - static char ok[] = "{\"ok\":true,"; - - ASSERT(size == 1); -DBGPRINTF("omelasticsearch request: %s\n", jsonData); -DBGPRINTF("omelasticsearch result: "); -for (i = 0; i < nmemb; i++) - DBGPRINTF("%c", p[i]); -DBGPRINTF("\n"); - - if (size == 1 && - nmemb > sizeof(ok)-1 && - strncmp(p, ok, sizeof(ok)-1) == 0) { - STATSCOUNTER_INC(indexSuccess, mutIndexSuccess); -dbgprintf("omelasticsearch ok\n"); - } else { -dbgprintf("omelasticsearch fail\n"); - STATSCOUNTER_INC(indexFailed, mutIndexFailed); - if (Debug) { - DBGPRINTF("omelasticsearch (fail) request: %s\n", jsonData); - DBGPRINTF("omelasticsearch (fail) result: "); - for (i = 0; i < nmemb; i++) - DBGPRINTF("%c", p[i]); - DBGPRINTF("\n"); - } + instanceData *pData = (instanceData*) userdata; + char *buf; + size_t newlen; + + newlen = pData->replyLen + size*nmemb; + if((buf = realloc(pData->reply, newlen + 1)) == NULL) { + DBGPRINTF("omelasticsearch: realloc failed in curlResult\n"); + return 0; /* abort due to failure */ } - return size * nmemb; + memcpy(buf+pData->replyLen, p, size*nmemb); + pData->replyLen = newlen; + pData->reply = buf; + return size*nmemb; } @@ -533,6 +689,7 @@ setInstParamDefaults(instanceData *pData) pData->asyncRepl = 0; pData->bulkmode = 0; pData->tplName = NULL; + pData->errorFile = NULL; } BEGINnewActInst @@ -552,6 +709,8 @@ CODESTARTnewActInst continue; if(!strcmp(actpblk.descr[i].name, "server")) { pData->server = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "errorfile")) { + pData->errorFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "serverport")) { pData->port = (int) pvals[i].val.d.n, NULL; } else if(!strcmp(actpblk.descr[i].name, "uid")) { @@ -693,6 +852,14 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct +BEGINdoHUP +CODESTARTdoHUP + if(pData->fdErrFile != -1) { + close(pData->fdErrFile); + pData->fdErrFile = -1; + } +ENDdoHUP + BEGINmodExit CODESTARTmodExit @@ -707,6 +874,7 @@ CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_doHUP CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt diff --git a/plugins/omjournal/Makefile.am b/plugins/omjournal/Makefile.am new file mode 100644 index 00000000..4cfbbd96 --- /dev/null +++ b/plugins/omjournal/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = omjournal.la + +omjournal_la_SOURCES = omjournal.c +omjournal_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) $(LIBSYSTEMD_JOURNAL_CFLAGS) +omjournal_la_LDFLAGS = -module -avoid-version +omjournal_la_LIBADD = $(LIBSYSTEMD_JOURNAL_LIBS) + +EXTRA_DIST = diff --git a/plugins/omjournal/omjournal.c b/plugins/omjournal/omjournal.c new file mode 100644 index 00000000..c340287f --- /dev/null +++ b/plugins/omjournal/omjournal.c @@ -0,0 +1,185 @@ +/* omjournal.c + * send messages to the Linux Journal. This is meant to be used + * in cases where journal serves as the whole system log database. + * Note that we may get into a loop if journald re-injects messages + * into the syslog stream and we read that via imuxsock. Thus there + * is an option in imuxsock to ignore messages from ourselves + * (actually from our pid). So there are some module-interdependencies. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include <systemd/sd-journal.h> + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("omjournal") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + + +typedef struct _instanceData { +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +BEGINnewActInst +CODESTARTnewActInst + /* Note: we currently do not have any parameters, so we do not need + * the lst ptr. However, we will most probably need params in the + * future. + */ + DBGPRINTF("newActInst (mmjournal)\n"); + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + /*setInstParamDefaults(pData);*/ +CODE_STD_FINALIZERnewActInst +/* cnfparamvalsDestruct(pvals, &actpblk);*/ +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +BEGINdoAction + msg_t *pMsg; + uchar *tag; + int lenTag; + int sev; + int r; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + MsgGetSeverity(pMsg, &sev); + getTAG(pMsg, &tag, &lenTag); + /* we can use more properties here, but let's see if there + * is some real user interest. We can always add later... + */ + r = sd_journal_send("MESSAGE=%s", getMSG(pMsg), + "PRIORITY=%d", sev, + "SYSLOG_FACILITY=%d", pMsg->iFacility, + "SYSLOG_IDENTIFIER=%s", tag, + NULL); + /* FIXME: think about what to do with errors ;) */ +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":omjournal:", sizeof(":omjournal:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "omjournal supports only v6+ config format, use: " + "action(type=\"omjournal\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("omjournal: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/omlibdbi/omlibdbi.c b/plugins/omlibdbi/omlibdbi.c index 99bcaf9d..390e59d5 100644 --- a/plugins/omlibdbi/omlibdbi.c +++ b/plugins/omlibdbi/omlibdbi.c @@ -81,15 +81,36 @@ typedef struct configSettings_s { uchar *dbName; /* database to use */ } configSettings_t; static configSettings_t cs; +uchar *pszFileDfltTplName; /* name of the default template to use */ + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ + uchar *dbiDrvrDir; /* where do the dbi drivers reside? */ + uchar *tplName; /* default template */ +}; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ +static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ + /* tables for interfacing with the v6 config system */ +/* module-global parameters */ +static struct cnfparamdescr modpdescr[] = { + { "template", eCmdHdlrGetWord, 0 }, + { "driverdirectory", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr + }; /* action (instance) parameters */ static struct cnfparamdescr actpdescr[] = { { "server", eCmdHdlrGetWord, 1 }, { "db", eCmdHdlrGetWord, 1 }, { "uid", eCmdHdlrGetWord, 1 }, { "pwd", eCmdHdlrGetWord, 1 }, - { "driverdirectory", eCmdHdlrGetWord, 0 }, { "driver", eCmdHdlrGetWord, 1 }, { "template", eCmdHdlrGetWord, 0 } }; @@ -99,6 +120,20 @@ static struct cnfparamblk actpblk = actpdescr }; +/* this function gets the default template. It coordinates action between + * old-style and new-style configuration parts. + */ +static inline uchar* +getDfltTpl(void) +{ + if(loadModConf != NULL && loadModConf->tplName != NULL) + return loadModConf->tplName; + else if(pszFileDfltTplName == NULL) + return (uchar*)" StdDBFmt"; + else + return pszFileDfltTplName; +} + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars @@ -144,7 +179,6 @@ static void closeConn(instanceData *pData) BEGINfreeInstance CODESTARTfreeInstance closeConn(pData); - free(pData->dbiDrvrDir); free(pData->drvrName); free(pData->host); free(pData->usrName); @@ -302,6 +336,79 @@ CODESTARTdoAction ENDdoAction +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; + pModConf->tplName = NULL; + bLegacyCnfModGlobalsPermitted = 1; +ENDbeginCnfLoad + +BEGINsetModCnf + struct cnfparamvals *pvals = NULL; + int i; +CODESTARTsetModCnf + pvals = nvlstGetParams(lst, &modpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "omlibdbi: error processing " + "module config parameters [module(...)]"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(Debug) { + dbgprintf("module (global) param blk for omlibdbi:\n"); + cnfparamsPrint(&modpblk, pvals); + } + + for(i = 0 ; i < modpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(modpblk.descr[i].name, "template")) { + loadModConf->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + if(pszFileDfltTplName != NULL) { + errmsg.LogError(0, RS_RET_DUP_PARAM, "omlibdbi: warning: default template " + "was already set via legacy directive - may lead to inconsistent " + "results."); + } + } else if(!strcmp(modpblk.descr[i].name, "driverdirectory")) { + loadModConf->dbiDrvrDir = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("omlibdbi: program error, non-handled " + "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); + } + } + bLegacyCnfModGlobalsPermitted = 0; +finalize_it: + if(pvals != NULL) + cnfparamvalsDestruct(pvals, &modpblk); +ENDsetModCnf + +BEGINendCnfLoad +CODESTARTendCnfLoad + loadModConf = NULL; /* done loading */ + /* free legacy config vars */ + free(pszFileDfltTplName); + pszFileDfltTplName = NULL; +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf + free(pModConf->tplName); + free(pModConf->dbiDrvrDir); +ENDfreeCnf + + + + static inline void setInstParamDefaults(instanceData *pData) { @@ -311,6 +418,7 @@ setInstParamDefaults(instanceData *pData) BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; CODESTARTnewActInst if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { @@ -332,28 +440,19 @@ CODESTARTnewActInst pData->usrName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "pwd")) { pData->pwd = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(actpblk.descr[i].name, "driverdirectory")) { - pData->dbiDrvrDir = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "driver")) { pData->drvrName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { - dbgprintf("ommysql: program error, non-handled " + dbgprintf("omlibdbi: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); } } - if(pData->tplName == NULL) { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*) strdup(" StdDBFmt"), - OMSR_RQD_TPL_OPT_SQL)); - } else { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, - (uchar*) strdup((char*) pData->tplName), - OMSR_RQD_TPL_OPT_SQL)); - } + tplToUse = (pData->tplName == NULL) ? (uchar*)strdup((char*)getDfltTpl()) : pData->tplName; + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_RQD_TPL_OPT_SQL)); CODE_STD_FINALIZERnewActInst -dbgprintf("XXXX: added param, iRet %d\n", iRet); cnfparamvalsDestruct(pvals, &actpblk); ENDnewActInst @@ -380,19 +479,17 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* NULL values are supported because drivers have different needs. * They will err out on connect. -- rgerhards, 2008-02-15 */ - if(cs.host != NULL) - if((pData->host = (uchar*) strdup((char*)cs.host)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + if(cs.host != NULL) + CHKmalloc(pData->host = (uchar*) strdup((char*)cs.host)); if(cs.usrName != NULL) - if((pData->usrName = (uchar*) strdup((char*)cs.usrName)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - if(cs.dbName != NULL) - if((pData->dbName = (uchar*) strdup((char*)cs.dbName)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - if(cs.pwd != NULL) - if((pData->pwd = (uchar*) strdup((char*)cs.pwd)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pData->usrName = (uchar*) strdup((char*)cs.usrName)); + if(cs.dbName != NULL) + CHKmalloc(pData->dbName = (uchar*) strdup((char*)cs.dbName)); + if(cs.pwd != NULL) + CHKmalloc(pData->pwd = (uchar*) strdup((char*)cs.pwd)); if(cs.dbiDrvrDir != NULL) - if((pData->dbiDrvrDir = (uchar*) strdup((char*)cs.dbiDrvrDir)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - - CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_RQD_TPL_OPT_SQL, (uchar*) " StdDBFmt")); - + CHKmalloc(loadModConf->dbiDrvrDir = (uchar*) strdup((char*)cs.dbiDrvrDir)); + CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_RQD_TPL_OPT_SQL, getDfltTpl())); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -413,6 +510,8 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt @@ -444,7 +543,7 @@ INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbidriverdirectory", 0, eCmdHdlrGetWord, NULL, &cs.dbiDrvrDir, STD_LOADABLE_MODULE_ID)); + CHKiRet(regCfSysLineHdlr2((uchar *)"actionlibdbidriverdirectory", 0, eCmdHdlrGetWord, NULL, &cs.dbiDrvrDir, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbidriver", 0, eCmdHdlrGetWord, NULL, &cs.drvrName, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbihost", 0, eCmdHdlrGetWord, NULL, &cs.host, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbiusername", 0, eCmdHdlrGetWord, NULL, &cs.usrName, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/ommail/ommail.c b/plugins/ommail/ommail.c index d70fa30a..6044d2e9 100644 --- a/plugins/ommail/ommail.c +++ b/plugins/ommail/ommail.c @@ -689,6 +689,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/ommongodb/ommongodb.c b/plugins/ommongodb/ommongodb.c index d246fab4..dd997410 100644 --- a/plugins/ommongodb/ommongodb.c +++ b/plugins/ommongodb/ommongodb.c @@ -68,6 +68,7 @@ typedef struct _instanceData { uchar *pwd; uchar *dbNcoll; uchar *tplName; + int bErrMsgPermitted; /* only one errmsg permitted per connection */ } instanceData; @@ -139,19 +140,21 @@ static void reportMongoError(instanceData *pData) { char errStr[1024]; - errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", - rs_strerror_r(errno, errStr, sizeof(errStr))); -#if 0 gchar *err; - if(mongo_sync_cmd_get_last_error(pData->conn, (gchar*)pData->db, &err) == TRUE) { - errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", err); - } else { - errmsg.LogError(0, RS_RET_ERR, "ommongodb: we had an error, but can " - "not obtain specifics"); + int eno; + + if(pData->bErrMsgPermitted) { + eno = errno; + if(mongo_sync_cmd_get_last_error(pData->conn, (gchar*)pData->db, &err) == TRUE) { + errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", err); + } else { + DBGPRINTF("ommongodb: we had an error, but can not obtain specifics, " + "using plain old errno error message generator\n"); + errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", + rs_strerror_r(eno, errStr, sizeof(errStr))); + } + pData->bErrMsgPermitted = 0; } -#else - (void)pData; -#endif } @@ -224,20 +227,20 @@ static bson * getDefaultBSON(msg_t *pMsg) { bson *doc = NULL; - uchar *procid; short unsigned procid_free; size_t procid_len; - uchar *tag; short unsigned tag_free; size_t tag_len; - uchar *pid; short unsigned pid_free; size_t pid_len; - uchar *sys; short unsigned sys_free; size_t sys_len; - uchar *msg; short unsigned msg_free; size_t msg_len; + uchar *procid; short unsigned procid_free; rs_size_t procid_len; + uchar *tag; short unsigned tag_free; rs_size_t tag_len; + uchar *pid; short unsigned pid_free; rs_size_t pid_len; + uchar *sys; short unsigned sys_free; rs_size_t sys_len; + uchar *msg; short unsigned msg_free; rs_size_t msg_len; int severity, facil; gint64 ts_gen, ts_rcv; /* timestamps: generated, received */ int secfrac; - procid = MsgGetProp(pMsg, NULL, PROP_PROGRAMNAME, NULL, &procid_len, &procid_free); - tag = MsgGetProp(pMsg, NULL, PROP_SYSLOGTAG, NULL, &tag_len, &tag_free); - pid = MsgGetProp(pMsg, NULL, PROP_PROCID, NULL, &pid_len, &pid_free); - sys = MsgGetProp(pMsg, NULL, PROP_HOSTNAME, NULL, &sys_len, &sys_free); - msg = MsgGetProp(pMsg, NULL, PROP_MSG, NULL, &msg_len, &msg_free); + procid = MsgGetProp(pMsg, NULL, PROP_PROGRAMNAME, NULL, &procid_len, &procid_free, NULL); + tag = MsgGetProp(pMsg, NULL, PROP_SYSLOGTAG, NULL, &tag_len, &tag_free, NULL); + pid = MsgGetProp(pMsg, NULL, PROP_PROCID, NULL, &pid_len, &pid_free, NULL); + sys = MsgGetProp(pMsg, NULL, PROP_HOSTNAME, NULL, &sys_len, &sys_free, NULL); + msg = MsgGetProp(pMsg, NULL, PROP_MSG, NULL, &msg_len, &msg_free, NULL); // TODO: move to datetime? Refactor in any case! rgerhards, 2012-03-30 ts_gen = (gint64) datetime.syslogTime2time_t(&pMsg->tTIMESTAMP) * 1000; /* ms! */ @@ -296,7 +299,9 @@ static bson *BSONFromJSONObject(struct json_object *json); static gboolean BSONAppendJSONObject(bson *doc, const gchar *name, struct json_object *json) { - switch(json_object_get_type(json)) { + switch(json != NULL ? json_object_get_type(json) : json_type_null) { + case json_type_null: + return bson_append_null(doc, name); case json_type_boolean: return bson_append_boolean(doc, name, json_object_get_boolean(json)); @@ -431,9 +436,11 @@ CODESTARTdoAction /* FIXME: is this a correct return code? */ ABORT_FINALIZE(RS_RET_ERR); } - if(!mongo_sync_cmd_insert(pData->conn, (char*)pData->dbNcoll, doc, NULL)) { - reportMongoError(pData); + if(mongo_sync_cmd_insert(pData->conn, (char*)pData->dbNcoll, doc, NULL)) { + pData->bErrMsgPermitted = 1; + } else { dbgprintf("ommongodb: insert error\n"); + reportMongoError(pData); ABORT_FINALIZE(RS_RET_SUSPENDED); } diff --git a/plugins/ommysql/ommysql.c b/plugins/ommysql/ommysql.c index 69ffb9ac..2dfa29de 100644 --- a/plugins/ommysql/ommysql.c +++ b/plugins/ommysql/ommysql.c @@ -189,7 +189,6 @@ static rsRetVal initMySQL(instanceData *pData, int bSilent) ASSERT(pData != NULL); ASSERT(pData->f_hmysql == NULL); - pData->f_hmysql = mysql_init(NULL); if(pData->f_hmysql == NULL) { errmsg.LogError(0, RS_RET_SUSPENDED, "can not initialize MySQL handle"); @@ -219,10 +218,12 @@ static rsRetVal initMySQL(instanceData *pData, int bSilent) pData->f_dbpwd, pData->f_dbname, pData->f_dbsrvPort, NULL, 0) == NULL) { reportDBError(pData, bSilent); closeMySQL(pData); /* ignore any error we may get */ - iRet = RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); } + mysql_autocommit(pData->f_hmysql, 0); } +finalize_it: RETiRet; } @@ -241,6 +242,7 @@ rsRetVal writeMySQL(uchar *psz, instanceData *pData) /* see if we are ready to proceed */ if(pData->f_hmysql == NULL) { CHKiRet(initMySQL(pData, 0)); + } /* try insert */ @@ -272,12 +274,28 @@ CODESTARTtryResume } ENDtryResume +BEGINbeginTransaction +CODESTARTbeginTransaction + CHKiRet(writeMySQL((uchar*)"START TRANSACTION", pData)); +finalize_it: +ENDbeginTransaction + BEGINdoAction CODESTARTdoAction dbgprintf("\n"); - iRet = writeMySQL(ppString[0], pData); + CHKiRet(writeMySQL(ppString[0], pData)); + iRet = RS_RET_DEFER_COMMIT; +finalize_it: ENDdoAction +BEGINendTransaction +CODESTARTendTransaction + if (mysql_commit(pData->f_hmysql) != 0) { + dbgprintf("mysql server error: transaction not committed\n"); + iRet = RS_RET_SUSPENDED; + } +ENDendTransaction + static inline void setInstParamDefaults(instanceData *pData) @@ -305,7 +323,7 @@ CODESTARTnewActInst CHKiRet(createInstance(&pData)); setInstParamDefaults(pData); - CODE_STD_STRING_REQUESTnewActInst(1) + CODE_STD_STRING_REQUESTparseSelectorAct(1) for(i = 0 ; i < actpblk.nParams ; ++i) { if(!pvals[i].bUsed) continue; @@ -437,6 +455,7 @@ BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt @@ -459,6 +478,11 @@ INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); + INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); + if(!bCoreSupportsBatching) { + errmsg.LogError(0, NO_ERRCODE, "ommysql: rsyslog core too old"); + ABORT_FINALIZE(RS_RET_ERR); + } /* we need to init the MySQL library. If that fails, we cannot run */ if( diff --git a/plugins/omprog/omprog.c b/plugins/omprog/omprog.c index 6978a9d0..e425b428 100644 --- a/plugins/omprog/omprog.c +++ b/plugins/omprog/omprog.c @@ -128,7 +128,12 @@ static void execBinary(instanceData *pData, int fdStdin) assert(pData != NULL); fclose(stdin); - dup(fdStdin); + if(dup(fdStdin) == -1) { + DBGPRINTF("omprog: dup() failed\n"); + /* do some more error handling here? Maybe if the module + * gets some more widespread use... + */ + } //fclose(stdout); /* we close all file handles as we fork soon diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c index 39ffe7fb..e55836c5 100644 --- a/plugins/omrelp/omrelp.c +++ b/plugins/omrelp/omrelp.c @@ -341,6 +341,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omruleset/omruleset.c b/plugins/omruleset/omruleset.c index 67aee97e..11765507 100644 --- a/plugins/omruleset/omruleset.c +++ b/plugins/omruleset/omruleset.c @@ -120,7 +120,11 @@ CODESTARTdoAction (char*) pData->pszRulesetName, pData->pRuleset); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); MsgSetRuleset(pMsg, pData->pRuleset); - submitMsg(pMsg); + /* Note: we intentionally use submitMsg2() here, as we process messages + * that were already run through the rate-limiter. So it is (at least) + * questionable if they were rate-limited again. + */ + submitMsg2(pMsg); finalize_it: ENDdoAction @@ -165,6 +169,9 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) p += sizeof(":omruleset:") - 1; /* eat indicator sequence (-1 because of '\0'!) */ CHKiRet(createInstance(&pData)); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: omruleset is deprecated, consider " + "using the 'call' statement instead"); + /* check if a non-standard template is to be applied */ if(*(p-1) == ';') --p; @@ -192,6 +199,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt @@ -237,6 +245,9 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(ruleset, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: omruleset is deprecated, consider " + "using the 'call' statement instead"); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionomrulesetrulesetname", 0, eCmdHdlrGetWord, setRuleset, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, diff --git a/plugins/omstdout/omstdout.c b/plugins/omstdout/omstdout.c index fb95e951..59f9c8bb 100644 --- a/plugins/omstdout/omstdout.c +++ b/plugins/omstdout/omstdout.c @@ -136,9 +136,13 @@ CODESTARTdoAction toWrite = (char*) ppString[0]; } len = strlen(toWrite); - write(1, toWrite, len); /* 1 is stdout! */ + /* the following if's are just to silence compiler warnings. If someone + * actually intends to use this module in production (why???), this code + * needs to be more solid. -- rgerhards, 2012-11-28 + */ + if(write(1, toWrite, len)) {}; /* 1 is stdout! */ if(pData->bEnsureLFEnding && toWrite[len-1] != '\n') { - write(1, "\n", 1); /* write missing LF */ + if(write(1, "\n", 1)) {}; /* write missing LF */ } ENDdoAction @@ -175,6 +179,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omtesting/omtesting.c b/plugins/omtesting/omtesting.c index ff290c94..c9f1e06b 100644 --- a/plugins/omtesting/omtesting.c +++ b/plugins/omtesting/omtesting.c @@ -313,6 +313,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omudpspoof/omudpspoof.c b/plugins/omudpspoof/omudpspoof.c index d2c5364c..4f37fd8c 100644 --- a/plugins/omudpspoof/omudpspoof.c +++ b/plugins/omudpspoof/omudpspoof.c @@ -5,7 +5,7 @@ * This file builds on UDP spoofing code contributed by * David Lang <david@lang.hm>. I then created a "real" rsyslog module * out of that code and omfwd. I decided to make it a separate module because - * omfwd already mixes up too many things (TCP & UDP & a differnt modes, + * omfwd already mixes up too many things (TCP & UDP & a different modes, * this has historic reasons), it would not be a good idea to also add * spoofing to it. And, looking at the requirements, there is little in * common between omfwd and this module. @@ -93,14 +93,19 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(net) typedef struct _instanceData { + uchar *tplName; /* name of assigned template */ uchar *host; uchar *port; + uchar *sourceTpl; + int mtu; int *pSockArray; /* sockets to use for UDP */ - int compressionLevel; /* 0 - no compression, else level for zlib */ struct addrinfo *f_addr; u_short sourcePort; u_short sourcePortStart; /* for sorce port iteration */ u_short sourcePortEnd; + int bReportLibnetInitErr; /* help prevent multiple error messages on init err */ + libnet_t *libnet_handle; + char errbuf[LIBNET_ERRBUF_SIZE]; } instanceData; #define DFLT_SOURCE_PORT_START 32000 @@ -116,6 +121,22 @@ typedef struct configSettings_s { } configSettings_t; static configSettings_t cs; +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "target", eCmdHdlrGetWord, 1 }, + { "port", eCmdHdlrGetWord, 0 }, + { "sourcetemplate", eCmdHdlrGetWord, 0 }, + { "sourceport.start", eCmdHdlrInt, 0 }, + { "sourceport.end", eCmdHdlrInt, 0 }, + { "mtu", eCmdHdlrInt, 0 }, + { "template", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + /* module-global parameters */ static struct cnfparamdescr modpdescr[] = { { "template", eCmdHdlrGetWord, 0 }, @@ -148,8 +169,6 @@ ENDinitConfVars /* add some variables needed for libnet */ -libnet_t *libnet_handle; -char errbuf[LIBNET_ERRBUF_SIZE]; pthread_mutex_t mutLibnet; /* forward definitions */ @@ -291,6 +310,9 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance + pData->libnet_handle = NULL; + pData->mtu = 1500; + pData->bReportLibnetInitErr = 1; ENDcreateInstance @@ -305,8 +327,12 @@ BEGINfreeInstance CODESTARTfreeInstance /* final cleanup */ closeUDPSockets(pData); + free(pData->tplName); free(pData->port); free(pData->host); + free(pData->sourceTpl); + if(pData->libnet_handle != NULL) + libnet_destroy(pData->libnet_handle); ENDfreeInstance @@ -331,16 +357,20 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) libnet_ptag_t ip, ipo; libnet_ptag_t udp; sbool bNeedUnlock = 0; + /* hdrOffs = fragmentation flags + offset (in bytes) + * divided by 8 */ + unsigned msgOffs, hdrOffs; + unsigned maxPktLen, pktLen; DEFiRet; if(pData->pSockArray == NULL) { CHKiRet(doTryResume(pData)); } - if(len > 1472) { - DBGPRINTF("omudpspoof: msg with length %d truncated to 1472 bytes: '%.768s'\n", + if(len > 65528) { + DBGPRINTF("omudpspoof: msg with length %d truncated to 64k: '%.768s'\n", len, msg); - len = 1472; + len = 65528; } ip = ipo = udp = 0; @@ -353,28 +383,43 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) bSendSuccess = RSFALSE; d_pthread_mutex_lock(&mutLibnet); bNeedUnlock = 1; - for (r = pData->f_addr; r; r = r->ai_next) { + for (r = pData->f_addr; r && bSendSuccess == RSFALSE ; r = r->ai_next) { tempaddr = (struct sockaddr_in *)r->ai_addr; - libnet_clear_packet(libnet_handle); + /* Getting max payload size (must be multiple of 8) */ + maxPktLen = (pData->mtu - LIBNET_IPV4_H) & ~0x07; + msgOffs = 0; + /* We're doing (payload size - UDP header size) and not + * checking if it's a multiple of 8 because we know the + * header is 8 bytes long */ + if(len > (maxPktLen - LIBNET_UDP_H) ) { + hdrOffs = IP_MF; + pktLen = maxPktLen - LIBNET_UDP_H; + } else { + hdrOffs = 0; + pktLen = len; + } + DBGPRINTF("omudpspoof: stage 1: MF:%d, hdrOffs %d, pktLen %d\n", + (hdrOffs & IP_MF) >> 13, (hdrOffs & 0x1FFF) << 3, pktLen); + libnet_clear_packet(pData->libnet_handle); /* note: libnet does need ports in host order NOT in network byte order! -- rgerhards, 2009-11-12 */ udp = libnet_build_udp( ntohs(pData->sourcePort),/* source port */ ntohs(tempaddr->sin_port),/* destination port */ - LIBNET_UDP_H + len, /* packet length */ + pktLen+LIBNET_UDP_H, /* packet length */ 0, /* checksum */ (u_char*)msg, /* payload */ - len, /* payload size */ - libnet_handle, /* libnet handle */ + pktLen, /* payload size */ + pData->libnet_handle, /* libnet handle */ udp); /* libnet id */ if (udp == -1) { - DBGPRINTF("Can't build UDP header: %s\n", libnet_geterror(libnet_handle)); + DBGPRINTF("omudpspoof: can't build UDP header: %s\n", libnet_geterror(pData->libnet_handle)); } ip = libnet_build_ipv4( - LIBNET_IPV4_H + len + LIBNET_UDP_H, /* length */ + LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, /* length */ 0, /* TOS */ 242, /* IP ID */ - 0, /* IP Frag */ + hdrOffs, /* IP Frag */ 64, /* TTL */ IPPROTO_UDP, /* protocol */ 0, /* checksum */ @@ -382,28 +427,87 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) tempaddr->sin_addr.s_addr, NULL, /* payload */ 0, /* payload size */ - libnet_handle, /* libnet handle */ + pData->libnet_handle, /* libnet handle */ ip); /* libnet id */ if (ip == -1) { - DBGPRINTF("Can't build IP header: %s\n", libnet_geterror(libnet_handle)); + DBGPRINTF("omudpspoof: can't build IP header: %s\n", libnet_geterror(pData->libnet_handle)); } /* Write it to the wire. */ - lsent = libnet_write(libnet_handle); - if (lsent == -1) { - DBGPRINTF("omudpspoof: write error: %s\n", libnet_geterror(libnet_handle)); + lsent = libnet_write(pData->libnet_handle); + dbgprintf("DDDD: omudpspoof stage 1 return state %d (expected %d), fd %d\n", lsent, + (int) (LIBNET_IPV4_H+LIBNET_UDP_H+pktLen), pData->libnet_handle->fd); + if(lsent != (int) (LIBNET_IPV4_H+LIBNET_UDP_H+pktLen)) { + /* note: access to fd is a libnet internal. If a newer version of libnet does + * not expose that member, we should simply remove it. However, while it is there + * it is useful for consolidating with strace output. + */ + DBGPRINTF("omudpspoof: write error (total len %d): pktLen %d, sent %d, fd %d: %s\n", + len, LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, lsent, pData->libnet_handle->fd, + libnet_geterror(pData->libnet_handle)); + if(lsent != -1) { + bSendSuccess = RSTRUE; + } } else { bSendSuccess = RSTRUE; - break; } - } - /* finished looping */ - if (bSendSuccess == RSFALSE) { - DBGPRINTF("error forwarding via udp, suspending\n"); - iRet = RS_RET_SUSPENDED; + msgOffs += pktLen; + + /* We need to get rid of the UDP header to build the other fragments */ + libnet_clear_packet(pData->libnet_handle); + ip = LIBNET_PTAG_INITIALIZER; + while(len > msgOffs ) { /* loop until all payload is sent */ + /* check if there will be more fragments */ + if((len - msgOffs) > maxPktLen) { + /* In IP's eyes, the UDP header in the first packet + * needs to be in the offset, so we add its size to + * the payload offset here */ + hdrOffs = IP_MF + (msgOffs + LIBNET_UDP_H)/8; + pktLen = maxPktLen; + } else { + /* See above */ + hdrOffs = (msgOffs + LIBNET_UDP_H)/8; + pktLen = len - msgOffs; + } + DBGPRINTF("omudpspoof: stage 2: MF:%d, hdrOffs %d, pktLen %d\n", + (hdrOffs & IP_MF) >> 13, (hdrOffs & 0x1FFF) << 3, pktLen); + ip = libnet_build_ipv4( + LIBNET_IPV4_H + pktLen, /* length */ + 0, /* TOS */ + 242, /* IP ID */ + hdrOffs, /* IP Frag */ + 64, /* TTL */ + IPPROTO_UDP, /* protocol */ + 0, /* checksum */ + source_ip.sin_addr.s_addr, + tempaddr->sin_addr.s_addr, + (u_int8_t*)(msg+msgOffs), /* payload */ + pktLen, /* payload size */ + pData->libnet_handle, /* libnet handle */ + ip); /* libnet id */ + if (ip == -1) { + DBGPRINTF("omudpspoof: can't build IP fragment header: %s\n", libnet_geterror(pData->libnet_handle)); + } + /* Write it to the wire. */ + lsent = libnet_write(pData->libnet_handle); + dbgprintf("DDDD: omudpspoof stage 1 return state %d (expected %d)\n", lsent, (int) (LIBNET_IPV4_H+pktLen)); + if(lsent != (int) (LIBNET_IPV4_H+pktLen)) { + DBGPRINTF("omudpspoof: fragment write error len %d, sent %d: %s\n", + LIBNET_IPV4_H+LIBNET_UDP_H+len, lsent, libnet_geterror(pData->libnet_handle)); + bSendSuccess = RSFALSE; + continue; + } + msgOffs += pktLen; + } } finalize_it: + if(iRet != RS_RET_OK) { + if(pData->libnet_handle != NULL) { + libnet_destroy(pData->libnet_handle); + pData->libnet_handle = NULL; + } + } if(bNeedUnlock) { d_pthread_mutex_unlock(&mutLibnet); } @@ -424,8 +528,32 @@ static rsRetVal doTryResume(instanceData *pData) if(pData->pSockArray != NULL) FINALIZE; + if(pData->host == NULL) + ABORT_FINALIZE(RS_RET_DISABLE_ACTION); + + if(pData->libnet_handle == NULL) { + /* Initialize the libnet library. Root priviledges are required. + * this initializes a IPv4 socket to use for forging UDP packets. + */ + pData->libnet_handle = libnet_init( + LIBNET_RAW4, /* injection type */ + NULL, /* network interface */ + pData->errbuf); /* errbuf */ + + if(pData->libnet_handle == NULL) { + if(pData->bReportLibnetInitErr) { + errmsg.LogError(0, RS_RET_ERR_LIBNET_INIT, "omudpsoof: error " + "initializing libnet - are you running as root?"); + pData->bReportLibnetInitErr = 0; + } + ABORT_FINALIZE(RS_RET_ERR_LIBNET_INIT); + } + } + DBGPRINTF("omudpspoof: libnit_init() ok\n"); + pData->bReportLibnetInitErr = 1; + /* The remote address is not yet known and needs to be obtained */ - DBGPRINTF(" %s\n", pData->host); + DBGPRINTF("omudpspoof trying resume for '%s'\n", pData->host); memset(&hints, 0, sizeof(hints)); /* port must be numeric, because config file syntax requires this */ hints.ai_flags = AI_NUMERICSERV; @@ -446,7 +574,8 @@ finalize_it: freeaddrinfo(pData->f_addr); pData->f_addr = NULL; } - iRet = RS_RET_SUSPENDED; + if(iRet != RS_RET_DISABLE_ACTION) + iRet = RS_RET_SUSPENDED; } RETiRet; @@ -460,67 +589,95 @@ ENDtryResume BEGINdoAction char *psz; /* temporary buffering */ - register unsigned l; + unsigned l; int iMaxLine; CODESTARTdoAction CHKiRet(doTryResume(pData)); - iMaxLine = glbl.GetMaxLine(); - - DBGPRINTF(" %s:%s/udpspoofs\n", pData->host, getFwdPt(pData)); + DBGPRINTF(" %s:%s/omudpspoof, src '%s', msg strt '%.256s'\n", pData->host, + getFwdPt(pData), ppString[1], ppString[0]); + iMaxLine = glbl.GetMaxLine(); psz = (char*) ppString[0]; l = strlen((char*) psz); if((int) l > iMaxLine) l = iMaxLine; -# ifdef USE_NETZIP - /* Check if we should compress and, if so, do it. We also - * check if the message is large enough to justify compression. - * The smaller the message, the less likely is a gain in compression. - * To save CPU cycles, we do not try to compress very small messages. - * What "very small" means needs to be configured. Currently, it is - * hard-coded but this may be changed to a config parameter. - * rgerhards, 2006-11-30 - */ - if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { - Bytef *out; - uLongf destLen = iMaxLine + iMaxLine/100 +12; /* recommended value from zlib doc */ - uLong srcLen = l; - int ret; - /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ - CHKmalloc(out = (Bytef*) MALLOC(destLen)); - out[0] = 'z'; - out[1] = '\0'; - ret = compress2((Bytef*) out+1, &destLen, (Bytef*) psz, - srcLen, pData->compressionLevel); - DBGPRINTF("Compressing message, length was %d now %d, return state %d.\n", - l, (int) destLen, ret); - if(ret != Z_OK) { - /* if we fail, we complain, but only in debug mode - * Otherwise, we are silent. In any case, we ignore the - * failed compression and just sent the uncompressed - * data, which is still valid. So this is probably the - * best course of action. - * rgerhards, 2006-11-30 - */ - DBGPRINTF("Compression failed, sending uncompressed message\n"); - } else if(destLen+1 < l) { - /* only use compression if there is a gain in using it! */ - DBGPRINTF("there is gain in compression, so we do it\n"); - psz = (char*) out; - l = destLen + 1; /* take care for the "z" at message start! */ - } - ++destLen; - } -# endif - CHKiRet(UDPSend(pData, ppString[1], psz, l)); finalize_it: ENDdoAction +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->tplName = NULL; + pData->sourcePortStart = DFLT_SOURCE_PORT_START; + pData->sourcePortEnd = DFLT_SOURCE_PORT_END; + pData->host = NULL; + pData->port = NULL; + pData->sourceTpl = (uchar*) strdup("RSYSLOG_omudpspoofDfltSourceTpl"); + pData->mtu = 1500; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + uchar *tplToUse; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (omudpspoof)\n"); + + pvals = nvlstGetParams(lst, &actpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "omudpspoof: mandatory " + "parameters missing"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(Debug) { + dbgprintf("action param blk in omudpspoof:\n"); + cnfparamsPrint(&actpblk, pvals); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "target")) { + pData->host = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "port")) { + pData->port = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "sourcetemplate")) { + free(pData->sourceTpl); + pData->sourceTpl = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "sourceport.start")) { + pData->sourcePortStart = (int) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "sourceport.end")) { + pData->sourcePortEnd = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "mtu")) { + pData->mtu = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "template")) { + pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + DBGPRINTF("omudpspoof: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + CODE_STD_STRING_REQUESTnewActInst(2) + pData->sourcePort = pData->sourcePortStart; + + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); + CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->sourceTpl), OMSR_NO_RQD_TPL_OPTS)); + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + BEGINparseSelectorAct uchar *sourceTpl; CODESTARTparseSelectorAct @@ -578,7 +735,6 @@ freeConfigVars(void) BEGINmodExit CODESTARTmodExit /* destroy the libnet state needed for forged UDP sources */ - libnet_destroy(libnet_handle); pthread_mutex_destroy(&mutLibnet); /* release what we no longer need */ objRelease(errmsg, CORE_COMPONENT); @@ -591,6 +747,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES ENDqueryEtryPt @@ -618,18 +775,6 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net,LM_NET_FILENAME)); - /* Initialize the libnet library. Root priviledges are required. - * this initializes a IPv4 socket to use for forging UDP packets. - */ - libnet_handle = libnet_init( - LIBNET_RAW4, /* injection type */ - NULL, /* network interface */ - errbuf); /* errbuf */ - - if(libnet_handle == NULL) { - errmsg.LogError(0, NO_ERRCODE, "Error initializing libnet, can not continue "); - ABORT_FINALIZE(RS_RET_ERR_LIBNET_INIT); - } pthread_mutex_init(&mutLibnet, NULL); CHKiRet(regCfSysLineHdlr((uchar *)"actionomudpspoofdefaulttemplate", 0, eCmdHdlrGetWord, setLegacyDfltTpl, NULL, NULL)); diff --git a/rsyslog.conf b/rsyslog.conf deleted file mode 100644 index 47fc4402..00000000 --- a/rsyslog.conf +++ /dev/null @@ -1,60 +0,0 @@ -# if you experience problems, check -# http://www.rsyslog.com/troubleshoot for assistance - -# rsyslog v3: load input modules -# If you do not load inputs, nothing happens! -# You may need to set the module load path if modules are not found. - -$ModLoad immark # provides --MARK-- message capability -$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) -$ModLoad imklog # kernel logging (formerly provided by rklogd) - -# Log all kernel messages to the console. -# Logging much else clutters up the screen. -#kern.* /dev/console - -# Log anything (except mail) of level info or higher. -# Don't log private authentication messages! -*.info;mail.none;authpriv.none;cron.none -/var/log/messages - -# The authpriv file has restricted access. -authpriv.* /var/log/secure - -# Log all the mail messages in one place. -mail.* -/var/log/maillog - - -# Log cron stuff -cron.* -/var/log/cron - -# Everybody gets emergency messages -*.emerg * - -# Save news errors of level crit and higher in a special file. -uucp,news.crit -/var/log/spooler - -# Save boot messages also to boot.log -local7.* /var/log/boot.log - -# Remote Logging (we use TCP for reliable delivery) -# An on-disk queue is created for this action. If the remote host is -# down, messages are spooled to disk and sent when it is up again. -#$WorkDirectory /rsyslog/spool # where to place spool files -#$ActionQueueFileName uniqName # unique name prefix for spool files -#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) -#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown -#$ActionQueueType LinkedList # run asynchronously -#$ActionResumeRetryCount -1 # infinite retries if host is down -# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional -#*.* @@remote-host:514 - - -# ######### Receiving Messages from Remote Hosts ########## -# TCP Syslog Server: -# provides TCP syslog reception and GSS-API (if compiled to support it) -#$ModLoad imtcp.so # load module -#$InputTCPServerRun 514 # start up TCP listener at port 514 - -# UDP Syslog Server: -#$ModLoad imudp.so # provides UDP syslog reception -#$UDPServerRun 514 # start a UDP syslog server at standard port 514 diff --git a/rsyslog.service.in b/rsyslog.service.in index 2265a491..08a4870f 100644 --- a/rsyslog.service.in +++ b/rsyslog.service.in @@ -2,7 +2,6 @@ Description=System Logging Service [Service] -ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service ExecStart=@sbindir@/rsyslogd -n Sockets=syslog.socket StandardOutput=null diff --git a/runtime/Makefile.am b/runtime/Makefile.am index 67e235a0..fbc92d9c 100644 --- a/runtime/Makefile.am +++ b/runtime/Makefile.am @@ -63,16 +63,15 @@ librsyslog_la_SOURCES = \ queue.h \ ruleset.c \ ruleset.h \ - rule.c \ - rule.h \ prop.c \ prop.h \ + ratelimit.c \ + ratelimit.h \ cfsysline.c \ cfsysline.h \ sd-daemon.c \ sd-daemon.h \ \ - \ ../action.h \ ../action.c \ ../threads.c \ diff --git a/runtime/batch.h b/runtime/batch.h index 944889bd..2ec07670 100644 --- a/runtime/batch.h +++ b/runtime/batch.h @@ -34,24 +34,21 @@ * main message queue. But over time, it could potentially be useful to split the two. * rgerhad, 2009-05-12 */ -typedef enum { - BATCH_STATE_RDY = 0, /* object ready for processing */ - BATCH_STATE_BAD = 1, /* unrecoverable failure while processing, do NOT resubmit to same action */ - BATCH_STATE_SUB = 2, /* message submitted for processing, outcome yet unknown */ - BATCH_STATE_COMM = 3, /* message successfully commited */ - BATCH_STATE_DISC = 4, /* discarded - processed OK, but do not submit to any other action */ -} batch_state_t; +#define BATCH_STATE_RDY 0 /* object ready for processing */ +#define BATCH_STATE_BAD 1 /* unrecoverable failure while processing, do NOT resubmit to same action */ +#define BATCH_STATE_SUB 2 /* message submitted for processing, outcome yet unknown */ +#define BATCH_STATE_COMM 3 /* message successfully commited */ +#define BATCH_STATE_DISC 4 /* discarded - processed OK, but do not submit to any other action */ +typedef unsigned char batch_state_t; /* an object inside a batch, including any information (state!) needed for it to "life". */ struct batch_obj_s { - obj_t *pUsrp; /* pointer to user object (most often message) */ - batch_state_t state; /* associated state */ + msg_t *pMsg; /* work variables for action processing; these are reused for each action (or block of * actions) */ - sbool bFilterOK; /* work area for filter processing (per action, reused!) */ sbool bPrevWasSuspended; /* following are caches to save allocs if not absolutely necessary */ uchar *staticActStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /**< for strings */ @@ -83,8 +80,16 @@ struct batch_s { int iDoneUpTo; /* all messages below this index have state other than RDY */ qDeqID deqID; /* ID of dequeue operation that generated this batch */ int *pbShutdownImmediate;/* end processing of this batch immediately if set to 1 */ + sbool *active; /* which messages are active for processing, NULL=all */ sbool bSingleRuleset; /* do all msgs of this batch use a single ruleset? */ batch_obj_t *pElem; /* batch elements */ + batch_state_t *eltState;/* state (array!) for individual objects. + NOTE: we have moved this out of batch_obj_t because we + get a *much* better cache hit ratio this way. So do not + move it back into this structure! Note that this is really + a HUGE saving, even if it doesn't look so (both profiler + data as well as practical tests indicate that!). + */ }; @@ -97,13 +102,13 @@ batchSetSingleRuleset(batch_t *pBatch, sbool val) { /* get the batches ruleset (if we have a single ruleset) */ static inline ruleset_t* batchGetRuleset(batch_t *pBatch) { - return (pBatch->nElem > 0) ? ((msg_t*) pBatch->pElem[0].pUsrp)->pRuleset : NULL; + return (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; } /* get the ruleset of a specifc element of the batch (index not verified!) */ static inline ruleset_t* batchElemGetRuleset(batch_t *pBatch, int i) { - return ((msg_t*) pBatch->pElem[i].pUsrp)->pRuleset; + return pBatch->pElem[i].pMsg->pRuleset; } /* get number of msgs for this batch */ @@ -119,8 +124,8 @@ batchNumMsgs(batch_t *pBatch) { */ static inline void batchSetElemState(batch_t *pBatch, int i, batch_state_t newState) { - if(pBatch->pElem[i].state != BATCH_STATE_DISC) - pBatch->pElem[i].state = newState; + if(pBatch->eltState[i] != BATCH_STATE_DISC) + pBatch->eltState[i] = newState; } @@ -129,23 +134,8 @@ batchSetElemState(batch_t *pBatch, int i, batch_state_t newState) { */ static inline int batchIsValidElem(batch_t *pBatch, int i) { - return(pBatch->pElem[i].bFilterOK && pBatch->pElem[i].state != BATCH_STATE_DISC); -} - - -/* copy one batch element to another. - * This creates a complete duplicate in those cases where - * it is needed. Use duplication only when absolutely necessary! - * Note that all working fields are reset to zeros. If that were - * not done, we would have potential problems with invalid - * or double pointer frees. - * rgerhards, 2010-06-10 - */ -static inline void -batchCopyElem(batch_obj_t *pDest, batch_obj_t *pSrc) { - memset(pDest, 0, sizeof(batch_obj_t)); - pDest->pUsrp = pSrc->pUsrp; - pDest->state = pSrc->state; + return( (pBatch->eltState[i] != BATCH_STATE_DISC) + && (pBatch->active == NULL || pBatch->active[i])); } @@ -166,6 +156,7 @@ batchFree(batch_t *pBatch) { } } free(pBatch->pElem); + free(pBatch->eltState); } @@ -179,6 +170,7 @@ batchInit(batch_t *pBatch, int maxElem) { pBatch->iDoneUpTo = 0; pBatch->maxElem = maxElem; CHKmalloc(pBatch->pElem = calloc((size_t)maxElem, sizeof(batch_obj_t))); + CHKmalloc(pBatch->eltState = calloc((size_t)maxElem, sizeof(batch_state_t))); // TODO: replace calloc by inidividual writes? finalize_it: RETiRet; diff --git a/runtime/conf.c b/runtime/conf.c index 488d1b86..c97391c6 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -61,17 +61,16 @@ #include "srUtils.h" #include "errmsg.h" #include "net.h" -#include "rule.h" #include "ruleset.h" #include "rsconf.h" #include "unicode-helper.h" +#include "rainerscript.h" #ifdef OS_SOLARIS # define NAME_MAX MAXNAMELEN #endif /* forward definitions */ -//static rsRetVal cfline(rsconf_t *conf, uchar *line, rule_t **pfCurr); /* static data */ @@ -79,7 +78,6 @@ DEFobjStaticHelpers DEFobjCurrIf(module) DEFobjCurrIf(errmsg) DEFobjCurrIf(net) -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) int bConfStrictScoping = 0; /* force strict scoping during config processing? */ @@ -326,14 +324,9 @@ cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int } -/* Helper to cfline(). This function takes the filter part of a traditional, PRI - * based line and decodes the PRIs given in the selector line. It processed the - * line up to the beginning of the action part. A pointer to that beginnig is - * passed back to the caller. - * rgerhards 2005-09-15 - */ +/* Decode a traditional PRI filter */ /* GPLv3 - stems back to sysklogd */ -rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) +rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]) { uchar *p; register uchar *q; @@ -347,22 +340,15 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) DEFiRet; ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ISOBJ_TYPE_assert(pRule, rule); - dbgprintf(" - traditional PRI filter '%s'\n", *pline); - errno = 0; /* keep strerror_r() stuff out of logerror messages */ + dbgprintf("Decoding traditional PRI filter '%s'\n", pline); - pRule->f_filter_type = FILTER_PRI; - /* Note: file structure is pre-initialized to zero because it was - * created with calloc()! - */ for (i = 0; i <= LOG_NFACILITIES; i++) { - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; } /* scan through the list of selectors */ - for (p = *pline; *p && *p != '\t' && *p != ' ';) { + for (p = pline; *p && *p != '\t' && *p != ' ';) { /* find the end of this facility name list */ for (q = p; *q && *q != '\t' && *q++ != '.'; ) continue; @@ -411,28 +397,28 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) for (i = 0; i <= LOG_NFACILITIES; i++) { if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pmask[i] = TABLE_ALLPRI; else - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] &= ~(1<<pri); + pmask[i] &= ~(1<<pri); else - pRule->f_filterData.f_pmask[i] |= (1<<pri); + pmask[i] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; else - pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pmask[i] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i] &= ~(1<<i2); + pmask[i] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i] |= (1<<i2); + pmask[i] |= (1<<i2); } } } @@ -447,27 +433,27 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pmask[i >> 3] = TABLE_ALLPRI; else - pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pmask[i >> 3] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<pri); + pmask[i >> 3] &= ~(1<<pri); else - pRule->f_filterData.f_pmask[i >> 3] |= (1<<pri); + pmask[i >> 3] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pmask[i >> 3] = TABLE_NOPRI; else - pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pmask[i >> 3] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<i2); + pmask[i >> 3] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i >> 3] |= (1<<i2); + pmask[i >> 3] |= (1<<i2); } } } @@ -478,11 +464,6 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) p = q; } - /* skip to action part */ - while (*p == '\t' || *p == ' ') - p++; - - *pline = p; RETiRet; } @@ -492,7 +473,7 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) * of the action part. A pointer to that beginnig is passed back to the caller. * rgerhards 2005-09-15 */ -rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) +rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt) { rsParsObj *pPars; cstr_t *pCSCompOp; @@ -501,16 +482,11 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) int iOffset; /* for compare operations */ ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(f != NULL); - dbgprintf(" - property-based filter '%s'\n", *pline); - errno = 0; /* keep strerror_r() stuff out of logerror messages */ - - f->f_filter_type = FILTER_PROP; + dbgprintf("Decoding property-based filter '%s'\n", pline); /* create parser object starting with line string without leading colon */ - if((iRet = rsParsConstructFromSz(&pPars, (*pline)+1)) != RS_RET_OK) { + if((iRet = rsParsConstructFromSz(&pPars, pline+1)) != RS_RET_OK) { errmsg.LogError(0, iRet, "Error %d constructing parser object - ignoring selector", iRet); return(iRet); } @@ -522,15 +498,15 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) rsParsDestruct(pPars); return(iRet); } - iRet = propNameToID(pCSPropName, &f->f_filterData.prop.propID); + iRet = propNameToID(pCSPropName, &stmt->d.s_propfilt.propID); if(iRet != RS_RET_OK) { errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); rsParsDestruct(pPars); return(iRet); } - if(f->f_filterData.prop.propID == PROP_CEE) { + if(stmt->d.s_propfilt.propID == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - if((f->f_filterData.prop.propName = + if((stmt->d.s_propfilt.propName = es_newStrFromBuf((char*)cstrGetSzStrNoNULL(pCSPropName)+2, cstrLen(pCSPropName)-2)) == NULL) { cstrDestruct(&pCSPropName); return(RS_RET_ERR); @@ -553,38 +529,38 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) */ if(rsCStrLen(pCSCompOp) > 0) { if(*rsCStrGetBufBeg(pCSCompOp) == '!') { - f->f_filterData.prop.isNegated = 1; + stmt->d.s_propfilt.isNegated = 1; iOffset = 1; /* ignore '!' */ } else { - f->f_filterData.prop.isNegated = 0; + stmt->d.s_propfilt.isNegated = 0; iOffset = 0; } } else { - f->f_filterData.prop.isNegated = 0; + stmt->d.s_propfilt.isNegated = 0; iOffset = 0; } if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "contains", 8)) { - f->f_filterData.prop.operation = FIOP_CONTAINS; + stmt->d.s_propfilt.operation = FIOP_CONTAINS; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isequal", 7)) { - f->f_filterData.prop.operation = FIOP_ISEQUAL; + stmt->d.s_propfilt.operation = FIOP_ISEQUAL; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isempty", 7)) { - f->f_filterData.prop.operation = FIOP_ISEMPTY; + stmt->d.s_propfilt.operation = FIOP_ISEMPTY; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "startswith", 10)) { - f->f_filterData.prop.operation = FIOP_STARTSWITH; + stmt->d.s_propfilt.operation = FIOP_STARTSWITH; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "regex", 5)) { - f->f_filterData.prop.operation = FIOP_REGEX; + stmt->d.s_propfilt.operation = FIOP_REGEX; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "ereregex", 8)) { - f->f_filterData.prop.operation = FIOP_EREREGEX; + stmt->d.s_propfilt.operation = FIOP_EREREGEX; } else { errmsg.LogError(0, NO_ERRCODE, "error: invalid compare operation '%s' - ignoring selector", (char*) rsCStrGetSzStrNoNULL(pCSCompOp)); } rsCStrDestruct(&pCSCompOp); /* no longer needed */ - if(f->f_filterData.prop.operation != FIOP_ISEMPTY) { + if(stmt->d.s_propfilt.operation != FIOP_ISEMPTY) { /* read compare value */ - iRet = parsQuotedCStr(pPars, &f->f_filterData.prop.pCSCompValue); + iRet = parsQuotedCStr(pPars, &stmt->d.s_propfilt.pCSCompValue); if(iRet != RS_RET_OK) { errmsg.LogError(0, iRet, "error %d compare value property - ignoring selector", iRet); rsParsDestruct(pPars); @@ -592,114 +568,10 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) } } - /* skip to action part */ - if((iRet = parsSkipWhitespace(pPars)) != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d skipping to action part - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - - /* cleanup */ - *pline = *pline + rsParsGetParsePointer(pPars) + 1; - /* we are adding one for the skipped initial ":" */ - return rsParsDestruct(pPars); } -/* - * Helper to cfline(). This function interprets a BSD host selector line - * from the config file ("+/-hostname"). It stores it for further reference. - * rgerhards 2005-10-19 - */ -rsRetVal cflineProcessHostSelector(uchar **pline) -{ - DEFiRet; - - ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(**pline == '-' || **pline == '+'); - - dbgprintf(" - host selector line\n"); - - /* check include/exclude setting */ - if(**pline == '+') { - eDfltHostnameCmpMode = HN_COMP_MATCH; - } else { /* we do not check for '-', it must be, else we wouldn't be here */ - eDfltHostnameCmpMode = HN_COMP_NOMATCH; - } - (*pline)++; /* eat + or - */ - - /* the below is somewhat of a quick hack, but it is efficient (this is - * why it is in here. "+*" resets the tag selector with BSD syslog. We mimic - * this, too. As it is easy to check that condition, we do not fire up a - * parser process, just make sure we do not address beyond our space. - * Order of conditions in the if-statement is vital! rgerhards 2005-10-18 - */ - if(**pline != '\0' && **pline == '*' && *(*pline+1) == '\0') { - dbgprintf("resetting BSD-like hostname filter\n"); - eDfltHostnameCmpMode = HN_NO_COMP; - if(pDfltHostnameCmp != NULL) { - CHKiRet(rsCStrSetSzStr(pDfltHostnameCmp, NULL)); - } - } else { - dbgprintf("setting BSD-like hostname filter to '%s'\n", *pline); - if(pDfltHostnameCmp == NULL) { - /* create string for parser */ - CHKiRet(rsCStrConstructFromszStr(&pDfltHostnameCmp, *pline)); - } else { /* string objects exists, just update... */ - CHKiRet(rsCStrSetSzStr(pDfltHostnameCmp, *pline)); - } - } - -finalize_it: - RETiRet; -} - - -/* - * Helper to cfline(). This function interprets a BSD tag selector line - * from the config file ("!tagname"). It stores it for further reference. - * rgerhards 2005-10-18 - */ -rsRetVal cflineProcessTagSelector(uchar **pline) -{ - DEFiRet; - - ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(**pline == '!'); - - dbgprintf(" - programname selector line\n"); - - (*pline)++; /* eat '!' */ - - /* the below is somewhat of a quick hack, but it is efficient (this is - * why it is in here. "!*" resets the tag selector with BSD syslog. We mimic - * this, too. As it is easy to check that condition, we do not fire up a - * parser process, just make sure we do not address beyond our space. - * Order of conditions in the if-statement is vital! rgerhards 2005-10-18 - */ - if(**pline != '\0' && **pline == '*' && *(*pline+1) == '\0') { - dbgprintf("resetting programname filter\n"); - if(pDfltProgNameCmp != NULL) { - rsCStrDestruct(&pDfltProgNameCmp); - } - } else { - dbgprintf("setting programname filter to '%s'\n", *pline); - if(pDfltProgNameCmp == NULL) { - /* create string for parser */ - CHKiRet(rsCStrConstructFromszStr(&pDfltProgNameCmp, *pline)); - } else { /* string objects exists, just update... */ - CHKiRet(rsCStrSetSzStr(pDfltProgNameCmp, *pline)); - } - } - -finalize_it: - RETiRet; -} - - /* process the action part of a selector line * rgerhards, 2007-08-01 */ @@ -735,13 +607,8 @@ rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction) if(iRet == RS_RET_OK || iRet == RS_RET_SUSPENDED) { if((iRet = addAction(&pAction, pMod, pModData, pOMSR, NULL, NULL, (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - else { - dbgprintf("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* here check if the module is compatible with select features + * (currently, we have no such features!) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ conf->actions.nbrActions++; /* one more active action! */ } @@ -831,7 +698,6 @@ CODESTARTObjClassExit(conf) objRelease(module, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); - objRelease(rule, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); ENDObjClassExit(conf) @@ -845,7 +711,6 @@ BEGINAbstractObjClassInit(conf, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANG CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* TODO: make this dependcy go away! */ - CHKiRet(objUse(rule, CORE_COMPONENT)); CHKiRet(objUse(ruleset, CORE_COMPONENT)); /* These commands will NOT be supported -- the new v6.3 config system provides diff --git a/runtime/conf.h b/runtime/conf.h index 018d9111..a1bb51ad 100644 --- a/runtime/conf.h +++ b/runtime/conf.h @@ -62,11 +62,8 @@ PROTOTYPEObj(conf); rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName); rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); -/* more dirt to cover the new config interface (will go away...) */ -rsRetVal cflineProcessTagSelector(uchar **pline); -rsRetVal cflineProcessHostSelector(uchar **pline); -rsRetVal cflineProcessTradPRIFilter(uchar **pline, rule_t *pRule); -rsRetVal cflineProcessPropFilter(uchar **pline, rule_t *f); +rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]); +rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt); rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction); extern EHostnameCmpMode eDfltHostnameCmpMode; extern cstr_t *pDfltHostnameCmp; diff --git a/runtime/datetime.c b/runtime/datetime.c index 0b9b1ae2..841ff625 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -61,8 +61,10 @@ timeval2syslogTime(struct timeval *tp, struct syslogTime *t) struct tm *tm; struct tm tmBuf; long lBias; + time_t secs; - tm = localtime_r((time_t*) &(tp->tv_sec), &tmBuf); + secs = tp->tv_sec; + tm = localtime_r(&secs, &tmBuf); t->year = tm->tm_year + 1900; t->month = tm->tm_mon + 1; @@ -180,12 +182,13 @@ getTime(time_t *ttSeconds) * the method always returns zero. * \retval The number parsed. */ -static int srSLMGParseInt32(uchar** ppsz, int *pLenStr) +static inline int +srSLMGParseInt32(uchar** ppsz, int *pLenStr) { register int i; i = 0; - while(*pLenStr > 0 && isdigit((int) **ppsz)) { + while(*pLenStr > 0 && **ppsz >= '0' && **ppsz <= '9') { i = i * 10 + **ppsz - '0'; ++(*ppsz); --(*pLenStr); @@ -902,6 +905,11 @@ time_t syslogTime2time_t(struct syslogTime *ts) case 12: MonthInDays = 334; //until 01 of December break; + default: /* this cannot happen (and would be a program error) + * but we need the code to keep the compiler silent. + */ + MonthInDays = 0; /* any value fits ;) */ + break; } diff --git a/runtime/debug.c b/runtime/debug.c index 307a8bb8..1f22d34e 100644 --- a/runtime/debug.c +++ b/runtime/debug.c @@ -44,6 +44,9 @@ #include <unistd.h> #include <fcntl.h> #include <sys/stat.h> +#ifdef HAVE_SYS_SYSCALL_H +# include <sys/syscall.h> +#endif #if _POSIX_TIMERS <= 0 #include <sys/time.h> #endif @@ -66,6 +69,7 @@ static int bPrintMutexAction = 0; /* shall mutex calls be printed to the debug l static int bPrintTime = 1; /* print a timestamp together with debug message */ static int bPrintAllDebugOnExit = 0; static int bAbortTrace = 1; /* print a trace after SIGABRT or SIGSEGV */ +static int bOutputTidToStderr = 0;/* output TID to stderr on thread creation */ static char *pszAltDbgFileName = NULL; /* if set, debug output is *also* sent to here */ static int altdbg = -1; /* and the handle for alternate debug output */ int stddbg = 1; /* the handle for regular debug output, set to stdout if not forking, -1 otherwise */ @@ -293,6 +297,21 @@ static inline void dbgFuncDBRemoveMutexLock(dbgFuncDB_t *pFuncDB, pthread_mutex_ /* ------------------------- END FuncDB utility functions ------------------------- */ +/* output the current thread ID to "relevant" places + * (what "relevant" means is determinded by various ways) + */ +void +dbgOutputTID(char* name) +{ +# ifdef HAVE_SYSCALL + if(bOutputTidToStderr) + fprintf(stderr, "thread tid %u, name '%s'\n", + (unsigned)syscall(SYS_gettid), name); + DBGPRINTF("thread created, tid %u, name '%s'\n", + (unsigned)syscall(SYS_gettid), name); +# endif +} + /* ########################################################################### * IMPORTANT NOTE * Mutex instrumentation reduces the code's concurrency and thus affects its @@ -902,8 +921,12 @@ do_dbgprint(uchar *pszObjName, char *pszMsg, size_t lenMsg) lenCopy = lenMsg; memcpy(pszWriteBuf + offsWriteBuf, pszMsg, lenCopy); offsWriteBuf += lenCopy; - if(stddbg != -1) write(stddbg, pszWriteBuf, offsWriteBuf); - if(altdbg != -1) write(altdbg, pszWriteBuf, offsWriteBuf); + /* the write is included in an "if" just to silence compiler + * warnings. Here, we really don't care if the write fails, we + * have no good response to that in any case... -- rgerhards, 2012-11-28 + */ + if(stddbg != -1) if(write(stddbg, pszWriteBuf, offsWriteBuf)){}; + if(altdbg != -1) if(write(altdbg, pszWriteBuf, offsWriteBuf)){}; bWasNL = (pszMsg[lenMsg - 1] == '\n') ? 1 : 0; } @@ -1321,6 +1344,7 @@ dbgGetRuntimeOptions(void) "PrintAllDebugInfoOnExit (not yet implemented)\n" "NoLogTimestamp\n" "Nostdoout\n" + "OutputTidToStderr\n" "filetrace=file (may be provided multiple times)\n" "DebugOnDemand - enables debugging on USR1, but does not turn on output\n" "\nSee debug.html in your doc set or http://www.rsyslog.com for details\n"); @@ -1354,6 +1378,8 @@ dbgGetRuntimeOptions(void) stddbg = -1; } else if(!strcasecmp((char*)optname, "noaborttrace")) { bAbortTrace = 0; + } else if(!strcasecmp((char*)optname, "outputtidtostderr")) { + bOutputTidToStderr = 1; } else if(!strcasecmp((char*)optname, "filetrace")) { if(*optval == '\0') { fprintf(stderr, "rsyslogd " VERSION " error: logfile debug option requires filename, " @@ -1372,6 +1398,25 @@ dbgGetRuntimeOptions(void) } +void +dbgSetDebugLevel(int level) +{ + Debug = level; + debugging_on = (level == DEBUG_FULL) ? 1 : 0; +} + +void +dbgSetDebugFile(uchar *fn) +{ + if(altdbg != -1) { + dbgprintf("switching to debug file %s\n", fn); + close(altdbg); + } + if((altdbg = open((char*)fn, O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY|O_CLOEXEC, S_IRUSR|S_IWUSR)) == -1) { + fprintf(stderr, "alternate debug file could not be opened, ignoring. Error: %s\n", strerror(errno)); + } +} + /* end support system to set debug options at runtime */ rsRetVal dbgClassInit(void) diff --git a/runtime/debug.h b/runtime/debug.h index 5bd26bd8..d8585fee 100644 --- a/runtime/debug.h +++ b/runtime/debug.h @@ -89,6 +89,8 @@ typedef struct dbgCallStack_s { /* prototypes */ rsRetVal dbgClassInit(void); rsRetVal dbgClassExit(void); +void dbgSetDebugFile(uchar *fn); +void dbgSetDebugLevel(int level); void sigsegvHdlr(int signum); void dbgoprint(obj_t *pObj, char *fmt, ...) __attribute__((format(printf, 2, 3))); void dbgprintf(char *fmt, ...) __attribute__((format(printf, 1, 2))); @@ -104,6 +106,7 @@ void dbgSetExecLocation(int iStackPtr, int line); void dbgSetThrdName(uchar *pszName); void dbgPrintAllDebugInfo(void); void *dbgmalloc(size_t size); +void dbgOutputTID(char* name); /* macros */ #ifdef DEBUGLESS diff --git a/runtime/dnscache.c b/runtime/dnscache.c index 32d6e425..2096aa36 100644 --- a/runtime/dnscache.c +++ b/runtime/dnscache.c @@ -7,7 +7,7 @@ * In any case, even the initial implementaton is far faster than what we had * before. -- rgerhards, 2011-06-06 * - * Copyright 2011 by Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2013 by Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -33,6 +33,7 @@ #include <signal.h> #include <netdb.h> #include <unistd.h> +#include <ctype.h> #include "syslogd-types.h" #include "glbl.h" @@ -40,46 +41,99 @@ #include "obj.h" #include "unicode-helper.h" #include "net.h" +#include "hashtable.h" +#include "prop.h" +#include "dnscache.h" -/* in this initial implementation, we use a simple, non-optimized at all - * linear list. - */ /* module data structures */ struct dnscache_entry_s { struct sockaddr_storage addr; - uchar *pszHostFQDN; - uchar *ip; + prop_t *fqdn; + prop_t *fqdnLowerCase; + prop_t *localName; /* only local name, without domain part (if configured so) */ + prop_t *ip; struct dnscache_entry_s *next; unsigned nUsed; }; typedef struct dnscache_entry_s dnscache_entry_t; struct dnscache_s { pthread_rwlock_t rwlock; - dnscache_entry_t *root; + struct hashtable *ht; unsigned nEntries; }; typedef struct dnscache_s dnscache_t; -#define MAX_CACHE_ENTRIES 1000 /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) DEFobjCurrIf(errmsg) +DEFobjCurrIf(prop) static dnscache_t dnsCache; +static prop_t *staticErrValue; + + +/* Our hash function. + * TODO: check how well it performs on socket addresses! + */ +unsigned int +hash_from_key_fn(void *k) +{ + int len; + uchar *rkey = (uchar*) k; /* we treat this as opaque bytes */ + unsigned hashval = 1; + len = SALEN((struct sockaddr*)k); + while(len--) + hashval = hashval * 33 + *rkey++; + + return hashval; +} + +static int +key_equals_fn(void *key1, void *key2) +{ + return (SALEN((struct sockaddr*)key1) == SALEN((struct sockaddr*) key2) + && !memcmp(key1, key2, SALEN((struct sockaddr*) key1))); +} + +/* destruct a cache entry. + * Precondition: entry must already be unlinked from list + */ +static void +entryDestruct(dnscache_entry_t *etry) +{ + if(etry->fqdn != NULL) + prop.Destruct(&etry->fqdn); + if(etry->fqdnLowerCase != NULL) + prop.Destruct(&etry->fqdnLowerCase); + if(etry->localName != NULL) + prop.Destruct(&etry->localName); + if(etry->ip != NULL) + prop.Destruct(&etry->ip); + free(etry); +} /* init function (must be called once) */ rsRetVal dnscacheInit(void) { DEFiRet; - dnsCache.root = NULL; + if((dnsCache.ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))entryDestruct)) == NULL) { + DBGPRINTF("dnscache: error creating hash table!\n"); + ABORT_FINALIZE(RS_RET_ERR); // TODO: make this degrade, but run! + } dnsCache.nEntries = 0; pthread_rwlock_init(&dnsCache.rwlock, NULL); CHKiRet(objGetObjInterface(&obj)); /* this provides the root pointer for all other queries */ CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + + prop.Construct(&staticErrValue); + prop.SetString(staticErrValue, (uchar*)"???", 3); + prop.ConstructFinalize(staticErrValue); finalize_it: RETiRet; } @@ -89,38 +143,20 @@ rsRetVal dnscacheDeinit(void) { DEFiRet; - //TODO: free cache elements dnsCache.root = NULL; + prop.Destruct(&staticErrValue); + hashtable_destroy(dnsCache.ht, 1); /* 1 => free all values automatically */ pthread_rwlock_destroy(&dnsCache.rwlock); objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); RETiRet; } -/* destruct a cache entry. - * Precondition: entry must already be unlinked from list - */ -static inline void -entryDestruct(dnscache_entry_t *etry) -{ - free(etry->pszHostFQDN); - free(etry->ip); - free(etry); -} - - static inline dnscache_entry_t* findEntry(struct sockaddr_storage *addr) { - dnscache_entry_t *etry; - for(etry = dnsCache.root ; etry != NULL ; etry = etry->next) { - if(SALEN((struct sockaddr*)addr) == SALEN((struct sockaddr*) &etry->addr) - && !memcmp(addr, &etry->addr, SALEN((struct sockaddr*) addr))) - break; /* in this case, we found our entry */ - } - if(etry != NULL) - ++etry->nUsed; /* this is *not* atomic, but we can live with an occasional loss! */ - return etry; + return((dnscache_entry_t*) hashtable_search(dnsCache.ht, addr)); } @@ -144,6 +180,73 @@ mygetnameinfo(const struct sockaddr *sa, socklen_t salen, } +/* get only the local part of the hostname and set it in cache entry */ +static inline void +setLocalHostName(dnscache_entry_t *etry) +{ + uchar *fqdnLower; + uchar *p; + int count; + int i; + uchar hostbuf[NI_MAXHOST]; + + if(glbl.GetPreserveFQDN()) { + prop.AddRef(etry->fqdnLowerCase); + etry->localName = etry->fqdnLowerCase; + goto done; + } + + /* strip domain, if configured for this entry */ + fqdnLower = propGetSzStr(etry->fqdnLowerCase); + p = (uchar*)strchr((char*)fqdnLower, '.'); /* find start of domain name "machine.example.com" */ + if(p == NULL) { /* do we have a domain part? */ + prop.AddRef(etry->fqdnLowerCase); /* no! */ + etry->localName = etry->fqdnLowerCase; + goto done; + } + + i = p - fqdnLower; /* length of hostname */ + memcpy(hostbuf, fqdnLower, i); + /* now check if we belong to any of the domain names that were specified + * in the -s command line option. If so, remove and we are done. + */ + if(glbl.GetStripDomains() != NULL) { + count=0; + while(glbl.GetStripDomains()[count]) { + if(strcmp((char*)(p + 1), glbl.GetStripDomains()[count]) == 0) { + prop.CreateStringProp(&etry->localName, hostbuf, i); + goto done; + } + count++; + } + } + /* if we reach this point, we have not found any domain we should strip. Now + * we try and see if the host itself is listed in the -l command line option + * and so should be stripped also. If so, we do it and return. Please note that + * -l list FQDNs, not just the hostname part. If it did just list the hostname, the + * door would be wide-open for all kinds of mixing up of hosts. Because of this, + * you'll see comparison against the full string (pszHostFQDN) below. + */ + if(glbl.GetLocalHosts() != NULL) { + count=0; + while(glbl.GetLocalHosts()[count]) { + if(!strcmp((char*)fqdnLower, (char*)glbl.GetLocalHosts()[count])) { + prop.CreateStringProp(&etry->localName, hostbuf, i); + goto done; + } + count++; + } + } + + /* at this point, we have not found anything, so we again use the + * already-created complete full name property. + */ + prop.AddRef(etry->fqdnLowerCase); + etry->localName = etry->fqdnLowerCase; +done: return; +} + + /* resolve an address. * * Please see http://www.hmug.org/man/3/getnameinfo.php (under Caveats) @@ -154,18 +257,19 @@ mygetnameinfo(const struct sockaddr *sa, socklen_t salen, * message should be processed (1) or discarded (0). */ static rsRetVal -resolveAddr(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) +resolveAddr(struct sockaddr_storage *addr, dnscache_entry_t *etry) { DEFiRet; int error; sigset_t omask, nmask; struct addrinfo hints, *res; + char szIP[80]; /* large enough for IPv6 */ + char fqdnBuf[NI_MAXHOST]; + rs_size_t fqdnLen; + rs_size_t i; - assert(addr != NULL); - assert(pszHostFQDN != NULL); - error = mygetnameinfo((struct sockaddr *)addr, SALEN((struct sockaddr *)addr), - (char*) ip, NI_MAXHOST, NULL, 0, NI_NUMERICHOST); + (char*) szIP, sizeof(szIP), NULL, 0, NI_NUMERICHOST); if(error) { dbgprintf("Malformed from address %s\n", gai_strerror(error)); ABORT_FINALIZE(RS_RET_INVALID_SOURCE); @@ -177,9 +281,8 @@ resolveAddr(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) pthread_sigmask(SIG_BLOCK, &nmask, &omask); error = mygetnameinfo((struct sockaddr *)addr, SALEN((struct sockaddr *) addr), - (char*)pszHostFQDN, NI_MAXHOST, NULL, 0, NI_NAMEREQD); + fqdnBuf, NI_MAXHOST, NULL, 0, NI_NAMEREQD); -dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); if(error == 0) { memset (&hints, 0, sizeof (struct addrinfo)); hints.ai_flags = AI_NUMERICHOST; @@ -188,7 +291,7 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); * because we should not have obtained a non-numeric address. If * we got a numeric one, someone messed with DNS! */ - if(getaddrinfo ((char*)pszHostFQDN, NULL, &hints, &res) == 0) { + if(getaddrinfo (fqdnBuf, NULL, &hints, &res) == 0) { uchar szErrMsg[1024]; freeaddrinfo (res); /* OK, we know we have evil. The question now is what to do about @@ -204,7 +307,7 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); snprintf((char*)szErrMsg, sizeof(szErrMsg) / sizeof(uchar), "Malicious PTR record, message dropped " "IP = \"%s\" HOST = \"%s\"", - ip, pszHostFQDN); + szIP, fqdnBuf); errmsg.LogError(0, RS_RET_MALICIOUS_ENTITY, "%s", szErrMsg); pthread_sigmask(SIG_SETMASK, &omask, NULL); ABORT_FINALIZE(RS_RET_MALICIOUS_ENTITY); @@ -219,92 +322,76 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); snprintf((char*)szErrMsg, sizeof(szErrMsg) / sizeof(uchar), "Malicious PTR record (message accepted, but used IP " "instead of PTR name: IP = \"%s\" HOST = \"%s\"", - ip, pszHostFQDN); + szIP, fqdnBuf); errmsg.LogError(0, NO_ERRCODE, "%s", szErrMsg); error = 1; /* that will trigger using IP address below. */ + } else {/* we have a valid entry, so let's create the respective properties */ + fqdnLen = strlen(fqdnBuf); + prop.CreateStringProp(&etry->fqdn, (uchar*)fqdnBuf, fqdnLen); + for(i = 0 ; i < fqdnLen ; ++i) + fqdnBuf[i] = tolower(fqdnBuf[i]); + prop.CreateStringProp(&etry->fqdnLowerCase, (uchar*)fqdnBuf, fqdnLen); } } pthread_sigmask(SIG_SETMASK, &omask, NULL); } -dbgprintf("dnscache: error %d, DisableDNS %d\n", error, glbl.GetDisableDNS()); - if(error || glbl.GetDisableDNS()) { - dbgprintf("Host name for your address (%s) unknown\n", ip); - strcpy((char*) pszHostFQDN, (char*)ip); - } finalize_it: - RETiRet; -} + if(iRet != RS_RET_OK) { + strcpy(szIP, "?error.obtaining.ip?"); + error = 1; /* trigger hostname copies below! */ + } + /* we need to create the inputName property (only once during our lifetime) */ + prop.CreateStringProp(&etry->ip, (uchar*)szIP, strlen(szIP)); -/* evict an entry from the cache. We should try to evict one that does - * not decrease the hit rate that much, but we do not try to hard currently - * (as the base cache data structure may change). - * This MUST NOT be called when the cache is empty! - * rgerhards, 2011-06-06 - */ -static inline void -evictEntry(void) -{ - dnscache_entry_t *prev, *evict, *prevEvict, *etry; - unsigned lowest; - - prev = prevEvict = NULL; - evict = dnsCache.root; - lowest = evict->nUsed; - for(etry = dnsCache.root->next ; etry != NULL ; etry = etry->next) { - if(etry->nUsed < lowest) { - evict = etry; - lowest = etry->nUsed; - prevEvict = prev; - } - prev = etry; - } + if(error || glbl.GetDisableDNS()) { + dbgprintf("Host name for your address (%s) unknown\n", szIP); + prop.AddRef(etry->ip); + etry->fqdn = etry->ip; + prop.AddRef(etry->ip); + etry->fqdnLowerCase = etry->ip; + } - /* found lowest, unlink */ - if(prevEvict == NULL) { /* remove root? */ - dnsCache.root = evict->next; - } else { - prevEvict = evict->next; - } - entryDestruct(evict); + setLocalHostName(etry); + + RETiRet; } -/* add a new entry to the cache. This means the address is resolved and - * then added to the cache. - */ static inline rsRetVal addEntry(struct sockaddr_storage *addr, dnscache_entry_t **pEtry) { - uchar pszHostFQDN[NI_MAXHOST]; - uchar ip[80]; /* 80 is safe for larges IPv6 addr */ - dnscache_entry_t *etry; + int r; + struct sockaddr_storage *keybuf; + dnscache_entry_t *etry = NULL; DEFiRet; - CHKiRet(resolveAddr(addr, pszHostFQDN, ip)); + CHKmalloc(etry = MALLOC(sizeof(dnscache_entry_t))); - CHKmalloc(etry->pszHostFQDN = ustrdup(pszHostFQDN)); - CHKmalloc(etry->ip = ustrdup(ip)); + CHKiRet(resolveAddr(addr, etry)); memcpy(&etry->addr, addr, SALEN((struct sockaddr*) addr)); etry->nUsed = 0; *pEtry = etry; - /* add to list. Currently, we place the new element always at - * the root node. This needs to be optimized later. 2011-06-06 - */ + CHKmalloc(keybuf = malloc(sizeof(struct sockaddr_storage))); + memcpy(keybuf, addr, sizeof(struct sockaddr_storage)); + pthread_rwlock_unlock(&dnsCache.rwlock); /* release read lock */ pthread_rwlock_wrlock(&dnsCache.rwlock); /* and re-aquire for writing */ - if(dnsCache.nEntries >= MAX_CACHE_ENTRIES) { - evictEntry(); + r = hashtable_insert(dnsCache.ht, keybuf, *pEtry); + if(r == 0) { + DBGPRINTF("dnscache: inserting element failed\n"); } - etry->next = dnsCache.root; - dnsCache.root = etry; pthread_rwlock_unlock(&dnsCache.rwlock); - pthread_rwlock_rdlock(&dnsCache.rwlock); /* TODO: optimize this! */ + pthread_rwlock_rdlock(&dnsCache.rwlock); /* we need this again */ finalize_it: + if(iRet != RS_RET_OK && etry != NULL) { + /* Note: sub-fields cannot be populated in this case */ + free(etry); + } RETiRet; } @@ -314,7 +401,7 @@ finalize_it: * TODO: implement! */ static inline rsRetVal -validateEntry(dnscache_entry_t *etry, struct sockaddr_storage *addr) +validateEntry(dnscache_entry_t __attribute__((unused)) *etry, struct sockaddr_storage __attribute__((unused)) *addr) { return RS_RET_OK; } @@ -322,10 +409,12 @@ validateEntry(dnscache_entry_t *etry, struct sockaddr_storage *addr) /* This is the main function: it looks up an entry and returns it's name * and IP address. If the entry is not yet inside the cache, it is added. - * If the entry can not be resolved, an error is reported back. + * If the entry can not be resolved, an error is reported back. If fqdn + * or fqdnLowerCase are NULL, they are not set. */ rsRetVal -dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) +dnscacheLookup(struct sockaddr_storage *addr, prop_t **fqdn, prop_t **fqdnLowerCase, + prop_t **localName, prop_t **ip) { dnscache_entry_t *etry; DEFiRet; @@ -338,18 +427,39 @@ dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) } else { CHKiRet(validateEntry(etry, addr)); } - // TODO/QUESTION: can we get rid of the strcpy? -dbgprintf("XXXX: hostn '%s', ip '%s'\n", etry->pszHostFQDN, etry->ip); - strcpy((char*)pszHostFQDN, (char*)etry->pszHostFQDN); - strcpy((char*)ip, (char*)etry->ip); + prop.AddRef(etry->ip); + *ip = etry->ip; + if(fqdn != NULL) { + prop.AddRef(etry->fqdn); + *fqdn = etry->fqdn; + } + if(fqdnLowerCase != NULL) { + prop.AddRef(etry->fqdnLowerCase); + *fqdnLowerCase = etry->fqdnLowerCase; + } + if(localName != NULL) { + prop.AddRef(etry->localName); + *localName = etry->localName; + } finalize_it: pthread_rwlock_unlock(&dnsCache.rwlock); -dbgprintf("XXXX: dnscacheLookup finished, iRet=%d\n", iRet); if(iRet != RS_RET_OK && iRet != RS_RET_ADDRESS_UNKNOWN) { DBGPRINTF("dnscacheLookup failed with iRet %d\n", iRet); - strcpy((char*) pszHostFQDN, "???"); - strcpy((char*) ip, "???"); + prop.AddRef(staticErrValue); + *ip = staticErrValue; + if(fqdn != NULL) { + prop.AddRef(staticErrValue); + *fqdn = staticErrValue; + } + if(fqdnLowerCase != NULL) { + prop.AddRef(staticErrValue); + *fqdnLowerCase = staticErrValue; + } + if(localName != NULL) { + prop.AddRef(staticErrValue); + *localName = staticErrValue; + } } RETiRet; } diff --git a/runtime/dnscache.h b/runtime/dnscache.h index 69f038ee..9c21a645 100644 --- a/runtime/dnscache.h +++ b/runtime/dnscache.h @@ -1,6 +1,6 @@ /* Definitions for dnscache module. * - * Copyright 2011-2012 Adiscon GmbH. + * Copyright 2011-2013 Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -24,6 +24,6 @@ rsRetVal dnscacheInit(void); rsRetVal dnscacheDeinit(void); -rsRetVal dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip); +rsRetVal dnscacheLookup(struct sockaddr_storage *addr, prop_t **fqdn, prop_t **fqdnLowerCase, prop_t **localName, prop_t **ip); #endif /* #ifndef INCLUDED_DNSCACHE_H */ diff --git a/runtime/glbl.c b/runtime/glbl.c index a0997829..b3fe3a1d 100644 --- a/runtime/glbl.c +++ b/runtime/glbl.c @@ -7,7 +7,7 @@ * * Module begun 2008-04-16 by Rainer Gerhards * - * Copyright 2008-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -82,6 +82,7 @@ static uchar *pszDfltNetstrmDrvrCAF = NULL; /* default CA file for the netstrm d static uchar *pszDfltNetstrmDrvrKeyFile = NULL; /* default key file for the netstrm driver (server) */ static uchar *pszDfltNetstrmDrvrCertFile = NULL; /* default cert file for the netstrm driver (server) */ static int bTerminateInputs = 0; /* global switch that inputs shall terminate ASAP (1=> terminate) */ +pid_t glbl_ourpid; #ifndef HAVE_ATOMIC_BUILTINS static DEF_ATOMIC_HELPER_MUT(mutTerminateInputs); #endif @@ -210,7 +211,7 @@ setLocalHostIPIF(void __attribute__((unused)) *pVal, uchar *pNewVal) if(propLocalIPIF != NULL) { errmsg.LogError(0, RS_RET_ERR, "$LocalHostIPIF is already set " "and cannot be reset; place it at TOP OF rsyslog.conf!"); - ABORT_FINALIZE(RS_RET_ERR_WRKDIR); + ABORT_FINALIZE(RS_RET_ERR); } localRet = net.GetIFIPAddr(pNewVal, AF_UNSPEC, myIP, (int) sizeof(myIP)); @@ -278,6 +279,28 @@ finalize_it: RETiRet; } + +static rsRetVal +setDebugFile(void __attribute__((unused)) *pVal, uchar *pNewVal) +{ + DEFiRet; + dbgSetDebugFile(pNewVal); + free(pNewVal); + RETiRet; +} + + +static rsRetVal +setDebugLevel(void __attribute__((unused)) *pVal, int level) +{ + DEFiRet; + dbgSetDebugLevel(level); + dbgprintf("debug level %d set via config file\n", level); + dbgprintf("This is rsyslog version " VERSION "\n"); + RETiRet; +} + + /* return our local IP. * If no local IP is set, "127.0.0.1" is selected *and* set. This * is an intensional side effect that we do in order to keep things @@ -610,6 +633,8 @@ BEGINAbstractObjClassInit(glbl, 1, OBJ_IS_CORE_MODULE) /* class, version */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); /* config handlers are never unregistered and need not be - we are always loaded ;) */ + CHKiRet(regCfSysLineHdlr((uchar *)"debugfile", 0, eCmdHdlrGetWord, setDebugFile, NULL, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"debuglevel", 0, eCmdHdlrInt, setDebugLevel, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"workdirectory", 0, eCmdHdlrGetWord, setWorkDir, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"dropmsgswithmaliciousdnsptrrecords", 0, eCmdHdlrBinary, NULL, &bDropMalPTRMsgs, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"defaultnetstreamdriver", 0, eCmdHdlrGetWord, NULL, &pszDfltNetstrmDrvr, NULL)); diff --git a/runtime/glbl.h b/runtime/glbl.h index d2d1e66a..e95e48f7 100644 --- a/runtime/glbl.h +++ b/runtime/glbl.h @@ -30,11 +30,14 @@ #ifndef GLBL_H_INCLUDED #define GLBL_H_INCLUDED +#include <sys/types.h> #include "rainerscript.h" #include "prop.h" #define glblGetIOBufSize() 4096 /* size of the IO buffer, e.g. for strm class */ +extern pid_t glbl_ourpid; + /* interfaces */ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ uchar* (*GetWorkDir)(void); @@ -86,6 +89,9 @@ ENDinterface(glbl) /* the remaining prototypes */ PROTOTYPEObj(glbl); +static inline pid_t glblGetOurPid(void) { return glbl_ourpid; } +static inline void glblSetOurPid(pid_t pid) { glbl_ourpid = pid; } + void glblPrepCnf(void); void glblProcessCnf(struct cnfobj *o); void glblDoneLoadCnf(void); diff --git a/runtime/hashtable.c b/runtime/hashtable.c index a01fa7d9..f718bd43 100644 --- a/runtime/hashtable.c +++ b/runtime/hashtable.c @@ -263,7 +263,7 @@ hashtable_destroy(struct hashtable *h, int free_values) /* some generic hash functions */ -/* one provided by Aaaron Wiebe based on perl's hashng algorithm +/* one provided by Aaaron Wiebe based on perl's hashing algorithm * (so probably pretty generic). Not for excessively large strings! */ unsigned int diff --git a/runtime/module-template.h b/runtime/module-template.h index 9dd759a5..fe74bac9 100644 --- a/runtime/module-template.h +++ b/runtime/module-template.h @@ -113,7 +113,7 @@ static rsRetVal modGetID(void **pID) \ /* macro to provide the v6 config system module name */ #define MODULE_CNFNAME(name) \ -static __attribute__((unused)) rsRetVal modGetCnfName(uchar **cnfName) \ +static rsRetVal modGetCnfName(uchar **cnfName) \ { \ *cnfName = (uchar*) name; \ return RS_RET_OK;\ @@ -246,7 +246,8 @@ static rsRetVal dbgPrintInstInfo(void *pModData)\ instanceData *pData = NULL; #define CODESTARTdbgPrintInstInfo \ - pData = (instanceData*) pModData; + pData = (instanceData*) pModData; \ + (void)pData; /* prevent compiler warning if unused! */ #define ENDdbgPrintInstInfo \ RETiRet;\ diff --git a/runtime/modules.c b/runtime/modules.c index 5706685f..9f7ff31c 100644 --- a/runtime/modules.c +++ b/runtime/modules.c @@ -313,7 +313,8 @@ finalize_it: /* get the name of a module */ -static uchar *modGetName(modInfo_t *pThis) +uchar * +modGetName(modInfo_t *pThis) { return((pThis->pszName == NULL) ? (uchar*) "" : pThis->pszName); } diff --git a/runtime/modules.h b/runtime/modules.h index 02e4a699..e42d19e1 100644 --- a/runtime/modules.h +++ b/runtime/modules.h @@ -190,8 +190,11 @@ ENDinterface(module) PROTOTYPEObj(module); /* in v6, we go back to in-core static link for core objects, at least those * that are not called from plugins. + * ... and we need to know that none of the module functions are called from plugins! + * rgerhards, 2012-09-24 */ rsRetVal modulesProcessCnf(struct cnfobj *o); +uchar *modGetName(modInfo_t *pThis); rsRetVal addModToCnfList(cfgmodules_etry_t *pNew, cfgmodules_etry_t *pLast); rsRetVal readyModForCnf(modInfo_t *pThis, cfgmodules_etry_t **ppNew, cfgmodules_etry_t **ppLast); #endif /* #ifndef MODULES_H_INCLUDED */ diff --git a/runtime/msg.c b/runtime/msg.c index 45ebf5f9..c302a050 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -7,7 +7,7 @@ * of the "old" message code without any modifications. However, it * helps to have things at the right place one we go to the meat of it. * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -36,14 +36,20 @@ #include <assert.h> #include <ctype.h> #include <sys/socket.h> +#if HAVE_SYSINFO_UPTIME #include <sys/sysinfo.h> +#endif #include <netdb.h> #include <libestr.h> -#include <libee/libee.h> +#include <json/json.h> +/* For struct json_object_iter, should not be necessary in future versions */ +#include <json/json_object_private.h> #if HAVE_MALLOC_H # include <malloc.h> #endif -#include <uuid/uuid.h> +#ifdef USE_LIBUUID + #include <uuid/uuid.h> +#endif #include "rsyslog.h" #include "srUtils.h" #include "stringbuf.h" @@ -57,6 +63,7 @@ #include "ruleset.h" #include "prop.h" #include "net.h" +#include "var.h" #include "rsconf.h" /* static data */ @@ -66,6 +73,19 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(regexp) DEFobjCurrIf(prop) DEFobjCurrIf(net) +DEFobjCurrIf(var) + +static char *two_digits[100] = { + "00", "01", "02", "03", "04", "05", "06", "07", "08", "09", + "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", + "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", + "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", + "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", + "50", "51", "52", "53", "54", "55", "56", "57", "58", "59", + "60", "61", "62", "63", "64", "65", "66", "67", "68", "69", + "70", "71", "72", "73", "74", "75", "76", "77", "78", "79", + "80", "81", "82", "83", "84", "85", "86", "87", "88", "89", + "90", "91", "92", "93", "94", "95", "96", "97", "98", "99"}; static struct { uchar *pszName; @@ -273,9 +293,15 @@ static char *syslog_fac_names[24] = { "kern", "user", "mail", "daemon", "auth", "news", "uucp", "cron", "authpriv", "ftp", "ntp", "audit", "alert", "clock", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7" }; +/* length of the facility names string (for optimizatiions) */ +static short len_syslog_fac_names[24] = { 4, 4, 4, 6, 4, 6, 3, + 4, 4, 4, 8, 3, 3, 5, + 5, 5, 6, 6, 6, 6, + 6, 6, 6, 6 }; /* table of severity names (in numerical order)*/ static char *syslog_severity_names[8] = { "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug" }; +static short len_syslog_severity_names[8] = { 5, 5, 4, 3, 7, 6, 4, 5 }; /* numerical values as string - this is the most efficient approach to convert severity * and facility values to a numerical string... -- rgerhars, 2009-06-17 @@ -291,6 +317,9 @@ static pthread_mutex_t mutTrimCtr; /* mutex to handle malloc trim */ /* some forward declarations */ static int getAPPNAMELen(msg_t *pM, sbool bLockMutex); +static rsRetVal jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate); +static uchar * jsonPathGetLeaf(uchar *name, int lenName); +static struct json_object *jsonDeepCopy(struct json_object *src); /* the locking and unlocking implementations: */ @@ -308,6 +337,47 @@ MsgUnlock(msg_t *pThis) } +/* set RcvFromIP name in msg object WITHOUT calling AddRef. + * rgerhards, 2013-01-22 + */ +static inline void +MsgSetRcvFromIPWithoutAddRef(msg_t *pThis, prop_t *new) +{ + if(pThis->pRcvFromIP != NULL) + prop.Destruct(&pThis->pRcvFromIP); + pThis->pRcvFromIP = new; +} + + +/* set RcvFrom name in msg object WITHOUT calling AddRef. + * rgerhards, 2013-01-22 + */ +void MsgSetRcvFromWithoutAddRef(msg_t *pThis, prop_t *new) +{ + assert(pThis != NULL); + + if(pThis->msgFlags & NEEDS_DNSRESOL) { + if(pThis->rcvFrom.pfrominet != NULL) + free(pThis->rcvFrom.pfrominet); + pThis->msgFlags &= ~NEEDS_DNSRESOL; + } else { + if(pThis->rcvFrom.pRcvFrom != NULL) + prop.Destruct(&pThis->rcvFrom.pRcvFrom); + } + pThis->rcvFrom.pRcvFrom = new; +} + + +/* rgerhards 2012-04-18: set associated ruleset (by ruleset name) + * If ruleset cannot be found, no update is done. + */ +static void +MsgSetRulesetByName(msg_t *pMsg, cstr_t *rulesetName) +{ + rulesetGetRuleset(runConf, &(pMsg->pRuleset), rsCStrGetSzStrNoNULL(rulesetName)); +} + + static inline int getProtocolVersion(msg_t *pM) { return(pM->iProtocolVersion); @@ -321,19 +391,18 @@ static inline rsRetVal resolveDNS(msg_t *pMsg) { rsRetVal localRet; prop_t *propFromHost = NULL; - prop_t *propFromHostIP = NULL; - uchar fromHost[NI_MAXHOST]; - uchar fromHostIP[NI_MAXHOST]; - uchar fromHostFQDN[NI_MAXHOST]; + prop_t *ip; + prop_t *localName; DEFiRet; MsgLock(pMsg); CHKiRet(objUse(net, CORE_COMPONENT)); if(pMsg->msgFlags & NEEDS_DNSRESOL) { - localRet = net.cvthname(pMsg->rcvFrom.pfrominet, fromHost, fromHostFQDN, fromHostIP); + localRet = net.cvthname(pMsg->rcvFrom.pfrominet, &localName, NULL, &ip); if(localRet == RS_RET_OK) { - MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); - CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); + /* we pass down the props, so no need for AddRef */ + MsgSetRcvFromWithoutAddRef(pMsg, localName); + MsgSetRcvFromIPWithoutAddRef(pMsg, ip); } } finalize_it: @@ -345,8 +414,6 @@ finalize_it: MsgUnlock(pMsg); if(propFromHost != NULL) prop.Destruct(&propFromHost); - if(propFromHostIP != NULL) - prop.Destruct(&propFromHostIP); RETiRet; } @@ -442,8 +509,10 @@ propNameStrToID(uchar *pName, propid_t *pPropID) *pPropID = PROP_MSGID; } else if(!strcmp((char*) pName, "parsesuccess")) { *pPropID = PROP_PARSESUCCESS; +#ifdef USE_LIBUUID } else if(!strcmp((char*) pName, "uuid")) { *pPropID = PROP_UUID; +#endif /* here start system properties (those, that do not relate to the message itself */ } else if(!strcmp((char*) pName, "$now")) { *pPropID = PROP_SYS_NOW; @@ -611,6 +680,7 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->iRefCount = 1; pM->iSeverity = -1; pM->iFacility = -1; + pM->iLenPROGNAME = -1; pM->offAfterPRI = 0; pM->offMSG = -1; pM->iProtocolVersion = 0; @@ -629,7 +699,6 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->pszTIMESTAMP3339 = NULL; pM->pszTIMESTAMP_MySQL = NULL; pM->pszTIMESTAMP_PgSQL = NULL; - pM->pCSProgName = NULL; pM->pCSStrucData = NULL; pM->pCSAPPNAME = NULL; pM->pCSPROCID = NULL; @@ -638,7 +707,7 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->pRcvFromIP = NULL; pM->rcvFrom.pRcvFrom = NULL; pM->pRuleset = NULL; - pM->event = NULL; + pM->json = NULL; memset(&pM->tRcvdAt, 0, sizeof(pM->tRcvdAt)); memset(&pM->tTIMESTAMP, 0, sizeof(pM->tTIMESTAMP)); pM->TAG.pszTAG = NULL; @@ -708,6 +777,19 @@ finalize_it: } +/* Special msg constructor, to be used when an object is deserialized. + * we do only the base init as we know the properties will be set in + * any case by the deserializer. We still do the "inexpensive" inits + * just to be on the safe side. The whole process needs to be + * refactored together with the msg serialization subsystem. + */ +rsRetVal +msgConstructForDeserializer(msg_t **ppThis) +{ + return msgBaseConstruct(ppThis); +} + + /* some free handlers for (slightly) complicated cases... All of them may be called * with an empty element. */ @@ -759,8 +841,8 @@ CODESTARTobjDestruct(msg) free(pThis->pszRcvdAt_PgSQL); free(pThis->pszTIMESTAMP_MySQL); free(pThis->pszTIMESTAMP_PgSQL); - if(pThis->pCSProgName != NULL) - rsCStrDestruct(&pThis->pCSProgName); + if(pThis->iLenPROGNAME >= CONF_PROGNAME_BUFSIZE) + free(pThis->PROGNAME.ptr); if(pThis->pCSStrucData != NULL) rsCStrDestruct(&pThis->pCSStrucData); if(pThis->pCSAPPNAME != NULL) @@ -769,8 +851,8 @@ CODESTARTobjDestruct(msg) rsCStrDestruct(&pThis->pCSPROCID); if(pThis->pCSMSGID != NULL) rsCStrDestruct(&pThis->pCSMSGID); - if(pThis->event != NULL) - ee_deleteEvent(pThis->event); + if(pThis->json != NULL) + json_object_put(pThis->json); if(pThis->pszUUID != NULL) free(pThis->pszUUID); # ifndef HAVE_ATOMIC_BUILTINS @@ -913,12 +995,14 @@ msg_t* MsgDup(msg_t* pOld) } } - tmpCOPYCSTR(ProgName); tmpCOPYCSTR(StrucData); tmpCOPYCSTR(APPNAME); tmpCOPYCSTR(PROCID); tmpCOPYCSTR(MSGID); + if(pOld->json != NULL) + pNew->json = jsonDeepCopy(pOld->json); + /* we do not copy all other cache properties, as we do not even know * if they are needed once again. So we let them re-create if needed. */ @@ -972,6 +1056,10 @@ static rsRetVal MsgSerialize(msg_t *pThis, strm_t *pStrm) CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFrom"), PROPTYPE_PSZ, (void*) psz)); psz = getRcvFromIP(pThis); CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFromIP"), PROPTYPE_PSZ, (void*) psz)); + if(pThis->json != NULL) { + psz = (uchar*) json_object_get_string(pThis->json); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("json"), PROPTYPE_PSZ, (void*) psz)); + } objSerializePTR(pStrm, pCSStrucData, CSTR); objSerializePTR(pStrm, pCSAPPNAME, CSTR); @@ -998,6 +1086,168 @@ finalize_it: } +/* This is a helper for MsgDeserialize that re-inits the var object. This + * whole construct should be replaced, var is really ready to be retired. + * But as an interim help during refactoring let's introduce this function + * here (and thus NOT as method of var object!). -- rgerhads, 2012-11-06 + */ +static inline void +reinitVar(var_t *pVar) +{ + rsCStrDestruct(&pVar->pcsName); /* no longer needed */ + if(pVar->varType == VARTYPE_STR) { + if(pVar->val.pStr != NULL) + rsCStrDestruct(&pVar->val.pStr); + } +} +/* deserialize the message again + * we deserialize the properties in the same order that we serialized them. Except + * for some checks to cover downlevel version, we do not need to do all these + * CPU intense name checkings. + */ +#define isProp(name) !rsCStrSzStrCmp(pVar->pcsName, (uchar*) name, sizeof(name) - 1) +rsRetVal +MsgDeserialize(msg_t *pMsg, strm_t *pStrm) +{ + prop_t *myProp; + prop_t *propRcvFrom = NULL; + prop_t *propRcvFromIP = NULL; + struct json_tokener *tokener; + struct json_object *json; + var_t *pVar = NULL; + DEFiRet; + + ISOBJ_TYPE_assert(pStrm, strm); + + CHKiRet(var.Construct(&pVar)); + CHKiRet(var.ConstructFinalize(pVar)); + + CHKiRet(objDeserializeProperty(pVar, pStrm)); + if(isProp("iProtocolVersion")) { + setProtocolVersion(pMsg, pVar->val.num); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("iSeverity")) { + pMsg->iSeverity = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("iFacility")) { + pMsg->iFacility = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("msgFlags")) { + pMsg->msgFlags = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("ttGenTime")) { + pMsg->ttGenTime = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("tRcvdAt")) { + memcpy(&pMsg->tRcvdAt, &pVar->val.vSyslogTime, sizeof(struct syslogTime)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("tTIMESTAMP")) { + memcpy(&pMsg->tTIMESTAMP, &pVar->val.vSyslogTime, sizeof(struct syslogTime)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszTAG")) { + MsgSetTAG(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), cstrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRawMsg")) { + MsgSetRawMsg(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr), cstrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszHOSTNAME")) { + MsgSetHOSTNAME(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszInputName")) { + /* we need to create a property */ + CHKiRet(prop.Construct(&myProp)); + CHKiRet(prop.SetString(myProp, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr))); + CHKiRet(prop.ConstructFinalize(myProp)); + MsgSetInputName(pMsg, myProp); + prop.Destruct(&myProp); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRcvFrom")) { + MsgSetRcvFromStr(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr), &propRcvFrom); + prop.Destruct(&propRcvFrom); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRcvFromIP")) { + MsgSetRcvFromIPStr(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr), &propRcvFromIP); + prop.Destruct(&propRcvFromIP); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("json")) { + tokener = json_tokener_new(); + json = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pVar->val.pStr), + cstrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSStrucData")) { + MsgSetStructuredData(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSAPPNAME")) { + MsgSetAPPNAME(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSPROCID")) { + MsgSetPROCID(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSMSGID")) { + MsgSetMSGID(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszUUID")) { + pMsg->pszUUID = ustrdup(rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRuleset")) { + MsgSetRulesetByName(pMsg, pVar->val.pStr); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + /* "offMSG" must always be our last field, so we use this as an + * indicator if the sequence is correct. This is a bit questionable, + * but on the other hand it works decently AND we will probably replace + * the whole persisted format soon in any case. -- rgerhards, 2012-11-06 + */ + if(!isProp("offMSG")) + ABORT_FINALIZE(RS_RET_DS_PROP_SEQ_ERR); + MsgSetMSGoffs(pMsg, pVar->val.num); +finalize_it: + if(pVar != NULL) + var.Destruct(&pVar); + RETiRet; +} +#undef isProp + + /* Increment reference count - see description of the "msg" * structure for details. As a convenience to developers, * this method returns the msg pointer that is passed to it. @@ -1092,32 +1342,33 @@ finalize_it: * The above definition has been taken from the FreeBSD syslogd sources. * * The program name is not parsed by default, because it is infrequently-used. - * If it is needed, this function should be called first. It checks if it is - * already set and extracts it, if not. - * * IMPORTANT: A locked message object must be provided, else a crash will occur. * rgerhards, 2005-10-19 */ -static rsRetVal aquireProgramName(msg_t *pM) +static inline rsRetVal +aquireProgramName(msg_t *pM) { - register int i; - uchar *pszTag; + int i; + uchar *pszTag, *pszProgName; DEFiRet; assert(pM != NULL); - if(pM->pCSProgName == NULL) { - /* ok, we do not yet have it. So let's parse the TAG to obtain it. */ - pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); - CHKiRet(cstrConstruct(&pM->pCSProgName)); - for( i = 0 - ; (i < pM->iLenTAG) && isprint((int) pszTag[i]) - && (pszTag[i] != '\0') && (pszTag[i] != ':') - && (pszTag[i] != '[') && (pszTag[i] != '/') - ; ++i) { - CHKiRet(cstrAppendChar(pM->pCSProgName, pszTag[i])); - } - CHKiRet(cstrFinalize(pM->pCSProgName)); + pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); + for( i = 0 + ; (i < pM->iLenTAG) && isprint((int) pszTag[i]) + && (pszTag[i] != '\0') && (pszTag[i] != ':') + && (pszTag[i] != '[') && (pszTag[i] != '/') + ; ++i) + ; /* just search end of PROGNAME */ + if(i < CONF_PROGNAME_BUFSIZE) { + pszProgName = pM->PROGNAME.szBuf; + } else { + CHKmalloc(pM->PROGNAME.ptr = malloc(i+1)); + pszProgName = pM->PROGNAME.ptr; } + memcpy((char*)pszProgName, (char*)pszTag, i); + pszProgName[i] = '\0'; + pM->iLenPROGNAME = i; finalize_it: RETiRet; } @@ -1142,6 +1393,7 @@ char *getProtocolVersionString(msg_t *pM) return(pM->iProtocolVersion ? "1" : "0"); } +#ifdef USE_LIBUUID /* note: libuuid seems not to be thread-safe, so we need * to get some safeguards in place. */ @@ -1196,6 +1448,7 @@ void getUUID(msg_t *pM, uchar **pBuf, int *piLen) } dbgprintf("[getUUID] END\n"); } +#endif void getRawMsg(msg_t *pM, uchar **pBuf, int *piLen) @@ -1215,6 +1468,14 @@ getRawMsg(msg_t *pM, uchar **pBuf, int *piLen) } +/* note: setMSGLen() is only for friends who really know what they + * do. Setting an invalid length can be desasterous! + */ +void setMSGLen(msg_t *pM, int lenMsg) +{ + pM->iLenMSG = lenMsg; +} + int getMSGLen(msg_t *pM) { return((pM == NULL) ? 0 : pM->iLenMSG); @@ -1664,16 +1925,6 @@ void MsgSetRuleset(msg_t *pMsg, ruleset_t *pRuleset) } -/* rgerhards 2012-04-18: set associated ruleset (by ruleset name) - * If ruleset cannot be found, no update is done. - */ -static void -MsgSetRulesetByName(msg_t *pMsg, cstr_t *rulesetName) -{ - rulesetGetRuleset(runConf, &(pMsg->pRuleset), rsCStrGetSzStrNoNULL(rulesetName)); -} - - /* set TAG in msg object * (rewritten 2009-06-18 rgerhards) */ @@ -1862,53 +2113,24 @@ static inline char *getStructuredData(msg_t *pM) return (char*) pszRet; } -/* check if we have a ProgramName, and, if not, try to aquire/emulate it. - * rgerhards, 2009-06-26 - */ -static inline void prepareProgramName(msg_t *pM, sbool bLockMutex) -{ - if(pM->pCSProgName == NULL) { - if(bLockMutex == LOCK_MUTEX) - MsgLock(pM); - - /* re-query as things might have changed during locking */ - if(pM->pCSProgName == NULL) - aquireProgramName(pM); - - if(bLockMutex == LOCK_MUTEX) - MsgUnlock(pM); - } -} - - -/* get the length of the "programname" sz string - * rgerhards, 2005-10-19 - */ -int getProgramNameLen(msg_t *pM, sbool bLockMutex) -{ - assert(pM != NULL); - prepareProgramName(pM, bLockMutex); - return (pM->pCSProgName == NULL) ? 0 : rsCStrLen(pM->pCSProgName); -} - - /* get the "programname" as sz string * rgerhards, 2005-10-19 */ uchar *getProgramName(msg_t *pM, sbool bLockMutex) { - uchar *pszRet; - - if(bLockMutex == LOCK_MUTEX) - MsgLock(pM); - prepareProgramName(pM, MUTEX_ALREADY_LOCKED); - if(pM->pCSProgName == NULL) - pszRet = UCHAR_CONSTANT(""); - else - pszRet = rsCStrGetSzStrNoNULL(pM->pCSProgName); - if(bLockMutex == LOCK_MUTEX) - MsgUnlock(pM); - return pszRet; + if(pM->iLenPROGNAME == -1) { + if(bLockMutex == LOCK_MUTEX) { + MsgLock(pM); + /* need to re-check, things may have change in between! */ + if(pM->iLenPROGNAME == -1) + aquireProgramName(pM); + MsgUnlock(pM); + } else { + aquireProgramName(pM); + } + } + return (pM->iLenPROGNAME < CONF_PROGNAME_BUFSIZE) ? pM->PROGNAME.szBuf + : pM->PROGNAME.ptr; } @@ -2023,18 +2245,8 @@ finalize_it: */ void MsgSetRcvFrom(msg_t *pThis, prop_t *new) { - assert(pThis != NULL); - prop.AddRef(new); - if(pThis->msgFlags & NEEDS_DNSRESOL) { - if(pThis->rcvFrom.pfrominet != NULL) - free(pThis->rcvFrom.pfrominet); - pThis->msgFlags &= ~NEEDS_DNSRESOL; - } else { - if(pThis->rcvFrom.pRcvFrom != NULL) - prop.Destruct(&pThis->rcvFrom.pRcvFrom); - } - pThis->rcvFrom.pRcvFrom = new; + MsgSetRcvFromWithoutAddRef(pThis, new); } @@ -2067,9 +2279,7 @@ rsRetVal MsgSetRcvFromIP(msg_t *pThis, prop_t *new) BEGINfunc prop.AddRef(new); - if(pThis->pRcvFromIP != NULL) - prop.Destruct(&pThis->pRcvFromIP); - pThis->pRcvFromIP = new; + MsgSetRcvFromIPWithoutAddRef(pThis, new); ENDfunc return RS_RET_OK; } @@ -2223,21 +2433,20 @@ void MsgSetRawMsgWOSize(msg_t *pMsg, char* pszRawMsg) /* Decode a priority into textual information like auth.emerg. - * The variable pRes must point to a user-supplied buffer and - * pResLen must contain its size. The pointer to the buffer + * The variable pRes must point to a user-supplied buffer. + * The pointer to the buffer * is also returned, what makes this functiona suitable for * use in printf-like functions. * Note: a buffer size of 20 characters is always sufficient. - * Interface to this function changed 2007-06-15 by RGerhards */ -char *textpri(char *pRes, size_t pResLen, int pri) +char *textpri(char *pRes, int pri) { assert(pRes != NULL); - assert(pResLen > 0); - - snprintf(pRes, pResLen, "%s.%s", syslog_fac_names[LOG_FAC(pri)], - syslog_severity_names[LOG_PRI(pri)]); - + memcpy(pRes, syslog_fac_names[LOG_FAC(pri)], len_syslog_fac_names[LOG_FAC(pri)]); + pRes[len_syslog_fac_names[LOG_FAC(pri)]] = '.'; + memcpy(pRes+len_syslog_fac_names[LOG_FAC(pri)]+1, + syslog_severity_names[LOG_PRI(pri)], + len_syslog_severity_names[LOG_PRI(pri)]+1 /* for \0! */); return pRes; } @@ -2251,40 +2460,48 @@ char *textpri(char *pRes, size_t pResLen, int pri) */ typedef enum ENOWType { NOW_NOW, NOW_YEAR, NOW_MONTH, NOW_DAY, NOW_HOUR, NOW_HHOUR, NOW_QHOUR, NOW_MINUTE } eNOWType; #define tmpBUFSIZE 16 /* size of formatting buffer */ -static uchar *getNOW(eNOWType eNow) +static uchar *getNOW(eNOWType eNow, struct syslogTime *t) { uchar *pBuf; - struct syslogTime t; if((pBuf = (uchar*) MALLOC(sizeof(uchar) * tmpBUFSIZE)) == NULL) { return NULL; } - datetime.getCurrTime(&t, NULL); + if(t->year == 0) { /* not yet set! */ + datetime.getCurrTime(t, NULL); + } + switch(eNow) { case NOW_NOW: - snprintf((char*) pBuf, tmpBUFSIZE, "%4.4d-%2.2d-%2.2d", t.year, t.month, t.day); + memcpy(pBuf, two_digits[t->year/100], 2); + memcpy(pBuf+2, two_digits[t->year%100], 2); + pBuf[4] = '-'; + memcpy(pBuf+5, two_digits[(int)t->month], 2); + pBuf[7] = '-'; + memcpy(pBuf+8, two_digits[(int)t->day], 3); break; case NOW_YEAR: - snprintf((char*) pBuf, tmpBUFSIZE, "%4.4d", t.year); + memcpy(pBuf, two_digits[t->year/100], 2); + memcpy(pBuf+2, two_digits[t->year%100], 3); break; case NOW_MONTH: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.month); + memcpy(pBuf, two_digits[(int)t->month], 3); break; case NOW_DAY: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.day); + memcpy(pBuf, two_digits[(int)t->day], 3); break; case NOW_HOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.hour); + memcpy(pBuf, two_digits[(int)t->hour], 3); break; case NOW_HHOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute / 30); + memcpy(pBuf, two_digits[t->hour/30], 3); break; case NOW_QHOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute / 15); + memcpy(pBuf, two_digits[t->hour/15], 3); break; case NOW_MINUTE: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute); + memcpy(pBuf, two_digits[(int)t->minute], 3); break; } @@ -2293,39 +2510,77 @@ static uchar *getNOW(eNOWType eNow) #undef tmpBUFSIZE /* clean up */ -/* Get a CEE-Property from libee. This function probably should be - * placed somewhere else, but this smells like a big restructuring - * useful in any case. So for the time being, I'll simply leave the - * function here, as the context seems good enough. -- rgerhards, 2010-12-01 - */ -static inline void -getCEEPropVal(msg_t *pMsg, es_str_t *propName, uchar **pRes, int *buflen, unsigned short *pbMustBeFreed) +/* Get a CEE-Property as string value*/ +rsRetVal +getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed) { - es_str_t *str = NULL; - int r; + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + struct json_object *field; + DEFiRet; if(*pbMustBeFreed) free(*pRes); *pRes = NULL; + // TODO: mutex? + if(pM->json == NULL) goto finalize_it; - if(pMsg->event == NULL) goto finalize_it; - r = ee_getEventFieldAsString(pMsg->event, propName, &str); - - if(r != EE_OK) { - DBGPRINTF("msgGtCEEVar: libee error %d during ee_getEventFieldAsString\n", r); - FINALIZE; + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + field = pM->json; + } else { + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + field = json_object_object_get(parent, (char*)leaf); + } + if(field != NULL) { + *pRes = (uchar*) strdup(json_object_get_string(field)); + *buflen = (int) ustrlen(*pRes); + *pbMustBeFreed = 1; } - *pRes = (unsigned char*) es_str2cstr(str, "#000"); - es_deleteStr(str); - *buflen = (int) ustrlen(*pRes); - *pbMustBeFreed = 1; finalize_it: + free(name); if(*pRes == NULL) { /* could not find any value, so set it to empty */ *pRes = (unsigned char*)""; *pbMustBeFreed = 0; } + RETiRet; +} + + +/* Get a CEE-Property as native json object + */ +rsRetVal +msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson) +{ + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + DEFiRet; + + // TODO: mutex? + if(pM->json == NULL) { + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + *pjson = pM->json; + FINALIZE; + } + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + *pjson = json_object_object_get(parent, (char*)leaf); + if(*pjson == NULL) { + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + +finalize_it: + free(name); + RETiRet; } @@ -2513,7 +2768,7 @@ finalize_it: * Parameter "bMustBeFreed" is set by this function. It tells the * caller whether or not the string returned must be freed by the * caller itself. It is is 0, the caller MUST NOT free it. If it is - * 1, the caller MUST free 1. Handling this wrongly leads to either + * 1, the caller MUST free it. Handling this wrongly leads to either * a memory leak of a program abort (do to double-frees or frees on * the constant memory pool). So be careful to do it right. * rgerhards 2004-11-23 @@ -2529,16 +2784,16 @@ finalize_it: *pPropLen = sizeof("**OUT OF MEMORY**") - 1; \ return(UCHAR_CONSTANT("**OUT OF MEMORY**"));} uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - propid_t propid, es_str_t *propName, size_t *pPropLen, - unsigned short *pbMustBeFreed) + propid_t propid, es_str_t *propName, rs_size_t *pPropLen, + unsigned short *pbMustBeFreed, struct syslogTime *ttNow) { uchar *pRes; /* result pointer */ - int bufLen = -1; /* length of string or -1, if not known */ + rs_size_t bufLen = -1; /* length of string or -1, if not known */ uchar *pBufStart; uchar *pBuf; int iLen; short iOffs; - es_str_t *str; /* for CEE handling, temp. string */ + enum tplFormatTypes datefmt; BEGINfunc assert(pMsg != NULL); @@ -2558,7 +2813,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, bufLen = getMSGLen(pMsg); break; case PROP_TIMESTAMP: - pRes = (uchar*)getTimeReported(pMsg, pTpe->data.field.eDateFormat); + if (pTpe != NULL) + datefmt = pTpe->data.field.eDateFormat; + else + datefmt = tplFmtDefault; + pRes = (uchar*)getTimeReported(pMsg, datefmt); break; case PROP_HOSTNAME: pRes = (uchar*)getHOSTNAME(pMsg); @@ -2588,7 +2847,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, RET_OUT_OF_MEMORY; } else { *pbMustBeFreed = 1; - pRes = (uchar*)textpri((char*)pBuf, 20, getPRIi(pMsg)); + pRes = (uchar*)textpri((char*)pBuf, getPRIi(pMsg)); } break; case PROP_IUT: @@ -2608,7 +2867,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, pRes = (uchar*)getSeverityStr(pMsg); break; case PROP_TIMEGENERATED: - pRes = (uchar*)getTimeGenerated(pMsg, pTpe->data.field.eDateFormat); + if (pTpe != NULL) + datefmt = pTpe->data.field.eDateFormat; + else + datefmt = tplFmtDefault; + pRes = (uchar*)getTimeGenerated(pMsg, datefmt); break; case PROP_PROGRAMNAME: pRes = getProgramName(pMsg, LOCK_MUTEX); @@ -2628,74 +2891,91 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, case PROP_MSGID: pRes = (uchar*)getMSGID(pMsg); break; +#ifdef USE_LIBUUID case PROP_UUID: getUUID(pMsg, &pRes, &bufLen); break; +#endif case PROP_PARSESUCCESS: pRes = (uchar*)getParseSuccess(pMsg); break; case PROP_SYS_NOW: - if((pRes = getNOW(NOW_NOW)) == NULL) { + if((pRes = getNOW(NOW_NOW, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 10; + } break; case PROP_SYS_YEAR: - if((pRes = getNOW(NOW_YEAR)) == NULL) { + if((pRes = getNOW(NOW_YEAR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 4; + } break; case PROP_SYS_MONTH: - if((pRes = getNOW(NOW_MONTH)) == NULL) { + if((pRes = getNOW(NOW_MONTH, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_DAY: - if((pRes = getNOW(NOW_DAY)) == NULL) { + if((pRes = getNOW(NOW_DAY, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_HOUR: - if((pRes = getNOW(NOW_HOUR)) == NULL) { + if((pRes = getNOW(NOW_HOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_HHOUR: - if((pRes = getNOW(NOW_HHOUR)) == NULL) { + if((pRes = getNOW(NOW_HHOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_QHOUR: - if((pRes = getNOW(NOW_QHOUR)) == NULL) { + if((pRes = getNOW(NOW_QHOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_MINUTE: - if((pRes = getNOW(NOW_MINUTE)) == NULL) { + if((pRes = getNOW(NOW_MINUTE, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_MYHOSTNAME: pRes = glbl.GetLocalHostName(); break; case PROP_CEE_ALL_JSON: - if(pMsg->event == NULL) { - if(*pbMustBeFreed == 1) - free(pRes); - pRes = (uchar*) "{}"; - *pbMustBeFreed = 0; + if(pMsg->json == NULL) { + if(*pbMustBeFreed == 1) + free(pRes); + pRes = (uchar*) "{}"; + bufLen = 2; + *pbMustBeFreed = 0; } else { - ee_fmtEventToJSON(pMsg->event, &str); - pRes = (uchar*) es_str2cstr(str, "#000"); - es_deleteStr(str); - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + pRes = (uchar*)strdup(json_object_get_string(pMsg->json)); + *pbMustBeFreed = 1; } break; case PROP_CEE: @@ -2708,8 +2988,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, *pbMustBeFreed = 0; break; case PROP_SYS_UPTIME: -# ifdef OS_SOLARIS - pRes = (uchar*) "UPTIME NOT available under Solaris"; +# ifndef HAVE_SYSINFO_UPTIME + /* An alternative on some systems (eg Solaris) is to scan + * /var/adm/utmpx for last boot time. + */ + pRes = (uchar*) "UPTIME NOT available on this system"; *pbMustBeFreed = 0; # else { @@ -2740,7 +3023,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } /* If we did not receive a template pointer, we are already done... */ - if(pTpe == NULL) { + if(pTpe == NULL || !pTpe->bComplexProcessing) { *pPropLen = (bufLen == -1) ? ustrlen(pRes) : bufLen; return pRes; } @@ -2962,6 +3245,8 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ ; /*DO NOTHING*/ } else { + if(iTo > bufLen) /* iTo is very large, if no to-position is set in the template! */ + iTo = bufLen; iLen = iTo - iFrom + 1; /* the +1 is for an actual char, NOT \0! */ pBufStart = pBuf = MALLOC((iLen + 1) * sizeof(char)); if(pBuf == NULL) { @@ -3327,9 +3612,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, jsonField(pTpe, &pRes, pbMustBeFreed, &bufLen); } - if(bufLen == -1) - bufLen = ustrlen(pRes); - *pPropLen = bufLen; + *pPropLen = (bufLen == -1) ? ustrlen(pRes) : bufLen; ENDfunc return(pRes); @@ -3346,29 +3629,25 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, es_str_t* msgGetCEEVarNew(msg_t *pMsg, char *name) { + uchar *leaf; + char *val; es_str_t *estr = NULL; - es_str_t *epropName = NULL; - struct ee_field *field; + struct json_object *json, *parent; ISOBJ_TYPE_assert(pMsg, msg); - if(pMsg->event == NULL) { + if(pMsg->json == NULL) { estr = es_newStr(1); goto done; } - - epropName = es_newStrFromCStr(name, strlen(name)); // TODO: optimize (in grammar!) - field = ee_getEventField(pMsg->event, epropName); - if(field != NULL) { - ee_getFieldAsString(field, &estr); - } - if(estr == NULL) { - DBGPRINTF("msgGetCEEVar: error obtaining var (field=%p, var='%s')\n", - field, name); - estr = es_newStrFromCStr("*ERROR*", sizeof("*ERROR*") - 1); + leaf = jsonPathGetLeaf((uchar*)name, strlen(name)); + if(jsonPathFindParent(pMsg, (uchar*)name, leaf, &parent, 1) != RS_RET_OK) { + estr = es_newStr(1); + goto done; } - es_deleteStr(epropName); - + json = json_object_object_get(parent, (char*)leaf); + val = (char*)json_object_get_string(json); + estr = es_newStrFromCStr(val, strlen(val)); done: return estr; } @@ -3379,7 +3658,7 @@ done: es_str_t* msgGetMsgVarNew(msg_t *pThis, uchar *name) { - size_t propLen; + rs_size_t propLen; uchar *pszProp = NULL; propid_t propid; unsigned short bMustBeFreed = 0; @@ -3390,7 +3669,7 @@ msgGetMsgVarNew(msg_t *pThis, uchar *name) /* always call MsgGetProp() without a template specifier */ /* TODO: optimize propNameToID() call -- rgerhards, 2009-06-26 */ propNameStrToID(name, &propid); - pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, NULL, &propLen, &bMustBeFreed); + pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, NULL, &propLen, &bMustBeFreed, NULL); estr = es_newStrFromCStr((char*)pszProp, propLen); if(bMustBeFreed) @@ -3412,6 +3691,8 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) prop_t *myProp; prop_t *propRcvFrom = NULL; prop_t *propRcvFromIP = NULL; + struct json_tokener *tokener; + struct json_object *json; DEFiRet; ISOBJ_TYPE_assert(pThis, msg); @@ -3466,6 +3747,12 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) MsgSetRulesetByName(pThis, pProp->val.pStr); } else if(isProp("pszMSG")) { dbgprintf("no longer supported property pszMSG silently ignored\n"); + } else if(isProp("json")) { + tokener = json_tokener_new(); + json = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pProp->val.pStr), + cstrLen(pProp->val.pStr)); + json_tokener_free(tokener); + msgAddJSON(pThis, (uchar*)"!", json); } else { dbgprintf("unknown supported property '%s' silently ignored\n", rsCStrGetSzStrNoNULL(pProp->pcsName)); @@ -3481,16 +3768,293 @@ finalize_it: * satisfies the base object class getSeverity semantics. * rgerhards, 2008-01-14 */ -static rsRetVal -MsgGetSeverity(obj_t_ptr pThis, int *piSeverity) +rsRetVal +MsgGetSeverity(msg_t *pMsg, int *piSeverity) { - ISOBJ_TYPE_assert(pThis, msg); - assert(piSeverity != NULL); - *piSeverity = ((msg_t*) pThis)->iSeverity; + *piSeverity = pMsg->iSeverity; return RS_RET_OK; } +static uchar * +jsonPathGetLeaf(uchar *name, int lenName) +{ + int i; + for(i = lenName ; name[i] != '!' && i >= 0 ; --i) + /* just skip */; + if(name[i] == '!') + ++i; + return name + i; +} + + +static rsRetVal +jsonPathFindNext(struct json_object *root, uchar **name, uchar *leaf, + struct json_object **found, int bCreate) +{ + uchar namebuf[1024]; + struct json_object *json; + size_t i; + uchar *p = *name; + DEFiRet; + + if(*p == '!') + ++p; + for(i = 0 ; *p && *p != '!' && p != leaf && i < sizeof(namebuf)-1 ; ++i, ++p) + namebuf[i] = *p; + if(i > 0) { + namebuf[i] = '\0'; + dbgprintf("AAAA: next JSONPath elt: '%s'\n", namebuf); + json = json_object_object_get(root, (char*)namebuf); + } else + json = root; + if(json == NULL) { + if(!bCreate) { + ABORT_FINALIZE(RS_RET_JNAME_INVALID); + } else { + json = json_object_new_object(); + json_object_object_add(root, (char*)namebuf, json); + } + } + + *name = p; + *found = json; +finalize_it: + RETiRet; +} + +static rsRetVal +jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate) +{ + DEFiRet; + *parent = pM->json; + while(name < leaf-1) { + jsonPathFindNext(*parent, &name, leaf, parent, bCreate); + } + RETiRet; +} + +static rsRetVal +jsonMerge(struct json_object *existing, struct json_object *json) +{ + /* TODO: check & handle duplicate names */ + DEFiRet; + struct json_object_iter it; + + json_object_object_foreachC(json, it) { +DBGPRINTF("AAAA jsonMerge adds '%s'\n", it.key); + json_object_object_add(existing, it.key, + json_object_get(it.val)); + } + /* note: json-c does ref counting. We added all descandants refcounts + * in the loop above. So when we now free(_put) the root object, only + * root gets freed(). + */ + json_object_put(json); + RETiRet; +} + +/* find a JSON structure element (field or container doesn't matter). */ +rsRetVal +jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres) +{ + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + struct json_object *field; + DEFiRet; + + if(pM->json == NULL) { + field = NULL; + goto finalize_it; + } + + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + field = pM->json; + } else { + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 0)); + field = json_object_object_get(parent, (char*)leaf); + } + *jsonres = field; + +finalize_it: + free(name); + RETiRet; +} + +rsRetVal +msgAddJSON(msg_t *pM, uchar *name, struct json_object *json) +{ + /* TODO: error checks! This is a quick&dirty PoC! */ + struct json_object *parent, *leafnode; + uchar *leaf; + DEFiRet; + + MsgLock(pM); + if(name[0] == '!' && name[1] == '\0') { + if(pM->json == NULL) + pM->json = json; + else + CHKiRet(jsonMerge(pM->json, json)); + } else { + if(pM->json == NULL) { + /* now we need a root obj */ + pM->json = json_object_new_object(); + } + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + leafnode = json_object_object_get(parent, (char*)leaf); + if(leafnode == NULL) { + json_object_object_add(parent, (char*)leaf, json); + } else { + if(json_object_get_type(json) == json_type_object) { + CHKiRet(jsonMerge(pM->json, json)); + } else { +//dbgprintf("AAAA: leafnode already exists, type is %d, update with %d\n", (int)json_object_get_type(leafnode), (int)json_object_get_type(json)); + /* TODO: improve the code below, however, the current + * state is not really bad */ + if(json_object_get_type(leafnode) == json_type_object) { + DBGPRINTF("msgAddJSON: trying to update a container " + "node with a leaf, name is '%s' - " + "forbidden\n", name); + json_object_put(json); + ABORT_FINALIZE(RS_RET_INVLD_SETOP); + } + /* json-c code indicates we can simply replace a + * json type. Unfortunaltely, this is not documented + * as part of the interface spec. We still use it, + * because it speeds up processing. If it does not work + * at some point, use + * json_object_object_del(parent, (char*)leaf); + * before adding. rgerhards, 2012-09-17 + */ + json_object_object_add(parent, (char*)leaf, json); + } + } + } + +finalize_it: + MsgUnlock(pM); + RETiRet; +} + +rsRetVal +msgDelJSON(msg_t *pM, uchar *name) +{ + struct json_object *parent, *leafnode; + uchar *leaf; + DEFiRet; + +dbgprintf("AAAA: unset variable '%s'\n", name); + MsgLock(pM); + if(name[0] == '!' && name[1] == '\0') { + /* strange, but I think we should permit this. After all, + * we trust rsyslog.conf to be written by the admin. + */ + DBGPRINTF("unsetting JSON root object\n"); + json_object_put(pM->json); + pM->json = NULL; + } else { + if(pM->json == NULL) { + /* now we need a root obj */ + pM->json = json_object_new_object(); + } + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + leafnode = json_object_object_get(parent, (char*)leaf); +DBGPRINTF("AAAA: unset found JSON value path '%s', " "leaf '%s', leafnode %p\n", name, leaf, leafnode); + if(leafnode == NULL) { + DBGPRINTF("unset JSON: could not find '%s'\n", name); + ABORT_FINALIZE(RS_RET_JNAME_NOTFOUND); + } else { + DBGPRINTF("deleting JSON value path '%s', " + "leaf '%s', type %d\n", + name, leaf, json_object_get_type(leafnode)); + json_object_object_del(parent, (char*)leaf); + } + } + +finalize_it: + MsgUnlock(pM); + RETiRet; +} + +static struct json_object * +jsonDeepCopy(struct json_object *src) +{ + struct json_object *dst = NULL, *json; + struct json_object_iter it; + int arrayLen, i; + + if(src == NULL) goto done; + + switch(json_object_get_type(src)) { + case json_type_boolean: + dst = json_object_new_boolean(json_object_get_boolean(src)); + break; + case json_type_double: + dst = json_object_new_double(json_object_get_double(src)); + break; + case json_type_int: + dst = json_object_new_int(json_object_get_int(src)); + break; + case json_type_string: + dst = json_object_new_string(json_object_get_string(src)); + break; + case json_type_object: + dst = json_object_new_object(); + json_object_object_foreachC(src, it) { + json = jsonDeepCopy(it.val); + json_object_object_add(dst, it.key, json); + } + break; + case json_type_array: + arrayLen = json_object_array_length(src); + dst = json_object_new_array(); + for(i = 0 ; i < arrayLen ; ++i) { + json = json_object_array_get_idx(src, i); + json = jsonDeepCopy(json); + json_object_array_add(dst, json); + } + break; + default:DBGPRINTF("jsonDeepCopy(): error unknown type %d\n", + json_object_get_type(src)); + dst = NULL; + break; + } +done: return dst; +} + + +rsRetVal +msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *v) +{ + struct json_object *json = NULL; + char *cstr; + DEFiRet; + switch(v->datatype) { + case 'S':/* string */ + cstr = es_str2cstr(v->d.estr, NULL); + json = json_object_new_string(cstr); + free(cstr); + break; + case 'N':/* number (integer) */ + json = json_object_new_int((int) v->d.n); + break; + case 'J':/* native JSON */ + json = jsonDeepCopy(v->d.json); + break; + default:DBGPRINTF("msgSetJSONFromVar: unsupported datatype %c\n", + v->datatype); + ABORT_FINALIZE(RS_RET_ERR); + } + msgAddJSON(pMsg, varname+1, json); +finalize_it: + RETiRet; +} + /* dummy */ rsRetVal msgQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } @@ -3503,11 +4067,10 @@ BEGINObjClassInit(msg, 1, OBJ_IS_CORE_MODULE) CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(var, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_SERIALIZE, MsgSerialize); - OBJSetMethodHandler(objMethod_SETPROPERTY, MsgSetProperty); - OBJSetMethodHandler(objMethod_GETSEVERITY, MsgGetSeverity); /* some more inits */ # if HAVE_MALLOC_TRIM INIT_ATOMIC_HELPER_MUT(mutTrimCtr); diff --git a/runtime/msg.h b/runtime/msg.h index c0b50709..edf5ed98 100644 --- a/runtime/msg.h +++ b/runtime/msg.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-13 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -30,6 +30,7 @@ #include <pthread.h> #include <libestr.h> +#include <json/json.h> #include "obj.h" #include "syslogd-types.h" #include "template.h" @@ -75,6 +76,7 @@ struct msg { int iLenMSG; /* Length of the MSG part */ int iLenTAG; /* Length of the TAG part */ int iLenHOSTNAME; /* Length of HOSTNAME */ + int iLenPROGNAME; /* Length of PROGNAME (-1 = not yet set) */ uchar *pszRawMsg; /* message as it was received on the wire. This is important in case we * need to preserve cryptographic verifiers. */ uchar *pszHOSTNAME; /* HOSTNAME from syslog message */ @@ -86,7 +88,6 @@ struct msg { char *pszTIMESTAMP3339; /* TIMESTAMP as RFC3339 formatted string (32 charcters at most) */ char *pszTIMESTAMP_MySQL;/* TIMESTAMP as MySQL formatted string (always 14 charcters) */ char *pszTIMESTAMP_PgSQL;/* TIMESTAMP as PgSQL formatted string (always 21 characters) */ - cstr_t *pCSProgName; /* the (BSD) program name */ cstr_t *pCSStrucData; /* STRUCTURED-DATA */ cstr_t *pCSAPPNAME; /* APP-NAME */ cstr_t *pCSPROCID; /* PROCID */ @@ -108,11 +109,15 @@ struct msg { it obviously is solved in way or another...). */ struct syslogTime tRcvdAt;/* time the message entered this program */ struct syslogTime tTIMESTAMP;/* (parsed) value of the timestamp */ - struct ee_event *event; /**< libee event */ + struct json_object *json; /* some fixed-size buffers to save malloc()/free() for frequently used fields (from the default templates) */ uchar szRawMsg[CONF_RAWMSG_BUFSIZE]; /* most messages are small, and these are stored here (without malloc/free!) */ uchar szHOSTNAME[CONF_HOSTNAME_BUFSIZE]; union { + uchar *ptr; /* pointer to progname value */ + uchar szBuf[CONF_PROGNAME_BUFSIZE]; + } PROGNAME; + union { uchar *pszTAG; /* pointer to tag value */ uchar szBuf[CONF_TAG_BUFSIZE]; } TAG; @@ -145,6 +150,8 @@ struct msg { PROTOTYPEObjClassInit(msg); rsRetVal msgConstruct(msg_t **ppThis); rsRetVal msgConstructWithTime(msg_t **ppThis, struct syslogTime *stTime, time_t ttGenTime); +rsRetVal msgConstructForDeserializer(msg_t **ppThis); +rsRetVal msgConstructFinalizer(msg_t *pThis); rsRetVal msgDestruct(msg_t **ppM); msg_t* MsgDup(msg_t* pOld); msg_t *MsgAddRef(msg_t *pM); @@ -171,8 +178,7 @@ void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg, size_t lenMsg); rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG); uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, propid_t propid, es_str_t *propName, - size_t *pPropLen, unsigned short *pbMustBeFreed); -char *textpri(char *pRes, size_t pResLen, int pri); + rs_size_t *pPropLen, unsigned short *pbMustBeFreed, struct syslogTime *ttNow); rsRetVal msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar); es_str_t* msgGetMsgVarNew(msg_t *pThis, uchar *name); uchar *getRcvFrom(msg_t *pM); @@ -182,21 +188,34 @@ char *getPRI(msg_t *pMsg); void getRawMsg(msg_t *pM, uchar **pBuf, int *piLen); rsRetVal msgGetCEEVar(msg_t *pThis, cstr_t *propName, var_t **ppVar); es_str_t* msgGetCEEVarNew(msg_t *pMsg, char *name); +rsRetVal msgAddJSON(msg_t *pM, uchar *name, struct json_object *json); +rsRetVal getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed); +rsRetVal MsgGetSeverity(msg_t *pThis, int *piSeverity); +rsRetVal MsgDeserialize(msg_t *pMsg, strm_t *pStrm); /* TODO: remove these five (so far used in action.c) */ uchar *getMSG(msg_t *pM); char *getHOSTNAME(msg_t *pM); char *getPROCID(msg_t *pM, sbool bLockMutex); char *getAPPNAME(msg_t *pM, sbool bLockMutex); +void setMSGLen(msg_t *pM, int lenMsg); int getMSGLen(msg_t *pM); char *getHOSTNAME(msg_t *pM); int getHOSTNAMELen(msg_t *pM); uchar *getProgramName(msg_t *pM, sbool bLockMutex); -int getProgramNameLen(msg_t *pM, sbool bLockMutex); uchar *getRcvFrom(msg_t *pM); rsRetVal propNameToID(cstr_t *pCSPropName, propid_t *pPropID); uchar *propIDToName(propid_t propID); +rsRetVal msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson); +rsRetVal msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *var); +rsRetVal msgDelJSON(msg_t *pMsg, uchar *varname); +rsRetVal jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres); + +static inline rsRetVal +msgUnsetJSON(msg_t *pMsg, uchar *varname) { + return msgDelJSON(pMsg, varname+1); +} /* ------------------------------ some inline functions ------------------------------ */ diff --git a/runtime/net.c b/runtime/net.c index dcf9cb52..b291213e 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -54,7 +54,11 @@ #include <fnmatch.h> #include <fcntl.h> #include <unistd.h> +#if HAVE_GETIFADDRS #include <ifaddrs.h> +#else +#include "compat/ifaddrs.h" +#endif /* HAVE_GETIFADDRS */ #include <sys/types.h> #include <arpa/inet.h> @@ -66,6 +70,7 @@ #include "errmsg.h" #include "net.h" #include "dnscache.h" +#include "prop.h" #ifdef OS_SOLARIS # define s6_addr32 _S6_un._S6_u32 @@ -79,6 +84,7 @@ MODULE_TYPE_NOKEEP DEFobjStaticHelpers DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) /* support for defining allowed TCP and UDP senders. We use the same * structure to implement this (a linked list), but we define two different @@ -575,7 +581,7 @@ static void clearAllowedSenders(uchar *pszType) { struct AllowedSenders *pPrev; - struct AllowedSenders *pCurr; + struct AllowedSenders *pCurr = NULL; if(setAllowRoot(&pCurr, pszType) != RS_RET_OK) return; /* if something went wrong, so let's leave */ @@ -983,7 +989,7 @@ MaskCmp(struct NetAddr *pAllow, uint8_t bits, struct sockaddr *pFrom, const char static int isAllowedSender2(uchar *pszType, struct sockaddr *pFrom, const char *pszFromHost, int bChkDNS) { struct AllowedSenders *pAllow; - struct AllowedSenders *pAllowRoot; + struct AllowedSenders *pAllowRoot = NULL; int bNeededDNS = 0; /* partial check because we could not resolve DNS? */ int ret; @@ -1111,98 +1117,15 @@ void debugListenInfo(int fd, char *type) } -/* Return a printable representation of a host address. - * Now (2007-07-16) also returns the full host name (if it could be obtained) - * in the second param [thanks to mildew@gmail.com for the patch]. - * The caller must provide buffer space for pszHost and pszHostFQDN. These - * buffers must be of size NI_MAXHOST. This is not checked here, because - * there is no way to check it. We use this way of doing things because it - * frees us from using dynamic memory allocation where it really does not - * pay. - * 2005-05-16 rgerhards: added IP representation. Must also be NI_MAXHOST +/* Return a printable representation of a host addresses. If + * a parameter is NULL, it is not set. rgerhards, 2013-01-22 */ -rsRetVal cvthname(struct sockaddr_storage *f, uchar *pszHost, uchar *pszHostFQDN, uchar *pszIP) +rsRetVal +cvthname(struct sockaddr_storage *f, prop_t **localName, prop_t **fqdn, prop_t **ip) { DEFiRet; - register uchar *p; - int count; - assert(f != NULL); - assert(pszHost != NULL); - assert(pszHostFQDN != NULL); - - iRet = dnscacheLookup(f, pszHostFQDN, pszIP); - - if(iRet == RS_RET_INVALID_SOURCE) { - strcpy((char*) pszHost, (char*) pszHostFQDN); /* we use whatever was provided as replacement */ - ABORT_FINALIZE(RS_RET_OK); /* this is handled, we are happy with it */ - } else if(iRet != RS_RET_OK) { - FINALIZE; /* we return whatever error state we have - can not handle it */ - } - - /* if we reach this point, we obtained a non-numeric hostname and can now process it */ - - /* Convert to lower case */ - for(p = pszHostFQDN ; *p ; p++) - if (isupper((int) *p)) - *p = tolower(*p); - - /* OK, the fqdn is now known. Now it is time to extract only the hostname - * part if we were instructed to do so. - */ - /* TODO: quick and dirty right now: we need to optimize that. We simply - * copy over the buffer and then use the old code. In the long term, that should - * be placed in its own function and probably outside of the net module (at least - * if should no longer reley on syslogd.c's global config-setting variables). - * Note that the old code always removes the local domain. We may want to - * make this in option in the long term. (rgerhards, 2007-09-11) - */ - strcpy((char*)pszHost, (char*)pszHostFQDN); - if( (glbl.GetPreserveFQDN() == 0) - && (p = (uchar*) strchr((char*)pszHost, '.'))) { /* find start of domain name "machine.example.com" */ - strcmp((char*)(p + 1), (char*)glbl.GetLocalDomain()); - if(strcmp((char*)(p + 1), (char*)glbl.GetLocalDomain()) == 0) { - *p = '\0'; /* simply terminate the string */ - } else { - /* now check if we belong to any of the domain names that were specified - * in the -s command line option. If so, remove and we are done. - * TODO: this must go away! -- rgerhards, 2008-04-16 - * For proper modularization, this must be done different, e.g. via a - * "to be stripped" property of *this* object itself. - */ - if(glbl.GetStripDomains() != NULL) { - count=0; - while(glbl.GetStripDomains()[count]) { - if (strcmp((char*)(p + 1), glbl.GetStripDomains()[count]) == 0) { - *p = '\0'; - FINALIZE; /* we are done */ - } - count++; - } - } - /* if we reach this point, we have not found any domain we should strip. Now - * we try and see if the host itself is listed in the -l command line option - * and so should be stripped also. If so, we do it and return. Please note that - * -l list FQDNs, not just the hostname part. If it did just list the hostname, the - * door would be wide-open for all kinds of mixing up of hosts. Because of this, - * you'll see comparison against the full string (pszHost) below. The termination - * still occurs at *p, which points at the first dot after the hostname. - * TODO: this must also go away - see comment above -- rgerhards, 2008-04-16 - */ - if(glbl.GetLocalHosts() != NULL) { - count=0; - while (glbl.GetLocalHosts()[count]) { - if (!strcmp((char*)pszHost, (char*)glbl.GetLocalHosts()[count])) { - *p = '\0'; - break; /* we are done */ - } - count++; - } - } - } - } - -finalize_it: + iRet = dnscacheLookup(f, NULL, fqdn, localName, ip); RETiRet; } @@ -1467,7 +1390,7 @@ finalize_it: */ static rsRetVal HasRestrictions(uchar *pszType, int *bHasRestrictions) { - struct AllowedSenders *pAllowRoot; + struct AllowedSenders *pAllowRoot = NULL; DEFiRet; CHKiRet(setAllowRoot(&pAllowRoot, pszType)); @@ -1577,6 +1500,7 @@ BEGINObjClassExit(net, OBJ_IS_LOADABLE_MODULE) /* CHANGE class also in END MACRO CODESTARTObjClassExit(net) /* release objects we no longer need */ objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); ENDObjClassExit(net) @@ -1589,6 +1513,7 @@ BEGINAbstractObjClassInit(net, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ ENDObjClassInit(net) diff --git a/runtime/net.h b/runtime/net.h index 1b41c81c..b196116b 100644 --- a/runtime/net.h +++ b/runtime/net.h @@ -1,6 +1,6 @@ /* Definitions for network-related stuff. * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -131,7 +131,7 @@ struct permittedPeers_s { /* interfaces */ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ - rsRetVal (*cvthname)(struct sockaddr_storage *f, uchar *pszHost, uchar *pszHostFQDN, uchar *pszIP); + rsRetVal (*cvthname)(struct sockaddr_storage *f, prop_t **localName, prop_t **fqdn, prop_t **ip); /* things to go away after proper modularization */ rsRetVal (*addAllowedSenderLine)(char* pName, uchar** ppRestOfConfLine); void (*PrintAllowedSenders)(int iListToPrint); @@ -156,8 +156,9 @@ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ /* data members - these should go away over time... TODO */ int *pACLAddHostnameOnFail; /* add hostname to acl when DNS resolving has failed */ int *pACLDontResolve; /* add hostname to acl instead of resolving it to IP(s) */ + /* v8 cvthname() signature change -- rgerhards, 2013-01-18 */ ENDinterface(net) -#define netCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ +#define netCURR_IF_VERSION 8 /* increment whenever you change the interface structure! */ /* prototypes */ PROTOTYPEObj(net); diff --git a/runtime/netstrm.c b/runtime/netstrm.c index 58f38280..c046cf52 100644 --- a/runtime/netstrm.c +++ b/runtime/netstrm.c @@ -271,11 +271,11 @@ GetRemoteHName(netstrm_t *pThis, uchar **ppsz) /* get remote IP - slim wrapper for NSD driver function */ static rsRetVal -GetRemoteIP(netstrm_t *pThis, uchar **ppsz) +GetRemoteIP(netstrm_t *pThis, prop_t **ip) { DEFiRet; ISOBJ_TYPE_assert(pThis, netstrm); - iRet = pThis->Drvr.GetRemoteIP(pThis->pDrvrData, ppsz); + iRet = pThis->Drvr.GetRemoteIP(pThis->pDrvrData, ip); RETiRet; } diff --git a/runtime/netstrm.h b/runtime/netstrm.h index ee8d9e59..4ef24229 100644 --- a/runtime/netstrm.h +++ b/runtime/netstrm.h @@ -49,7 +49,7 @@ BEGINinterface(netstrm) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Send)(netstrm_t *pThis, uchar *pBuf, ssize_t *pLenBuf); rsRetVal (*Connect)(netstrm_t *pThis, int family, unsigned char *port, unsigned char *host); rsRetVal (*GetRemoteHName)(netstrm_t *pThis, uchar **pszName); - rsRetVal (*GetRemoteIP)(netstrm_t *pThis, uchar **pszIP); + rsRetVal (*GetRemoteIP)(netstrm_t *pThis, prop_t **ip); rsRetVal (*SetDrvrMode)(netstrm_t *pThis, int iMode); rsRetVal (*SetDrvrAuthMode)(netstrm_t *pThis, uchar*); rsRetVal (*SetDrvrPermPeers)(netstrm_t *pThis, permittedPeers_t*); @@ -72,10 +72,11 @@ BEGINinterface(netstrm) /* name must also be changed in ENDinterface macro! */ /* v4 */ rsRetVal (*EnableKeepAlive)(netstrm_t *pThis); ENDinterface(netstrm) -#define netstrmCURR_IF_VERSION 5 /* increment whenever you change the interface structure! */ +#define netstrmCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ /* interface version 3 added GetRemAddr() * interface version 4 added EnableKeepAlive() -- rgerhards, 2009-06-02 * interface version 5 changed return of CheckConnection from void to rsRetVal -- alorbach, 2012-09-06 + * interface version 6 changed signature of GetRemoteIP() -- rgerhards, 2013-01-21 * */ /* prototypes */ diff --git a/runtime/nsd.h b/runtime/nsd.h index d7d6abbd..aa3662a4 100644 --- a/runtime/nsd.h +++ b/runtime/nsd.h @@ -59,7 +59,7 @@ BEGINinterface(nsd) /* name must also be changed in ENDinterface macro! */ uchar *pLstnPort, uchar *pLstnIP, int iSessMax); rsRetVal (*AcceptConnReq)(nsd_t *pThis, nsd_t **ppThis); rsRetVal (*GetRemoteHName)(nsd_t *pThis, uchar **pszName); - rsRetVal (*GetRemoteIP)(nsd_t *pThis, uchar **pszIP); + rsRetVal (*GetRemoteIP)(nsd_t *pThis, prop_t **ip); rsRetVal (*SetMode)(nsd_t *pThis, int mode); /* sets a driver specific mode - see driver doc for details */ rsRetVal (*SetAuthMode)(nsd_t *pThis, uchar*); /* sets a driver specific mode - see driver doc for details */ rsRetVal (*SetPermPeers)(nsd_t *pThis, permittedPeers_t*); /* sets driver permitted peers for auth needs */ @@ -80,10 +80,11 @@ BEGINinterface(nsd) /* name must also be changed in ENDinterface macro! */ /* v5 */ rsRetVal (*EnableKeepAlive)(nsd_t *pThis); ENDinterface(nsd) -#define nsdCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define nsdCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ /* interface version 4 added GetRemAddr() * interface version 5 added EnableKeepAlive() -- rgerhards, 2009-06-02 * interface version 6 changed return of CheckConnection from void to rsRetVal -- alorbach, 2012-09-06 + * interface version 7 changed signature ofGetRempoteIP() -- rgerhards, 2013-01-21 */ /* interface for the select call */ diff --git a/runtime/nsd_gtls.c b/runtime/nsd_gtls.c index 71eafbd2..0f66c766 100644 --- a/runtime/nsd_gtls.c +++ b/runtime/nsd_gtls.c @@ -1357,16 +1357,14 @@ GetRemAddr(nsd_t *pNsd, struct sockaddr_storage **ppAddr) } -/* get the remote host's IP address. The returned string must be freed by the - * caller. -- rgerhards, 2008-04-25 - */ +/* get the remote host's IP address. Caller must Destruct the object. */ static rsRetVal -GetRemoteIP(nsd_t *pNsd, uchar **ppszIP) +GetRemoteIP(nsd_t *pNsd, prop_t **ip) { DEFiRet; nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_gtls); - iRet = nsd_ptcp.GetRemoteIP(pThis->pTcp, ppszIP); + iRet = nsd_ptcp.GetRemoteIP(pThis->pTcp, ip); RETiRet; } diff --git a/runtime/nsd_ptcp.c b/runtime/nsd_ptcp.c index 12f891ea..f889a00e 100644 --- a/runtime/nsd_ptcp.c +++ b/runtime/nsd_ptcp.c @@ -2,7 +2,7 @@ * * An implementation of the nsd interface for plain tcp sockets. * - * Copyright 2007, 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -50,6 +50,7 @@ #include "nsdsel_ptcp.h" #include "nsdpoll_ptcp.h" #include "nsd_ptcp.h" +#include "prop.h" #include "dnscache.h" MODULE_TYPE_LIB @@ -62,6 +63,7 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(net) DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) +DEFobjCurrIf(prop) /* a few deinit helpers */ @@ -87,10 +89,9 @@ ENDobjConstruct(nsd_ptcp) BEGINobjDestruct(nsd_ptcp) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(nsd_ptcp) sockClose(&pThis->sock); - if(pThis->pRemHostIP != NULL) - free(pThis->pRemHostIP); - if(pThis->pRemHostName != NULL) - free(pThis->pRemHostName); + if(pThis->remoteIP != NULL) + prop.Destruct(&pThis->remoteIP); + free(pThis->pRemHostName); ENDobjDestruct(nsd_ptcp) @@ -251,32 +252,22 @@ Abort(nsd_t *pNsd) static rsRetVal FillRemHost(nsd_ptcp_t *pThis, struct sockaddr_storage *pAddr) { - uchar szIP[NI_MAXHOST] = ""; - uchar szHname[NI_MAXHOST] = ""; - size_t len; + prop_t *fqdn; DEFiRet; ISOBJ_TYPE_assert(pThis, nsd_ptcp); assert(pAddr != NULL); - CHKiRet(dnscacheLookup(pAddr, szHname, szIP)); + CHKiRet(dnscacheLookup(pAddr, &fqdn, NULL, NULL, &pThis->remoteIP)); /* We now have the names, so now let's allocate memory and store them permanently. * (side note: we may hold on to these values for quite a while, thus we trim their * memory consumption) */ - len = strlen((char*)szIP) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostIP = MALLOC(len)) == NULL) + if((pThis->pRemHostName = MALLOC(prop.GetStringLen(fqdn)+1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - memcpy(pThis->pRemHostIP, szIP, len); - - len = strlen((char*)szHname) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostName = MALLOC(len)) == NULL) { - free(pThis->pRemHostIP); /* prevent leak */ - pThis->pRemHostIP = NULL; - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - memcpy(pThis->pRemHostName, szHname, len); + memcpy(pThis->pRemHostName, propGetSzStr(fqdn), prop.GetStringLen(fqdn)+1); + prop.Destruct(&fqdn); finalize_it: RETiRet; @@ -460,7 +451,9 @@ LstnInit(netstrms_t *pNS, void *pUsr, rsRetVal(*fAddLstn)(void*,netstrm_t*), #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - dbgprintf("error %d while binding tcp socket\n", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; @@ -717,21 +710,16 @@ finalize_it: } -/* get the remote host's IP address. The returned string must be freed by the - * caller. - * rgerhards, 2008-04-24 +/* get the remote host's IP address. Caller must Destruct the object. */ static rsRetVal -GetRemoteIP(nsd_t *pNsd, uchar **ppszIP) +GetRemoteIP(nsd_t *pNsd, prop_t **ip) { DEFiRet; nsd_ptcp_t *pThis = (nsd_ptcp_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_ptcp); - assert(ppszIP != NULL); - - CHKmalloc(*ppszIP = (uchar*)strdup(pThis->pRemHostIP == NULL ? "" : (char*) pThis->pRemHostIP)); - -finalize_it: + prop.AddRef(pThis->remoteIP); + *ip = pThis->remoteIP; RETiRet; } @@ -777,6 +765,7 @@ CODESTARTObjClassExit(nsd_ptcp) /* release objects we no longer need */ objRelease(net, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(netstrm, DONT_LOAD_LIB); objRelease(netstrms, LM_NETSTRMS_FILENAME); @@ -791,6 +780,7 @@ BEGINObjClassInit(nsd_ptcp, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(net, CORE_COMPONENT)); CHKiRet(objUse(netstrms, LM_NETSTRMS_FILENAME)); CHKiRet(objUse(netstrm, DONT_LOAD_LIB)); diff --git a/runtime/nsd_ptcp.h b/runtime/nsd_ptcp.h index a1bcd646..ed6b8565 100644 --- a/runtime/nsd_ptcp.h +++ b/runtime/nsd_ptcp.h @@ -30,7 +30,7 @@ typedef nsd_if_t nsd_ptcp_if_t; /* we just *implement* this interface */ /* the nsd_ptcp object */ struct nsd_ptcp_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - uchar *pRemHostIP; /**< IP address of remote peer (currently used in server mode, only) */ + prop_t *remoteIP; /**< IP address of remote peer (currently used in server mode, only) */ uchar *pRemHostName; /**< host name of remote peer (currently used in server mode, only) */ struct sockaddr_storage remAddr; /**< remote addr as sockaddr - used for legacy ACL code */ int sock; /**< the socket we use for regular, single-socket, operations */ diff --git a/runtime/obj-types.h b/runtime/obj-types.h index da27a391..30a6a2c0 100644 --- a/runtime/obj-types.h +++ b/runtime/obj-types.h @@ -282,14 +282,12 @@ rsRetVal objName##ClassExit(void) \ rsRetVal OBJ##Destruct(OBJ##_t __attribute__((unused)) **ppThis) \ { \ DEFiRet; \ - int iCancelStateSave; \ OBJ##_t *pThis; #define CODESTARTobjDestruct(OBJ) \ ASSERT(ppThis != NULL); \ pThis = *ppThis; \ - ISOBJ_TYPE_assert(pThis, OBJ); \ - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + ISOBJ_TYPE_assert(pThis, OBJ); /* note: there was a long-time bug in the macro below that lead to *ppThis = NULL * only when the object was actually destructed. I discovered this issue during @@ -309,7 +307,6 @@ rsRetVal objName##ClassExit(void) \ free(pThis); \ } \ *ppThis = NULL; \ - pthread_setcancelstate(iCancelStateSave, NULL); \ RETiRet; \ } diff --git a/runtime/obj.c b/runtime/obj.c index eb151b67..63f1f38c 100644 --- a/runtime/obj.c +++ b/runtime/obj.c @@ -604,7 +604,7 @@ finalize_it: /* Deserialize a single property. Pointer must be positioned at begin of line. Whole line * up until the \n is read. */ -static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) +rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) { DEFiRet; number_t i; @@ -665,7 +665,7 @@ static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) if(c != '\n') ABORT_FINALIZE(RS_RET_INVALID_PROPFRAME); finalize_it: - if(Debug && iRet != RS_RET_OK) { + if(Debug && iRet != RS_RET_OK && iRet != RS_RET_NO_PROPLINE) { strm.GetCurrOffset(pStrm, &offs); dbgprintf("error %d deserializing property name, offset %lld, step %d\n", iRet, offs, step); @@ -767,21 +767,20 @@ finalize_it: * of the trailer. Header must already have been processed. * rgerhards, 2008-01-11 */ -static rsRetVal objDeserializeProperties(obj_t *pObj, objInfo_t *pObjInfo, strm_t *pStrm) +static rsRetVal objDeserializeProperties(obj_t *pObj, rsRetVal (*objSetProperty)(), strm_t *pStrm) { DEFiRet; var_t *pVar = NULL; ISOBJ_assert(pObj); ISOBJ_TYPE_assert(pStrm, strm); - ASSERT(pObjInfo != NULL); CHKiRet(var.Construct(&pVar)); CHKiRet(var.ConstructFinalize(pVar)); iRet = objDeserializeProperty(pVar, pStrm); while(iRet == RS_RET_OK) { - CHKiRet(pObjInfo->objMethods[objMethod_SETPROPERTY](pObj, pVar)); + CHKiRet(objSetProperty(pObj, pVar)); /* re-init var object - TODO: method of var! */ rsCStrDestruct(&pVar->pcsName); /* no longer needed */ if(pVar->varType == VARTYPE_STR) { @@ -848,7 +847,7 @@ Deserialize(void *ppObj, uchar *pszTypeExpected, strm_t *pStrm, rsRetVal (*fFixu CHKiRet(pObjInfo->objMethods[objMethod_CONSTRUCT](&pObj)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); /* check if we need to call a fixup function that modifies the object * before it is finalized. -- rgerhards, 2008-01-13 @@ -873,6 +872,104 @@ finalize_it: } +/* De-Serialize an object, with known constructur and destructor. Params like Deserialize(). + * rgerhards, 2012-11-03 + */ +rsRetVal +objDeserializeWithMethods(void *ppObj, uchar *pszTypeExpected, int lenTypeExpected, strm_t *pStrm, rsRetVal (*fFixup)(obj_t*,void*), void *pUsr, rsRetVal (*objConstruct)(), rsRetVal (*objConstructFinalize)(), rsRetVal (*objDeserialize)()) +{ + DEFiRet; + rsRetVal iRetLocal; + obj_t *pObj = NULL; + int oVers = 0; /* keep compiler happy, but it is totally useless but takes up some execution time... */ + cstr_t *pstrID = NULL; + + assert(ppObj != NULL); + assert(pszTypeExpected != NULL); + ISOBJ_TYPE_assert(pStrm, strm); + + /* we de-serialize the header. if all goes well, we are happy. However, if + * we experience a problem, we try to recover. We do this by skipping to + * the next object header. This is defined via the line-start cookies. In + * worst case, we exhaust the queue, but then we receive EOF return state, + * from objDeserializeTryRecover(), what will cause us to ultimately give up. + * rgerhards, 2008-07-08 + */ + do { + iRetLocal = objDeserializeHeader((uchar*) "Obj", &pstrID, &oVers, pStrm); + if(iRetLocal != RS_RET_OK) { + dbgprintf("objDeserialize error %d during header processing - " + "trying to recover\n", iRetLocal); + CHKiRet(objDeserializeTryRecover(pStrm)); + } + } while(iRetLocal != RS_RET_OK); + + if(rsCStrSzStrCmp(pstrID, pszTypeExpected, lenTypeExpected)) + ABORT_FINALIZE(RS_RET_INVALID_OID); + + CHKiRet(objConstruct(&pObj)); + + /* we got the object, now we need to fill the properties */ + CHKiRet(objDeserialize(pObj, pStrm)); + CHKiRet(objDeserializeTrailer(pStrm)); /* do trailer checks */ + + /* check if we need to call a fixup function that modifies the object + * before it is finalized. -- rgerhards, 2008-01-13 + */ + if(fFixup != NULL) + CHKiRet(fFixup(pObj, pUsr)); + + /* we have a valid object, let's finalize our work and return */ + if(objConstructFinalize != NULL) { + CHKiRet(objConstructFinalize(pObj)); + } + + *((obj_t**) ppObj) = pObj; + +finalize_it: + if(iRet != RS_RET_OK && pObj != NULL) + free(pObj); /* TODO: check if we can call destructor 2008-01-13 rger */ + + if(pstrID != NULL) + rsCStrDestruct(&pstrID); + + RETiRet; +} + +/* This is a dummy deserializer, to be used for the delete queue reader + * specifically. This is kind of a hack, but also to be replace (hopefully) soon + * by totally different code. So let's make it as simple as possible... + * rgerhards, 2012-11-06 + */ +rsRetVal +objDeserializeDummy(obj_t __attribute__((unused)) *pObj, strm_t *pStrm) +{ + DEFiRet; + var_t *pVar = NULL; + + CHKiRet(var.Construct(&pVar)); + CHKiRet(var.ConstructFinalize(pVar)); + + iRet = objDeserializeProperty(pVar, pStrm); + while(iRet == RS_RET_OK) { + /* this loop does actually NOGHTING but read the file... */ + /* re-init var object - TODO: method of var! */ + rsCStrDestruct(&pVar->pcsName); /* no longer needed */ + if(pVar->varType == VARTYPE_STR) { + if(pVar->val.pStr != NULL) + rsCStrDestruct(&pVar->val.pStr); + } + iRet = objDeserializeProperty(pVar, pStrm); + } +finalize_it: + if(iRet == RS_RET_NO_PROPLINE) + iRet = RS_RET_OK; /* NO_PROPLINE is OK and a kind of EOF! */ + if(pVar != NULL) + var.Destruct(&pVar); + RETiRet; +} + + /* De-Serialize an object, but treat it as property bag. * rgerhards, 2008-01-11 */ @@ -909,7 +1006,7 @@ objDeserializeObjAsPropBag(obj_t *pObj, strm_t *pStrm) CHKiRet(FindObjInfo(pstrID, &pObjInfo)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); finalize_it: if(pstrID != NULL) @@ -961,7 +1058,7 @@ DeserializePropBag(obj_t *pObj, strm_t *pStrm) CHKiRet(FindObjInfo(pstrID, &pObjInfo)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); finalize_it: if(pstrID != NULL) diff --git a/runtime/obj.h b/runtime/obj.h index 32f7ef09..27d32b7a 100644 --- a/runtime/obj.h +++ b/runtime/obj.h @@ -83,10 +83,7 @@ ((obj_t*) (pThis))->pObjInfo = pObjInfoOBJ; \ ((obj_t*) (pThis))->pszName = NULL #endif -#define objDestruct(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_DESTRUCT])(&pThis) #define objSerialize(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_SERIALIZE]) -#define objGetSeverity(pThis, piSever) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_GETSEVERITY])(pThis, piSever) -#define objDebugPrint(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_DEBUGPRINT])(pThis) #define OBJSetMethodHandler(methodID, pHdlr) \ CHKiRet(obj.InfoSetMethod(pObjInfoOBJ, methodID, (rsRetVal (*)(void*)) pHdlr)) @@ -121,6 +118,9 @@ ENDinterface(obj) rsRetVal objGetObjInterface(obj_if_t *pIf); PROTOTYPEObjClassInit(obj); PROTOTYPEObjClassExit(obj); +rsRetVal objDeserializeWithMethods(void *ppObj, uchar *pszTypeExpected, int lenTypeExpected, strm_t *pStrm, rsRetVal (*fFixup)(obj_t*,void*), void *pUsr, rsRetVal (*objConstruct)(), rsRetVal (*objConstructFinalize)(), rsRetVal (*objDeserialize)()); +rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm); +rsRetVal objDeserializeDummy(obj_t *pObj, strm_t *pStrm); /* the following definition is only for "friends" */ diff --git a/runtime/objomsr.c b/runtime/objomsr.c index 9cf3781b..e63eb681 100644 --- a/runtime/objomsr.c +++ b/runtime/objomsr.c @@ -42,9 +42,7 @@ rsRetVal OMSRdestruct(omodStringRequest_t *pThis) /* free the strings */ if(pThis->ppTplName != NULL) { for(i = 0 ; i < pThis->iNumEntries ; ++i) { - if(pThis->ppTplName[i] != NULL) { - free(pThis->ppTplName[i]); - } + free(pThis->ppTplName[i]); } free(pThis->ppTplName); } diff --git a/runtime/parser.c b/runtime/parser.c index 645ea0f4..74b28f4c 100644 --- a/runtime/parser.c +++ b/runtime/parser.c @@ -143,6 +143,14 @@ finalize_it: RETiRet; } +void +printParserList(parserList_t *pList) +{ + while(pList != NULL) { + dbgprintf("parser: %s\n", pList->pParser->pName); + pList = pList->pNext; + } +} /* find a parser based on the provided name */ static rsRetVal @@ -354,11 +362,10 @@ SanitizeMsg(msg_t *pMsg) */ int bNeedSanitize = 0; for(iSrc = 0 ; iSrc < lenMsg ; iSrc++) { - if(iscntrl(pszMsg[iSrc])) { + if(pszMsg[iSrc] < 32) { if(bSpaceLFOnRcv && pszMsg[iSrc] == '\n') pszMsg[iSrc] = ' '; - else - if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { + else if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { bNeedSanitize = 1; if (!bSpaceLFOnRcv) break; @@ -375,7 +382,9 @@ SanitizeMsg(msg_t *pMsg) FINALIZE; } - /* now copy over the message and sanitize it */ + /* now copy over the message and sanitize it. Note that up to iSrc-1 there was + * obviously no need to sanitize, so we can go over that quickly... + */ iMaxLine = glbl.GetMaxLine(); maxDest = lenMsg * 4; /* message can grow at most four-fold */ if(maxDest > iMaxLine) @@ -384,9 +393,13 @@ SanitizeMsg(msg_t *pMsg) pDst = szSanBuf; else CHKmalloc(pDst = MALLOC(sizeof(uchar) * (iMaxLine + 1))); - iSrc = iDst = 0; + if(iSrc > 0) { + iSrc--; /* go back to where everything is OK */ + memcpy(pDst, pszMsg, iSrc); /* fast copy known good */ + } + iDst = iSrc; while(iSrc < lenMsg && iDst < maxDest - 3) { /* leave some space if last char must be escaped */ - if(iscntrl((int) pszMsg[iSrc]) && (pszMsg[iSrc] != '\t' || bEscapeTab)) { + if((pszMsg[iSrc] < 32) && (pszMsg[iSrc] != '\t' || bEscapeTab)) { /* note: \0 must always be escaped, the rest of the code currently * can not handle it! -- rgerhards, 2009-08-26 */ diff --git a/runtime/parser.h b/runtime/parser.h index f214ba0c..87a6269e 100644 --- a/runtime/parser.h +++ b/runtime/parser.h @@ -62,6 +62,7 @@ BEGINinterface(parser) /* name must also be changed in ENDinterface macro! */ ENDinterface(parser) #define parserCURR_IF_VERSION 1 /* increment whenever you change the interface above! */ +void printParserList(parserList_t *pList); /* prototypes */ PROTOTYPEObj(parser); diff --git a/runtime/prop.c b/runtime/prop.c index 9d5927fd..cb89fac0 100644 --- a/runtime/prop.c +++ b/runtime/prop.c @@ -100,7 +100,7 @@ static int GetStringLen(prop_t *pThis) /* get string */ -static rsRetVal GetString(prop_t *pThis, uchar **ppsz, int *plen) +rsRetVal GetString(prop_t *pThis, uchar **ppsz, int *plen) { BEGINfunc ISOBJ_TYPE_assert(pThis, prop); diff --git a/runtime/prop.h b/runtime/prop.h index 40a35f9b..c7564e6b 100644 --- a/runtime/prop.h +++ b/runtime/prop.h @@ -52,6 +52,13 @@ ENDinterface(prop) #define propCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +/* get classic c-style string */ +static inline uchar * +propGetSzStr(prop_t *pThis) +{ + return(pThis->len < CONF_PROP_BUFSIZE) ? pThis->szVal.sz : pThis->szVal.psz; +} + /* prototypes */ PROTOTYPEObj(prop); diff --git a/runtime/queue.c b/runtime/queue.c index 09ba7e67..4c8d3ac5 100644 --- a/runtime/queue.c +++ b/runtime/queue.c @@ -59,7 +59,6 @@ #include "datetime.h" #include "unicode-helper.h" #include "statsobj.h" -#include "msg.h" /* TODO: remove once we remove MsgAddRef() call */ #ifdef OS_SOLARIS # include <sched.h> @@ -74,7 +73,7 @@ DEFobjCurrIf(datetime) DEFobjCurrIf(statsobj) /* forward-definitions */ -static inline rsRetVal doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr); +static inline rsRetVal doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg); static rsRetVal qqueueChkPersist(qqueue_t *pThis, int nUpdates); static rsRetVal RateLimiter(qqueue_t *pThis); static int qqueueChkStopWrkrDA(qqueue_t *pThis); @@ -83,7 +82,7 @@ static rsRetVal ConsumerDA(qqueue_t *pThis, wti_t *pWti); static rsRetVal batchProcessed(qqueue_t *pThis, wti_t *pWti); static rsRetVal qqueueMultiEnqObjNonDirect(qqueue_t *pThis, multi_submit_t *pMultiSub); static rsRetVal qqueueMultiEnqObjDirect(qqueue_t *pThis, multi_submit_t *pMultiSub); -static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr); +static rsRetVal qAddDirect(qqueue_t *pThis, msg_t *pMsg); static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qConstructDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis); @@ -127,11 +126,11 @@ static struct cnfparamblk pblk = }; /* debug aid */ -static void displayBatchState(batch_t *pBatch) +static inline void displayBatchState(batch_t *pBatch) { int i; for(i = 0 ; i < pBatch->nElem ; ++i) { - DBGPRINTF("displayBatchState %p[%d]: %d\n", pBatch, i, pBatch->pElem[i].state); + DBGPRINTF("displayBatchState %p[%d]: %d\n", pBatch, i, pBatch->eltState[i]); } } @@ -232,12 +231,19 @@ getQueueTypeName(queueType_t t) switch(t) { case QUEUETYPE_FIXED_ARRAY: r = "FixedArray"; + break; case QUEUETYPE_LINKEDLIST: r = "LinkedList"; + break; case QUEUETYPE_DISK: r = "Disk"; + break; case QUEUETYPE_DIRECT: r = "Direct"; + break; + default: + r = "invalid/unknown queue mode"; + break; } return r; } @@ -308,16 +314,16 @@ getLogicalQueueSize(qqueue_t *pThis) */ static inline void queueDrain(qqueue_t *pThis) { - void *pUsr; + msg_t *pMsg; ASSERT(pThis != NULL); BEGINfunc DBGOPRINT((obj_t*) pThis, "queue (type %d) will lose %d messages, destroying...\n", pThis->qType, pThis->iQueueSize); /* iQueueSize is not decremented by qDel(), so we need to do it ourselves */ while(ATOMIC_DEC_AND_FETCH(&pThis->iQueueSize, &pThis->mutQueueSize) > 0) { - pThis->qDeq(pThis, &pUsr); - if(pUsr != NULL) { - objDestruct(pUsr); + pThis->qDeq(pThis, &pMsg); + if(pMsg != NULL) { + msgDestruct(&pMsg); } pThis->qDel(pThis); } @@ -407,7 +413,7 @@ StartDA(qqueue_t *pThis) */ pThis->pqDA->pqParent = pThis; - CHKiRet(qqueueSetpUsr(pThis->pqDA, pThis->pUsr)); + CHKiRet(qqueueSetpAction(pThis->pqDA, pThis->pAction)); CHKiRet(qqueueSetsizeOnDiskMax(pThis->pqDA, pThis->sizeOnDiskMax)); CHKiRet(qqueueSetiDeqSlowdown(pThis->pqDA, pThis->iDeqSlowdown)); CHKiRet(qqueueSetMaxFileSize(pThis->pqDA, pThis->iMaxFileSize)); @@ -542,7 +548,7 @@ static rsRetVal qDestructFixedArray(qqueue_t *pThis) } -static rsRetVal qAddFixedArray(qqueue_t *pThis, void* in) +static rsRetVal qAddFixedArray(qqueue_t *pThis, msg_t* in) { DEFiRet; @@ -556,7 +562,7 @@ static rsRetVal qAddFixedArray(qqueue_t *pThis, void* in) } -static rsRetVal qDeqFixedArray(qqueue_t *pThis, void **out) +static rsRetVal qDeqFixedArray(qqueue_t *pThis, msg_t **out) { DEFiRet; @@ -617,7 +623,7 @@ static rsRetVal qDestructLinkedList(qqueue_t __attribute__((unused)) *pThis) RETiRet; } -static rsRetVal qAddLinkedList(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddLinkedList(qqueue_t *pThis, msg_t* pMsg) { qLinkedList_t *pEntry; DEFiRet; @@ -625,7 +631,7 @@ static rsRetVal qAddLinkedList(qqueue_t *pThis, void* pUsr) CHKmalloc((pEntry = (qLinkedList_t*) MALLOC(sizeof(qLinkedList_t)))); pEntry->pNext = NULL; - pEntry->pUsr = pUsr; + pEntry->pMsg = pMsg; if(pThis->tVars.linklist.pDelRoot == NULL) { pThis->tVars.linklist.pDelRoot = pThis->tVars.linklist.pDeqRoot = pThis->tVars.linklist.pLast = pEntry; @@ -643,14 +649,13 @@ finalize_it: } -static rsRetVal qDeqLinkedList(qqueue_t *pThis, obj_t **ppUsr) +static rsRetVal qDeqLinkedList(qqueue_t *pThis, msg_t **ppMsg) { qLinkedList_t *pEntry; DEFiRet; pEntry = pThis->tVars.linklist.pDeqRoot; - ISOBJ_TYPE_assert(pEntry->pUsr, msg); - *ppUsr = pEntry->pUsr; + *ppMsg = pEntry->pMsg; pThis->tVars.linklist.pDeqRoot = pEntry->pNext; RETiRet; @@ -740,18 +745,12 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) { DEFiRet; strm_t *psQIF = NULL; - uchar pszQIFNam[MAXFNAME]; - size_t lenQIFNam; struct stat stat_buf; ISOBJ_TYPE_assert(pThis, qqueue); - /* Construct file name */ - lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), "%s/%s.qi", - (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); - /* check if the file exists */ - if(stat((char*) pszQIFNam, &stat_buf) == -1) { + if(stat((char*) pThis->pszQIFNam, &stat_buf) == -1) { if(errno == ENOENT) { DBGOPRINT((obj_t*) pThis, "clean startup, no .qi file found\n"); ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); @@ -766,7 +765,7 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) CHKiRet(strm.Construct(&psQIF)); CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_READ)); CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.SetFName(psQIF, pThis->pszQIFNam, pThis->lenQIFNam)); CHKiRet(strm.ConstructFinalize(psQIF)); /* first, we try to read the property bag for ourselfs */ @@ -778,9 +777,7 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) CHKiRet(obj.Deserialize(&pThis->tVars.disk.pReadDel, (uchar*) "strm", psQIF, (rsRetVal(*)(obj_t*,void*))qqueueLoadPersStrmInfoFixup, pThis)); - /* create a duplicate for the read "pointer". - */ - + /* create a duplicate for the read "pointer". */ CHKiRet(strm.Dup(pThis->tVars.disk.pReadDel, &pThis->tVars.disk.pReadDeq)); CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pReadDeq, 0)); /* deq must NOT delete the files! */ CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pReadDeq)); @@ -799,7 +796,7 @@ finalize_it: strm.Destruct(&psQIF); if(iRet != RS_RET_OK) { - DBGOPRINT((obj_t*) pThis, "error %d reading .qi file - can not read persisted info (if any)\n", + DBGOPRINT((obj_t*) pThis, "state %d reading .qi file - can not read persisted info (if any)\n", iRet); } @@ -891,7 +888,7 @@ static rsRetVal qDestructDisk(qqueue_t *pThis) RETiRet; } -static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddDisk(qqueue_t *pThis, msg_t* pMsg) { DEFiRet; number_t nWriteCount; @@ -899,7 +896,7 @@ static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) ASSERT(pThis != NULL); CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, &nWriteCount)); - CHKiRet((objSerialize(pUsr))(pUsr, pThis->tVars.disk.pWrite)); + CHKiRet((objSerialize(pMsg))(pMsg, pThis->tVars.disk.pWrite)); CHKiRet(strm.Flush(pThis->tVars.disk.pWrite)); CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, NULL)); /* no more counting for now... */ @@ -909,7 +906,7 @@ static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) * the in-memory representation. The instance will be re-created upon * dequeue. -- rgerhards, 2008-07-09 */ - objDestruct(pUsr); + msgDestruct(&pMsg); DBGOPRINT((obj_t*) pThis, "write wrote %lld octets to disk, queue disk size now %lld octets, EnqOnly:%d\n", nWriteCount, pThis->tVars.disk.sizeOnDisk, pThis->bEnqOnly); @@ -919,43 +916,11 @@ finalize_it: } -static rsRetVal qDeqDisk(qqueue_t *pThis, void **ppUsr) -{ - DEFiRet; - iRet = obj.Deserialize(ppUsr, (uchar*) "msg", pThis->tVars.disk.pReadDeq, NULL, NULL); - RETiRet; -} - - -static rsRetVal qDelDisk(qqueue_t *pThis) +static rsRetVal qDeqDisk(qqueue_t *pThis, msg_t **ppMsg) { - obj_t *pDummyObj; /* we need to deserialize it... */ DEFiRet; - - int64 offsIn; - int64 offsOut; - - CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pReadDel, &offsIn)); - CHKiRet(obj.Deserialize(&pDummyObj, (uchar*) "msg", pThis->tVars.disk.pReadDel, NULL, NULL)); - objDestruct(pDummyObj); - CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pReadDel, &offsOut)); - - /* This time it is a bit tricky: we free disk space only upon file deletion. So we need - * to keep track of what we have read until we get an out-offset that is lower than the - * in-offset (which indicates file change). Then, we can subtract the whole thing from - * the on-disk size. -- rgerhards, 2008-01-30 - */ - if(offsIn < offsOut) { - pThis->tVars.disk.bytesRead += offsOut - offsIn; - } else { - pThis->tVars.disk.sizeOnDisk -= pThis->tVars.disk.bytesRead; - pThis->tVars.disk.bytesRead = offsOut; - DBGOPRINT((obj_t*) pThis, "a file has been deleted, now %lld octets disk space used\n", pThis->tVars.disk.sizeOnDisk); - /* awake possibly waiting enq process */ - pthread_cond_signal(&pThis->notFull); /* we hold the mutex while we are in here! */ - } - -finalize_it: + iRet = objDeserializeWithMethods(ppMsg, (uchar*) "msg", 3, pThis->tVars.disk.pReadDeq, NULL, + NULL, msgConstructForDeserializer, NULL, MsgDeserialize); RETiRet; } @@ -972,10 +937,12 @@ static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis) return RS_RET_OK; } -static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddDirect(qqueue_t *pThis, msg_t* pMsg) { batch_t singleBatch; batch_obj_t batchObj; + batch_state_t batchState = BATCH_STATE_RDY; + sbool active = 1; int i; DEFiRet; @@ -992,17 +959,17 @@ static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr) */ memset(&batchObj, 0, sizeof(batch_obj_t)); memset(&singleBatch, 0, sizeof(batch_t)); - batchObj.state = BATCH_STATE_RDY; - batchObj.pUsrp = (obj_t*) pUsr; - batchObj.bFilterOK = 1; + batchObj.pMsg = pMsg; singleBatch.nElem = 1; /* there always is only one in direct mode */ singleBatch.pElem = &batchObj; - iRet = pThis->pConsumer(pThis->pUsr, &singleBatch, &pThis->bShutdownImmediate); + singleBatch.eltState = &batchState; + singleBatch.active = &active; + iRet = pThis->pConsumer(pThis->pAction, &singleBatch, &pThis->bShutdownImmediate); /* delete the batch string params: TODO: create its own "class" for this */ for(i = 0 ; i < CONF_OMOD_NUMSTRINGS_MAXSIZE ; ++i) { free(batchObj.staticActStrings[i]); } - objDestruct(pUsr); + msgDestruct(&pMsg); RETiRet; } @@ -1024,7 +991,7 @@ rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch) * We use our knowledge about the batch_t structure below, but without that, we * pay a too-large performance toll... -- rgerhards, 2009-04-22 */ - iRet = pThis->pConsumer(pThis->pUsr, pBatch, &pThis->bShutdownImmediate); + iRet = pThis->pConsumer(pThis->pAction, pBatch, &pThis->bShutdownImmediate); RETiRet; } @@ -1045,13 +1012,13 @@ static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis) * things truely different. -- rgerhards, 2008-02-12 */ static rsRetVal -qqueueAdd(qqueue_t *pThis, void *pUsr) +qqueueAdd(qqueue_t *pThis, msg_t *pMsg) { DEFiRet; ASSERT(pThis != NULL); - CHKiRet(pThis->qAdd(pThis, pUsr)); + CHKiRet(pThis->qAdd(pThis, pMsg)); if(pThis->qType != QUEUETYPE_DIRECT) { ATOMIC_INC(&pThis->iQueueSize, &pThis->mutQueueSize); @@ -1067,7 +1034,7 @@ finalize_it: /* generic code to dequeue a queue entry */ static rsRetVal -qqueueDeq(qqueue_t *pThis, void **ppUsr) +qqueueDeq(qqueue_t *pThis, msg_t **ppMsg) { DEFiRet; @@ -1078,7 +1045,7 @@ qqueueDeq(qqueue_t *pThis, void **ppUsr) * If we decrement, however, we may lose a message. But that is better than * losing the whole process because it loops... -- rgerhards, 2008-01-03 */ - iRet = pThis->qDeq(pThis, ppUsr); + iRet = pThis->qDeq(pThis, ppMsg); ATOMIC_INC(&pThis->nLogDeq, &pThis->mutLogDeq); // DBGOPRINT((obj_t*) pThis, "entry deleted, size now log %d, phys %d entries\n", @@ -1176,11 +1143,11 @@ tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) rsRetVal iRetLocal; DEFiRet; -RUNLOG_STR("trying to shutdown workers within Action Timeout"); ISOBJ_TYPE_assert(pThis, qqueue); ASSERT(pThis->pqParent == NULL); /* detect invalid calling sequence */ /* instruct workers to finish ASAP, even if still work exists */ + DBGOPRINT((obj_t*) pThis, "trying to shutdown workers within Action Timeout"); DBGOPRINT((obj_t*) pThis, "setting EnqOnly mode\n"); pThis->bEnqOnly = 1; pThis->bShutdownImmediate = 1; @@ -1374,7 +1341,7 @@ finalize_it: } -/* set default inisde queue object suitable for action queues. +/* set default inside queue object suitable for action queues. * This shall be called directly after queue construction. This functions has * been added in support of the new v6 config system. It expect properly pre-initialized * objects, but we need to differentiate between ruleset main and action queues. @@ -1408,6 +1375,36 @@ qqueueSetDefaultsActionQueue(qqueue_t *pThis) } +/* set defaults inside queue object suitable for main/ruleset queues. + * See queueSetDefaultsActionQueue() for more details and background. + */ +void +qqueueSetDefaultsRulesetQueue(qqueue_t *pThis) +{ + pThis->qType = QUEUETYPE_FIXED_ARRAY; /* type of the main message queue above */ + pThis->iMaxQueueSize = 50000; /* size of the main message queue above */ + pThis->iDeqBatchSize = 1024; /* default batch size */ + pThis->iHighWtrMrk = 45000; /* high water mark for disk-assisted queues */ + pThis->iLowWtrMrk = 20000; /* low water mark for disk-assisted queues */ + pThis->iDiscardMrk = 49500; /* begin to discard messages */ + pThis->iDiscardSeverity = 8; /* turn off */ + pThis->iNumWorkerThreads = 1; /* number of worker threads for the mm queue above */ + pThis->iMaxFileSize = 16*1024*1024; + pThis->iPersistUpdCnt = 0; /* persist queue info every n updates */ + pThis->bSyncQueueFiles = 0; + pThis->toQShutdown = 1500; /* queue shutdown */ + pThis->toActShutdown = 1000; /* action shutdown (in phase 2) */ + pThis->toEnq = 2000; /* timeout for queue enque */ + pThis->toWrkShutdown = 60000; /* timeout for worker thread shutdown */ + pThis->iMinMsgsPerWrkr = 1000; /* minimum messages per worker needed to start a new one */ + pThis->bSaveOnShutdown = 1; /* save queue on shutdown (when DA enabled)? */ + pThis->sizeOnDiskMax = 0; /* unlimited */ + pThis->iDeqSlowdown = 0; + pThis->iDeqtWinFromHr = 0; + pThis->iDeqtWinToHr = 25; /* disable time-windowed dequeuing by default */ +} + + /* This function checks if the provided message shall be discarded and does so, if needed. * In DA mode, we do not discard any messages as we assume the disk subsystem is fast enough to * provide real-time creation of spool files. @@ -1421,22 +1418,21 @@ qqueueSetDefaultsActionQueue(qqueue_t *pThis) * the return state! * rgerhards, 2008-01-24 */ -static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, void *pUsr) +static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, msg_t *pMsg) { DEFiRet; rsRetVal iRetLocal; int iSeverity; ISOBJ_TYPE_assert(pThis, qqueue); - ISOBJ_assert(pUsr); if(pThis->iDiscardMrk > 0 && iQueueSize >= pThis->iDiscardMrk) { - iRetLocal = objGetSeverity(pUsr, &iSeverity); + iRetLocal = MsgGetSeverity(pMsg, &iSeverity); if(iRetLocal == RS_RET_OK && iSeverity >= pThis->iDiscardSeverity) { DBGOPRINT((obj_t*) pThis, "queue nearly full (%d entries), discarded severity %d message\n", iQueueSize, iSeverity); STATSCOUNTER_INC(pThis->ctrNFDscrd, pThis->mutCtrNFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } else { DBGOPRINT((obj_t*) pThis, "queue nearly full (%d entries), but could not drop msg " @@ -1455,13 +1451,32 @@ static inline rsRetVal DoDeleteBatchFromQStore(qqueue_t *pThis, int nElem) { int i; + off64_t bytesDel; DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); /* now send delete request to storage driver */ - for(i = 0 ; i < nElem ; ++i) { - pThis->qDel(pThis); + if(pThis->qType == QUEUETYPE_DISK) { + strmMultiFileSeek(pThis->tVars.disk.pReadDel, pThis->tVars.disk.deqFileNumOut, + pThis->tVars.disk.deqOffs, &bytesDel); + /* We need to correct the on-disk file size. This time it is a bit tricky: + * we free disk space only upon file deletion. So we need to keep track of what we + * have read until we get an out-offset that is lower than the in-offset (which + * indicates file change). Then, we can subtract the whole thing from the on-disk + * size. -- rgerhards, 2008-01-30 + */ + if(bytesDel != 0) { + pThis->tVars.disk.sizeOnDisk -= bytesDel; + DBGOPRINT((obj_t*) pThis, "a %lld octet file has been deleted, now %lld octets disk " + "space used\n", bytesDel, pThis->tVars.disk.sizeOnDisk); + /* awake possibly waiting enq process */ + pthread_cond_signal(&pThis->notFull); /* we hold the mutex while we are in here! */ + } + } else { /* memory queue */ + for(i = 0 ; i < nElem ; ++i) { + pThis->qDel(pThis); + } } /* iQueueSize is not decremented by qDel(), so we need to do it ourselves */ @@ -1522,7 +1537,7 @@ static inline rsRetVal DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) { int i; - void *pUsr; + msg_t *pMsg; int nEnqueued = 0; rsRetVal localRet; DEFiRet; @@ -1531,17 +1546,16 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) assert(pBatch != NULL); for(i = 0 ; i < pBatch->nElem ; ++i) { - pUsr = pBatch->pElem[i].pUsrp; - if( pBatch->pElem[i].state == BATCH_STATE_RDY - || pBatch->pElem[i].state == BATCH_STATE_SUB) { - localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, - (obj_t*)MsgAddRef((msg_t*) pUsr)); + pMsg = pBatch->pElem[i].pMsg; + if( pBatch->eltState[i] == BATCH_STATE_RDY + || pBatch->eltState[i] == BATCH_STATE_SUB) { + localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); ++nEnqueued; if(localRet != RS_RET_OK) { DBGPRINTF("error %d re-enqueuing unprocessed data element - discarded\n", localRet); } } - objDestruct(pUsr); + msgDestruct(&pMsg); } DBGPRINTF("we deleted %d objects and enqueued %d objects\n", i-nEnqueued, nEnqueued); @@ -1573,7 +1587,7 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz int nDiscarded; int nDeleted; int iQueueSize; - void *pUsr; + msg_t *pMsg; rsRetVal localRet; DEFiRet; @@ -1581,11 +1595,14 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz DeleteProcessedBatch(pThis, &pWti->batch); nDequeued = nDiscarded = 0; + if(pThis->qType == QUEUETYPE_DISK) { + pThis->tVars.disk.deqFileNumIn = strmGetCurrFileNum(pThis->tVars.disk.pReadDeq); + } while((iQueueSize = getLogicalQueueSize(pThis)) > 0 && nDequeued < pThis->iDeqBatchSize) { - CHKiRet(qqueueDeq(pThis, &pUsr)); + CHKiRet(qqueueDeq(pThis, &pMsg)); /* check if we should discard this element */ - localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr); + localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pMsg); if(localRet == RS_RET_QUEUE_FULL) { ++nDiscarded; continue; @@ -1594,12 +1611,16 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz } /* all well, use this element */ - pWti->batch.pElem[nDequeued].pUsrp = pUsr; - pWti->batch.pElem[nDequeued].state = BATCH_STATE_RDY; - pWti->batch.pElem[nDequeued].bFilterOK = 1; // TODO: think again if we can handle that with more performance + pWti->batch.pElem[nDequeued].pMsg = pMsg; + pWti->batch.eltState[nDequeued] = BATCH_STATE_RDY; ++nDequeued; } + if(pThis->qType == QUEUETYPE_DISK) { + strm.GetCurrOffset(pThis->tVars.disk.pReadDeq, &pThis->tVars.disk.deqOffs); + pThis->tVars.disk.deqFileNumOut = strmGetCurrFileNum(pThis->tVars.disk.pReadDeq); + } + /* it is sufficient to persist only when the bulk of work is done */ qqueueChkPersist(pThis, nDequeued+nDiscarded+nDeleted); @@ -1607,7 +1628,6 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz pWti->batch.nElemDeq = nDequeued + nDiscarded; pWti->batch.deqID = getNextDeqID(pThis); *piRemainingQueueSize = iQueueSize; - finalize_it: RETiRet; } @@ -1643,7 +1663,6 @@ DequeueConsumable(qqueue_t *pThis, wti_t *pWti) pthread_cond_broadcast(&pThis->belowLightDlyWtrMrk); } - // TODO: MULTI: check physical queue size? pthread_cond_signal(&pThis->notFull); /* WE ARE NO LONGER PROTECTED BY THE MUTEX */ @@ -1837,7 +1856,7 @@ ConsumerReg(qqueue_t *pThis, wti_t *pWti) /* at this spot, we may be cancelled */ pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &iCancelStateSave); - CHKiRet(pThis->pConsumer(pThis->pUsr, &pWti->batch, &pThis->bShutdownImmediate)); + CHKiRet(pThis->pConsumer(pThis->pAction, &pWti->batch, &pThis->bShutdownImmediate)); /* we now need to check if we should deliberately delay processing a bit * and, if so, do that. -- rgerhards, 2008-01-30 @@ -1895,13 +1914,9 @@ ConsumerDA(qqueue_t *pThis, wti_t *pWti) /* iterate over returned results and enqueue them in DA queue */ for(i = 0 ; i < pWti->batch.nElem && !pThis->bShutdownImmediate ; i++) { - /* TODO: we must add a generic "addRef" mechanism, because the disk queue enqueue destructs - * the message. So far, we simply assume we always have msg_t, what currently is always the case. - * rgerhards, 2009-05-28 - */ - CHKiRet(qqueueEnqObj(pThis->pqDA, eFLOWCTL_NO_DELAY, - (obj_t*)MsgAddRef((msg_t*)(pWti->batch.pElem[i].pUsrp)))); - pWti->batch.pElem[i].state = BATCH_STATE_COMM; /* commited to other queue! */ + CHKiRet(qqueueEnqMsg(pThis->pqDA, eFLOWCTL_NO_DELAY, + MsgAddRef(pWti->batch.pElem[i].pMsg))); + pWti->batch.eltState[i] = BATCH_STATE_COMM; /* commited to other queue! */ } /* but now cancellation is no longer permitted */ @@ -1981,6 +1996,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ { DEFiRet; uchar pszBuf[64]; + uchar pszQIFNam[MAXFNAME]; int wrk; uchar *qName; size_t lenBuf; @@ -2003,8 +2019,8 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->qConstruct = qConstructLinkedList; pThis->qDestruct = qDestructLinkedList; pThis->qAdd = qAddLinkedList; - pThis->qDeq = (rsRetVal (*)(qqueue_t*,void**)) qDeqLinkedList; - pThis->qDel = (rsRetVal (*)(qqueue_t*)) qDelLinkedList; + pThis->qDeq = qDeqLinkedList; + pThis->qDel = qDelLinkedList; pThis->MultiEnq = qqueueMultiEnqObjNonDirect; break; case QUEUETYPE_DISK: @@ -2012,10 +2028,16 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->qDestruct = qDestructDisk; pThis->qAdd = qAddDisk; pThis->qDeq = qDeqDisk; - pThis->qDel = qDelDisk; + pThis->qDel = NULL; /* delete for disk handled via special code! */ pThis->MultiEnq = qqueueMultiEnqObjNonDirect; /* special handling */ pThis->iNumWorkerThreads = 1; /* we need exactly one worker */ + /* pre-construct file name for .qi file */ + pThis->lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), + "%s/%s.qi", (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); + pThis->pszQIFNam = ustrdup(pszQIFNam); + DBGOPRINT((obj_t*) pThis, ".qi file name is '%s', len %d\n", pThis->pszQIFNam, + (int) pThis->lenQIFNam); break; case QUEUETYPE_DIRECT: pThis->qConstruct = qConstructDirect; @@ -2075,6 +2097,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->pqParent == NULL ? 0 : 1, pThis->iFullDlyMrk, pThis->iLightDlyMrk, pThis->iDeqBatchSize); + pThis->bQueueStarted = 1; if(pThis->qType == QUEUETYPE_DIRECT) FINALIZE; /* with direct queues, we are already finished... */ @@ -2105,7 +2128,6 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ * the case when a disk queue has been loaded. If we did not start it here, it would never start. */ qqueueAdviseMaxWorkers(pThis); - pThis->bQueueStarted = 1; /* support statistics gathering */ qName = obj.GetName((obj_t*)pThis); @@ -2142,7 +2164,7 @@ finalize_it: } -/* persist the queue to disk. If we have something to persist, we first +/* persist the queue to disk (write the .qi file). If we have something to persist, we first * save the information on the queue properties itself and then we call * the queue-type specific drivers. * Variable bIsCheckpoint is set to 1 if the persist is for a checkpoint, @@ -2153,8 +2175,6 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) { DEFiRet; strm_t *psQIF = NULL; /* Queue Info File */ - uchar pszQIFNam[MAXFNAME]; - size_t lenQIFNam; ASSERT(pThis != NULL); @@ -2172,13 +2192,9 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) DBGOPRINT((obj_t*) pThis, "persisting queue to disk, %d entries...\n", getPhysicalQueueSize(pThis)); - /* Construct file name */ - lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), "%s/%s.qi", - (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); - if((bIsCheckpoint != QUEUE_CHECKPOINT) && (getPhysicalQueueSize(pThis) == 0)) { if(pThis->bNeedDelQIF) { - unlink((char*)pszQIFNam); + unlink((char*)pThis->pszQIFNam); pThis->bNeedDelQIF = 0; } /* indicate spool file needs to be deleted */ @@ -2191,7 +2207,7 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_WRITE_TRUNC)); CHKiRet(strm.SetbSync(psQIF, pThis->bSyncQueueFiles)); CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.SetFName(psQIF, pThis->pszQIFNam, pThis->lenQIFNam)); CHKiRet(strm.ConstructFinalize(psQIF)); /* first, write the property bag for ourselfs @@ -2203,7 +2219,6 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) CHKiRet(obj.BeginSerializePropBag(psQIF, (obj_t*) pThis)); objSerializeSCALAR(psQIF, iQueueSize, INT); objSerializeSCALAR(psQIF, tVars.disk.sizeOnDisk, INT64); - objSerializeSCALAR(psQIF, tVars.disk.bytesRead, INT64); CHKiRet(obj.EndSerialize(psQIF)); /* now persist the stream info */ @@ -2303,73 +2318,75 @@ DoSaveOnShutdown(qqueue_t *pThis) /* destructor for the queue object */ BEGINobjDestruct(qqueue) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(qqueue) - /* shut down all workers - * We do not need to shutdown workers when we are in enqueue-only mode or we are a - * direct queue - because in both cases we have none... ;) - * with a child! -- rgerhards, 2008-01-28 - */ - if(pThis->qType != QUEUETYPE_DIRECT && !pThis->bEnqOnly && pThis->pqParent == NULL - && pThis->pWtpReg != NULL) - ShutdownWorkers(pThis); + if(pThis->bQueueStarted) { + /* shut down all workers + * We do not need to shutdown workers when we are in enqueue-only mode or we are a + * direct queue - because in both cases we have none... ;) + * with a child! -- rgerhards, 2008-01-28 + */ + if(pThis->qType != QUEUETYPE_DIRECT && !pThis->bEnqOnly && pThis->pqParent == NULL + && pThis->pWtpReg != NULL) + ShutdownWorkers(pThis); - if(pThis->bIsDA && getPhysicalQueueSize(pThis) > 0 && pThis->bSaveOnShutdown) { - CHKiRet(DoSaveOnShutdown(pThis)); - } + if(pThis->bIsDA && getPhysicalQueueSize(pThis) > 0 && pThis->bSaveOnShutdown) { + CHKiRet(DoSaveOnShutdown(pThis)); + } - /* finally destruct our (regular) worker thread pool - * Note: currently pWtpReg is never NULL, but if we optimize our logic, this may happen, - * e.g. when they are not created in enqueue-only mode. We already check the condition - * as this may otherwise be very hard to find once we optimize (and have long forgotten - * about this condition here ;) - * rgerhards, 2008-01-25 - */ - if(pThis->qType != QUEUETYPE_DIRECT && pThis->pWtpReg != NULL) { - wtpDestruct(&pThis->pWtpReg); - } + /* finally destruct our (regular) worker thread pool + * Note: currently pWtpReg is never NULL, but if we optimize our logic, this may happen, + * e.g. when they are not created in enqueue-only mode. We already check the condition + * as this may otherwise be very hard to find once we optimize (and have long forgotten + * about this condition here ;) + * rgerhards, 2008-01-25 + */ + if(pThis->qType != QUEUETYPE_DIRECT && pThis->pWtpReg != NULL) { + wtpDestruct(&pThis->pWtpReg); + } - /* Now check if we actually have a DA queue and, if so, destruct it. - * Note that the wtp must be destructed first, it may be in cancel cleanup handler - * *right now* and actually *need* to access the queue object to persist some final - * data (re-queueing case). So we need to destruct the wtp first, which will make - * sure all workers have terminated. Please note that this also generates a situation - * where it is possible that the DA queue has a parent pointer but the parent has - * no WtpDA associated with it - which is perfectly legal thanks to this code here. - */ - if(pThis->pWtpDA != NULL) { - wtpDestruct(&pThis->pWtpDA); - } - if(pThis->pqDA != NULL) { - qqueueDestruct(&pThis->pqDA); - } + /* Now check if we actually have a DA queue and, if so, destruct it. + * Note that the wtp must be destructed first, it may be in cancel cleanup handler + * *right now* and actually *need* to access the queue object to persist some final + * data (re-queueing case). So we need to destruct the wtp first, which will make + * sure all workers have terminated. Please note that this also generates a situation + * where it is possible that the DA queue has a parent pointer but the parent has + * no WtpDA associated with it - which is perfectly legal thanks to this code here. + */ + if(pThis->pWtpDA != NULL) { + wtpDestruct(&pThis->pWtpDA); + } + if(pThis->pqDA != NULL) { + qqueueDestruct(&pThis->pqDA); + } - /* persist the queue (we always do that - queuePersits() does cleanup if the queue is empty) - * This handler is most important for disk queues, it will finally persist the necessary - * on-disk structures. In theory, other queueing modes may implement their other (non-DA) - * methods of persisting a queue between runs, but in practice all of this is done via - * disk queues and DA mode. Anyhow, it doesn't hurt to know that we could extend it here - * if need arises (what I doubt...) -- rgerhards, 2008-01-25 - */ - CHKiRet_Hdlr(qqueuePersist(pThis, QUEUE_NO_CHECKPOINT)) { - DBGOPRINT((obj_t*) pThis, "error %d persisting queue - data lost!\n", iRet); - } + /* persist the queue (we always do that - queuePersits() does cleanup if the queue is empty) + * This handler is most important for disk queues, it will finally persist the necessary + * on-disk structures. In theory, other queueing modes may implement their other (non-DA) + * methods of persisting a queue between runs, but in practice all of this is done via + * disk queues and DA mode. Anyhow, it doesn't hurt to know that we could extend it here + * if need arises (what I doubt...) -- rgerhards, 2008-01-25 + */ + CHKiRet_Hdlr(qqueuePersist(pThis, QUEUE_NO_CHECKPOINT)) { + DBGOPRINT((obj_t*) pThis, "error %d persisting queue - data lost!\n", iRet); + } - /* finally, clean up some simple things... */ - if(pThis->pqParent == NULL) { - /* if we are not a child, we allocated our own mutex, which we now need to destroy */ - pthread_mutex_destroy(pThis->mut); - free(pThis->mut); - } - pthread_mutex_destroy(&pThis->mutThrdMgmt); - pthread_cond_destroy(&pThis->notFull); - pthread_cond_destroy(&pThis->notEmpty); - pthread_cond_destroy(&pThis->belowFullDlyWtrMrk); - pthread_cond_destroy(&pThis->belowLightDlyWtrMrk); + /* finally, clean up some simple things... */ + if(pThis->pqParent == NULL) { + /* if we are not a child, we allocated our own mutex, which we now need to destroy */ + pthread_mutex_destroy(pThis->mut); + free(pThis->mut); + } + pthread_mutex_destroy(&pThis->mutThrdMgmt); + pthread_cond_destroy(&pThis->notFull); + pthread_cond_destroy(&pThis->notEmpty); + pthread_cond_destroy(&pThis->belowFullDlyWtrMrk); + pthread_cond_destroy(&pThis->belowLightDlyWtrMrk); - DESTROY_ATOMIC_HELPER_MUT(pThis->mutQueueSize); - DESTROY_ATOMIC_HELPER_MUT(pThis->mutLogDeq); + DESTROY_ATOMIC_HELPER_MUT(pThis->mutQueueSize); + DESTROY_ATOMIC_HELPER_MUT(pThis->mutLogDeq); - /* type-specific destructor */ - iRet = pThis->qDestruct(pThis); + /* type-specific destructor */ + iRet = pThis->qDestruct(pThis); + } free(pThis->pszFilePrefix); free(pThis->pszSpoolDir); @@ -2431,7 +2448,7 @@ finalize_it: * rgerhards, 2009-06-16 */ static inline rsRetVal -doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) +doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg) { DEFiRet; int err; @@ -2440,7 +2457,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) STATSCOUNTER_INC(pThis->ctrEnqueued, pThis->mutCtrEnqueued); /* first check if we need to discard this message (which will cause CHKiRet() to exit) */ - CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr)); + CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pMsg)); /* handle flow control * There are two different flow control mechanisms: basic and advanced flow control. @@ -2517,7 +2534,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) if(pThis->toEnq == 0 || pThis->bEnqOnly) { DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL - configured for immediate discarding.\n"); STATSCOUNTER_INC(pThis->ctrFDscrd, pThis->mutCtrFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } else { DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL - waiting %dms to drain.\n", pThis->toEnq); @@ -2529,7 +2546,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) if(pthread_cond_timedwait(&pThis->notFull, pThis->mut, &t) != 0) { DBGOPRINT((obj_t*) pThis, "enqueueMsg: cond timeout, dropping message!\n"); STATSCOUNTER_INC(pThis->ctrFDscrd, pThis->mutCtrFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } dbgoprint((obj_t*) pThis, "enqueueMsg: wait solved queue full condition, enqueing\n"); @@ -2537,7 +2554,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) } /* and finally enqueue the message */ - CHKiRet(qqueueAdd(pThis, pUsr)); + CHKiRet(qqueueAdd(pThis, pMsg)); STATSCOUNTER_SETMAX_NOMUT(pThis->ctrMaxqsize, pThis->iQueueSize); finalize_it: @@ -2613,11 +2630,11 @@ finalize_it: * Enqueues the new element and awakes worker thread. */ rsRetVal -qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr) +qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg) { DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); - iRet = qAddDirect(pThis, pUsr); + iRet = qAddDirect(pThis, pMsg); RETiRet; } @@ -2626,7 +2643,7 @@ qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr) * Enqueues the new element and awakes worker thread. */ rsRetVal -qqueueEnqObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) +qqueueEnqMsg(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg) { DEFiRet; int iCancelStateSave; @@ -2638,7 +2655,7 @@ qqueueEnqObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) d_pthread_mutex_lock(pThis->mut); } - CHKiRet(doEnqSingleObj(pThis, flowCtlType, pUsr)); + CHKiRet(doEnqSingleObj(pThis, flowCtlType, pMsg)); qqueueChkPersist(pThis, 1); @@ -2669,6 +2686,15 @@ qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals) return RS_RET_OK; } + +/* are any queue params set at all? 1 - yes, 0 - no */ +int +queueCnfParamsSet(struct cnfparamvals *pvals) +{ + return cnfparamvalsIsSet(&pblk, pvals); +} + + /* apply all params from param block to queue. Must be called before * finalizing. This supports the v6 config system. Defaults were already * set during queue creation. The pvals object is destructed by this @@ -2758,7 +2784,7 @@ DEFpropSetMeth(qqueue, iLightDlyMrk, int) DEFpropSetMeth(qqueue, bIsDA, int) DEFpropSetMeth(qqueue, iMinMsgsPerWrkr, int) DEFpropSetMeth(qqueue, bSaveOnShutdown, int) -DEFpropSetMeth(qqueue, pUsr, void*) +DEFpropSetMeth(qqueue, pAction, action_t*) DEFpropSetMeth(qqueue, iDeqSlowdown, int) DEFpropSetMeth(qqueue, iDeqBatchSize, int) DEFpropSetMeth(qqueue, sizeOnDiskMax, int64) @@ -2781,8 +2807,6 @@ static rsRetVal qqueueSetProperty(qqueue_t *pThis, var_t *pProp) pThis->iQueueSize = pProp->val.num; } else if(isProp("tVars.disk.sizeOnDisk")) { pThis->tVars.disk.sizeOnDisk = pProp->val.num; - } else if(isProp("tVars.disk.bytesRead")) { - pThis->tVars.disk.bytesRead = pProp->val.num; } else if(isProp("qType")) { if(pThis->qType != pProp->val.num) ABORT_FINALIZE(RS_RET_QTYPE_MISMATCH); diff --git a/runtime/queue.h b/runtime/queue.h index edb770c6..886fac8d 100644 --- a/runtime/queue.h +++ b/runtime/queue.h @@ -51,7 +51,7 @@ typedef enum { /* list member definition for linked list types of queues: */ typedef struct qLinkedList_S { struct qLinkedList_S *pNext; - void *pUsr; + msg_t *pMsg; } qLinkedList_t; @@ -71,7 +71,7 @@ struct queue_s { int iMinMsgsPerWrkr;/* minimum nbr of msgs per worker thread, if more, a new worker is started until max wrkrs */ wtp_t *pWtpDA; wtp_t *pWtpReg; - void *pUsr; /* a global, user-supplied pointer. Is passed back to consumer. */ + action_t *pAction; /* for action queues, ptr to action object; for main queues unused */ int iUpdsSincePersist;/* nbr of queue updates since the last persist call */ int iPersistUpdCnt; /* persits queue info after this nbr of updates - 0 -> persist only on shutdown */ sbool bSyncQueueFiles;/* if working with files, sync them after each write? */ @@ -111,8 +111,8 @@ struct queue_s { /* type-specific handlers (set during construction) */ rsRetVal (*qConstruct)(struct queue_s *pThis); rsRetVal (*qDestruct)(struct queue_s *pThis); - rsRetVal (*qAdd)(struct queue_s *pThis, void *pUsr); - rsRetVal (*qDeq)(struct queue_s *pThis, void **ppUsr); + rsRetVal (*qAdd)(struct queue_s *pThis, msg_t *pMsg); + rsRetVal (*qDeq)(struct queue_s *pThis, msg_t **ppMsg); rsRetVal (*qDel)(struct queue_s *pThis); /* end type-specific handler */ /* public entry points (set during construction, permit to set best algorithm for params selected) */ @@ -135,6 +135,8 @@ struct queue_s { size_t lenSpoolDir; uchar *pszFilePrefix; size_t lenFilePrefix; + uchar *pszQIFNam; /* full .qi file name, based on parts above */ + size_t lenQIFNam; int iNumberFiles; /* how many files make up the queue? */ int64 iMaxFileSize; /* max size for a single queue file */ int64 sizeOnDiskMax; /* maximum size on disk allowed */ @@ -145,7 +147,8 @@ struct queue_s { struct queue_s *pqParent;/* pointer to the parent (if this is a child queue) */ int bDAEnqOnly; /* EnqOnly setting for DA queue */ /* now follow queueing mode specific data elements */ - union { /* different data elements based on queue type (qType) */ + //union { /* different data elements based on queue type (qType) */ + struct { /* different data elements based on queue type (qType) */ struct { long deqhead, head, tail; void** pBuf; /* the queued user data structure */ @@ -157,7 +160,9 @@ struct queue_s { } linklist; struct { int64 sizeOnDisk; /* current amount of disk space used */ - int64 bytesRead; /* number of bytes read from current (undeleted!) file */ + int64 deqOffs; /* offset after dequeue batch - used for file deleter */ + int deqFileNumIn; /* same for the circular file numbers, mainly for */ + int deqFileNumOut;/* deleting finished files */ strm_t *pWrite; /* current file to be written */ strm_t *pReadDeq; /* current file for dequeueing */ strm_t *pReadDel; /* current file for deleting */ @@ -184,8 +189,8 @@ struct queue_s { /* prototypes */ rsRetVal qqueueDestruct(qqueue_t **ppThis); -rsRetVal qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr); -rsRetVal qqueueEnqObj(qqueue_t *pThis, flowControl_t flwCtlType, void *pUsr); +rsRetVal qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg); +rsRetVal qqueueEnqMsg(qqueue_t *pThis, flowControl_t flwCtlType, msg_t *pMsg); rsRetVal qqueueStart(qqueue_t *pThis); rsRetVal qqueueSetMaxFileSize(qqueue_t *pThis, size_t iMaxFileSize); rsRetVal qqueueSetFilePrefix(qqueue_t *pThis, uchar *pszPrefix, size_t iLenPrefix); @@ -193,7 +198,9 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*, int*)); rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch); rsRetVal qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals); +int queueCnfParamsSet(struct cnfparamvals *pvals); rsRetVal qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals); +void qqueueSetDefaultsRulesetQueue(qqueue_t *pThis); void qqueueSetDefaultsActionQueue(qqueue_t *pThis); void qqueueDbgPrint(qqueue_t *pThis); @@ -213,7 +220,7 @@ PROTOTYPEpropSetMeth(qqueue, iDiscardMrk, int); PROTOTYPEpropSetMeth(qqueue, iDiscardSeverity, int); PROTOTYPEpropSetMeth(qqueue, iMinMsgsPerWrkr, int); PROTOTYPEpropSetMeth(qqueue, bSaveOnShutdown, int); -PROTOTYPEpropSetMeth(qqueue, pUsr, void*); +PROTOTYPEpropSetMeth(qqueue, pAction, action_t*); PROTOTYPEpropSetMeth(qqueue, iDeqSlowdown, int); PROTOTYPEpropSetMeth(qqueue, sizeOnDiskMax, int64); PROTOTYPEpropSetMeth(qqueue, iDeqBatchSize, int); diff --git a/runtime/ratelimit.c b/runtime/ratelimit.c new file mode 100644 index 00000000..4b618fb5 --- /dev/null +++ b/runtime/ratelimit.c @@ -0,0 +1,359 @@ +/* ratelimit.c + * support for rate-limiting sources, including "last message + * repeated n times" processing. + * + * Copyright 2012 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> + +#include "rsyslog.h" +#include "errmsg.h" +#include "ratelimit.h" +#include "datetime.h" +#include "parser.h" +#include "unicode-helper.h" +#include "msg.h" +#include "rsconf.h" +#include "dirty.h" + +/* definitions for objects we access */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) +DEFobjCurrIf(datetime) +DEFobjCurrIf(parser) + +/* static data */ + +/* generate a "repeated n times" message */ +static inline msg_t * +ratelimitGenRepMsg(ratelimit_t *ratelimit) +{ + msg_t *repMsg; + size_t lenRepMsg; + uchar szRepMsg[1024]; + + if(ratelimit->nsupp == 1) { /* we simply use the original message! */ + repMsg = MsgAddRef(ratelimit->pMsg); + } else {/* we need to duplicate, original message may still be in use in other + * parts of the system! */ + if((repMsg = MsgDup(ratelimit->pMsg)) == NULL) { + DBGPRINTF("Message duplication failed, dropping repeat message.\n"); + goto done; + } + lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), + " message repeated %d times: [%.800s]", + ratelimit->nsupp, getMSG(ratelimit->pMsg)); + MsgReplaceMSG(repMsg, szRepMsg, lenRepMsg); + } + +done: return repMsg; +} + +static inline rsRetVal +doLastMessageRepeatedNTimes(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRepMsg) +{ + int bNeedUnlockMutex = 0; + rsRetVal localRet; + DEFiRet; + + if((pMsg->msgFlags & NEEDS_PARSING) != 0) { + if((localRet = parser.ParseMsg(pMsg)) != RS_RET_OK) { + DBGPRINTF("Message discarded, parsing error %d\n", localRet); + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } + } + + if(ratelimit->bThreadSafe) { + pthread_mutex_lock(&ratelimit->mut); + bNeedUnlockMutex = 1; + } + + if( ratelimit->pMsg != NULL && + getMSGLen(pMsg) == getMSGLen(ratelimit->pMsg) && + !ustrcmp(getMSG(pMsg), getMSG(ratelimit->pMsg)) && + !strcmp(getHOSTNAME(pMsg), getHOSTNAME(ratelimit->pMsg)) && + !strcmp(getPROCID(pMsg, LOCK_MUTEX), getPROCID(ratelimit->pMsg, LOCK_MUTEX)) && + !strcmp(getAPPNAME(pMsg, LOCK_MUTEX), getAPPNAME(ratelimit->pMsg, LOCK_MUTEX))) { + ratelimit->nsupp++; + DBGPRINTF("msg repeated %d times\n", ratelimit->nsupp); + /* use current message, so we have the new timestamp + * (means we need to discard previous one) */ + msgDestruct(&ratelimit->pMsg); + ratelimit->pMsg = pMsg; + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } else {/* new message, do "repeat processing" & save it */ + if(ratelimit->pMsg != NULL) { + if(ratelimit->nsupp > 0) { + *ppRepMsg = ratelimitGenRepMsg(ratelimit); + ratelimit->nsupp = 0; + } + msgDestruct(&ratelimit->pMsg); + } + ratelimit->pMsg = MsgAddRef(pMsg); + } + +finalize_it: + if(bNeedUnlockMutex) + pthread_mutex_unlock(&ratelimit->mut); + RETiRet; +} + + +/* helper: tell how many messages we lost due to linux-like ratelimiting */ +static inline void +tellLostCnt(ratelimit_t *ratelimit) +{ + uchar msgbuf[1024]; + if(ratelimit->missed) { + snprintf((char*)msgbuf, sizeof(msgbuf), + "%s: %u messages lost due to rate-limiting", + ratelimit->name, ratelimit->missed); + logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); + ratelimit->missed = 0; + } +} + +/* Linux-like ratelimiting, modelled after the linux kernel + * returns 1 if message is within rate limit and shall be + * processed, 0 otherwise. + * This implementation is NOT THREAD-SAFE and must not + * be called concurrently. + */ +static inline int +withinRatelimit(ratelimit_t *ratelimit, time_t tt) +{ + int ret; + uchar msgbuf[1024]; + + if(ratelimit->interval == 0) { + ret = 1; + goto finalize_it; + } + + assert(ratelimit->burst != 0); + + if(ratelimit->begin == 0) + ratelimit->begin = tt; + + /* resume if we go out of out time window */ + if(tt > ratelimit->begin + ratelimit->interval) { + tellLostCnt(ratelimit); + ratelimit->begin = 0; + ratelimit->done = 0; + } + + /* do actual limit check */ + if(ratelimit->burst > ratelimit->done) { + ratelimit->done++; + ret = 1; + } else { + if(ratelimit->missed == 0) { + snprintf((char*)msgbuf, sizeof(msgbuf), + "%s: begin to drop messages due to rate-limiting", + ratelimit->name); + logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); + } + ratelimit->missed++; + ret = 0; + } + +finalize_it: + return ret; +} + + +/* ratelimit a message, that means: + * - handle "last message repeated n times" logic + * - handle actual (discarding) rate-limiting + * This function returns RS_RET_OK, if the caller shall process + * the message regularly and RS_RET_DISCARD if the caller must + * discard the message. The caller should also discard the message + * if another return status occurs. This places some burden on the + * caller logic, but provides best performance. Demanding this + * cooperative mode can enable a faulty caller to thrash up part + * of the system, but we accept that risk (a faulty caller can + * always do all sorts of evil, so...) + * If *ppRepMsg != NULL on return, the caller must enqueue that + * message before the original message. + */ +rsRetVal +ratelimitMsg(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRepMsg) +{ + DEFiRet; + + *ppRepMsg = NULL; + if(ratelimit->interval) { + if(withinRatelimit(ratelimit, pMsg->ttGenTime) == 0) { + msgDestruct(&pMsg); + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } + } + if(ratelimit->bReduceRepeatMsgs) { + CHKiRet(doLastMessageRepeatedNTimes(ratelimit, pMsg, ppRepMsg)); + } +finalize_it: + RETiRet; +} + +/* returns 1, if the ratelimiter performs any checks and 0 otherwise */ +int +ratelimitChecked(ratelimit_t *ratelimit) +{ + return ratelimit->interval || ratelimit->bReduceRepeatMsgs; +} + + +/* add a message to a ratelimiter/multisubmit structure. + * ratelimiting is automatically handled according to the ratelimit + * settings. + * if pMultiSub == NULL, a single-message enqueue happens (under reconsideration) + */ +rsRetVal +ratelimitAddMsg(ratelimit_t *ratelimit, multi_submit_t *pMultiSub, msg_t *pMsg) +{ + rsRetVal localRet; + msg_t *repMsg; + DEFiRet; + + if(pMultiSub == NULL) { + localRet = ratelimitMsg(ratelimit, pMsg, &repMsg); + if(repMsg != NULL) + CHKiRet(submitMsg2(repMsg)); + if(localRet == RS_RET_OK) + CHKiRet(submitMsg2(pMsg)); + } else { + localRet = ratelimitMsg(ratelimit, pMsg, &repMsg); + if(repMsg != NULL) { + pMultiSub->ppMsgs[pMultiSub->nElem++] = repMsg; + if(pMultiSub->nElem == pMultiSub->maxElem) + CHKiRet(multiSubmitMsg2(pMultiSub)); + } + if(localRet == RS_RET_OK) { + pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; + if(pMultiSub->nElem == pMultiSub->maxElem) + CHKiRet(multiSubmitMsg2(pMultiSub)); + } + } + +finalize_it: + RETiRet; +} + + +/* modname must be a static name (usually expected to be the module + * name and MUST be present. dynname may be NULL and can be used for + * dynamic information, e.g. PID or listener IP, ... + * Both values should be kept brief. + */ +rsRetVal +ratelimitNew(ratelimit_t **ppThis, char *modname, char *dynname) +{ + ratelimit_t *pThis; + char namebuf[256]; + DEFiRet; + + CHKmalloc(pThis = calloc(1, sizeof(ratelimit_t))); + if(modname == NULL) + modname ="*ERROR:MODULE NAME MISSING*"; + + if(dynname == NULL) { + pThis->name = strdup(modname); + } else { + snprintf(namebuf, sizeof(namebuf), "%s[%s]", + modname, dynname); + namebuf[sizeof(namebuf)-1] = '\0'; /* to be on safe side */ + pThis->name = strdup(namebuf); + } + pThis->bReduceRepeatMsgs = loadConf->globals.bReduceRepeatMsgs; + *ppThis = pThis; +finalize_it: + RETiRet; +} + + +/* enable linux-like ratelimiting */ +void +ratelimitSetLinuxLike(ratelimit_t *ratelimit, unsigned short interval, unsigned short burst) +{ + ratelimit->interval = interval; + ratelimit->burst = burst; + ratelimit->done = 0; + ratelimit->missed = 0; + ratelimit->begin = 0; +} + + +/* enable thread-safe operations mode. This make sure that + * a single ratelimiter can be called from multiple threads. As + * this causes some overhead and is not always required, it needs + * to be explicitely enabled. This operation cannot be undone + * (think: why should one do that???) + */ +void +ratelimitSetThreadSafe(ratelimit_t *ratelimit) +{ + ratelimit->bThreadSafe = 1; + pthread_mutex_init(&ratelimit->mut, NULL); +} + +void +ratelimitDestruct(ratelimit_t *ratelimit) +{ + msg_t *pMsg; + if(ratelimit->pMsg != NULL) { + if(ratelimit->nsupp > 0) { + pMsg = ratelimitGenRepMsg(ratelimit); + if(pMsg != NULL) + submitMsg2(pMsg); + } + msgDestruct(&ratelimit->pMsg); + } + tellLostCnt(ratelimit); + if(ratelimit->bThreadSafe) + pthread_mutex_destroy(&ratelimit->mut); + free(ratelimit->name); + free(ratelimit); +} + +void +ratelimitModExit(void) +{ + objRelease(datetime, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); +} + +rsRetVal +ratelimitModInit(void) +{ + DEFiRet; + CHKiRet(objGetObjInterface(&obj)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(parser, CORE_COMPONENT)); +finalize_it: + RETiRet; +} + diff --git a/runtime/ratelimit.h b/runtime/ratelimit.h new file mode 100644 index 00000000..820817bc --- /dev/null +++ b/runtime/ratelimit.h @@ -0,0 +1,51 @@ +/* header for ratelimit.c + * + * Copyright 2012 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_RATELIMIT_H +#define INCLUDED_RATELIMIT_H + +struct ratelimit_s { + char *name; /**< rate limiter name, e.g. for user messages */ + /* support for Linux kernel-type ratelimiting */ + unsigned short interval; + unsigned short burst; + unsigned done; + unsigned missed; + time_t begin; + /* support for "last message repeated n times */ + int bReduceRepeatMsgs; /**< shall we do "last message repeated n times" processing? */ + unsigned nsupp; /**< nbr of msgs suppressed */ + msg_t *pMsg; + sbool bThreadSafe; /**< do we need to operate in Thread-Safe mode? */ + pthread_mutex_t mut; /**< mutex if thread-safe operation desired */ +}; + +/* prototypes */ +rsRetVal ratelimitNew(ratelimit_t **ppThis, char *modname, char *dynname); +void ratelimitSetThreadSafe(ratelimit_t *ratelimit); +void ratelimitSetLinuxLike(ratelimit_t *ratelimit, unsigned short interval, unsigned short burst); +rsRetVal ratelimitMsg(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRep); +rsRetVal ratelimitAddMsg(ratelimit_t *ratelimit, multi_submit_t *pMultiSub, msg_t *pMsg); +void ratelimitDestruct(ratelimit_t *pThis); +int ratelimitChecked(ratelimit_t *ratelimit); +rsRetVal ratelimitModInit(void); +void ratelimitModExit(void); + +#endif /* #ifndef INCLUDED_RATELIMIT_H */ diff --git a/runtime/rsconf.c b/runtime/rsconf.c index 118e9c11..d8b81f1b 100644 --- a/runtime/rsconf.c +++ b/runtime/rsconf.c @@ -36,7 +36,6 @@ #include "rsyslog.h" #include "obj.h" #include "srUtils.h" -#include "rule.h" #include "ruleset.h" #include "modules.h" #include "conf.h" @@ -68,9 +67,9 @@ #include "dirty.h" #include "template.h" +extern char* yytext; /* static data */ DEFobjStaticHelpers -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) DEFobjCurrIf(module) DEFobjCurrIf(conf) @@ -116,8 +115,8 @@ void cnfDoCfsysline(char *ln); */ BEGINobjConstruct(rsconf) /* be sure to specify the object type also in END macro! */ pThis->globals.bDebugPrintTemplateList = 1; - pThis->globals.bDebugPrintModuleList = 1; - pThis->globals.bDebugPrintCfSysLineHandlerList = 1; + pThis->globals.bDebugPrintModuleList = 0; + pThis->globals.bDebugPrintCfSysLineHandlerList = 0; pThis->globals.bLogStatusMsgs = DFLT_bLogStatusMsgs; pThis->globals.bErrMsgToStderr = 1; pThis->globals.umask = -1; @@ -254,54 +253,6 @@ CODESTARTobjDebugPrint(rsconf) ENDobjDebugPrint(rsconf) -rsRetVal -cnfDoActlst(struct cnfactlst *actlst, rule_t *pRule) -{ - struct cnfcfsyslinelst *cflst; - action_t *pAction; - uchar *str; - rsRetVal localRet; - DEFiRet; - - while(actlst != NULL) { - dbgprintf("aclst %p: ", actlst); - if(actlst->actType == CNFACT_V2) { - dbgprintf("v6+ action object\n"); - if(actionNewInst(actlst->data.lst, &pAction) == RS_RET_OK) { - iRet = llAppend(&(pRule)->llActList, NULL, (void*) pAction); - } else { - errmsg.LogError(0, RS_RET_ERR, "errors occured in file '%s' " - "around line %d", actlst->cnfFile, actlst->lineno); - } - } else { - DBGPRINTF("legacy action line:%s\n", actlst->data.legActLine); - str = (uchar*) actlst->data.legActLine; - if((localRet = cflineDoAction(loadConf, &str, &pAction)) != RS_RET_OK) { - uchar szErrLoc[MAXFNAME + 64]; - if(localRet != RS_RET_OK_WARN) { - DBGPRINTF("legacy action line NOT successfully processed\n"); - } - snprintf((char*)szErrLoc, sizeof(szErrLoc) / sizeof(uchar), - "%s, line %d", actlst->cnfFile, actlst->lineno); - errmsg.LogError(0, NO_ERRCODE, "the last %s occured in %s:\"%s\"", - (localRet == RS_RET_OK_WARN) ? "warning" : "error", - (char*)szErrLoc, (char*)actlst->data.legActLine); - if(localRet != RS_RET_OK_WARN) { - ABORT_FINALIZE(localRet); - } - } - iRet = llAppend(&(pRule)->llActList, NULL, (void*) pAction); - } - for( cflst = actlst->syslines - ; cflst != NULL ; cflst = cflst->next) { - cnfDoCfsysline(cflst->line); - } - actlst = actlst->next; - } -finalize_it: - RETiRet; -} - /* This function returns the current date in different * variants. It is used to construct the $NOW series of * system properties. The returned buffer must be freed @@ -341,6 +292,9 @@ getNOW(eNOWType eNow, es_str_t **estr) case NOW_MINUTE: len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.minute); break; + default: + len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "*invld eNow*"); + break; } /* now create a string object out of it and hand that over to the var */ @@ -397,15 +351,11 @@ inputProcessCnf(struct cnfobj *o) pvals = nvlstGetParams(o->nvlst, &inppblk, NULL); if(pvals == NULL) { - ABORT_FINALIZE(RS_RET_ERR); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); } DBGPRINTF("input param blk after inputProcessCnf:\n"); cnfparamsPrint(&inppblk, pvals); typeIdx = cnfparamGetIdx(&inppblk, "type"); - if(pvals[typeIdx].bUsed == 0) { - errmsg.LogError(0, RS_RET_CONF_RQRD_PARAM_MISSING, "input type missing"); - ABORT_FINALIZE(RS_RET_CONF_RQRD_PARAM_MISSING); // TODO: move this into rainerscript handlers - } cnfModName = (uchar*)es_str2cstr(pvals[typeIdx].val.d.estr, NULL); if((pMod = module.FindWithCnfName(loadConf, cnfModName, eMOD_IN)) == NULL) { errmsg.LogError(0, RS_RET_MOD_UNKNOWN, "input module name '%s' is unknown", cnfModName); @@ -435,9 +385,6 @@ parser_errmsg(char *fmt, ...) va_start(ap, fmt); if(vsnprintf(errBuf, sizeof(errBuf), fmt, ap) == sizeof(errBuf)) errBuf[sizeof(errBuf)-1] = '\0'; -dbgprintf("XXXX: msg: %s\n", errBuf); -dbgprintf("XXXX: cnfcurrfn: %s\n", cnfcurrfn); -dbgprintf("XXXX: yylineno: %d\n", yylineno); errmsg.LogError(0, RS_RET_CONF_PARSE_ERROR, "error during parsing file %s, on or before line %d: %s", cnfcurrfn, yylineno, errBuf); @@ -447,7 +394,7 @@ dbgprintf("XXXX: yylineno: %d\n", yylineno); int yyerror(char *s) { - parser_errmsg("%s", s); + parser_errmsg("%s on token '%s'", s, yytext); return 0; } void cnfDoObj(struct cnfobj *o) @@ -463,75 +410,35 @@ void cnfDoObj(struct cnfobj *o) case CNFOBJ_MODULE: modulesProcessCnf(o); break; - case CNFOBJ_ACTION: - actionProcessCnf(o); - break; case CNFOBJ_INPUT: inputProcessCnf(o); break; case CNFOBJ_TPL: - tplProcessCnf(o); + if(tplProcessCnf(o) != RS_RET_OK) + parser_errmsg("error processing template object"); + break; + case CNFOBJ_RULESET: + rulesetProcessCnf(o); break; case CNFOBJ_PROPERTY: case CNFOBJ_CONSTANT: /* these types are processed at a later stage */ bChkUnuse = 0; break; + default: + dbgprintf("cnfDoObj program error: unexpected object type %u\n", + o->objType); + break; } if(bChkUnuse) nvlstChkUnused(o->nvlst); cnfobjDestruct(o); } -void cnfDoRule(struct cnfrule *cnfrule) +void cnfDoScript(struct cnfstmt *script) { - rule_t *pRule; - uchar *str; - rsRetVal iRet = RS_RET_OK; //DEFiRet; - - dbgprintf("cnf:global:rule\n"); - cnfrulePrint(cnfrule); - - CHKiRet(rule.Construct(&pRule)); /* create "fresh" selector */ - CHKiRet(rule.SetAssRuleset(pRule, ruleset.GetCurrent(loadConf))); - CHKiRet(rule.ConstructFinalize(pRule)); - - switch(cnfrule->filttype) { - case CNFFILT_NONE: - break; - case CNFFILT_PRI: - str = (uchar*) cnfrule->filt.s; - iRet = cflineProcessTradPRIFilter(&str, pRule); - break; - case CNFFILT_PROP: - dbgprintf("%s\n", cnfrule->filt.s); - str = (uchar*) cnfrule->filt.s; - iRet = cflineProcessPropFilter(&str, pRule); - break; - case CNFFILT_SCRIPT: - pRule->f_filter_type = FILTER_EXPR; - pRule->f_filterData.expr = cnfrule->filt.expr; - break; - } - /* we now check if there are some global (BSD-style) filter conditions - * and, if so, we copy them over. rgerhards, 2005-10-18 - */ - if(pDfltProgNameCmp != NULL) { - CHKiRet(rsCStrConstructFromCStr(&(pRule->pCSProgNameComp), pDfltProgNameCmp)); - } - - if(eDfltHostnameCmpMode != HN_NO_COMP) { - pRule->eHostnameCmpMode = eDfltHostnameCmpMode; - CHKiRet(rsCStrConstructFromCStr(&(pRule->pCSHostnameComp), pDfltHostnameCmp)); - } - - cnfDoActlst(cnfrule->actlst, pRule); - - CHKiRet(ruleset.AddRule(rule.GetAssRuleset(pRule), &pRule)); - -finalize_it: - //TODO: do something with error states - cnfruleDestruct(cnfrule); + dbgprintf("cnf:global:script\n"); + ruleset.AddScript(ruleset.GetCurrent(loadConf), script); } void cnfDoCfsysline(char *ln) @@ -545,13 +452,21 @@ void cnfDoCfsysline(char *ln) void cnfDoBSDTag(char *ln) { DBGPRINTF("cnf:global:BSD tag: %s\n", ln); - cflineProcessTagSelector((uchar**)&ln); + errmsg.LogError(0, RS_RET_BSD_BLOCKS_UNSUPPORTED, + "BSD-style blocks are no longer supported in rsyslog, " + "see http://www.rsyslog.com/g/BSD for details and a " + "solution (Block '%s')", ln); + free(ln); } void cnfDoBSDHost(char *ln) { DBGPRINTF("cnf:global:BSD host: %s\n", ln); - cflineProcessHostSelector((uchar**)&ln); + errmsg.LogError(0, RS_RET_BSD_BLOCKS_UNSUPPORTED, + "BSD-style blocks are no longer supported in rsyslog, " + "see http://www.rsyslog.com/g/BSD for details and a " + "solution (Block '%s')", ln); + free(ln); } es_str_t* @@ -565,6 +480,9 @@ cnfGetVar(char *name, void *usrptr) estr = msgGetCEEVarNew((msg_t*) usrptr, name+2); else estr = msgGetMsgVarNew((msg_t*) usrptr, (uchar*)name+1); + } else { /* if this happens, we have a program logic error */ + estr = es_newStrFromCStr("err: var must start with $", + strlen("err: var must start with $")); } if(Debug) { char *s; @@ -841,7 +759,7 @@ activateMainQueue() { DEFiRet; /* create message queue */ - CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"))) { + CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"), NULL)) { /* no queue is fatal, we need to give up in that case... */ fprintf(stderr, "fatal error %d: could not create message queue - rsyslogd can not run!\n", iRet); FINALIZE; @@ -961,6 +879,7 @@ setCurrRuleset(void __attribute__((unused)) *pVal, uchar *pszName) CHKiRet(ruleset.Construct(&pRuleset)); CHKiRet(ruleset.SetName(pRuleset, pszName)); CHKiRet(ruleset.ConstructFinalize(ourConf, pRuleset)); + rulesetSetCurrRulesetPtr(pRuleset); } else { ABORT_FINALIZE(localRet); } @@ -1162,6 +1081,7 @@ initLegacyConf(void) ruleset.Construct(&pRuleset); ruleset.SetName(pRuleset, UCHAR_CONSTANT("RSYSLOG_DefaultRuleset")); ruleset.ConstructFinalize(loadConf, pRuleset); + rulesetSetCurrRulesetPtr(pRuleset); /* now register config handlers */ CHKiRet(regCfSysLineHdlr((uchar *)"sleep", 0, eCmdHdlrGoneAway, @@ -1378,6 +1298,7 @@ ourConf = loadConf; // TODO: remove, once ourConf is gone! ABORT_FINALIZE(RS_RET_NO_ACTIONS); } tellLexEndParsing(); + rulesetOptimizeAll(loadConf); tellCoreConfigLoadDone(); tellModulesConfigLoadDone(); @@ -1436,7 +1357,6 @@ ENDobjQueryInterface(rsconf) BEGINObjClassInit(rsconf, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(ruleset, CORE_COMPONENT)); - CHKiRet(objUse(rule, CORE_COMPONENT)); CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(conf, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); @@ -1453,7 +1373,6 @@ ENDObjClassInit(rsconf) /* De-initialize the rsconf class. */ BEGINObjClassExit(rsconf, OBJ_IS_CORE_MODULE) /* class, version */ - objRelease(rule, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); objRelease(module, CORE_COMPONENT); objRelease(conf, CORE_COMPONENT); diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index cbab06b7..047dfa9b 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -72,7 +72,6 @@ #include "glbl.h" #include "errmsg.h" #include "prop.h" -#include "rule.h" #include "ruleset.h" #include "parser.h" #include "strgen.h" @@ -171,8 +170,6 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(glblClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "msg"; CHKiRet(msgClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "rule"; - CHKiRet(ruleClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "ruleset"; CHKiRet(rulesetClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "wti"; @@ -220,7 +217,6 @@ rsrtExit(void) confClassExit(); glblClassExit(); rulesetClassExit(); - ruleClassExit(); objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */ } diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index a6e4b100..e7a5dffb 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -46,6 +46,7 @@ #define CONF_HOSTNAME_MAXSIZE 512 /* a value that is deemed far too large for any valid HOSTNAME */ #define CONF_RAWMSG_BUFSIZE 101 #define CONF_TAG_BUFSIZE 32 +#define CONF_PROGNAME_BUFSIZE 16 #define CONF_HOSTNAME_BUFSIZE 32 #define CONF_PROP_BUFSIZE 16 /* should be close to sizeof(ptr) or lighly above it */ #define CONF_MIN_SIZE_FOR_COMPRESS 60 /* config param: minimum message size to try compression. The smaller @@ -69,6 +70,7 @@ * approach taken here is considered appropriate. * rgerhards, 2010-06-24 */ +#define CONF_NUM_MULTISUB 1024 /* default number of messages per multisub structure */ /* ############################################################# * * # End Config Settings # * @@ -322,7 +324,7 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_ERR_DOOR = -2147, /**< some problems with handling the Solaris door functionality */ RS_RET_NO_SRCNAME_TPL = -2150, /**< sourcename template was not specified where one was needed (omudpspoof spoof addr) */ RS_RET_HOST_NOT_SPECIFIED = -2151, /**< (target) host was not specified where it was needed */ - RS_RET_ERR_LIBNET_INIT = -2152, /**< error initializing libnet */ + RS_RET_ERR_LIBNET_INIT = -2152, /**< error initializing libnet, e.g. because not running as root */ RS_RET_FORCE_TERM = -2153, /**< thread was forced to terminate by bShallShutdown, a state, not an error */ RS_RET_RULES_QUEUE_EXISTS = -2154,/**< we were instructed to create a new ruleset queue, but one already exists */ RS_RET_NO_CURR_RULESET = -2155,/**< no current ruleset exists (but one is required) */ @@ -376,15 +378,34 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_LEGA_ACT_NOT_SUPPORTED = -2215, /**< the module (no longer) supports legacy action syntax */ RS_RET_MAX_OMSR_REACHED = -2216, /**< max nbr of string requests reached, not supported by core */ RS_RET_UID_MISSING = -2217, /**< a user id is missing (but e.g. a password provided) */ + RS_RET_DATAFAIL = -2218, /**< data passed to action caused failure */ /* reserved for pre-v6.5 */ RS_RET_DUP_PARAM = -2220, /**< config parameter is given more than once */ RS_RET_MODULE_ALREADY_IN_CONF = -2221, /**< module already in current configuration */ RS_RET_PARAM_NOT_PERMITTED = -2222, /**< legacy parameter no longer permitted (usally already set by v2) */ RS_RET_NO_JSON_PASSING = -2223, /**< rsyslog core does not support JSON-passing plugin API */ RS_RET_MOD_NO_INPUT_STMT = -2224, /**< (input) module does not support input() statement */ + RS_RET_NO_CEE_MSG = -2225, /**< the message being processed is NOT CEE-enhanced */ + + /**** up to 2300 is reserved for v6 use ****/ + RS_RET_JNAME_NO_ROOT = -2301, /**< root element is missing in JSON path */ + RS_RET_JNAME_INVALID = -2302, /**< JSON path is invalid */ + RS_RET_JSON_PARSE_ERR = -2303, /**< we had a problem parsing JSON (or extra data) */ + RS_RET_BSD_BLOCKS_UNSUPPORTED = -2304, /**< BSD-style config blocks are no longer supported */ + RS_RET_JNAME_NOTFOUND = -2305, /**< JSON name not found (does not exist) */ + RS_RET_INVLD_SETOP = -2305, /**< invalid variable set operation, incompatible type */ + RS_RET_RULESET_EXISTS = -2306,/**< ruleset already exists */ + RS_RET_DEPRECATED = -2307,/**< deprecated functionality is used */ + RS_RET_DS_PROP_SEQ_ERR = -2308,/**< property sequence error deserializing object */ + RS_RET_TPL_INVLD_PROP = -2309,/**< property name error in template (unknown name) */ + RS_RET_NO_RULEBASE = -2310,/**< mmnormalize: rulebase can not be found or otherwise invalid */ + RS_RET_INVLD_MODE = -2311,/**< invalid mode specified in configuration */ + RS_RET_INVLD_ANON_BITS = -2312,/**< mmanon: invalid number of bits to anonymize specified */ + RS_RET_REPLCHAR_IGNORED = -2313,/**< mmanon: replacementChar parameter is ignored */ /* RainerScript error messages (range 1000.. 1999) */ RS_RET_SYSVAR_NOT_FOUND = 1001, /**< system variable could not be found (maybe misspelled) */ + RS_RET_FIELD_NOT_FOUND = 1002, /**< field() function did not find requested field */ /* some generic error/status codes */ RS_RET_OK = 0, /**< operation successful */ diff --git a/runtime/rule.c b/runtime/rule.c deleted file mode 100644 index fc1e740f..00000000 --- a/runtime/rule.c +++ /dev/null @@ -1,479 +0,0 @@ -/* rule.c - rsyslog's rule object - * - * See file comment in rule.c for the overall structure of rule processing. - * - * Module begun 2009-06-10 by Rainer Gerhards - * - * Copyright 2009-2012 Adiscon GmbH. - * - * This file is part of the rsyslog runtime library. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * -or- - * see COPYING.ASL20 in the source distribution - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "config.h" -#include <stdlib.h> -#include <string.h> -#include <assert.h> -#include <ctype.h> - -#include "rsyslog.h" -#include "obj.h" -#include "action.h" -#include "rule.h" -#include "errmsg.h" -#include "srUtils.h" -#include "batch.h" -#include "parserif.h" -#include "unicode-helper.h" - -/* static data */ -DEFobjStaticHelpers -DEFobjCurrIf(errmsg) - - -/* support for simple textual representation of FIOP names - * rgerhards, 2005-09-27 - */ -static char* -getFIOPName(unsigned iFIOP) -{ - char *pRet; - switch(iFIOP) { - case FIOP_CONTAINS: - pRet = "contains"; - break; - case FIOP_ISEQUAL: - pRet = "isequal"; - break; - case FIOP_STARTSWITH: - pRet = "startswith"; - break; - case FIOP_REGEX: - pRet = "regex"; - break; - case FIOP_EREREGEX: - pRet = "ereregex"; - break; - case FIOP_ISEMPTY: - pRet = "isempty"; - break; - default: - pRet = "NOP"; - break; - } - return pRet; -} - - -/* iterate over all actions, this is often needed, for example when HUP processing - * must be done or a shutdown is pending. - */ -static rsRetVal -iterateAllActions(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) -{ - return llExecFunc(&pThis->llActList, pFunc, pParam); -} - - -/* helper to processMsg(), used to call the configured actions. It is - * executed from within llExecFunc() of the action list. - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(processBatchDoActions) -{ - DEFiRet; - rsRetVal iRetMod; /* return value of module - we do not always pass that back */ - action_t *pAction = (action_t*) pData; - batch_t *pBatch = (batch_t*) pParam; - - DBGPRINTF("Processing next action\n"); - iRetMod = pAction->submitToActQ(pAction, pBatch); - - RETiRet; -} - - -/* This functions looks at the given message and checks if it matches the - * provided filter condition. - */ -static rsRetVal -shouldProcessThisMessage(rule_t *pRule, msg_t *pMsg, sbool *bProcessMsg) -{ - DEFiRet; - unsigned short pbMustBeFreed; - uchar *pszPropVal; - int bRet = 0; - size_t propLen; - - ISOBJ_TYPE_assert(pRule, rule); - assert(pMsg != NULL); - - /* we first have a look at the global, BSD-style block filters (for tag - * and host). Only if they match, we evaluate the actual filter. - * rgerhards, 2005-10-18 - */ - if(pRule->eHostnameCmpMode == HN_NO_COMP) { - /* EMPTY BY INTENSION - we check this value first, because - * it is the one most often used, so this saves us time! - */ - } else if(pRule->eHostnameCmpMode == HN_COMP_MATCH) { - if(rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, so we are already done... */ - DBGPRINTF("hostname filter '+%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } else { /* must be -hostname */ - if(!rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, SO WE ARe already done... */ - DBGPRINTF("hostname filter '-%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } - - if(pRule->pCSProgNameComp != NULL) { - int bInv = 0, bEqv = 0, offset = 0; - if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp)) == '-') { - if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp) + 1) == '-') - offset = 1; - else { - bInv = 1; - offset = 1; - } - } - if(!rsCStrOffsetSzStrCmp(pRule->pCSProgNameComp, offset, - (uchar*) getProgramName(pMsg, LOCK_MUTEX), getProgramNameLen(pMsg, LOCK_MUTEX))) - bEqv = 1; - - if((!bEqv && !bInv) || (bEqv && bInv)) { - /* not equal or inverted selection, so we are already done... */ - DBGPRINTF("programname filter '%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp), getProgramName(pMsg, LOCK_MUTEX)); - FINALIZE; - } - } - - /* done with the BSD-style block filters */ - - if(pRule->f_filter_type == FILTER_PRI) { - /* skip messages that are incorrect priority */ - if ( (pRule->f_filterData.f_pmask[pMsg->iFacility] == TABLE_NOPRI) || \ - ((pRule->f_filterData.f_pmask[pMsg->iFacility] & (1<<pMsg->iSeverity)) == 0) ) - bRet = 0; - else - bRet = 1; - DBGPRINTF("testing filter, f_pmask %d, result %d\n", pRule->f_filterData.f_pmask[pMsg->iFacility], bRet); - } else if(pRule->f_filter_type == FILTER_EXPR) { - bRet = cnfexprEvalBool(pRule->f_filterData.expr, pMsg); - DBGPRINTF("result of rainerscript filter evaluation: %d\n", bRet); - } else { - assert(pRule->f_filter_type == FILTER_PROP); /* assert() just in case... */ - if(pRule->f_filterData.prop.propID == PROP_INVALID) { - DBGPRINTF("invalid property ID, filter always returns 0\n"); - bRet = 0; - } else { - pszPropVal = MsgGetProp(pMsg, NULL, pRule->f_filterData.prop.propID, - pRule->f_filterData.prop.propName, &propLen, &pbMustBeFreed); - - /* Now do the compares (short list currently ;)) */ - switch(pRule->f_filterData.prop.operation ) { - case FIOP_CONTAINS: - if(rsCStrLocateInSzStr(pRule->f_filterData.prop.pCSCompValue, (uchar*) pszPropVal) != -1) - bRet = 1; - break; - case FIOP_ISEMPTY: - if(propLen == 0) - bRet = 1; /* process message! */ - break; - case FIOP_ISEQUAL: - if(rsCStrSzStrCmp(pRule->f_filterData.prop.pCSCompValue, - pszPropVal, ustrlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_STARTSWITH: - if(rsCStrSzStrStartsWithCStr(pRule->f_filterData.prop.pCSCompValue, - pszPropVal, ustrlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_REGEX: - if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 0, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - case FIOP_EREREGEX: - if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 1, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - default: - /* here, it handles NOP (for performance reasons) */ - assert(pRule->f_filterData.prop.operation == FIOP_NOP); - bRet = 1; /* as good as any other default ;) */ - break; - } - - /* now check if the value must be negated */ - if(pRule->f_filterData.prop.isNegated) - bRet = (bRet == 1) ? 0 : 1; - - if(Debug) { - char *cstr; - if(pRule->f_filterData.prop.propID == PROP_CEE) { - cstr = es_str2cstr(pRule->f_filterData.prop.propName, NULL); - dbgprintf("Filter: check for CEE property '%s' (value '%s') ", - cstr, pszPropVal); - free(cstr); - } else { - dbgprintf("Filter: check for property '%s' (value '%s') ", - propIDToName(pRule->f_filterData.prop.propID), pszPropVal); - } - if(pRule->f_filterData.prop.isNegated) - dbgprintf("NOT "); - if(pRule->f_filterData.prop.operation == FIOP_ISEMPTY) { - dbgprintf("%s : %s\n", - getFIOPName(pRule->f_filterData.prop.operation), - bRet ? "TRUE" : "FALSE"); - } else { - dbgprintf("%s '%s': %s\n", - getFIOPName(pRule->f_filterData.prop.operation), - rsCStrGetSzStrNoNULL(pRule->f_filterData.prop.pCSCompValue), - bRet ? "TRUE" : "FALSE"); - } - } - - /* cleanup */ - if(pbMustBeFreed) - free(pszPropVal); - } - } - -finalize_it: - *bProcessMsg = bRet; - RETiRet; -} - - - -/* Process (consume) a batch of messages. Calls the actions configured. - * rgerhards, 2005-10-13 - */ -static rsRetVal -processBatch(rule_t *pThis, batch_t *pBatch) -{ - int i; - rsRetVal localRet; - DEFiRet; - - ISOBJ_TYPE_assert(pThis, rule); - assert(pBatch != NULL); - - /* first check the filters and reset status variables */ - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - localRet = shouldProcessThisMessage(pThis, (msg_t*)(pBatch->pElem[i].pUsrp), - &(pBatch->pElem[i].bFilterOK)); - if(localRet != RS_RET_OK) { - DBGPRINTF("processBatch: iRet %d returned from shouldProcessThisMessage, " - "ignoring message\n", localRet); - pBatch->pElem[i].bFilterOK = 0; - } - if(pBatch->pElem[i].bFilterOK) { - /* re-init only when actually needed (cache write cost!) */ - pBatch->pElem[i].bPrevWasSuspended = 0; - } - } - CHKiRet(llExecFunc(&pThis->llActList, processBatchDoActions, pBatch)); - -finalize_it: - RETiRet; -} - - -/* Standard-Constructor - */ -BEGINobjConstruct(rule) /* be sure to specify the object type also in END macro! */ -ENDobjConstruct(rule) - - -/* ConstructionFinalizer - * rgerhards, 2008-01-09 - */ -static rsRetVal -ruleConstructFinalize(rule_t *pThis) -{ - DEFiRet; - ISOBJ_TYPE_assert(pThis, rule); - - /* note: actionDestruct is from action.c API! */ - CHKiRet(llInit(&pThis->llActList, actionDestruct, NULL, NULL)); - -finalize_it: - RETiRet; -} - - -/* destructor for the rule object */ -BEGINobjDestruct(rule) /* be sure to specify the object type also in END and CODESTART macros! */ -CODESTARTobjDestruct(rule) - if(pThis->pCSHostnameComp != NULL) - rsCStrDestruct(&pThis->pCSHostnameComp); - if(pThis->pCSProgNameComp != NULL) - rsCStrDestruct(&pThis->pCSProgNameComp); - - if(pThis->f_filter_type == FILTER_PROP) { - if(pThis->f_filterData.prop.pCSCompValue != NULL) - rsCStrDestruct(&pThis->f_filterData.prop.pCSCompValue); - if(pThis->f_filterData.prop.regex_cache != NULL) - rsCStrRegexDestruct(&pThis->f_filterData.prop.regex_cache); - if(pThis->f_filterData.prop.propName != NULL) - es_deleteStr(pThis->f_filterData.prop.propName); - } else if(pThis->f_filter_type == FILTER_EXPR) { - cnfexprDestruct(pThis->f_filterData.expr); - } - - llDestroy(&pThis->llActList); -ENDobjDestruct(rule) - - -/* set the associated ruleset */ -static rsRetVal -setAssRuleset(rule_t *pThis, ruleset_t *pRuleset) -{ - DEFiRet; - ISOBJ_TYPE_assert(pThis, rule); - ISOBJ_TYPE_assert(pRuleset, ruleset); - pThis->pRuleset = pRuleset; - RETiRet; -} - -/* get the associated ruleset (may be NULL if not set!) */ -static ruleset_t* -getAssRuleset(rule_t *pThis) -{ - ISOBJ_TYPE_assert(pThis, rule); - return pThis->pRuleset; -} - - -/* helper to DebugPrint, to print out all actions via - * the llExecFunc() facility. - */ -DEFFUNC_llExecFunc(dbgPrintInitInfoAction) -{ - DEFiRet; - iRet = actionDbgPrint((action_t*) pData); - dbgprintf("\n"); - RETiRet; -} - - -/* debugprint for the rule object */ -BEGINobjDebugPrint(rule) /* be sure to specify the object type also in END and CODESTART macros! */ - int i; - char *cstr; -CODESTARTobjDebugPrint(rule) - dbgoprint((obj_t*) pThis, "rsyslog rule:\n"); - if(pThis->pCSProgNameComp != NULL) - dbgprintf("tag: '%s'\n", rsCStrGetSzStrNoNULL(pThis->pCSProgNameComp)); - if(pThis->eHostnameCmpMode != HN_NO_COMP) - dbgprintf("hostname: %s '%s'\n", - pThis->eHostnameCmpMode == HN_COMP_MATCH ? - "only" : "allbut", - rsCStrGetSzStrNoNULL(pThis->pCSHostnameComp)); - if(pThis->f_filter_type == FILTER_PRI) { - for (i = 0; i <= LOG_NFACILITIES; i++) - if (pThis->f_filterData.f_pmask[i] == TABLE_NOPRI) - dbgprintf(" X "); - else - dbgprintf("%2X ", pThis->f_filterData.f_pmask[i]); - } else if(pThis->f_filter_type == FILTER_EXPR) { - dbgprintf("EXPRESSION-BASED Filter: can currently not be displayed"); - } else { - dbgprintf("PROPERTY-BASED Filter:\n"); - dbgprintf("\tProperty.: '%s'\n", propIDToName(pThis->f_filterData.prop.propID)); - if(pThis->f_filterData.prop.propID != PROP_INVALID) { - if(pThis->f_filterData.prop.propName != NULL) { - cstr = es_str2cstr(pThis->f_filterData.prop.propName, NULL); - dbgprintf("\tCEE-Prop.: '%s'\n", cstr); - free(cstr); - } - dbgprintf("\tOperation: "); - if(pThis->f_filterData.prop.isNegated) - dbgprintf("NOT "); - dbgprintf("'%s'\n", getFIOPName(pThis->f_filterData.prop.operation)); - dbgprintf("\tValue....: '%s'\n", - rsCStrGetSzStrNoNULL(pThis->f_filterData.prop.pCSCompValue)); - } - dbgprintf("\tAction...: "); - } - - dbgprintf("\nActions:\n"); - llExecFunc(&pThis->llActList, dbgPrintInitInfoAction, NULL); /* actions */ - - dbgprintf("\n"); -ENDobjDebugPrint(rule) - - -/* queryInterface function - * rgerhards, 2008-02-21 - */ -BEGINobjQueryInterface(rule) -CODESTARTobjQueryInterface(rule) - if(pIf->ifVersion != ruleCURR_IF_VERSION) { /* check for current version, increment on each change */ - ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); - } - - /* ok, we have the right interface, so let's fill it - * Please note that we may also do some backwards-compatibility - * work here (if we can support an older interface version - that, - * of course, also affects the "if" above). - */ - pIf->Construct = ruleConstruct; - pIf->ConstructFinalize = ruleConstructFinalize; - pIf->Destruct = ruleDestruct; - pIf->DebugPrint = ruleDebugPrint; - - pIf->IterateAllActions = iterateAllActions; - pIf->ProcessBatch = processBatch; - pIf->SetAssRuleset = setAssRuleset; - pIf->GetAssRuleset = getAssRuleset; -finalize_it: -ENDobjQueryInterface(rule) - - -/* Exit the rule class. - * rgerhards, 2009-04-06 - */ -BEGINObjClassExit(rule, OBJ_IS_CORE_MODULE) /* class, version */ - objRelease(errmsg, CORE_COMPONENT); -ENDObjClassExit(rule) - - -/* Initialize the rule class. Must be called as the very first method - * before anything else is called inside this class. - * rgerhards, 2008-02-19 - */ -BEGINObjClassInit(rule, 1, OBJ_IS_CORE_MODULE) /* class, version */ - /* request objects we use */ - CHKiRet(objUse(errmsg, CORE_COMPONENT)); - - /* set our own handlers */ - OBJSetMethodHandler(objMethod_DEBUGPRINT, ruleDebugPrint); - OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, ruleConstructFinalize); -ENDObjClassInit(rule) - -/* vi:set ai: - */ diff --git a/runtime/rule.h b/runtime/rule.h deleted file mode 100644 index 1b07279b..00000000 --- a/runtime/rule.h +++ /dev/null @@ -1,78 +0,0 @@ -/* The rule object. - * - * This implements rules within rsyslog. - * - * Copyright 2009-2012 Adiscon GmbH. - * - * This file is part of the rsyslog runtime library. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * -or- - * see COPYING.ASL20 in the source distribution - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -#ifndef INCLUDED_RULE_H -#define INCLUDED_RULE_H - -#include "libestr.h" -#include "linkedlist.h" -#include "regexp.h" -#include "rainerscript.h" - -/* the rule object */ -struct rule_s { - BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - /* filter properties */ - enum { - FILTER_PRI = 0, /* traditional PRI based filer */ - FILTER_PROP = 1, /* extended filter, property based */ - FILTER_EXPR = 2 /* extended filter, expression based */ - } f_filter_type; - EHostnameCmpMode eHostnameCmpMode; - cstr_t *pCSHostnameComp; /* hostname to check */ - cstr_t *pCSProgNameComp; /* tag to check or NULL, if not to be checked */ - union { - u_char f_pmask[LOG_NFACILITIES+1]; /* priority mask */ - struct { - fiop_t operation; - regex_t *regex_cache; /* cache for compiled REs, if such are used */ - cstr_t *pCSCompValue; /* value to "compare" against */ - sbool isNegated; - propid_t propID; /* ID of the requested property */ - es_str_t *propName; /* name of property for CEE-based filters */ - } prop; - struct cnfexpr *expr; /* expression object */ - } f_filterData; - - ruleset_t *pRuleset; /* associated ruleset */ - linkedList_t llActList; /* list of configured actions */ -}; - -/* interfaces */ -BEGINinterface(rule) /* name must also be changed in ENDinterface macro! */ - INTERFACEObjDebugPrint(rule); - rsRetVal (*Construct)(rule_t **ppThis); - rsRetVal (*ConstructFinalize)(rule_t __attribute__((unused)) *pThis); - rsRetVal (*Destruct)(rule_t **ppThis); - rsRetVal (*IterateAllActions)(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void *pParam); - rsRetVal (*ProcessBatch)(rule_t *pThis, batch_t *pBatch); - rsRetVal (*SetAssRuleset)(rule_t *pThis, ruleset_t*); - ruleset_t* (*GetAssRuleset)(rule_t *pThis); -ENDinterface(rule) -#define ruleCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ -/* change for v2: ProcessMsg replaced by ProcessBatch - 2010-06-10 */ - - -/* prototypes */ -PROTOTYPEObj(rule); - -#endif /* #ifndef INCLUDED_RULE_H */ diff --git a/runtime/ruleset.c b/runtime/ruleset.c index 5cb34148..66b38fc9 100644 --- a/runtime/ruleset.c +++ b/runtime/ruleset.c @@ -11,27 +11,24 @@ * * Module begun 2009-06-10 by Rainer Gerhards * - * Copyright 2009-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * - * The rsyslog runtime library is free software: you can redistribute it and/or modify - * it under the terms of the GNU Lesser General Public License as published by - * the Free Software Foundation, either version 3 of the License, or - * (at your option) any later version. - * - * The rsyslog runtime library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. - * - * A copy of the GPL can be found in the file "COPYING" in this distribution. - * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. */ - #include "config.h" #include <stdlib.h> #include <assert.h> @@ -42,22 +39,36 @@ #include "cfsysline.h" #include "msg.h" #include "ruleset.h" -#include "rule.h" #include "errmsg.h" #include "parser.h" #include "batch.h" #include "unicode-helper.h" #include "rsconf.h" +#include "action.h" +#include "rainerscript.h" +#include "srUtils.h" +#include "modules.h" #include "dirty.h" /* for main ruleset queue creation */ /* static data */ DEFobjStaticHelpers DEFobjCurrIf(errmsg) -DEFobjCurrIf(rule) DEFobjCurrIf(parser) +/* tables for interfacing with the v6 config system (as far as we need to) */ +static struct cnfparamdescr rspdescr[] = { + { "name", eCmdHdlrString, CNFPARAM_REQUIRED }, + { "parser", eCmdHdlrArray, 0 } +}; +static struct cnfparamblk rspblk = + { CNFPARAMBLK_VERSION, + sizeof(rspdescr)/sizeof(struct cnfparamdescr), + rspdescr + }; + /* forward definitions */ static rsRetVal processBatch(batch_t *pBatch); +static rsRetVal scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active); /* ---------- linked-list key handling functions (ruleset) ---------- */ @@ -73,45 +84,61 @@ rulesetKeyDestruct(void __attribute__((unused)) *pData) /* ---------- END linked-list key handling functions (ruleset) ---------- */ +/* iterate over all actions in a script (stmt subtree) */ +static void +scriptIterateAllActions(struct cnfstmt *root, rsRetVal (*pFunc)(void*, void*), void* pParam) +{ + struct cnfstmt *stmt; + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + switch(stmt->nodetype) { + case S_NOP: + case S_STOP: + case S_CALL:/* call does not need to do anything - done in called ruleset! */ + break; + case S_ACT: + DBGPRINTF("iterateAllActions calling into action %p\n", stmt->d.act); + pFunc(stmt->d.act, pParam); + break; + case S_IF: + if(stmt->d.s_if.t_then != NULL) + scriptIterateAllActions(stmt->d.s_if.t_then, + pFunc, pParam); + if(stmt->d.s_if.t_else != NULL) + scriptIterateAllActions(stmt->d.s_if.t_else, + pFunc, pParam); + break; + case S_PRIFILT: + if(stmt->d.s_prifilt.t_then != NULL) + scriptIterateAllActions(stmt->d.s_prifilt.t_then, + pFunc, pParam); + if(stmt->d.s_prifilt.t_else != NULL) + scriptIterateAllActions(stmt->d.s_prifilt.t_else, + pFunc, pParam); + break; + case S_PROPFILT: + scriptIterateAllActions(stmt->d.s_propfilt.t_then, + pFunc, pParam); + break; + default: + dbgprintf("error: unknown stmt type %u during iterateAll\n", + (unsigned) stmt->nodetype); + break; + } + } +} /* driver to iterate over all of this ruleset actions */ typedef struct iterateAllActions_s { rsRetVal (*pFunc)(void*, void*); void *pParam; } iterateAllActions_t; -DEFFUNC_llExecFunc(doIterateRulesetActions) -{ - DEFiRet; - rule_t* pRule = (rule_t*) pData; - iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; - iRet = rule.IterateAllActions(pRule, pMyParam->pFunc, pMyParam->pParam); - RETiRet; -} -/* iterate over all actions of THIS rule set. - */ -static rsRetVal -iterateRulesetAllActions(ruleset_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) -{ - iterateAllActions_t params; - DEFiRet; - assert(pFunc != NULL); - - params.pFunc = pFunc; - params.pParam = pParam; - CHKiRet(llExecFunc(&(pThis->llRules), doIterateRulesetActions, ¶ms)); - -finalize_it: - RETiRet; -} - - /* driver to iterate over all actions */ DEFFUNC_llExecFunc(doIterateAllActions) { DEFiRet; ruleset_t* pThis = (ruleset_t*) pData; iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; - iRet = iterateRulesetAllActions(pThis, pMyParam->pFunc, pMyParam->pParam); + scriptIterateAllActions(pThis->root, pMyParam->pFunc, pMyParam->pParam); RETiRet; } /* iterate over ALL actions present in the WHOLE system. @@ -134,30 +161,10 @@ finalize_it: } - -/* helper to processBatch(), used to call the configured actions. It is - * executed from within llExecFunc() of the action list. - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(processBatchDoRules) -{ - rsRetVal iRet; - ISOBJ_TYPE_assert(pData, rule); - DBGPRINTF("Processing next rule\n"); - iRet = rule.ProcessBatch((rule_t*) pData, (batch_t*) pParam); - DBGPRINTF("ruleset: get iRet %d from rule.ProcessMsg()\n", iRet); - return iRet; -} - - - /* This function is similar to processBatch(), but works on a batch that * contains rules from multiple rulesets. In this case, we can not push * the whole batch through the ruleset. Instead, we examine it and * partition it into sub-rulesets which we then push through the system. - * Note that when we evaluate which message must be processed, we do NOT need - * to look at bFilterOK, because this value is only set in a later processing - * stage. Doing so caused a bug during development ;) * rgerhards, 2010-06-15 */ static inline rsRetVal @@ -174,7 +181,7 @@ processBatchMultiRuleset(batch_t *pBatch) do { bHaveUnprocessed = 0; /* search for first unprocessed element */ - for(iStart = 0 ; iStart < pBatch->nElem && pBatch->pElem[iStart].state == BATCH_STATE_DISC ; ++iStart) + for(iStart = 0 ; iStart < pBatch->nElem && pBatch->eltState[iStart] == BATCH_STATE_DISC ; ++iStart) /* just search, no action */; if(iStart == pBatch->nElem) break; /* everything processed */ @@ -187,11 +194,11 @@ processBatchMultiRuleset(batch_t *pBatch) for(i = iStart ; i < pBatch->nElem ; ++i) { if(batchElemGetRuleset(pBatch, i) == currRuleset) { /* for performance reasons, we copy only those members that we actually need */ - snglRuleBatch.pElem[iNew].pUsrp = pBatch->pElem[i].pUsrp; - snglRuleBatch.pElem[iNew].state = pBatch->pElem[i].state; + snglRuleBatch.pElem[iNew].pMsg = pBatch->pElem[i].pMsg; + snglRuleBatch.eltState[iNew] = pBatch->eltState[i]; ++iNew; /* We indicate the element also as done, so it will not be processed again */ - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } else { bHaveUnprocessed = 1; } @@ -207,6 +214,335 @@ finalize_it: RETiRet; } +/* return a new "active" structure for the batch. Free with freeActive(). */ +static inline sbool *newActive(batch_t *pBatch) +{ + return malloc(sizeof(sbool) * batchNumMsgs(pBatch)); + +} +static inline void freeActive(sbool *active) { free(active); } + + +/* for details, see scriptExec() header comment! */ +/* call action for all messages with filter on */ +static rsRetVal +execAct(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + DEFiRet; +dbgprintf("RRRR: execAct [%s]: batch of %d elements, active %p\n", modGetName(stmt->d.act->pMod), batchNumMsgs(pBatch), active); + pBatch->active = active; + stmt->d.act->submitToActQ(stmt->d.act, pBatch); + RETiRet; +} + +static rsRetVal +execSet(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + int i; + struct var result; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + cnfexprEval(stmt->d.s_set.expr, &result, pBatch->pElem[i].pMsg); + msgSetJSONFromVar(pBatch->pElem[i].pMsg, stmt->d.s_set.varname, + &result); + varDelete(&result); + } + } + RETiRet; +} + +static rsRetVal +execUnset(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + int i; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + msgUnsetJSON(pBatch->pElem[i].pMsg, stmt->d.s_unset.varname); + } + } + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +/* "stop" simply discards the filtered items - it's just a (hopefully more intuitive + * shortcut for users. + */ +static rsRetVal +execStop(batch_t *pBatch, sbool *active) +{ + int i; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + pBatch->eltState[i] = BATCH_STATE_DISC; + } + } + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +// save current filter, evaluate new one +// perform then (if any message) +// if ELSE given: +// set new filter, inverted +// perform else (if any messages) +static rsRetVal +execIf(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *newAct; + int i; + sbool bRet; + DEFiRet; + newAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + FINALIZE; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + if(active == NULL || active[i]) { + bRet = cnfexprEvalBool(stmt->d.s_if.expr, pBatch->pElem[i].pMsg); + } else + bRet = 0; + newAct[i] = bRet; + DBGPRINTF("batch: item %d: expr eval: %d\n", i, bRet); + } + + if(stmt->d.s_if.t_then != NULL) { + scriptExec(stmt->d.s_if.t_then, pBatch, newAct); + } + if(stmt->d.s_if.t_else != NULL) { + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + FINALIZE; + if(pBatch->eltState[i] != BATCH_STATE_DISC) + newAct[i] = !newAct[i]; + } + scriptExec(stmt->d.s_if.t_else, pBatch, newAct); + } + freeActive(newAct); +finalize_it: + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +static void +execPRIFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *newAct; + msg_t *pMsg; + int bRet; + int i; + newAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + pMsg = pBatch->pElem[i].pMsg; + if(active == NULL || active[i]) { + if( (stmt->d.s_prifilt.pmask[pMsg->iFacility] == TABLE_NOPRI) || + ((stmt->d.s_prifilt.pmask[pMsg->iFacility] + & (1<<pMsg->iSeverity)) == 0) ) + bRet = 0; + else + bRet = 1; + } else + bRet = 0; + newAct[i] = bRet; + DBGPRINTF("batch: item %d PRIFILT %d\n", i, newAct[i]); + } + + if(stmt->d.s_prifilt.t_then != NULL) { + scriptExec(stmt->d.s_prifilt.t_then, pBatch, newAct); + } + if(stmt->d.s_prifilt.t_else != NULL) { + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] != BATCH_STATE_DISC) + newAct[i] = !newAct[i]; + } + scriptExec(stmt->d.s_prifilt.t_else, pBatch, newAct); + } + freeActive(newAct); +} + + +/* helper to execPROPFILT(), as the evaluation itself is quite lengthy */ +static int +evalPROPFILT(struct cnfstmt *stmt, msg_t *pMsg) +{ + unsigned short pbMustBeFreed; + uchar *pszPropVal; + int bRet = 0; + rs_size_t propLen; + + if(stmt->d.s_propfilt.propID == PROP_INVALID) + goto done; + + pszPropVal = MsgGetProp(pMsg, NULL, stmt->d.s_propfilt.propID, + stmt->d.s_propfilt.propName, &propLen, + &pbMustBeFreed, NULL); + + /* Now do the compares (short list currently ;)) */ + switch(stmt->d.s_propfilt.operation ) { + case FIOP_CONTAINS: + if(rsCStrLocateInSzStr(stmt->d.s_propfilt.pCSCompValue, (uchar*) pszPropVal) != -1) + bRet = 1; + break; + case FIOP_ISEMPTY: + if(propLen == 0) + bRet = 1; /* process message! */ + break; + case FIOP_ISEQUAL: + if(rsCStrSzStrCmp(stmt->d.s_propfilt.pCSCompValue, + pszPropVal, propLen) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_STARTSWITH: + if(rsCStrSzStrStartsWithCStr(stmt->d.s_propfilt.pCSCompValue, + pszPropVal, propLen) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_REGEX: + if(rsCStrSzStrMatchRegex(stmt->d.s_propfilt.pCSCompValue, + (unsigned char*) pszPropVal, 0, &stmt->d.s_propfilt.regex_cache) == RS_RET_OK) + bRet = 1; + break; + case FIOP_EREREGEX: + if(rsCStrSzStrMatchRegex(stmt->d.s_propfilt.pCSCompValue, + (unsigned char*) pszPropVal, 1, &stmt->d.s_propfilt.regex_cache) == RS_RET_OK) + bRet = 1; + break; + default: + /* here, it handles NOP (for performance reasons) */ + assert(stmt->d.s_propfilt.operation == FIOP_NOP); + bRet = 1; /* as good as any other default ;) */ + break; + } + + /* now check if the value must be negated */ + if(stmt->d.s_propfilt.isNegated) + bRet = (bRet == 1) ? 0 : 1; + + if(Debug) { + char *cstr; + if(stmt->d.s_propfilt.propID == PROP_CEE) { + cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + DBGPRINTF("Filter: check for CEE property '%s' (value '%s') ", + cstr, pszPropVal); + free(cstr); + } else { + DBGPRINTF("Filter: check for property '%s' (value '%s') ", + propIDToName(stmt->d.s_propfilt.propID), pszPropVal); + } + if(stmt->d.s_propfilt.isNegated) + DBGPRINTF("NOT "); + if(stmt->d.s_propfilt.operation == FIOP_ISEMPTY) { + DBGPRINTF("%s : %s\n", + getFIOPName(stmt->d.s_propfilt.operation), + bRet ? "TRUE" : "FALSE"); + } else { + DBGPRINTF("%s '%s': %s\n", + getFIOPName(stmt->d.s_propfilt.operation), + rsCStrGetSzStrNoNULL(stmt->d.s_propfilt.pCSCompValue), + bRet ? "TRUE" : "FALSE"); + } + } + + /* cleanup */ + if(pbMustBeFreed) + free(pszPropVal); +done: + return bRet; +} + +/* for details, see scriptExec() header comment! */ +static void +execPROPFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *thenAct; + sbool bRet; + int i; + thenAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + if(active == NULL || active[i]) { + bRet = evalPROPFILT(stmt, pBatch->pElem[i].pMsg); + } else + bRet = 0; + thenAct[i] = bRet; + DBGPRINTF("batch: item %d PROPFILT %d\n", i, thenAct[i]); + } + + scriptExec(stmt->d.s_propfilt.t_then, pBatch, thenAct); + freeActive(thenAct); +} + +/* The rainerscript execution engine. It is debatable if that would be better + * contained in grammer/rainerscript.c, HOWEVER, that file focusses primarily + * on the parsing and object creation part. So as an actual executor, it is + * better suited here. + * param active: if NULL, all messages are active (to be processed), if non-null + * this is an array of the same size as the batch. If 1, the message + * is to be processed, otherwise not. + * NOTE: this function must receive batches which contain a single ruleset ONLY! + * rgerhards, 2012-09-04 + */ +static rsRetVal +scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active) +{ + DEFiRet; + struct cnfstmt *stmt; + + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { +dbgprintf("RRRR: scriptExec: batch of %d elements, active %p, stmt %p, nodetype %u\n", batchNumMsgs(pBatch), active, stmt, stmt->nodetype); + switch(stmt->nodetype) { + case S_NOP: + break; + case S_STOP: + execStop(pBatch, active); + break; + case S_ACT: + execAct(stmt, pBatch, active); + break; + case S_SET: + execSet(stmt, pBatch, active); + break; + case S_UNSET: + execUnset(stmt, pBatch, active); + break; + case S_CALL: + DBGPRINTF("calling ruleset\n"); // TODO: add Name + scriptExec(stmt->d.s_call.stmt, pBatch, active); + break; + case S_IF: + execIf(stmt, pBatch, active); + break; + case S_PRIFILT: + execPRIFILT(stmt, pBatch, active); + break; + case S_PROPFILT: + execPROPFILT(stmt, pBatch, active); + break; + default: + dbgprintf("error: unknown stmt type %u during exec\n", + (unsigned) stmt->nodetype); + break; + } + } + RETiRet; +} + + /* Process (consume) a batch of messages. Calls the actions configured. * If the whole batch uses a singel ruleset, we can process the batch as * a whole. Otherwise, we need to process it slower, on a message-by-message @@ -226,7 +562,7 @@ processBatch(batch_t *pBatch) if(pThis == NULL) pThis = ourConf->rulesets.pDflt; ISOBJ_TYPE_assert(pThis, ruleset); - CHKiRet(llExecFunc(&pThis->llRules, processBatchDoRules, pBatch)); + CHKiRet(scriptExec(pThis->root, pBatch, NULL)); } else { CHKiRet(processBatchMultiRuleset(pBatch)); } @@ -248,34 +584,21 @@ GetParserList(rsconf_t *conf, msg_t *pMsg) } -/* Add a new rule to the end of the current rule set. We do a number - * of checks and ignore the rule if it does not pass them. - */ -static rsRetVal -addRule(ruleset_t *pThis, rule_t **ppRule) +/* Add a script block to the current ruleset */ +static void +addScript(ruleset_t *pThis, struct cnfstmt *script) { - int iActionCnt; - DEFiRet; - - ISOBJ_TYPE_assert(pThis, ruleset); - ISOBJ_TYPE_assert(*ppRule, rule); - - CHKiRet(llGetNumElts(&(*ppRule)->llActList, &iActionCnt)); - if(iActionCnt == 0) { - errmsg.LogError(0, NO_ERRCODE, "warning: selector line without actions will be discarded"); - rule.Destruct(ppRule); - } else { - CHKiRet(llAppend(&pThis->llRules, NULL, *ppRule)); - DBGPRINTF("selector line successfully processed, %d actions\n", iActionCnt); + if(pThis->last == NULL) + pThis->root = pThis->last = script; + else { + pThis->last->next = script; + pThis->last = script; } - -finalize_it: - RETiRet; } /* set name for ruleset */ -static rsRetVal setName(ruleset_t *pThis, uchar *pszName) +static rsRetVal rulesetSetName(ruleset_t *pThis, uchar *pszName) { DEFiRet; free(pThis->pszName); @@ -344,8 +667,7 @@ finalize_it: } -/* Set a new current rule set. If the ruleset can not be found, no change happens. - */ +/* Set a new current rule set. If the ruleset can not be found, no change happens */ static rsRetVal SetCurrRuleset(rsconf_t *conf, uchar *pszName) { @@ -362,23 +684,11 @@ finalize_it: } -/* destructor we need to destruct rules inside our linked list contents. - */ -static rsRetVal -doRuleDestruct(void *pData) -{ - rule_t *pRule = (rule_t *) pData; - DEFiRet; - rule.Destruct(&pRule); - RETiRet; -} - - /* Standard-Constructor */ BEGINobjConstruct(ruleset) /* be sure to specify the object type also in END macro! */ - CHKiRet(llInit(&pThis->llRules, doRuleDestruct, NULL, NULL)); -finalize_it: + pThis->root = NULL; + pThis->last = NULL; ENDobjConstruct(ruleset) @@ -399,9 +709,6 @@ rulesetConstructFinalize(rsconf_t *conf, ruleset_t *pThis) CHKmalloc(keyName = ustrdup(pThis->pszName)); CHKiRet(llAppend(&(conf->rulesets.llRulesets), keyName, pThis)); - /* this now also is the new current ruleset */ - conf->rulesets.pCurr = pThis; - /* and also the default, if so far none has been set */ if(conf->rulesets.pDflt == NULL) conf->rulesets.pDflt = pThis; @@ -421,8 +728,8 @@ CODESTARTobjDestruct(ruleset) if(pThis->pParserLst != NULL) { parser.DestructParserList(&pThis->pParserLst); } - llDestroy(&pThis->llRules); free(pThis->pszName); + cnfstmtDestruct(pThis->root); ENDobjDestruct(ruleset) @@ -456,16 +763,13 @@ rulesetDestructForLinkedList(void *pData) return rulesetDestruct(&pThis); } -/* helper for debugPrint(), initiates rule printing */ -DEFFUNC_llExecFunc(doDebugPrintRule) -{ - return rule.DebugPrint((rule_t*) pData); -} /* debugprint for the ruleset object */ BEGINobjDebugPrint(ruleset) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDebugPrint(ruleset) dbgoprint((obj_t*) pThis, "rsyslog ruleset %s:\n", pThis->pszName); - llExecFunc(&pThis->llRules, doDebugPrintRule, NULL); + cnfstmtPrint(pThis->root, 0); + dbgoprint((obj_t*) pThis, "ruleset %s assigned parser list:\n", pThis->pszName); + printParserList(pThis->pParserLst); ENDobjDebugPrint(ruleset) @@ -486,6 +790,40 @@ debugPrintAll(rsconf_t *conf) RETiRet; } +static inline void +rulesetOptimize(ruleset_t *pRuleset) +{ + if(Debug) { + dbgprintf("ruleset '%s' before optimization:\n", + pRuleset->pszName); + rulesetDebugPrint((ruleset_t*) pRuleset); + } + cnfstmtOptimize(pRuleset->root); + if(Debug) { + dbgprintf("ruleset '%s' after optimization:\n", + pRuleset->pszName); + rulesetDebugPrint((ruleset_t*) pRuleset); + } +} + +/* helper for rulsetOptimizeAll(), optimizes a single ruleset */ +DEFFUNC_llExecFunc(doRulesetOptimizeAll) +{ + rulesetOptimize((ruleset_t*) pData); + return RS_RET_OK; +} +/* optimize all rulesets + */ +rsRetVal +rulesetOptimizeAll(rsconf_t *conf) +{ + DEFiRet; + dbgprintf("begin ruleset optimization phase\n"); + llExecFunc(&(conf->rulesets.llRulesets), doRulesetOptimizeAll, NULL); + dbgprintf("ruleset optimization phase finished.\n"); + RETiRet; +} + /* Create a ruleset-specific "main" queue for this ruleset. If one is already * defined, an error message is emitted but nothing else is done. @@ -517,7 +855,7 @@ doRulesetCreateQueue(rsconf_t *conf, int *pNewVal) rsname = (conf->rulesets.pCurr->pszName == NULL) ? (uchar*) "[ruleset]" : conf->rulesets.pCurr->pszName; DBGPRINTF("adding a ruleset-specific \"main\" queue for ruleset '%s'\n", rsname); - CHKiRet(createMainQueue(&conf->rulesets.pCurr->pQueue, rsname)); + CHKiRet(createMainQueue(&conf->rulesets.pCurr->pQueue, rsname, NULL)); finalize_it: RETiRet; @@ -539,13 +877,11 @@ rulesetCreateQueue(void __attribute__((unused)) *pVal, int *pNewVal) * rgerhards, 2009-11-04 */ static rsRetVal -doRulesetAddParser(rsconf_t *conf, uchar *pName) +doRulesetAddParser(ruleset_t *pRuleset, uchar *pName) { parser_t *pParser; DEFiRet; - assert(conf->rulesets.pCurr != NULL); - CHKiRet(objUse(parser, CORE_COMPONENT)); iRet = parser.FindParser(&pParser, pName); if(iRet == RS_RET_PARSER_NOT_FOUND) { @@ -557,9 +893,9 @@ doRulesetAddParser(rsconf_t *conf, uchar *pName) FINALIZE; } - CHKiRet(parser.AddParserToList(&conf->rulesets.pCurr->pParserLst, pParser)); + CHKiRet(parser.AddParserToList(&pRuleset->pParserLst, pParser)); - DBGPRINTF("added parser '%s' to ruleset '%s'\n", pName, conf->rulesets.pCurr->pszName); + DBGPRINTF("added parser '%s' to ruleset '%s'\n", pName, pRuleset->pszName); finalize_it: d_free(pName); /* no longer needed */ @@ -570,7 +906,72 @@ finalize_it: static rsRetVal rulesetAddParser(void __attribute__((unused)) *pVal, uchar *pName) { - return doRulesetAddParser(ourConf, pName); + return doRulesetAddParser(ourConf->rulesets.pCurr, pName); +} + + +/* Process ruleset() objects */ +rsRetVal +rulesetProcessCnf(struct cnfobj *o) +{ + struct cnfparamvals *pvals; + struct cnfparamvals *queueParams; + rsRetVal localRet; + uchar *rsName = NULL; + uchar *parserName; + int nameIdx, parserIdx; + ruleset_t *pRuleset; + struct cnfarray *ar; + int i; + uchar *rsname; + DEFiRet; + + pvals = nvlstGetParams(o->nvlst, &rspblk, NULL); + if(pvals == NULL) { + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + DBGPRINTF("ruleset param blk after rulesetProcessCnf:\n"); + cnfparamsPrint(&rspblk, pvals); + nameIdx = cnfparamGetIdx(&rspblk, "name"); + rsName = (uchar*)es_str2cstr(pvals[nameIdx].val.d.estr, NULL); + localRet = rulesetGetRuleset(loadConf, &pRuleset, rsName); + if(localRet == RS_RET_OK) { + errmsg.LogError(0, RS_RET_RULESET_EXISTS, + "error: ruleset '%s' specified more than once", + rsName); + cnfstmtDestruct(o->script); + ABORT_FINALIZE(RS_RET_RULESET_EXISTS); + } else if(localRet != RS_RET_NOT_FOUND) { + ABORT_FINALIZE(localRet); + } + CHKiRet(rulesetConstruct(&pRuleset)); + CHKiRet(rulesetSetName(pRuleset, rsName)); + CHKiRet(rulesetConstructFinalize(loadConf, pRuleset)); + addScript(pRuleset, o->script); + + /* we have only two params, so we do NOT do the usual param loop */ + parserIdx = cnfparamGetIdx(&rspblk, "parser"); + if(parserIdx != -1 && pvals[parserIdx].bUsed) { + ar = pvals[parserIdx].val.d.ar; + for(i = 0 ; i < ar->nmemb ; ++i) { + parserName = (uchar*)es_str2cstr(ar->arr[i], NULL); + doRulesetAddParser(pRuleset, parserName); + free(parserName); + } + } + + /* pick up ruleset queue parameters */ + qqueueDoCnfParams(o->nvlst, &queueParams); + if(queueCnfParamsSet(queueParams)) { + rsname = (pRuleset->pszName == NULL) ? (uchar*) "[ruleset]" : pRuleset->pszName; + DBGPRINTF("adding a ruleset-specific \"main\" queue for ruleset '%s'\n", rsname); + CHKiRet(createMainQueue(&pRuleset->pQueue, rsname, queueParams)); + } + +finalize_it: + free(rsName); + cnfparamvalsDestruct(pvals, &rspblk); + RETiRet; } @@ -595,9 +996,9 @@ CODESTARTobjQueryInterface(ruleset) pIf->IterateAllActions = iterateAllActions; pIf->DestructAllActions = destructAllActions; - pIf->AddRule = addRule; + pIf->AddScript = addScript; pIf->ProcessBatch = processBatch; - pIf->SetName = setName; + pIf->SetName = rulesetSetName; pIf->DebugPrintAll = debugPrintAll; pIf->GetCurrent = GetCurrent; pIf->GetRuleset = rulesetGetRuleset; @@ -614,7 +1015,6 @@ ENDobjQueryInterface(ruleset) */ BEGINObjClassExit(ruleset, OBJ_IS_CORE_MODULE) /* class, version */ objRelease(errmsg, CORE_COMPONENT); - objRelease(rule, CORE_COMPONENT); objRelease(parser, CORE_COMPONENT); ENDObjClassExit(ruleset) @@ -626,7 +1026,6 @@ ENDObjClassExit(ruleset) BEGINObjClassInit(ruleset, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); - CHKiRet(objUse(rule, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, rulesetDebugPrint); diff --git a/runtime/ruleset.h b/runtime/ruleset.h index f4443e18..cbf8243b 100644 --- a/runtime/ruleset.h +++ b/runtime/ruleset.h @@ -25,13 +25,15 @@ #include "queue.h" #include "linkedlist.h" +#include "rsconf.h" /* the ruleset object */ struct ruleset_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - linkedList_t llRules; /* this is NOT a pointer - no typo here ;) */ uchar *pszName; /* name of our ruleset */ qqueue_t *pQueue; /* "main" message queue, if the ruleset has its own (else NULL) */ + struct cnfstmt *root; + struct cnfstmt *last; parserList_t *pParserLst;/* list of parsers to use for this ruleset */ }; @@ -42,9 +44,7 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Construct)(ruleset_t **ppThis); rsRetVal (*ConstructFinalize)(rsconf_t *conf, ruleset_t __attribute__((unused)) *pThis); rsRetVal (*Destruct)(ruleset_t **ppThis); - rsRetVal (*IterateAllActions)(rsconf_t *conf, rsRetVal (*pFunc)(void*, void*), void* pParam); rsRetVal (*DestructAllActions)(rsconf_t *conf); - rsRetVal (*AddRule)(ruleset_t *pThis, rule_t **ppRule); rsRetVal (*SetName)(ruleset_t *pThis, uchar *pszName); rsRetVal (*ProcessBatch)(batch_t*); rsRetVal (*GetRuleset)(rsconf_t *conf, ruleset_t **ppThis, uchar*); @@ -60,8 +60,12 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ * removed conf ptr from SetName, AddRule as the flex/bison based * system uses globals in any case. */ + /* v7, 2012-09-04 */ + /* AddRule() removed */ + /*TODO:REMOVE*/rsRetVal (*IterateAllActions)(rsconf_t *conf, rsRetVal (*pFunc)(void*, void*), void* pParam); + void (*AddScript)(ruleset_t *pThis, struct cnfstmt *script); ENDinterface(ruleset) -#define rulesetCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define rulesetCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ /* prototypes */ @@ -87,5 +91,16 @@ rulesetGetName(ruleset_t *pRuleset) } +/* we will most probably convert this module back to traditional C + * calling sequence, so here we go... + */ rsRetVal rulesetGetRuleset(rsconf_t *conf, ruleset_t **ppRuleset, uchar *pszName); +rsRetVal rulesetOptimizeAll(rsconf_t *conf); +rsRetVal rulesetProcessCnf(struct cnfobj *o); + +/* Set a current rule set to already-known pointer */ +static inline void +rulesetSetCurrRulesetPtr(ruleset_t *pRuleset) { + loadConf->rulesets.pCurr = pRuleset; +} #endif /* #ifndef INCLUDED_RULESET_H */ diff --git a/runtime/srutils.c b/runtime/srutils.c index f420c0f7..7b485b23 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -92,6 +92,9 @@ syslogName_t syslogFacNames[] = { #if defined(LOG_FTP) {"ftp", LOG_FTP}, #endif +#if defined(LOG_AUDIT) + {"audit", LOG_AUDIT}, +#endif {"local0", LOG_LOCAL0}, {"local1", LOG_LOCAL1}, {"local2", LOG_LOCAL2}, @@ -524,8 +527,7 @@ char *rs_strerror_r(int errnum, char *buf, size_t buflen) { } -/* Decode a symbolic name to a numeric value - */ +/* Decode a symbolic name to a numeric value */ int decodeSyslogName(uchar *name, syslogName_t *codetab) { register syslogName_t *c; @@ -535,22 +537,23 @@ int decodeSyslogName(uchar *name, syslogName_t *codetab) ASSERT(name != NULL); ASSERT(codetab != NULL); - dbgprintf("symbolic name: %s", name); - if (isdigit((int) *name)) - { - dbgprintf("\n"); + DBGPRINTF("symbolic name: %s", name); + if(isdigit((int) *name)) { + DBGPRINTF("\n"); return (atoi((char*) name)); } strncpy((char*) buf, (char*) name, 79); - for (p = buf; *p; p++) + for(p = buf; *p; p++) { if (isupper((int) *p)) *p = tolower((int) *p); - for (c = codetab; c->c_name; c++) - if (!strcmp((char*) buf, (char*) c->c_name)) - { - dbgprintf(" ==> %d\n", c->c_val); + } + for(c = codetab; c->c_name; c++) { + if(!strcmp((char*) buf, (char*) c->c_name)) { + DBGPRINTF(" ==> %d\n", c->c_val); return (c->c_val); } + } + DBGPRINTF("\n"); return (-1); } diff --git a/runtime/stream.c b/runtime/stream.c index 742799d2..3e890c71 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -74,11 +74,12 @@ DEFobjStaticHelpers DEFobjCurrIf(zlibw) /* forward definitions */ -static rsRetVal strmFlushInternal(strm_t *pThis); +static rsRetVal strmFlushInternal(strm_t *pThis, int bFlushZip); static rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); static rsRetVal strmCloseFile(strm_t *pThis); static void *asyncWriterThread(void *pPtr); -static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); +static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush); +static rsRetVal doZipFinish(strm_t *pThis); static rsRetVal strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); @@ -341,7 +342,10 @@ static rsRetVal strmCloseFile(strm_t *pThis) (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName); if(pThis->tOperationsMode != STREAMMODE_READ) { - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); + if(pThis->iZipLevel) { + doZipFinish(pThis); + } if(pThis->bAsyncWrite) { strmWaitAsyncWriterDone(pThis); } @@ -361,7 +365,13 @@ static rsRetVal strmCloseFile(strm_t *pThis) pThis->fdDir = -1; } - if(pThis->bDeleteOnClose && pThis->pszCurrFName != NULL) { + if(pThis->bDeleteOnClose) { + if(pThis->pszCurrFName == NULL) { + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, + pThis->iFileNumDigits)); + } + DBGPRINTF("strmCloseFile: deleting '%s'\n", pThis->pszCurrFName); if(unlink((char*) pThis->pszCurrFName) == -1) { char errStr[1024]; int err = errno; @@ -369,12 +379,13 @@ static rsRetVal strmCloseFile(strm_t *pThis) DBGPRINTF("error %d unlinking '%s' - ignored: %s\n", errno, pThis->pszCurrFName, errStr); } - free(pThis->pszCurrFName); /* no longer needed in any case (just for open) */ + free(pThis->pszCurrFName); pThis->pszCurrFName = NULL; } pThis->iCurrOffs = 0; /* we are back at begin of file */ +finalize_it: RETiRet; } @@ -585,25 +596,33 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) * mode = 2 LF <not whitespace> mode, a log line starts at the beginning of a line, but following lines that are indented are part of the same log entry * This modal interface is not nearly as flexible as being able to define a regex for when a new record starts, but it's also not nearly as hard (or as slow) to implement */ - DEFiRet; uchar c; uchar finished; + rsRetVal readCharRet; + DEFiRet; ASSERT(pThis != NULL); ASSERT(ppCStr != NULL); CHKiRet(cstrConstruct(ppCStr)); - - /* now read the line */ CHKiRet(strmReadChar(pThis, &c)); - if (mode == 0){ - while(c != '\n') { + + if(mode == 0) { + /* append previous message to current message if necessary */ + if(pThis->prevLineSegment != NULL) { + CHKiRet(cstrAppendCStr(*ppCStr, pThis->prevLineSegment)); + cstrDestruct(&pThis->prevLineSegment); + } + while(c != '\n') { CHKiRet(cstrAppendChar(*ppCStr, c)); - CHKiRet(strmReadChar(pThis, &c)); + readCharRet = strmReadChar(pThis, &c); + if(readCharRet == RS_RET_EOF) {/* end of file reached without \n? */ + CHKiRet(rsCStrConstructFromCStr(&pThis->prevLineSegment, *ppCStr)); + } + CHKiRet(readCharRet); } CHKiRet(cstrFinalize(*ppCStr)); - } - if (mode == 1){ + } else if(mode == 1) { finished=0; while(finished == 0){ if(c != '\n') { @@ -624,8 +643,7 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) } } CHKiRet(cstrFinalize(*ppCStr)); - } - if (mode == 2){ + } else if(mode == 2) { /* indented follow-up lines */ finished=0; while(finished == 0){ @@ -675,9 +693,11 @@ BEGINobjConstruct(strm) /* be sure to specify the object type also in END macro! pThis->fd = -1; pThis->fdDir = -1; pThis->iUngetC = -1; + pThis->bVeryReliableZip = 0; pThis->sType = STREAMTYPE_FILE_SINGLE; pThis->sIOBufSize = glblGetIOBufSize(); pThis->tOpenMode = 0600; + pThis->prevLineSegment = NULL; ENDobjConstruct(strm) @@ -777,6 +797,7 @@ stopWriter(strm_t *pThis) BEGINobjDestruct(strm) /* be sure to specify the object type also in END and CODESTART macros! */ int i; CODESTARTobjDestruct(strm) + /* we need to stop the ZIP writer */ if(pThis->bAsyncWrite) /* Note: mutex will be unlocked in stopWriter! */ d_pthread_mutex_lock(&pThis->mut); @@ -919,14 +940,14 @@ finalize_it: /* write memory buffer to a stream object. */ static inline rsRetVal -doWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf) +doWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush) { DEFiRet; ASSERT(pThis != NULL); if(pThis->iZipLevel) { - CHKiRet(doZipWrite(pThis, pBuf, lenBuf)); + CHKiRet(doZipWrite(pThis, pBuf, lenBuf, bFlush)); } else { /* write without zipping */ CHKiRet(strmPhysWrite(pThis, pBuf, lenBuf)); @@ -971,7 +992,7 @@ doAsyncWriteInternal(strm_t *pThis, size_t lenBuf) * the background thread. -- rgerhards, 2009-07-07 */ static rsRetVal -strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlushZip) { DEFiRet; @@ -990,7 +1011,7 @@ strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) if(pThis->bAsyncWrite) { CHKiRet(doAsyncWriteInternal(pThis, lenBuf)); } else { - CHKiRet(doWriteInternal(pThis, pBuf, lenBuf)); + CHKiRet(doWriteInternal(pThis, pBuf, lenBuf, bFlushZip)); } @@ -1011,17 +1032,20 @@ asyncWriterThread(void *pPtr) sbool bTimedOut = 0; strm_t *pThis = (strm_t*) pPtr; int err; + uchar thrdName[256] = "rs:"; ISOBJ_TYPE_assert(pThis, strm); BEGINfunc + ustrncpy(thrdName+3, pThis->pszFName, sizeof(thrdName)-4); + dbgOutputTID((char*)thrdName); # if HAVE_PRCTL && defined PR_SET_NAME - if(prctl(PR_SET_NAME, "rs:asyn strmwr", 0, 0, 0) != 0) { + if(prctl(PR_SET_NAME, (char*)thrdName, 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", "stream writer"); } # endif + d_pthread_mutex_lock(&pThis->mut); while(1) { /* loop broken inside */ - d_pthread_mutex_lock(&pThis->mut); while(pThis->iCnt == 0) { if(pThis->bStopWriter) { pthread_cond_broadcast(&pThis->isEmpty); @@ -1030,18 +1054,17 @@ asyncWriterThread(void *pPtr) } if(bTimedOut && pThis->iBufPtr > 0) { /* if we timed out, we need to flush pending data */ - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); bTimedOut = 0; - continue; /* now we should have data */ + d_pthread_mutex_unlock(&pThis->mut); + continue; } bTimedOut = 0; timeoutComp(&t, pThis->iFlushInterval * 1000); /* *1000 millisconds */ if(pThis->bDoTimedWait) { if((err = pthread_cond_timedwait(&pThis->notEmpty, &pThis->mut, &t)) != 0) { - if(err == ETIMEDOUT) { - bTimedOut = 1; - } else { - bTimedOut = 1; + bTimedOut = 1; /* simulate in any case */ + if(err != ETIMEDOUT) { char errStr[1024]; rs_strerror_r(err, errStr, sizeof(errStr)); DBGPRINTF("stream async writer timeout with error (%d): %s - ignoring\n", @@ -1056,8 +1079,12 @@ asyncWriterThread(void *pPtr) bTimedOut = 0; /* we may have timed out, but there *is* work to do... */ iDeq = pThis->iDeq++ % STREAM_ASYNC_NUMBUFS; - doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf); + + /* now we can do the actual write in parallel */ + d_pthread_mutex_unlock(&pThis->mut); + doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf, 0); // TODO: flush state // TODO: error check????? 2009-07-06 + d_pthread_mutex_lock(&pThis->mut); --pThis->iCnt; if(pThis->iCnt < STREAM_ASYNC_NUMBUFS) { @@ -1065,8 +1092,8 @@ asyncWriterThread(void *pPtr) if(pThis->iCnt == 0) pthread_cond_broadcast(&pThis->isEmpty); } - d_pthread_mutex_unlock(&pThis->mut); } + d_pthread_mutex_unlock(&pThis->mut); finalize_it: ENDfunc @@ -1128,6 +1155,7 @@ strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) DEFiRet; ISOBJ_TYPE_assert(pThis, strm); + DBGPRINTF("strmPhysWrite, stream %p, len %d\n", pThis, (int) lenBuf); if(pThis->fd == -1) CHKiRet(strmOpenFile(pThis)); @@ -1166,63 +1194,97 @@ finalize_it: * rgerhards, 2009-06-04 */ static rsRetVal -doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush) { - z_stream zstrm; int zRet; /* zlib return state */ - sbool bzInitDone = RSFALSE; DEFiRet; + unsigned outavail; assert(pThis != NULL); assert(pBuf != NULL); - /* allocate deflate state */ - zstrm.zalloc = Z_NULL; - zstrm.zfree = Z_NULL; - zstrm.opaque = Z_NULL; - zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ - /* see note in file header for the params we use with deflateInit2() */ - zRet = zlibw.DeflateInit2(&zstrm, pThis->iZipLevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); - if(zRet != Z_OK) { - DBGPRINTF("error %d returned from zlib/deflateInit2()\n", zRet); - ABORT_FINALIZE(RS_RET_ZLIB_ERR); + if(!pThis->bzInitDone) { + /* allocate deflate state */ + pThis->zstrm.zalloc = Z_NULL; + pThis->zstrm.zfree = Z_NULL; + pThis->zstrm.opaque = Z_NULL; + /* see note in file header for the params we use with deflateInit2() */ + zRet = zlibw.DeflateInit2(&pThis->zstrm, pThis->iZipLevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateInit2()\n", zRet); + ABORT_FINALIZE(RS_RET_ZLIB_ERR); + } + pThis->bzInitDone = RSTRUE; } - bzInitDone = RSTRUE; /* now doing the compression */ - zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ - zstrm.avail_in = lenBuf; + pThis->zstrm.next_in = (Bytef*) pBuf; + pThis->zstrm.avail_in = lenBuf; /* run deflate() on buffer until everything has been compressed */ do { - DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", zstrm.avail_in, zstrm.total_in); - zstrm.avail_out = pThis->sIOBufSize; - zstrm.next_out = pThis->pZipBuf; - zRet = zlibw.Deflate(&zstrm, Z_FINISH); /* no bad return value */ - DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, zstrm.avail_out); - assert(zRet != Z_STREAM_ERROR); /* state not clobbered */ - if(zstrm.avail_out == pThis->sIOBufSize) - break; /* this is valid, indicates end of compression --> see zlib howto */ - CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, pThis->sIOBufSize - zstrm.avail_out)); - } while (zstrm.avail_out == 0); - assert(zstrm.avail_in == 0); /* all input will be used */ + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", pThis->zstrm.avail_in, pThis->zstrm.total_in); + pThis->zstrm.avail_out = pThis->sIOBufSize; + pThis->zstrm.next_out = pThis->pZipBuf; + zRet = zlibw.Deflate(&pThis->zstrm, bFlush ? Z_SYNC_FLUSH : Z_NO_FLUSH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pThis->zstrm.avail_out); + outavail =pThis->sIOBufSize - pThis->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, outavail)); + } + } while (pThis->zstrm.avail_out == 0); finalize_it: - if(bzInitDone) { - zRet = zlibw.DeflateEnd(&zstrm); - if(zRet != Z_OK) { - DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); - } + if(pThis->bzInitDone && pThis->bVeryReliableZip) { + doZipFinish(pThis); } - RETiRet; } + +/* finish zlib buffer, to be called before closing the ZIP file (if + * running in stream mode). + */ +static rsRetVal +doZipFinish(strm_t *pThis) +{ + int zRet; /* zlib return state */ + DEFiRet; + unsigned outavail; + assert(pThis != NULL); + + if(!pThis->bzInitDone) + goto done; + + pThis->zstrm.avail_in = 0; + /* run deflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", pThis->zstrm.avail_in, pThis->zstrm.total_in); + pThis->zstrm.avail_out = pThis->sIOBufSize; + pThis->zstrm.next_out = pThis->pZipBuf; + zRet = zlibw.Deflate(&pThis->zstrm, Z_FINISH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pThis->zstrm.avail_out); + outavail = pThis->sIOBufSize - pThis->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, outavail)); + } + } while (pThis->zstrm.avail_out == 0); + +finalize_it: + zRet = zlibw.DeflateEnd(&pThis->zstrm); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); + } + + pThis->bzInitDone = 0; +done: RETiRet; +} + /* flush stream output buffer to persistent storage. This can be called at any time * and is automatically called when the output buffer is full. * rgerhards, 2008-01-10 */ static rsRetVal -strmFlushInternal(strm_t *pThis) +strmFlushInternal(strm_t *pThis, int bFlushZip) { DEFiRet; @@ -1232,7 +1294,7 @@ strmFlushInternal(strm_t *pThis) (long) pThis->iBufPtr, (pThis->iBufPtr == 0) ? " (no need to flush)" : ""); if(pThis->tOperationsMode != STREAMMODE_READ && pThis->iBufPtr > 0) { - iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr); + iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr, bFlushZip); } RETiRet; @@ -1254,7 +1316,7 @@ strmFlush(strm_t *pThis) if(pThis->bAsyncWrite) d_pthread_mutex_lock(&pThis->mut); - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 1)); finalize_it: if(pThis->bAsyncWrite) @@ -1277,7 +1339,7 @@ static rsRetVal strmSeek(strm_t *pThis, off64_t offs) if(pThis->fd == -1) { CHKiRet(strmOpenFile(pThis)); } else { - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 0)); } long long i; DBGOPRINT((obj_t*) pThis, "file %d seek, pos %llu\n", pThis->fd, (long long unsigned) offs); @@ -1289,6 +1351,56 @@ finalize_it: RETiRet; } +/* multi-file seek, seeks to file number & offset within file. This + * is a support function for the queue, in circular mode. DO NOT USE + * IT FOR OTHER NEEDS - it may not work as expected. It will + * seek to the new position and delete interim files, as it skips them. + * Note: this code can be removed when the queue gets a new disk store + * handler (if and when it does ;)). + * The output parameter bytesDel receives the number of bytes that have + * been deleted (if a file is deleted) or 0 if nothing was deleted. + * rgerhards, 2012-11-07 + */ +rsRetVal +strmMultiFileSeek(strm_t *pThis, int FNum, off64_t offs, off64_t *bytesDel) +{ + struct stat statBuf; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, strm); + + if(FNum == 0 && offs == 0) { /* happens during queue init */ + *bytesDel = 0; + FINALIZE; + } + + if(pThis->iCurrFNum != FNum) { + /* Note: we assume that no more than one file is skipped - an + * assumption that is being used also by the whole rest of the + * code and most notably the queue subsystem. + */ + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, + pThis->iFileNumDigits)); + stat((char*)pThis->pszCurrFName, &statBuf); + *bytesDel = statBuf.st_size; + DBGPRINTF("strmMultiFileSeek: detected new filenum, was %d, new %d, " + "deleting '%s' (%lld bytes)\n", pThis->iCurrFNum, FNum, + pThis->pszCurrFName, (long long) *bytesDel); + unlink((char*)pThis->pszCurrFName); + free(pThis->pszCurrFName); + pThis->pszCurrFName = NULL; + pThis->iCurrFNum = FNum; + } else { + *bytesDel = 0; + } + pThis->iCurrOffs = offs; + +finalize_it: + RETiRet; +} + + /* seek to current offset. This is primarily a helper to readjust the OS file * pointer after a strm object has been deserialized. @@ -1320,7 +1432,7 @@ static rsRetVal strmWriteChar(strm_t *pThis, uchar c) /* if the buffer is full, we need to flush before we can write */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 0)); } /* we now always have space for one character, so we simply copy it */ *(pThis->pIOBuf + pThis->iBufPtr) = c; @@ -1380,17 +1492,17 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) ASSERT(pThis != NULL); ASSERT(pBuf != NULL); -//DBGPRINTF("strmWrite(%p, '%65.65s', %ld);, disabled %d, sizelim %ld, size %lld\n", pThis, pBuf,lenBuf, pThis->bDisabled, pThis->iSizeLimit, pThis->iCurrOffs); - if(pThis->bAsyncWrite) - d_pthread_mutex_lock(&pThis->mut); - + /* DEV DEBUG ONLY DBGPRINTF("strmWrite(%p[%s], '%65.65s', %ld);, disabled %d, sizelim %ld, size %lld\n", pThis, pThis->pszCurrFName, pBuf,(long) lenBuf, pThis->bDisabled, (long) pThis->iSizeLimit, (long long) pThis->iCurrOffs); */ if(pThis->bDisabled) ABORT_FINALIZE(RS_RET_STREAM_DISABLED); + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + iOffset = 0; do { if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis, 0)); /* get a new buffer for rest of data */ } iWrite = pThis->sIOBufSize - pThis->iBufPtr; /* this fits in current buf */ if(iWrite > lenBuf) @@ -1405,7 +1517,7 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) * write it. This seems more natural than waiting (hours?) for the next message... */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis, 0)); /* get a new buffer for rest of data */ } finalize_it: @@ -1433,6 +1545,7 @@ DEFpropSetMeth(strm, tOperationsMode, int) DEFpropSetMeth(strm, tOpenMode, mode_t) DEFpropSetMeth(strm, sType, strmType_t) DEFpropSetMeth(strm, iZipLevel, int) +DEFpropSetMeth(strm, bVeryReliableZip, int) DEFpropSetMeth(strm, bSync, int) DEFpropSetMeth(strm, sIOBufSize, size_t) DEFpropSetMeth(strm, iSizeLimit, off_t) @@ -1564,7 +1677,7 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) ISOBJ_TYPE_assert(pThis, strm); ISOBJ_TYPE_assert(pStrm, strm); - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); CHKiRet(obj.BeginSerialize(pStrm, (obj_t*) pThis)); objSerializeSCALAR(pStrm, iCurrFNum, INT); @@ -1584,6 +1697,8 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) l = pThis->iCurrOffs; objSerializeSCALAR_VAR(pStrm, iCurrOffs, INT64, l); + objSerializePTR(pStrm, prevLineSegment, PSZ); + CHKiRet(obj.EndSerialize(pStrm)); finalize_it: @@ -1689,6 +1804,8 @@ static rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) CHKiRet(strmSetiFileNumDigits(pThis, pProp->val.num)); } else if(isProp("bDeleteOnClose")) { CHKiRet(strmSetbDeleteOnClose(pThis, pProp->val.num)); + } else if(isProp("prevLineSegment")) { + CHKiRet(rsCStrConstructFromCStr(&pThis->prevLineSegment, pProp->val.pStr)); } finalize_it: @@ -1757,6 +1874,7 @@ CODESTARTobjQueryInterface(strm) pIf->SettOpenMode = strmSettOpenMode; pIf->SetsType = strmSetsType; pIf->SetiZipLevel = strmSetiZipLevel; + pIf->SetbVeryReliableZip = strmSetbVeryReliableZip; pIf->SetbSync = strmSetbSync; pIf->SetsIOBufSize = strmSetsIOBufSize; pIf->SetiSizeLimit = strmSetiSizeLimit; diff --git a/runtime/stream.h b/runtime/stream.h index a01929f2..b7e74074 100644 --- a/runtime/stream.h +++ b/runtime/stream.h @@ -124,6 +124,8 @@ typedef struct strm_s { sbool bAsyncWrite; /* do asynchronous writes (always if a flush interval is given) */ sbool bStopWriter; /* shall writer thread terminate? */ sbool bDoTimedWait; /* instruct writer thread to do a times wait to support flush timeouts */ + sbool bzInitDone; /* did we do an init of zstrm already? */ + sbool bVeryReliableZip; /* shall we write interim headers to create a very reliable ZIP file? */ int iFlushInterval; /* flush in which interval - 0, no flushing */ pthread_mutex_t mut;/* mutex for flush in async mode */ pthread_cond_t notFull; @@ -132,6 +134,7 @@ typedef struct strm_s { unsigned short iEnq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ unsigned short iDeq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ short iCnt; /* current nbr of elements in buffer */ + z_stream zstrm; /* zip stream to use */ struct { uchar *pBuf; size_t lenBuf; @@ -141,6 +144,7 @@ typedef struct strm_s { off_t iSizeLimit; /* file size limit, 0 = no limit */ uchar *pszSizeLimitCmd; /* command to carry out when size limit is reached */ sbool bIsTTY; /* is this a tty file? */ + cstr_t *prevLineSegment; /* for ReadLine, previous, unwritten part of file */ } strm_t; @@ -181,11 +185,18 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ INTERFACEpropSetMeth(strm, pszSizeLimitCmd, uchar*); /* v6 added */ rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, int mode); + /* v7 added 2012-09-14 */ + INTERFACEpropSetMeth(strm, bVeryReliableZip, int); ENDinterface(strm) -#define strmCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define strmCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ +static inline int +strmGetCurrFileNum(strm_t *pStrm) { + return pStrm->iCurrFNum; +} /* prototypes */ PROTOTYPEObjClassInit(strm); +rsRetVal strmMultiFileSeek(strm_t *pThis, int fileNum, off64_t offs, off64_t *bytesDel); #endif /* #ifndef STREAM_H_INCLUDED */ diff --git a/runtime/stringbuf.c b/runtime/stringbuf.c index e7fd72c2..d9f80231 100644 --- a/runtime/stringbuf.c +++ b/runtime/stringbuf.c @@ -482,6 +482,8 @@ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) register uchar *pC; rsCHECKVALIDOBJECT(pThis, OIDrsCStr); + if(pThis->iStrLen == 0) + goto done; /* empty string -> nothing to trim ;) */ i = pThis->iStrLen; pC = pThis->pBuf + i - 1; while(i > 0 && isspace((int)*pC)) { @@ -492,7 +494,7 @@ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) pThis->iStrLen = i; pThis->pBuf[pThis->iStrLen] = '0'; /* we always have this space */ - return RS_RET_OK; +done: return RS_RET_OK; } /* compare two string objects - works like strcmp(), but operates @@ -868,13 +870,7 @@ int rsCStrSzStrCmp(cstr_t *pCS1, uchar *psz, size_t iLenSz) * length, so we need to actually check if they * are equal. */ - register size_t i; - for(i = 0 ; i < iLenSz ; ++i) { - if(pCS1->pBuf[i] != psz[i]) - return pCS1->pBuf[i] - psz[i]; - } - /* if we arrive here, the strings are equal */ - return 0; + return strncmp((char*)pCS1->pBuf, (char*)psz, iLenSz); } else return pCS1->iStrLen - iLenSz; diff --git a/runtime/strms_sess.c b/runtime/strms_sess.c index d14f0b37..2537e8d8 100644 --- a/runtime/strms_sess.c +++ b/runtime/strms_sess.c @@ -38,12 +38,14 @@ #include "errmsg.h" #include "netstrm.h" #include "msg.h" +#include "prop.h" #include "datetime.h" /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) DEFobjCurrIf(errmsg) DEFobjCurrIf(netstrm) DEFobjCurrIf(datetime) @@ -86,7 +88,8 @@ CODESTARTobjDestruct(strms_sess) } /* now destruct our own properties */ free(pThis->fromHost); - free(pThis->fromHostIP); + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); ENDobjDestruct(strms_sess) @@ -111,17 +114,18 @@ SetHost(strms_sess_t *pThis, uchar *pszHost) RETiRet; } -/* set the remote host's IP. Note that the caller *hands over* the string. That is, +/* set the remote host's IP. Note that the caller *hands over* the property. That is, * the caller no longer controls it once SetHostIP() has received it. Most importantly, - * the caller must not free it. -- rgerhards, 2008-05-16 + * the caller must not destruct it. -- rgerhards, 2008-05-16 */ static rsRetVal -SetHostIP(strms_sess_t *pThis, uchar *pszHostIP) +SetHostIP(strms_sess_t *pThis, prop_t *ip) { DEFiRet; ISOBJ_TYPE_assert(pThis, strms_sess); - free(pThis->fromHostIP); - pThis->fromHostIP = pszHostIP; + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); + pThis->fromHostIP = ip; RETiRet; } @@ -188,8 +192,8 @@ Close(strms_sess_t *pThis) netstrm.Destruct(&pThis->pStrm); free(pThis->fromHost); pThis->fromHost = NULL; /* not really needed, but... */ - free(pThis->fromHostIP); - pThis->fromHostIP = NULL; /* not really needed, but... */ + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); RETiRet; } @@ -284,6 +288,7 @@ BEGINObjClassInit(strms_sess, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANGE CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(netstrm, LM_NETSTRMS_FILENAME)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); iMaxLine = glbl.GetMaxLine(); /* get maximum size we currently support */ diff --git a/runtime/strms_sess.h b/runtime/strms_sess.h index 5c0309f8..86f692a8 100644 --- a/runtime/strms_sess.h +++ b/runtime/strms_sess.h @@ -33,9 +33,8 @@ struct strms_sess_s { strmsrv_t *pSrv; /* pointer back to my server (e.g. for callbacks) */ strmLstnPortList_t *pLstnInfo; /* pointer back to listener info */ netstrm_t *pStrm; -// uchar *pMsg; /* message (fragment) received */ uchar *fromHost; - uchar *fromHostIP; + prop_t *fromHostIP; void *pUsr; /* a user-pointer */ }; @@ -54,15 +53,17 @@ BEGINinterface(strms_sess) /* name must also be changed in ENDinterface macro! * rsRetVal (*SetUsrP)(strms_sess_t*, void*); void* (*GetUsrP)(strms_sess_t*); rsRetVal (*SetHost)(strms_sess_t *pThis, uchar*); - rsRetVal (*SetHostIP)(strms_sess_t *pThis, uchar*); + rsRetVal (*SetHostIP)(strms_sess_t *pThis, prop_t*); rsRetVal (*SetStrm)(strms_sess_t *pThis, netstrm_t*); rsRetVal (*SetOnMsgReceive)(strms_sess_t *pThis, rsRetVal (*OnMsgReceive)(strms_sess_t*, uchar*, int)); ENDinterface(strms_sess) -#define strms_sessCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define strms_sessCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface changes * to version v2, rgerhards, 2009-05-22 * - Data structures changed * - SetLstnInfo entry point added + * version 3, rgerhads, 2013-01-21: + * - signature of SetHostIP() changed */ diff --git a/runtime/strmsrv.c b/runtime/strmsrv.c index 8310e832..e8b544b8 100644 --- a/runtime/strmsrv.c +++ b/runtime/strmsrv.c @@ -70,6 +70,7 @@ #include "netstrm.h" #include "nssel.h" #include "errmsg.h" +#include "prop.h" #include "unicode-helper.h" MODULE_TYPE_LIB @@ -89,6 +90,7 @@ DEFobjCurrIf(net) DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) DEFobjCurrIf(nssel) +DEFobjCurrIf(prop) /* forward definitions */ static rsRetVal create_strm_socket(strmsrv_t *pThis); @@ -418,7 +420,7 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes int iSess = -1; struct sockaddr_storage *addr; uchar *fromHostFQDN = NULL; - uchar *fromHostIP = NULL; + prop_t *ip = NULL; ISOBJ_TYPE_assert(pThis, strmsrv); assert(pLstnInfo != NULL); @@ -444,7 +446,7 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes /* get the host name */ CHKiRet(netstrm.GetRemoteHName(pNewStrm, &fromHostFQDN)); - CHKiRet(netstrm.GetRemoteIP(pNewStrm, &fromHostIP)); + CHKiRet(netstrm.GetRemoteIP(pNewStrm, &ip)); CHKiRet(netstrm.GetRemAddr(pNewStrm, &addr)); /* TODO: check if we need to strip the domain name here -- rgerhards, 2008-04-24 */ @@ -467,8 +469,8 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes */ CHKiRet(strms_sess.SetHost(pSess, fromHostFQDN)); fromHostFQDN = NULL; /* we handed this string over */ - CHKiRet(strms_sess.SetHostIP(pSess, fromHostIP)); - fromHostIP = NULL; /* we handed this string over */ + CHKiRet(strms_sess.SetHostIP(pSess, ip)); + ip = NULL; /* we handed this string over */ CHKiRet(strms_sess.SetStrm(pSess, pNewStrm)); pNewStrm = NULL; /* prevent it from being freed in error handler, now done in strms_sess! */ CHKiRet(strms_sess.ConstructFinalize(pSess)); @@ -489,7 +491,8 @@ finalize_it: if(pNewStrm != NULL) netstrm.Destruct(&pNewStrm); free(fromHostFQDN); - free(fromHostIP); + if(ip != NULL) + prop.Destruct(&ip); } RETiRet; @@ -908,6 +911,7 @@ CODESTARTObjClassExit(strmsrv) objRelease(strms_sess, DONT_LOAD_LIB); objRelease(conf, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(netstrms, DONT_LOAD_LIB); objRelease(nssel, DONT_LOAD_LIB); @@ -930,6 +934,7 @@ BEGINObjClassInit(strmsrv, 1, OBJ_IS_LOADABLE_MODULE) /* class, version - CHANGE CHKiRet(objUse(strms_sess, DONT_LOAD_LIB)); CHKiRet(objUse(conf, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, strmsrvDebugPrint); diff --git a/runtime/typedefs.h b/runtime/typedefs.h index 4e7f1622..5cc24e4a 100644 --- a/runtime/typedefs.h +++ b/runtime/typedefs.h @@ -92,6 +92,10 @@ typedef struct cfgmodules_etry_s cfgmodules_etry_t; typedef struct outchannels_s outchannels_t; typedef struct modConfData_s modConfData_t; typedef struct instanceConf_s instanceConf_t; +typedef struct ratelimit_s ratelimit_t; +typedef struct action_s action_t; +typedef int rs_size_t; /* we do never need more than 2Gig strings, signed permits to + * use -1 as a special flag. */ typedef rsRetVal (*prsf_t)(struct vmstk_s*, int); /* pointer to a RainerScript function */ typedef uint64 qDeqID; /* queue Dequeue order ID. 32 bits is considered dangerously few */ @@ -162,6 +166,7 @@ typedef enum cslCmdHdlrType { eCmdHdlrSeverity, eCmdHdlrGetWord, eCmdHdlrString, + eCmdHdlrArray, eCmdHdlrQueueType, eCmdHdlrGoneAway /* statment existed, but is no longer supported */ } ecslCmdHdrlType; diff --git a/runtime/wtp.c b/runtime/wtp.c index a53a9888..f8d3588b 100644 --- a/runtime/wtp.c +++ b/runtime/wtp.c @@ -383,6 +383,7 @@ wtpWorker(void *arg) /* the arg is actually a wti object, even though we are in } # endif + dbgOutputTID((char*)thrdName); pthread_cleanup_push(wtpWrkrExecCancelCleanup, pWti); wtiWorker(pWti); pthread_cleanup_pop(0); diff --git a/tcps_sess.c b/tcps_sess.c index e7149cb7..5821e443 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -47,6 +47,7 @@ #include "msg.h" #include "datetime.h" #include "prop.h" +#include "ratelimit.h" #include "debug.h" @@ -139,24 +140,20 @@ finalize_it: RETiRet; } -/* set the remote host's IP. Note that the caller *hands over* the string. That is, +/* set the remote host's IP. Note that the caller *hands over* the property. That is, * the caller no longer controls it once SetHostIP() has received it. Most importantly, - * the caller must not free it. -- rgerhards, 2008-05-16 + * the caller must not destruct it. -- rgerhards, 2008-05-16 */ static rsRetVal -SetHostIP(tcps_sess_t *pThis, uchar *pszHostIP) +SetHostIP(tcps_sess_t *pThis, prop_t *ip) { DEFiRet; - ISOBJ_TYPE_assert(pThis, tcps_sess); - if(pThis->fromHostIP == NULL) - CHKiRet(prop.Construct(&pThis->fromHostIP)); - - CHKiRet(prop.SetString(pThis->fromHostIP, pszHostIP, ustrlen(pszHostIP))); - -finalize_it: - free(pszHostIP); + if(pThis->fromHostIP != NULL) { + prop.Destruct(&pThis->fromHostIP); + } + pThis->fromHostIP = ip; RETiRet; } @@ -264,14 +261,7 @@ defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttG MsgSetRuleset(pMsg, pThis->pLstnInfo->pRuleset); STATSCOUNTER_INC(pThis->pLstnInfo->ctrSubmit, pThis->pLstnInfo->mutCtrSubmit); - if(pMultiSub == NULL) { - CHKiRet(submitMsg(pMsg)); - } else { - pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; - if(pMultiSub->nElem == pMultiSub->maxElem) - CHKiRet(multiSubmitMsg(pMultiSub)); - } - + ratelimitAddMsg(pThis->pLstnInfo->ratelimiter, pMultiSub, pMsg); finalize_it: /* reset status variables */ @@ -368,7 +358,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt ISOBJ_TYPE_assert(pThis, tcps_sess); if(pThis->inputState == eAtStrtFram) { - if(pThis->bSuppOctetFram && isdigit((int) c)) { + if(pThis->bSuppOctetFram && c >= '0' && c <= '9') { pThis->inputState = eInOctetCnt; pThis->iOctetsRemain = 0; pThis->eFraming = TCP_FRAMING_OCTET_COUNTING; @@ -379,7 +369,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt } if(pThis->inputState == eInOctetCnt) { - if(isdigit(c)) { + if(c >= '0' && c <= '9') { /* isdigit() the faster way */ pThis->iOctetsRemain = pThis->iOctetsRemain * 10 + c - '0'; } else { /* done with the octet count, so this must be the SP terminator */ DBGPRINTF("TCP Message with octet-counter, size %d.\n", pThis->iOctetsRemain); @@ -487,11 +477,7 @@ DataRcvd(tcps_sess_t *pThis, char *pData, size_t iLen) while(pData < pEnd) { CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); } - - if(multiSub.nElem > 0) { - /* submit anything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&multiSub)); - } + iRet = multiSubmitFlush(&multiSub); finalize_it: RETiRet; diff --git a/tcps_sess.h b/tcps_sess.h index 054ce397..4506cf07 100644 --- a/tcps_sess.h +++ b/tcps_sess.h @@ -66,16 +66,18 @@ BEGINinterface(tcps_sess) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetLstnInfo)(tcps_sess_t *pThis, tcpLstnPortList_t *pLstnInfo); rsRetVal (*SetUsrP)(tcps_sess_t*, void*); rsRetVal (*SetHost)(tcps_sess_t *pThis, uchar*); - rsRetVal (*SetHostIP)(tcps_sess_t *pThis, uchar*); + rsRetVal (*SetHostIP)(tcps_sess_t *pThis, prop_t*); rsRetVal (*SetStrm)(tcps_sess_t *pThis, netstrm_t*); rsRetVal (*SetMsgIdx)(tcps_sess_t *pThis, int); rsRetVal (*SetOnMsgReceive)(tcps_sess_t *pThis, rsRetVal (*OnMsgReceive)(tcps_sess_t*, uchar*, int)); ENDinterface(tcps_sess) -#define tcps_sessCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define tcps_sessCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface changes * to version v2, rgerhards, 2009-05-22 * - Data structures changed * - SetLstnInfo entry point added + * version 3, rgerhards, 2013-01-21: + * - signature of SetHostIP() changed */ @@ -72,6 +72,7 @@ #include "nspoll.h" #include "errmsg.h" #include "ruleset.h" +#include "ratelimit.h" #include "unicode-helper.h" @@ -151,6 +152,9 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort, int bSuppOctetFram) snprintf((char*)statname, sizeof(statname), "%s(%s)", pThis->pszInputName, pszPort); statname[sizeof(statname)-1] = '\0'; /* just to be on the save side... */ CHKiRet(statsobj.SetName(pEntry->stats, statname)); + CHKiRet(ratelimitNew(&pEntry->ratelimiter, "tcperver", NULL)); + ratelimitSetLinuxLike(pEntry->ratelimiter, pThis->ratelimitInterval, pThis->ratelimitBurst); + ratelimitSetThreadSafe(pEntry->ratelimiter); STATSCOUNTER_INIT(pEntry->ctrSubmit, pEntry->mutCtrSubmit); CHKiRet(statsobj.AddCounter(pEntry->stats, UCHAR_CONSTANT("submitted"), ctrType_IntCtr, &(pEntry->ctrSubmit))); @@ -295,6 +299,7 @@ static void deinit_tcp_listener(tcpsrv_t *pThis) while(pEntry != NULL) { free(pEntry->pszPort); prop.Destruct(&pEntry->pInputName); + ratelimitDestruct(pEntry->ratelimiter); pDel = pEntry; pEntry = pEntry->pNext; free(pDel); @@ -419,7 +424,7 @@ SessAccept(tcpsrv_t *pThis, tcpLstnPortList_t *pLstnInfo, tcps_sess_t **ppSess, int iSess = -1; struct sockaddr_storage *addr; uchar *fromHostFQDN = NULL; - uchar *fromHostIP = NULL; + prop_t *fromHostIP; ISOBJ_TYPE_assert(pThis, tcpsrv); assert(pLstnInfo != NULL); @@ -470,7 +475,6 @@ SessAccept(tcpsrv_t *pThis, tcpLstnPortList_t *pLstnInfo, tcps_sess_t **ppSess, CHKiRet(tcps_sess.SetHost(pSess, fromHostFQDN)); fromHostFQDN = NULL; /* we handed this string over */ CHKiRet(tcps_sess.SetHostIP(pSess, fromHostIP)); - fromHostIP = NULL; /* we handed this string over */ CHKiRet(tcps_sess.SetStrm(pSess, pNewStrm)); pNewStrm = NULL; /* prevent it from being freed in error handler, now done in tcps_sess! */ CHKiRet(tcps_sess.SetMsgIdx(pSess, 0)); @@ -493,7 +497,6 @@ finalize_it: if(pNewStrm != NULL) netstrm.Destruct(&pNewStrm); free(fromHostFQDN); - free(fromHostIP); } RETiRet; @@ -913,6 +916,8 @@ BEGINobjConstruct(tcpsrv) /* be sure to specify the object type also in END macr pThis->addtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; pThis->bDisableLFDelim = 0; pThis->OnMsgReceive = NULL; + pThis->ratelimitInterval = 0; + pThis->ratelimitBurst = 10000; pThis->bUseFlowControl = 1; ENDobjConstruct(tcpsrv) @@ -1120,6 +1125,17 @@ finalize_it: } +/* Set the linux-like ratelimiter settings */ +static rsRetVal +SetLinuxLikeRatelimiters(tcpsrv_t *pThis, int ratelimitInterval, int ratelimitBurst) +{ + DEFiRet; + pThis->ratelimitInterval = ratelimitInterval; + pThis->ratelimitBurst = ratelimitBurst; + RETiRet; +} + + /* Set the ruleset (ptr) to use */ static rsRetVal SetRuleset(tcpsrv_t *pThis, ruleset_t *pRuleset) @@ -1270,6 +1286,7 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetCBOnErrClose = SetCBOnErrClose; pIf->SetOnMsgReceive = SetOnMsgReceive; pIf->SetRuleset = SetRuleset; + pIf->SetLinuxLikeRatelimiters = SetLinuxLikeRatelimiters; pIf->SetNotificationOnRemoteClose = SetNotificationOnRemoteClose; finalize_it: @@ -42,6 +42,7 @@ struct tcpLstnPortList_s { ruleset_t *pRuleset; /**< associated ruleset */ statsobj_t *stats; /**< associated stats object */ sbool bSuppOctetFram; /**< do we support octect-counted framing? (if no->legay only!)*/ + ratelimit_t *ratelimiter; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) tcpLstnPortList_t *pNext; /**< next port or NULL */ }; @@ -70,6 +71,8 @@ struct tcpsrv_s { int addtlFrameDelim; /**< additional frame delimiter for plain TCP syslog framing (e.g. to handle NetScreen) */ int bDisableLFDelim; /**< if 1, standard LF frame delimiter is disabled (*very dangerous*) */ + int ratelimitInterval; + int ratelimitBurst; tcps_sess_t **pSessions;/**< array of all of our sessions */ void *pUsr; /**< a user-settable pointer (provides extensibility for "derived classes")*/ /* callbacks */ @@ -142,8 +145,10 @@ BEGINinterface(tcpsrv) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetUseFlowControl)(tcpsrv_t*, int); /* added v11 -- rgerhards, 2011-05-09 */ rsRetVal (*SetKeepAlive)(tcpsrv_t*, int); + /* added v13 -- rgerhards, 2012-10-15 */ + rsRetVal (*SetLinuxLikeRatelimiters)(tcpsrv_t *pThis, int interval, int burst); ENDinterface(tcpsrv) -#define tcpsrvCURR_IF_VERSION 12 /* increment whenever you change the interface structure! */ +#define tcpsrvCURR_IF_VERSION 13 /* increment whenever you change the interface structure! */ /* change for v4: * - SetAddtlFrameDelim() added -- rgerhards, 2008-12-10 * - SetInputName() added -- rgerhards, 2008-12-10 @@ -44,6 +44,7 @@ #include "errmsg.h" #include "strgen.h" #include "rsconf.h" +#include "msg.h" #include "unicode-helper.h" /* static data */ @@ -54,9 +55,10 @@ DEFobjCurrIf(strgen) /* tables for interfacing with the v6 config system */ static struct cnfparamdescr cnfparamdescr[] = { { "name", eCmdHdlrString, 1 }, - { "type", eCmdHdlrString, 0 }, + { "type", eCmdHdlrString, 1 }, { "string", eCmdHdlrString, 0 }, { "plugin", eCmdHdlrString, 0 }, + { "subtree", eCmdHdlrString, 0 }, { "option.stdsql", eCmdHdlrBinary, 0 }, { "option.sql", eCmdHdlrBinary, 0 }, { "option.json", eCmdHdlrBinary, 0 } @@ -139,14 +141,16 @@ finalize_it: * offers big performance improvements. * rewritten 2009-06-19 rgerhards */ -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf) +rsRetVal +tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf, + struct syslogTime *ttNow) { DEFiRet; struct templateEntry *pTpe; size_t iBuf; unsigned short bMustBeFreed = 0; uchar *pVal; - size_t iLenVal = 0; + rs_size_t iLenVal = 0; assert(pTpl != NULL); assert(pMsg != NULL); @@ -158,6 +162,23 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t * FINALIZE; } + if(pTpl->subtree != NULL) { + /* only a single CEE subtree must be provided */ + /* note: we could optimize the code below, however, this is + * not worth the effort, as this passing mode is not expected + * in subtree mode and so most probably only used for debug & test. + */ + getCEEPropVal(pMsg, pTpl->subtree, &pVal, &iLenVal, &bMustBeFreed); + if(iLenVal >= (rs_size_t)*pLenBuf) /* we reserve one char for the final \0! */ + CHKiRet(ExtendBuf(ppBuf, pLenBuf, iLenVal + 1)); + memcpy(*ppBuf, pVal, iLenVal+1); + if(bMustBeFreed) + free(pVal); + FINALIZE; + } + + /* we have a "regular" template with template entries */ + /* loop through the template. We obtain one value * and copy it over to our dynamic string buffer. Then, we * free the obtained value (if requested). We continue this @@ -172,7 +193,8 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t * bMustBeFreed = 0; } else if(pTpe->eEntryType == FIELD) { pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &iLenVal, &bMustBeFreed); + pTpe->data.field.propName, &iLenVal, + &bMustBeFreed, ttNow); /* we now need to check if we should use SQL option. In this case, * we must go over the generated string and escape '\'' characters. * rgerhards, 2005-09-22: the option values below look somewhat misplaced, @@ -226,13 +248,14 @@ finalize_it: * is indicated by a NULL pointer. * rgerhards, 2009-04-03 */ -rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) +rsRetVal +tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime *ttNow) { DEFiRet; struct templateEntry *pTpe; uchar **pArr; int iArr; - size_t propLen; + rs_size_t propLen; unsigned short bMustBeFreed; uchar *pVal; @@ -240,11 +263,24 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) assert(pMsg != NULL); assert(ppArr != NULL); + if(pTpl->subtree) { + /* Note: this mode is untested, as there is no official plugin + * using array passing, so I simply could not test it. + */ + CHKmalloc(pArr = calloc(2, sizeof(uchar*))); + getCEEPropVal(pMsg, pTpl->subtree, &pVal, &propLen, &bMustBeFreed); + if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ + pArr[0] = pVal; /* ... so we can use it! */ + } else { + CHKmalloc(pArr[0] = (uchar*)strdup((char*) pVal)); + } + FINALIZE; + } + /* loop through the template. We obtain one value, create a * private copy (if necessary), add it to the string array * and then on to the next until we have processed everything. */ - CHKmalloc(pArr = calloc(pTpl->tpenElements + 1, sizeof(uchar*))); iArr = 0; @@ -254,7 +290,8 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) CHKmalloc(pArr[iArr] = (uchar*)strdup((char*) pTpe->data.constant.pConstant)); } else if(pTpe->eEntryType == FIELD) { pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, &bMustBeFreed); + pTpe->data.field.propName, &propLen, + &bMustBeFreed, ttNow); if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ pArr[iArr] = pVal; /* ... so we can use it! */ } else { @@ -277,15 +314,28 @@ finalize_it: * tpltoString(). * rgerhards, 2012-08-29 */ -rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson) +rsRetVal +tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson, struct syslogTime *ttNow) { struct templateEntry *pTpe; - size_t propLen; + rs_size_t propLen; unsigned short bMustBeFreed; uchar *pVal; struct json_object *json, *jsonf; + rsRetVal localRet; DEFiRet; + if(pTpl->subtree != NULL){ + localRet = jsonFind(pMsg, pTpl->subtree, pjson); + if(*pjson == NULL) { + /* we need to have a root object! */ + *pjson = json_object_new_object(); + } else { + json_object_get(*pjson); /* inc refcount */ + } + FINALIZE; + } + json = json_object_new_object(); for(pTpe = pTpl->pEntryRoot ; pTpe != NULL ; pTpe = pTpe->pNext) { if(pTpe->eEntryType == CONSTANT) { @@ -294,19 +344,34 @@ rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjso jsonf = json_object_new_string((char*) pTpe->data.constant.pConstant); json_object_object_add(json, (char*)pTpe->fieldName, jsonf); } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, &bMustBeFreed); - if(pTpe->data.field.options.bMandatory || propLen > 0) { - jsonf = json_object_new_string_len((char*)pVal, propLen); - json_object_object_add(json, (char*)pTpe->fieldName, jsonf); - } - if(bMustBeFreed) { /* json-c makes its own private copy! */ - free(pVal); + if(pTpe->data.field.propid == PROP_CEE) { + localRet = msgGetCEEPropJSON(pMsg, pTpe->data.field.propName, &jsonf); + if(localRet == RS_RET_OK) { + json_object_object_add(json, (char*)pTpe->fieldName, json_object_get(jsonf)); + } else { + DBGPRINTF("tplToJSON: error %d looking up property\n", + localRet); + if(pTpe->data.field.options.bMandatory) { + json_object_object_add(json, (char*)pTpe->fieldName, NULL); + } + } + } else { + pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, + pTpe->data.field.propName, &propLen, + &bMustBeFreed, ttNow); + if(pTpe->data.field.options.bMandatory || propLen > 0) { + jsonf = json_object_new_string_len((char*)pVal, propLen); + json_object_object_add(json, (char*)pTpe->fieldName, jsonf); + } + if(bMustBeFreed) { /* json-c makes its own private copy! */ + free(pVal); + } } } } - *pjson = (iRet == RS_RET_OK) ? json : NULL; + +finalize_it: RETiRet; } @@ -364,7 +429,7 @@ static void doEmergencyEscape(register uchar *p, int mode) * 2005-09-22 rgerhards */ rsRetVal -doEscape(uchar **pp, size_t *pLen, unsigned short *pbMustBeFreed, int mode) +doEscape(uchar **pp, rs_size_t *pLen, unsigned short *pbMustBeFreed, int mode) { DEFiRet; uchar *p = NULL; @@ -487,14 +552,20 @@ tplConstruct(rsconf_t *conf) /* helper to tplAddLine. Parses a constant and generates * the necessary structure. - * returns: 0 - ok, 1 - failure + * Paramter "bDoEscapes" is to support legacy vs. v6+ config system. In + * legacy, we must do escapes ourselves, whereas v6+ passes in already + * escaped strings (which we are NOT permitted to further escape, this would + * cause invalid result strings!). Note: if escapes are not permitted, + * quotes (") are just a regular character and do NOT terminate the constant! */ -static int do_Constant(unsigned char **pp, struct template *pTpl) +static rsRetVal +do_Constant(unsigned char **pp, struct template *pTpl, int bDoEscapes) { register unsigned char *p; cstr_t *pStrB; struct templateEntry *pTpe; int i; + DEFiRet; assert(pp != NULL); assert(*pp != NULL); @@ -502,13 +573,12 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) p = *pp; - if(cstrConstruct(&pStrB) != RS_RET_OK) - return 1; + CHKiRet(cstrConstruct(&pStrB)); /* process the message and expand escapes * (additional escapes can be added here if needed) */ - while(*p && *p != '%' && *p != '\"') { - if(*p == '\\') { + while(*p && *p != '%' && !(bDoEscapes && *p == '\"')) { + if(bDoEscapes && *p == '\\') { switch(*++p) { case '\0': /* the best we can do - it's invalid anyhow... */ @@ -557,7 +627,7 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) if((pTpe = tpeConstruct(pTpl)) == NULL) { rsCStrDestruct(&pStrB); - return 1; + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } pTpe->eEntryType = CONSTANT; cstrFinalize(pStrB); @@ -567,12 +637,12 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) * 2005-09-09 rgerhards */ pTpe->data.constant.iLenConstant = rsCStrLen(pStrB); - if(cstrConvSzStrAndDestruct(pStrB, &pTpe->data.constant.pConstant, 0) != RS_RET_OK) - return 1; + CHKiRet(cstrConvSzStrAndDestruct(pStrB, &pTpe->data.constant.pConstant, 0)); *pp = p; - return 0; +finalize_it: + RETiRet; } @@ -677,11 +747,11 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe) /* helper to tplAddLine. Parses a parameter and generates * the necessary structure. - * returns: 0 - ok, 1 - failure */ -static int do_Parameter(unsigned char **pp, struct template *pTpl) +static rsRetVal +do_Parameter(uchar **pp, struct template *pTpl) { - unsigned char *p; + uchar *p; cstr_t *pStrProp; cstr_t *pStrField = NULL; struct templateEntry *pTpe; @@ -693,21 +763,15 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) unsigned char *regex_char; unsigned char *regex_end; #endif + DEFiRet; assert(pp != NULL); assert(*pp != NULL); assert(pTpl != NULL); - p = (unsigned char*) *pp; - - if(cstrConstruct(&pStrProp) != RS_RET_OK) - return 1; - - if((pTpe = tpeConstruct(pTpl)) == NULL) { - /* TODO: add handler */ - dbgprintf("Could not allocate memory for template parameter!\n"); - return 1; - } + p = (uchar*) *pp; + CHKiRet(cstrConstruct(&pStrProp)); + CHKmalloc(pTpe = tpeConstruct(pTpl)); pTpe->eEntryType = FIELD; while(*p && *p != '%' && *p != ':') { @@ -719,19 +783,22 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) cstrFinalize(pStrProp); if(propNameToID(pStrProp, &pTpe->data.field.propid) != RS_RET_OK) { + errmsg.LogError(0, RS_RET_TPL_INVLD_PROP, "template '%s': invalid parameter '%s'", + pTpl->pszName, cstrGetSzStrNoNULL(pStrProp)); cstrDestruct(&pStrProp); - return 1; + ABORT_FINALIZE(RS_RET_TPL_INVLD_PROP); } if(pTpe->data.field.propid == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - if((pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(pStrProp)+2, cstrLen(pStrProp)-2)) == NULL) { + if((pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(pStrProp)+1, cstrLen(pStrProp)-1)) == NULL) { cstrDestruct(&pStrProp); - return 1; + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } } /* Check frompos, if it has an R, then topos should be a regex */ if(*p == ':') { + pTpe->bComplexProcessing = 1; ++p; /* eat ':' */ #ifdef FEATURE_REGEXP if(*p == 'R') { @@ -791,8 +858,8 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) * comma itself is already part of the next field. */ } else { - errmsg.LogError(0, NO_ERRCODE, "error: invalid regular expression type, rest of line %s", - (char*) p); + errmsg.LogError(0, NO_ERRCODE, "template %s error: invalid regular expression type, rest of line %s", + pTpl->pszName, (char*) p); } } @@ -898,9 +965,7 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) #ifdef FEATURE_REGEXP if (pTpe->data.field.has_regex) { - dbgprintf("debug: has regex \n"); - /* APR 2005-09 I need the string that represent the regex */ /* The regex end is: "--end" */ /* TODO : this is hardcoded and cant be escaped, please change */ @@ -916,16 +981,13 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) if(regex_char == NULL) { dbgprintf("Could not allocate memory for template parameter!\n"); pTpe->data.field.has_regex = 0; - return 1; - /* TODO: RGer: check if we can recover better... (probably not) */ + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } /* Get the regex string for compiling later */ memcpy(regex_char, p, longitud); regex_char[longitud] = '\0'; - dbgprintf("debug: regex detected: '%s'\n", regex_char); - /* Now i compile the regex */ /* Remember that the re is an attribute of the Template entry */ if((iRetLocal = objUse(regexp, LM_REGEXP_FILENAME)) == RS_RET_OK) { @@ -1010,8 +1072,7 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) /* check field name */ if(*p == ':') { ++p; /* eat ':' */ - if(cstrConstruct(&pStrField) != RS_RET_OK) - return 1; + CHKiRet(cstrConstruct(&pStrField)); while(*p != ':' && *p != '%' && *p != '\0') { cstrAppendChar(pStrField, *p); ++p; @@ -1031,18 +1092,18 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) } } else { pTpe->fieldName = ustrdup(cstrGetSzStrNoNULL(pStrField)); - pTpe->lenFieldName = cstrLen(pStrProp); + pTpe->lenFieldName = ustrlen(pTpe->fieldName); cstrDestruct(&pStrField); } - if(pTpe->fieldName == NULL) - return 1; - + if(pTpe->fieldName == NULL) { + DBGPRINTF("template/do_Parameter: fieldName is NULL!\n"); + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } cstrDestruct(&pStrProp); - if(*p) ++p; /* eat '%' */ - *pp = p; - return 0; +finalize_it: + RETiRet; } @@ -1108,7 +1169,6 @@ struct template *tplAddLine(rsconf_t *conf, char* pName, uchar** ppRestOfConfLin assert(pName != NULL); assert(ppRestOfConfLine != NULL); - if((pTpl = tplConstruct(conf)) == NULL) return NULL; @@ -1171,10 +1231,13 @@ struct template *tplAddLine(rsconf_t *conf, char* pName, uchar** ppRestOfConfLin break; case '%': /* parameter */ ++p; /* eat '%' */ - do_Parameter(&p, pTpl); + if(do_Parameter(&p, pTpl) != RS_RET_OK) { + dbgprintf("tplAddLine error: parameter invalid"); + return NULL; + }; break; default: /* constant */ - do_Constant(&p, pTpl); + do_Constant(&p, pTpl, 1); break; } if(*p == '"') {/* end of template string? */ @@ -1235,7 +1298,7 @@ static rsRetVal createConstantTpe(struct template *pTpl, struct cnfobj *o) { struct templateEntry *pTpe; - es_str_t *value; + es_str_t *value = NULL; /* init just to keep compiler happy - mandatory parameter */ int i; struct cnfparamvals *pvals = NULL; uchar *outname = NULL; @@ -1292,6 +1355,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) int fielddelim = 9; /* default is HT (USACSII 9) */ int re_matchToUse = 0; int re_submatchToUse = 0; + int bComplexProcessing = 0; char *re_expr = NULL; struct cnfparamvals *pvals = NULL; enum {F_NONE, F_CSV, F_JSON, F_JSONF} formatType = F_NONE; @@ -1317,23 +1381,31 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) free(tmpstr); } else if(!strcmp(pblkProperty.descr[i].name, "droplastlf")) { droplastlf = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "mandatory")) { mandatory = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "spifno1stsp")) { spifno1stsp = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "outname")) { outname = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(pblkProperty.descr[i].name, "position.from")) { frompos = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "position.to")) { topos = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "field.number")) { fieldnum = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "field.delimiter")) { fielddelim = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "regex.expression")) { re_expr = es_str2cstr(pvals[i].val.d.estr, NULL); + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "regex.type")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"BRE", sizeof("BRE")-1)) { re_type = TPL_REGEX_BRE; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"ERE", sizeof("ERE")-1)) { @@ -1346,6 +1418,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "regex.nomatchmode")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"DFLT", sizeof("DFLT")-1)) { re_nomatchType = TPL_REGEX_NOMATCH_USE_DFLTSTR; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"BLANK", sizeof("BLANK")-1)) { @@ -1362,10 +1435,13 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "regex.match")) { + bComplexProcessing = 1; re_matchToUse = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "regex.submatch")) { + bComplexProcessing = 1; re_submatchToUse = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "format")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"csv", sizeof("csv")-1)) { formatType = F_CSV; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"json", sizeof("json")-1)) { @@ -1380,6 +1456,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "controlcharacters")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"escape", sizeof("escape")-1)) { controlchr = CC_ESCAPE; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"space", sizeof("space")-1)) { @@ -1394,6 +1471,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "securepath")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"drop", sizeof("drop")-1)) { secpath = SP_DROP; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"replace", sizeof("replace")-1)) { @@ -1406,6 +1484,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "caseconversion")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"lower", sizeof("lower")-1)) { caseconv = tplCaseConvLower; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"upper", sizeof("upper")-1)) { @@ -1475,8 +1554,8 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) CHKiRet(propNameToID(name, &pTpe->data.field.propid)); if(pTpe->data.field.propid == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(name)+2, - cstrLen(name)-2); + pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(name)+1, + cstrLen(name)-1); } pTpe->data.field.options.bDropLastLF = droplastlf; pTpe->data.field.options.bSPIffNo1stSP = spifno1stsp; @@ -1524,6 +1603,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) pTpe->fieldName = outname; if(outname != NULL) pTpe->lenFieldName = ustrlen(outname); + pTpe->bComplexProcessing = bComplexProcessing; pTpe->data.field.eDateFormat = datefmt; if(fieldnum != -1) { pTpe->data.field.has_fields = 1; @@ -1604,12 +1684,14 @@ tplProcessCnf(struct cnfobj *o) { struct template *pTpl = NULL; struct cnfparamvals *pvals = NULL; - int lenName; + int lenName = 0; /* init just to keep compiler happy: mandatory parameter */ char *name = NULL; uchar *tplStr = NULL; uchar *plugin = NULL; + es_str_t *subtree = NULL; uchar *p; - enum { T_STRING, T_PLUGIN, T_LIST } tplType; + enum { T_STRING, T_PLUGIN, T_LIST, T_SUBTREE } + tplType = T_STRING; /* init just to keep compiler happy: mandatory parameter */ int i; int o_sql=0, o_stdsql=0, o_json=0; /* options */ int numopts; @@ -1632,6 +1714,8 @@ tplProcessCnf(struct cnfobj *o) tplType = T_PLUGIN; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"list", sizeof("list")-1)) { tplType = T_LIST; + } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"subtree", sizeof("subtree")-1)) { + tplType = T_SUBTREE; } else { uchar *typeStr = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); errmsg.LogError(0, RS_RET_ERR, "invalid template type '%s'", @@ -1641,6 +1725,22 @@ tplProcessCnf(struct cnfobj *o) } } else if(!strcmp(pblk.descr[i].name, "string")) { tplStr = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(pblk.descr[i].name, "subtree")) { + uchar *st_str = es_getBufAddr(pvals[i].val.d.estr); + if(st_str[0] != '$' || st_str[1] != '!') { + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_ERR, "invalid subtree " + "parameter, variable must start with '$!' but " + "var name is '%s'", cstr); + free(cstr); + free(name); /* overall assigned */ + ABORT_FINALIZE(RS_RET_ERR); + } else { + /* TODO: unify strings! */ + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + subtree = es_newStrFromBuf(cstr+1, es_strlen(pvals[i].val.d.estr)-1); + free(cstr); + } } else if(!strcmp(pblk.descr[i].name, "plugin")) { plugin = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(pblk.descr[i].name, "option.stdsql")) { @@ -1682,6 +1782,19 @@ tplProcessCnf(struct cnfobj *o) } } + if(subtree == NULL) { + if(tplType == T_SUBTREE) { + errmsg.LogError(0, RS_RET_ERR, "template '%s' of type subtree needs " + "subtree parameter", name); + ABORT_FINALIZE(RS_RET_ERR); + } + } else { + if(tplType != T_SUBTREE) { + errmsg.LogError(0, RS_RET_ERR, "template '%s' is not a subtree " + "template but has a subtree specified - ignored", name); + } + } + if(o->subobjs == NULL) { if(tplType == T_LIST) { errmsg.LogError(0, RS_RET_ERR, "template '%s' of type list has " @@ -1719,10 +1832,10 @@ tplProcessCnf(struct cnfobj *o) switch(*p) { case '%': /* parameter */ ++p; /* eat '%' */ - do_Parameter(&p, pTpl); + CHKiRet(do_Parameter(&p, pTpl)); break; default: /* constant */ - do_Constant(&p, pTpl); + do_Constant(&p, pTpl, 0); break; } } @@ -1739,6 +1852,8 @@ tplProcessCnf(struct cnfobj *o) break; case T_LIST: createListTpl(pTpl, o); break; + case T_SUBTREE: pTpl->subtree = subtree; + break; } pTpl->optFormatEscape = NO_ESCAPE; @@ -1840,8 +1955,9 @@ void tplDeleteAll(rsconf_t *conf) } pTplDel = pTpl; pTpl = pTpl->pNext; - if(pTplDel->pszName != NULL) - free(pTplDel->pszName); + free(pTplDel->pszName); + if(pTplDel->subtree != NULL) + es_deleteStr(pTplDel->subtree); free(pTplDel); } ENDfunc @@ -1898,8 +2014,9 @@ void tplDeleteNew(rsconf_t *conf) } pTplDel = pTpl; pTpl = pTpl->pNext; - if(pTplDel->pszName != NULL) - free(pTplDel->pszName); + free(pTplDel->pszName); + if(pTplDel->subtree != NULL) + es_deleteStr(pTplDel->subtree); free(pTplDel); } ENDfunc @@ -1965,8 +2082,14 @@ void tplPrintList(rsconf_t *conf) case tplFmtUnixDate: dbgprintf("[Format as Unix timestamp] "); break; + case tplFmtSecFrac: + dbgprintf("[fractional seconds, only] "); + break; + case tplFmtRFC3164BuggyDate: + dbgprintf("[Format as buggy RFC3164-Date] "); + break; default: - dbgprintf("[INVALID eDateFormat %d] ", pTpe->data.field.eDateFormat); + dbgprintf("[UNKNOWN eDateFormat %d] ", pTpe->data.field.eDateFormat); } switch(pTpe->data.field.eCaseConv) { case tplCaseConvNo: @@ -2022,6 +2145,8 @@ void tplPrintList(rsconf_t *conf) } break; } + if(pTpe->bComplexProcessing) + dbgprintf("[COMPLEX]"); dbgprintf("\n"); pTpe = pTpe->pNext; } @@ -2035,8 +2160,6 @@ int tplGetEntryCount(struct template *pTpl) return(pTpl->tpenElements); } -/* our init function. TODO: remove once converted to a class - */ rsRetVal templateInit() { DEFiRet; @@ -39,7 +39,8 @@ struct template { struct template *pNext; char *pszName; int iLenName; - rsRetVal (*pStrgen)(msg_t*, uchar**, size_t *); /* name of strgen to use (bound if non-NULL!) */ + rsRetVal (*pStrgen)(msg_t*, uchar**, size_t *); + es_str_t *subtree; /* subtree name for subtree-type templates */ int tpenElements; /* number of elements in templateEntry list */ struct templateEntry *pEntryRoot; struct templateEntry *pEntryLast; @@ -71,6 +72,7 @@ struct templateEntry { enum EntryTypes eEntryType; uchar *fieldName; /**< field name to be used for structured output */ int lenFieldName; + sbool bComplexProcessing; /**< set if complex processing (options, etc) is required */ union { struct { uchar *pConstant; /* pointer to constant value */ @@ -141,15 +143,16 @@ void tplDeleteNew(rsconf_t *conf); void tplPrintList(rsconf_t *conf); void tplLastStaticInit(rsconf_t *conf, struct template *tpl); rsRetVal ExtendBuf(uchar **pBuf, size_t *pLenBuf, size_t iMinSize); +int tplRequiresDateCall(struct template *pTpl); /* note: if a compiler warning for undefined type tells you to look at this * code line below, the actual cause is that you currently MUST include template.h * BEFORE msg.h, even if your code file does not actually need it. * rgerhards, 2007-08-06 */ -rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr); -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz, size_t *); -rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **); -rsRetVal doEscape(uchar **pp, size_t *pLen, unsigned short *pbMustBeFreed, int escapeMode); +rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime *ttNow); +rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz, size_t *, struct syslogTime *ttNow); +rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **, struct syslogTime *ttNow); +rsRetVal doEscape(uchar **pp, rs_size_t *pLen, unsigned short *pbMustBeFreed, int escapeMode); rsRetVal templateInit(); rsRetVal tplProcessCnf(struct cnfobj *o); diff --git a/tests/Makefile.am b/tests/Makefile.am index e7122653..b339e797 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1,6 +1,6 @@ if ENABLE_TESTBENCH # TODO: reenable TESTRUNS = rt_init rscript -check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq msleep randomgen diagtalker uxsockrcvr syslog_caller syslog_inject inputfilegen +check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq msleep randomgen diagtalker uxsockrcvr syslog_caller syslog_inject inputfilegen minitcpsrv TESTS = $(TESTRUNS) #TESTS = $(TESTRUNS) cfg.sh @@ -65,9 +65,18 @@ TESTS += \ failover-no-basic.sh \ rcvr_fail_restore.sh \ rscript_contains.sh \ + rscript_field.sh \ + rscript_stop.sh \ + rscript_stop2.sh \ + rscript_prifilt.sh \ + rscript_optimizer1.sh \ + rscript_ruleset_call.sh \ + cee_simple.sh \ + cee_diskqueue.sh \ incltest.sh \ incltest_dir.sh \ incltest_dir_wildcard.sh \ + incltest_dir_empty_wildcard.sh \ linkedlistqueue.sh if HAVE_VALGRIND @@ -172,6 +181,10 @@ test_files = testbench.h runtime-dummy.c EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ validation-run.sh \ + tls-certs/ca-key.pem \ + tls-certs/ca.pem \ + tls-certs/cert.pem \ + tls-certs/key.pem \ testsuites/x.509/ca.pem \ testsuites/x.509/ca-key.pem \ testsuites/x.509/client-cert.pem \ @@ -270,10 +283,28 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/arrayqueue.conf \ rscript_contains.sh \ testsuites/rscript_contains.conf \ + rscript_field.sh \ + testsuites/rscript_field.conf \ + rscript_stop.sh \ + testsuites/rscript_stop.conf \ + rscript_stop2.sh \ + testsuites/rscript_stop2.conf \ + rscript_prifilt.sh \ + testsuites/rscript_prifilt.conf \ + rscript_optimizer1.sh \ + testsuites/rscript_optimizer1.conf \ + rscript_ruleset_call.sh \ + testsuites/rscript_ruleset_call.conf \ + cee_simple.sh \ + testsuites/cee_simple.conf \ + cee_diskqueue.sh \ + testsuites/cee_diskqueue.conf \ incltest.sh \ testsuites/incltest.conf \ incltest_dir.sh \ testsuites/incltest_dir.conf \ + incltest_dir_empty_wildcard.sh \ + testsuites/incltest_dir_empty_wildcard.conf \ incltest_dir_wildcard.sh \ testsuites/incltest_dir_wildcard.conf \ testsuites/incltest.d/include.conf \ @@ -505,6 +536,9 @@ if ENABLE_GNUTLS tcpflood_LDADD += -lgcrypt endif +minitcpsrv_SOURCES = minitcpsrvr.c +minitcpsrv_LDADD = $(SOL_LIBS) + syslog_caller_SOURCES = syslog_caller.c syslog_caller_LDADD = $(SOL_LIBS) diff --git a/tests/cee_diskqueue.sh b/tests/cee_diskqueue.sh new file mode 100755 index 00000000..4e19855b --- /dev/null +++ b/tests/cee_diskqueue.sh @@ -0,0 +1,14 @@ +# check if CEE properties are properly saved & restored to/from disk queue +# added 2012-09-19 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[cee_diskqueue.sh\]: CEE and diskqueue test +source $srcdir/diag.sh init +source $srcdir/diag.sh startup cee_diskqueue.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/cee_simple.sh b/tests/cee_simple.sh new file mode 100755 index 00000000..32f56393 --- /dev/null +++ b/tests/cee_simple.sh @@ -0,0 +1,13 @@ +# added 2012-09-19 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[cee_simple.sh\]: basic CEE property test +source $srcdir/diag.sh init +source $srcdir/diag.sh startup cee_simple.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/diag.sh b/tests/diag.sh index bd38b29d..2fdcbfb9 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -16,7 +16,6 @@ case $1 in 'init') $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason cp $srcdir/testsuites/diag-common.conf diag-common.conf cp $srcdir/testsuites/diag-common2.conf diag-common2.conf - rm -f rsyslog.action.*.include rm -f rsyslogd.started work-*.conf rsyslog.random.data rm -f rsyslogd2.started work-*.conf rm -f work rsyslog.out.log rsyslog2.out.log rsyslog.out.log.save # common work files @@ -24,6 +23,10 @@ case $1 in rm -f rsyslog.out.*.log work-presort rsyslog.pipe rm -f rsyslog.input rsyslog.empty rm -f core.* vgcore.* + # Note: rsyslog.action.*.include must NOT be deleted, as it + # is used to setup some parameters BEFORE calling init. This + # happens in chained test scripts. Delete on exit is fine, + # though. mkdir test-spool ;; 'exit') rm -f rsyslogd.started work-*.conf diag-common.conf diff --git a/tests/imptcp_addtlframedelim.sh b/tests/imptcp_addtlframedelim.sh index b26fc85b..00276ab3 100755 --- a/tests/imptcp_addtlframedelim.sh +++ b/tests/imptcp_addtlframedelim.sh @@ -3,7 +3,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_addtlframedelim.sh\]: test imptcp additional frame delimiter -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imptcp_addtlframedelim.conf source $srcdir/diag.sh tcpflood -m20000 -F0 -P129 diff --git a/tests/imptcp_conndrop.sh b/tests/imptcp_conndrop.sh index 684de6b5..0cf0ba5e 100755 --- a/tests/imptcp_conndrop.sh +++ b/tests/imptcp_conndrop.sh @@ -4,9 +4,8 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_conndrop.sh\]: test imptcp with random connection drops -cat rsyslog.action.1.include source $srcdir/diag.sh init -source $srcdir/diag.sh startup imptcp_large.conf +source $srcdir/diag.sh startup imptcp_conndrop.conf # 100 byte messages to gain more practical data use source $srcdir/diag.sh tcpflood -c20 -m50000 -r -d100 -P129 -D sleep 4 # due to large messages, we need this time for the tcp receiver to settle... diff --git a/tests/imptcp_large.sh b/tests/imptcp_large.sh index b4d130bb..43027069 100755 --- a/tests/imptcp_large.sh +++ b/tests/imptcp_large.sh @@ -4,7 +4,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_large.sh\]: test imptcp with large-size messages -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imptcp_large.conf # send 4000 messages of 10.000bytes plus header max, randomized diff --git a/tests/imtcp_addtlframedelim.sh b/tests/imtcp_addtlframedelim.sh index 8de7ca58..4c1fd9cb 100755 --- a/tests/imtcp_addtlframedelim.sh +++ b/tests/imtcp_addtlframedelim.sh @@ -3,7 +3,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imtcp_addtlframedelim.sh\]: test imtcp additional frame delimiter -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp_addtlframedelim.conf source $srcdir/diag.sh tcpflood -m20000 -F0 -P129 diff --git a/tests/incltest_dir_empty_wildcard.sh b/tests/incltest_dir_empty_wildcard.sh new file mode 100755 index 00000000..6cdb3b21 --- /dev/null +++ b/tests/incltest_dir_empty_wildcard.sh @@ -0,0 +1,13 @@ +# This test checks if an empty includeConfig directory causes problems. It +# should not, as this is a valid situation that by default exists on many +# distros. +echo =============================================================================== +echo \[incltest_dir_empty_wildcard.sh\]: test $IncludeConfig for \"empty\" wildcard +source $srcdir/diag.sh init +source $srcdir/diag.sh startup incltest_dir_empty_wildcard.conf +# 100 messages are enough - the question is if the include is read ;) +source $srcdir/diag.sh injectmsg 0 100 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 99 +source $srcdir/diag.sh exit diff --git a/tests/minitcpsrvr.c b/tests/minitcpsrvr.c new file mode 100644 index 00000000..8ac59f25 --- /dev/null +++ b/tests/minitcpsrvr.c @@ -0,0 +1,62 @@ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/socket.h> +#include <fcntl.h> +#include <unistd.h> +#include <arpa/inet.h> + +static void +errout(char *reason) +{ + perror(reason); + exit(1); +} + +int +main(int argc, char *argv[]) +{ + int fds; + int fdc; + int fdf; + struct sockaddr_in srvAddr; + struct sockaddr_in cliAddr; + unsigned int srvAddrLen; + unsigned int cliAddrLen; + char wrkBuf[4096]; + ssize_t nRead; + + if(argc != 4) { + fprintf(stderr, "usage: minitcpsrvr ip-addr port outfile\n"); + exit(1); + } + + if(!strcmp(argv[3], "-")) { + fdf = 1; + } else { + fdf = open(argv[3], O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR|S_IWUSR); + if(fdf == -1) errout(argv[3]); + } + + fds = socket(AF_INET, SOCK_STREAM, 0); + srvAddr.sin_family = AF_INET; + srvAddr.sin_addr.s_addr = inet_addr(argv[1]); + srvAddr.sin_port = htons(atoi(argv[2])); + srvAddrLen = sizeof(srvAddr); + if(bind(fds, (struct sockaddr *)&srvAddr, srvAddrLen) != 0) + errout("bind"); + if(listen(fds, 20) != 0) errout("listen"); + cliAddrLen = sizeof(cliAddr); + + fdc = accept(fds, (struct sockaddr *)&cliAddr, &cliAddrLen); + while(1) { + nRead = read(fdc, wrkBuf, sizeof(wrkBuf)); + if(nRead == 0) break; + if(write(fdf, wrkBuf, nRead) != nRead) + errout("write"); + } + /* let the OS do the cleanup */ + return 0; +} diff --git a/tests/nettester.c b/tests/nettester.c index 9e68ebcc..4e4fe55a 100644 --- a/tests/nettester.c +++ b/tests/nettester.c @@ -243,7 +243,7 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) { int pipefd[2]; pid_t cpid; - char *newargv[] = {"../tools/rsyslogd", "dummy", "-c4", "-u2", "-n", "-irsyslog.pid", + char *newargv[] = {"../tools/rsyslogd", "dummy", "-u2", "-n", "-irsyslog.pid", "-M../runtime/.libs:../.libs", NULL, NULL}; char confFile[1024]; diff --git a/tests/queue-persist-drvr.sh b/tests/queue-persist-drvr.sh index 53fbcb8b..de597308 100755 --- a/tests/queue-persist-drvr.sh +++ b/tests/queue-persist-drvr.sh @@ -24,6 +24,7 @@ source $srcdir/diag.sh check-mainq-spool echo "#" > work-delay.conf source $srcdir/diag.sh startup queue-persist.conf source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +./msleep 500 $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 4999 source $srcdir/diag.sh exit diff --git a/tests/rscript_field.sh b/tests/rscript_field.sh new file mode 100755 index 00000000..e989e666 --- /dev/null +++ b/tests/rscript_field.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_field.sh\]: testing rainerscript field\(\) function +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_field.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_optimizer1.sh b/tests/rscript_optimizer1.sh new file mode 100755 index 00000000..1d2dcf87 --- /dev/null +++ b/tests/rscript_optimizer1.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_optimizer1.sh\]: testing rainerscript optimizer +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_optimizer1.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_prifilt.sh b/tests/rscript_prifilt.sh new file mode 100755 index 00000000..815492ab --- /dev/null +++ b/tests/rscript_prifilt.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_prifilt.sh\]: testing rainerscript prifield\(\) function +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_prifilt.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_ruleset_call.sh b/tests/rscript_ruleset_call.sh new file mode 100755 index 00000000..e29f21da --- /dev/null +++ b/tests/rscript_ruleset_call.sh @@ -0,0 +1,13 @@ +# added 2012-10-29 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_ruleset_call.sh\]: testing rainerscript ruleset\(\) and call statement +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_ruleset_call.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_stop.sh b/tests/rscript_stop.sh new file mode 100755 index 00000000..e532a522 --- /dev/null +++ b/tests/rscript_stop.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_stop.sh\]: testing rainerscript STOP statement +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_stop.conf +source $srcdir/diag.sh injectmsg 0 8000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_stop2.sh b/tests/rscript_stop2.sh new file mode 100755 index 00000000..eae36cce --- /dev/null +++ b/tests/rscript_stop2.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_stop2.sh\]: testing rainerscript STOP statement, alternate method +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_stop2.conf +source $srcdir/diag.sh injectmsg 0 8000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/runtime-dummy.c b/tests/runtime-dummy.c index 5a9039bf..f6f2d07f 100644 --- a/tests/runtime-dummy.c +++ b/tests/runtime-dummy.c @@ -30,7 +30,6 @@ #include "rsyslog.h" int bReduceRepeatMsgs = 0; -int repeatinterval = 30; int bActExecWhenPrevSusp = 0; int iActExecOnceInterval = 1; int MarkInterval = 30; diff --git a/tests/tcp_forwarding_tpl.sh b/tests/tcp_forwarding_tpl.sh new file mode 100755 index 00000000..61114507 --- /dev/null +++ b/tests/tcp_forwarding_tpl.sh @@ -0,0 +1,30 @@ +# This test tests tcp forwarding with assigned template. To do so, a simple +# tcp listener service is started. +# added 2012-10-30 by Rgerhards. Released under GNU GPLv3+ +echo =============================================================================== +echo \[tcp_forwarding_tpl.sh\]: test for tcp forwarding with assigned template + +# create the pipe and start a background process that copies data from +# it to the "regular" work file +source $srcdir/diag.sh init +./minitcpsrvr 127.0.0.1 13514 rsyslog.out.log & +BGPROCESS=$! +echo background minitcpsrvr process id is $BGPROCESS + +# now do the usual run +source $srcdir/diag.sh startup tcp_forwarding_tpl.conf +# 10000 messages should be enough +source $srcdir/diag.sh injectmsg 0 10000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown + +# note: minitcpsrvr shuts down automatically if the connection is closed! +# (we still leave the code here in in case we need it later) +#echo shutting down minitcpsrv... +#kill $BGPROCESS +#wait $BGPROCESS +#echo background process has terminated, continue test... + +# and continue the usual checks +source $srcdir/diag.sh seq-check 0 9999 +source $srcdir/diag.sh exit diff --git a/tests/tcpflood.c b/tests/tcpflood.c index 4a3199c8..b3cef2e0 100644 --- a/tests/tcpflood.c +++ b/tests/tcpflood.c @@ -258,7 +258,7 @@ int openConnections(void) return setupUDP(); if(bShowProgress) - write(1, " open connections", sizeof(" open connections")-1); + if(write(1, " open connections", sizeof(" open connections")-1)){} # ifdef ENABLE_GNUTLS sessArray = calloc(numConnections, sizeof(gnutls_session_t)); # endif @@ -278,7 +278,7 @@ int openConnections(void) } if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d open connections\n", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)) {} } return 0; @@ -303,12 +303,12 @@ void closeConnections(void) return; if(bShowProgress) - write(1, " close connections", sizeof(" close connections")-1); + if(write(1, " close connections", sizeof(" close connections")-1)){} for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)){} } } if(sockArray[i] != -1) { @@ -325,7 +325,7 @@ void closeConnections(void) } if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d close connections\n", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)){} } } @@ -347,7 +347,7 @@ genMsg(char *buf, size_t maxBuf, int *pLenBuf, struct instdata *inst) /* get message from file */ do { done = 1; - *pLenBuf = fread(buf, 1, 1024, dataFP); + *pLenBuf = fread(buf, 1, MAX_EXTRADATA_LEN + 1024, dataFP); if(*pLenBuf == 0) { if(--numFileIterations > 0) { rewind(dataFP); diff --git a/tests/testsuites/cee_diskqueue.conf b/tests/testsuites/cee_diskqueue.conf new file mode 100644 index 00000000..a9b98e80 --- /dev/null +++ b/tests/testsuites/cee_diskqueue.conf @@ -0,0 +1,9 @@ +$IncludeConfig diag-common.conf + +global(workDirectory="/tmp") +template(name="outfmt" type="string" string="%$!usr!msg:F,58:2%\n") + +set $!usr!msg = $msg; +if $msg contains 'msgnum' then + action(type="omfile" file="./rsyslog.out.log" template="outfmt" + queue.type="disk" queue.filename="rsyslog-act1") diff --git a/tests/testsuites/cee_simple.conf b/tests/testsuites/cee_simple.conf new file mode 100644 index 00000000..1bcf83c1 --- /dev/null +++ b/tests/testsuites/cee_simple.conf @@ -0,0 +1,6 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="string" string="%$!usr!msg:F,58:2%\n") +set $!usr!msg = $msg; +if $msg contains 'msgnum' then + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/complex1.conf b/tests/testsuites/complex1.conf index 9e2441d4..9b6a9f35 100644 --- a/tests/testsuites/complex1.conf +++ b/tests/testsuites/complex1.conf @@ -3,6 +3,8 @@ $MaxMessageSize 10k $IncludeConfig diag-common.conf +$MainMsgQueueTimeoutEnqueue 5000 + $ModLoad ../plugins/imtcp/.libs/imtcp $MainMsgQueueTimeoutShutdown 10000 @@ -13,6 +15,7 @@ $template dynfile,"rsyslog.out.%inputname%.%msg:F,58:2%.log" $Ruleset R13514 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 @@ -36,6 +39,7 @@ $InputTCPServerRun 13514 $Ruleset R13515 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 @@ -60,6 +64,7 @@ $InputTCPServerRun 13515 $Ruleset R13516 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 diff --git a/tests/testsuites/imptcp_addtlframedelim.conf b/tests/testsuites/imptcp_addtlframedelim.conf index eb7ed0c4..bf302fb4 100644 --- a/tests/testsuites/imptcp_addtlframedelim.conf +++ b/tests/testsuites/imptcp_addtlframedelim.conf @@ -9,5 +9,4 @@ $template outfmt,"%msg:F,58:2%\n" $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/imptcp_conndrop.conf b/tests/testsuites/imptcp_conndrop.conf index 677e33f6..77a5d79a 100644 --- a/tests/testsuites/imptcp_conndrop.conf +++ b/tests/testsuites/imptcp_conndrop.conf @@ -12,5 +12,4 @@ $template dynfile,"rsyslog.out.log" # trick to use relative path names! $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ?dynfile;outfmt diff --git a/tests/testsuites/imptcp_large.conf b/tests/testsuites/imptcp_large.conf index 677e33f6..77a5d79a 100644 --- a/tests/testsuites/imptcp_large.conf +++ b/tests/testsuites/imptcp_large.conf @@ -12,5 +12,4 @@ $template dynfile,"rsyslog.out.log" # trick to use relative path names! $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ?dynfile;outfmt diff --git a/tests/testsuites/imtcp_addtlframedelim.conf b/tests/testsuites/imtcp_addtlframedelim.conf index 3b4759c5..6558c519 100644 --- a/tests/testsuites/imtcp_addtlframedelim.conf +++ b/tests/testsuites/imtcp_addtlframedelim.conf @@ -9,5 +9,4 @@ $template outfmt,"%msg:F,58:2%\n" $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/incltest_dir_empty_wildcard.conf b/tests/testsuites/incltest_dir_empty_wildcard.conf new file mode 100644 index 00000000..5e750c5b --- /dev/null +++ b/tests/testsuites/incltest_dir_empty_wildcard.conf @@ -0,0 +1,11 @@ +# see .sh file for description +# rgerhards, 2009-11-30 +$IncludeConfig diag-common.conf + +# the following include leads to no files actually being included +# but MUST NOT abort rsyslog's startup sequence. No files matching +# the wildcard is valid (as long as the path exists)! +$IncludeConfig testsuites/incltest.d/*.conf-not-there + +$template outfmt,"%msg:F,58:2%\n" +:msg, contains, "msgnum:" ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/rscript_field.conf b/tests/testsuites/rscript_field.conf new file mode 100644 index 00000000..d7eb9066 --- /dev/null +++ b/tests/testsuites/rscript_field.conf @@ -0,0 +1,11 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if $msg contains 'msgnum' then { + set $!usr!msgnum = field($msg, 58, 2); + action(type="omfile" file="./rsyslog.out.log" template="outfmt") +} diff --git a/tests/testsuites/rscript_optimizer1.conf b/tests/testsuites/rscript_optimizer1.conf new file mode 100644 index 00000000..7720af7a --- /dev/null +++ b/tests/testsuites/rscript_optimizer1.conf @@ -0,0 +1,12 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + +/* tcpflood uses local4.=debug */ +if prifilt("syslog.*") then + stop # it actually doesn't matter what we do here +else + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/rscript_prifilt.conf b/tests/testsuites/rscript_prifilt.conf new file mode 100644 index 00000000..8cb13a0f --- /dev/null +++ b/tests/testsuites/rscript_prifilt.conf @@ -0,0 +1,10 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + +/* tcpflood uses local4.=debug, we use a bit more generic filter */ +if prifilt("local4.*") then + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/rscript_ruleset_call.conf b/tests/testsuites/rscript_ruleset_call.conf new file mode 100644 index 00000000..96eab293 --- /dev/null +++ b/tests/testsuites/rscript_ruleset_call.conf @@ -0,0 +1,22 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + + +# we deliberately include continue/stop to make sure we have more than +# one statement. This catches grammar erorrs +ruleset(name="rs2") { + continue + action(type="omfile" file="./rsyslog.out.log" template="outfmt") + stop +} + +# this time we make sure a single statement is properly supported +ruleset(name="rs1") { + call rs2 +} + +if $msg contains 'msgnum' then call rs1 diff --git a/tests/testsuites/rscript_stop.conf b/tests/testsuites/rscript_stop.conf new file mode 100644 index 00000000..ab9569e5 --- /dev/null +++ b/tests/testsuites/rscript_stop.conf @@ -0,0 +1,13 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if $msg contains 'msgnum' then { + set $!usr!msgnum = field($msg, 58, 2); + if cnum($!usr!msgnum) >= 5000 then + stop + action(type="omfile" file="./rsyslog.out.log" template="outfmt") +} diff --git a/tests/testsuites/rscript_stop2.conf b/tests/testsuites/rscript_stop2.conf new file mode 100644 index 00000000..9ac9143e --- /dev/null +++ b/tests/testsuites/rscript_stop2.conf @@ -0,0 +1,18 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if not ($msg contains 'msgnum') then + stop + +set $!usr!msgnum = field($msg, 58, 2); +if cnum($!usr!msgnum) >= 5000 then + stop +/* We could use yet another method, but we like to have the action statement + * without a filter in rsyslog.conf top level hierarchy - so this test, as + * a side-effect, also tests this ability. + */ +action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/tcp_forwarding_tpl.conf b/tests/testsuites/tcp_forwarding_tpl.conf new file mode 100644 index 00000000..686c73ac --- /dev/null +++ b/tests/testsuites/tcp_forwarding_tpl.conf @@ -0,0 +1,7 @@ +$IncludeConfig diag-common.conf +$MainMsgQueueTimeoutShutdown 10000 +template(name="outfmt" type="string" string="%msg:F,58:2%\n") + +if $msg contains "msgnum:" then + action(type="omfwd" template="outfmt" + target="127.0.0.1" port="13514" protocol="tcp") diff --git a/tests/threadingmq.sh b/tests/threadingmq.sh index ea5d7837..98f195d4 100755 --- a/tests/threadingmq.sh +++ b/tests/threadingmq.sh @@ -11,6 +11,9 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup threadingmq.conf source $srcdir/diag.sh injectmsg 0 100000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +# we give an extra seconds for things to settle, especially +# important on slower test machines +./msleep 1000 source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 99999 source $srcdir/diag.sh exit diff --git a/tests/tls-certs/ca-key.pem b/tests/tls-certs/ca-key.pem new file mode 100644 index 00000000..181a8ad9 --- /dev/null +++ b/tests/tls-certs/ca-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXQIBAAKBgQDDaz5X5YIruPH0wukMPik7xIKqrpCcr8Gm28oz5h4GtX253eWr +piBuk2a/f/CKDjeuqmiWqTs90PFNb+Z1c+Yzvagqv80VzZwDI4RcrwlNaKrBz/9X +iowCcoV8s7GvV2vtZEPSThNzz4FYkxCMvbOYZeJIYQVhZggUcuadfhmDIwIDAQAB +AoGAIG5AUD2jmYDzD+UhiultVgtkifyNaEtsuQsZu/zbt85P2VQ0z4SINlbvrXvc +iJ9tEzzEPa3udHGj/MTDe3OAB4TK5tImX1pe2gw+zaOB/DaH5i4QhXeltU7epCHF +oUv9EVNzL8Bl00MFiWcLY0LisQVfHeW5rcN9U7EbvTlWbRkCQQDR2/Qn1ceavwDU +qYt2TbEicJVC8aQMYYyc6Xvi4mZaNa8gGCpWpurgQop0Ln0QE8vA0601UVs6N3tm +g8FJ8rXpAkEA7mKCtp2MXCbHMdkZCyQt6drUYCyU9N/HtmBEtFGpqt1PdMyUI07m +rlVFDwUH9JFmg18RP1X2ufj7+ZbJzaMtKwJBAJgbw1Z0P19Mfj+mPC2dlnyN+cIx +/2Px+Mdq/J6w1tsf+jVbDqUMC0ZNNKmNYJycnJzBUNRKicMin9DoQttkjrECQQCC +s/aRY+6adBSRi0QE7NBTwUzicm81mCDrKPtilsfdTDyNgMHUXiVy/oO/yXVkLfi0 +HQLa5CpEK3UUkw2Qt2BDAkA0XXvQzW0+tEHiktLNljIluhiyOAx2bBywY/9Qmn6C +hv4sOSCzTR39jNmuNZ0X6ZZvt4VsWTHhpche/ud1+3p6 +-----END RSA PRIVATE KEY----- diff --git a/tests/tls-certs/ca.pem b/tests/tls-certs/ca.pem new file mode 100644 index 00000000..6324c7d5 --- /dev/null +++ b/tests/tls-certs/ca.pem @@ -0,0 +1,15 @@ +-----BEGIN CERTIFICATE----- +MIICYjCCAc2gAwIBAgIBATALBgkqhkiG9w0BAQUwWDELMAkGA1UEBhMCREUxHTAb +BgNVBAoTFHJzeXNsb2cgdGVzdCByb290IENBMQswCQYDVQQLEwJDQTEdMBsGA1UE +AxMUcnN5c2xvZy10ZXN0LXJvb3QtY2EwHhcNMDgwNTIwMTI1ODEyWhcNMTgwNTE4 +MTI1ODI0WjBYMQswCQYDVQQGEwJERTEdMBsGA1UEChMUcnN5c2xvZyB0ZXN0IHJv +b3QgQ0ExCzAJBgNVBAsTAkNBMR0wGwYDVQQDExRyc3lzbG9nLXRlc3Qtcm9vdC1j +YTCBnDALBgkqhkiG9w0BAQEDgYwAMIGIAoGAw2s+V+WCK7jx9MLpDD4pO8SCqq6Q +nK/BptvKM+YeBrV9ud3lq6YgbpNmv3/wig43rqpolqk7PdDxTW/mdXPmM72oKr/N +Fc2cAyOEXK8JTWiqwc//V4qMAnKFfLOxr1dr7WRD0k4Tc8+BWJMQjL2zmGXiSGEF +YWYIFHLmnX4ZgyMCAwEAAaNDMEEwDwYDVR0TAQH/BAUwAwEB/zAPBgNVHQ8BAf8E +BQMDBwYAMB0GA1UdDgQWBBQzYQQgUm0YLNdarJnc2c1LxYVClDALBgkqhkiG9w0B +AQUDgYEAuGWtH7Jkpa0n/izqQ5ddDQP/LT6taivCwlpEYEU9aumpQPWWxtYywKaP +RfM1JTMLAiYd8MS7TJ8TYRvvR32Y02Y+OhXn11xERkWvBT2M9yzqX6hDfRueN7RT +fPWsfm/NBTVojzjaECcTFenZid7PC5JiFbcU6PSUMZ49/JPhxAo= +-----END CERTIFICATE----- diff --git a/tests/tls-certs/cert.pem b/tests/tls-certs/cert.pem new file mode 100644 index 00000000..6b5b13cd --- /dev/null +++ b/tests/tls-certs/cert.pem @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIIChjCCAfGgAwIBAgIBADALBgkqhkiG9w0BAQUwWDELMAkGA1UEBhMCREUxHTAb +BgNVBAoTFHJzeXNsb2cgdGVzdCByb290IENBMQswCQYDVQQLEwJDQTEdMBsGA1UE +AxMUcnN5c2xvZy10ZXN0LXJvb3QtY2EwHhcNMDgwNTIwMTMwNDE5WhcNMTgwNTE4 +MTMwNDI2WjA6MQswCQYDVQQGEwJERTEQMA4GA1UEChMHcnN5c2xvZzEZMBcGA1UE +CxMQdGVzdCBjZXJ0aWZpY2F0ZTCBnDALBgkqhkiG9w0BAQEDgYwAMIGIAoGAxmHe +fztJgaGxFYEceiUg0hdMlRVWBqoZelJ8BeXTDnXcu/5F2HtM+l+QDyDaGjKlx+NI +K4rkj7d6Wd3AKPgOYS0VSDZe3a1xf9rRYzOthWTv7tYi4/LTqPXqN5lKE71dgrB/ +/gOmvV/1YD776FIxVGCSAT0hHwkFC3slmpJSwD8CAwEAAaOBhDCBgTAMBgNVHRMB +Af8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHREECzAJ +ggdyc3lzbG9nMB0GA1UdDgQWBBQYu6eC9UALvC+5K5VOnFRi5OC98TAfBgNVHSME +GDAWgBQzYQQgUm0YLNdarJnc2c1LxYVClDALBgkqhkiG9w0BAQUDgYEAXaymqsG9 +PNBhhWIRFvXCDMaDM71vUtgSFoNUbxIV607ua2HQosPPM4EHIda6N6hdBK1bMQoG +yqBwhvw0JVaVaO70Kbs2m2Ypk3YcpJtRqyp8q8+2y/w1Mk1QazFZC29aYgX2iNVf +X4/x38YEL7Gu5vqPrTn++agnV4ZXECKuvLQ= +-----END CERTIFICATE----- diff --git a/tests/tls-certs/key.pem b/tests/tls-certs/key.pem new file mode 100644 index 00000000..3ff507f0 --- /dev/null +++ b/tests/tls-certs/key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICWwIBAAKBgQDGYd5/O0mBobEVgRx6JSDSF0yVFVYGqhl6UnwF5dMOddy7/kXY +e0z6X5APINoaMqXH40griuSPt3pZ3cAo+A5hLRVINl7drXF/2tFjM62FZO/u1iLj +8tOo9eo3mUoTvV2CsH/+A6a9X/VgPvvoUjFUYJIBPSEfCQULeyWaklLAPwIDAQAB +AoGARIwKqmHc+0rYenq7UUVE+vMMBjNyHyllVkvsCMmpzMRS+i5ZCf1I0vZ0O5X5 +ZrX7bH8PL+R1J2eZgjXKMR3NMZBuyKHewItD9t2rIC0eD/ITlwq3VybbaMsw666e +INxSmax+dS5CEcLevHHP3c+Q7S7QAFiWV43TdFUGXWJktIkCQQDPQ5WAZ+/Tvv0Q +vtRjXMeTVaw/bSuKNUeDzFkmGyePnFeCReNFtJLE9PFSQWcPuYcbZgU59JTfA5ac +Un+cHm31AkEA9Qek+q7PcJ+kON9E6SNodCZn6gLyHjnWrq4tf8pZO3NvoX2QiuD4 +rwF7KWjr6q1JzADpLtwXnuYEhyiLFjJA4wJAcElMCEnG2y+ASH8p7z7HfKGQdLg/ +O1wMB3JA5e0WLK5lllUogI4IaZ3N02NNY25+rLBDqpc/w+ZcxQnIypqNtQJATs9p +ofON5wSB1oUBbhckZo9fxuWxqEUkJsUA/2Q+9R843XE8h166vdc1HOmRT8bywHne +hmLl+gazmCFTMw1wzwJAHng+3zGUl4D8Ov3MPFD6hwYYK6/pEdtz/NUsCSazF7eK +XuuP+DXPHNhXOuF1A3tP74pfc/fC1uCUH2G5z3Fy0Q== +-----END RSA PRIVATE KEY----- @@ -183,9 +183,11 @@ static void* thrdStarter(void *arg) assert(pThis != NULL); assert(pThis->pUsrThrdMain != NULL); + ustrncpy(thrdName+3, pThis->name, 20); + dbgOutputTID((char*)thrdName); + # if HAVE_PRCTL && defined PR_SET_NAME /* set thread name - we ignore if the call fails, has no harsh consequences... */ - ustrncpy(thrdName+3, pThis->name, 20); if(prctl(PR_SET_NAME, thrdName, 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", pThis->name); } else { diff --git a/tools/Makefile.am b/tools/Makefile.am index e634076c..9d9bd352 100644 --- a/tools/Makefile.am +++ b/tools/Makefile.am @@ -36,7 +36,7 @@ rsyslogd_SOURCES = \ pidfile.h \ \ ../dirty.h -rsyslogd_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) $(CNF_LIBS) +rsyslogd_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) # note: it looks like librsyslog.la must be explicitely given on LDDADD, # otherwise dependencies are not properly calculated (resulting in a # potentially incomplete build, a problem we had several times...) diff --git a/tools/logctl.c b/tools/logctl.c index df332bc2..1ab8ead0 100644 --- a/tools/logctl.c +++ b/tools/logctl.c @@ -143,7 +143,6 @@ struct ofields* get_data(struct results *res) struct ofields *fields; const char *msg; const char *prog; - const char *level; const char *syslog_tag; gint64 date_r; bson_cursor *c; @@ -263,7 +262,7 @@ struct select_doc* create_select() struct query_doc* create_query(struct queryopt *opt) { struct query_doc *qu_doc; - bson *query_what, *order_what, *order_how, *msg_what, *date_what; + bson *query_what, *order_what, *msg_what, *date_what; struct tm tm; time_t t; gint64 ts; @@ -417,7 +416,6 @@ int main (int argc, char *argv[]) struct queryopt opt; struct ofields *fields; - struct bson_doc *doc; struct select_doc *s_doc; struct query_doc *qu_doc; struct db_connect *db_conn; diff --git a/tools/omdiscard.c b/tools/omdiscard.c index 182c4b63..15c6ea82 100644 --- a/tools/omdiscard.c +++ b/tools/omdiscard.c @@ -35,6 +35,7 @@ #include "syslogd-types.h" #include "omdiscard.h" #include "module-template.h" +#include "errmsg.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -42,6 +43,7 @@ MODULE_TYPE_NOKEEP /* internal structures */ DEF_OMOD_STATIC_DATA +DEFobjCurrIf(errmsg); typedef struct _instanceData { EMPTY_STRUCT @@ -92,8 +94,10 @@ CODE_STD_STRING_REQUESTparseSelectorAct(0) p = *pp; if(*p == '~') { - /* TODO: check the rest of the selector line - error reporting */ dbgprintf("discard\n"); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: ~ action " + "is deprecated, consider using the 'stop' " + "statement instead"); } else { iRet = RS_RET_CONFLINE_UNPROCESSED; } @@ -103,6 +107,7 @@ ENDparseSelectorAct BEGINmodExit CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); ENDmodExit @@ -116,6 +121,7 @@ BEGINmodInit(Discard) CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); ENDmodInit /* * vi:set ai: diff --git a/tools/omfile.c b/tools/omfile.c index 715b218c..1c65fc59 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -68,6 +68,7 @@ #include "stream.h" #include "unicode-helper.h" #include "atomic.h" +#include "statsobj.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -81,6 +82,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(strm) +DEFobjCurrIf(statsobj) /* for our current LRU mechanism, we need a monotonically increasing counters. We use * it much like a "Lamport logical clock": we do not need the actual time, we just need @@ -156,6 +158,13 @@ typedef struct _instanceData { int iFlushInterval; /* how fast flush buffer on inactivity? */ sbool bFlushOnTXEnd; /* flush write buffers when transaction has ended? */ sbool bUseAsyncWriter; /* use async stream writer? */ + sbool bVeryRobustZip; + statsobj_t *stats; /* dynafile, primarily cache stats */ + STATSCOUNTER_DEF(ctrRequests, mutCtrRequests); + STATSCOUNTER_DEF(ctrLevel0, mutCtrLevel0); + STATSCOUNTER_DEF(ctrEvict, mutCtrEvict); + STATSCOUNTER_DEF(ctrMiss, mutCtrMiss); + STATSCOUNTER_DEF(ctrMax, mutCtrMax); } instanceData; @@ -205,6 +214,7 @@ static struct cnfparamdescr actpdescr[] = { { "ziplevel", eCmdHdlrInt, 0 }, /* legacy: omfileziplevel */ { "flushinterval", eCmdHdlrInt, 0 }, /* legacy: omfileflushinterval */ { "asyncwriting", eCmdHdlrBinary, 0 }, /* legacy: omfileasyncwriting */ + { "veryrobustzip", eCmdHdlrBinary, 0 }, { "flushontxend", eCmdHdlrBinary, 0 }, /* legacy: omfileflushontxend */ { "iobuffersize", eCmdHdlrSize, 0 }, /* legacy: omfileiobuffersize */ { "dirowner", eCmdHdlrUID, 0 }, /* legacy: dirowner */ @@ -269,7 +279,8 @@ CODESTARTdbgPrintInstInfo dbgprintf("\tflush on TX end=%d\n", pData->bFlushOnTXEnd); dbgprintf("\tflush interval=%d\n", pData->iFlushInterval); dbgprintf("\tfile cache size=%d\n", pData->iDynaFileCacheSize); - dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "yes" : "no"); + dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "on" : "off"); + dbgprintf("\tvery robust zip: %s\n", pData->bCreateDirs ? "on" : "off"); dbgprintf("\tfile owner %d, group %d\n", (int) pData->fileUID, (int) pData->fileGID); dbgprintf("\tdirectory owner %d, group %d\n", (int) pData->dirUID, (int) pData->dirGID); dbgprintf("\tdir create mode 0%3.3o, file create mode 0%3.3o\n", @@ -292,7 +303,7 @@ setLegacyDfltTpl(void __attribute__((unused)) *pVal, uchar* newVal) if(loadModConf != NULL && loadModConf->tplName != NULL) { free(newVal); - errmsg.LogError(0, RS_RET_ERR, "omfile default template already set via module " + errmsg.LogError(0, RS_RET_ERR, "omfile: default template already set via module " "global parameter - can no longer be changed"); ABORT_FINALIZE(RS_RET_ERR); } @@ -536,6 +547,7 @@ prepareFile(instanceData *pData, uchar *newFileName) CHKiRet(strm.SetFName(pData->pStrm, szBaseName, ustrlen(szBaseName))); CHKiRet(strm.SetDir(pData->pStrm, szDirName, ustrlen(szDirName))); CHKiRet(strm.SetiZipLevel(pData->pStrm, pData->iZipLevel)); + CHKiRet(strm.SetbVeryReliableZip(pData->pStrm, pData->bVeryRobustZip)); CHKiRet(strm.SetsIOBufSize(pData->pStrm, (size_t) pData->iIOBufSize)); CHKiRet(strm.SettOperationsMode(pData->pStrm, STREAMMODE_WRITE_APPEND)); CHKiRet(strm.SettOpenMode(pData->pStrm, cs.fCreateMode)); @@ -593,7 +605,8 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) && !ustrcmp(newFileName, pCache[pData->iCurrElt]->pName)) { /* great, we are all set */ pCache[pData->iCurrElt]->clkTickAccessed = getClockFileAccess(); - // LRU needs only a strictly monotonically increasing counter, so such a one could do + STATSCOUNTER_INC(pData->ctrLevel0, pData->mutCtrLevel0); + /* LRU needs only a strictly monotonically increasing counter, so such a one could do */ FINALIZE; } @@ -625,6 +638,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) } /* we have not found an entry */ + STATSCOUNTER_INC(pData->ctrMiss, pData->mutCtrMiss); /* invalidate iCurrElt as we may error-exit out of this function when the currrent * iCurrElt has been freed or otherwise become unusable. This is a precaution, and @@ -642,6 +656,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) if(iFirstFree == -1 && (pData->iCurrCacheSize < pData->iDynaFileCacheSize)) { /* there is space left, so set it to that index */ iFirstFree = pData->iCurrCacheSize++; + STATSCOUNTER_SETMAX_NOMUT(pData->ctrMax, (unsigned) pData->iCurrCacheSize); } /* Note that the following code sequence does not work with the cache entry itself, @@ -650,6 +665,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) */ if(iFirstFree == -1) { dynaFileDelCacheEntry(pCache, iOldest, 0); + STATSCOUNTER_INC(pData->ctrEvict, pData->mutCtrEvict); iFirstFree = iOldest; /* this one *is* now free ;) */ } else { /* we need to allocate memory for the cache structure */ @@ -843,7 +859,12 @@ BEGINendTransaction CODESTARTendTransaction /* Note: pStrm may be NULL if there was an error opening the stream */ if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { - CHKiRet(strm.Flush(pData->pStrm)); + /* if we have an async writer, it controls the flush via + * a timeout. However, without it, we actually need to flush, + * else incomplete records are written. + */ + if(!pData->bUseAsyncWriter) + CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: ENDendTransaction @@ -851,7 +872,10 @@ ENDendTransaction BEGINdoAction CODESTARTdoAction - DBGPRINTF("file to log to: %s\n", pData->f_fname); + DBGPRINTF("file to log to: %s\n", + (pData->bDynamicName) ? ppString[1] : pData->f_fname); + DBGPRINTF("omfile: start of data: '%.128s'\n", ppString[0]); + STATSCOUNTER_INC(pData->ctrRequests, pData->mutCtrRequests); CHKiRet(writeFile(ppString, iMsgOpts, pData)); if(!bCoreSupportsBatching && pData->bFlushOnTXEnd) { CHKiRet(strm.Flush(pData->pStrm)); @@ -878,14 +902,53 @@ setInstParamDefaults(instanceData *pData) pData->bCreateDirs = 1; pData->bSyncFile = 0; pData->iZipLevel = 0; + pData->bVeryRobustZip = 0; pData->bFlushOnTXEnd = FLUSHONTX_DFLT; pData->iIOBufSize = IOBUF_DFLT_SIZE; pData->iFlushInterval = FLUSH_INTRVL_DFLT; pData->bUseAsyncWriter = USE_ASYNCWRITER_DFLT; } + +static rsRetVal +setupInstStatsCtrs(instanceData *pData) +{ + uchar ctrName[512]; + DEFiRet; + + if(!pData->bDynamicName) { + FINALIZE; + } + + /* support statistics gathering */ + snprintf((char*)ctrName, sizeof(ctrName), "dynafile cache %s", pData->f_fname); + ctrName[sizeof(ctrName)-1] = '\0'; /* be on the save side */ + CHKiRet(statsobj.Construct(&(pData->stats))); + CHKiRet(statsobj.SetName(pData->stats, ctrName)); + STATSCOUNTER_INIT(pData->ctrRequests, pData->mutCtrRequests); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("requests"), + ctrType_IntCtr, &(pData->ctrRequests))); + STATSCOUNTER_INIT(pData->ctrLevel0, pData->mutCtrLevel0); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("level0"), + ctrType_IntCtr, &(pData->ctrLevel0))); + STATSCOUNTER_INIT(pData->ctrMiss, pData->mutCtrMiss); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("missed"), + ctrType_IntCtr, &(pData->ctrMiss))); + STATSCOUNTER_INIT(pData->ctrEvict, pData->mutCtrEvict); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("evicted"), + ctrType_IntCtr, &(pData->ctrEvict))); + STATSCOUNTER_INIT(pData->ctrMax, pData->mutCtrMax); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("maxused"), + ctrType_IntCtr, &(pData->ctrMax))); + CHKiRet(statsobj.ConstructFinalize(pData->stats)); + +finalize_it: + RETiRet; +} + BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; CODESTARTnewActInst DBGPRINTF("newActInst (omfile)\n"); @@ -914,6 +977,8 @@ CODESTARTnewActInst pData->iZipLevel = (int) pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "flushinterval")) { pData->iFlushInterval = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "veryrobustzip")) { + pData->bVeryRobustZip = pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "asyncwriting")) { pData->bUseAsyncWriter = pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "flushontxend")) { @@ -960,7 +1025,8 @@ CODESTARTnewActInst ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); } - CHKiRet(OMSRsetEntry(*ppOMSR, 0, ustrdup(getDfltTpl()), OMSR_NO_RQD_TPL_OPTS)); + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); if(pData->bDynamicName) { /* "filename" is actually a template name, we need this as string 1. So let's add it @@ -974,6 +1040,7 @@ CODESTARTnewActInst pData->iCurrElt = -1; /* no current element */ } // TODO: add pData->iSizeLimit = 0; /* default value, use outchannels to configure! */ + setupInstStatsCtrs(pData); CODE_STD_FINALIZERnewActInst cnfparamvalsDestruct(pvals, &actpblk); @@ -1062,6 +1129,8 @@ CODESTARTparseSelectorAct pData->iIOBufSize = (int) cs.iIOBufSize; pData->iFlushInterval = cs.iFlushInterval; pData->bUseAsyncWriter = cs.bUseAsyncWriter; + pData->bVeryRobustZip = 0; /* cannot be specified via legacy conf */ + setupInstStatsCtrs(pData); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -1088,7 +1157,6 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a cs.bUseAsyncWriter = USE_ASYNCWRITER_DFLT; free(pszFileDfltTplName); pszFileDfltTplName = NULL; - return RS_RET_OK; } @@ -1110,6 +1178,7 @@ BEGINmodExit CODESTARTmodExit objRelease(errmsg, CORE_COMPONENT); objRelease(strm, CORE_COMPONENT); + objRelease(statsobj, CORE_COMPONENT); DESTROY_ATOMIC_HELPER_MUT(mutClock); ENDmodExit @@ -1132,6 +1201,7 @@ CODEmodInit_QueryRegCFSLineHdlr INITLegCnfVars CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(strm, CORE_COMPONENT)); + CHKiRet(objUse(statsobj, CORE_COMPONENT)); INIT_ATOMIC_HELPER_MUT(mutClock); diff --git a/tools/omfwd.c b/tools/omfwd.c index 2fd24bdf..129392d2 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -138,6 +138,7 @@ static struct cnfparamdescr actpdescr[] = { { "streamdriverauthmode", eCmdHdlrGetWord, 0 }, { "streamdriverpermittedpeers", eCmdHdlrGetWord, 0 }, { "resendlastmsgonreconnect", eCmdHdlrBinary, 0 }, + { "template", eCmdHdlrGetWord, 0 }, }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -760,6 +761,7 @@ setInstParamDefaults(instanceData *pData) BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; rsRetVal localRet; CODESTARTnewActInst @@ -881,7 +883,8 @@ CODESTARTnewActInst } CODE_STD_STRING_REQUESTnewActInst(1) - CHKiRet(OMSRsetEntry(*ppOMSR, 0, ustrdup(getDfltTpl()), OMSR_NO_RQD_TPL_OPTS)); + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); CHKiRet(initTCP(pData)); CODE_STD_FINALIZERnewActInst diff --git a/tools/pidfile.c b/tools/pidfile.c index e9601232..8298b94e 100644 --- a/tools/pidfile.c +++ b/tools/pidfile.c @@ -55,7 +55,8 @@ int read_pid (char *pidfile) if (!(f=fopen(pidfile,"r"))) return 0; - fscanf(f,"%d", &pid); + if(fscanf(f,"%d", &pid) != 1) + pid = 0; fclose(f); return pid; } @@ -113,7 +114,8 @@ int write_pid (char *pidfile) #if HAVE_FLOCK if (flock(fd, LOCK_EX|LOCK_NB) == -1) { - fscanf(f, "%d", &pid); + if(fscanf(f, "%d", &pid) != 1) + pid = 0; fclose(f); printf("Can't lock, lock is held by pid %d.\n", pid); return 0; diff --git a/tools/recover_qi.pl b/tools/recover_qi.pl index 4e2cf9d5..4e2cf9d5 100644..100755 --- a/tools/recover_qi.pl +++ b/tools/recover_qi.pl diff --git a/tools/rsyslog.conf.5 b/tools/rsyslog.conf.5 index dcc9b7c7..fe9e083b 100644 --- a/tools/rsyslog.conf.5 +++ b/tools/rsyslog.conf.5 @@ -17,7 +17,7 @@ .\" along with this program; if not, write to the Free Software .\" Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. .\" -.TH RSYSLOG.CONF 5 "11 July 2008" "Version 3.18.0" "Linux System Administration" +.TH RSYSLOG.CONF 5 "22 October 2012" "Version 7.2.0" "Linux System Administration" .SH NAME rsyslog.conf \- rsyslogd(8) configuration file .SH DESCRIPTION @@ -335,13 +335,6 @@ Rsyslog offers three different types "filter conditions": * expression-based filters .RE -.SS Blocks -Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block of lines is separated from -the previous block by a program or hostname specification. A block will only log messages -corresponding to the most recent program and hostname specifications given. Thus, a block which -selects "ppp" as the program, directly followed by a block that selects messages from the -hostname "dialhost", then the second block will only log messages from the ppp program on dialhost. - .SS Selectors .B Selectors are the traditional way of filtering syslog messages. They have been kept in rsyslog with their original syntax, because it is well-known, highly @@ -422,7 +415,7 @@ To escape: .sp 0 \\ = \\\\ --> '\\' is used to escape (as in C) .sp 0 -$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n" +$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n" Properties can be accessed by the property replacer (see there for details). @@ -494,7 +487,7 @@ A template that resembles traditional syslogd file output: .RS $template TraditionalFormat,"%timegenerated% %HOSTNAME% .sp 0 -%syslogtag%%msg:::drop-last-lf%\n" +%syslogtag%%msg:::drop-last-lf%\\n" .RE A template that tells you a little more about the message: @@ -502,7 +495,7 @@ A template that tells you a little more about the message: .RS $template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%, .sp 0 -%syslogtag%,%msg%\n" +%syslogtag%,%msg%\\n" .RE A template for RFC 3164 format: @@ -514,7 +507,7 @@ $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" A template for the format traditionally used for user messages: .sp .RS -$template usermsg," XXXX%syslogtag%%msg%\n\r" +$template usermsg," XXXX%syslogtag%%msg%\\n\\r" .RE And a template with the traditional wall-message format: diff --git a/tools/rsyslogd.8 b/tools/rsyslogd.8 index 36f29769..620006f2 100644 --- a/tools/rsyslogd.8 +++ b/tools/rsyslogd.8 @@ -10,6 +10,7 @@ rsyslogd \- reliable and extended syslogd .RB [ " \-6 " ] .RB [ " \-A " ] .RB [ " \-d " ] +.RB [ " \-D " ] .RB [ " \-f " .I config file ] @@ -120,10 +121,15 @@ If neither -4 nor -6 is given, listens to all configured addresses of the system. .TP .BI "\-c " "version" -This option has been obsolted and has no function any longer. It is still +This option has been obsoleted and has no function any longer. It is still accepted in order not to break existing scripts. However, future versions may not support it. .TP +.B "\-D" +Runs the Bison config parser in debug mode. This may help when hard to find +syntax errors are reported. Please note that the output generated is deeply +technical and orignally targeted towards developers. +.TP .B "\-d" Turns on debug mode. Using this the daemon will not proceed a .BR fork (2) diff --git a/tools/syslogd.c b/tools/syslogd.c index a89c7e57..c2928a8f 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -44,7 +44,6 @@ #include "rsyslog.h" #define DEFUPRI (LOG_USER|LOG_NOTICE) -#define TIMERINTVL 30 /* interval for checking flush, mark */ #include <unistd.h> #include <stdlib.h> @@ -93,6 +92,8 @@ #include <zlib.h> #endif +extern int yydebug; /* interface to flex */ + #include <netdb.h> #include "pidfile.h" @@ -119,13 +120,13 @@ #include "batch.h" #include "unicode-helper.h" #include "ruleset.h" -#include "rule.h" #include "net.h" #include "prop.h" #include "rsconf.h" #include "dnscache.h" #include "sd-daemon.h" #include "rainerscript.h" +#include "ratelimit.h" /* definitions for objects we access */ DEFobjCurrIf(obj) @@ -134,7 +135,6 @@ DEFobjCurrIf(datetime) /* TODO: make go away! */ DEFobjCurrIf(conf) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) DEFobjCurrIf(prop) DEFobjCurrIf(parser) @@ -195,7 +195,6 @@ static prop_t *pInternalInputName = NULL; /* there is only one global inputName static uchar *ConfFile = (uchar*) _PATH_LOGCONF; /* read-only after startup */ static char *PidFile = _PATH_LOGPID; /* read-only after startup */ -static pid_t myPid; /* our pid for use in self-generated messages, e.g. on startup */ /* mypid is read-only after the initial fork() */ static int bHadHUP = 0; /* did we have a HUP? */ @@ -205,13 +204,6 @@ static int bFinished = 0; /* used by termination signal handler, read-only excep */ int iConfigVerify = 0; /* is this just a config verify run? */ -/* Intervals at which we flush out "message repeated" messages, - * in seconds after previous message is logged. After each flush, - * we move to the next interval until we reach the largest. - * TODO: this shall go into action object! -- rgerhards, 2008-01-29 - */ -int repeatinterval[2] = { 30, 60 }; /* # of secs before flush */ - #define LIST_DELIMITER ':' /* delimiter between two hosts */ static pid_t ppid; /* This is a quick and dirty hack used for spliting main/startup thread */ @@ -222,6 +214,8 @@ struct queuefilenames_s { } *queuefilenames = NULL; +static ratelimit_t *dflt_ratelimiter = NULL; /* ratelimiter for submits without explicit one */ +static ratelimit_t *internalMsg_ratelimiter = NULL; /* ratelimiter for rsyslog-own messages */ int MarkInterval = 20 * 60; /* interval between marks in seconds - read-only after startup */ int send_to_all = 0; /* send message to all IPv4/IPv6 addresses */ static int NoFork = 0; /* don't fork - don't run in daemon mode - read-only after startup */ @@ -413,7 +407,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla CHKiRet(prop.Destruct(&pProp)); CHKiRet(MsgSetRcvFromIPStr(pMsg, hnameIP, ustrlen(hnameIP), &pProp)); CHKiRet(prop.Destruct(&pProp)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(submitMsg2(pMsg)); finalize_it: RETiRet; @@ -433,6 +427,12 @@ submitErrMsg(int iErr, uchar *msg) } +static inline rsRetVal +submitMsgWithDfltRatelimiter(msg_t *pMsg) +{ + return ratelimitAddMsg(dflt_ratelimiter, NULL, pMsg); +} + /* rgerhards 2004-11-09: the following is a function that can be used * to log a message orginating from the syslogd itself. */ @@ -484,50 +484,13 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) /* we have the queue, so we can simply provide the * message to the queue engine. */ - submitMsg(pMsg); + ratelimitAddMsg(internalMsg_ratelimiter, NULL, pMsg); + //submitMsgWithDfltRatelimiter(pMsg); } finalize_it: RETiRet; } -/* check message against ACL set - * rgerhards, 2009-11-16 - */ -#if 0 -static inline rsRetVal -chkMsgAgainstACL() { - /* if we reach this point, we had a good receive and can process the packet received */ - /* check if we have a different sender than before, if so, we need to query some new values */ - if(net.CmpHost(&frominet, frominetPrev, socklen) != 0) { - CHKiRet(net.cvthname(&frominet, fromHost, fromHostFQDN, fromHostIP)); - memcpy(frominetPrev, &frominet, socklen); /* update cache indicator */ - /* Here we check if a host is permitted to send us - * syslog messages. If it isn't, we do not further - * process the message but log a warning (if we are - * configured to do this). - * rgerhards, 2005-09-26 - */ - *pbIsPermitted = net.isAllowedSender((uchar*)"UDP", - (struct sockaddr *)&frominet, (char*)fromHostFQDN); - - if(!*pbIsPermitted) { - DBGPRINTF("%s is not an allowed sender\n", (char*)fromHostFQDN); - if(glbl.GetOption_DisallowWarning) { - time_t tt; - - datetime.GetTime(&tt); - if(tt > ttLastDiscard + 60) { - ttLastDiscard = tt; - errmsg.LogError(0, NO_ERRCODE, - "UDP message from disallowed sender %s discarded", - (char*)fromHost); - } - } - } - } -} -#endif - /* preprocess a batch of messages, that is ready them for actual processing. This is done * as a first stage and totally in parallel to any other worker active in the system. So @@ -536,9 +499,9 @@ chkMsgAgainstACL() { */ static inline rsRetVal preprocessBatch(batch_t *pBatch) { - uchar fromHost[NI_MAXHOST]; - uchar fromHostIP[NI_MAXHOST]; - uchar fromHostFQDN[NI_MAXHOST]; + prop_t *ip; + prop_t *fqdn; + prop_t *localName; prop_t *propFromHost = NULL; prop_t *propFromHostIP = NULL; int bSingleRuleset; @@ -550,31 +513,31 @@ preprocessBatch(batch_t *pBatch) { DEFiRet; bSingleRuleset = 1; - batchRuleset = (pBatch->nElem > 0) ? ((msg_t*) pBatch->pElem[0].pUsrp)->pRuleset : NULL; + batchRuleset = (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; for(i = 0 ; i < pBatch->nElem && !*(pBatch->pbShutdownImmediate) ; i++) { - pMsg = (msg_t*) pBatch->pElem[i].pUsrp; + pMsg = pBatch->pElem[i].pMsg; if((pMsg->msgFlags & NEEDS_ACLCHK_U) != 0) { DBGPRINTF("msgConsumer: UDP ACL must be checked for message (hostname-based)\n"); - if(net.cvthname(pMsg->rcvFrom.pfrominet, fromHost, fromHostFQDN, fromHostIP) != RS_RET_OK) + if(net.cvthname(pMsg->rcvFrom.pfrominet, &localName, &fqdn, &ip) != RS_RET_OK) continue; bIsPermitted = net.isAllowedSender2((uchar*)"UDP", - (struct sockaddr *)pMsg->rcvFrom.pfrominet, (char*)fromHostFQDN, 1); + (struct sockaddr *)pMsg->rcvFrom.pfrominet, (char*)propGetSzStr(fqdn), 1); if(!bIsPermitted) { DBGPRINTF("Message from '%s' discarded, not a permitted sender host\n", - fromHostFQDN); - pBatch->pElem[i].state = BATCH_STATE_DISC; + propGetSzStr(fqdn)); + pBatch->eltState[i] = BATCH_STATE_DISC; } else { /* save some of the info we obtained */ - MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); - CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); + MsgSetRcvFrom(pMsg, localName); + CHKiRet(MsgSetRcvFromIP(pMsg, ip)); pMsg->msgFlags &= ~NEEDS_ACLCHK_U; } } if((pMsg->msgFlags & NEEDS_PARSING) != 0) { if((localRet = parser.ParseMsg(pMsg)) != RS_RET_OK) { DBGPRINTF("Message discarded, parsing error %d\n", localRet); - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } } if(pMsg->pRuleset != batchRuleset) @@ -609,7 +572,7 @@ msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch, int *pbShu //do not have this yet and so we emulate -- 2010-06-10 int i; for(i = 0 ; i < pBatch->nElem && !*pbShutdownImmediate ; i++) { - pBatch->pElem[i].state = BATCH_STATE_COMM; + pBatch->eltState[i] = BATCH_STATE_COMM; } RETiRet; } @@ -620,7 +583,7 @@ int i; * rgerhards, 2008-02-13 */ rsRetVal -submitMsg(msg_t *pMsg) +submitMsg2(msg_t *pMsg) { qqueue_t *pQueue; ruleset_t *pRuleset; @@ -633,26 +596,31 @@ submitMsg(msg_t *pMsg) /* if a plugin logs a message during shutdown, the queue may no longer exist */ if(pQueue == NULL) { - DBGPRINTF("submitMsg() could not submit message - " + DBGPRINTF("submitMsg2() could not submit message - " "queue does (no longer?) exist - ignored\n"); FINALIZE; } - qqueueEnqObj(pQueue, pMsg->flowCtlType, (void*) pMsg); + qqueueEnqMsg(pQueue, pMsg->flowCtlType, pMsg); finalize_it: RETiRet; } +rsRetVal +submitMsg(msg_t *pMsg) +{ + return submitMsgWithDfltRatelimiter(pMsg); +} + /* submit multiple messages at once, very similar to submitMsg, just * for multi_submit_t. All messages need to go into the SAME queue! * rgerhards, 2009-06-16 */ rsRetVal -multiSubmitMsg(multi_submit_t *pMultiSub) +multiSubmitMsg2(multi_submit_t *pMultiSub) { - int i; qqueue_t *pQueue; ruleset_t *pRuleset; DEFiRet; @@ -677,8 +645,23 @@ multiSubmitMsg(multi_submit_t *pMultiSub) finalize_it: RETiRet; } +rsRetVal +multiSubmitMsg(multi_submit_t *pMultiSub) /* backward compat. level */ +{ + return multiSubmitMsg2(pMultiSub); +} +/* flush multiSubmit, e.g. at end of read records */ +rsRetVal +multiSubmitFlush(multi_submit_t *pMultiSub) +{ + DEFiRet; + if(pMultiSub->nElem > 0) { + iRet = multiSubmitMsg2(pMultiSub); + } + RETiRet; +} static void @@ -697,43 +680,6 @@ reapchild() } -/* helper to doFlushRptdMsgs() to flush the individual action links via llExecFunc - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(flushRptdMsgsActions) -{ - action_t *pAction = (action_t*) pData; - assert(pAction != NULL); - - BEGINfunc - d_pthread_mutex_lock(&pAction->mutAction); - /* TODO: time() performance: the call below could be moved to - * the beginn of the llExec(). This makes it slightly less correct, but - * in an acceptable way. -- rgerhards, 2008-09-16 - */ - if (pAction->f_prevcount && datetime.GetTime(NULL) >= REPEATTIME(pAction)) { - DBGPRINTF("flush %s: repeated %d times, %d sec.\n", - module.GetStateName(pAction->pMod), pAction->f_prevcount, - repeatinterval[pAction->f_repeatcount]); - actionWriteToAction(pAction); - BACKOFF(pAction); - } - d_pthread_mutex_unlock(&pAction->mutAction); - - ENDfunc - return RS_RET_OK; /* we ignore errors, we can not do anything either way */ -} - - -/* This method flushes repeat messages. - */ -static void -doFlushRptdMsgs(void) -{ - ruleset.IterateAllActions(runConf, flushRptdMsgsActions, NULL); -} - - static void debug_switch() { time_t tTime; @@ -774,8 +720,11 @@ static void debug_switch() * a minimal delay, but it is much cleaner than the approach of doing everything * inside the signal handler. * rgerhards, 2005-10-26 - * Note: we do not call DBGPRINTF() as this may cause us to block in case something - * with the threading is wrong. + * Note: + * - we do not call DBGPRINTF() as this may cause us to block in case something + * with the threading is wrong. + * - we do not really care about the return state of write(), but we need this + * strange check we do to silence compiler warnings (thanks, Ubuntu!) */ static void doDie(int sig) { @@ -783,11 +732,13 @@ static void doDie(int sig) # define MSG2 "DoDie called 5 times - unconditional exit\n" static int iRetries = 0; /* debug aid */ dbgprintf(MSG1); - if(Debug == DEBUG_FULL) - write(1, MSG1, sizeof(MSG1) - 1); + if(Debug == DEBUG_FULL) { + if(write(1, MSG1, sizeof(MSG1) - 1)) {} + } if(iRetries++ == 4) { - if(Debug == DEBUG_FULL) - write(1, MSG2, sizeof(MSG2) - 1); + if(Debug == DEBUG_FULL) { + if(write(1, MSG2, sizeof(MSG2) - 1)) {} + } abort(); } bFinished = sig; @@ -843,7 +794,7 @@ die(int sig) (void) snprintf(buf, sizeof(buf) / sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"]" " exiting on signal %d.", - (int) myPid, sig); + (int) glblGetOurPid(), sig); errno = 0; logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)buf, 0); } @@ -1106,7 +1057,7 @@ finalize_it: * the time being (remember that we want to restructure config processing at large!). * rgerhards, 2009-10-27 */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName) +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams) { struct queuefilenames_s *qfn; uchar *qfname = NULL; @@ -1122,60 +1073,65 @@ rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName) /* name our main queue object (it's not fatal if it fails...) */ obj.SetName((obj_t*) (*ppQueue), pszQueueName); - /* ... set some properties ... */ -# define setQPROP(func, directive, data) \ - CHKiRet_Hdlr(func(*ppQueue, data)) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } -# define setQPROPstr(func, directive, data) \ - CHKiRet_Hdlr(func(*ppQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } + if(queueParams == NULL) { /* use legacy parameters? */ + /* ... set some properties ... */ + # define setQPROP(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data)) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } + # define setQPROPstr(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } - if(ourConf->globals.mainQ.pszMainMsgQFName != NULL) { - /* check if the queue file name is unique, else emit an error */ - for(qfn = queuefilenames ; qfn != NULL ; qfn = qfn->next) { - dbgprintf("check queue file name '%s' vs '%s'\n", qfn->name, ourConf->globals.mainQ.pszMainMsgQFName ); - if(!ustrcmp(qfn->name, ourConf->globals.mainQ.pszMainMsgQFName)) { - snprintf((char*)qfrenamebuf, sizeof(qfrenamebuf), "%d-%s-%s", - ++qfn_renamenum, ourConf->globals.mainQ.pszMainMsgQFName, - (pszQueueName == NULL) ? "NONAME" : (char*)pszQueueName); - qfname = ustrdup(qfrenamebuf); - errmsg.LogError(0, NO_ERRCODE, "Error: queue file name '%s' already in use " - " - using '%s' instead", ourConf->globals.mainQ.pszMainMsgQFName, qfname); - break; + if(ourConf->globals.mainQ.pszMainMsgQFName != NULL) { + /* check if the queue file name is unique, else emit an error */ + for(qfn = queuefilenames ; qfn != NULL ; qfn = qfn->next) { + dbgprintf("check queue file name '%s' vs '%s'\n", qfn->name, ourConf->globals.mainQ.pszMainMsgQFName ); + if(!ustrcmp(qfn->name, ourConf->globals.mainQ.pszMainMsgQFName)) { + snprintf((char*)qfrenamebuf, sizeof(qfrenamebuf), "%d-%s-%s", + ++qfn_renamenum, ourConf->globals.mainQ.pszMainMsgQFName, + (pszQueueName == NULL) ? "NONAME" : (char*)pszQueueName); + qfname = ustrdup(qfrenamebuf); + errmsg.LogError(0, NO_ERRCODE, "Error: queue file name '%s' already in use " + " - using '%s' instead", ourConf->globals.mainQ.pszMainMsgQFName, qfname); + break; + } } + if(qfname == NULL) + qfname = ustrdup(ourConf->globals.mainQ.pszMainMsgQFName); + qfn = malloc(sizeof(struct queuefilenames_s)); + qfn->name = qfname; + qfn->next = queuefilenames; + queuefilenames = qfn; } - if(qfname == NULL) - qfname = ustrdup(ourConf->globals.mainQ.pszMainMsgQFName); - qfn = malloc(sizeof(struct queuefilenames_s)); - qfn->name = qfname; - qfn->next = queuefilenames; - queuefilenames = qfn; - } - setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", ourConf->globals.mainQ.iMainMsgQueMaxFileSize); - setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", ourConf->globals.mainQ.iMainMsgQueMaxDiskSpace); - setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", ourConf->globals.mainQ.iMainMsgQueDeqBatchSize); - setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", qfname); - setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", ourConf->globals.mainQ.iMainMsgQPersistUpdCnt); - setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", ourConf->globals.mainQ.bMainMsgQSyncQeueFiles); - setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", ourConf->globals.mainQ.iMainMsgQtoQShutdown ); - setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", ourConf->globals.mainQ.iMainMsgQtoActShutdown); - setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", ourConf->globals.mainQ.iMainMsgQtoWrkShutdown); - setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", ourConf->globals.mainQ.iMainMsgQtoEnq); - setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", ourConf->globals.mainQ.iMainMsgQHighWtrMark); - setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", ourConf->globals.mainQ.iMainMsgQLowWtrMark); - setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", ourConf->globals.mainQ.iMainMsgQDiscardMark); - setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", ourConf->globals.mainQ.iMainMsgQDiscardSeverity); - setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", ourConf->globals.mainQ.iMainMsgQWrkMinMsgs); - setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", ourConf->globals.mainQ.bMainMsgQSaveOnShutdown); - setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", ourConf->globals.mainQ.iMainMsgQDeqSlowdown); - setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", ourConf->globals.mainQ.iMainMsgQueueDeqtWinFromHr); - setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", ourConf->globals.mainQ.iMainMsgQueueDeqtWinToHr); - -# undef setQPROP -# undef setQPROPstr + setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", ourConf->globals.mainQ.iMainMsgQueMaxFileSize); + setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", ourConf->globals.mainQ.iMainMsgQueMaxDiskSpace); + setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", ourConf->globals.mainQ.iMainMsgQueDeqBatchSize); + setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", qfname); + setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", ourConf->globals.mainQ.iMainMsgQPersistUpdCnt); + setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", ourConf->globals.mainQ.bMainMsgQSyncQeueFiles); + setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", ourConf->globals.mainQ.iMainMsgQtoQShutdown ); + setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", ourConf->globals.mainQ.iMainMsgQtoActShutdown); + setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", ourConf->globals.mainQ.iMainMsgQtoWrkShutdown); + setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", ourConf->globals.mainQ.iMainMsgQtoEnq); + setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", ourConf->globals.mainQ.iMainMsgQHighWtrMark); + setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", ourConf->globals.mainQ.iMainMsgQLowWtrMark); + setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", ourConf->globals.mainQ.iMainMsgQDiscardMark); + setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", ourConf->globals.mainQ.iMainMsgQDiscardSeverity); + setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", ourConf->globals.mainQ.iMainMsgQWrkMinMsgs); + setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", ourConf->globals.mainQ.bMainMsgQSaveOnShutdown); + setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", ourConf->globals.mainQ.iMainMsgQDeqSlowdown); + setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", ourConf->globals.mainQ.iMainMsgQueueDeqtWinFromHr); + setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", ourConf->globals.mainQ.iMainMsgQueueDeqtWinToHr); + + # undef setQPROP + # undef setQPROPstr + } else { /* use new style config! */ + qqueueSetDefaultsRulesetQueue(*ppQueue); + qqueueApplyCnfParam(*ppQueue, queueParams); + } /* ... and finally start the queue! */ CHKiRet_Hdlr(qqueueStart(*ppQueue)) { @@ -1212,7 +1168,7 @@ init(void) snprintf(bufStartUpMsg, sizeof(bufStartUpMsg)/sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] start", - (int) myPid); + (int) glblGetOurPid()); logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)bufStartUpMsg, 0); } @@ -1254,7 +1210,7 @@ static inline void processImInternal(void) msg_t *pMsg; while(iminternalRemoveMsg(&pMsg) == RS_RET_OK) { - submitMsg(pMsg); + submitMsgWithDfltRatelimiter(pMsg); } } @@ -1291,7 +1247,7 @@ doHUP(void) snprintf(buf, sizeof(buf) / sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed", - (int) myPid); + (int) glblGetOurPid()); errno = 0; logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)buf, 0); } @@ -1321,49 +1277,22 @@ mainloop(void) while(!bFinished){ /* this is now just a wait - please note that we do use a near-"eternal" - * timeout of 1 day if we do not have repeated message reduction turned on - * (which it is not by default). This enables us to help safe the environment + * timeout of 1 day. This enables us to help safe the environment * by not unnecessarily awaking rsyslog on a regular tick (just think * powertop, for example). In that case, we primarily wait for a signal, * but a once-a-day wakeup should be quite acceptable. -- rgerhards, 2008-06-09 */ - tvSelectTimeout.tv_sec = (runConf->globals.bReduceRepeatMsgs == 1) ? TIMERINTVL : 86400 /*1 day*/; - //tvSelectTimeout.tv_sec = TIMERINTVL; /* TODO: change this back to the above code when we have a better solution for apc */ + tvSelectTimeout.tv_sec = 86400 /*1 day*/; tvSelectTimeout.tv_usec = 0; select(1, NULL, NULL, NULL, &tvSelectTimeout); if(bFinished) - break; /* exit as quickly as possible - see long comment below */ - - /* If we received a HUP signal, we call doFlushRptdMsgs() a bit early. This - * doesn't matter, because doFlushRptdMsgs() checks timestamps. What may happen, - * however, is that the too-early call may lead to a bit too-late output - * of "last message repeated n times" messages. But that is quite acceptable. - * rgerhards, 2007-12-21 - * ... and just to explain, we flush here because that is exactly what the mainloop - * shall do - provide a periodic interval in which not-yet-flushed messages will - * be flushed. Be careful, there is a potential race condition: doFlushRptdMsgs() - * needs to aquire a lock on the action objects. If, however, long-running consumers - * cause the main queue worker threads to lock them for a long time, we may receive - * a starvation condition, resulting in the mainloop being held on lock for an extended - * period of time. That, in turn, could lead to unresponsiveness to termination - * requests. It is especially important that the bFinished flag is checked before - * doFlushRptdMsgs() is called (I know because I ran into that situation). I am - * not yet sure if the remaining probability window of a termination-related - * problem is large enough to justify changing the code - I would consider it - * extremely unlikely that the problem ever occurs in practice. Fixing it would - * require not only a lot of effort but would cost considerable performance. So - * for the time being, I think the remaining risk can be accepted. - * rgerhards, 2008-01-10 - */ - if(runConf->globals.bReduceRepeatMsgs == 1) - doFlushRptdMsgs(); + break; /* exit as quickly as possible */ if(bHadHUP) { doHUP(); bHadHUP = 0; continue; } - // TODO: remove execScheduled(); /* handle Apc calls (if any) */ } ENDfunc } @@ -1409,6 +1338,11 @@ static void printVersion(void) #else printf("\tRuntime Instrumentation (slow code):\tNo\n"); #endif +#ifdef USE_LIBUUID + printf("\tuuid support:\t\t\t\tYes\n"); +#else + printf("\tuuid support:\t\t\t\tNo\n"); +#endif printf("\nSee http://www.rsyslog.com for more information.\n"); } @@ -1437,8 +1371,6 @@ InitGlobalClasses(void) CHKiRet(objUse(module, CORE_COMPONENT)); pErrObj = "datetime"; CHKiRet(objUse(datetime, CORE_COMPONENT)); - pErrObj = "rule"; - CHKiRet(objUse(rule, CORE_COMPONENT)); pErrObj = "ruleset"; CHKiRet(objUse(ruleset, CORE_COMPONENT)); pErrObj = "conf"; @@ -1461,6 +1393,7 @@ InitGlobalClasses(void) CHKiRet(objUse(net, LM_NET_FILENAME)); dnscacheInit(); initRainerscript(); + ratelimitModInit(); finalize_it: if(iRet != RS_RET_OK) { @@ -1492,7 +1425,6 @@ GlobalClassExit(void) objRelease(prop, CORE_COMPONENT); objRelease(conf, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); - objRelease(rule, CORE_COMPONENT); parserClassExit(); /* this is hack, currently core_modules do not get this automatically called */ rsconfClassExit(); /* this is hack, currently core_modules do not get this automatically called */ objRelease(datetime, CORE_COMPONENT); @@ -1500,6 +1432,7 @@ GlobalClassExit(void) /* TODO: implement the rest of the deinit */ /* dummy "classes */ strExit(); + ratelimitModExit(); #if 0 CHKiRet(objGetObjInterface(&obj)); /* this provides the root pointer for all other queries */ @@ -1770,7 +1703,7 @@ doGlblProcessInit(void) fputs("Pidfile (and pid) already exist.\n", stderr); exit(1); /* exit during startup - questionable */ } - myPid = getpid(); /* save our pid for further testing (also used for messages) */ + glblSetOurPid(getpid()); memset(&sigAct, 0, sizeof (sigAct)); sigemptyset(&sigAct.sa_mask); @@ -1825,7 +1758,7 @@ int realMain(int argc, char **argv) * of other options, we do this during the inital option processing. * rgerhards, 2008-04-04 */ - while((ch = getopt(argc, argv, "46a:Ac:def:g:hi:l:m:M:nN:op:qQr::s:t:T:u:vwx")) != EOF) { + while((ch = getopt(argc, argv, "46a:Ac:dDef:g:hi:l:m:M:nN:op:qQr::s:t:T:u:vwx")) != EOF) { switch((char)ch) { case '4': case '6': @@ -1853,11 +1786,15 @@ int realMain(int argc, char **argv) CHKiRet(bufOptAdd(ch, optarg)); break; case 'c': /* compatibility mode */ - fprintf(stderr, "rsyslogd: error: option -c is no longer supported - ignored"); + fprintf(stderr, "rsyslogd: error: option -c is no longer supported - ignored\n"); break; case 'd': /* debug - must be handled now, so that debug is active during init! */ debugging_on = 1; Debug = 1; + yydebug = 1; + break; + case 'D': /* BISON debug */ + yydebug = 1; break; case 'e': /* log every message (no repeat message supression) */ bEOptionWasGiven = 1; @@ -2034,6 +1971,12 @@ int realMain(int argc, char **argv) } CHKiRet(localRet); + CHKiRet(ratelimitNew(&dflt_ratelimiter, "rsyslogd", "dflt")); + /* TODO: add linux-type limiting capability */ + CHKiRet(ratelimitNew(&internalMsg_ratelimiter, "rsyslogd", "internal_messages")); + ratelimitSetLinuxLike(internalMsg_ratelimiter, 5, 500); + /* TODO: make internalMsg ratelimit settings configurable */ + if(bChDirRoot) { if(chdir("/") != 0) fprintf(stderr, "Can not do 'cd /' - still trying to run\n"); @@ -2057,7 +2000,7 @@ int realMain(int argc, char **argv) } /* Send a signal to the parent so it can terminate. */ - if(myPid != ppid) + if(glblGetOurPid() != ppid) kill(ppid, SIGTERM); |