diff options
265 files changed, 23151 insertions, 5420 deletions
@@ -1,4 +1,782 @@ --------------------------------------------------------------------------- +Version 7.4.3 [v7.4-stable] 2013-07-18 +- bugfix: $QHOUR/$HHOUR were always "00" or "01" + regression some time between v5 and here + Thanks to forum user rjmcinty for reporting this bug +--------------------------------------------------------------------------- +Version 7.4.3 [v7.4-stable] 2013-07-18 +- bugfix: memory leak if disk queues were used and json data present +- bugfix: CEE/json data was lost during disk queue operation +- bugfix: potential segfault during startup on invalid config + could happen if invalid actions were present, which could lead + to improper handling in optimizer. +- bugfix: 100% CPU utilization when DA queue became full +- bugfix: omlibdbi did not properly close connection on some errors + This happened to errors occuring in Begin/End Transaction entry + points. +- cosmetic bugfix: file name buffer was not freed on disk queue destruction + This was an extremely small one-time per run memleak, so nothing of + concern. However, it bugs under valgrind and similar memory debuggers. +- fix build on FreeBSD + Thanks to Christiano Rolim for the patch +--------------------------------------------------------------------------- +Version 7.4.2 [v7.4-stable] 2013-07-04 +- bugfix: in RFC5425 TLS, multiple wildcards in auth could cause segfault +- bugfix: RainerScript object required parameters were not properly + checked - this clould result to segfaults on startup if parameters + were missing. +- bugfix: double-free in omelasticsearch + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=461 + a security advisory for this bug is available at: + http://www.lsexperts.de/advisories/lse-2013-07-03.txt + CVE: CVE-2013-4758 + PLEASE NOTE: This issue only existed if omelasticsearch was used + in a non-default configuration, where the "errorfile" parameter + was specified. Without that parameter set, the bug could not + be triggered. + Thanks to Markus Vervier and Marius Ionescu for providing a detailled + bug report. Special thanks to Markus for coordinating his security + advisory with us. +- bugfix: omrelp potential segfault at startup on invalid config parameters +- bugfix: small memory leak when $uptime property was used +- bugfix: potential segfault on rsyslog termination in imudp + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=456 +- bugfix: lmsig_gt abort on invalid configuration parameters + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=448 + Thanks to Risto Laanoja for the patch. +- imtcp: fix typo in "listner" parameter, which is "listener" + Currently, both names are accepted. +- solved build problems on FreeBSD + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=457 + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=458 + Thanks to Christiano for reproting and suggesting patches +- solved build problems on CENTOS5 +--------------------------------------------------------------------------- +Version 7.4.1 [v7.4-stable] 2013-06-17 +- imjournal: add ratelimiting capability + The original imjournal code did not support ratelimiting at all. We + now have our own ratelimiter. This can mitigate against journal + database corruption, when the journal re-sends old data. This is a + current bug in systemd journal, but we won't outrule this to happen + in the future again. So it is better to have a safeguard in place. + By default, we permit 20,000 messages witin 10 minutes. This may + be a bit restrictive, but given the risk potential it seems reasonable. + Users requiring larger traffic flows can always adjust the value. +- bugfix: potential loop in rate limiting + if the message that tells about rate-limiting gets rate-limited itself, + it will potentially create and endless loop +- bugfix: potential segfault in imjournal if journal DB is corrupted +- bugfix: prevent a segfault in imjournal if state file is not defined +- bugfix imzmq3: potential segfault on startup + if no problem happend at startup, everything went fine + Thanks to Hongfei Cheng and Brian Knox for the patch +--------------------------------------------------------------------------- +Version 7.4.0 [v7.4-stable] 2013-06-06 +This starts a new stable branch based on 7.3.15 plus the following changes: +- add --enable-cached-man-pages ./configure option + permits to build rsyslog on a system where rst2man is not installed. In + that case, cached versions of the man pages are used (they were built + during "make dist", so they should be current for the version in + question. +- doc bugfix: ReadMode wrong in imfile doc, two values were swapped + Thanks to jokajak@gmail.com for mentioning this + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=450 +- imjournal: no longer do periodic wakeup +- bugfix: potential hang *in debug mode* on rsyslogd termination + This ONLY affected rsyslogd if it were running with debug output + enabled. +- bugfix: $template statement with multiple spaces lead to invalid tpl name + If multiple spaces were used in front of the template name, all but one + of them became actually part of the template name. So + $template a,"..." would be name " a", and as such "a" was not + available, e.g. in + *.* /var/log/file;a + This is a legacy config problem. As it was unreported for many years, + no backport of the fix to old versions will happen. + This is a long-standing bug that was only recently reported by forum + user mc-sim. + Reference: http://kb.monitorware.com/post23448.html +- 0mq fixes; credits to Hongfei Cheng and Brian Knox +--------------------------------------------------------------------------- +Version 7.3.15 [beta] 2013-05-15 +- bugfix: problem in build system (especially when cross-compiling) + Thanks to Tomas Heinrich and winfried_mb2@xmsnet.nl for the patch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=445 +- bugfix: imjournal had problem with systemd journal API change +- imjournal: now obtain and include PID +- bugfix: .logsig files had tlv16 indicator bit at wrong offset +- bugfix: omrelp legacy config parameters set a timeout of zero + which lead the legacy config to be unusable. +- bugfix: segfault on startup if a disk queue was configure without file + name + Now this triggers an error message and the queue is changed to + linkedList type. +- bugfix: invalid addressing in string class (recent regression) +--------------------------------------------------------------------------- +Version 7.3.14 [beta] 2013-05-06 +- bugfix: some man pages were not properly installed + either rscryutil or rsgtutil man was installed, but not both + Thanks to Marius Tomaschewski for the patch. +- bugfix: potential segfault on startup when builtin module was specified + in module() statement. + Thanks to Marius Tomaschewski for reporting the bug. +- bugfix: segfault due to invalid dynafile cache handling + Accidently, the old-style cache size parameter was used when the + dynafile cache was created in a RainerScript action. If the old-style + size was lower than the one actually set, this lead to misadressing + when the size was overrun, and that could lead to all kinds of + "interesting things", often in segfaults. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=440 +--------------------------------------------------------------------------- +Version 7.3.13 [beta] 2013-04-29 +- added omrabbitmq module (contributed, untested) + Note: this is unsupported and as such was moved immediately into the + beta version. + Thanks to Vaclav Tomec for providing this module. +- bugfix: build problem when --enable-encryption was not selected + Thanks to Michael Biebl for fixing this. +- doc bugfix: omfile parameter "VeryRobustZip" was documentas as + "VeryReliableZip" + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=437 + Thanks to Thomas Doll for reporting this. +--------------------------------------------------------------------------- +Version 7.3.12 [devel] 2013-04-25 +- added doc for omelasticsearch + Thanks to Radu Gheorghe for the doc contribution. +- omelasticsearch: _id field support for bulk operations + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=392 + Thanks to Jérôme Renard for the idea and patches. +- max number of templates for plugin use has been increased to five +- platform compatibility enhancement: solve compile issue with libgcrypt + do not use GCRY_CIPHER_MODE_AESWRAP where not available +- fix compile on Solaris + Thanks to Martin Carpenter for the patch. +- bugfix: off-by-one error in handling local FQDN name (regression) + A remporary buffer was allocated one byte too small. Did only + affect startup, not actual operations. Came up during routine tests, + and can have no effect once the engine runs. Bug was introduced in + 7.3.11. +- bugfix: build problems on Solaris + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=436 +- bugfix: block size limit was not properly honored +- bugfix: potential segfault in guardtime signature provider + it could segfault if an error was reported by the GuardTime API, because + an invalid free could happen then +--------------------------------------------------------------------------- +Version 7.3.11 [devel] 2013-04-23 +- added support for encrypting log files +- omhiredis: added support for redis pipeline support + Thanks to Brian Knox for the patch. +- bugfix: $PreserveFQDN is not properly working + Thanks to Louis Bouchard for the patch + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=426 +- bugfix: imuxsock aborted due to problem in ratelimiting code + Thanks to Tomas Heinrich for the patch. +- bugfix: imuxsock aborted under some conditions + regression from ratelimiting enhancements - this was a different one + to the one Tomas Heinrich patched. +- bugfix: timestamp problems in imkmsg +--------------------------------------------------------------------------- +Version 7.3.10 [devel] 2013-04-10 +- added RainerScript re_extract() function +- omrelp: added support for RainerScript-based configuration +- omrelp: added ability to specify session timeout +- templates now permit substring extraction relative to end-of-string +- bugfix: failover/action suspend did not work correctly + This was experienced if the retry action took more than one second + to complete. For suspending, a cached timestamp was used, and if the + retry took longer, that timestamp was already in the past. As a + result, the action never was kept in suspended state, and as such + no failover happened. The suspend functionalit now does no longer use + the cached timestamp (should not have any performance implication, as + action suspend occurs very infrequently). +- bugfix: gnutls RFC5425 driver had some undersized buffers + Thanks to Tomas Heinrich for the patch. +- bugfix: nested if/prifilt conditions did not work properly + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=415 +- bugfix: imuxsock aborted under some conditions + regression from ratelimiting enhancements +- bugfix: build problems on Solaris + Thanks to Martin Carpenter for the patches. +--------------------------------------------------------------------------- +Version 7.3.9 [devel] 2013-03-27 +- support for signing logs added +- imudp: now supports user-selectable inputname +- omlibdbi: now supports transaction interface + if recent enough lbdbi is present +- imuxsock: add ability to NOT create/delete sockets during startup and + shutdown + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=259 +- imfile: errors persisting state file are now reported + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=292 +- imfile: now detects file change when rsyslog was inactive + Previosly, this case could not be detected, so if a file was overwritten + or rotated away while rsyslog was stopped, some data was missing. This + is now detected and the new file being forwarded right from the + beginning. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=228 +- updated systemd files to match current systemd source +- bugfix: imudp scheduling parameters did affect main thread, not imudp + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=409 +- bugfix: build problem on platforms without GLOB_NOMAGIC +- bugfix: build problems on non-Linux platforms +- bugfix: stdout/stderr were not closed on forking + but were closed when running in the forground - this was just reversed + of what it should be. This is a regression of a recent change. +--------------------------------------------------------------------------- +Version 7.3.8 [devel] 2013-03-18 +- imrelp: now supports listening to IPv4/v6 only instead of always both + build now requires librelp 1.0.2 + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=378 +- bugfix: mmanon did not build on some platforms (e.g. Ubuntu) +- bugfix: segfault in expression optimizer + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=423 +- bugfix: imuxsock was missing SysSock.ParseTrusted module parameter + To use that functionality, legacy rsyslog.conf syntax had to be used. + Also, the doc was missing information on the "ParseTrusted" set of + config directives. +- bugfix: include files got included in the wrong order + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=411 + This happens if an $IncludeConfig directive was done on multiple + files (e.g. the distro default of $IncludeConfig /etc/rsyslog.d/*.conf). + In that case, the order of include file processing is reversed, which + could lead to all sorts of problems. + Thanks to Nathan Stratton Treadway for his great analysis of the problem, + which made bug fixing really easy. +--------------------------------------------------------------------------- +Version 7.3.7 [devel] 2013-03-12 +- add support for anonymizing IPv4 addresses +- add support for writing to the Linux Journal (omjournal) +- imuxsock: add capability to ignore messages from ourselfes + This helps prevent message routing loops, and is vital to have + if omjournal is used together with traditional syslog. +- field() function now supports a string as field delimiter +- added ability to configure debug system via rsyslog.conf +- bugfix: imuxsock segfault when system log socket was used +- bugfix: mmjsonparse segfault if new-style config was used +- bugfix: script == comparison did not work properly on JSON objects +- bugfix: field() function did never return "***FIELD NOT FOUND***" + instead it returned "***ERROR in field() FUNCTION***" in that case +--------------------------------------------------------------------------- +Version 7.3.6 [devel] 2013-01-28 +- greatly improved speed of large-array [N]EQ RainerScript comparisons + Thanks to David Lang for a related discussion that inspired the idea + to do this with a much simpler (yet sufficient) approach than orignally + planned for. +- greatly improved speed of DNS cache for large cache sizes +- general performance improvements +- omfile: added stats counters for dynafile caches +- omfile: improved async writing, finally enabled full async write + also fixed a couple of smaller issues along that way +- impstats: added ability to write stats records to local file + and avoid going through the syslog log stream. syslog logging can now + also be turned off (see doc for details). +- bugfix: imklog issued wrong facility in error messages + ...what could lead to problems in other parts of the code +- fix compile problem in imklog +- added capability to output thread-id-to-function debug info + This is a useful debug aid, but nothing of concern for regular users. +--------------------------------------------------------------------------- +Version 7.3.5 [devel] 2012-12-19 +- ommysql: addded batching/transaction support +- enhanced script optimizer to optimize common PRI-based comparisons + These constructs are especially used in SUSE default config files, + but also by many users (as they are more readable than the equivalent + PRI-based filter). +- omudpspoof: add support for new config system +- omudpspoof: add support for packets larger than 1472 bytes + On Ethernet, they need to be transmitted in multiple fragments. While + it is known that fragmentation can cause issues, it is the best choice + to be made in that case. Also improved debug output. +- bugfix: omudpspoof failed depending on the execution environment + The v7 engine closes fds, and closed some of libnet's fds as well, what + lead to problems (unfortunately, at least some libnet versions do not + report a proper error state but still "success"...). The order of libnet + calls has been adjusted to by in sync with what the core engine does. +- bugfix: segfault on imuxsock startup if system log socket is used + and no ratelimiting supported. Happens only during initial config + read phase, once this is over, everything works stable. +- bugfix: mmnormalize build problems +- bugfix: mmnormalize could abort rsyslog if config parameter was in error +- bugfix: no error message for invalid string template parameters + rather a malformed template was generated, and error information emitted + at runtime. However, this could be quite confusing. Note that with this + "bugfix" user experience changes: formerly, rsyslog and the affected + actions properly started up, but the actions did not produce proper + data. Now, there are startup error messages and the actions are NOT + executed (due to missing template due to template error). +- bugfix[minor]: invalid error code when mmnormalize could not access + rulebase +- bugfix(kind of): script optimizer did not work for complex boolean + expressions +- doc bugfix: corrections and improvements in mmnormalize html doc page +- bugfix: some message properties could be garbled due to race condition + This happened only on very high volume systems, if the same message was + being processed by two different actions. This was a regression caused + by the new config processor, which did no longer properly enable msg + locking in multithreaded cases. The bugfix is actually a refactoring of + the msg locking code - we no longer do unlocked operations, as the use + case for it has mostly gone away. It is potentially possible only at + very low-end systems, and there the small additional overhead of doing + the locking does not really hurt. Instead, the removal of that + capability can actually slightly improve performance in common cases, + as the code path is smaller and requires slightly less memory writes. + That probably outperforms the extra locking overhead (which in the + low-end case always happens in user space, without need for kernel + support as we can always directly aquire the lock - there is no + contention at all). +- build system cleanup (thanks to Michael Biebl for this!) +- bugfix: omelasticsearch did not properly compile on some platforms + due to missing libmath. Thanks to Michael Biebl for the fix +--------------------------------------------------------------------------- +Version 7.3.4 [devel] 2012-11-23 +- further (and rather drastically) improved disk queue performance + we now save one third of the IO calls +- imklog: added ParseKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- imklog: added KeepKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- bugfix: improper handling of backslash in string-type template()s +- bugfix: leading quote (") in string-type template() lead to thight loop + on startup +- bugfix: no error msg on invalid field option in legacy/string template +- bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds + ... actually, they are microseconds. So the fractional part of the + timestamp was not properly formatted. (import from 5.10.2) + Thanks to Marius Tomaschewski for the bug report and the patch idea. +--------------------------------------------------------------------------- +Version 7.3.3 [devel] 2012-11-07 +- improved disk queue performance +- bugfix: dynafile zip files could be corrupted + This could happen if a dynafile was destructed before the first write. + In practice, this could happen if few lines were written to a file and + it then became evicted from the dynafile cache. This would probably + look very random, because it depended on the timing in regard to + message volume and dynafile cache size. +--------------------------------------------------------------------------- +Version 7.3.2 [devel] 2012-10-30 +- mmnormalize: support for v6+ config interface added +- mmjsonparse: support for v6+ config interface added +--------------------------------------------------------------------------- +Version 7.3.2 [devel] 2012-10-30 +- totally reworked ratelimiting and "last message repeated n times" + all over rsyslog code. Each of the supported inputs now supports + linux-like ratelimiting (formerly only imuxsock did). Also, the + "last message repeated n times" is now processed at the input side + and no longer at the output side of rsyslog processing. This + provides the basis for new future additions as well as usually more + performance and a much simpler output part (which can be even further + refactored). +- imtcp: support for Linux-Type ratelimiting added +- imptcp: support for Linux-Type ratelimiting added +- imudp enhancements: + * support for input batching added (performance improvement) + * support for Linux-Type ratelimiting added +- permited action-like statements (stop, call, ...) in action lists +- bugfix: segfault on startup when modules using MSG_PASSING mode are used +- omelasticsearch: support for writing data errors to local file added +- omelasticsearch: fix check for bulk processing status response +--------------------------------------------------------------------------- +Version 7.3.1 [devel] 2012-10-19 +- optimized template processing performance, especially for $NOW family + of properties +- change lumberjack cookie to "@cee:" from "@cee: " + CEE originally specified the cookie with SP, whereas other lumberjack + tools used it without space. In order to keep interop with lumberjack, + we now use the cookie without space as well. I hope this can be changed + in CEE as well when it is released at a later time. + Thanks to Miloslav Trmač for pointing this out and a similiar v7 patch. +- bugfix: imuxsock and imklog truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski, Florian Piekert and Milan Bartos for their help in + solving this issue. +- bugfix: imuxsock did not properly honor $LocalHostIPIF +--------------------------------------------------------------------------- +Version 7.3.0 [devel] 2012-10-09 +- omlibdbi improvements, added + * support for config load phases & module() parameters + * support for default templates + * driverdirectory is now cleanly a global parameter, but can no longer + be specified as an action paramter. Note that in previous versions + this parameter was ignored in all but the first action definition +- improved omfile zip writer to increase compression + This was achieved by somewhat reducing the robustness of the zip archive. + This is controlled by the new action parameter "VeryReliableZip". +---------------------------------------------------------------------------- +Version 7.2.8 [v7-stable] 2013-0?-?? +- bugfix: potential segfault on startup when builtin module was specified + in module() statement. + Thanks to Marius Tomaschewski for reporting the bug. +- bugfix: segfault due to invalid dynafile cache handling + Accidently, the old-style cache size parameter was used when the + dynafile cache was created in a RainerScript action. If the old-style + size was lower than the one actually set, this lead to misadressing + when the size was overrun, and that could lead to all kinds of + "interesting things", often in segfaults. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=440 +---------------------------------------------------------------------------- +Version 7.2.7 [v7-stable] 2013-04-17 +- rsyslogd startup information is now properly conveyed back to init + when privileges are beging dropped + Actually, we have moved termination of the parent in front of the + priv drop. So it shall work now in all cases. See code comments in + commit for more details. +- If forking, the parent now waits for a maximum of 60 seconds for + termination by the child +- improved debugging support in forked (auto-backgrounding) mode + The rsyslog debug log file is now continued to be written across the + fork. +- updated systemd files to match current systemd source +- bugfix: failover/action suspend did not work correctly + This was experienced if the retry action took more than one second + to complete. For suspending, a cached timestamp was used, and if the + retry took longer, that timestamp was already in the past. As a + result, the action never was kept in suspended state, and as such + no failover happened. The suspend functionalit now does no longer use + the cached timestamp (should not have any performance implication, as + action suspend occurs very infrequently). +- bugfix: nested if/prifilt conditions did not work properly + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=415 +- bugfix: script == comparison did not work properly on JSON objects + [backport from 7.3 branch] +- bugfix: imudp scheduling parameters did affect main thread, not imudp + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=409 +- bugfix: imuxsock rate-limiting could not be configured via legacy conf + Rate-limiting for the system socket could not be configured via legacy + configuration directives. However, the new-style RainerScript config + options worked. + Thanks to Milan Bartos for the patch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=390 +- bugfix: using group resolution could lead to endless loop + Thanks to Tomas Heinrich for the patch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=310 +- bugfix: $mmnormalizeuseramsg paramter was specified with wrong type + Thank to Renzhong Zhang for alerting us of the problem. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=420 +- bugfix: RainerScript getenv() function caused segfault when var was + not found. + Thanks to Philippe Muller for the patch. +- bugfix: several issues in imkmsg + see bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=421#c8 +- bugfix: imuxsock was missing SysSock.ParseTrusted module parameter + To use that functionality, legacy rsyslog.conf syntax had to be used. + Also, the doc was missing information on the "ParseTrusted" set of + config directives. +- bugfix: parameter action.execOnlyWhenPreviousIsSuspended was accidently + of integer-type. For obvious reasons, it needs to be boolean. Note + that this change can break existing configurations if they circumvented + the problem by using 0/1 values. +- doc bugfix: rsyslog.conf man page had invalid file format info + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=418 +---------------------------------------------------------------------------- +Version 7.2.6 [v7-stable] 2013-03-05 +- slightly improved config parser error messages when invalid escapes happen +- bugfix: include files got included in the wrong order + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=411 + This happens if an $IncludeConfig directive was done on multiple + files (e.g. the distro default of $IncludeConfig /etc/rsyslog.d/*.conf). + In that case, the order of include file processing is reversed, which + could lead to all sorts of problems. + Thanks to Nathan Stratton Treadway for his great analysis of the problem, + which made bug fixing really easy. +- bugfix: omelasticsearch failed when authentication data was provided + ... at least in most cases it emitted an error message: + "snprintf failed when trying to build auth string" + Thanks to Joerg Heinemann for alerting us. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=404 +- bugfix: some property-based filter were incorrectly parsed + This usually lead to a syntax error on startup and rsyslogd not actually + starting up. The problem was the regex, which did not care for double + quote characters to follow in the action part - unfortunately something + that can frequently happen with v6+ format. An example: + :programname, isequal, "as" {action(type="omfile" ...) } + Here, the part + :programname, isequal, "as" {action(type="omfile" + was treated as the property filter, and the rest as action part. + Obviously, this did not work out. Unfortunately, such situations usually + resulted in very hard to understand error messages. +---------------------------------------------------------------------------- +Version 7.2.5 [v7-stable] 2013-01-08 +- build system cleanup (thanks to Michael Biebl for this!) +- bugfix: omelasticsearch did not properly compile on some platforms + due to missing libmath. Thanks to Michael Biebl for the fix +- bugfix: invalid DST handling under Solaris + Thanks to Scott Severtson for the patch. +- bugfix: on termination, actions were incorrectly called + The problem was that incomplete fiter evaluation was done *during the + shutdown phase*. This affected only the LAST batches being processed. No + problem existed during the regular run. Could usually only happen on + very busy systems, which were still busy during shutdown. +- bugfix: very large memory consumption (and probably out of memory) when + FromPos was specified in template, but ToPos not. + Thanks to Radu Gheorghe for alerting us of this bug. +- bugfix: timeval2syslogTime cause problems on some platforms + due to invalid assumption on structure data types. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=394 + Thanks to David Hill for the patch [under ASL2.0 as per email conversation + 2013-01-03]. +- bugfix: compile errors in im3195 + Thanks to Martin Körper for the patch +- bugfix: doGetFileCreateMode() had invalid validity check ;) + Thanks to Chandler Latour for the patch. +- bugfix: mmjsonparse errornously returned action error when no CEE cookie + was present. +---------------------------------------------------------------------------- +Version 7.2.4 [v7-stable] 2012-12-07 +- enhance: permit RFC3339 timestamp in local log socket messages + Thanks to Sebastien Ponce for the patch. +- imklog: added ParseKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- fix missing functionality: ruleset(){} could not specify ruleset queue + The "queue.xxx" parameter set was not supported, and legacy ruleset + config statements did not work (by intention). The fix introduces the + "queue.xxx" parameter set. It has some regression potential, but only + for the new functionality. Note that using that interface it is possible + to specify duplicate queue file names, which will cause trouble. This + will be solved in v7.3, because there is a too-large regression + potential for the v7.2 stable branch. +- imklog: added KeepKernelTimestamp parameter (import from 5.10.2) + Thanks to Marius Tomaschewski for the patch. +- bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds + ... actually, they are microseconds. So the fractional part of the + timestamp was not properly formatted. (import from 5.10.2) + Thanks to Marius Tomaschewski for the bug report and the patch idea. +- bugfix: supportoctetcountedframing parameter did not work in imptcp +- bugfix: modules not (yet) supporting new conf format were not properly + registered. This lead to a "module not found" error message instead of + the to-be-expected "module does not support new style" error message. + That invalid error message could be quite misleading and actually stop + people from addressing the real problem (aka "go nuts" ;)) +- bugfix: template "type" parameter is mandatory (but was not) +- bugfix: some message properties could be garbled due to race condition + This happened only on very high volume systems, if the same message was + being processed by two different actions. This was a regression caused + by the new config processor, which did no longer properly enable msg + locking in multithreaded cases. The bugfix is actually a refactoring of + the msg locking code - we no longer do unlocked operations, as the use + case for it has mostly gone away. It is potentially possible only at + very low-end systems, and there the small additional overhead of doing + the locking does not really hurt. Instead, the removal of that + capability can actually slightly improve performance in common cases, + as the code path is smaller and requires slightly less memory writes. + That probably outperforms the extra locking overhead (which in the + low-end case always happens in user space, without need for kernel + support as we can always directly aquire the lock - there is no + contention at all). +---------------------------------------------------------------------------- +Version 7.2.3 [v7-stable] 2012-10-21 +- regression fix: rsyslogd terminated when wild-card $IncludeConfig did not + find actual include files. For example, if this directive is present: + $IncludeConfig /etc/rsyslog.d/*.conf + and there are no *.conf files in /etc/rsyslog.d (but rsyslog.d exists), + rsyslogd will emit an error message and terminate. Previous (and expected) + behaviour is that an empty file set is no problem. HOWEVER, if the + directory itself does not exist, this is flagged as an error and will + load to termination (no startup). + Unfortunately, this is often the case by default in many distros, so this + actually prevents rsyslog startup. +---------------------------------------------------------------------------- +Version 7.2.2 [v7-stable] 2012-10-16 +- doc improvements +- enabled to build without libuuid, at loss of uuid functionality + this enables smoother builds on older systems that do not support + libuuid. Loss of functionality should usually not matter too much as + uuid support has only recently been added and is very seldom used. +- bugfix: omfwd did not properly support "template" parameter +- bugfix: potential segfault when re_match() function was used + Thanks to oxpa for the patch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=371 +- bugfix: potential abort of imtcp on rsyslogd shutdown +- bugfix: imzmq3 segfault with PULL subscription + Thanks to Martin Nilsson for the patch. +- bugfix: improper handling of backslash in string-type template()s +- bugfix: leading quote (") in string-type template() lead to thight loop + on startup +- bugfix: no error msg on invalid field option in legacy/string template +- bugfix: potential segfault due to invalid param handling in comparisons + This could happen in RainerScript comparisons (like contains); in some + cases an unitialized variable was accessed, which could lead to an + invalid free and in turn to a segfault. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=372 + Thanks to Georgi Georgiev for reporting this bug and his great help + in solving it. +- bugfix: no error msg on unreadable $IncludeConfig path +- bugfix: $IncludeConfig did not correctly process directories + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=376 + The testbench was also enhanced to check for these cases. + Thanks to Georgi Georgiev for the bug report. +- bugfix: make rsyslog compile on kfreebsd again + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=380 + Thanks to Guillem Jover for the patch. +- bugfix: garbled message if field name was used with jsonf property option + The length for the field name was invalidly computed, resulting in either + truncated field names or including extra random data. If the random data + contained NULs, the rest of the message became unreadable. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=374 +- bugfix: potential segfault at startup with property-based filter + If the property name was followed by a space before the comma, rsyslogd + aborted on startup. Note that no segfault could happen if the initial + startup went well (this was a problem with the config parser). + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=381 +- bugfix: imfile discarded some file parts + File lines that were incomplete (LF missing) *at the time imfile polled + the file* were partially discarded. That part of the line that was read + without the LF was discarded, and the rest of the line was submitted in + the next polling cycle. This is now changed so that the partial content + is saved until the complete line is read. Note that the patch affects + only read mode 0. + Thanks to Milan Bartos for providing the base idea for the solution. +---------------------------------------------------------------------------- +Version 7.2.1 [v7-stable] 2012-10-29 +- bugfix: ruleset()-object did only support a single statement +- added -D rsyslogd option to enable config parser debug mode +- improved syntax error messages by outputting the error token +- the rsyslog core now suspeneds actions after 10 failures in a row + This was former the case after 1,000 failures and could cause rsyslog + to be spammed/ressources misused. See the v6 compatibility doc for more + details. +- ommongodb rate-limits error messages to prevent spamming the syslog + closes (for v7.2): http://bugzilla.adiscon.com/show_bug.cgi?id=366 +---------------------------------------------------------------------------- +Version 7.2.0 [v7-stable] 2012-10-22 +This starts a new stable branch based on 7.1.12 plus the following changes: +- bugfix: imuxsock did not properly honor $LocalHostIPIF +- omruleset/omdiscard do no longer issue "deprecated" warings, as 7.1 + grammar does not permit to use the replacements under all circumstances +---------------------------------------------------------------------------- +Version 7.1.12 [beta] 2012-10-18 +- minor updates to better support newer systemd developments + Thanks to Michael Biebl for the patches. +- build system cleanup + Thanks to Michael Biebl for the patch series. +- cleanup: removed remains of -c option (compatibility mode) + both from code & doc and emitted warning message if still used + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=361 + Thanks to Michael Biebl for reporting & suggestions +- bugfix: imklog truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski and Florian Piekert for their help in solving this issue. +---------------------------------------------------------------------------- +Version 7.1.11 [beta] 2012-10-16 +- bugfix: imuxsock truncated head of received message + This happened only under some circumstances. Thanks to Marius + Tomaschewski, Florian Piekert and Milan Bartos for their help in + solving this issue. +- bugfix: do not crash if set statement is used with date field + Thanks to Miloslav Trmač for the patch. +- change lumberjack cookie to "@cee:" from "@cee: " + CEE originally specified the cookie with SP, whereas other lumberjack + tools used it without space. In order to keep interop with lumberjack, + we now use the cookie without space as well. I hope this can be changed + in CEE as well when it is released at a later time. + Thanks to Miloslav Trmač for pointing this out and a similiar v7 patch. +- added deprecated note to omruleset (plus clue to use "call") +- added deprecated note to discard action (plus clue to use "stop") +--------------------------------------------------------------------------- +Version 7.1.10 [beta] 2012-10-11 + - bugfix: m4 directory was not present in release tarball + - bugfix: small memory leak with string-type templates + - bugfix: small memory leak when template was specified in omfile + - bugfix: some config processing warning messages were treated as errors + - bugfix: small memory leak when processing action() statements + - bugfix: unknown action() parameters were not reported +--------------------------------------------------------------------------- +Version 7.1.9 [beta] 2012-10-09 +- bugfix: comments inside objects (e.g. action()) were not properly handled +- bugfix: in (non)equal comparisons the position of arrays influenced result + This behaviour is OK for "contains"-type of comparisons (which have quite + different semantics), but not for == and <>, which shall be commutative. + This has been fixed now, so there is no difference any longer if the + constant string array is the left or right hand operand. We solved this + via the optimizer, as it keeps the actual script execution code small. +--------------------------------------------------------------------------- +Version 7.1.8 [beta] 2012-10-02 +- bugfix: ruleset(){} directive errornously changed default ruleset + much like the $ruleset legacy conf statement. This potentially lead + to statements being assigned to the wrong ruleset. +- improved module doc +- added "parser" parameter to ruleset(), so that parser chain can be + configured +- implemented "continue" RainerScript statement +--------------------------------------------------------------------------- +Version 7.1.7 [devel] 2012-10-01 +- implemented RainerScript "call" statement +- implemented RainerScript array-based string comparison operations +- implemented imtcp "permittedPeers" module-global parameter +- imudp: support for specifying multiple ports via array added +--------------------------------------------------------------------------- +Version 7.1.6 [devel] 2012-09-28 +- implemented RainerScript input() statement, including support for it + in major input plugins +- implemented RainerScript ruleset() statement +--------------------------------------------------------------------------- +Version 7.1.5 [devel] 2012-09-25 +- implemented RainerScript prifield() function +- implemented RainerScript field() function +- added new module imkmsg to process structured kernel log + Thanks to Milan Bartos for contributing this module +- implemented basic RainerScript optimizer, which will speed up script + operations +- bugfix: invalid free if function re_match() was incorrectly used + if the config file parser detected that param 2 was not constant, some + data fields were not initialized. The destructor did not care about that. + This bug happened only if rsyslog startup was unclean. +--------------------------------------------------------------------------- +Version 7.1.4 [devel] 2012-09-19 +- implemented ability for CEE-based properties to be stored in disk queues +- implemented string concatenation in expressions via &-operator +- implemented json subtree copy in variable assignment +- implemented full JSON support for variable manipulation +- introduced "subtree"-type templates +- bugfix: omfile action did not respect "template" parameter + ... and used default template in all cases +- bugfix: MsgDup() did not copy CEE structure + This function was called at various places, most importantly during + "last messages repeated n times" processing and omruleset. If CEE(JSON) + data was present, it was lost as part of the copy process. +- bugfix: debug output indicated improper queue type +--------------------------------------------------------------------------- +Version 7.1.3 [devel] 2012-09-17 +- introduced "set" and "unset" config statements +- bugfix: missing support for escape sequences in RainerScript + only \' was supported. Now the usual set is supported. Note that v5 + used \x as escape where x was any character (e.g. "\n" meant "n" and NOT + LF). This also means there is some incompatibility to v5 for well-know + sequences. Better break it now than later. +- bugfix: invalid property name in property-filter could cause abort + if action chaining (& operator) was used + http://bugzilla.adiscon.com/show_bug.cgi?id=355 + Thanks to pilou@gmx.com for the bug report +--------------------------------------------------------------------------- +Version 7.1.2 [devel] 2012-09-12 +- bugfix: messages were duplicated, sometimes massively + regression from new code in 7.1.1 and reason for early release +- bugfix: remove invalid socket option call from imuxsock + Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom +- bugfix: abort when invalid property name was configured + in property-based filter +- bugfix: multiple rulesets did no longer work correctly (7.1.1 regression) +--------------------------------------------------------------------------- +Version 7.1.1 [devel] 2012-09-11 +- MAJOR NEW FEATURE: rulengine now fully supports nesting + including if ... then ... else ... constructs. This is a big change + and it obviously has a lot of bug potential. +- BSD-style (filter) blocks are no longer supported + see http://www.rsyslog.com/g/BSD for details and solution +- imuxsock now stores trusted properties by default in the CEE root + This was done in order to keep compatible with other implementations of + the lumberjack schema + Thanks to Miloslav Trmač for pointing to this. +- bugfix: string-generating templates caused abort if CEE field could not + be found +--------------------------------------------------------------------------- +Version 7.1.0 [devel] 2012-09-06 +- added support for hierarchical properties (CEE/lumberjack) +- added pure JSON output plugin parameter passing mode +- ommongodb now supports templates +- bugfix: imtcp could abort on exit due to invalid free() +- imported bugfixes from 6.4.1 +--------------------------------------------------------------------------- Version 6.6.1 [v6-stable] 2012-10-?? - bugfix: build problems on some platforms - bugfix: misaddressing of $mmnormalizeuserawmsg parameter @@ -66,13 +844,9 @@ Version 6.5.1 [beta] 2012-10-11 - bugfix: imtcp could abort on exit due to invalid free() - bugfix: remove invalid socket option call from imuxsock Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom -- bugfix: missing support for escape sequences in RainerScript - only \' was supported. Now the usual set is supported. Note that v5 - used \x as escape where x was any character (e.g. "\n" meant "n" and NOT - LF). This also means there is some incompatibility to v5 for well-know - sequences. Better break it now than later. -- bugfix: remove invalid socket option call from imuxsock - Thanks to Cristian Ionescu-Idbohrn and Jonny Törnbom +- added pure JSON output plugin parameter passing mode +- ommongodb now supports templates +- bugfix: imtcp could abort on exit due to invalid free() - bugfix: missing support for escape sequences in RainerScript only \' was supported. Now the usual set is supported. Note that v5 used \x as escape where x was any character (e.g. "\n" meant "n" and NOT @@ -738,7 +1512,7 @@ Version 5.10.2 [V5-STABLE], 201?-??-?? Thanks to Tomas Heinrich for the patch. - bugfix[kind of]: omudpspoof discarded messages >1472 bytes (MTU size) it now truncates these message, but ensures they are sent. Note that - 7.2.5+ will switch to fragmented UDP messages instead (up to 64K) + 7.3.5+ will switch to fragmented UDP messages instead (up to 64K) --------------------------------------------------------------------------- Version 5.10.1 [V5-STABLE], 2012-10-17 - bugfix: imuxsock and imklog truncated head of received message diff --git a/Makefile.am b/Makefile.am index 98708c80..567b8769 100644 --- a/Makefile.am +++ b/Makefile.am @@ -58,10 +58,11 @@ CLEANFILES = \ endif EXTRA_DIST = \ - freebsd/rsyslogd \ - slackware/rc.rsyslogd \ + platform/README \ + platform/freebsd/rsyslogd \ + platform/slackware/rc.rsyslogd \ + platform/redhat/rsyslog.conf \ contrib/README \ - rsyslog.conf \ COPYING \ COPYING.LESSER \ COPYING.ASL20 \ @@ -70,7 +71,7 @@ EXTRA_DIST = \ contrib/gnutls/key.pem \ rsyslog.service.in -SUBDIRS = doc runtime grammar . plugins/immark plugins/imuxsock plugins/imtcp plugins/imudp plugins/omtesting +SUBDIRS = doc compat runtime grammar . plugins/immark plugins/imuxsock plugins/imtcp plugins/imudp plugins/omtesting if ENABLE_RSYSLOGD SUBDIRS += tools @@ -80,6 +81,10 @@ if ENABLE_IMKLOG SUBDIRS += plugins/imklog endif +if ENABLE_IMKMSG +SUBDIRS += plugins/imkmsg +endif + if ENABLE_IMPSTATS SUBDIRS += plugins/impstats endif @@ -160,6 +165,10 @@ if ENABLE_OMZMQ3 SUBDIRS += plugins/omzmq3 endif +if ENABLE_OMRABBITMQ +SUBDIRS += plugins/omrabbitmq +endif + if ENABLE_IMZMQ3 SUBDIRS += plugins/imzmq3 endif @@ -172,6 +181,14 @@ if ENABLE_OMHDFS SUBDIRS += plugins/omhdfs endif +if ENABLE_OMJOURNAL +SUBDIRS += plugins/omjournal +endif + +if ENABLE_IMJOURNAL +SUBDIRS += plugins/imjournal +endif + if ENABLE_ELASTICSEARCH SUBDIRS += plugins/omelasticsearch endif @@ -220,6 +237,10 @@ if ENABLE_MMAUDIT SUBDIRS += plugins/mmaudit endif +if ENABLE_MMANON +SUBDIRS += plugins/mmanon +endif + if ENABLE_ORACLE SUBDIRS += plugins/omoracle endif @@ -240,7 +261,6 @@ SUBDIRS += tests # in a make distcheck is so that we detect code that accidently was not updated # when some global update happened. DISTCHECK_CONFIGURE_FLAGS= --enable-gssapi_krb5 \ - --enable-gnutls \ --enable-imfile \ --enable-snmp \ --enable-libdbi \ @@ -266,6 +286,7 @@ DISTCHECK_CONFIGURE_FLAGS= --enable-gssapi_krb5 \ --enable-pmsnare \ --enable-mmsnmptrapd \ --enable-elasticsearch \ + --enable-valgrind \ --with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir) # temporarily disable these checks for make distcheck 2012-09-06 rgerhards # --enable-extended-tests \ @@ -12,11 +12,11 @@ * necessary to triple-check that everything works well in *all* modes. * The different modes (and calling sequence) are: * - * if set iExecEveryNthOccur > 1 || f_ReduceRepeated || iSecsExecOnceInterval + * if set iExecEveryNthOccur > 1 || iSecsExecOnceInterval * - doSubmitToActionQComplexBatch * - helperSubmitToActionQComplexBatch * - doActionCallAction - * handles duplicate message processing, but in essence calls + * handles mark message reduction, but in essence calls * - actionWriteToAction * - qqueueEnqObj * (now queue engine processing) @@ -188,7 +188,7 @@ static struct cnfparamdescr cnfparamdescr[] = { { "action.execonlyeverynthtime", eCmdHdlrInt, 0 }, /* legacy: actionexeconlyeverynthtime */ { "action.execonlyeverynthtimetimeout", eCmdHdlrInt, 0 }, /* legacy: actionexeconlyeverynthtimetimeout */ { "action.execonlyonceeveryinterval", eCmdHdlrInt, 0 }, /* legacy: actionexeconlyonceeveryinterval */ - { "action.execonlywhenpreviousissuspended", eCmdHdlrInt, 0 }, /* legacy: actionexeconlywhenpreviousissuspended */ + { "action.execonlywhenpreviousissuspended", eCmdHdlrBinary, 0 }, /* legacy: actionexeconlywhenpreviousissuspended */ { "action.repeatedmsgcontainsoriginalmsg", eCmdHdlrBinary, 0 }, /* legacy: repeatedmsgcontainsoriginalmsg */ { "action.resumeretrycount", eCmdHdlrInt, 0 }, /* legacy: actionresumeretrycount */ { "action.resumeinterval", eCmdHdlrInt, 0 } @@ -289,6 +289,11 @@ rsRetVal actionDestruct(action_t *pThis) DEFiRet; ASSERT(pThis != NULL); + if(!strcmp((char*)modGetName(pThis->pMod), "builtin:omdiscard")) { + /* discard actions will be optimized out */ + FINALIZE; + } + if(pThis->pQueue != NULL) { qqueueDestruct(&pThis->pQueue); } @@ -302,16 +307,13 @@ rsRetVal actionDestruct(action_t *pThis) if(pThis->pMod != NULL) pThis->pMod->freeInstance(pThis->pModData); - if(pThis->f_pMsg != NULL) - msgDestruct(&pThis->f_pMsg); - pthread_mutex_destroy(&pThis->mutAction); pthread_mutex_destroy(&pThis->mutActExec); d_free(pThis->pszName); d_free(pThis->ppTpl); +finalize_it: d_free(pThis); - RETiRet; } @@ -362,6 +364,10 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) ASSERT(pThis != NULL); + if(!strcmp((char*)modGetName(pThis->pMod), "builtin:omdiscard")) { + /* discard actions will be optimized out */ + FINALIZE; + } /* generate a friendly name for us action stats */ if(pThis->pszName == NULL) { snprintf((char*) pszAName, sizeof(pszAName)/sizeof(uchar), "action %d", iActionNbr); @@ -401,16 +407,11 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) * mode is much faster processing (and simpler code) -- rgerhards, 2010-06-08 */ if( pThis->iExecEveryNthOccur > 1 - || pThis->f_ReduceRepeated || pThis->iSecsExecOnceInterval ) { DBGPRINTF("info: firehose mode disabled for action because " - "iExecEveryNthOccur=%d, " - "ReduceRepeated=%d, " - "iSecsExecOnceInterval=%d\n", - pThis->iExecEveryNthOccur, pThis->f_ReduceRepeated, - pThis->iSecsExecOnceInterval - ); + "iExecEveryNthOccur=%d, iSecsExecOnceInterval=%d\n", + pThis->iExecEveryNthOccur, pThis->iSecsExecOnceInterval); pThis->submitToActQ = doSubmitToActionQComplexBatch; } else if(pThis->bWriteAllMarkMsgs == RSFALSE) { /* nearly full-speed submission mode, default case */ @@ -429,7 +430,7 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) CHKiRet(qqueueConstruct(&pThis->pQueue, cs.ActionQueType, 1, cs.iActionQueueSize, (rsRetVal (*)(void*, batch_t*, int*))processBatchMain)); obj.SetName((obj_t*) pThis->pQueue, pszAName); - qqueueSetpUsr(pThis->pQueue, pThis); + qqueueSetpAction(pThis->pQueue, pThis); if(queueParams == NULL) { /* use legacy params? */ /* ... set some properties ... */ @@ -604,13 +605,17 @@ static void actionDisable(action_t *pThis) * CPU time. TODO: maybe a config option for that? * rgerhards, 2007-08-02 */ -static inline void actionSuspend(action_t *pThis, time_t ttNow) +static inline void actionSuspend(action_t *pThis) { - if(ttNow == NO_TIME_PROVIDED) - datetime.GetTime(&ttNow); + time_t ttNow; + + /* note: we can NOT use a cached timestamp, as time may have evolved + * since caching, and this would break logic (and it actually did so!) + */ + datetime.GetTime(&ttNow); pThis->ttResumeRtry = ttNow + pThis->iResumeInterval * (pThis->iNbrResRtry / 10 + 1); actionSetState(pThis, ACT_STATE_SUSP); - DBGPRINTF("earliest retry=%d\n", (int) pThis->ttResumeRtry); + DBGPRINTF("action suspended, earliest retry=%d\n", (int) pThis->ttResumeRtry); } @@ -630,7 +635,7 @@ static inline void actionSuspend(action_t *pThis, time_t ttNow) * of its inability to recover. -- rgerhards, 2010-04-26. */ static inline rsRetVal -actionDoRetry(action_t *pThis, time_t ttNow, int *pbShutdownImmediate) +actionDoRetry(action_t *pThis, int *pbShutdownImmediate) { int iRetries; int iSleepPeriod; @@ -641,24 +646,28 @@ actionDoRetry(action_t *pThis, time_t ttNow, int *pbShutdownImmediate) iRetries = 0; while((*pbShutdownImmediate == 0) && pThis->eState == ACT_STATE_RTRY) { + DBGPRINTF("actionDoRetry: enter loop, iRetries=%d\n", iRetries); iRet = pThis->pMod->tryResume(pThis->pModData); - if((pThis->iResumeOKinRow > 999) && (pThis->iResumeOKinRow % 1000 == 0)) { + DBGPRINTF("actionDoRetry: action->tryResume returned %d\n", iRet); + if((pThis->iResumeOKinRow > 9) && (pThis->iResumeOKinRow % 10 == 0)) { bTreatOKasSusp = 1; pThis->iResumeOKinRow = 0; } else { bTreatOKasSusp = 0; } if((iRet == RS_RET_OK) && (!bTreatOKasSusp)) { + DBGPRINTF("actionDoRetry: had success RDY again (iRet=%d)\n", iRet); actionSetState(pThis, ACT_STATE_RDY); } else if(iRet == RS_RET_SUSPENDED || bTreatOKasSusp) { /* max retries reached? */ + DBGPRINTF("actionDoRetry: check for max retries, iResumeRetryCount %d, iRetries %d\n", + pThis->iResumeRetryCount, iRetries); if((pThis->iResumeRetryCount != -1 && iRetries >= pThis->iResumeRetryCount)) { - actionSuspend(pThis, ttNow); + actionSuspend(pThis); } else { ++pThis->iNbrResRtry; ++iRetries; iSleepPeriod = pThis->iResumeInterval; - ttNow += iSleepPeriod; /* not truly exact, but sufficiently... */ srSleep(iSleepPeriod, 0); if(*pbShutdownImmediate) { ABORT_FINALIZE(RS_RET_FORCE_TERM); @@ -705,7 +714,7 @@ static rsRetVal actionTryResume(action_t *pThis, int *pbShutdownImmediate) if(pThis->eState == ACT_STATE_RTRY) { if(ttNow == NO_TIME_PROVIDED) /* use cached result if we have it */ datetime.GetTime(&ttNow); - CHKiRet(actionDoRetry(pThis, ttNow, pbShutdownImmediate)); + CHKiRet(actionDoRetry(pThis, pbShutdownImmediate)); } if(Debug && (pThis->eState == ACT_STATE_RTRY ||pThis->eState == ACT_STATE_SUSP)) { @@ -765,7 +774,6 @@ rsRetVal actionDbgPrint(action_t *pThis) pThis->pMod->dbgPrintInstInfo(pThis->pModData); dbgprintf("\n"); dbgprintf("\tInstance data: 0x%lx\n", (unsigned long) pThis->pModData); - dbgprintf("\tRepeatedMsgReduction: %d\n", pThis->f_ReduceRepeated); dbgprintf("\tResume Interval: %d\n", pThis->iResumeInterval); if(pThis->eState == ACT_STATE_SUSP) { dbgprintf("\tresume next retry: %u, number retries: %d", @@ -792,7 +800,8 @@ rsRetVal actionDbgPrint(action_t *pThis) /* prepare the calling parameters for doAction() * rgerhards, 2009-05-07 */ -static rsRetVal prepareDoActionParams(action_t *pAction, batch_obj_t *pElem) +static rsRetVal +prepareDoActionParams(action_t *pAction, batch_obj_t *pElem, struct syslogTime *ttNow) { int i; msg_t *pMsg; @@ -802,23 +811,23 @@ static rsRetVal prepareDoActionParams(action_t *pAction, batch_obj_t *pElem) ASSERT(pAction != NULL); ASSERT(pElem != NULL); - pMsg = (msg_t*) pElem->pUsrp; + pMsg = pElem->pMsg; /* here we must loop to process all requested strings */ for(i = 0 ; i < pAction->iNumTpls ; ++i) { switch(pAction->eParamPassing) { case ACT_STRING_PASSING: CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(pElem->staticActStrings[i]), - &pElem->staticLenStrings[i])); + &pElem->staticLenStrings[i], ttNow)); pElem->staticActParams[i] = pElem->staticActStrings[i]; break; case ACT_ARRAY_PASSING: - CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pElem->staticActParams[i]))); + CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pElem->staticActParams[i]), ttNow)); break; case ACT_MSG_PASSING: pElem->staticActParams[i] = (void*) pMsg; break; case ACT_JSON_PASSING: - CHKiRet(tplToJSON(pAction->ppTpl[i], pMsg, &json)); + CHKiRet(tplToJSON(pAction->ppTpl[i], pMsg, &json, ttNow)); pElem->staticActParams[i] = (void*) json; break; default:dbgprintf("software bug/error: unknown pAction->eParamPassing %d in prepareDoActionParams\n", @@ -851,9 +860,12 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) ASSERT(pAction != NULL); + if(pAction->eParamPassing == ACT_STRING_PASSING || pAction->eParamPassing == ACT_MSG_PASSING) + goto done; /* we need to do nothing with these types! */ + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { pElem = &(pBatch->pElem[i]); - if(pElem->bFilterOK && pElem->state != BATCH_STATE_DISC) { + if(batchIsValidElem(pBatch, i)) { switch(pAction->eParamPassing) { case ACT_ARRAY_PASSING: ppMsgs = (uchar***) pElem->staticActParams; @@ -870,19 +882,6 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) } } break; - case ACT_STRING_PASSING: - case ACT_MSG_PASSING: - /* nothing to do in that case */ - /* TODO ... and yet we do something ;) This is considered not - * really needed, but I was not bold enough to remove that while - * fixing the stable. It should be removed in a devel version - * soon (I really don't see a reason why we would need it). - * rgerhards, 2010-12-16 - */ - for(j = 0 ; j < pAction->iNumTpls ; ++j) { - ((uchar**)pElem->staticActParams)[j] = NULL; - } - break; case ACT_JSON_PASSING: for(j = 0 ; j < pAction->iNumTpls ; ++j) { json_object_put((struct json_object*) @@ -890,11 +889,15 @@ static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) pElem->staticActParams[j] = NULL; } break; + case ACT_STRING_PASSING: + case ACT_MSG_PASSING: + /* can never happen, just to keep compiler happy! */ + break; } } } - RETiRet; +done: RETiRet; } @@ -958,6 +961,8 @@ actionProcessMessage(action_t *pThis, msg_t *pMsg, void *actParams, int *pbShutd ISOBJ_TYPE_assert(pMsg, msg); CHKiRet(actionPrepare(pThis, pbShutdownImmediate)); + if(pThis->pMod->mod.om.SetShutdownImmdtPtr != NULL) + pThis->pMod->mod.om.SetShutdownImmdtPtr(pThis->pModData, pbShutdownImmediate); if(pThis->eState == ACT_STATE_ITX) CHKiRet(actionCallDoAction(pThis, pMsg, actParams)); @@ -1051,9 +1056,8 @@ tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) /* NOTE: do NOT extend the filter below! Anything else must be done on the * enq side of the queue (see file header comment)! -- rgerhards, 2011-06-15 */ - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC) { - pMsg = (msg_t*) pBatch->pElem[i].pUsrp; + if(batchIsValidElem(pBatch, i)) { + pMsg = pBatch->pElem[i].pMsg; localRet = actionProcessMessage(pAction, pMsg, pBatch->pElem[i].staticActParams, pBatch->pbShutdownImmediate); DBGPRINTF("action %p call returned %d\n", pAction, localRet); @@ -1076,11 +1080,11 @@ tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) ++iCommittedUpTo; //pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; } - pBatch->pElem[i].state = BATCH_STATE_SUB; + pBatch->eltState[i] = BATCH_STATE_SUB; } else if(localRet == RS_RET_DEFER_COMMIT) { - pBatch->pElem[i].state = BATCH_STATE_SUB; + pBatch->eltState[i] = BATCH_STATE_SUB; } else if(localRet == RS_RET_DISCARDMSG) { - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } else { dbgprintf("tryDoAction: unexpected error code %d[nElem %d, Commited UpTo %d], finalizing\n", localRet, *pnElem, iCommittedUpTo); @@ -1117,6 +1121,7 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) assert(pBatch != NULL); + DBGPRINTF("submitBatch: enter, nElem %d\n", nElem); wasDoneTo = pBatch->iDoneUpTo; bDone = 0; do { @@ -1138,13 +1143,14 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) || localRet == RS_RET_DEFER_COMMIT) { bDone = 1; } else if(localRet == RS_RET_SUSPENDED) { - ; /* do nothing, this will retry the full batch */ + DBGPRINTF("action ret RS_RET_SUSPENDED - retry full batch\n"); + /* do nothing, this will retry the full batch */ } else if(localRet == RS_RET_ACTION_FAILED) { /* in this case, everything not yet committed is BAD */ for(i = pBatch->iDoneUpTo ; i < wasDoneTo + nElem ; ++i) { - if( pBatch->pElem[i].state != BATCH_STATE_DISC - && pBatch->pElem[i].state != BATCH_STATE_COMM ) { - pBatch->pElem[i].state = BATCH_STATE_BAD; + if( pBatch->eltState[i] != BATCH_STATE_DISC + && pBatch->eltState[i] != BATCH_STATE_COMM ) { + pBatch->eltState[i] = BATCH_STATE_BAD; pBatch->pElem[i].bPrevWasSuspended = 1; STATSCOUNTER_INC(pAction->ctrFail, pAction->mutCtrFail); } @@ -1173,6 +1179,29 @@ finalize_it: } +/* copy "active" array of batch, as we need to modify it. The caller + * must make sure the new array is freed and the orginal batch + * pointer is restored (thus the caller must save it). If active + * is currently NULL, this is properly handled. + * Note: the batches active pointer is modified, so it must be + * saved BEFORE calling this function! + * rgerhards, 2012-09-12 + */ +static rsRetVal +copyActive(batch_t *pBatch) +{ + sbool *active; + DEFiRet; + + CHKmalloc(active = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); + if(pBatch->active == NULL) + memset(active, 1, batchNumMsgs(pBatch)); + else + memcpy(active, pBatch->active, batchNumMsgs(pBatch)); + pBatch->active = active; +finalize_it: + RETiRet; +} /* The following function prepares a batch for processing, that it is * reinitializes batch states, generates strings and does everything else @@ -1183,19 +1212,29 @@ finalize_it: * rgerhards, 2010-06-14 */ static inline rsRetVal -prepareBatch(action_t *pAction, batch_t *pBatch) +prepareBatch(action_t *pAction, batch_t *pBatch, sbool **activeSave, int *bMustRestoreActivePtr) { int i; batch_obj_t *pElem; + struct syslogTime ttNow; DEFiRet; + /* indicate we have not yet read the date */ + ttNow.year = 0; + pBatch->iDoneUpTo = 0; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { pElem = &(pBatch->pElem[i]); - if(pElem->bFilterOK && pElem->state != BATCH_STATE_DISC) { - pElem->state = BATCH_STATE_RDY; - if(prepareDoActionParams(pAction, pElem) != RS_RET_OK) - pElem->bFilterOK = RSFALSE; + if(batchIsValidElem(pBatch, i)) { + pBatch->eltState[i] = BATCH_STATE_RDY; + if(prepareDoActionParams(pAction, pElem, &ttNow) != RS_RET_OK) { + /* make sure we have our copy of "active" array */ + if(!*bMustRestoreActivePtr) { + *activeSave = pBatch->active; + copyActive(pBatch); + } + pBatch->active[i] = RSFALSE; + } } } RETiRet; @@ -1228,14 +1267,18 @@ static rsRetVal processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) { int *pbShutdownImmdtSave; + sbool *activeSave; + int bMustRestoreActivePtr = 0; rsRetVal localRet; DEFiRet; assert(pBatch != NULL); - pbShutdownImmdtSave = pBatch->pbShutdownImmediate; - pBatch->pbShutdownImmediate = pbShutdownImmediate; - CHKiRet(prepareBatch(pAction, pBatch)); + if(pbShutdownImmediate != NULL) { + pbShutdownImmdtSave = pBatch->pbShutdownImmediate; + pBatch->pbShutdownImmediate = pbShutdownImmediate; + } + CHKiRet(prepareBatch(pAction, pBatch, &activeSave, &bMustRestoreActivePtr)); /* We now must guard the output module against execution by multiple threads. The * plugin interface specifies that output modules must not be thread-safe (except @@ -1258,9 +1301,15 @@ processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) if(iRet == RS_RET_OK) iRet = localRet; + + if(bMustRestoreActivePtr) { + free(pBatch->active); + pBatch->active = activeSave; + } finalize_it: - pBatch->pbShutdownImmediate = pbShutdownImmdtSave; + if(pbShutdownImmediate != NULL) + pBatch->pbShutdownImmediate = pbShutdownImmdtSave; RETiRet; } #pragma GCC diagnostic warning "-Wempty-body" @@ -1343,9 +1392,9 @@ doSubmitToActionQ(action_t *pAction, msg_t *pMsg) STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); if(pAction->pQueue->qType == QUEUETYPE_DIRECT) - iRet = qqueueEnqObjDirect(pAction->pQueue, (void*) MsgAddRef(pMsg)); + iRet = qqueueEnqMsgDirect(pAction->pQueue, MsgAddRef(pMsg)); else - iRet = qqueueEnqObj(pAction->pQueue, eFLOWCTL_NO_DELAY, (void*) MsgAddRef(pMsg)); + iRet = qqueueEnqMsg(pAction->pQueue, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); finalize_it: RETiRet; @@ -1360,14 +1409,10 @@ finalize_it: * be filtered out before calling us (what is done currently!). */ rsRetVal -actionWriteToAction(action_t *pAction) +actionWriteToAction(action_t *pAction, msg_t *pMsg) { - msg_t *pMsgSave; /* to save current message pointer, necessary to restore - it in case it needs to be updated (e.g. repeated msgs) */ DEFiRet; - pMsgSave = NULL; /* indicate message poiner not saved */ - /* first, we check if the action should actually be called. The action-specific * $ActionExecOnlyEveryNthTime permits us to execute an action only every Nth * time. So we need to check if we need to drop the (otherwise perfectly executable) @@ -1394,43 +1439,6 @@ actionWriteToAction(action_t *pAction) } } - /* then check if this is a regular message or the repeation of - * a previous message. If so, we need to change the message text - * to "last message repeated n times" and then go ahead and write - * it. Please note that we can not modify the message object, because - * that would update it in other selectors as well. As such, we first - * need to create a local copy of the message, which we than can update. - * rgerhards, 2007-07-10 - */ - if(pAction->f_prevcount > 1) { - msg_t *pMsg; - size_t lenRepMsg; - uchar szRepMsg[1024]; - - if((pMsg = MsgDup(pAction->f_pMsg)) == NULL) { - /* it failed - nothing we can do against it... */ - DBGPRINTF("Message duplication failed, dropping repeat message.\n"); - ABORT_FINALIZE(RS_RET_ERR); - } - - if(pAction->bRepMsgHasMsg == 0) { /* old format repeat message? */ - lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " last message repeated %d times", - pAction->f_prevcount); - } else { - lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " message repeated %d times: [%.800s]", - pAction->f_prevcount, getMSG(pAction->f_pMsg)); - } - - /* We now need to update the other message properties. Please note that digital - * signatures inside the message are also invalidated. - */ - datetime.getCurrTime(&(pMsg->tRcvdAt), &(pMsg->ttGenTime)); - memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); - MsgReplaceMSG(pMsg, szRepMsg, lenRepMsg); - pMsgSave = pAction->f_pMsg; /* save message pointer for later restoration */ - pAction->f_pMsg = pMsg; /* use the new msg (pointer will be restored below) */ - } - DBGPRINTF("Called action(complex case), logging to %s\n", module.GetStateName(pAction->pMod)); /* now check if we need to drop the message because otherwise the action would be too @@ -1451,31 +1459,14 @@ actionWriteToAction(action_t *pAction) /* we use reception time, not dequeue time - this is considered more appropriate and also faster ;) * rgerhards, 2008-09-17 */ pAction->tLastExec = getActNow(pAction); /* re-init time flags */ - pAction->f_time = pAction->f_pMsg->ttGenTime; + pAction->f_time = pMsg->ttGenTime; /* When we reach this point, we have a valid, non-disabled action. * So let's enqueue our message for execution. -- rgerhards, 2007-07-24 */ - iRet = doSubmitToActionQ(pAction, pAction->f_pMsg); - - if(iRet == RS_RET_OK) - pAction->f_prevcount = 0; /* message processed, so we start a new cycle */ + iRet = doSubmitToActionQ(pAction, pMsg); finalize_it: - if(pMsgSave != NULL) { - /* we had saved the original message pointer. That was - * done because we needed to create a temporary one - * (most often for "message repeated n time" handling). If so, - * we need to restore the original one now, so that procesing - * can continue as normal. We also need to discard the temporary - * one, as we do not like memory leaks ;) Please note that the original - * message object will be discarded by our callers, so this is nothing - * of our business. rgerhards, 2007-07-10 - */ - msgDestruct(&pAction->f_pMsg); - pAction->f_pMsg = pMsgSave; /* restore it */ - } - RETiRet; } @@ -1489,7 +1480,7 @@ doActionCallAction(action_t *pAction, batch_t *pBatch, int idxBtch) msg_t *pMsg; DEFiRet; - pMsg = (msg_t*)(pBatch->pElem[idxBtch].pUsrp); + pMsg = pBatch->pElem[idxBtch].pMsg; pAction->tActNow = -1; /* we do not yet know our current time (clear prev. value) */ /* don't output marks to recently written outputs */ @@ -1498,43 +1489,8 @@ doActionCallAction(action_t *pAction, batch_t *pBatch, int idxBtch) ABORT_FINALIZE(RS_RET_OK); } - /* suppress duplicate messages */ - if ((pAction->f_ReduceRepeated == 1) && pAction->f_pMsg != NULL && - (pMsg->msgFlags & MARK) == 0 && getMSGLen(pMsg) == getMSGLen(pAction->f_pMsg) && - !ustrcmp(getMSG(pMsg), getMSG(pAction->f_pMsg)) && - !strcmp(getHOSTNAME(pMsg), getHOSTNAME(pAction->f_pMsg)) && - !strcmp(getPROCID(pMsg, LOCK_MUTEX), getPROCID(pAction->f_pMsg, LOCK_MUTEX)) && - !strcmp(getAPPNAME(pMsg, LOCK_MUTEX), getAPPNAME(pAction->f_pMsg, LOCK_MUTEX))) { - pAction->f_prevcount++; - DBGPRINTF("msg repeated %d times, %ld sec of %d.\n", - pAction->f_prevcount, (long) getActNow(pAction) - pAction->f_time, - repeatinterval[pAction->f_repeatcount]); - /* use current message, so we have the new timestamp (means we need to discard previous one) */ - msgDestruct(&pAction->f_pMsg); - pAction->f_pMsg = MsgAddRef(pMsg); - /* If domark would have logged this by now, flush it now (so we don't hold - * isolated messages), but back off so we'll flush less often in the future. - */ - if(getActNow(pAction) > REPEATTIME(pAction)) { - iRet = actionWriteToAction(pAction); - BACKOFF(pAction); - } - } else {/* new message, save it */ - /* first check if we have a previous message stored - * if so, emit and then discard it first - */ - if(pAction->f_pMsg != NULL) { - if(pAction->f_prevcount > 0) - actionWriteToAction(pAction); - /* we do not care about iRet above - I think it's right but if we have - * some troubles, you know where to look at ;) -- rgerhards, 2007-08-01 - */ - msgDestruct(&pAction->f_pMsg); - } - pAction->f_pMsg = MsgAddRef(pMsg); - /* call the output driver */ - iRet = actionWriteToAction(pAction); - } + /* call the output driver */ + iRet = actionWriteToAction(pAction, pMsg); finalize_it: /* we need to update the batch to handle failover processing correctly */ @@ -1564,7 +1520,8 @@ DEFFUNC_llExecFunc(doActivateActions) } actionDisable(pThis); } - DBGPRINTF("Action %p: queue %p started\n", pThis, pThis->pQueue); + DBGPRINTF("Action %s[%p]: queue %p started\n", modGetName(pThis->pMod), + pThis, pThis->pQueue); ENDfunc return RS_RET_OK; /* we ignore errors, we can not do anything either way */ } @@ -1598,22 +1555,15 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) time_t now = 0; time_t lastAct; int i; - int bModifiedFilter; - sbool FilterSave[1024]; - sbool *pFilterSave; + sbool *activeSave; DEFiRet; - if(batchNumMsgs(pBatch) <= (int) (sizeof(FilterSave)/sizeof(sbool))) { - pFilterSave = FilterSave; - } else { - CHKmalloc(pFilterSave = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); - } + activeSave = pBatch->active; + copyActive(pBatch); - bModifiedFilter = 0; for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(!pBatch->pElem[i].bFilterOK) + if((pBatch->eltState[i] == BATCH_STATE_DISC) || !pBatch->active[i]) continue; - pFilterSave[i] = pBatch->pElem[i].bFilterOK; if(now == 0) { now = datetime.GetTime(NULL); /* good time call - the only one done */ } @@ -1622,17 +1572,17 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) * also faster ;) -- rgerhards, 2008-09-17 */ do { lastAct = pAction->f_time; - if(((msg_t*)(pBatch->pElem[i].pUsrp))->msgFlags & MARK) { + if(pBatch->pElem[i].pMsg->msgFlags & MARK) { if((now - lastAct) < MarkInterval / 2) { - pBatch->pElem[i].bFilterOK = 0; - bModifiedFilter = 1; - DBGPRINTF("action was recently called, ignoring mark message\n"); + pBatch->active[i] = 0; + DBGPRINTF("batch item %d: action was recently called, ignoring " + "mark message\n", i); break; /* do not update timestamp for non-written mark messages */ } } } while(ATOMIC_CAS_time_t(&pAction->f_time, lastAct, - ((msg_t*)(pBatch->pElem[i].pUsrp))->ttGenTime, &pAction->mutCAS) == 0); - if(pBatch->pElem[i].bFilterOK) { + pBatch->pElem[i].pMsg->ttGenTime, &pAction->mutCAS) == 0); + if(pBatch->active[i]) { DBGPRINTF("Called action(NotAllMark), processing batch[%d] via '%s'\n", i, module.GetStateName(pAction->pMod)); } @@ -1640,17 +1590,8 @@ doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) iRet = doSubmitToActionQBatch(pAction, pBatch); - if(bModifiedFilter) { - /* in this case, we need to restore previous state */ - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - /* note: clang static code analyzer reports a false positive below */ - pBatch->pElem[i].bFilterOK = pFilterSave[i]; - } - } - -finalize_it: - if(pFilterSave != FilterSave) - free(pFilterSave); + free(pBatch->active); + pBatch->active = activeSave; RETiRet; } @@ -1660,8 +1601,7 @@ countStatsBatchEnq(action_t *pAction, batch_t *pBatch) { int i; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC) { + if( batchIsValidElem(pBatch, i)) { STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); } } @@ -1675,18 +1615,13 @@ countStatsBatchEnq(action_t *pAction, batch_t *pBatch) static inline rsRetVal doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) { - sbool FilterSave[1024]; - sbool *pFilterSave; sbool bNeedSubmit; - sbool bModifiedFilter; + sbool *activeSave; int i; DEFiRet; - if(batchNumMsgs(pBatch) <= (int) (sizeof(FilterSave)/sizeof(sbool))) { - pFilterSave = FilterSave; - } else { - CHKmalloc(pFilterSave = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); - } + activeSave = pBatch->active; + copyActive(pBatch); /* note: for direct mode, we need to adjust the filter property. For non-direct * this is not necessary, because in that case we enqueue only what actually needs @@ -1694,37 +1629,25 @@ doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) */ if(pAction->bExecWhenPrevSusp) { bNeedSubmit = 0; - bModifiedFilter = 0; for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - pFilterSave[i] = pBatch->pElem[i].bFilterOK; if(!pBatch->pElem[i].bPrevWasSuspended) { - DBGPRINTF("action enq stage: change bFilterOK to 0 due to " + DBGPRINTF("action enq stage: change active to 0 due to " "failover case in elem %d\n", i); - pBatch->pElem[i].bFilterOK = 0; - bModifiedFilter = 1; + pBatch->active[i] = 0; } - if(pBatch->pElem[i].bFilterOK && pBatch->pElem[i].state != BATCH_STATE_DISC) { + if(batchIsValidElem(pBatch, i)) { STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); bNeedSubmit = 1; } - DBGPRINTF("action %p[%d]: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, i, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p[%d]: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, i, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); } if(bNeedSubmit) { /* note: stats were already computed above */ iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); } else { - DBGPRINTF("no need to submit batch, all bFilterOK==0 or discarded\n"); - } - if(bModifiedFilter) { - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, - pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - /* note: clang static code analyzer reports a false positive below */ - pBatch->pElem[i].bFilterOK = pFilterSave[i]; - } + DBGPRINTF("no need to submit batch, all invalid\n"); } } else { if(GatherStats) @@ -1732,7 +1655,8 @@ doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); } -finalize_it: + free(pBatch->active); + pBatch->active = activeSave; RETiRet; } @@ -1755,13 +1679,12 @@ doSubmitToActionQBatch(action_t *pAction, batch_t *pBatch) * TODO: optimize this, we may do at least a multi-submit! */ for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC + if( batchIsValidElem(pBatch, i) && (pAction->bExecWhenPrevSusp == 0 || pBatch->pElem[i].bPrevWasSuspended == 1)) { - doSubmitToActionQ(pAction, (msg_t*)(pBatch->pElem[i].pUsrp)); + doSubmitToActionQ(pAction, pBatch->pElem[i].pMsg); } } } @@ -1784,11 +1707,10 @@ helperSubmitToActionQComplexBatch(action_t *pAction, batch_t *pBatch) DBGPRINTF("Called action %p (complex case), logging to %s\n", pAction, module.GetStateName(pAction->pMod)); for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: filterOK:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, pBatch->pElem[i].bFilterOK, pBatch->pElem[i].state, + DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", + pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( pBatch->pElem[i].bFilterOK - && pBatch->pElem[i].state != BATCH_STATE_DISC + if( batchIsValidElem(pBatch, i) && ((pAction->bExecWhenPrevSusp == 0) || pBatch->pElem[i].bPrevWasSuspended) ) { doActionCallAction(pAction, pBatch, i); } @@ -1858,7 +1780,6 @@ actionApplyCnfParam(action_t *pAction, struct cnfparamvals *pvals) } - /* add an Action to the current selector * The pOMSR is freed, as it is not needed after this function. * Note: this function pulls global data that specifies action config state. @@ -1923,7 +1844,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, && (pAction->ppTpl[i] = tplFind(ourConf, (char*)pTplName, strlen((char*)pTplName))) == NULL) { snprintf(errMsg, sizeof(errMsg) / sizeof(char), - " Could not find template '%s' - action disabled\n", + " Could not find template '%s' - action disabled", pTplName); errno = 0; errmsg.LogError(0, RS_RET_NOT_FOUND, "%s", errMsg); @@ -1954,17 +1875,11 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, pAction->pMod = pMod; pAction->pModData = pModData; - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) { - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - } else { - DBGPRINTF("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* check if the module is compatible with select features (currently no such features exist) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ if(bSuspended) - actionSuspend(pAction, datetime.GetTime(NULL)); /* "good" time call, only during init and unavoidable */ + actionSuspend(pAction); CHKiRet(actionConstructFinalize(pAction, queueParams)); @@ -2060,13 +1975,8 @@ actionNewInst(struct nvlst *lst, action_t **ppAction) if((iRet = addAction(&pAction, pMod, pModData, pOMSR, paramvals, queueParams, (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - else { - DBGPRINTF("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* check if the module is compatible with select features + * (currently no such features exist) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ loadConf->actions.nbrActions++; /* one more active action! */ } @@ -2078,33 +1988,6 @@ finalize_it: RETiRet; } - -/* Process a rsyslog v6 action config object (the now-primary config method). - * rgerhards, 2011-07-19 - */ -rsRetVal -actionProcessCnf(struct cnfobj __attribute__((unused)) *o) -{ - DEFiRet; -#if 0 /* we need to check if we actually need this functionality -- later! */ -// This is for STAND-ALONE actions at the conf file TOP level - struct cnfparamvals *paramvals; - - paramvals = nvlstGetParams(o->nvlst, &pblk, NULL); - if(paramvals == NULL) { - iRet = RS_RET_ERR; - goto finalize_it; - } - DBGPRINTF("action param blk after actionProcessCnf:\n"); - cnfparamsPrint(&pblk, paramvals); - - /* now find module to activate */ -finalize_it: -#endif - RETiRet; -} - - /* TODO: we are not yet a real object, the ClassInit here just looks like it is.. */ rsRetVal actionClassInit(void) @@ -46,9 +46,8 @@ typedef enum { /* the following struct defines the action object data structure */ -typedef struct action_s action_t; struct action_s { - time_t f_time; /* used for "message repeated n times" - be careful, old, old code */ + time_t f_time; /* used for "max. n messages in m seconds" processing */ time_t tActNow; /* the current time for an action execution. Initially set to -1 and populated on an as-needed basis. This is a performance optimization. */ time_t tLastExec; /* time this action was last executed */ @@ -69,9 +68,6 @@ struct action_s { struct modInfo_s *pMod;/* pointer to output module handling this selector */ void *pModData; /* pointer to module data - content is module-specific */ sbool bRepMsgHasMsg; /* "message repeated..." has msg fragment in it (0-no, 1-yes) */ - short f_ReduceRepeated;/* reduce repeated lines 0 - no, 1 - yes */ - int f_prevcount; /* repetition cnt of prevline */ - int f_repeatcount; /* number of "repeated" msgs */ rsRetVal (*submitToActQ)(action_t *, batch_t *);/* function submit message to action queue */ rsRetVal (*qConstruct)(struct queue_s *pThis); enum { ACT_STRING_PASSING = 0, ACT_ARRAY_PASSING = 1, ACT_MSG_PASSING = 2, @@ -80,10 +76,6 @@ struct action_s { int iNumTpls; /* number of array entries for template element below */ struct template **ppTpl;/* array of template to use - strings must be passed to doAction * in this order. */ - msg_t *f_pMsg; /* pointer to the message (this will replace the other vars with msg - * content later). This is preserved after the message has been - * processed - it is also used to detect duplicates. - */ qqueue_t *pQueue; /* action queue */ pthread_mutex_t mutAction; /* primary action mutex */ pthread_mutex_t mutActExec; /* mutex to guard actual execution of doAction for single-threaded modules */ @@ -104,7 +96,7 @@ rsRetVal actionDestruct(action_t *pThis); rsRetVal actionDbgPrint(action_t *pThis); rsRetVal actionSetGlobalResumeInterval(int iNewVal); rsRetVal actionDoAction(action_t *pAction); -rsRetVal actionWriteToAction(action_t *pAction); +rsRetVal actionWriteToAction(action_t *pAction, msg_t *pMsg); rsRetVal actionCallHUPHdlr(action_t *pAction); rsRetVal actionClassInit(void); rsRetVal addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringRequest_t *pOMSR, struct cnfparamvals *actParams, struct cnfparamvals *queueParams, int bSuspended); diff --git a/compat/Makefile.am b/compat/Makefile.am new file mode 100644 index 00000000..f580a380 --- /dev/null +++ b/compat/Makefile.am @@ -0,0 +1,6 @@ +noinst_LTLIBRARIES = compat.la + +compat_la_SOURCES = getifaddrs.c ifaddrs.h +compat_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +compat_la_LDFLAGS = -module -avoid-version +compat_la_LIBADD = $(IMUDP_LIBS) diff --git a/compat/getifaddrs.c b/compat/getifaddrs.c new file mode 100755 index 00000000..25d04d4d --- /dev/null +++ b/compat/getifaddrs.c @@ -0,0 +1,283 @@ +#include "config.h" +#ifndef HAVE_GETIFADDRS +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ + +/* + * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. + */ + +#include <netdb.h> +#include <nss_dbdefs.h> +#include <netinet/in.h> +#include <sys/socket.h> +#include <string.h> +#include <stdio.h> +#include <sys/sockio.h> +#include <sys/types.h> +#include <stdlib.h> +#include <net/if.h> +#include <ifaddrs.h> + +/* Normally this is defined in <net/if.h> but was new for Solaris 11 */ +#ifndef LIFC_ENABLED +#define LIFC_ENABLED 0x20 +#endif + +int getallifaddrs(sa_family_t af, struct ifaddrs **ifap, int64_t flags); +int getallifs(int s, sa_family_t af, struct lifreq **lifr, int *numifs, + int64_t lifc_flags); + +/* + * Create a linked list of `struct ifaddrs' structures, one for each + * address that is UP. If successful, store the list in *ifap and + * return 0. On errors, return -1 and set `errno'. + * + * The storage returned in *ifap is allocated dynamically and can + * only be properly freed by passing it to `freeifaddrs'. + */ +int +getifaddrs(struct ifaddrs **ifap) +{ + int err; + char *cp; + struct ifaddrs *curr; + + if (ifap == NULL) { + errno = EINVAL; + return (-1); + } + *ifap = NULL; + err = getallifaddrs(AF_UNSPEC, ifap, LIFC_ENABLED); + if (err == 0) { + for (curr = *ifap; curr != NULL; curr = curr->ifa_next) { + if ((cp = strchr(curr->ifa_name, ':')) != NULL) + *cp = '\0'; + } + } + return (err); +} + +void +freeifaddrs(struct ifaddrs *ifa) +{ + struct ifaddrs *curr; + + while (ifa != NULL) { + curr = ifa; + ifa = ifa->ifa_next; + free(curr->ifa_name); + free(curr->ifa_addr); + free(curr->ifa_netmask); + free(curr->ifa_dstaddr); + free(curr); + } +} + +/* + * Returns all addresses configured on the system. If flags contain + * LIFC_ENABLED, only the addresses that are UP are returned. + * Address list that is returned by this function must be freed + * using freeifaddrs(). + */ +int +getallifaddrs(sa_family_t af, struct ifaddrs **ifap, int64_t flags) +{ + struct lifreq *buf = NULL; + struct lifreq *lifrp; + struct lifreq lifrl; + int ret; + int s, n, numifs; + struct ifaddrs *curr, *prev; + sa_family_t lifr_af; + int sock4; + int sock6; + int err; + + if ((sock4 = socket(AF_INET, SOCK_DGRAM, 0)) < 0) + return (-1); + if ((sock6 = socket(AF_INET6, SOCK_DGRAM, 0)) < 0) { + err = errno; + close(sock4); + errno = err; + return (-1); + } + +retry: + /* Get all interfaces from SIOCGLIFCONF */ + ret = getallifs(sock4, af, &buf, &numifs, (flags & ~LIFC_ENABLED)); + if (ret != 0) + goto fail; + + /* + * Loop through the interfaces obtained from SIOCGLIFCOMF + * and retrieve the addresses, netmask and flags. + */ + prev = NULL; + lifrp = buf; + *ifap = NULL; + for (n = 0; n < numifs; n++, lifrp++) { + + /* Prepare for the ioctl call */ + (void) strncpy(lifrl.lifr_name, lifrp->lifr_name, + sizeof (lifrl.lifr_name)); + lifr_af = lifrp->lifr_addr.ss_family; + if (af != AF_UNSPEC && lifr_af != af) + continue; + + s = (lifr_af == AF_INET ? sock4 : sock6); + + if (ioctl(s, SIOCGLIFFLAGS, (caddr_t)&lifrl) < 0) + goto fail; + if ((flags & LIFC_ENABLED) && !(lifrl.lifr_flags & IFF_UP)) + continue; + + /* + * Allocate the current list node. Each node contains data + * for one ifaddrs structure. + */ + curr = calloc(1, sizeof (struct ifaddrs)); + if (curr == NULL) + goto fail; + + if (prev != NULL) { + prev->ifa_next = curr; + } else { + /* First node in the linked list */ + *ifap = curr; + } + prev = curr; + + curr->ifa_flags = lifrl.lifr_flags; + if ((curr->ifa_name = strdup(lifrp->lifr_name)) == NULL) + goto fail; + + curr->ifa_addr = malloc(sizeof (struct sockaddr_storage)); + if (curr->ifa_addr == NULL) + goto fail; + (void) memcpy(curr->ifa_addr, &lifrp->lifr_addr, + sizeof (struct sockaddr_storage)); + + /* Get the netmask */ + if (ioctl(s, SIOCGLIFNETMASK, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_netmask = malloc(sizeof (struct sockaddr_storage)); + if (curr->ifa_netmask == NULL) + goto fail; + (void) memcpy(curr->ifa_netmask, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + + /* Get the destination for a pt-pt interface */ + if (curr->ifa_flags & IFF_POINTOPOINT) { + if (ioctl(s, SIOCGLIFDSTADDR, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_dstaddr = malloc( + sizeof (struct sockaddr_storage)); + if (curr->ifa_dstaddr == NULL) + goto fail; + (void) memcpy(curr->ifa_dstaddr, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + } else if (curr->ifa_flags & IFF_BROADCAST) { + if (ioctl(s, SIOCGLIFBRDADDR, (caddr_t)&lifrl) < 0) + goto fail; + curr->ifa_broadaddr = malloc( + sizeof (struct sockaddr_storage)); + if (curr->ifa_broadaddr == NULL) + goto fail; + (void) memcpy(curr->ifa_broadaddr, &lifrl.lifr_addr, + sizeof (struct sockaddr_storage)); + } + + } + free(buf); + close(sock4); + close(sock6); + return (0); +fail: + err = errno; + free(buf); + freeifaddrs(*ifap); + *ifap = NULL; + if (err == ENXIO) + goto retry; + close(sock4); + close(sock6); + errno = err; + return (-1); +} + +/* + * Do a SIOCGLIFCONF and store all the interfaces in `buf'. + */ +int +getallifs(int s, sa_family_t af, struct lifreq **lifr, int *numifs, + int64_t lifc_flags) +{ + struct lifnum lifn; + struct lifconf lifc; + size_t bufsize; + char *tmp; + caddr_t *buf = (caddr_t *)lifr; + + lifn.lifn_family = af; + lifn.lifn_flags = lifc_flags; + + *buf = NULL; +retry: + if (ioctl(s, SIOCGLIFNUM, &lifn) < 0) + goto fail; + + /* + * When calculating the buffer size needed, add a small number + * of interfaces to those we counted. We do this to capture + * the interface status of potential interfaces which may have + * been plumbed between the SIOCGLIFNUM and the SIOCGLIFCONF. + */ + bufsize = (lifn.lifn_count + 4) * sizeof (struct lifreq); + + if ((tmp = realloc(*buf, bufsize)) == NULL) + goto fail; + + *buf = tmp; + lifc.lifc_family = af; + lifc.lifc_flags = lifc_flags; + lifc.lifc_len = bufsize; + lifc.lifc_buf = *buf; + if (ioctl(s, SIOCGLIFCONF, (char *)&lifc) < 0) + goto fail; + + *numifs = lifc.lifc_len / sizeof (struct lifreq); + if (*numifs >= (lifn.lifn_count + 4)) { + /* + * If every entry was filled, there are probably + * more interfaces than (lifn.lifn_count + 4). + * Redo the ioctls SIOCGLIFNUM and SIOCGLIFCONF to + * get all the interfaces. + */ + goto retry; + } + return (0); +fail: + free(*buf); + *buf = NULL; + return (-1); +} +#endif /* HAVE_GETIFADDRS */ diff --git a/compat/ifaddrs.h b/compat/ifaddrs.h new file mode 100755 index 00000000..ec359c13 --- /dev/null +++ b/compat/ifaddrs.h @@ -0,0 +1,89 @@ +#include "config.h" +#ifndef HAVE_GETIFADDRS +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ +/* + * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. + */ +#ifndef _IFADDRS_H +#define _IFADDRS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <sys/types.h> + +/* + * The `getifaddrs' function generates a linked list of these structures. + * Each element of the list describes one network interface. + */ +#if defined(_INT64_TYPE) +struct ifaddrs { + struct ifaddrs *ifa_next; /* Pointer to the next structure. */ + char *ifa_name; /* Name of this network interface. */ + uint64_t ifa_flags; /* Flags as from SIOCGLIFFLAGS ioctl. */ + struct sockaddr *ifa_addr; /* Network address of this interface. */ + struct sockaddr *ifa_netmask; /* Netmask of this interface. */ + union { + /* + * At most one of the following two is valid. If the + * IFF_BROADCAST bit is set in `ifa_flags', then + * `ifa_broadaddr' is valid. If the IFF_POINTOPOINT bit is + * set, then `ifa_dstaddr' is valid. It is never the case that + * both these bits are set at once. + */ + struct sockaddr *ifu_broadaddr; + struct sockaddr *ifu_dstaddr; + } ifa_ifu; + void *ifa_data; /* Address-specific data (may be unused). */ +/* + * This may have been defined in <net/if.h>. + */ +#ifndef ifa_broadaddr +#define ifa_broadaddr ifa_ifu.ifu_broadaddr /* broadcast address */ +#endif +#ifndef ifa_dstaddr +#define ifa_dstaddr ifa_ifu.ifu_dstaddr /* other end of p-to-p link */ +#endif +}; +#endif + +/* + * Create a linked list of `struct ifaddrs' structures, one for each + * network interface on the host machine. If successful, store the + * list in *ifap and return 0. On errors, return -1 and set `errno'. + * + * The storage returned in *ifap is allocated dynamically and can + * only be properly freed by passing it to `freeifaddrs'. + */ +extern int getifaddrs(struct ifaddrs **); + +/* Reclaim the storage allocated by a previous `getifaddrs' call. */ +extern void freeifaddrs(struct ifaddrs *); + + +#ifdef __cplusplus +} +#endif + +#endif /* _IFADDRS_H */ +#endif /* HAVE_GETIFADDRS */ diff --git a/configure.ac b/configure.ac index 6d79a981..e16c1f57 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[6.6.0],[rsyslog@lists.adiscon.com]) +AC_INIT([rsyslog],[7.4.3],[rsyslog@lists.adiscon.com]) AM_INIT_AUTOMAKE m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) @@ -32,11 +32,8 @@ AC_CANONICAL_HOST PKG_PROG_PKG_CONFIG # modules we require -PKG_CHECK_MODULES(LIBESTR, libestr >= 0.1.2) -PKG_CHECK_MODULES(LIBEE, libee >= 0.4.0) +PKG_CHECK_MODULES(LIBESTR, libestr >= 0.1.5) PKG_CHECK_MODULES([JSON_C], [json]) -AC_SUBST([JSON_CFLAGS]) -AC_SUBST([JSON_LIBS]) case "${host}" in *-*-linux*) @@ -57,6 +54,10 @@ case "${host}" in AC_DEFINE([OS_SOLARIS], [1], [Indicator for a Solaris OS]) AC_DEFINE([_POSIX_PTHREAD_SEMANTICS], [1], [Use POSIX pthread semantics]) SOL_LIBS="-lsocket -lnsl" + # Solaris libuuid does not ship with a pkgconfig file so override the appropriate + # variables (but only if they have not been set by the user). + LIBUUID_CFLAGS=${LIBUUID_CFLAGS:= } + LIBUUID_LIBS=${LIBUUID_LIBS:=-luuid} AC_SUBST(SOL_LIBS) ;; esac @@ -118,7 +119,10 @@ AC_TYPE_SIGNAL AC_FUNC_STAT AC_FUNC_STRERROR_R AC_FUNC_VPRINTF -AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync lseek64]) +AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64]) + +# getifaddrs is in libc (mostly) or in libsocket (eg Solaris 11) or not defined (eg Solaris 10) +AC_SEARCH_LIBS([getifaddrs], [socket], [AC_DEFINE(HAVE_GETIFADDRS, [1], [set define])]) # the check below is probably ugly. If someone knows how to do it in a better way, please # let me know! -- rgerhards, 2010-10-06 @@ -126,6 +130,9 @@ AC_CHECK_DECL([SCM_CREDENTIALS], [AC_DEFINE(HAVE_SCM_CREDENTIALS, [1], [set defi #include <sys/socket.h>]) AC_CHECK_DECL([SO_TIMESTAMP], [AC_DEFINE(HAVE_SO_TIMESTAMP, [1], [set define])], [], [#include <sys/types.h> #include <sys/socket.h>]) +AC_CHECK_DECL([SYS_gettid], [AC_DEFINE(HAVE_SYS_gettid, [1], [set define])], [], [#include <sys/syscall.h>]) +AC_CHECK_MEMBER([struct sysinfo.uptime], [AC_DEFINE(HAVE_SYSINFO_UPTIME, [1], [set define])], [], [#include <sys/sysinfo.h>]) +AC_CHECK_DECL([GLOB_NOMAGIC], [AC_DEFINE(HAVE_GLOB_NOMAGIC, [1], [set define])], [], [#include <glob.h>]) # Check for MAXHOSTNAMELEN AC_MSG_CHECKING(for MAXHOSTNAMELEN) @@ -149,7 +156,7 @@ RS_ATOMIC_OPERATIONS RS_ATOMIC_OPERATIONS_64BIT # fall back to POSIX sems for atomic operations (cpu expensive) -AC_CHECK_HEADERS([semaphore.h]) +AC_CHECK_HEADERS([semaphore.h sys/syscall.h]) # Additional module directories @@ -247,22 +254,7 @@ AM_CONDITIONAL(ENABLE_GSSAPI, test x$enable_gssapi_krb5 = xyes) # multithreading via pthreads -AC_ARG_ENABLE(pthreads, - [AS_HELP_STRING([--enable-pthreads],[Enable multithreading via pthreads @<:@default=yes@:>@])], - [case "${enableval}" in - yes) enable_pthreads="yes" ;; - no) enable_pthreads="no" ;; - *) AC_MSG_ERROR(bad value ${enableval} for --enable-pthreads) ;; - esac], - [enable_pthreads=yes] -) - -if test "x$enable_pthreads" = "xno"; then - AC_MSG_ERROR(rsyslog v3+ does no longer support single threading mode -- use a previous version for that); -fi - -if test "x$enable_pthreads" != "xno"; then - AC_CHECK_HEADERS( +AC_CHECK_HEADERS( [pthread.h], [ AC_CHECK_LIB( @@ -283,8 +275,7 @@ if test "x$enable_pthreads" != "xno"; then ) ], [AC_MSG_FAILURE([pthread is missing])] - ) -fi +) AC_CHECK_FUNCS( [pthread_setschedparam], @@ -332,27 +323,32 @@ AM_CONDITIONAL(ENABLE_IMKLOG_BSD, test x$os_type = xbsd) AM_CONDITIONAL(ENABLE_IMKLOG_LINUX, test x$os_type = xlinux) AM_CONDITIONAL(ENABLE_IMKLOG_SOLARIS, test x$os_type = xsolaris) +# kmsg +AC_ARG_ENABLE(kmsg, + [AS_HELP_STRING([--enable-kmsg],[Kmsg structured kernel logs functionality @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_kmsg="yes" ;; + no) enable_kmsg="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-kmsg) ;; + esac], + [enable_kmsg="no"] +) +AM_CONDITIONAL(ENABLE_IMKMSG, test x$enable_kmsg = xyes) -# -# SYSLOG_UNIXAF -# -AC_MSG_CHECKING(for SYSLOG_UNIXAF support) -AC_ARG_ENABLE([unix], - [AS_HELP_STRING([--disable-unix], [Disable support for unix])], - [ - if test "x${enableval}" = "xyes"; then - AC_MSG_RESULT([yes]) - AC_DEFINE([SYSLOG_UNIXAF], [1], [Description]) - else - AC_MSG_RESULT([no]) - fi - ], - [ - # enabled by default - AC_MSG_RESULT([yes]) - AC_DEFINE([SYSLOG_UNIXAF], [1], [Description]) - ]) - +# imjournal +AC_ARG_ENABLE(imjournal, + [AS_HELP_STRING([--enable-imjournal],[Systemd journal message import @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_imjournal="yes" ;; + no) enable_imjournal="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-imjournal) ;; + esac], + [enable_imjournal="no"] +) +if test "x$enable_imjournal" = "xyes"; then + PKG_CHECK_MODULES([LIBSYSTEMD_JOURNAL], [libsystemd-journal >= 197]) +fi +AM_CONDITIONAL(ENABLE_IMJOURNAL, test x$enable_imjournal = xyes) # inet AC_ARG_ENABLE(inet, @@ -686,27 +682,16 @@ if test "x$enable_libdbi" = "xyes"; then [dbi_initialize_r], [AC_DEFINE([HAVE_DBI_R], [1], [Define to 1 if libdbi supports the new plugin-safe interface])] ) + AC_CHECK_LIB( + [dbi], + [dbi_conn_transaction_begin], + [AC_DEFINE([HAVE_DBI_TXSUPP], [1], [Define to 1 if libdbi supports transactions])] + ) fi AM_CONDITIONAL(ENABLE_OMLIBDBI, test x$enable_libdbi = xyes) AC_SUBST(LIBDBI_CFLAGS) AC_SUBST(LIBDBI_LIBS) -# libuuid support -AC_CHECK_HEADERS( - [uuid/uuid.h],, - [AC_MSG_FAILURE([libuuid is missing])] -) -AC_CHECK_LIB( - [uuid], - [uuid_generate], - [LIBUUID_CFLAGS="" - LIBUUID_LIBS="-luuid" - ], - [AC_MSG_FAILURE([libuuid library is missing])] -) -AC_SUBST(LIBUUID_CFLAGS) -AC_SUBST(LIBUUID_LIBS) - # SNMP support AC_ARG_ENABLE(snmp, [AS_HELP_STRING([--enable-snmp],[Enable SNMP support @<:@default=no@:>@])], @@ -736,6 +721,23 @@ AC_SUBST(SNMP_CFLAGS) AC_SUBST(SNMP_LIBS) +# uuid support +AC_ARG_ENABLE(uuid, + [AS_HELP_STRING([--enable-uuid],[Enable support for uuid generation @<:@default=yes@:>@])], + [case "${enableval}" in + yes) enable_uuid="yes" ;; + no) enable_uuid="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-uuid) ;; + esac], + [enable_uuid=yes] +) +if test "x$enable_uuid" = "xyes"; then + PKG_CHECK_MODULES([LIBUUID], [uuid]) + AC_DEFINE(USE_LIBUUID, 1, [Define if you want to enable libuuid support]) +fi +AM_CONDITIONAL(ENABLE_UUID, test x$enable_uuid = xyes) + + # elasticsearch support AC_ARG_ENABLE(elasticsearch, [AS_HELP_STRING([--enable-elasticsearch],[Enable elasticsearch output module @<:@default=no@:>@])], @@ -747,27 +749,10 @@ AC_ARG_ENABLE(elasticsearch, [enable_elasticsearch=no] ) if test "x$enable_elasticsearch" = "xyes"; then - AC_CHECK_PROG( - [HAVE_CURL_CONFIG], - [curl-config], - [yes],,, - ) - if test "x${HAVE_CURL_CONFIG}" != "xyes"; then - AC_MSG_FAILURE([curl-config not found in PATH]) - fi - AC_CHECK_LIB( - [curl], - [curl_global_init], - [CURL_CFLAGS="`curl-config --cflags`" - CURL_LIBS="`curl-config --libs`" - ], - [AC_MSG_FAILURE([curl library is missing])], - [`curl-config --libs --cflags`] - ) + PKG_CHECK_MODULES([CURL], [libcurl]) + LT_LIB_M fi AM_CONDITIONAL(ENABLE_ELASTICSEARCH, test x$enable_elasticsearch = xyes) -AC_SUBST(CURL_CFLAGS) -AC_SUBST(CURL_LIBS) # GnuTLS support @@ -785,8 +770,40 @@ if test "x$enable_gnutls" = "xyes"; then AC_DEFINE([ENABLE_GNUTLS], [1], [Indicator that GnuTLS is present]) fi AM_CONDITIONAL(ENABLE_GNUTLS, test x$enable_gnutls = xyes) -AC_SUBST(GNUTLS_CFLAGS) -AC_SUBST(GNUTLS_LIBS) + +# libgcrypt support +AC_ARG_ENABLE(libgcrypt, + [AS_HELP_STRING([--enable-libgcrypt],[Enable log file encryption support (libgcrypt) @<:@default=yes@:>@])], + [case "${enableval}" in + yes) enable_libgcrypt="yes" ;; + no) enable_libgcrypt="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-libgcrypt) ;; + esac], + [enable_libgcrypt=yes] +) +if test "x$enable_libgcrypt" = "xyes"; then + AC_CHECK_PROG( + [HAVE_LIBGCRYPT_CONFIG], + [libgcrypt-config], + [yes],,, + ) + if test "x${HAVE_LIBGCRYPT_CONFIG}" != "xyes"; then + AC_MSG_FAILURE([libgcrypt-config not found in PATH]) + fi + AC_CHECK_LIB( + [gcrypt], + [gcry_cipher_open], + [LIBGCRYPT_CFLAGS="`libgcrypt-config --cflags`" + LIBGCRYPT_LIBS="`libgcrypt-config --libs`" + ], + [AC_MSG_FAILURE([libgcrypt is missing])], + [`libgcrypt-config --libs --cflags`] + ) + AC_DEFINE([ENABLE_LIBGCRYPT], [1], [Indicator that LIBGCRYPT is present]) +fi +AM_CONDITIONAL(ENABLE_LIBGCRYPT, test x$enable_libgcrypt = xyes) +AC_SUBST(LIBGCRYPT_CFLAGS) +AC_SUBST(LIBGCRYPT_LIBS) # support for building the rsyslogd runtime @@ -802,7 +819,6 @@ AC_ARG_ENABLE(rsyslogrt, if test "x$enable_rsyslogrt" = "xyes"; then RSRT_CFLAGS1="-I\$(top_srcdir)/runtime -I\$(top_srcdir) -I\$(top_srcdir)/grammar" RSRT_LIBS1="\$(top_builddir)/runtime/librsyslog.la" - #??CNF_LIBS="\$(top_builddir)/grammar/libgrammar.la" fi AM_CONDITIONAL(ENABLE_RSYSLOGRT, test x$enable_rsyslogrt = xyes) RSRT_CFLAGS="\$(RSRT_CFLAGS1) \$(LIBESTR_CFLAGS) \$(JSON_C_FLAGS)" @@ -901,11 +917,10 @@ AC_ARG_ENABLE(mmnormalize, [enable_mmnormalize=no] ) if test "x$enable_mmnormalize" = "xyes"; then + PKG_CHECK_MODULES(LIBEE, libee >= 0.4.0) PKG_CHECK_MODULES(LIBLOGNORM, lognorm >= 0.3.1) fi AM_CONDITIONAL(ENABLE_MMNORMALIZE, test x$enable_mmnormalize = xyes) -AC_SUBST(LOGNORM_CFLAGS) -AC_SUBST(LOGNORM_LIBS) # mmnjsonparse @@ -918,13 +933,9 @@ AC_ARG_ENABLE(mmjsonparse, esac], [enable_mmjsonparse=no] ) -if test "x$enable_mmjsonparse" = "xyes"; then - PKG_CHECK_MODULES(LIBLOGNORM, lognorm >= 0.3.1) -fi AM_CONDITIONAL(ENABLE_MMJSONPARSE, test x$enable_mmjsonparse = xyes) - # mmaudit AC_ARG_ENABLE(mmaudit, [AS_HELP_STRING([--enable-mmaudit],[Enable building mmaudit support @<:@default=no@:>@])], @@ -935,12 +946,22 @@ AC_ARG_ENABLE(mmaudit, esac], [enable_mmaudit=no] ) -if test "x$enable_mmaudit" = "xyes"; then - PKG_CHECK_MODULES(LIBLOGNORM, lognorm >= 0.3.1) -fi AM_CONDITIONAL(ENABLE_MMAUDIT, test x$enable_mmaudit = xyes) +# mmanon +AC_ARG_ENABLE(mmanon, + [AS_HELP_STRING([--enable-mmanon],[Enable building mmanon support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmanon="yes" ;; + no) enable_mmanon="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmanon) ;; + esac], + [enable_mmanon=no] +) +AM_CONDITIONAL(ENABLE_MMANON, test x$enable_mmanon = xyes) + + # RELP support AC_ARG_ENABLE(relp, [AS_HELP_STRING([--enable-relp],[Enable RELP support @<:@default=no@:>@])], @@ -952,11 +973,49 @@ AC_ARG_ENABLE(relp, [enable_relp=no] ) if test "x$enable_relp" = "xyes"; then - PKG_CHECK_MODULES(RELP, relp >= 1.0.1) + PKG_CHECK_MODULES(RELP, relp >= 1.0.3) fi AM_CONDITIONAL(ENABLE_RELP, test x$enable_relp = xyes) -AC_SUBST(RELP_CFLAGS) -AC_SUBST(RELP_LIBS) + + +# GuardTime support +AC_ARG_ENABLE(guardtime, + [AS_HELP_STRING([--enable-guardtime],[Enable log file signing support (via GuardTime) @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_guardtime="yes" ;; + no) enable_guardtime="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-guardtime) ;; + esac], + [enable_guardtime=no] +) +if test "x$enable_guardtime" = "xyes"; then + PKG_CHECK_MODULES(GUARDTIME, libgt >= 0.3.1) +fi +AM_CONDITIONAL(ENABLE_GUARDTIME, test x$enable_guardtime = xyes) + + +# Support using cached man file copies, to avoid the need for rst2man +# in the build environment +AC_ARG_ENABLE(cached_man_pages, + [AS_HELP_STRING([--enable-cached-man-pages],[Enable using cached versions of man files (avoid rst2man) @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_cached_man_pages="yes" ;; + no) enable_cached_man_pages="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-cached-man-pages) ;; + esac], + [enable_cached_man_pages=no] +) +if test "x$enable_cached_man_pages" = "xno"; then +# obtain path for rst2man + if test "x$enable_libgcrypt" = "xyes" || \ + test "x$enable_guardtime" = "xyes"; then + AC_PATH_PROG([RST2MAN], [rst2man]) + if test "x${RST2MAN}" == "x"; then + AC_MSG_FAILURE([rst2man not found in PATH]) + fi + fi +fi + # RFC 3195 support @@ -973,8 +1032,6 @@ if test "x$enable_rfc3195" = "xyes"; then PKG_CHECK_MODULES(LIBLOGGING, liblogging >= 0.7.1) fi AM_CONDITIONAL(ENABLE_RFC3195, test x$enable_rfc3195 = xyes) -AC_SUBST(LIBLOGGING_CFLAGS) -AC_SUBST(LIBLOGGING_LIBS) # enable/disable the testbench (e.g. because some important parts @@ -1110,6 +1167,21 @@ AC_ARG_ENABLE(omstdout, ) AM_CONDITIONAL(ENABLE_OMSTDOUT, test x$enable_omstdout = xyes) +# settings for omjournal +AC_ARG_ENABLE(omjournal, + [AS_HELP_STRING([--enable-omjournal],[Compiles omjournal @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_omjournal="yes" ;; + no) enable_omjournal="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-omjournal) ;; + esac], + [enable_omjournal=no] +) +if test "x$enable_omjournal" = "xyes"; then + PKG_CHECK_MODULES([LIBSYSTEMD_JOURNAL], [libsystemd-journal >= 197]) +fi +AM_CONDITIONAL(ENABLE_OMJOURNAL, test x$enable_omjournal = xyes) + # settings for pmlastmsg AC_ARG_ENABLE(pmlastmsg, @@ -1178,7 +1250,7 @@ AM_CONDITIONAL(ENABLE_PMRFC3164SD, test x$enable_pmrfc3164sd = xyes) # settings for omruleset AC_ARG_ENABLE(omruleset, - [AS_HELP_STRING([--enable-omruleset],[Compiles ruleset forwarding module @<:@default=yes@:>@])], + [AS_HELP_STRING([--enable-omruleset],[Compiles ruleset forwarding module @<:@default=no@:>@])], [case "${enableval}" in yes) enable_omruleset="yes" ;; no) enable_omruleset="no" ;; @@ -1207,9 +1279,6 @@ fi AM_CONDITIONAL(ENABLE_GUI, test x$enable_gui = xyes) -AC_SUBST(RELP_CFLAGS) -AC_SUBST(RELP_LIBS) - # settings for omuxsock AC_ARG_ENABLE(omuxsock, [AS_HELP_STRING([--enable-omuxsock],[Compiles omuxsock module @<:@default=no@:>@])], @@ -1280,8 +1349,6 @@ AC_ARG_ENABLE(ommongodb, ) if test "x$enable_ommongodb" = "xyes"; then PKG_CHECK_MODULES(LIBMONGO_CLIENT, libmongo-client >= 0.1.4) - AC_SUBST(LIBMONGO_CLIENT_CFLAGS) - AC_SUBST(LIBMONGO_CLIENT_LIBS) fi AM_CONDITIONAL(ENABLE_OMMONGODB, test x$enable_ommongodb = xyes) # end of mongodb code @@ -1298,8 +1365,6 @@ AC_ARG_ENABLE(imzmq3, ) if test "x$enable_imzmq3" = "xyes"; then PKG_CHECK_MODULES(CZMQ, libczmq >= 1.1.0) - AC_SUBST(CZMQ_CFLAGS) - AC_SUBST(CZMQ_LIBS) fi AM_CONDITIONAL(ENABLE_IMZMQ3, test x$enable_imzmq3 = xyes) @@ -1317,13 +1382,31 @@ AC_ARG_ENABLE(omzmq3, ) if test "x$enable_omzmq3" = "xyes"; then PKG_CHECK_MODULES(CZMQ, libczmq >= 1.1.0) - AC_SUBST(CZMQ_CFLAGS) - AC_SUBST(CZMQ_LIBS) fi AM_CONDITIONAL(ENABLE_OMZMQ3, test x$enable_omzmq3 = xyes) # END ZMQ3 SUPPORT +# BEGIN RABBITMQ OUTPUT SUPPORT + +AC_ARG_ENABLE(omrabbitmq, + [AS_HELP_STRING([--enable-omrabbitmq],[Compiles omrabbitmq output module @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_omrabbitmq="yes" ;; + no) enable_omrabbitmq="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-omrabbitmq) ;; + esac], + [enable_omrabbitmq=no] +) +if test "x$enable_omrabbitmq" = "xyes"; then + PKG_CHECK_MODULES(RABBITMQ, librabbitmq >= 0.2.0) + AC_SUBST(RABBITMQ_CFLAGS) + AC_SUBST(RABBITMQ_LIBS) +fi +AM_CONDITIONAL(ENABLE_OMRABBITMQ, test x$enable_omrabbitmq = xyes) + +# END RABBITMQ SUPPORT + # HIREDIS SUPPORT AC_ARG_ENABLE(omhiredis, @@ -1338,8 +1421,6 @@ AC_ARG_ENABLE(omhiredis, # if test "x$enable_omhiredis" = "xyes"; then PKG_CHECK_MODULES(HIREDIS, hiredis >= 0.10.1) - AC_SUBST(HIREDIS_CFLAGS) - AC_SUBST(HIREDIS_LIBS) fi AM_CONDITIONAL(ENABLE_OMHIREDIS, test x$enable_omhiredis = xyes) @@ -1347,6 +1428,7 @@ AM_CONDITIONAL(ENABLE_OMHIREDIS, test x$enable_omhiredis = xyes) AC_CONFIG_FILES([Makefile \ runtime/Makefile \ + compat/Makefile \ grammar/Makefile \ tools/Makefile \ doc/Makefile \ @@ -1355,11 +1437,14 @@ AC_CONFIG_FILES([Makefile \ plugins/im3195/Makefile \ plugins/imgssapi/Makefile \ plugins/imuxsock/Makefile \ + plugins/imjournal/Makefile \ plugins/immark/Makefile \ plugins/imklog/Makefile \ + plugins/imkmsg/Makefile \ plugins/omhdfs/Makefile \ plugins/omprog/Makefile \ plugins/omstdout/Makefile \ + plugins/omjournal/Makefile \ plugins/pmrfc3164sd/Makefile \ plugins/pmlastmsg/Makefile \ plugins/pmcisconames/Makefile \ @@ -1374,7 +1459,7 @@ AC_CONFIG_FILES([Makefile \ plugins/impstats/Makefile \ plugins/imrelp/Makefile \ plugins/imdiag/Makefile \ - plugins/imzmq3/Makefile \ + plugins/imzmq3/Makefile \ plugins/omtesting/Makefile \ plugins/omgssapi/Makefile \ plugins/ommysql/Makefile \ @@ -1386,11 +1471,13 @@ AC_CONFIG_FILES([Makefile \ plugins/omoracle/Makefile \ plugins/omudpspoof/Makefile \ plugins/ommongodb/Makefile \ - plugins/omhiredis/Makefile \ - plugins/omzmq3/Makefile \ + plugins/omhiredis/Makefile \ + plugins/omzmq3/Makefile \ + plugins/omrabbitmq/Makefile \ plugins/mmnormalize/Makefile \ plugins/mmjsonparse/Makefile \ plugins/mmaudit/Makefile \ + plugins/mmanon/Makefile \ plugins/omelasticsearch/Makefile \ plugins/sm_cust_bindcdr/Makefile \ plugins/mmsnmptrapd/Makefile \ @@ -1408,10 +1495,16 @@ echo " Zlib compression support enabled: $enable_zlib" echo " rsyslog runtime will be built: $enable_rsyslogrt" echo " rsyslogd will be built: $enable_rsyslogd" echo " GUI components will be built: $enable_gui" +echo " cached man files will be used: $enable_cached_man_pages" echo " Unlimited select() support enabled: $enable_unlimited_select" +echo " uuid support enabled: $enable_uuid" +echo " Log file signing support: $enable_guardtime" +echo " Log file encryption support: $enable_libgcrypt" +echo " anonymization support enabled: $enable_mmanon" echo echo "---{ input plugins }---" echo " Klog functionality enabled: $enable_klog ($os_type)" +echo " /dev/kmsg functionality enabled: $enable_kmsg" echo " plain tcp input module enabled: $enable_imptcp" echo " threaded plain tcp input module enabled: $enable_imttcp" echo " imdiag enabled: $enable_imdiag" @@ -1419,17 +1512,20 @@ echo " file input module enabled: $enable_imfile" echo " Solaris input module enabled: $enable_imsolaris" echo " periodic statistics module enabled: $enable_impstats" echo " imzmq3 input module enabled: $enable_imzmq3" +echo " imjournal input module enabled: $enable_imjournal" echo echo "---{ output plugins }---" echo " Mail support enabled: $enable_mail" echo " omprog module will be compiled: $enable_omprog" echo " omstdout module will be compiled: $enable_omstdout" +echo " omjournal module will be compiled: $enable_omjournal" echo " omhdfs module will be compiled: $enable_omhdfs" echo " omelasticsearch module will be compiled: $enable_elasticsearch" echo " omruleset module will be compiled: $enable_omruleset" echo " omudpspoof module will be compiled: $enable_omudpspoof" echo " omuxsock module will be compiled: $enable_omuxsock" echo " omzmq3 module will be compiled: $enable_omzmq3" +echo " omrabbitmq module will be compiled: $enable_omrabbitmq" echo echo "---{ parser modules }---" echo " pmrfc3164sd module will be compiled: $enable_pmrfc3164sd" @@ -27,26 +27,18 @@ #ifndef DIRTY_H_INCLUDED #define DIRTY_H_INCLUDED 1 -rsRetVal multiSubmitMsg(multi_submit_t *pMultiSub); -rsRetVal submitMsg(msg_t *pMsg); +rsRetVal __attribute__((deprecated)) multiSubmitMsg(multi_submit_t *pMultiSub); +rsRetVal multiSubmitMsg2(multi_submit_t *pMultiSub); /* friends only! */ +rsRetVal submitMsg2(msg_t *pMsg); +rsRetVal __attribute__((deprecated)) submitMsg(msg_t *pMsg); +rsRetVal multiSubmitFlush(multi_submit_t *pMultiSub); rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags); -rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime, ruleset_t *pRuleset); +rsRetVal __attribute__((deprecated)) parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime, ruleset_t *pRuleset); rsRetVal diagGetMainMsgQSize(int *piSize); /* for imdiag */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName); +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams); -/* Intervals at which we flush out "message repeated" messages, - * in seconds after previous message is logged. After each flush, - * we move to the next interval until we reach the largest. - * TODO: move this to action object! Only action.c and syslogd.c use it. - */ extern int MarkInterval; -extern int repeatinterval[2]; extern qqueue_t *pMsgQueue; /* the main message queue */ extern int iConfigVerify; /* is this just a config verify run? */ extern int bHaveMainQueue; -#define MAXREPEAT ((int)((sizeof(repeatinterval) / sizeof(repeatinterval[0])) - 1)) -#define REPEATTIME(f) ((f)->f_time + repeatinterval[(f)->f_repeatcount]) -#define BACKOFF(f) { if (++(f)->f_repeatcount > MAXREPEAT) \ - (f)->f_repeatcount = MAXREPEAT; \ - } #endif /* #ifndef DIRTY_H_INCLUDED */ diff --git a/doc/Makefile.am b/doc/Makefile.am index 1ae1c68d..e1757644 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -34,11 +34,15 @@ html_files = \ ompipe.html \ omfwd.html \ omfile.html \ + omjournal.html \ + imjournal.html \ + mmanon.html \ omusrmsg.html \ omstdout.html \ omudpspoof.html \ omruleset.html \ omsnmp.html \ + sigprov_gt.html \ ommysql.html \ omoracle.html \ omlibdbi.html \ @@ -70,6 +74,7 @@ html_files = \ tls_cert_client.html \ tls_cert_scenario.html \ rainerscript.html \ + lookup_tables.html \ rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ @@ -115,13 +120,13 @@ html_files = \ gssapi.html \ licensing.html \ mmnormalize.html \ + mmjsonparse.html \ ommail.html \ omuxsock.html \ omrelp.html \ syslog_parsing.html \ troubleshoot.html \ rsyslog_conf_actions.html \ - rsyslog_conf_examples.html \ rsyslog_conf_filter.html \ rsyslog_conf_global.html \ rsyslog_conf_modules.html \ @@ -130,6 +135,7 @@ html_files = \ rsyslog_conf_nomatch.html \ queues_analogy.html \ multi_ruleset.html \ + multi_ruleset_legacy_format.html \ dev_oplugins.html \ free_support.html \ imudp.html \ @@ -140,11 +146,13 @@ html_files = \ rsconf1_abortonuncleanconfig.html \ rsconf1_maxopenfiles.html \ rsconf1_omfileforcechown.html \ - rsyslog_conf_file_syntax_differences.html \ - rsyslog_conf_lines.html \ rsyslog_queue_pointers.jpeg \ rsyslog_queue_pointers2.jpeg \ v6compatibility.html \ + v7compatibility.html \ + rsyslog_conf_basic_structure.html \ + rsyslog_conf_sysklogd_compatibility.html \ + imkmsg.html \ src/classes.dia grfx_files = \ diff --git a/doc/confsamples/normalization.conf b/doc/confsamples/normalization.conf new file mode 100644 index 00000000..7cfd92ef --- /dev/null +++ b/doc/confsamples/normalization.conf @@ -0,0 +1,187 @@ +# this is a config sample for log normalization, but can +# be used as a more complex general sample. +# It is based on a plain standard rsyslog.conf for Red Hat systems. +# +# NOTE: Absolute path names for modules are used in this config +# so that we can run a different rsyslog version alongside the +# regular system-installed rsyslogd. Remove these path names +# for production environment. + +#### MODULES #### + +# we do not run imuxsock as we don't want to mess with the main system logger +#module(load="/home/rger/proj/rsyslog/plugins/imuxsock/.libs/imuxsock") # provides support for local system logging (e.g. via logger command) +#module(load="imklog") # provides kernel logging support (previously done by rklogd) +module(load="/home/rger/proj/rsyslog/plugins/imudp/.libs/imudp") # Provides UDP syslog reception +module(load="/home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp") +module(load="/home/rger/proj/rsyslog/plugins/mmjsonparse/.libs/mmjsonparse") +module(load="/home/rger/proj/rsyslog/plugins/mmnormalize/.libs/mmnormalize") + +/* We assume to have all TCP logging (for simplicity) + * Note that we use different ports to point different sources + * to the right rule sets for normalization. While there are + * other methods (e.g. based on tag or source), using multiple + * ports is both the easiest as well as the fastest. + */ +input(type="imtcp" port="13514" Ruleset="WindowsRsyslog") +input(type="imtcp" port="13515" Ruleset="LinuxPlainText") +input(type="imtcp" port="13516" Ruleset="WindowsSnare") + +#debug: +action(type="omfile" file="/home/rger/proj/rsyslog/logfile") + +/* This ruleset handles structured logging. + * It is the only one ever called for remote machines + * but executed in addition to the standard action for + * the local machine. The ultimate goal is to forward + * to some Vendor's analysis tool (which digests a + * structured log format, here we use Lumberjack). + */ +template(name="lumberjack" type="string" string="%$!all-json%\n") + + +/* the rsyslog Windows Agent uses native Lumberjack format + * (better said: is configured to use it) + */ +ruleset(name="WindowsRsyslog") { + action(type="mmjsonparse") + if $parsesuccess == "OK" then { + if $!id == 4634 then + set $!usr!type = "logoff"; + else if $!id == 4624 then + set $!usr!type = "logon"; + set $!usr!rcvdfrom = $!source; + set $!usr!rcvdat = $timereported; + set $!usr!user = $!TargetDomainName & "\\" & $!TargetUserName; + call outwriter + } +} + +/* This handles clumsy snare format. Note that "#011" are + * the escape sequences for tab chars used by snare. + */ +ruleset(name="WindowsSnare") { + set $!usr!type = field($rawmsg, "#011", 6); + if $!usr!type == 4634 then { + set $!usr!type = "logoff"; + set $!doProces = 1; + } else if $!usr!type == 4624 then { + set $!usr!type = "logon"; + set $!doProces = 1; + } else + set $!doProces = 0; + if $!doProces == 1 then { + set $!usr!rcvdfrom = field($rawmsg, 32, 4); + set $!usr!rcvdat = field($rawmsg, "#011", 5); + /* we need to fix up the snare date */ + set $!usr!rcvdat = field($!usr!rcvdat, 32, 2) & " " & + field($!usr!rcvdat, 32, 3) & " " & + field($!usr!rcvdat, 32, 4); + set $!usr!user = field($rawmsg, "#011", 8); + call outwriter + } +} + +/* plain Linux log messages (here: ssh and sudo) need to be + * parsed - we use mmnormalize for fast and efficient parsing + * here. + */ +ruleset(name="LinuxPlainText") { + action(type="mmnormalize" + rulebase="/home/rger/proj/rsyslog/linux.rb" userawmsg="on") + if $parsesuccess == "OK" and $!user != "" then { + if $!type == "opened" then + set $!usr!type = "logon"; + else if $!type == "closed" then + set $!usr!type = "logoff"; + set $!usr!rcvdfrom = $!rcvdfrom; + set $!usr!rcvdat = $!rcvdat; + set $!usr!user = $!user; + call outwriter + } +} + +/* with CSV, we the reader must receive information on the + * field names via some other method (e.g. tool configuration, + * prepending of a header to the written CSV-file). All of + * this is highly dependant on the actual CSV dialect needed. + * Below, we cover the basics. + */ +template(name="csv" type="list") { + property(name="$!usr!rcvdat" format="csv") + constant(value=",") + property(name="$!usr!rcvdfrom" format="csv") + constant(value=",") + property(name="$!usr!user" format="csv") + constant(value=",") + property(name="$!usr!type" format="csv") + constant(value="\n") +} + +/* template for Lumberjack-style logging. Note that the extra + * LF at the end is just for wrinting it to file - it MUST NOT + * be included for messages intended to be sent to a remote system. + * For the latter use case, the syslog header must also be prepended, + * something we have also not done for simplicity (as we write to files). + * Note that we use a JSON-shortcut: If a tree name is specified, JSON + * for its whole subtree is generated. Thus, we only need to specify the + * $!usr top node to get everytihing we need. + */ +template(name="cee" type="string" string="@cee: %$!usr%\n") + + +/* this ruleset simulates forwarding to the final destination */ +ruleset(name="outwriter"){ + action(type="omfile" + file="/home/rger/proj/rsyslog/logfile.csv" template="csv") + action(type="omfile" + file="/home/rger/proj/rsyslog/logfile.cee" template="cee") +} + + +/* below is just the usual "uninteresting" stuff... + * Note that this goes into the default rule set. So + * local logging is handled "as usual" without the need + * for any extra effort. + */ + + +#### GLOBAL DIRECTIVES #### + +# Use default timestamp format +$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + +# Include all config files in /etc/rsyslog.d/ +# commented out not to interfere with the system rsyslogd +# (just for this test configuration!) +#$IncludeConfig /etc/rsyslog.d/*.conf + + +#### RULES #### + +# Log all kernel messages to the console. +# Logging much else clutters up the screen. +#kern.* /dev/console + +# Log anything (except mail) of level info or higher. +# Don't log private authentication messages! +*.info;mail.none;authpriv.none;cron.none /var/log/messages + +# The authpriv file has restricted access. +authpriv.* /var/log/secure + +# Log all the mail messages in one place. +mail.* /var/log/maillog + + +# Log cron stuff +cron.* /var/log/cron + +# Everybody gets emergency messages +*.emerg :omusrmsg:* + +# Save news errors of level crit and higher in a special file. +uucp,news.crit /var/log/spooler + +# Save boot messages also to boot.log +local7.* /var/log/boot.log diff --git a/doc/cryprov_gcry.html b/doc/cryprov_gcry.html new file mode 100644 index 00000000..2568add9 --- /dev/null +++ b/doc/cryprov_gcry.html @@ -0,0 +1,121 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>libgcryt Log Crypto Provider (gcry)</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> + +<h1>libgcrypt Log Crypto Provider (gcry)</h1> +<p><b>Signature Provider Name: gt</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Supported Since: </b>since 7.3.10 +<p><b>Description</b>:</p> +<p>Provides encryption support to rsyslog. +</p> + +<p><b>Configuration Parameters</b>:</p> +<p>Crypto providers are loaded by omfile, when the +provider is selected in its "cry.providerName" parameter. +Parameters for the provider are given in the omfile action instance +line. +<p>This provider creates an encryption information file with the same base name but +the extension ".encinfo" for each log file (both for fixed-name files +as well as dynafiles). Both files together form a set. So you need to +archive both in order to prove integrity. +<ul> +<li><b>cry.algo</b> <Encryption Algorithm><br> +The algorithm (cipher) to be used for encryption. +The default algorithm is "AES128". +<br>Currently, the following Algorithms are supported: + <ul> + <li>3DES + <li>CAST5 + <li>BLOWFISH + <li>AES128 + <li>AES192 + <li>AES256 + <li>TWOFISH + <li>TWOFISH128 + <li>ARCFOUR + <li>DES + <li>SERPENT128 + <li>SERPENT192 + <li>SERPENT256 + <li>RFC2268_40 + <li>SEED + <li>CAMELLIA128 + <li>CAMELLIA192 + <li>CAMELLIA256 + </ul> + <br> + The actual availability of an algorithms depends on which ones + are compiled into libgcrypt. Note that some versions of libgcrypt + simply abort the process (rsyslogd in this case!) if a supported + algorithm is select but not available due to libgcrypt build + settings. There is nothing rsyslog can do against this. So in + order to avoid production downtime, always check carefully when + you change the algorithm. +</li> +<li><b>cry.mode</b> <Algorithm Mode><br> +The encryption mode to be used. Default ist Cipher Block Chaining (CBC). +Note that not all encryption modes can be used together with all +algorithms. +<br>Currently, the following modes are supported: + <ul> + <li>ECB + <li>CFB + <li>CBC + <li>STREAM + <li>OFB + <li>CTR + <li>AESWRAP + </ul> +<li><b>cry.key</b> <encryption key><br> + TESTING AID, NOT FOR PRODUCTION USE. This uses the KEY specified + inside rsyslog.conf. This is the actual key, and as such this mode + is highly insecure. However, it can be useful for intial testing + steps. This option may be removed in the future. +</li> +<li><b>cry.keyfile</b> <filename><br> + Reads the key from the specified file. The file must contain the key, only, + no headers or other meta information. Keyfiles can be generated via the + rscrytool utility. +</li> +<li><b>cry.keyprogram</b> <path to program><br> + If given, the key is provided by a so-called "key program". This program + is executed and must return the key to (as well as some meta information) + via stdout. The core idea of key programs is that using this interface the + user can implement as complex (and secure) method to obtain keys as + desired, all without the need to make modifications to rsyslog. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>currently none known +</li> +</ul> +<p><b>Samples:</b></p> +<p>This encrypts a log file. Default parameters are used, they key is +provided via a keyfile. +</p> +<textarea rows="3" cols="60"> +action(type="omfile" file="/var/log/somelog" + cry.provider="gcry" keyfile="/secured/path/to/keyfile") +</textarea> +Note that the keyfile can be generated via the rscrytool utility (see its +documentation for how to actually do that). + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2013 by +<a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/debug.html b/doc/debug.html index 6aeb7975..557ca6d3 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -49,8 +49,14 @@ FileTrace=vm.c FileTrace=expr.c"</li> <li><b>Debug</b> - if present, turns on the debug system and enables debug output <li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>OutputTidToStderr</b> - if present, makes rsyslog output information about +the thread id (tid) of newly create processesto stderr. Note that not necessarily +all new threads are reported (depends on the code, e.g. of plugins). This is +only available under Linux. This usually does NOT work when privileges have +been dropped (that's not a bug, but the way it is). <li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> </ul> +<p>Individual options are separated by spaces.</p> </ul> <h3>Why Environment Variables?</h3> <p>You may ask why we use environment variables for debug-system parameters and not @@ -70,6 +76,26 @@ rsyslog core, we get a number of data structures wrong. <p>For these reasons, we utilize environment variables to initialize and configure the debugging system. We understand this may be somewhat painful, but now you know there are at least some good reasons for doing so. +<p>HOWEVER, if you have a too hard time to set debug instructions using the environment +variables, there is a cure, described in the next paragraph. + +<h2>Enabling Debug via rsyslog.conf</h2> +<p>As described in the previous paragraph, enabling debug via rsyslog.conf +may not be perfect for some debugging needs, but basic debug output will work - and +that is what most often is requried. There are limited options available, but these +cover the most important use cases. +<p>Debug processing is done via legacy config statements. There currently +is no plan to move these over to the v6+ config system. Availabe settings are +<ul> +<li>$DebugFile <filename> - sets the debug file name +<li>$DebugLevel <0|1|2> - sets the respective debug level, where +0 means debug off, 1 is debug on demand activated (but debug mode off) +and 2 is full debug mode. +</ul> +<p>Note that in theory it is forbidden to specify these parameters more +than once. However, we do not enforce that and if it happens results +are undefined. + <h2>Getting debug information from a running Instance</h2> <p>It is possible to obtain debugging information from a running instance, but this requires some setup. We assume that the instance runs in the background, so debug output to @@ -138,7 +164,7 @@ instance of rsyslogd can be aborted by pressing ctl-c. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/history.html b/doc/history.html index a06aaf5d..57b64004 100644 --- a/doc/history.html +++ b/doc/history.html @@ -118,7 +118,25 @@ use case. On February, 28th rsyslog 3.12.0 was released, the first version to contain expression support. This also meant that rsyslog from that date on supported all syslog-ng major features, but had a number of major features exlusive to it. With 3.12.0, I consider -rsyslog fully superior to syslog-ng (except for platform support).</p><p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> +rsyslog fully superior to syslog-ng (except for platform support).</p> + +<p>Following the Fedora Developer's conference in Brno <b>2012</b>, rsyslog +got very serious on implementing <b>structured logging</b> in +project Lumberjack (CEE) style. Project Lumberjack was a much broader +effort and brought closer collaboration with the syslog-ng folks, which +helped to maintain and improve interoperability. In the +<b>late winter/spring/summer 2012</b> timeframe numerous engine enhancements +were made and plugins written (among them the first "official" interfaces +to the Linux audit subsystem). At the end of the year, this culminated in the +rsyslog 7, which not only implemented Lumberjack but also was the first one +to support full condition nesting in rsyslog.conf (and a ton of other features as +well). + +<p>In <b>spring 2013</b> major new security features were engineered, +namely anonymization support, as well as log file signing and +encryption capabilities. + +<p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> to get some more insight into the development and futures of rsyslog and syslog in general. Don't be shy to post to either the blog or the <a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog forums</a>.</p> @@ -126,4 +144,4 @@ Don't be shy to post to either the blog or the <ul> <li><a href="http://www.rsyslog.com/Topic4.phtml">the rsyslog change log</a></li> </ul> -</body></html>
\ No newline at end of file +</body></html> diff --git a/doc/imfile.html b/doc/imfile.html index 1594cdce..942fe531 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -14,7 +14,7 @@ a syslog message. A standard text file is a file consisting of printable characters with lines being delimited by LF.</p> <p>The file is read line-by-line and any line read is passed to -rsyslog's rule engine. The rule engine applies filter conditons and +rsyslog's rule engine. The rule engine applies filter conditions and selects which actions needs to be carried out. Empty lines are <b>not</b> processed, as they would result in empty syslog records. They are simply ignored.</p> @@ -49,9 +49,9 @@ releases of imfile may support per-file polling intervals, but currently this is not the case. If multiple PollingInterval statements are present in rsyslog.conf, only the last one is used.<br> A short poll interval provides more rapid message forwarding, but -requires more system ressources. While it is possible, we stongly +requires more system resources. While it is possible, we stongly recommend not to set the polling interval to 0 seconds. That will make -rsyslogd become a CPU hog, taking up considerable ressources. It is +rsyslogd become a CPU hog, taking up considerable resources. It is supported, however, for the few very unusual situations where this level may be needed. Even if you need quick response, 1 seconds should be well enough. Please note that imfile keeps reading files as long as @@ -61,15 +61,15 @@ nothing is left to be processed.</li> <p><b>Action Directives</b></p> <ul> -<li><strong>File /path/to/file</strong><br> +<li><strong>(required) File /path/to/file</strong><br> The file being monitored. So far, this must be an absolute name (no macros or templates)</li> -<li><span style="font-weight: bold;">Tag +<li><span style="font-weight: bold;">(required) Tag tag:</span><br> The tag to be used for messages that originate from this file. If you would like to see the colon after the tag, you need to specify it here (as shown above).</li> -<li><span style="font-weight: bold;">StateFile +<li><span style="font-weight: bold;">(required) StateFile <name-of-state-file></span><br> Rsyslog must keep track of which parts of the to be monitored file it already processed. This is done in the state file. This file always is @@ -77,7 +77,9 @@ created in the rsyslog working directory (configurable via $WorkDirectory). Be careful to use unique names for different files being monitored. If there are duplicates, all sorts of "interesting" things may happen. Rsyslog currently does not check if a name is -specified multiple times.</li> +specified multiple times. +Note that when $WorkDirectory is not set or set to a non-writable +location, the state file will not be generated.</li> <li><span style="font-weight: bold;">Facility facility</span><br> The syslog facility to be assigned to lines read. Can be specified in @@ -91,9 +93,8 @@ textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for "info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is "notice".</li> <li><b>PersistStateInterval</b> [lines]</b><br> -Available in 4.7.3+, 5.6.2+<br> Specifies how often the state file shall be written when processing the input -file. The default value is 0, which means a new state file is only written when +file. The <strong>default</strong> value is 0, which means a new state file is only written when the monitored files is being closed (end of rsyslogd execution). Any other value n means that the state file is written every time n file lines have been processed. This setting can be used to guard against message duplication due @@ -101,9 +102,11 @@ to fatal errors (like power fail). Note that this setting affects imfile performance, especially when set to a low value. Frequently writing the state file is very time consuming. <li><b>ReadMode</b> [mode]</b><br> -Available in 5.7.5+ -<li><b>MaxLinesAtOnce</b> [number]</b><br> -Available in 5.9.0+ +This mode should defined when having multiline messages. The value can range from 0-2 and determines the multiline detection method. +<br>0 (<strong>default</strong>) - line based (Each line is a new message) +<br>1 - paragraph (There is a blank line between log messages) +<br>2 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it) +<li><b>MaxLinesAtOnce</b> [number]</b> <br> This is useful if multiple files need to be monitored. If set to 0, each file will be fully processed and then processing switches to the next file @@ -111,17 +114,15 @@ will be fully processed and then processing switches to the next file [number] lines is processed in sequence for each file, and then the file is switched. This provides a kind of mutiplexing the load of multiple files and probably leads to a more natural distribution of events when multiple busy files -are monitored. The default is 1024. -<li><b>MaxSubmitAtOnce</b> [number]</b><br> -Available in 5.9.0+ +are monitored. The <strong>default</strong> is 1024. +<li><b>MaxSubmitAtOnce</b> [number]</b> <br> This is an expert option. It can be used to set the maximum input batch size that -imfile can generate. The default is 1024, which is suitable for a wide range of +imfile can generate. The <strong>default</strong> is 1024, which is suitable for a wide range of applications. Be sure to understand rsyslog message batch processing before you modify this option. If you do not know what this doc here talks about, this is a good indication that you should NOT modify the default. -<li><b>Ruleset</b> <ruleset><br> -Available in 5.7.5+, 6.1.5+ +<li><b>Ruleset</b> <ruleset> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> @@ -142,17 +143,17 @@ your distro puts rsyslog's config files). Note that only commands actually needed need to be specified. The second file uses less commands and uses defaults instead.<br> </p> -<textarea rows="15" cols="60">module(load="folder/to/rsyslog/plugins/imfile/.libs/imfile" PollingInterval="10") #needs to be done just once +<textarea rows="15" cols="60">module(load="imfile" PollingInterval="10") #needs to be done just once # File 1 input(type="imfile" File="/path/to/file1" -Tag="tag1" -StateFile="/var/spool/rsyslog/statefile1" -Severity="error" -Facility="local7") + Tag="tag1" + StateFile="statefile1" + Severity="error" + Facility="local7") # File 2 input(type="imfile" File="/path/to/file2" -Tag="tag2" -StateFile="/var/spool/rsyslog/statefile2") + Tag="tag2" + StateFile="statefile2") # ... and so on ... # </textarea> @@ -181,12 +182,16 @@ directive, no file monitoring will take place.</li> seconds</span><br> equivalent to: PollingInterva</li> <li><b>$InputFilePersistStateInterval</b> [lines]</b><br> +Available in 4.7.3+, 5.6.2+<br> equivalent to: PersistStateInterval <li><b>$InputFileReadMode</b> [mode]</b><br> +Available in 5.7.5+<br> equivalent to: ReadMode <li><b>$InputFileMaxLinesAtOnce</b> [number]</b><br> +Available in 5.9.0+<br> equivalent to: MaxLinesAtOnce <li>$InputFileBindRuleset <ruleset><br> +Available in 5.7.5+, 6.1.5+<br> equivalent to: Ruleset </li> </ul> <b>Caveats/Known Bugs:</b> @@ -207,8 +212,7 @@ your distro puts rsyslog's config files). Note that only commands actually needed need to be specified. The second file uses less commands and uses defaults instead.<br> </p> -<textarea rows="15" cols="60">$ModLoad imfile # -needs to be done just once +<textarea rows="15" cols="60">$ModLoad imfile # needs to be done just once # File 1 $InputFileName /path/to/file1 $InputFileTag tag1: diff --git a/doc/imjournal.html b/doc/imjournal.html new file mode 100644 index 00000000..a4b232e8 --- /dev/null +++ b/doc/imjournal.html @@ -0,0 +1,111 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Systemd Journal Input Module</title></head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Systemd Journal Input Module</h1> +<p><b>Module Name: imjournal</b></p> +<p><b>Author: </b>Milan Bartos +<mbartos@redhat.com></p> +<p><b>Description</b>:</p> +<p>Provides the ability to import structured log messages from systemd journal +to syslog.</p> +<p>Note that this module reads the journal database, what is considered a +relativly performance-intense operation. As such, the performance of a +configuration utilizing this +module may be notably slower then when using +<a href="imuxsock.html">imuxsock</a>. The journal provides imuxsock with a +copy of all "classical" syslog messages, however, it does not provide +structured data. If the latter is needed, imjournal must be used. Otherwise, +imjournal may be simply replaced by imuxsock. +<p>We suggest to check out our short presentation on +<a href="http://youtu.be/GTS7EuSdFKE">rsyslog journal integration</a> to +learn more details of anticipated use cases. + +<p><b>Warning:</b> Some versions of systemd journal have problems with database +corruption, which leads to the journal to return the same data endlessly +in a thight loop. This results in massive message duplication inside rsyslog +probably resulting in a denial-of-service when the system ressouces get +exhausted. This can be somewhat mitigated by using proper rate-limiters, but +even then there are spikes of old data which are endlessly repeated. By default, +ratelimiting is activated and permits to process 20,000 messages within 10 +minutes, what should be well enough for most use cases. If insufficient, use +the parameters described below to adjust the permitted volume. +<b>It is strongly recommended to use this plugin only if there +is hard need to do so.</b> + +<p><b>Configuration Directives</b>:</p> +<p><b>Module Directives</b></p> +<ul> +<li><b>PersistStateInterval</b> number-of-messages<br> +This is a global setting. It specifies how often should the journal state be persisted. +The persists happens after each <i>number-of-messages</i>. +This option is useful for rsyslog to start reding from the last journal message it read. + +<li><b>StateFile</b> /path/to/file<br> +This is a global setting. It specifies where the state file for persisting +journal state is located. + +<li><b>ratelimit.interval</b> seconds (default: 600)<br> +Specifies the interval in seconds onto which rate-limiting is to be applied. +If more than ratelimit.burst messages are read during that interval, further +messages up to the end of the interval are discarded. The number of messages +discarded is emitted at the end of the interval (if there were any discards). +<br>Setting this to value zero turns off ratelimiting. Note that it is +<b>not recommended to turn of ratelimiting</b>, except that you know for +sure journal database entries will never be corrupted. Without ratelimiting, +a corrupted systemd journal database may cause a kind of denial of service (we +are stressing this point as multiple users have reported us such problems +with the journal database - information current as of June 2013). + +<li><b>ratelimit.burst</b> messages (default: 20000)<br> +Specifies the maximum number of messages that can be emitted within the +ratelimit.interval interval. For futher information, see description there. + +<li><b>IgnorePreviousMessages</b> [<b>off</b>/on]<br> +This option specifies whether imjournal should ignore messages currently in +journal and read only new messages. This option is only used when there is +no StateFile to avoid message loss. +</ul> + +<b>Caveats/Known Bugs:</b> +<p> +<ul> +<li>As stated above, a corrupted systemd journal database can cause major +problems, depending on what the corruption results in. This is beyond the +control of the rsyslog team. +</ul> +</p> +<p><b>Sample:</b></p> +<p> +The following example shows pulling structured imjournal messages and saving them into /var/log/ceelog. +</p> +<textarea rows="11" cols="80"> +module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file") #load imjournal module +module(load="mmjsonparse") #load mmjsonparse module for structured logs + +template(name="CEETemplate" type="string" + string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n" + ) #template for messages + +action(type="mmjsonparse") +action(type="omfile" file="/var/log/ceelog" template="CEETemplate") +</textarea> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><b>$imjournalPersistStateInterval</b><br> +Equivalent to: PersistStateInterval</li> +<li><b>$imjournalStateFile</b><br> +Equivalent to: StateFile</li> +<li><b>$imjournalRatelimitInterval</b><br> +Equivalent to: ratelimit.interval</li> +<li><b>$imjournalRatelimitBurst</b><br> +Equivalent to: ratelimit.burst</li> +<li><strong>$ImjournalIgnorePreviousMessages</strong><br> +Equivalent to: ignorePreviousMessages</li> +</ul> + +</body> +</html> diff --git a/doc/imklog.html b/doc/imklog.html index 05292ddf..1f195b16 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -15,7 +15,10 @@ syslog engine.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><strong>$KLogInternalMsgFacility +<li><strong>LogPath</strong><br> +The path to the Kernel log. This value should only be changed if you really know what +you are doing.</li> +<li><strong>InternalMsgFacility <facility></strong><br> The facility which messages internally generated by imklog will have. imklog generates some messages of itself (e.g. on problems, startup and @@ -26,13 +29,54 @@ need to specify this configuratin directive - it is included primarily for few limited cases where it is needed for good reason. Bottom line: if you don't have a good idea why you should use this setting, do not touch it.</li> -<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility -[on/<span style="font-style: italic;">off</span>]<br> -</span>At least under BSD the kernel log may contain entries +<li><b>PermitNonKernelFacility [on/<i>off</i>]</b><br> +At least under BSD the kernel log may contain entries with non-kernel facilities. This setting controls how those are handled. The default is "off", in which case these messages are ignored. Switch it to on to submit non-kernel messages to rsyslog -processing.<span style="font-weight: bold;"></span></li> +processing.</li> +<li><b>ParseKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled and the kernel creates a timestamp for its log messages, this timestamp will be +parsed and converted into regular message time instead to use the receive time of the kernel +message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the +clock used by the kernel to create the timestamps is not supposed to be as accurate as the +monotonic clock required to convert it. Depending on the hardware and kernel, it can result +in message time differences between kernel and system messages which occurred at same time. +<li><b>KeepKernelTimeStamp</b> [on/<b>off</b>]<br> +If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin +of in each message rather than to remove it, when it could be parsed and converted into +local time for use as regular message time. Only used when <b>ParseKernelTimestamp</b> is on. +<li><b>ConsoleLogLevel</b> [<i>number</i>] +(former klogd -c option) -- sets the console log level. If specified, only messages with +up to the specified level are printed to the console. The default is -1, which means that +the current settings are not modified. To get this behavior, do not specify +ConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time +it is changed, the previous definition is re-set. The one activate will be that one that is +active when imklog actually starts processing. In short words: do not specify this +directive more than once! +</ul> +<b>Caveats/Known Bugs:</b> +<p>This is obviously platform specific and requires platform +drivers. +Currently, imklog functionality is available on Linux and BSD.</p> +<p>This module is <b>not supported on Solaris</b> and not needed there. +For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the kernel log. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling kernel messages.<br> +</p> +<textarea rows="4" cols="60">module(load="imklog") +</textarea> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><strong>$KLogInternalMsgFacility +<facility></strong><br> +equivalent to: InternalMsgFacility</li> +<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility +[on/<span style="font-style: italic;">off</span>]<br> +equivalent to: PermitNonKernelFacility</li> <li><span style="font-weight: bold;"></span>$DebugPrintKernelSymbols [on/<b>off</b>]<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -50,14 +94,7 @@ it except if you have a very good reason. If you have one, let us know because otherwise new versions will no longer support it.<br> Linux only, ignored on other platforms (but may be specified)</li> <li><b>$klogConsoleLogLevel</b> [<i>number</i>] -(former klogd -c option) -- sets the console log level. If specified, only messages with -up to the specified level are printed to the console. The default is -1, which means that -the current settings are not modified. To get this behavior, do not specify -$klogConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time -it is changed, the previous definition is re-set. The one activate will be that one that is -active when imklog actually starts processing. In short words: do not specify this -directive more than once! -<br><b>Linux only</b>, ignored on other platforms (but may be specified)</li> +<br>equivalent to: ConsoleLogLevel</li> <li><b>$klogUseSyscallInterface</b> [on/<b>off</b>] -- former klogd -s option<br> Linux only, ignored on other platforms (but may be specified)</li> @@ -65,40 +102,17 @@ Linux only, ignored on other platforms (but may be specified)</li> former klogd -2 option<br> Linux only, ignored on other platforms (but may be specified)<br style="font-weight: bold;"> </li> -<li><b>$klogParseKernelTimestamp</b> [on/<b>off</b>] -If enabled and the kernel creates a timestamp for its log messages, this timestamp will be -parsed and converted into regular message time instead to use the receive time of the kernel -message (as in 5.8.x and before). Default is to not parse the kernel timestamp, because the -clock used by the kernel to create the timestamps is not supposed to be as accurate as the -monotonic clock required to convert it. Depending on the hardware and kernel, it can result -in message time differences between kernel and system messages which occurred at same time. -</li> -<li><b>$klogKeepKernelTimestamp</b> [on/<b>off</b>] -If enabled, this option causes to keep the [timestamp] provided by the kernel at the begin -of in each message rather than to remove it, when it could be parsed and converted into -local time for use as regular message time. Only used, when $klogParseKernelTimestamp is on. -</li> +<li><b>$klogParseKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: ParseKernelTimeStamp</li> +<li><b>$klogKeepKernelTimeStamp</b> [on/<b>off</b>]<br> +equivalent to: KeepKernelTimeStamp</li> </ul> -<b>Caveats/Known Bugs:</b> -<p>This is obviously platform specific and requires platform -drivers. -Currently, imklog functionality is available on Linux and BSD.</p> -<p>This module is <b>not supported on Solaris</b> and not needed there. -For Solaris kernel input, use <a href="imsolaris.html">imsolaris</a>.</p> -<p><b>Sample:</b></p> -<p>The following sample pulls messages from the kernel log. All -parameters are left by default, which is usually a good idea. Please -note that loading the plugin is sufficient to activate it. No directive -is needed to start pulling kernel messages.<br> -</p> -<textarea rows="15" cols="60">$ModLoad imklog -</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imkmsg.html b/doc/imkmsg.html new file mode 100644 index 00000000..23b96147 --- /dev/null +++ b/doc/imkmsg.html @@ -0,0 +1,50 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>/dev/kmsg Log Input Module (imkmsg)</title> + +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>/dev/kmsg Log Input Module</h1> +<p><b>Module Name: imkmsg</b></p> +<p><b>Authors: </b>Rainer Gerhards +<rgerhards@adiscon.com><br /> +Milan Bartos +<mbartos@redhat.com></p> +<p><b>Description</b>:</p> +<p>Reads messages from the /dev/kmsg structured kernel log and submits them to the +syslog engine.</p> +<p> +The printk log buffer constains log records. These records are exported by /dev/kmsg +device as structured data in the following format:<br /> + "level,sequnum,timestamp;<message text>\n"<br /> +There could be continuation lines starting with space that contains key/value pairs.<br /> +<br /> +Log messages are parsed as necessary into rsyslog msg_t structure. Continuation lines are parsed +as json key/value pairs and added into rsyslog's message json representation. +</p> +<p><b>Configuration Directives</b>:</p> +<p>This module has no configuration directives.</p> +<b>Caveats/Known Bugs:</b> +<p>This module can't be used together with imklog module. When using one of them, make sure the other +one is not enabled.</p> +<p>This is Linux specific module and requires /dev/kmsg device with structured kernel logs.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the /dev/kmsg log device. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling messages.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imkmsg +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/impstats.html b/doc/impstats.html index 64b04a30..8db9c6f6 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -16,26 +16,66 @@ availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version). <p>The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). -Be sure that your configuration records the counter messages (default is syslog.info). +Be sure that your configuration records the counter messages (default is syslog.=info). +Besides logging to the regular syslog stream, the module can also be configured to +write statistics data into a (local) file. <p>Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be noticable (for high-load environments). <p>The rsyslog website has an updated overview of available <a href="http://rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a>. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Module Confguration Parameters</b>:</p> +<p>This module supports module parameters, only. <ul> -<li>$PStatInterval <Seconds><br> -Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the -actual interval may be a bit longer. We do not try to be precise and so the interval is -actually a sleep period which is entered after generating all messages. So the actual -interval is what is configured here plus the actual time required to generate messages. -In general, the difference should not really matter. -<li>$PStatFacility <numerical facility><br> -The numerical syslog facility code to be used for generated messages. Default -is 5 (syslog).This is useful for filtering messages.</li> -<li>$PStatSeverity <numerical severity><br> -The numerical syslog severity code to be used for generated messages. Default -is 6 (info).This is useful for filtering messages.</li> + <li><strong>interval </strong>[seconds] (default 300 [5minutes])<br> + Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the + actual interval may be a bit longer. We do not try to be precise and so the interval is + actually a sleep period which is entered after generating all messages. So the actual + interval is what is configured here plus the actual time required to generate messages. + In general, the difference should not really matter. + <br></li> + <li><strong>facility </strong>[templateName]<br> + The numerical syslog facility code to be used for generated messages. Default + is 5 (syslog). This is useful for filtering messages. + <br></li> + <li><strong>severity </strong>[templateName]<br> + The numerical syslog severity code to be used for generated messages. Default + is 6 (info).This is useful for filtering messages. + <br></li> + <li><strong>format </strong>[json/cee/<b>legacy</b>](rsyslog v6.3.8+ only)<br> + Specifies the format of emitted stats messages. The default of "legacy" is + compatible with pre v6-rsyslog. The other options provide support for + structured formats (note the "cee" is actually "project lumberack" logging). + <br></li> + <li><strong>log.syslog </strong>[<b>on</b>/off] - available since 7.3.6<br> + This is a boolean setting specifying if data should be sent + to the usual syslog stream. This is useful if custom formatting + or more elaborate processing is desired. However, output is placed + under the same restrictions as regular syslog data, especially in + regard to the queue position (stats data may sit for an extended + period of time in queues if they are full).<br></li> + <li><strong>log.file </strong>[file name] - available since 7.3.6<br> + If specified, statistics data is written the specified file. For + robustness, this should be a local file. The file format cannot be + customized, it consists of a date header, followed by a colon, + followed by the actual statistics record, all on one line. Only + very limited error handling is done, so if things go wrong stats + records will probably be lost. Logging to file an be a useful + alternative if for some reasons (e.g. full queues) the regular + syslog stream method shall not be used solely. Note that turning + on file logging does NOT turn of syslog logging. If that is desired + log.syslog="off" must be explicitely set. + <br></li> + +</ul> +<p><b>Legacx Configuration Directives</b>:</p> +A limited set of parameters can also be set via the legacy configuration +syntax. Note that this is intended as an upward compatibilit layer, so +newer features are intentionally <b>not</b> available via legacy directives. +<ul> +<li>$PStatInterval <Seconds> - same as the "interval" parameter. +<li>$PStatFacility <numerical facility> - same as the "facility" parameter. +<li>$PStatSeverity <numerical severity> - same as the "severity" parameter. <li>$PStatJSON <on/<b>off</b>> (rsyslog v6.3.8+ only)<br> If set to on, stats messages are emitted as structured cee-enhanced syslog. If set to off, legacy format is used (which is compatible with pre v6-rsyslog). @@ -45,23 +85,45 @@ set to off, legacy format is used (which is compatible with pre v6-rsyslog). <ul> <li>This module MUST be loaded right at the top of rsyslog.conf, otherwise stats may not get turned on in all places.</li> -<li>experimental code</li> </ul> -<p><b>Sample:</b></p> +<p><b>Samples:</b></p> <p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:<br> </p> -<textarea rows="8" cols="60">$ModLoad impstats +<textarea rows="5" cols="60">module(load="impstats" interval="600" severity="7") + +# to actually gather the data: +syslog.=debug /var/log/rsyslog-stats +</textarea> +<p><b>Legacy Sample:</b></p> +<p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:</p> +<textarea rows="6" cols="60">$ModLoad impstats $PStatInterval 600 $PStatSeverity 7 -syslog.debug /var/log/rsyslog-stats +syslog.=debug /var/log/rsyslog-stats </textarea> +<p>In the next sample, the default interval of 5 minutes is used. However, this time +stats data is NOT emitted to the syslog stream but to a local file instead. +<p> +<textarea rows="3" cols="70">module(load="impstats" interval="600" severity="7" + log.syslog="off" /* need to turn log stream logging off! */ + log.file="/path/to/local/stats.log") +</textarea> +<p>And finally, we log to both the regular syslog log stream as well as a file. +Within the log stream, we forward the data records to another server: +<p> +<textarea rows="4" cols="70">module(load="impstats" interval="600" severity="7" + log.file="/path/to/local/stats.log") + +syslog.=debug @central.example.net +</textarea> + <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imptcp.html b/doc/imptcp.html index d301b76f..aece428d 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -13,18 +13,34 @@ <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via plain TCP syslog. This is a specialised input plugin tailored for high performance on Linux. It will -probably not run on any other platform. Also, it does no provide TLS services. +probably not run on any other platform. Also, it does not provide TLS services. Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a>. <p>This module has no limit on the number of listeners and sessions that can be used. -<p>Multiple receivers may be configured by -specifying $InputPTCPServerRun multiple times. </p> <p><b>Configuration Directives</b>:</p> <p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> -<li><b>AddTLFrameDelimiter</b> <Delimiter><br> + +<p><b>Module Parameters</b>:</p> +<p>These paramters can be used with the "module()" statement. They apply +globaly to all inputs defined by the module. +<ul> +<li>Threads <number><br> +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +helper threads (but not more than there are CPUs/Cores) can help improving +performance. The default value is two, which means there +is a default thread count of three (the main input thread plus two +helpers). +No more than 16 threads can be set (if tried to, rsyslog always resorts to 16). +</ul> +<p><b>Input Parameters</b>:</p> +<p>These parameters can be used with the "input()" statement. They apply to the +input they are specified with. +<ul> +<li><b>AddtlFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -78,13 +94,15 @@ name is not strictly necessary, but can be useful to apply filtering based on wh the message was received from. <li><b>Ruleset</b> <name><br> Binds specified ruleset to next server defined. -<!--<li>$InputPTCPHelperThreads <number><br> -Number of helper worker threads to process incoming messages. These -threads are utilized to pull data off the network. On a busy system, additional -helper threads (but not more than there are CPUs/Cores) can help improving -performance. The default value is two.--> <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> @@ -93,13 +111,11 @@ On multi-homed machines, specifies to which local address the listerner should b <p><b>Sample:</b></p> <p>This sets up a TCP server on port 514:<br> </p> -<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imptcp/.libs/imptcp") # needs to be done just once +<textarea rows="4" cols="60">module(load="/folder/to/rsyslog/plugins/imptcp/.libs/imptcp") # needs to be done just once input(type="imptcp" port="514") </textarea> <p><b>Legacy Configuration Directives</b>:</p> -<p>This plugin has config directives similar named as imtcp, but they all have <b>P</b>TCP in -their name instead of just TCP. Note that only a subset of the parameters are supported. <ul> <li>$InputPTCPServerAddtlFrameDelimiter <Delimiter><br> Equivalent to: AddTLFrameDelimiter</li> @@ -122,11 +138,8 @@ Equivalent to: Port </li> Equivalent to: Name </li> <li>$InputPTCPServerBindRuleset <name><br> Equivalent to: Ruleset </li> -<li>$InputPTCPHelperThreads <number><br> -Number of helper worker threads to process incoming messages. These -threads are utilized to pull data off the network. On a busy system, additional -helper threads (but not more than there are CPUs/Cores) can help improving -performance. The default value is two. +<li>$InputPTCPServerHelperThreads <number><br> +Equivalent to: threads </li> <li>$InputPTCPServerListenIP <name><br> Equivalent to: Address </li> </ul> @@ -137,7 +150,7 @@ Equivalent to: Address </li> <p><b>Sample:</b></p> <p>This sets up a TCP server on port 514:<br> </p> -<textarea rows="15" cols="60">$ModLoad imptcp # +<textarea rows="3" cols="60">$ModLoad imptcp # needs to be done just once $InputPTCPServerRun 514 </textarea> @@ -146,7 +159,7 @@ $InputPTCPServerRun 514 <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2010-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imrelp.html b/doc/imrelp.html index 80ddfd53..f7fcc4b3 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -30,14 +30,12 @@ Clients send messages to the RELP server via omrelp.</p> <p><b>Configuration Directives</b>:</p> <ul> -<li><b>Ruleset</b> <name> (available in 6.3.6+)</br> -Binds the specified ruleset to all RELP listeners. <li><b>Port</b> <port><br> Starts a RELP server on selected port</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>see description</li> +<li>ruleset can only be bound via legacy configuration format</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> @@ -47,14 +45,14 @@ not specific ones. This is due to a currently existing limitation in librelp. <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> </p> -<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imrelp/.libs/imrelp") # needs to be done just once +<textarea rows="15" cols="60">module(load="imrelp") # needs to be done just once input(type="imrelp" port="20514") </textarea> <p><b>Legacy Configuration Directives</b>:</p> <ul> <li>InputRELPServerBindRuleset <name> (available in 6.3.6+)</br> -equivalent to: RuleSet +Binds the specified ruleset to all RELP listeners. <li>InputRELPServerRun <port><br> equivalent to: Port</li> </ul> diff --git a/doc/imtcp.html b/doc/imtcp.html index 649b08f8..b9f0b056 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -17,13 +17,11 @@ Encryption is natively provided by selecting the approprioate network stream driver and can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> (an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> -<p>Multiple receivers may be configured by specifying -$InputTCPServerRun multiple times. This is available since version 4.3.1, earlier -versions do NOT support it. -</p> + <p><b>Configuration Directives</b>:</p> +<p><b>Global Directives</b>:</p> <ul> -<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +<li><b>AddtlFrameDelimiter</b> <Delimiter><br> This directive permits to specify an additional frame delimiter for plain tcp syslog. The industry-standard specifies using the LF character as frame delimiter. Some vendors, notable Juniper in their NetScreen products, use an invalid frame delimiter, in Juniper's @@ -43,7 +41,7 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. -<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +<li><b>DisableLFDelimiter</b> <on/<b>off</b>><br> Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames. However, some users brought up the case that it may be useful to define a different delimiter and totally disable LF as a delimiter (the use case named were multi-line messages). This mode @@ -51,16 +49,14 @@ is non-standard and will probably come with a lot of problems. However, as there for it and it is relatively easy to support, we do so. Be sure to turn this setting to "on" only if you exactly know what you are doing. You may run into all sorts of troubles, so be prepared to wrangle with that! -<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +<li><b>NotifyOnConnectionClose</b> [on/<b>off</b>]<br> instructs imtcp to emit a message if the remote peer closes a connection.<br> <b>Important:</b> This directive is global to all listeners and must be given right after loading imtcp, otherwise it may have no effect.</li> -<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +<li><b>KeepAlive</b> <on/<b>off</b>><br> enable of disable keep-alive packets at the tcp socket layer. The default is to disable them.</li> -<li><b>$InputTCPServerRun</b> <port><br> -Starts a TCP server on selected port</li> -<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +<li><b>FlowControl</b> <<b>on</b>/off><br> This setting specifies whether some message flow control shall be exercised on the related TCP input. If set to on, messages are handled as "light delayable", which means the sender is throttled a bit when the queue becomes near-full. This is done in order @@ -69,29 +65,93 @@ may have some undesired effect in some configurations. Still, we consider this a a useful setting and thus it is the default. To turn the handling off, simply configure that explicitely. </li> -<li><b>$InputTCPMaxListeners</b> <number><br> +<li><b>MaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li><b>$InputTCPMaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> -<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +<li><b>MaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>StreamDriver.Mode</b> <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> -<li><b>$InputTCPServerInputName</b> <name><br> -Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a -name is not strictly necessary, but can be useful to apply filtering based on which input -the message was received from. -<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +<li><b>StreamDriver.AuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> -<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +<li><b>PermittedPeer</b> <id-string><br> Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected -AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> -<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeer may not be set in anonymous modes. +<br>PermittedPeer may be set either to a single peer or an array of peers either of type IP or name, depending on the tls certificate. +<br>Single peer: PermittedPeer="127.0.0.1" +<br>Array of peers: PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."]</li> +</ul> +<p><b>Action Directives</b>:</p> +<ul> +<li><b>Port</b> <port><br> +Starts a TCP server on selected port</li> +<li><b>Name</b> <name><br> +Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a +name is not strictly necessary, but can be useful to apply filtering based on which input +the message was received from. +<li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> -<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +<li><b>SupportOctetCountedFraming</b> <<b>on</b>|off><br> If set to "on", the legacy octed-counted framing (similar to RFC5425 framing) is activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> +(which includes the functionality of imtcp)</li> +</ul> +<p><b>Example:</b></p> +<p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> +</p> +<textarea rows="15" cols="60">module(load="imtcp" MaxSessions="500") +input(type="imtcp" port="514") +</textarea> +<p>Note that the global parameters (here: max sessions) need to be set when the module is loaded. Otherwise, the parameters will not apply. +</p> + +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li><b>$InputTCPServerAddtlFrameDelimiter <Delimiter></b><br> +equivalent to: AddtlFrameDelimiter +<li><b>$InputTCPServerDisableLFDelimiter</b> <on/<b>off</b>> (available since 5.5.3)<br> +equivalent to: DisableLFDelimiter +<li><b>$InputTCPServerNotifyOnConnectionClose</b> [on/<b>off</b>] (available since 4.5.5)<br> +equivalent to: NotifyOnConnectionClose<br> +</li> +<li><b>$InputTCPServerKeepAlive</b> <on/<b>off</b>><br> +equivalent to: KeepAlive</li> +<li><b>$InputTCPServerRun</b> <port><br> +equivalent to: Port</li> +<li><b>$InputTCPFlowControl</b> <<b>on</b>/off><br> +equivalent to: FlowControl +</li> +<li><b>$InputTCPMaxListeners</b> <number><br> +equivalent to: MaxListeners</li> +<li><b>$InputTCPMaxSessions</b> <number><br> +equivalent to: MaxSessions</li> +<li><b>$InputTCPServerStreamDriverMode</b> <number><br> +equivalent to: StreamDriver.Mode</li> +<li><b>$InputTCPServerInputName</b> <name><br> +equivalent to: Name +<li><b>$InputTCPServerStreamDriverAuthMode</b> <mode-string><br> +equivalent to: StreamDriver.AuthMode</li> +<li><b>$InputTCPServerStreamDriverPermittedPeer</b> <id-string><br> +equivalent to: PermittedPeer.</li> +<li><b>$InputTCPServerBindRuleset</b> <ruleset><br> +equivalent to: Ruleset</a>.</li> +<li><b>$InputTCPSupportOctetCountedFraming</b> <<b>on</b>|off><br> +equivalent to: SupportOctetCountedFraming +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> diff --git a/doc/imudp.html b/doc/imudp.html index 3512d474..a8dbca31 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -18,8 +18,8 @@ multiple input actions. </p> -<p><b>Configuration Directives</b>:</p> -<p><b>Global Directives</b>:</p> +<p><b>Configuration Parameters</b>:</p> +<p><b>Module Parameters</b>:</p> <ul> <li><b>TimeRequery</b> <nbr-of-times><br> this is a performance @@ -33,30 +33,84 @@ the value, the less precise the timestamp. <li><b>SchedulingPolicy</b> <rr/fifo/other><br> Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time -processing under Linux (and thus reduce chance of packet loss). Available since 4.7.4+, 5.7.3+, 6.1.3+. +processing under Linux (and thus reduce chance of packet loss). <li><b>SchedulingPriority</b> <number><br> -Scheduling priority to use. Available since 4.7.4+, 5.7.3+, 6.1.3+. +Scheduling priority to use. </ul> -<p><b>Action Directives</b>:</p> +<p><b>Input Parameters</b>:</p> <ul> <li><b>Address</b> <IP><br> local IP address (or name) the UDP listens should bind to</li> <li><b>Port</b> <port><br> -default 514, start UDP server on this port</li> +default 514, start UDP server on this port. Either a single port can be specified or an array of ports. If multiple ports are specified, a listener will be automatically started for each port. Thus, no additional inputs need to be configured. +<br>Single port: Port="514" +<br>Array of ports: Port=["514","515","10514","..."]</li> <li><b>Ruleset</b> <ruleset><br> Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> +<li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting +interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number +of seconds (5 recommended) to activate rate-limiting. +</li> +<li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting +burst in number of messages. Default is 10,000. +</li> +<li><b>InputName</b> [name] - (available since 7.3.9) specifies the value of +the inputname. In older versions, this was always "imudp" for all listeners, +which still i the default. +Starting with 7.3.9 it can be set to different values for each listener. +Note that when a single input statement defines multipe listner ports, the +inputname will be the same for all of them. If you want to differentiate in that +case, use "InputName.AppendPort" to make them unique. +Note that the "InputName" parameter can be an empty string. In that case, the +corresponding inputname property will obviously also be the empty string. This +is primarily meant to be used togehter with "InputName.AppendPort" to set the +inputname equal to the port. +</li> +<li><b>InputName.AppendPort</b> [on/<b>off</b>] - (available since 7.3.9) +appends the port the the inputname. Note that when no inputname is specified, +the default of "imudp" is used and the port is appended to that default. So, +for example, a listner port of 514 in that case will lead to an inputname +of "imudp514". The ability to append a port is most useful when multiple ports +are defined for a single input and each of the inputnames shall be unique. +Note that there currently is no differentiation between IPv4/v6 listeners on +the same port. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>currently none known</li> +<li>Scheduling parameters are set <b>after</b> privileges have been dropped. +In most cases, this means that setting them will not be possible after +privilege drop. This may be worked around by using a sufficiently-privileged +user account. +</li> </ul> -<p><b>Sample:</b></p> +<p><b>Samples:</b></p> <p>This sets up an UPD server on port 514:<br> </p> -<textarea rows="15" cols="60">module(load="/folder/to/rsyslog/plugins/imudp/.libs/imudp") # needs to be done just once +<textarea rows="3" cols="60">module(load="imudp") # needs to be done just once input(type="imudp" port="514") </textarea> +<p>In the next example, we set up three listeners at ports 10514, 10515 and 10516 +and assign a listner name of "udp" to it, followed by the port number: +</p> +<textarea rows="4" cols="60">module(load="imudp") +input(type="imudp" port=["10514","10515","10516"] + inputname="udp" inputname.appendPort="on") +</textarea> + +<p>The next example is almost equal to the previous one, but +now the inputname property will just be set to the port number. +So if a message was received on port 10515, the input name will be +"10515" in this example whereas it was "udp10515" in the previous one. +Note that to do that we set the inputname to the empty string. +</p> +<textarea rows="4" cols="60">module(load="imudp") +input(type="imudp" port=["10514","10515","10516"] + inputname="" inputname.appendPort="on") +</textarea> + + <p><b>Legacy Configuration Directives</b>:</p> <p>Multiple receivers may be configured by specifying $UDPServerRun multiple times. @@ -70,28 +124,25 @@ equivalent to: Port </li> equivalent to: TimeRequery <li>$InputUDPServerBindRuleset <ruleset><br> equivalent to: Ruleset </li> -<li>$IMUDPSchedulingPolicy <rr/fifo/other><br> +<li>$IMUDPSchedulingPolicy <rr/fifo/other> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPolicy -<li>$IMUDPSchedulingPriority <number><br> +<li>$IMUDPSchedulingPriority <number> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPriority </ul> -<b>Caveats/Known Bugs:</b> -<ul> -<li>currently none known</li> -</ul> <p><b>Sample:</b></p> <p>This sets up an UPD server on port 514:<br> </p> -<textarea rows="15" cols="60">$ModLoad imudp # needs to be done just once +<textarea rows="3" cols="60">$ModLoad imudp # needs to be done just once $UDPServerRun 514 </textarea> + <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2009-2013 by +<a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index bd207a37..0affe8c3 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -65,6 +65,12 @@ you must turn it on (via SysSock.Annotate and Annotate). <li><b>SysSock.IgnoreTimestamp</b> [<b>on</b>/off]<br> Ignore timestamps included in the messages, applies to messages received via the system log socket. </li> +<li><b>SysSock.IgnoreOwnMessages</b> [<b>on</b>/off] (available since 7.3.7)<br> +Ignores messages that originated from the same instance of rsyslogd. There usually +is no reason to receive messages from ourselfs. This setting is vital +when writing messages to the Linux journal. See <a href="omjournal.html">omjournal</a> +module documentation for a more in-depth description. +</li> <li><b>SysSock.Use</b> (imuxsock) [on/<b>off</b>] do NOT listen for the local log socket. This is most useful if you run multiple instances of rsyslogd where only one shall handle the system log socket. @@ -77,7 +83,7 @@ to the system log socket. <li><b>SysSock.UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0. +with earlier versions of rsyslog. </li> <li><b>SysSock.RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. @@ -92,17 +98,33 @@ messages that shall be rate-limited. </li> <li><b>SysSock.Annotate</b> <on/<b>off</b>> turn on annotation/trusted properties for the system log socket.</li> +<li><b>SysSock.ParseTrusted</b> <on/<b>off</b>> if Annotation is turned on, create +JSON/lumberjack properties out of the trusted properties (which can be accessed +via RainerScript JSON Variables, e.g. "$!pid") instead of adding them to the message. +</li> +<li><b>SysSock.Unlink</b> <<b>on</b>/off> (available since 7.3.9)<br> +if turned on (default), the system socket is unlinked and re-created when +opened and also unlinked when finally closed. Note that this setting has +no effect when running under systemd control (because systemd handles +the socket). +</li> </ul> <p><b>Input Instance Parameters</b></p> <ul> <li><b>IgnoreTimestamp</b> [<b>on</b>/off] <br>Ignore timestamps included in the message. Applies to the next socket being added.</li> +<li><b>IgnoreOwnMessages</b> [<b>on</b>/off] (available since 7.3.7)<br> +Ignore messages that originated from the same instance of rsyslogd. There usually +is no reason to receive messages from ourselfs. This setting is vital +when writing messages to the Linux journal. See <a href="omjournal.html">omjournal</a> +module documentation for a more in-depth description. +</li> <li><b>FlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the next socket.</li> <li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number -of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen in 5.9.6+, +of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> @@ -112,7 +134,7 @@ burst in number of messages. Default is 200. <li><b>RateLimit.Severity</b> [numerical severity] - specifies the severity of messages that shall be rate-limited. </li> -<!--<li><b>LocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified +<!--<li><b>LocalIPIF</b> [interface name] - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. @@ -120,7 +142,7 @@ the default of "127.0.0.1" is used. <li><b>UsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall be obtained from the log socket itself. If so, the TAG part of the message is rewritten. It is recommended to turn this option on, but the default is "off" to keep compatible -with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> +with earlier versions of rsyslog. </li> <li><b>UseSysTimeStamp</b> [<b>on</b>/off] instructs imuxsock to obtain message time from the system (via control messages) insted of using time recorded inside the message. This may be most useful in combination with systemd. Note: @@ -139,7 +161,7 @@ being reset to "off" after the Socket directive, so if you would have for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those -sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> +sockets even though their directories do not yet exist.</li> <li><b>Socket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> <li><b>HostName</b> <hostname> permits to override the hostname that shall be used inside messages taken from the <b>next</b> Socket socket. Note that @@ -148,6 +170,15 @@ will only affect the next one and then automatically be reset. This functionalit that the local hostname can be overridden in cases where that is desired.</li> <li><b>Annotate</b> <on/<b>off</b>> turn on annotation/trusted properties for the non-system log socket in question.</li> +<li><b>ParseTrusted</b> <on/<b>off</b>> equivalent to the SysSock.ParseTrusted module +parameter, but applies to the input that is being defined. +<li><b>Unlink</b> <<b>on</b>/off> (available since 7.3.9)<br> +if turned on (default), the socket is unlinked and re-created when +opened and also unlinked when finally closed. Set it to off if you +handle socket creation yourself. Note that handling socket creation +oneself has the advantage that a limited amount of messages may be +queued by the OS if rsyslog is not running. +</li> </ul> <b>Caveats/Known Bugs:</b><br> @@ -160,12 +191,20 @@ change the array size in imuxsock.c. <p>The following sample is the minimum setup required to accept syslog messages from applications running on the local system.<br> </p> -<textarea rows="2" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +<textarea rows="2" cols="70">module(load="imuxsock" # needs to be done just once SysSock.FlowControl="on") # enable flow control (use if needed) </textarea> + +<p>The following sample is similiar to the first one, but enables trusted +properties, which are put into JSON/lumberjack variables. +<br> +</p> +<textarea rows="2" cols="70">module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on") +</textarea> + <p>The following sample is a configuration where rsyslogd pulls logs from two jails, and assigns different hostnames to each of the jails: </p> -<textarea rows="6" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock") # needs to be done just once +<textarea rows="6" cols="70">module(load="imuxsock") # needs to be done just once input(type="imuxsock" HostName="jail1.example.net" Socket="/jail/1/dev/log") input(type="imuxsock" HostName="jail2.example.net" Socket="/jail/2/dev/log") @@ -176,18 +215,18 @@ system. As rsyslogd starts up before the sshd, it needs to create the socket directories, because it otherwise can not open the socket and thus not listen to openssh messages. Note that it is vital not to place any other socket between the CreatePath and the Socket.</p> -<textarea rows="6" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock") # needs to be done just once +<textarea rows="6" cols="70">module(load="imuxsock") # needs to be done just once input(type="imuxsock" Socket="/var/run/sshd/dev/log" CreatePath="on") </textarea> <p>The following sample is used to turn off input rate limiting on the system log socket. -<textarea rows="4" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +<textarea rows="4" cols="70">module(load="imuxsock" # needs to be done just once SysSock.RateLimit.Interval="0") # turn off rate limiting </textarea> <p>The following sample is used activate message annotation and thus trusted properties on the system log socket. -<textarea rows="4" cols="70">module(load="/folder/to/rsyslog/plugins/imuxsock/.libs/imuxsock" # needs to be done just once +<textarea rows="4" cols="70">module(load="imuxsock" # needs to be done just once SysSock.Annotate="on") </textarea> @@ -195,39 +234,43 @@ SysSock.Annotate="on") <p><b>Legacy Configuration Directives</b>:</p> <ul> <li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] -<br>Please see: IgnoreTimestamp.</li> -<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - Please see: FlowControl .</li> -<li><b>$IMUXSockRateLimitInterval</b> [number] - Please see: RateLimit.Interval +<br>equivalent to: IgnoreTimestamp.</li> +<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - equivalent to: FlowControl .</li> +<li><b>$IMUXSockRateLimitInterval</b> [number] - equivalent to: RateLimit.Interval </li> -<li><b>$IMUXSockRateLimitBurst</b> [number] - Please see: RateLimit.Burst +<li><b>$IMUXSockRateLimitBurst</b> [number] - equivalent to: RateLimit.Burst </li> -<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - Please see: RateLimit.Severity +<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - equivalent to: RateLimit.Severity </li> <li><b>$IMUXSockLocalIPIF</b> [interface name] - (available since 5.9.6) - if provided, the IP of the specified interface (e.g. "eth0") shall be used as fromhost-ip for imuxsock-originating messages. If this directive is not given OR the interface cannot be found (or has no IP address), the default of "127.0.0.1" is used. </li> -<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - Please see: UsePIDFromSystem.</li> -<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] Please see: UseSysTimeStamp .<br> +<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$InputUnixListenSocketUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: UseSysTimeStamp .<br> <li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> -Please see: SysSock.IgnoreTimestamp.</li> -<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] Please see: SysSock.Use</li> -<li><b>$SystemLogSocketName</b> <name-of-socket> Please see: SysSock.Name</li> -<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - Please see: SysSock.FlowControl.</li> -<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - Please see: SysSock.UsePIDFromSystem.</li> -<li><b>$SystemLogRateLimitInterval</b> [number] - Please see: SysSock.RateLimit.Interval. +equivalent to: SysSock.IgnoreTimestamp.</li> +<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] equivalent to: SysSock.Use</li> +<li><b>$SystemLogSocketName</b> <name-of-socket> equivalent to: SysSock.Name</li> +<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - equivalent to: SysSock.FlowControl.</li> +<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - equivalent to: SysSock.UsePIDFromSystem. +<br>This option was introduced in 5.7.0.</li> +<li><b>$SystemLogRateLimitInterval</b> [number] - equivalent to: SysSock.RateLimit.Interval. </li> -<li><b>$SystemLogRateLimitBurst</b> [number] - Please see: SysSock.RateLimit.Burst +<li><b>$SystemLogRateLimitBurst</b> [number] - equivalent to: SysSock.RateLimit.Burst </li> -<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - Please see: SysSock.RateLimit.Severity +<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - equivalent to: SysSock.RateLimit.Severity </li> -<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] Please see: SysSock.UseSysTimeStamp. -<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - Please see: CreatePath</li> -<li><b>$AddUnixListenSocket</b> <name-of-socket> Please see: Socket </li> -<li><b>$InputUnixListenSocketHostName</b> <hostname> Please see: HostName.</li> -<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> Please see: Annotate.</li> -<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> Please see: SysSock.Annotate.</li> +<li><b>$SystemLogUseSysTimeStamp</b> [<b>on</b>/off] equivalent to: SysSock.UseSysTimeStamp. +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - equivalent to: CreatePath +<br>[available since 4.7.0 and 5.3.0]</li> +<li><b>$AddUnixListenSocket</b> <name-of-socket> equivalent to: Socket </li> +<li><b>$InputUnixListenSocketHostName</b> <hostname> equivalent to: HostName.</li> +<li><b>$InputUnixListenSocketAnnotate</b> <on/<b>off</b>> equivalent to: Annotate.</li> +<li><b>$SystemLogSocketAnnotate</b> <on/<b>off</b>> equivalent to: SysSock.Annotate.</li> +<li><b>$SystemLogSocketParseTrusted</b> <on/<b>off</b>> equivalent to: SysSock.ParseTrusted.</li> </ul> <b>Caveats/Known Bugs:</b><br> @@ -280,7 +323,7 @@ $SystemLogSocketAnnotate on <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html new file mode 100644 index 00000000..d72810f1 --- /dev/null +++ b/doc/lookup_tables.html @@ -0,0 +1,205 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Lookup Tables</title> +</head> + +<body> +<h1>Lookup Tables</h1> + +<p><b><font color="red">NOTE: this is</font> proposed functionality, which is +<font color="red">NOT YET IMPLEMENTED</font>!</b> + +<p><b>Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices).</b> +<p>The base idea is to use a message variable as an index into a table which then +returns another value. For example, $fromhost-ip could be used as an index, with +the table value representing the type of server or the department or remote office +it is located in. A main point with lookup tables is that the lookup is very fast. +So while lookup tables can be emulated with if-elseif constructs, they are generally +much faster. Also, it is possible to reload lookup tables during rsyslog runtime without +the need for a full restart. +<p>The lookup tables itself exists in a separate configuration file (one per table). This +file is loaded on rsyslog startup and when a reload is requested. +<p>There are different types of lookup tables: +<ul> +<li><b>string</b> - the value to be looked up is an arbitrary string. Only exact +some strings match. +<li><b>array</b> - the value to be looked up is an integer number from a consequtive set. +The set does not need to start at zero or one, but there must be no number missing. So, for example +5,6,7,8,9 would be a valid set of index values, while 1,2,4,5 would not be (due to missing +2). +A match happens if the requested number is present. +<li><b>sparseArray</b> - the value to be looked up is an integer value, but there may +be gaps inside the set of values (usually there are large gaps). A typical use case would +be the matching of IPv4 address information. A match happens on the first value that is +less than or equal to the requested value. +</ul> +<p>Note that index integer numbers are represented by unsigned 32 bits. +<p>Lookup tables can be access via the lookup() built-in function. The core idea is to +set a local variable to the lookup result and later on use that local variable in templates. +<p>More details on usage now follow. +<h2>Lookup Table File Format</h2> +<p>Lookup table files contain a single JSON object. This object contains of a header and a +table part. +<h3>Header</h3> +<p>The header is the top-level json. It has paramters "version", "nomatch", and "type". +The version parameter +must be given and must always be one for this version of rsyslog. The nomatch +parameter is optional. If specified, it contains the value to be used if lookup() +is provided an index value for which no entry exists. The default for +"nomatch" is the empty string. Type specifies the type of lookup to be done. +<h3>Table</h3> +This must be an array of elements, even if only a single value exists (for obvious +reasons, we do not expect this to occur often). Each array element must contain two +fields "index" and "value". +<h3>Example</h3> +<p>This is a sample of how an ip-to-office mapping may look like: +<pre> +{ "version":1, "nomatch":"unk", "type":"string", + "table":[ {"index":"10.0.1.1", "value":"A" }, + {"index":"10.0.1.2", "value":"A" }, + {"index":"10.0.1.3", "value":"A" }, + {"index":"10.0.2.1", "value":"B" }, + {"index":"10.0.2.2", "value":"B" }, + {"index":"10.0.2.3", "value":"B" } + ] +} +</pre> +Note: if a different IP comes in, the value "unk" +is returend thanks to the nomatch parameter in +the first line. +<p> +<h2>RainerScript Statements</h2> +<h3>lookup_table() Object</h3> +<p>This statement defines and intially loads a lookup table. Its format is +as follows: +<pre> +lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (mandatory)<br> + Defines the name of lookup table for further reference + inside the configuration. Names must be unique. Note that + it is possible, though not advisible, to have different + names for the same file. + <li><b>file</b> (mandatory)<br> + Specifies the full path for the lookup table file. This file + must be readable for the user rsyslog is run under (important + when dropping privileges). It must point to a valid lookup + table file as described above. + <li><b>reloadOnHUP</b> (optional, default "on")<br> + Specifies if the table shall automatically be reloaded + as part of HUP processing. For static tables, the + default is "off" and specifying "on" triggers an + error message. Note that the default of "on" may be + somewhat suboptimal performance-wise, but probably + is what the user intuitively expects. Turn it off + if you know that you do not need the automatic + reload capability. +</ul> + +<h3>lookup() Function</h3> +<p>This function is used to actually do the table lookup. Format: +<pre> +lookup_table("name", indexvalue) +</pre> +<h4>Parameters</h4> +<ul> + <li><b>return value</b><br> + The function returns the string that is associated with the + given indexvalue. If the indexvalue is not present inside the + lookup table, the "nomatch" string is returned (or an empty string + if it is not defined). + <li><b>name</b> (constant string)<br> + The lookup table to be used. Note that this must be specificed as a + constant. In theory, variable table names could be made possible, but + their runtime behaviour is not as good as for static names, and we do + not (yet) see good use cases where dynamic table names could be useful. + <li><b>indexvalue</b> (expression)<br> + The value to be looked up. While this is an arbitrary RainerScript expression, + it's final value is always converted to a string in order to conduct + the lookup. For example, "lookup(table, 3+4)" would be exactly the same + as "lookup(table, "7")". In most cases, indexvalue will probably be + a single variable, but it could also be the result of all RainerScript-supported + expression types (like string concatenation or substring extraction). + Valid samples are "lookup(name, $fromhost-ip & $hostname)" or + "lookup(name, substr($fromhost-ip, 0, 5))" as well as of course the + usual "lookup(table, $fromhost-ip)". +</ul> + + +<h3>load_lookup_table Statement</h3> + +<p><b>Note: in the final implementation, this MAY be implemented as an action. +This is a low-level decesion that must be made during the detail development +process. Parameters and semantics will remain the same of this happens.</b> + +<p>This statement is used to reload a lookup table. It will fail if +the table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight performance +hit during the load phase. It is assume that always a triggering condition +is used to load the table. +<pre> +load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value") +</pre> +<h4>Parameters</h4> +<ul> + <li><b>name</b> (string)<br> + The lookup table to be used. + <li><b>errOnFail</b> (boolean, default "on")<br> + Specifies whether or not an error message is to be emitted if + there are any problems reloading the lookup table. + <li><b>valueOnFail</b> (optional, string)<br> + This parameter affects processing if the lookup table cannot + be loaded for some reason: If the parameter is not present, + the previous table will be kept in use. If the parameter is + given, the previous table will no longer be used, and instead + an empty table be with nomath=valueOnFail be generated. In short, + that means when the parameter is set and the reload fails, + all matches will always return what is specified in valueOnFail. +</ul> + +<h3>Usage example</h3> +<p>For clarity, we show only those parts of rsyslog.conf that affect +lookup tables. We use the remote office example that an example lookup +table file is given above for. +<pre> +lookup_table(name="ip2office" file="/path/to/ipoffice.lu" + reloadOnHUP="off") + + +template(name="depfile" type="string" + string="/var/log/%$usr.dep%/messages") + +set $usr.dep = lookup("ip2office", $fromhost-ip); +action(type="omfile" dynfile="depfile") + +# support for reload "commands" +if $fromhost-ip == "10.0.1.123" + and $msg contains "reload office lookup table" + then + load_lookup_table(name="ip2office" errOnFail="on") +</pre> + +<p>Note: for performance reasons, it makes sense to put the reload command into +a dedicated ruleset, bound to a specific listener - which than should also +be sufficiently secured, e.g. via TLS mutual auth. + +<h2>Implementation Details</h2> +<p>The lookup table functionality is implemented via highly efficient algorithms. +The string lookup is based on a parse tree and has O(1) time complexity. The array +lookup is also O(1). In case of sparseArray, we have O(log n). +<p>To preserve space and, more important, increase cache hit performance, equal +data values are only stored once, no matter how often a lookup index points to them. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/manual.html b/doc/manual.html index 9c7c677e..ed22967d 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -13,13 +13,13 @@ It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its <a href="features.html"> advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_tls.html">encryption protected syslog</a> relay chains while at the same time being very easy to setup for the -novice user. And as we know what enterprise users really need, there is -also <a href="http://www.rsyslog.com/professional-services">professional -rsyslog support</a> available directly from the source!</p> +novice user. And as we know what enterprise users really need, there are +also <a href="http://www.rsyslog.com/professional-services"> rsyslog +professional services</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 6.6.0 (v6-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 7.4.3 (v7.4-stable branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -35,12 +35,13 @@ if you upgrade from v4, read the <a href="v5compatibility.html">rsyslog v5 compatibility notes</a>, and if you upgrade from v5, read the <a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. +if you upgrade from v6, read the +<a href="v7compatibility.html">rsyslog v7 compatibility notes</a>. <p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> <p><b>Follow the links below for the</b></p> <ul> <li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> -<li><a href="http://www.rsyslog.com/doc/node1.html">rsyslog.conf, new RainerScript-based format (v6+)</a></li> <li><a href="rsyslog_conf.html">configuration file format (rsyslog.conf)</a></li> <li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> <li> <a href="property_replacer.html">property replacer, an important core component</a></li> diff --git a/doc/mmanon.html b/doc/mmanon.html new file mode 100644 index 00000000..16065a1f --- /dev/null +++ b/doc/mmanon.html @@ -0,0 +1,119 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>IP Address Anonimization Module (mmanon)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>IP Address Anonimization Module (mmanon)</h1> +<p><b>Module Name: mmanon</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.3.7</p> +<p><b>Description</b>:</p> +<p>The mmanon module permits to anonymize IP addresses. It is a message +modification module that actually changes the IP address inside the message, +so after calling mmanon, the original message can no longer be obtained. +Note that anonymization will break digital signatures on the message, if +they exist. +<p><i>How are IP-Addresses defined?</i> +<p>We assume that an IP address consists of four octets in dotted notation, +where each of the octets has a value between 0 and 255, inclusively. After +the last octet, there must be either a space or a colon. So, for example, +"1.2.3.4 Test" and "1.2.3.4:514 Test" are detected as containing valid IP +addresses, whereas this is not the case for "1.2.300.4 Test" or +"1.2.3.4-Test". The message text may contain multiple addresses. If so, +each of them is anonimized (according to the same rules). +<b>Important:</b> We may change the set of acceptable characters after +the last octet in the future, if there are good reasons to do so. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>mode</b> - default "rewrite"<br> +There exists the "simple" and "rewrite" mode. In simple mode, only octets +as whole can be anonymized and the length of the message is never changed. +This means that when the last three octets of the address 10.1.12.123 are +anonymized, the result will be 10.0.00.000. This means that the length of the +original octets is still visible and may be used to draw some privacy-evasive +conclusions. This mode is slightly faster than "overwrite" mode, and this +may matter in high throughput environments.<br> +The default "rewrite" mode will do full anonymization of any number of bits +and it will also normlize the address, so that no information about the +original IP address is available. So in the above example, 10.1.12.123 would +be anonymized to 10.0.0.0. +<li><b>ipv4.bits</b> - default 16<br> +This set the number of bits that should be anonymized (bits are from the +right, so lower bits are anonymized first). This setting permits to save +network information while still anonymizing user-specific data. The more +bits you discard, the better the anonymization obviously is. The default +of 16 bits reflects what German data privacy rules consider as being +sufficinetly anonymized. We assume, this can also be used as a rough +but conservative guideline for other countries.<br> +Note: when in simple mode, only bits on a byte boundary can be specified. +As such, any value other than 8, 16, 24 or 32 is invalid. If an invalid +value is given, it is rounded to the next byte boundary (so we favor stronger +anonymization in that case). For example, a bit value of 12 will become 16 in +simple mode (an error message is also emitted). +<li><b>replacementChar</b> - default "x"<br> +In simple mode, this sets the character +that the to-be-anonymized part of the IP address is to be overwritten +with. In rewrite mode, this parameter is <b>not permitted</b>, as in +this case we need not necessarily rewrite full octets. As such, the anonymized +part is always zero-filled and replacementChar is of no use. If it is +specified, an error message is emitted and the parameter ignored. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li><b>only IPv4</b> is supported +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we write one file without anonymization and another one +with the message anonymized. Note that once mmanon has run, access to the +original message is no longer possible (execept if stored in user +variables before anonymization). +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>This next snippet is almost identical to the first one, but +here we anonymize the full IPv4 address. Note that by +modifying the number of bits, you can anonymize different parts +of the address. Keep in mind that in simple mode (used here), the bit values +must match IP address bytes, so for IPv4 only the values 8, 16, 24 and +32 are valid. Also, in this example the replacement is done +via asterisks instead of lower-case "x"-letters. Also keep in mind that +"replacementChar" can only be set in simple mode. +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon" ipv4.bits="32" mode="simple" replacementChar="*") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>The next snippet is also based on the first one, but anonimzes an +"odd" number of bits, 12. The value of 12 is used by some folks as a +compromise between keeping privacy and still permiting to gain some +more in-depth insight from log files. Note that anonymizing 12 bits +may be insufficient to fulfill legal requirements (if such exist). +<p><textarea rows="5" cols="60">module(load="mmanon") +action(type="omfile" file="/path/to/non-anon.log") +action(type="mmanon" ipv4.bits="12") +action(type="omfile" file="/path/to/anon.log") +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmjsonparse.html b/doc/mmjsonparse.html new file mode 100644 index 00000000..c2c862d7 --- /dev/null +++ b/doc/mmjsonparse.html @@ -0,0 +1,45 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>CEE/lumberjack JSON support Module (mmjsonparse)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Log Message Normalization Module</h1> +<p><b>Module Name: mmjsonparse</b></p> +<p><b>Available since: </b>6.6.0+ +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides support for parsing structured log messages +that follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked +and, if present, the JSON-encoded structured message content is parsed. +The properties are than available as original message properties. +</p> +<p><b>Action specific Configuration Directives</b>:</p> +<p>currently none +<ul> +<p><b>Legacy Configuration Directives</b>:</p> +<p>none +<b>Caveats/Known Bugs:</b> +<p>None known at this time. +</ul> +<p><b>Sample:</b></p> +<p>This activates the module and applies normalization to all messages:<br> +</p> +<textarea rows="2" cols="60">module(load="mmjsonparse") +action(type="mmjsonparse") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="2" cols="60">$ModLoad mmjsonparse +*.* :mmjsonparse: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2012 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html index 82f9b6a2..787bd957 100644 --- a/doc/mmnormalize.html +++ b/doc/mmnormalize.html @@ -11,37 +11,53 @@ <p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> <p><b>Description</b>:</p> <p>This module provides the capability to normalize log messages via -<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to libee, unstructured text, +<a href="http://www.liblognorm.com">liblognorm</a>. Thanks to liblognorm, unstructured text, like usually found in log messages, can very quickly be parsed and put into -a normal form. This is done so quickly, that it usually should be possible +a normal form. This is done so quickly, that it should be possible to normalize events in realtime. -<p>This module is implemented via the output module interface. That means that +<p>This module is implemented via the output module interface. This means that mmnormalize should be called just like an action. After it has been called, -the normalized message properties are avaialable and can be access. These properties -are called the "CEE" properties, because liblognorm creates a format that is -inspired by the CEE approach. +the normalized message properties are avaialable and can be accessed. These properties +are called the "CEE/lumberjack" properties, because liblognorm creates a format that is +inspired by the CEE/lumberjack approach. +<p><b>Please note:</b> CEE/lumberjack properties are different from regular properties. +They have always "$!" prepended to the property name given in the rulebase. Such a +property needs to be called with <b>%$!propertyname%</b>. <p>Note that mmnormalize should only be called once on each message. Behaviour is -undifined if multiple calls to mmnormalize happen for the same message. +undefined if multiple calls to mmnormalize happen for the same message. </p> -<p><b>Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> -<li>$mmnormalizeRuleBase <rulebase-file><br> -Specifies which rulebase file is to use. This file is loaded. If there are +<li><b>ruleBase</b> [word]<br> +Specifies which rulebase file is to use. If there are multiple mmnormalize instances, each one can use a different file. However, a single instance can use only a single file. This parameter MUST be given, -because normalization can only happen based on a rulebase. -<li>$mmnormalizeUseRawMsg <on/off><br> +because normalization can only happen based on a rulebase. It is recommended +that an absolute path name is given. Information on how to create the rulebase +can be found in the <a href="http://www.liblognorm.com/files/manual/index.html">liblognorm manual</a>. +<li><b>useRawMsg</b> [boolean]<br> Specifies if the raw message should be used for normalization (on) or just the MSG part of the message (off). Default is "off". </ul> +<p><b>Legacy Configuration Directives</b>:</p> +<ul> +<li>$mmnormalizeRuleBase <rulebase-file> - equivalent to the "ruleBase" +parameter. +<li>$mmnormalizeUseRawMsg <on/off> - equivalent to the "useRawMsg" +parameter. +</ul> <b>Caveats/Known Bugs:</b> <p>None known at this time. </ul> <p><b>Sample:</b></p> <p>This activates the module and applies normalization to all messages:<br> </p> -<textarea rows="8" cols="60">$ModLoad mmnormalize -$mmnormalizeRuleBase rulebase.rb +<textarea rows="2" cols="60">module(load="mmnormalize") +action(type="mmnormalize" ruleBase="/path/to/rulebase.rb") +</textarea> +<p>The same in legacy format:</p> +<textarea rows="3" cols="60">$ModLoad mmnormalize +$mmnormalizeRuleBase /path/to/rulebase.rb *.* :mmnormalize: </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] @@ -49,7 +65,7 @@ $mmnormalizeRuleBase rulebase.rb <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2010-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index da65b4ba..83c495ca 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -31,7 +31,7 @@ You can think of a traditional config file just as a single default rule set, wh automatically bound to each of the inputs. This is even what actually happens. When rsyslog.conf is processed, the config file parser looks for the directive -<pre>$RuleSet <name> +<pre>ruleset(name="rulesetname"); </pre> <p>Where name is any name the user likes (but must not start with "RSYSLOG_", which @@ -63,7 +63,7 @@ to seperate the messages by any other method. <p>Binding to rulesets is input-specifc. For imtcp, this is done via the -<pre>$InputTCPServerBindRuleset <name> +<pre>input(type="imptcp" port="514" ruleset="rulesetname"); </pre> directive. Note that "name" must be the name of a ruleset that is already defined @@ -116,8 +116,12 @@ filters on the message, processes it and then discards it: <pre> # ... module loading ... # process remote messages -:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile -& ~ +if $fromhost-ip == '192.168.152.137' then { + action(type="omfile" file="/var/log/remotefile02") + stop + } + + # only messages not from 192.0.21 make it past this point # The authpriv file has restricted access. @@ -131,7 +135,7 @@ cron.* /var/log/cron ... more ... </pre> -<p>Note the tilde character, which is the discard action!. Also note that we assume that +<p>Note that "stop" is the discard action!. Also note that we assume that 192.0.2.1 is the sole remote sender (to keep it simple). <p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception @@ -141,66 +145,15 @@ case and bind it to the receiver. This may be written as follows: # ... module loading ... # process remote messages # define new ruleset and add rules to it: -$RuleSet remote -*.* /var/log/remotefile +ruleset(name="remote"){ + action(type="omfile" file="/var/log/remotefile") +} # only messages not from 192.0.21 make it past this point -# bind ruleset to tcp listener -$InputTCPServerBindRuleset remote -# and activate it: -$InputTCPServerRun 10514 - -# switch back to the default ruleset: -$RuleSet RSYSLOG_DefaultRuleset -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... +# bind ruleset to tcp listener and activate it: +input(type="imptcp" port="10514" ruleset="remote") </pre> -<p>Here, we need to switch back to the default ruleset after we have defined our custom -one. This is why I recommend a different ordering, which I find more intuitive. The sample -below has it, and it leads to the same results: - -<pre> -# ... module loading ... -# at first, this is a copy of the unmodified rsyslog.conf -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... -# end of the "regular" rsyslog.conf. Now come the new definitions: - -# process remote messages -# define new ruleset and add rules to it: -$RuleSet remote -*.* /var/log/remotefile - -# bind ruleset to tcp listener -$InputTCPServerBindRuleset remote -# and activate it: -$InputTCPServerRun 10514 -</pre> - -<p>Here, we do not switch back to the default ruleset, because this is not needed as it is -completely defined when we begin the "remote" ruleset. - -<p>Now look at the examples and compare them to the single-ruleset solution. You will notice -that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use -"*.*" as all messages now means all messages that are being processed by this -rule set and all of them come in via the TCP receiver! This is what makes using multiple -rulesets so much easier. - <h3>Split local and remote logging for three different ports</h3> <p>This example is almost like the first one, but it extends it a little bit. While it is very similar, I hope it is different enough to provide a useful example why you may want @@ -217,47 +170,34 @@ written to 10516's general log file. <pre> # ... module loading ... -# at first, this is a copy of the unmodified rsyslog.conf -# The authpriv file has restricted access. -authpriv.* /var/log/secure -# Log all the mail messages in one place. -mail.* /var/log/maillog -# Log cron stuff -cron.* /var/log/cron -# Everybody gets emergency messages -*.emerg * -... more ... -# end of the "regular" rsyslog.conf. Now come the new definitions: - # process remote messages -#define rulesets first -$RuleSet remote10514 -*.* /var/log/remote10514 - -$RuleSet remote10515 -*.* /var/log/remote10515 - -$RuleSet remote10516 -mail.* /var/log/mail10516 -& ~ -# note that the discard-action will prevent this messag from -# being written to the remote10516 file - as usual... -*.* /var/log/remote10516 - -# and now define listners bound to the relevant ruleset -$InputTCPServerBindRuleset remote10514 -$InputTCPServerRun 10514 - -$InputTCPServerBindRuleset remote10515 -$InputTCPServerRun 10515 - -$InputTCPServerBindRuleset remote10516 -$InputTCPServerRun 10516 +ruleset(name="remote10514"){ + action(type="omfile" file="/var/log/remote10514") +} + +ruleset(name="remote10515"){ + action(type="omfile" file="/var/log/remote10515") +} + +ruleset(name="test1"){ + if prifilt("mail.*") then { + /var/log/mail10516 + stop + # note that the stop-command will prevent this message from + # being written to the remote10516 file - as usual... + } + /var/log/remote10516 +} + + +# and now define listeners bound to the relevant ruleset +input(type="imptcp" port="10514" ruleset="remote10514") +input(type="imptcp" port="10515" ruleset="remote10515") +input(type="imptcp" port="10516" ruleset="remote10516") </pre> -<p>Note that the "mail.*" rule inside the "remote10516" ruleset does -not affect processing inside any other rule set, including the default rule set. + <h2>Performance</h2> @@ -289,10 +229,6 @@ dedicated queue for each of the inputs. <p>By default, rulesets do <b>not</b> have their own queue. It must be activated via the <a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> directive. -<h3>Future Enhancements</h3> -<p>In the long term, multiple rule sets will probably lay the foundation for even better -optimizations. So it is not a bad idea to get aquainted with them. - <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> diff --git a/doc/multi_ruleset_legacy_format.html b/doc/multi_ruleset_legacy_format.html new file mode 100644 index 00000000..273a4a09 --- /dev/null +++ b/doc/multi_ruleset_legacy_format.html @@ -0,0 +1,192 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Multiple Rulesets in legacy format</title></head> +<body> +<h1>Multiple Rulesets in rsyslog</h1> +<p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports +multiple rulesets within a single configuration. +This is especially useful for routing the recpetion of remote messages to a set of specific rules. +Note that the input module must support binding to non-standard rulesets, so the functionality +may not be available with all inputs.<p> +<b>Attention: this guide is shortened and only contains the samples in legacy format.</b> +Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html<a> + + +<h2>Examples</h2> +<h3>Split local and remote logging</h3> +<p>Let's say you have a pretty standard system that logs its local messages to the usual +bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf +might look like this: + +<pre> +# ... module loading ... +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Now, you want to add receive messages from a remote system and log these to +a special file, but you do not want to have these messages written to the files +specified above. The traditional approach is to add a rule in front of all others that +filters on the message, processes it and then discards it: + +<pre> +# ... module loading ... +# process remote messages +:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile +& ~ +# only messages not from 192.0.21 make it past this point + +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Note the tilde character, which is the discard action!. Also note that we assume that +192.0.2.1 is the sole remote sender (to keep it simple). + +<p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception +case and bind it to the receiver. This may be written as follows: + +<pre> +# ... module loading ... +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile +# only messages not from 192.0.21 make it past this point + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 + +# switch back to the default ruleset: +$RuleSet RSYSLOG_DefaultRuleset +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Here, we need to switch back to the default ruleset after we have defined our custom +one. This is why I recommend a different ordering, which I find more intuitive. The sample +below has it, and it leads to the same results: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 +</pre> + +<p>Here, we do not switch back to the default ruleset, because this is not needed as it is +completely defined when we begin the "remote" ruleset. + +<p>Now look at the examples and compare them to the single-ruleset solution. You will notice +that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use +"*.*" as all messages now means all messages that are being processed by this +rule set and all of them come in via the TCP receiver! This is what makes using multiple +rulesets so much easier. + +<h3>Split local and remote logging for three different ports</h3> +<p>This example is almost like the first one, but it extends it a little bit. While it is +very similar, I hope it is different enough to provide a useful example why you may want +to have more than two rulesets. + +<p>Again, we would like to use the "regular" log files for local logging, only. But +this time we set up three syslog/tcp listeners, each one listening to a different +port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into +different files. Also, logs received from 10516 (and only from that port!) with +"mail.*" priority, shall be written into a specif file and <b>not</b> be +written to 10516's general log file. + +<p>This is the config: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages + +#define rulesets first +$RuleSet remote10514 +*.* /var/log/remote10514 + +$RuleSet remote10515 +*.* /var/log/remote10515 + +$RuleSet remote10516 +mail.* /var/log/mail10516 +& ~ +# note that the discard-action will prevent this messag from +# being written to the remote10516 file - as usual... +*.* /var/log/remote10516 + +# and now define listeners bound to the relevant ruleset +$InputTCPServerBindRuleset remote10514 +$InputTCPServerRun 10514 + +$InputTCPServerBindRuleset remote10515 +$InputTCPServerRun 10515 + +$InputTCPServerBindRuleset remote10516 +$InputTCPServerRun 10516 +</pre> + +<p>Note that the "mail.*" rule inside the "remote10516" ruleset does +not affect processing inside any other rule set, including the default rule set. + + +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omelasticsearch.html b/doc/omelasticsearch.html new file mode 100644 index 00000000..618b7065 --- /dev/null +++ b/doc/omelasticsearch.html @@ -0,0 +1,177 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> + <head> + <meta content="en" http-equiv="Content-Language" /> + <title>Elasticsearch Output Module</title> + </head> + <body> + <p> + <a href="rsyslog_conf_modules.html">back</a></p> + <h1> + Elasticsearch Output Module</h1> + <p> + <b>Module Name: omelasticsearch</b></p> + <p> + <b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> + <p> + <b>Available since: </b>6.4.0+</p> + <p> + <b>Description</b>:</p> + <p> + This module provides native support for logging to <a href="http://www.elasticsearch.org/">Elasticsearch</a>.</p> + <p> + <b>Action Parameters</b>:</p> + <ul> + <li> + <b>server</b><br /> + Host name or IP address of the Elasticsearch server. Defaults to "localhost"</li> + <li> + <b>serverport</b><br /> + HTTP port to connect to Elasticsearch. Defaults to 9200</li> + <li> + <b>searchIndex</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#index">Elasticsearch index</a> to send your logs to. Defaults to "system"</li> + <li> + <b>dynSearchIndex </b><on/<b>off</b>><br /> + Whether the string provided for <strong>searchIndex</strong> should be taken as a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a>. Defaults to "off", which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let's assume you define a template named "date-days" containing "%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", if you say searchIndex="date-days", each log will be sent to and index named after the first 10 characters of the timestamp, like "2013-03-22".</li> + <li> + <b>searchType</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#type">Elasticsearch type</a> to send your index to. Defaults to "events"</li> + <li> + <b>dynSearchType</b> <on/<strong>off</strong>><br /> + Like <strong>dynSearchIndex</strong>, it allows you to specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> for <strong>searchType</strong>, instead of a static string.</li> + <li> + <strong>asyncrepl </strong><on/<strong>off</strong>><br /> + By default, an indexing operation returns after all <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard">replica shards</a> have indexed the document. With asyncrepl="on" it will return after it was indexed on the <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard">primary shard</a> only - thus trading some consistency for speed.</li> + <li> + <strong>timeout</strong><br /> + How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to "1m".</li> + <li> + <strong>template</strong><br /> + This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:</li> + </ul> + <pre> +$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}" +</pre> + <p> + Which will produce this sort of documents (pretty-printed here for readability):</p> + <ul> + </ul> + <pre> +{ + "message": " this is a test message", + "fromhost": "test-host", + "facility": "user", + "priority": "info", + "timereported": "2013-03-12T18:05:01.344864+02:00", + "timegenerated": "2013-03-12T18:05:01.344864+02:00" +}</pre> + <ul> + <li> + <strong>bulkmode </strong><on/<strong>off</strong>><br /> + The default "off" setting means logs are shipped one by one. Each in its own HTTP request, using the <a href="http://www.elasticsearch.org/guide/reference/api/index_.html">Index API</a>. Set it to "on" and it will use Elasticsearch's <a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">Bulk API</a> to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the <a href="http://www.rsyslog.com/doc/node35.html">dequeue batch size</a>. More information about queues can be found <a href="http://www.rsyslog.com/doc/node32.html">here</a>.</li> + <li> + <strong>parent</strong><br /> + Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the <a href="http://www.elasticsearch.org/guide/reference/mapping/parent-field.html">parent field</a> in your <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> for that to work. By default, logs are indexed without a parent.</li> + <li> + <strong>dynParent </strong><on/<strong>off</strong>><br /> + Using the same parent for all the logs sent in the same action is quite unlikely. So you'd probably want to turn this "on" and specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> that will provide meaningful parent IDs for your logs.</li> + <li> + <strong>uid</strong><br /> + If you have basic HTTP authentication deployed (eg: through the <a href="https://github.com/Asquera/elasticsearch-http-basic">elasticsearch-basic plugin</a>), you can specify your user-name here.</li> + <li> + <strong>pwd</strong><br /> + Password for basic authentication.</li> + </ul> + <p> + <b>Samples:</b></p> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + outputs all logs to Elasticsearch using the default settings</li> + </ul> + <pre> +module(load="omelasticsearch") +*.* action(type="omelasticsearch")</pre> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + defines a template that will make the JSON contain the following properties (more info about what properties you can use <a href="http://www.rsyslog.com/doc/property_replacer.html">here</a>): + <ul> + <li> + RFC-3339 timestamp when the event was generated</li> + <li> + the message part of the event</li> + <li> + hostname of the system that generated the message</li> + <li> + severity of the event, as a string</li> + <li> + facility, as a string</li> + <li> + the tag of the event</li> + </ul> + </li> + <li> + outputs to Elasticsearch with the following settings + <ul> + <li> + host name of the server is myserver.local</li> + <li> + port is 9200</li> + <li> + JSON docs will look as defined in the template above</li> + <li> + index will be "test-index"</li> + <li> + type will be "test-type"</li> + <li> + activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300</li> + <li> + retry indefinitely if the HTTP request failed (eg: if the target server is down)</li> + </ul> + </li> + </ul> + <pre> +module(load="omelasticsearch") +template(name="testTemplate" + type="list" + option.json="on") { + constant(value="{") + constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") + constant(value="\",\"message\":\"") property(name="msg") + constant(value="\",\"host\":\"") property(name="hostname") + constant(value="\",\"severity\":\"") property(name="syslogseverity-text") + constant(value="\",\"facility\":\"") property(name="syslogfacility-text") + constant(value="\",\"syslogtag\":\"") property(name="syslogtag") + constant(value="\"}") + } +*.* action(type="omelasticsearch" + server="myserver.local" + serverport="9200" + template="testTemplate" + searchIndex="test-index" + searchType="test-type" + bulkmode="on" + queue.type="linkedlist" + queue.size="5000" + queue.dequeuebatchsize="300" + action.resumeretrycount="-1")</pre> + <p> + </p> + <pre> +</pre> + <p> + [<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + <p> + <font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br /> + Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the ASL 2.0.</font></p> + </body> +</html> + diff --git a/doc/omfile.html b/doc/omfile.html index 23ecc034..72320921 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -13,14 +13,14 @@ <p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> <p> </p> -<p><b>Global Configuration Directives</b>:</p> +<p><b>Module Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> sets a new default template for file actions.<br></li> </ul> <p> </p> -<p><b>Action specific Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -28,6 +28,16 @@ <li><strong>ZipLevel </strong>0..9 [default 0]<br> if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + <li><b>VeryRobustZip</b> [<b>on</b>/off] (v7.3.0+) - if ZipLevel is greater 0, + then this setting controls if extra headers are written to make the resulting file + extra hardened against malfunction. If set to off, data appended to previously unclean + closed files may not be accessible without extra tools. + Note that this risk is usually expected to be bearable, and thus "off" is the default mode. + The extra headers considerably + degrade compression, files with this option set to "on" may be four to five times as + large as files processed in "off" mode. + </li><br> + <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> Defines a template to be used for the output. <br></li><br> @@ -47,10 +57,10 @@ Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.<br></li><br> <li><strong>FileOwner </strong><br> - Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + Set the file owner for files newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> <li><strong>FileGroup </strong><br> - Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + Set the group for files newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> <li><strong>DirCreateMode </strong>[defaul 0700]<br> This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> @@ -59,7 +69,7 @@ The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> <li><strong>FailOnCHOwnFailure </strong>on/off [default on]<br> - This option modifies behaviour of dynaFile creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> + This option modifies behaviour of file creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> <li><strong>CreateDirs </strong>on/off [default on]<br> create directories on an as-needed basis<br></li><br> @@ -73,77 +83,106 @@ <li><strong>DynaFile </strong><br> For each message, the file name is generated based on the given template. Then, this file is opened. As with the ``file'' property, data is appended if the file already exists. If the file does not exist, a new file is created. A cache of recent files is kept. Note that this cache can consume quite some memory (especially if large buffer sizes are used). Files are kept open as long as they stay inside the cache. Currently, files are only evicted from the cache when there is need to do so (due to insufficient cache size). To force-close (and evict) a dynafile from cache, send a HUP signal to rsyslogd. <br></li><br> + <li><b>Sig.Provider </b>[ProviderName]<br> + Selects a signature provider for log signing. Currently, + there only is one provider called + "<a href="sigprov_gt.html">gt</a>".<br></li><br> + + <li><b>Cry.Provider </b>[ProviderName]<br> + Selects a crypto provider for log encryption. Currently, + there only is one provider called + "<a href="cryprov_gcry.html">gcry</a>".<br></li><br> + <li><strong>Template </strong>[templateName]<br> sets a new default template for file actions.<br></li><br> </ul> -<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Caveats/Known Bugs:</b></p> +<ul> +<li>One needs to be careful with log rotation if signatures and/or encryption +are being used. These create side-files, which form a set and must be kept +together. +<br> +For signatures, the ".sigstate" file must NOT be rotated away if +signature chains are to be build across multiple files. This is because +.sigstate contains just global information for the whole file set. However, +all other files need to be rotated together. The proper sequence is to + <ol> + <li> move all files inside the file set + <li> only AFTER this is completely done, HUP rsyslog + </ol> +This sequence will ensure that all files inside the set are atomically +closed and in sync. HUPing only after a subset of files have been moved +results in inconsistencies and will most probably render the file set +unusable. +</li> +</ul> <p><b>Sample:</b></p> <p>The following command writes all syslog messages into a file.</p> -<textarea rows="5" cols="60">Module (path="builtin:omfile") +<textarea rows="5" cols="60">Module (load="builtin:omfile") *.* action(type="omfile" -DirCreateMode="0700" -FileCreateMode="0644" -File="/var/log/messages") + DirCreateMode="0700" + FileCreateMode="0644" + File="/var/log/messages") </textarea> <br><br> <p><b>Legacy Configuration Directives</b>:</p> <ul> - <li><strong>$DynaFileCacheSize </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + <li><strong>$DynaFileCacheSize </strong><br> + equivalent to the "dynaFileCacheSize" parameter<br></li><br> - <li><strong>$OMFileZipLevel </strong>0..9 [default 0]<br> - if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> + <li><strong>$OMFileZipLevel </strong><br> + equivalent to the "zipLevel" parameter<br></li><br> - <li><strong>$OMFileFlushInterval </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + <li><strong>$OMFileFlushInterval </strong><br> + equivalent to the "flushInterval" parameter<br></li><br> - <li><strong>$OMFileASyncWriting </strong>on/off [default off]<br> - if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable FlushInterval, AsyncWriting must be set to "on". Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is "on" but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. <br></li><br> + <li><strong>$OMFileASyncWriting </strong><br> + equivalent to the "asyncWriting" parameter<br></li><br> - <li><strong>$OMFileFlushOnTXEnd </strong>on/off [default on]<br> - Omfile has the capability to write output using a buffered writer. Disk writes are only done when the buffer is full. So if an error happens during that write, data is potentially lost. In cases where this is unacceptable, set FlushOnTXEnd to on. Then, data is written at the end of each transaction (for pre-v5 this means after each log message) and the usual error recovery thus can handle write errors without data loss. Note that this option severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -on- is primarily an aid to preserve the traditional syslogd behaviour.<br></li><br> + <li><strong>$OMFileFlushOnTXEnd </strong><br> + equivalent to the "flushOnTXEnd" parameter<br></li><br> - <li><strong>$OMFileIOBufferSize </strong><size_nbr>, default 4k<br> - size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)<br></li><br> + <li><strong>$OMFileIOBufferSize </strong><br> + equivalent to the "IOBufferSize" parameter<br></li><br> <li><strong>$DirOwner </strong><br> - Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + equivalent to the "dirOwner" parameter<br></li><br> <li><strong>$DirGroup </strong><br> - Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + equivalent to the "dirGroup" parameter<br></li><br> <li><strong>$FileOwner </strong><br> - Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + equivalent to the "fileOwner" parameter<br></li><br> <li><strong>$FileGroup </strong><br> - Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + equivalent to the "fileGroup" parameter<br></li><br> - <li><strong>$DirCreateMode </strong>[defaul 0700]<br> - This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> + <li><strong>$DirCreateMode </strong><br> + equivalent to the "dirCreateMode" parameter<br></li><br> - <li><strong>$FileCreateMode </strong>[default 0644]<br> - The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> + <li><strong>$FileCreateMode </strong><br> + equivalent to the "fileCreateMode" parameter<br></li><br> - <li><strong>$FailOnCHOwnFailure </strong>on/off [default on]<br> - This option modifies behaviour of dynaFile creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> + <li><strong>$FailOnCHOwnFailure </strong><br> + equivalent to the "failOnChOwnFailure" parameter<br></li><br> <li><strong>$F$OMFileForceCHOwn </strong><br> - force ownership change for all files<br></li><br> + equivalent to the "ForceChOwn" parameter<br></li><br> - <li><strong>$CreateDirs </strong>on/off [default on]<br> - create directories on an as-needed basis<br></li><br> + <li><strong>$CreateDirs </strong><br> + equivalent to the "createDirs" parameter<br></li><br> - <li><strong>$ActionFileEnableSync </strong>on/off [default off]<br> - enables file syncing capability of omfile.<br></li><br> + <li><strong>$ActionFileEnableSync </strong><br> + equivalent to the "enableSync" parameter<br></li><br> - <li><strong>$ActionFileDefaultTemplate </strong>[templateName]<br> - sets a new default template for file actions.<br></li><br> + <li><strong>$ActionFileDefaultTemplate </strong><br> + equivalent to the "template" module parameter<br></li><br> <li><strong>$ResetConfigVariables </strong><br> - Resets all configuration variables to their default value. Any settings made will not be applied to configuration lines following the $ResetConfigVariables. This is a good method to make sure no side-effects exists from previous directives. This directive has no parameters.<br></li><br> + Resets all configuration variables to their default value.<br></li><br> </ul> @@ -160,7 +199,7 @@ $FileCreateMode 0644 index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omfwd.html b/doc/omfwd.html index 5599ae39..53f9e527 100644 --- a/doc/omfwd.html +++ b/doc/omfwd.html @@ -16,7 +16,7 @@ <p><b>Global Configuration Directives</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li> + sets a non-standard default template for this module.<br></li> </ul> <p> </p> @@ -59,7 +59,7 @@ <p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> <p><b>Sample:</b></p> <p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> -<textarea rows="5" cols="60">Module (path="builtin:omfwd") +<textarea rows="5" cols="60">Module (load="builtin:omfwd") *.* action(type="omfwd" Target="192.168.2.11" Port="10514" diff --git a/doc/omjournal.html b/doc/omjournal.html new file mode 100644 index 00000000..6124e40c --- /dev/null +++ b/doc/omjournal.html @@ -0,0 +1,86 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Linux Journal Output Module (omjournal)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Linux Journal Output Module (omjournal)</h1> +<p><b>Module Name: omjournal</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.3.7</p> +<p><b>Description</b>:</p> +<p>The omjournal output module provides an interface to the Linux journal. +It is meant to be used in those cases where the Linux journal is being used +as the sole system log database. With omjournal, messages from various +sources (e.g. files and remote devices) can also be written to the journal +and processed by its tools. +<p>A typical use case we had on our mind is a SOHO environment, where the +user wants to include syslog data obtained from the local router to be +part of the journal data. +<p>We suggest to check out our short presentation on +<a href="http://youtu.be/GTS7EuSdFKE">rsyslog journal integration</a> to +learn more details of anticipated use cases. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<p>Currently none. + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>One needs to be careful that no message routing loop is created. The +systemd journal forwards messages it receives to the traditional syslog +system (if present). That means rsyslog will receive the same message that +it just wrote as new input on imuxsock. If not handled specially and assuming +all messages be written to the journal, the message would be emitted to the +journal again and a deadly loop is started. +<p>To prevent that, imuxsock by default does not accept messages originating +from its own process ID, aka it ignores messages from the current instance of +rsyslogd. However, this setting can be changed, and if so the problem may occur. +</ul> + +<p><b>Sample:</b></p> +<p>We assume we have a DSL router inside the network and would like to +receive its syslog message into the journal. Note that this configuration can be +used without havoing any other syslog functionality at all (most importantly, there +is no need to write any file to /var/log!). We assume syslog over UDP, as this +is the most probable choice for the SOHO environment that this use case reflects. +To log to syslog data to the journal, add the following snippet to rsyslog.conf: +<textarea rows="20" cols="60">/* first, we make sure all necessary + * modules are present: + */ +module(load="imudp") # input module for UDP syslog +module(load="omjournal") # output module for journal + +/* then, define the actual server that listens to the + * router. Note that 514 is the default port for UDP + * syslog and that we use a dedicated ruleset to + * avoid mixing messages with the local log stream + * (if there is any). + */ +input(type="imudp" port="514" ruleset="writeToJournal") + +/* inside that ruleset, we just write data to the journal: */ +ruleset(name="writeToJournal") { + action(type="omjournal") +} +</textarea> +<p>Note that this can be your sole rsyslog.conf if you do not use rsyslog +for anything else than receving the router syslog messages. +<p>If you do not receive messages, <b>you probably need to enable inbound UDP +syslog traffic in your firewall</b>. + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html index 008dcb81..e47c7f57 100644 --- a/doc/omlibdbi.html +++ b/doc/omlibdbi.html @@ -54,7 +54,23 @@ dlopen()ed plugin (as omlibdbi is). So in short, you probably save you a lot of headache if you make sure you have at least libdbi version 0.8.3 on your system. </p> -<p><b>Action Parameters</b>:</p> +<p><b>Module Parameters</b></p> +<ul> +<li><b>template</b><br> +The default template to use. This template is used when no template is +explicitely specified in the action() statement. +<li><b>driverdirectory</b><br> +Path to the libdbi drivers. Usually, +you do not need to set it. If you installed libdbi-drivers at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <b>not</b> use this configuration directive. +Usually, everything works just fine. +Note that this was an action() paramter in rsyslog versions below 7.3.0. +However, only the first action's driverdirectory parameter was actually used. +This has been cleaned up in 7.3.0, where this now is a module paramter. +</li> +</ul> +<p><b>Action Parameters</b></p> <ul> <li><b>server</b><br>Name or address of the MySQL server <li><b>db</b><br>Database to use @@ -68,24 +84,18 @@ writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", "freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use ompgsql instead).</li> -<li><b>driverdirectory</b><br> -Path to the libdbi drivers. Usually, -you do not need to set it. If you installed libdbi-drivers at a -non-standard location, you may need to specify the directory here. If -you are unsure, do <b>not</b> use this configuration directive. -Usually, everything works just fine.</li> </ul> <p><b>Legacy (pre-v6) Configuration Directives</b>:</p> +<p>It is strongly recommended NOT to use legacy format. <ul> -<li><b>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</b> +<li><i>$ActionLibdbiDriverDirectory /path/to/dbd/drivers</i> - like the driverdirectory action parameter. -<li><strong>$ActionLibdbiDriver drivername</strong><br> - like the drivername action parameter. -<li><span style="font-weight: bold;">$ActionLibdbiHost hostname</span> - like the server action parameter -The host to connect to.</li> -<li><b>$ActionLibdbiUserName user</b> - like the uid action parameter -<li><b>$ActionlibdbiPassword</b> - like the pwd action parameter -<li><b>$ActionlibdbiDBName db</b> - like the db action parameter -<li><b>selector line: :omlibdbi:<i>;template</i></b><br> +<li><i>$ActionLibdbiDriver drivername</i> - like the drivername action parameter +<li><i>$ActionLibdbiHost hostname</i> - like the server action parameter +<li><i>$ActionLibdbiUserName user</i> - like the uid action parameter +<li><i>$ActionlibdbiPassword</i> - like the pwd action parameter +<li><i>$ActionlibdbiDBName db</i> - like the db action parameter +<li><i>selector line: :omlibdbi:<code>;template</code></i><br> executes the recently configured omlibdbi action. The ;template part is optional. If no template is provided, a default template is used (which is currently optimized for MySQL - sorry, folks...)</li> @@ -114,14 +124,14 @@ database "syslog_db" on mysqlsever.example.com. The server is MySQL and being accessed under the account of "user" with password "pwd" (if you have empty passwords, just remove the $ActionLibdbiPassword line).<br> </p> -<textarea rows="5" cols="60">$ModLoad omlibdbi +<textarea rows="5" cols="60">module(load="omlibdbi") *.* action(type="omlibdbi" driver="mysql" server="mysqlserver.example.com" db="syslog_db" uid="user" pwd="pwd" </textarea> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>The same as above, but in legacy config format (pre rsyslog-v6): -<textarea rows="10" cols="60">$ModLoad omlibdbi +<textarea rows="8" cols="60">$ModLoad omlibdbi $ActionLibdbiDriver mysql $ActionLibdbiHost mysqlserver.example.com $ActionLibdbiUserName user diff --git a/doc/ommongodb.html b/doc/ommongodb.html new file mode 100644 index 00000000..a6112642 --- /dev/null +++ b/doc/ommongodb.html @@ -0,0 +1,60 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>MongoDB Output Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>MongoDB Output Module</h1> +<p><b>Module Name: ommongodb</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides native support for logging to MongoDB. +</p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>server</b><br>Name or address of the MongoDB server +<li><b>serverport</b><br>Permits to select +a non-standard port for the MongoDB server. The default is 0, which means the +system default port is used. There is no need to specify this parameter unless +you know the server is running on a non-standard listen port. +<li><b>db</b><br>Database to use +<li><b>collection</b><br>Collection to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +</ul> +<p>Note rsyslog contains a canned default template to write to the MongoDB. It +will be used automatically if no other template is specified to be used. This template is: +<p> +<textarea rows="5" cols="80">template(name="BSON" type="string" string="\"sys\" : \"%hostname%\", \"time\" : \"%timereported:::rfc3339%\", \"time_rcvd\" : \"%timegenerated:::rfc3339%\", \"msg\" : \"%msg%\", \"syslog_fac\" : \"%syslogfacility%\", \"syslog_sever\" : \"%syslogseverity%\", \"syslog_tag\" : \"%syslogtag%\", \"procid\" : \"%programname%\", \"pid\" : \"%procid%\", \"level\" : \"%syslogpriority-text%\"") +</textarea> +<p>This creates the BSON document needed for MongoDB if no template is specified. The default +schema is aligned to CEE and project lumberjack. As such, the field names are standard +lumberjack field names, and <b>not</b> +<a href="property_replacer.html">rsyslog property names</a>. When specifying templates, be sure +to use rsyslog property names as given in the table. If you would like to use lumberjack-based +field names inside MongoDB (which probably is useful depending on the use case), you need to +select fields names based on the lumberjack schema. +If you just want to use a subset of the fields, but with lumberjack names, you can look up the +mapping in the default template. For example, the lumberjack field "level" contains the rsyslog +property "syslogpriority-text". +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog" and into the collection "log" on mongosever.example.com. The server is +being accessed under the account of "user" with password "pwd". +</p> +<textarea rows="5" cols="80">module(load="ommongodb") +*.* action(type="ommongodb" server="mongoserver.example.com" db="syslog" collection="log" uid="user" pwd="pwd") +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the ASL 2.0.</font></p> +</body></html> diff --git a/doc/omrelp.html b/doc/omrelp.html index 22e6845f..8858f884 100644 --- a/doc/omrelp.html +++ b/doc/omrelp.html @@ -16,10 +16,34 @@ RELP protocol. For RELP's advantages over plain tcp syslog, please see the documentation for <a href="imrelp.html">imrelp</a> (the server counterpart). </p> <span style="font-weight: bold;">Setup</span> -<p>Please note the <a href="http://www.librelp.com">librelp</a> +<p>Please note that <a href="http://www.librelp.com">librelp</a> is required for imrelp (it provides the core relp protocol implementation).</p> -<p><b>Configuration Directives</b>:</p> +<p><b>Action Configuration Parameters</b>:</p> +<p>This module supports RainerScript configuration starting with +rsyslog 7.3.10. For older versions, legacy configuration directives +must be used. +<ul> + <li><b>target </b>(mandatory)<br> + The target server to connect to. + </li> + <li><b>template </b>(not mandatory, default "RSYSLOG_ForwardFormat")<br> + Defines the template to be used for the output. + </li> + <li><b>timeout </b>(not mandatory, default 90)<br> + Timeout for relp sessions. If set too low, valid sessions + may be considered dead and tried to recover. + </li> +</ul> +<p><b>Sample:</b></p> +<p>The following sample sends all messages to the central server +"centralserv" at port 2514 (note that that server must run imrelp on +port 2514). +</p> +<textarea rows="3" cols="60">module(load="omrelp") +action(type="omrelp" target="centralserv" port="2514") +</textarea> +<p><b>Legacy Configuration Directives</b>:</p> <p>This module uses old-style action configuration to keep consistent with the forwarding rule. So far, no additional configuration directives can be specified. To send a message via RELP, @@ -33,18 +57,15 @@ use</p> <b>Caveats/Known Bugs:</b> <p>See <a href="imrelp.html">imrelp</a>, which documents them. </p> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>The following sample sends all messages to the central server "centralserv" at port 2514 (note that that server must run imrelp on -port 2514). Rsyslog's high-precision timestamp format is used, thus the -special "RSYSLOG_ForwardFormat" (case sensitive!) template is used.<br> +port 2514). </p> -<textarea rows="15" cols="60">$ModLoad omrelp -# forward messages to the remote server "myserv" on -# port 2514 -*.* :omrelp:centralserv:2514;RSYSLOG_ForwardFormat +<textarea rows="3" cols="60">$ModLoad omrelp +*.* :omrelp:centralserv:2514 </textarea> -Note: to use IPv6 addresses, encode them in [::1] format. +<p>Note: to use IPv6 addresses, encode them in [::1] format. <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/omudpspoof.html b/doc/omudpspoof.html index df14bbe1..930412c8 100644 --- a/doc/omudpspoof.html +++ b/doc/omudpspoof.html @@ -7,46 +7,161 @@ <h1>UDP spoofing output module (omudpspoof)</h1> <p><b>Module Name: omstdout</b></p> -<p><b>Author: </b>David Lang <david@lang.hm> and Rainer Gerhards -<rgerhards@adiscon.com></p> -<p><b>Available Since</b>: 5.1.3</p> +<p><b>Authors: </b>Rainer Gerhards <rgerhards@adiscon.com> +and David Lang <david@lang.hm> +</p> +<p><b>Available Since</b>: 5.1.3 / v7 config since 7.2.5</p> <p><b>Description</b>:</p> <p>This module is similar to the regular UDP forwarder, but permits to spoof the sender address. Also, it enables to circle through a number of source ports. -<p><b>Configuration Directives</b>:</p> +<p><b>Important:</b> This module requires root priveleges for its low-level +socket access. As such, the <b>module will not work if rsyslog is configured to +drop privileges</b>. + +<p><b>load() Parameters</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a non-standard default template for this module.<br></li> + +</ul> +<p> </p> +<p><b>action() parameters</b>:</p> +<ul> + <li><strong>Target </strong>string<br> + Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> + + <li><strong>Port </strong>[Integer, Default 514]<br> + Name or numerical value of port to use when connecting to target. <br></li><br> + + <li><b>Template</b>[Word]<br> + Template to use as message text. + <br></li><br> + + <li><strong>SourceTemplate </strong>[Word]<br> + This is the name of the template that contains a + numerical IP address that is to be used as the source system IP address. + While it may often be a constant value, it can be generated as usual via the + property replacer, as long as it is a valid IPv4 address. If not specified, the + build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined + as follows:<br> + template(name="RSYSLOG_omudpspoofDfltSourceTpl" type="string" string="%fromhost-ip%")<br> + So in essence, the default template spoofs the address of the system the message + was received from. This is considered the most important use case. + <br></li><br> + + <li><b>SourcePortStart</b>[Word]<br> + Specifies the start value for circeling the source ports. Must be less than or + equal to the end value. Default is 32000. + <br></li><br> + + <li><b>SourcePortEnd</b>[Word]<br> + Specifies the ending value for circeling the source ports. Must be less than or + equal to the start value. Default is 42000. + <br></li><br> + + <li><b>mtu</b>[Integer, default 1500]<br> + Maximum MTU supported by the network. Default respects Ethernet and must + usually not be adjusted. Setting a too-high MTU can lead to message loss, + too low to excess message fragmentation. Change only if you really know what + you are doing. This is always given in number of bytes. + <br></li><br> +</ul> +<p><b>pre-v7 Configuration Directives</b>:</p> <ul> -<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename><br> -This is the name of the template that contains a -numerical IP address that is to be used as the source system IP address. -While it may often be a constant value, it can be generated as usual via the -property replacer, as long as it is a valid IPv4 address. If not specified, the -build-in default template RSYSLOG_omudpspoofDfltSourceTpl is used. This template is defined -as follows:<br> -$template RSYSLOG_omudpspoofDfltSourceTpl,"%fromhost-ip%"<br> -So in essence, the default template spoofs the address of the system the message -was received from. This is considered the most important use case. -<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname><br> -Host that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofTargetPort</b> <port><br> -Remote port that the messages shall be sent to. -<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename><br> -This setting instructs omudpspoof to use a template different from the -default template for all of its actions that do not have a template specified -explicitely. -<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number><br> -Specifies the start value for circeling the source ports. Must be less than or -equal to the end value. Default is 32000. -<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number><br> -Specifies the ending value for circeling the source ports. Must be less than or -equal to the start value. Default is 42000. +<li><b>$ActionOMOMUDPSpoofSourceNameTemplate</b> <templatename> +- equivalent to the "sourceTemplate" parameter. +<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofTargetPort</b> <port> - equivalent to the "target" parameter. +<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename> +- equivalent to the "template" load() parameter. +<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number> +- equivalent to the "SourcePortStart" parameter. +<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number> +- equivalent to the "SourcePortEnd" parameter. </ul> <b>Caveats/Known Bugs:</b> <ul> <li><b>IPv6</b> is currently not supported. If you need this capability, please let us know via the rsyslog mailing list. +<li>Versions shipped prior to rsyslog 7.2.5 do not support message sizes over 1472 bytes (more +pricesely: over the network-supported MTU). Starting with 7.2.5, those messages will be +fragmented, up to a total upper limit of 64K (induced by UDP). Message sizes over +64K will be truncated. For older versions, messages over 1472 may be totally discarded +or truncated, depending on version and environment. </ul> -<p><b>Sample:</b></p> + +<p><b>Config Samples</b></p> +<p>The following sample forwards all syslog messages in standard form to the +remote server server.example.com. The original sender's address is used. We do not +care about the source port. This example is considered the typical use case for +omudpspoof. +</p> +<textarea rows="3" cols="80">module(load="omudpspoof") +action(type="omudpspoof" target="server.example.com") +</textarea> + +<p>The following sample forwards all syslog messages in unmodified form to the +remote server server.example.com. The sender address 192.0.2.1 with fixed +source port 514 is used. +</p> +<textarea rows="7" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514) +</textarea> +<p>The following sample is exatly like the previous, but it specifies a larger size +MTU. If, for example, the envrionment supports Jumbo Ethernet frames, increasing the +MTU is useful as it reduces packet fragmentation, which most often is the source of +problems. Note that setting the MTU to a value larger than the local-attached network +supports will lead to send errors and loss of message. So use with care! +</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>Of course, the action can be combined with any type of filter, for +example a tradition PRI filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +local0.* action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>... or any complex expression-based filter:</p> +<textarea rows="8" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") +</textarea> +<p>and of course it can also be combined with as many other actions +as one likes:</p> +<textarea rows="11" cols="80">module(load="omudpspoof") +template(name="spoofaddr" type="string" string="192.0.2.1") +template(name="spooftemplate" type="string" string="%rawmsg%") +if prifilt("local0.*") and $msg contains "error" then { + action(type="omudpspoof" target="server.example.com" + sourcetemplate="spoofaddr" template="spooftemplate" + sourceport.start="514" sourceport.end="514 + mtu="8000") + action(type="omfile" file="/var/log/somelog") + stop # or whatever... +} +</textarea> + +<p><b>Legacy Sample (pre-v7):</b></p> <p>The following sample forwards all syslog messages in standard form to the remote server server.example.com. The original sender's address is used. We do not care about the source port. This example is considered the typical use case for diff --git a/doc/property_replacer.html b/doc/property_replacer.html index dc09d33c..13ff41c3 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -228,7 +228,15 @@ for filtering in a generic way)</td> <td>This is the "bridge" to syslog message normalization (via <a href="mmnormalize.html">mmnormalize</a>): name is a name defined inside the normalization rule. It has the value selected by the rule -or none if no rule with this field did match. +or none if no rule with this field did match. You can also use these +properties to specify JSON fields from the CEE-enhanced syslog +message, once you parse it with <a href="mmjsonparse.html">mmjsonparse</a> +</td> +</tr> +<tr> +<td><b>$!all-json</b></td> +<td>This is the JSON part of the CEE-enhanced syslog message, which +can be parsed with <a href="mmjsonparse.html">mmjsonparse</a> </td> </tr> </tbody> @@ -405,6 +413,12 @@ option when forwarding to remote hosts - they may treat the date as invalid <td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> </tr> <tr> +<td>pos-end-relative</td> + <td>the from and to position is relative to the end of the string + instead of the usual start of string. (available since rsyslog v7.3.10) + </td> +</tr> +<tr> <td><b>ControlCharacters</b></td> <td>Option values for how to process control characters</td> </tr> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index fcc2674d..7cbbfa9f 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -51,6 +51,11 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. +<h2>Lookup Tables</h2> +<p><a href="lookup_tables.html">Lookup tables</a> are a powerful construct +to obtain "class" information based on message content (e.g. to build +log file names for different server types, departments or remote +offices). <h2>Functions</h2> <p>RainerScript supports a currently quite limited set of functions: <ul> @@ -61,6 +66,33 @@ variable, if it exists. Returns an empty string if it does not exist. <li>cstr(expr) - converts expr to a string value <li>cnum(expr) - converts expr to a number (integer) <li>re_match(expr, re) - returns 1, if expr matches re, 0 otherwise +<li>re_extract(expr, re, match, submatch, no-found) - extracts +data from a string (property) via a regular expression match. +POSIX ERE regular expressions are used. The variable "match" contains +the number of the match to use. This permits to pick up more than the +first expression match. Submatch is the submatch to match (max 50 supported). +The "no-found" parameter specifies which string is to be returned in case when +the regular expression is not found. Note that match and submatch start with +zero. It currently is not possible to extract more than one submatch with +a single call. +<li>field(str, delim, matchnbr) - returns a field-based substring. str is the string +to search, delim is the delimiter and matchnbr is the match to search +for (the first match starts at 1). This works similar as the field based +property-replacer option. +Versions prior to 7.3.7 only support a single character as delimiter character. +Starting with version 7.3.7, a full string can be used as delimiter. If a single +character is being used as delimiter, delim is the numerical ascii value of the +field delimiter character (so that non-printable characters can by specified). If a +string is used as delmiter, a multi-character string (e.g. "#011") is to be +specified. Samples:<br> +set $!usr!field = field($msg, 32, 3); -- the third field, delimited by space<br> +set $!usr!field = field($msg, "#011", 3); -- the third field, delmited by "#011"<br> +Note that when a single character is specified as string [field($msg, ",", 3)] a +string-based extraction is done, which is more performance intense than the +equivalent single-character [field($msg, 44 ,3)] extraction. +<li>prifilt(constant) - mimics a traditional PRI-based filter (like "*.*" or +"mail.info"). The traditional filter string must be given as a <b>constant string</b>. +Dynamic string evaluation is not permitted (for performance reasons). </ul> <p>The following example can be used to build a dynamic filter based on some environment variable: @@ -72,7 +104,7 @@ if $msg contains getenv('TRIGGERVAR') then /path/to/errfile <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/rsconf1_rulesetcreatemainqueue.html b/doc/rsconf1_rulesetcreatemainqueue.html index 5c1e0dec..d09f95ce 100644 --- a/doc/rsconf1_rulesetcreatemainqueue.html +++ b/doc/rsconf1_rulesetcreatemainqueue.html @@ -58,7 +58,7 @@ mail.* /var/log/mail10516 # being written to the remote10516 file - as usual... *.* /var/log/remote10516 -# and now define listners bound to the relevant ruleset +# and now define listeners bound to the relevant ruleset $InputTCPServerBindRuleset remote10514 $InputTCPServerRun 10514 diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html index ef29c2a8..433456c1 100644 --- a/doc/rsconf1_rulesetparser.html +++ b/doc/rsconf1_rulesetparser.html @@ -9,7 +9,7 @@ <p><b>Type:</b> ruleset-specific configuration directive</p> <p><b>Parameter Values:</b> string</p> <p><b>Available since:</b> 5.3.4+</p> -<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc5425</p> +<p><b>Default:</b> rsyslog.rfc5424 followed by rsyslog.rfc3164</p> <p><b>Description:</b></p> <p> This directive permits to specify which diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 6aa2e460..c5f4d2e3 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -21,16 +21,15 @@ especially useful while you are migrating from syslogd to rsyslogd.</p> <p><b>Follow the links below to learn more about specific topics:</b></p> <ul> -<li><a href="rsyslog_conf_modules.html">Modules</a></li> -<li><a href="rsyslog_conf_lines.html">Lines</a></li> -<li><a href="rsyslog_conf_global.html">Configuration Directives</a></li> <li><a href="rsyslog_conf_basic_structure.html">Basic Structure</a></li> +<li><a href="rsyslog_conf_modules.html">Modules</a></li> <li><a href="rsyslog_conf_templates.html">Templates</a></li> -<li><a href="rsyslog_conf_output.html">Output Channels</a></li> <li><a href="rsyslog_conf_filter.html">Filter Conditions</a></li> -<li><a href="rsyslog_conf_actions.html">Actions</a></li> -<li><a href="rsyslog_conf_file_syntax_differences.html">Configuration File Syntax Differences</a></li> -<li><a href="rsyslog_conf_examples.html">Examples</a></li> +<li><a href="rsyslog_conf_actions.html">Actions (legacy format)</a></li> +<li><a href="rsyslog_conf_output.html">Output Channels</a></li> +<!--<li><a href="rsyslog_conf_examples.html">Examples</a></li>--> +<li><a href="rsyslog_conf_global.html">Legacy Configuration Directives</a></li> +<li><a href="rsyslog_conf_sysklogd_compatibility.html">sysklogd compatibility</a></li> </ul> <p>[<a href="rsyslog_conf.html">back to top</a>] @@ -38,7 +37,7 @@ especially useful while you are migrating from syslogd to rsyslogd.</p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 2e2293ce..fa240d97 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -4,12 +4,82 @@ <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> <h2>Actions</h2> -<p>The action field of a rule describes what to do with the -message. In general, message content is written to a kind of "logfile". -But also other actions might be done, like writing to a database table -or forwarding to another host.<br> -<br> -Templates can be used with all actions. If used, the specified template +Action object describe what is to be done with a message. They are +implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. +<p>The action object has different parameters: +<ul> +<li>those that apply to all actions and are action specific. These + are documented below. +<li>parameters for the action queue. While they also apply to + all parameters, they are queue-specific, not action-specific (they + are the same that are used in rulesets, for example). +<li>action-specific parameters. These are specific to a certain + type of actions. They are documented by the output module + in question. +</ul> +<h3>General Action Parameters</h3> +<ul> + <li><b>name</b> word + <br>used for statistics gathering and documentation + <li><b>type</b> string + <br>Mandatory parameter for every action. The name of the module that should be used. </li> + <li><b>action.writeAllMarkMessages</b> on/off + <br>Normally, mark messages are written to actions only if the action was not recently executed (by default, recently means within the past 20 minutes). If this setting is switched to "on", mark messages are always sent to actions, no matter how recently they have been executed. In this mode, mark messages can be used as a kind of heartbeat. Note that this option auto-resets to "off", so if you intend to use it with multiple actions, it must be specified in front off all selector lines that should provide this functionality. </li> + <li><b>action.execOnlyEveryNthTime</b> integer + <br>If configured, the next action will only be executed every n-th time. For example, if configured to 3, the first two messages that go into the action will be dropped, the 3rd will actually cause the action to execute, the 4th and 5th will be dropped, the 6th executed under the action, ... and so on. Note: this setting is automatically re-set when the actual action is defined.</li> + <li><b>action.execOnlyEveryNthTimeout</b> integer + <br>Has a meaning only if Action.ExecOnlyEveryNthTime is also configured for the same action. If so, the timeout setting specifies after which period the counting of "previous actions" expires and a new action count is begun. Specify 0 (the default) to disable timeouts. +Why is this option needed? Consider this case: a message comes in at, eg., 10am. That's count 1. Then, nothing happens for the next 10 hours. At 8pm, the next one occurs. That's count 2. Another 5 hours later, the next message occurs, bringing the total count to 3. Thus, this message now triggers the rule. +The question is if this is desired behavior? Or should the rule only be triggered if the messages occur within an e.g. 20 minute window? If the later is the case, you need a +<br>Action.ExecOnlyEveryNthTimeTimeout="1200" +<br>This directive will timeout previous messages seen if they are older than 20 minutes. In the example above, the count would now be always 1 and consequently no rule would ever be triggered. </li> + <li><b>action.execOnlyOnceEveryInterval</b> integer + <br>Execute action only if the last execute is at last <seconds> seconds in the past (more info in ommail, but may be used with any action)</li> + <li><b>action.execOnlyWhenpReviousIsSuspended</b> on/off + <br>This directive allows to specify if actions should always be executed ("off," the default) or only if the previous action is suspended ("on"). This directive works hand-in-hand with the multiple actions per selector feature. It can be used, for example, to create rules that automatically switch destination servers or databases to a (set of) backup(s), if the primary server fails. Note that this feature depends on proper implementation of the suspend feature in the output module. All built-in output modules properly support it (most importantly the database write and the syslog message forwarder).</li> + <li><b>action.repeatedmsgcontainsoriginalmsg</b> on/off + <br>"last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most probably more (this may change from version to version, thus no specific limit is given). The bottom line is that n is large enough to get a good idea which message was repeated but it is not necessarily large enough for the whole message. (Introduced with 4.1.5). Once set, it affects all following actions.</li> + <li><b>action.resumeRetryCount</b> integer + <br>[default 0, -1 means eternal]</li> + <li><b>action.resumeInterval</b> integer + <br>Sets the ActionResumeInterval for the action. The interval provided is always in seconds. Thus, multiply by 60 if you need minutes and 3,600 if you need hours (not recommended). +When an action is suspended (e.g. destination can not be connected), the action is resumed for the configured interval. Thereafter, it is retried. If multiple retires fail, the interval is automatically extended. This is to prevent excessive ressource use for retires. After each 10 retries, the interval is extended by itself. To be precise, the actual interval is (numRetries / 10 + 1) * Action.ResumeInterval. so after the 10th try, it by default is 60 and after the 100th try it is 330.</li> +</ul> + + +<h2>Legacy Format</h2> +<p><b>Be warned that legacy action format is hard to get right. It is +recommended to use RainerScript-Style action format whenever possible!</b> +A key problem with legacy format is that a single action is defined via +multiple configurations lines, which may be spread all across rsyslog.conf. +Even the definition of multiple actions may be intermixed (often not +intentional!). If legacy actions format needs to be used (e.g. some modules +may not yet implement the RainerScript format), it is strongly recommended +to place all configuration statements pertaining to a single action +closely together. +<p>Please also note that legacy action parameters <b>do not</b> affect +RainerScript action objects. So if you define for example: + +<code><pre> +$actionResumeRetryCount 10 +action(type="omfwd" target="server1.example.net") +@@server2.example.net +</pre></code> + +server1's "action.resumeRetryCount" parameter is <b>not</b> set, instead +server2's is! +<p>A goal of the new RainerScript action format was to avoid confusion +which parameters are actually used. As such, it would be counter-productive +to honor legacy action parameters inside a RainerScript definition. As +result, both types of action definitions are strictly (and nicely) +separated from each other. The bottom line is that if RainerScript actions +are used, one does not need to care about which legacy action parameters may +(still...) be in effect. +<p> +<p>Note that not all modules necessarily support legacy action format. +Especially newer modules are recommended to NOT support it. +<h3>Legacy Description</h3> +<p>Templates can be used with many actions. If used, the specified template is used to generate the message content (instead of the default template). To specify a template, write a semicolon after the action value immediately followed by the template name.<br> @@ -331,13 +401,12 @@ one template name for each given action. The default template is specific to each action. For a description of what a template is and what you can do with it, see "TEMPLATES" at the top of this document.</p> - <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_basic_structure.html b/doc/rsyslog_conf_basic_structure.html index 4ce78de0..f5d4891a 100644 --- a/doc/rsyslog_conf_basic_structure.html +++ b/doc/rsyslog_conf_basic_structure.html @@ -1,33 +1,101 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html><head><title>Basic Structure - rsyslog.conf</title></head> <body> +<h1>Basic rsyslog.conf Structure</h1> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Basic Structure</h1> -<p>Rsyslog supports standard sysklogd's configuration file format -and extends it. So in general, you can take a "normal" syslog.conf and -use it together with rsyslogd. It will understand everything. However, -to use most of rsyslogd's unique features, you need to add extended -configuration directives.</p> -<p>Rsyslogd supports the classical, selector-based rule lines. -They are still at the heart of it and all actions are initiated via -rule lines. A rule lines is any line not starting with a $ or the -comment sign (#). Lines starting with $ carry rsyslog-specific -directives.</p> -<p>Every rule line consists of two fields, a selector field and -an action field. These two fields are separated by one or more spaces -or tabs. The selector field specifies a pattern of facilities and -priorities belonging to the specified action.<br> -<br> -Lines starting with a hash mark ("#'') and empty lines are ignored. -</p> +<p>Rsyslog supports three different types of configuration statements +concurrently: +<ul> +<li><b>sysklogd</b> - this is the plain old format, thaught everywhere +and still pretty useful for simple use cases. Note that some very +few constructs are no longer supported because they are incompatible +with newer features. These are mentioned in the compatibility docs. +<li><b>legacy rsyslog</b> - these are statements that begin with a dollar +sign. They set some configuration parameters and modify e.g. the way +actions operate. This is the only format supported in pre-v6 versions of +rsyslog. It is still fully supported in v6 and above. Note that some +plugins and features may still only be available through legacy format +(because plugins need to be explicitely upgraded to use the new style +format, and this hasn't happened to all plugins). +<li><b>RainerScript</b> - the new style format. This is the best and most +precise format to be used for more complex cases. The rest of this page +assumes RainerScript based rsyslog.conf. +</ul> +<p>The rsyslog.conf files consists of statements. For old style (sysklogd & legacy +rsyslog), lines do matter. For new style (RainerScript) line spacing is irrelevant. +Most importantly, this means with new style actions and all other objects can split +across lines as users want to. +<h2>Comments</h2> +<p>There are two types of comments: +<ul> +<li><b>#-Comments</b> - start with a hash sign (#) and run to the end of the line +<li><b>C-style Comments</b> - start with /* and end with */, just like in the C +programming language. They can be used to comment out multiple lines at one. Comment +nesting is not supported, but #-Comments can be contained inside a C-style comment. +</ul> + +<h2>Processing Order</h2> +<p>Directives are processed from the top of rsyslog.conf to the bottom. Sequence +matters. For example, if you stop processing of a message, obviously all statements +after the stop statement are never evaluated. + +<h3>Flow Control Statements</h3> +<ul> +<li><b>if expr then ... else ...</b> - conditional execution +<li><b>stop</b> - stops processing the current message +<li><b>call</b> - calls a ruleset (just like a subroutine call) +<li><b>continue</b> - a NOP, useful e.g. inside the then part of an if +</ul> + +<h3>Data Manipulation Statements</h3> +<ul> +<li><b>set</b> - <a href="http://www.rsyslog.com/how-to-set-variables-in-rsyslog-v7/">sets</a> +a user variable +<li><b>unset</b> - deletes a previously set user variable +</ul> + +<h2>Inputs</h2> +<p>Every input requires an input module to be loaded and a listener defined for it. +Full details can be found inside the <a href="rsyslog_conf_modules.html">rsyslog +modules</a> documentation. Once loaded, inputs are defined via the +<b>input()</b> object. + +<h2>Outputs</h2> +<p>Outputs are also called "actions". A small set of actions is pre-loaded (like +the output file writer, which is used in almost every rsyslog.conf), others must +be loaded just like inputs. +<p>An action is invoked via the <b>action(type="type" ...)</b> object. Type is +mandatory and must contain the name of the plugin to be called (e.g. "omfile" or +"ommongodb"). Other paramters may be present. Their type and use depends on +the output plugin in question. + +<h2>Rulesets and Rules</h2> +<p>Rulesets and rules form the basis of rsyslog processing. In short, a rule +is a way how rsyslog shall process a specific message. Usually, there is a type +of filter (if-statement) in front of the rule. Complex nesting of rules is possible, +much like in a programming language. +<p>Rulesets are containers for rules. A single ruleset can contain many rules. In +the programming language analogy, one may think of a ruleset like being a program. +A ruleset can be "bound" (assigned) to a specific input. In the analogy, this means that when +a message comes in via that input, the "program" (ruleset) bound to it will be executed +(but not any other!). +<p>There is detail documentation available for +<a href="multi_ruleset.html">rsyslog rulesets</a>. +<p>For quick reference, rulesets are defined as follows: +<pre> +ruleset(name="rulesetname") { + action(type="omfile" file="/path/to/file") + action(type="..." ...) + /* and so on... */ +} +</pre> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_examples.html b/doc/rsyslog_conf_examples.html deleted file mode 100644 index b46460e5..00000000 --- a/doc/rsyslog_conf_examples.html +++ /dev/null @@ -1,209 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Examples - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">back</a> -<h2>Examples</h2> -<p>Below are example for templates and selector lines. I hope -they are self-explanatory. If not, please see -www.monitorware.com/rsyslog/ for advise.</p> -<h3>TEMPLATES</h3> -<p>Please note that the samples are split across multiple lines. -A template MUST NOT actually be split across multiple lines.<br> -<br> -A template that resembles traditional syslogd file output:<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> -%syslogtag%%msg:::drop-last-lf%\n"<br> -<br> -A template that tells you a little more about the message:<br> -$template -precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> -%syslogtag%,%msg%\n"<br> -<br> -A template for RFC 3164 format:<br> -$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% -%syslogtag%%msg%"<br> -<br> -A template for the format traditonally used for user messages:<br> -$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> -<br> -And a template with the traditonal wall-message format:<br> -$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at -%timegenerated%<br> -<br> -A template that can be used for the database write (please note the SQL<br> -template option)<br> -$template MySQLInsert,"insert iut, message, receivedat values<br> -('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')<br> -into systemevents\r\n", SQL<br> -<br> -The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> -format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> -format, you do not feel bad if you don't know it ;)). It's interesting -to see how it takes different parts out of the date stamps. What -happens is that the date stamp is split into the actual date and time -and the these two are combined with just a comma in between them.<br> -<br> -$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> -%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,<br> -%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,<br> -%syslogtag%%msg%\n"</p> -<h3>SELECTOR LINES</h3> -<p># Store critical stuff in critical<br> -#<br> -*.=crit;kern.none /var/adm/critical<br> -<br> -This will store all messages with the priority crit in the file -/var/adm/critical, except for any kernel message.<br> -<br> -<br> -# Kernel messages are first, stored in the kernel<br> -# file, critical messages and higher ones also go<br> -# to another host and to the console. Messages to<br> -# the host finlandia are forwarded in RFC 3164<br> -# format (using the template defined above).<br> -#<br> -kern.* /var/adm/kernel<br> -kern.crit @finlandia;RFC3164fmt<br> -kern.crit /dev/console<br> -kern.info;kern.!err /var/adm/kernel-info<br> -<br> -The first rule direct any message that has the kernel facility to the -file /var/adm/kernel.<br> -<br> -The second statement directs all kernel messages of the priority crit -and higher to the remote host finlandia. This is useful, because if the -host crashes and the disks get irreparable errors you might not be able -to read the stored messages. If they're on a remote host, too, you -still can try to find out the reason for the crash.<br> -<br> -The third rule directs these messages to the actual console, so the -person who works on the machine will get them, too.<br> -<br> -The fourth line tells rsyslogd to save all kernel messages that come -with priorities from info up to warning in the file -/var/adm/kernel-info. Everything from err and higher is excluded.<br> -<br> -<br> -# The tcp wrapper loggs with mail.info, we display<br> -# all the connections on tty12<br> -#<br> -mail.=info /dev/tty12<br> -<br> -This directs all messages that uses mail.info (in source LOG_MAIL | -LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper -tcpd(8) uses this as it's default.<br> -<br> -<br> -# Store all mail concerning stuff in a file<br> -#<br> -mail.*;mail.!=info /var/adm/mail<br> -<br> -This pattern matches all messages that come with the mail facility, -except for the info priority. These will be stored in the file -/var/adm/mail.<br> -<br> -<br> -# Log all mail.info and news.info messages to info<br> -#<br> -mail,news.=info /var/adm/info<br> -<br> -This will extract all messages that come either with mail.info or with -news.info and store them in the file /var/adm/info.<br> -<br> -<br> -# Log info and notice messages to messages file<br> -#<br> -*.=info;*.=notice;\<br> -mail.none /var/log/messages<br> -<br> -This lets rsyslogd log all messages that come with either the info or -the notice facility into the file /var/log/messages, except for all<br> -messages that use the mail facility.<br> -<br> -<br> -# Log info messages to messages file<br> -#<br> -*.=info;\<br> -mail,news.none /var/log/messages<br> -<br> -This statement causes rsyslogd to log all messages that come with the -info priority to the file /var/log/messages. But any message coming -either with the mail or the news facility will not be stored.<br> -<br> -<br> -# Emergency messages will be displayed using wall<br> -#<br> -*.=emerg *<br> -<br> -This rule tells rsyslogd to write all emergency messages to all -currently logged in users. This is the wall action.<br> -<br> -<br> -# Messages of the priority alert will be directed<br> -# to the operator<br> -#<br> -*.alert root,rgerhards<br> -<br> -This rule directs all messages with a priority of alert or higher to -the terminals of the operator, i.e. of the users "root'' and -"rgerhards'' if they're logged in.<br> -<br> -<br> -*.* @finlandia<br> -<br> -This rule would redirect all messages to a remote host called -finlandia. This is useful especially in a cluster of machines where all -syslog messages will be stored on only one machine.<br> -<br> -In the format shown above, UDP is used for transmitting the message. -The destination port is set to the default auf 514. Rsyslog is also -capable of using much more secure and reliable TCP sessions for message -forwarding. Also, the destination port can be specified. To select TCP, -simply add one additional @ in front of the host name (that is, @host -is UPD, @@host is TCP). For example:<br> -<br> -<br> -*.* @@finlandia<br> -<br> -To specify the destination port on the remote machine, use a colon -followed by the port number after the machine name. The following -forwards to port 1514 on finlandia:<br> -<br> -<br> -*.* @@finlandia:1514<br> -<br> -This syntax works both with TCP and UDP based syslog. However, you will -probably primarily need it for TCP, as there is no well-accepted port -for this transport (it is non-standard). For UDP, you can usually stick -with the default auf 514, but might want to modify it for security rea-<br> -sons. If you would like to do that, it's quite easy:<br> -<br> -<br> -*.* @finlandia:1514<br> -<br> -<br> -<br> -*.* >dbhost,dbname,dbuser,dbpassword;dbtemplate<br> -<br> -This rule writes all message to the database "dbname" hosted on -"dbhost". The login is done with user "dbuser" and password -"dbpassword". The actual table that is updated is specified within the -template (which contains the insert statement). The template is called -"dbtemplate" in this case.</p> -<p>:msg,contains,"error" @errorServer</p> -<p>This rule forwards all messages that contain the word "error" -in the msg part to the server "errorServer". Forwarding is via UDP. -Please note the colon in fron</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_file_syntax_differences.html b/doc/rsyslog_conf_file_syntax_differences.html deleted file mode 100644 index bfac8926..00000000 --- a/doc/rsyslog_conf_file_syntax_differences.html +++ /dev/null @@ -1,32 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Configuration File Syntax Differences - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Configuration File Syntax Differences</h1> -<p>Rsyslogd uses a slightly different syntax for its -configuration file than the original BSD sources. Originally all -messages of a specific priority and above were forwarded to the log -file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd -more flexible and to use it in a more intuitive manner.<br> -<br> -The original BSD syslogd doesn't understand spaces as separators -between the selector and the action field.<br> -<br> -When compared to syslogd from sysklogd package, rsyslogd offers -additional -<a href="features.html">features</a> (like template -and database support). For obvious reasons, the syntax for defining -such features is available in rsyslogd, only.</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 3 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_filter.html b/doc/rsyslog_conf_filter.html index fbced4a3..a795193f 100644 --- a/doc/rsyslog_conf_filter.html +++ b/doc/rsyslog_conf_filter.html @@ -4,38 +4,95 @@ <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> <h2>Filter Conditions</h2> -<p>Rsyslog offers four different types "filter conditions":</p> +<p>Rsyslog offers three different types "filter conditions":</p> <ul> -<li>BSD-style blocks</li> +<li><a href="http://www.rainerscript.com/">RainerScript</a>-based filters</li> <li>"traditional" severity and facility based selectors</li> <li>property-based filters</li> -<li>expression-based filters</li> </ul> -<h3>Blocks</h3> -<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each -block of lines is separated from the previous block by a program or -hostname specification. A block will only log messages corresponding to -the most recent program and hostname specifications given. Thus, a -block which selects ‘ppp’ as the program, directly followed by a block -that selects messages from the hostname ‘dialhost’, then the second -block will only log messages from the ppp program on dialhost. -</p> -<p>A program specification is a line beginning with ‘!prog’ and -the following blocks will be associated with calls to syslog from that -specific program. A program specification for ‘foo’ will also match any -message logged by the kernel with the prefix ‘foo: ’. Alternatively, a -program specification ‘-foo’ causes the following blocks to be applied -to messages from any program but the one specified. A hostname -specification of the form ‘+hostname’ and the following blocks will be -applied to messages received from the specified hostname. -Alternatively, a hostname specification ‘-hostname’ causes the -following blocks to be applied to messages from any host but the one -specified. If the hostname is given as ‘@’, the local hostname will be -used. (NOT YET IMPLEMENTED) A program or hostname specification may be -reset by giving the program or hostname as ‘*’.</p> -<p>Please note that the "#!prog", "#+hostname" and "#-hostname" -syntax available in BSD syslogd is not supported by rsyslogd. By -default, no hostname or program is set.</p> +<h3>RainerScript-Based Filters</h3> +RainerScript based filters are the prime means of creating complex rsyslog configuration. +The permit filtering on arbitrary complex expressions, which can include boolean, +arithmetic and string operations. They also support full nesting of filters, just +as you know from other scripting environments. +<br> +Scripts based filters are indicated by the keyword "if", as usual. +They have this format:<br> +<br> +if expr then block else block +<br> +"If" and "then" are fixed keywords that mus be present. "expr" is a +(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for +details. +The keyword "else" and its associated block is optional. Note that a block can contain either +a single action (chain), or an arbitrary complex script enclosed in curly braces, e.g.: +<br> +<pre> +if $programname == 'prog1' then { + action(type="omfile" file="/var/log/prog1.log") + if $msg contains 'test' then + action(type="omfile" file="/var/log/prog1test.log") + else + action(type="omfile" file="/var/log/prog1notest.log") +} +</pre> +<br> +Other types of filtes can also be combined with the pure RainerScript ones. This makes +it particularly easy to migrate from early config files to RainerScript. Also, the traditional +syslog PRI-based filters are a good and easy to use addition. While they are legacy, we still +recommend there use where they are up to the job. We do NOT, however, recommend property-based +filters any longer. As an example, the following is perfectly valid: +<br> +<pre> +if $fromhost == 'host1' then { + mail.* action(type="omfile" file="/var/log/host1/mail.log") + *.err /var/log/host1/errlog # this is also still valid + # + # more "old-style rules" ... + # +} else { + mail.* action(type="omfile" file="/var/log/mail.log") + *.err /var/log/errlog + # + # more "old-style rules" ... + # +} +</pre> +<br> + +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. +If you don't like that, you can of course also use the textual property +- just be sure to use the right one. As expression support is enhanced, +this will change. For example, if you would like to filter on message +that have facility local0, start with "DEVNAME" and have either +"error1" or "error0" in their message content, you could use the +following filter:<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +Please note that the above <span style="font-weight: bold;">must +all be on one line</span>! And if you would like to store all +messages except those that contain "error1" or "error0", you just need +to add a "not":<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> +($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +If you would like to do case-insensitive comparisons, use +"contains_i" instead of "contains" and "startswith_i" instead of +"startswith".<br> +<br> +Regular expressions are supported via functions (see function list). + <h3>Selectors</h3> <p><b>Selectors are the traditional way of filtering syslog messages.</b> They have been kept in rsyslog with their original @@ -140,9 +197,14 @@ of the property value. For example, if you search for "val" with <p>it will be a match if msg contains "values are in this message" but it won't match if the msg contains "There are values in this message" (in the later case, contains would match). Please note -that "startswith" is by far faster than regular expressions. So even -once they are implemented, it can make very much sense -(performance-wise) to use "startswith".</p> +that "startswith" is by far faster than regular expressions. So +it makes very much sense (performance-wise) to use "startswith".</p> +<p>Note: when processing syslog messages, please note that $msg usually +starts with a space. The reason for this is RFC3164. Please read the +<a href="http://www.rsyslog.com/log-normalization-and-the-leading-space/">detail +description</a> of what that means to you. In short, you need to make sure +that you include the first space if you use "startswith", otherwise you will +not get matches. </td> </tr> <tr> @@ -213,71 +275,6 @@ supported (except for "not" as outlined above). Please note that while it is possible to query facility and severity via property-based filters, it is far more advisable to use classic selectors (see above) for those cases.</p> -<h3>Expression-Based Filters</h3> -Expression based filters allow -filtering on arbitrary complex expressions, which can include boolean, -arithmetic and string operations. Expression filters will evolve into a -full configuration scripting language. Unfortunately, their syntax will -slightly change during that process. So if you use them now, you need -to be prepared to change your configuration files some time later. -However, we try to implement the scripting facility as soon as possible -(also in respect to stage work needed). So the window of exposure is -probably not too long.<br> -<br> -Expression based filters are indicated by the keyword "if" in column 1 -of a new line. They have this format:<br> -<br> -if expr then action-part-of-selector-line<br> -<br> -"If" and "then" are fixed keywords that mus be present. "expr" is a -(potentially quite complex) expression. So the <a href="expression.html">expression documentation</a> for -details. "action-part-of-selector-line" is an action, just as you know -it (e.g. "/var/log/logfile" to write to that file).<br> -<br> -A few quick samples:<br> -<br> -<code> -*.* /var/log/file1 # the traditional way<br> -if $msg contains 'error' then /var/log/errlog # the expression-based way<br> -</code> -<br> -Right now, you need to specify numerical values if you would like to -check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. -If you don't like that, you can of course also use the textual property -- just be sure to use the right one. As expression support is enhanced, -this will change. For example, if you would like to filter on message -that have facility local0, start with "DEVNAME" and have either -"error1" or "error0" in their message content, you could use the -following filter:<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -Please note that the above <span style="font-weight: bold;">must -all be on one line</span>! And if you would like to store all -messages except those that contain "error1" or "error0", you just need -to add a "not":<br> -<br> -<code> -if $syslogfacility-text == 'local0' and $msg -startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> -($msg contains 'error1' or $msg contains -'error0') then /var/log/somelog<br> -</code> -<br> -If you would like to do case-insensitive comparisons, use -"contains_i" instead of "contains" and "startswith_i" instead of -"startswith".<br> -<br> -Note that regular expressions are currently NOT -supported in expression-based filters. These will be added later when -function support is added to the expression engine (the reason is that -regular expressions will be a separate loadable module, which requires -some more prequisites before it can be implemented).<br> - <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index a4d760eb..e48ed6d3 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -17,12 +17,12 @@ appear as implementation progresses. many parameter settings modify queue parameters. If in doubt, use the default, it is usually well-chosen and applicable in most cases.</p> <ul> -<li><a href="rsconf1_abortonuncleanconfig.html">$AbortOnUncleanConfig</a> - abort startup if there is +<li><a href="rsconf1_abortonuncleanconfig.html"><b>$AbortOnUncleanConfig</b></a> - abort startup if there is any issue with the config file</li> <li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> -<li>$ActionName <a_single_word> - used primarily for documentation, e.g. when +<li><b>$ActionName</b> <a_single_word> - used primarily for documentation, e.g. when generating a configuration graph. Available sice 4.3.1. -<li>$ActionExecOnlyOnceEveryInterval <seconds> - +<li><b>$ActionExecOnlyOnceEveryInterval</b> <seconds> - execute action only if the last execute is at last <seconds> seconds in the past (more info in <a href="ommail.html">ommail</a>, but may be used with any action)</li> @@ -46,60 +46,60 @@ The question is if this is desired behavior? Or should the rule only be triggered if the messages occur within an e.g. 20 minute window? If the later is the case, you need a <br> -$ActionExecOnlyEveryNthTimeTimeout 1200 +<b>$ActionExecOnlyEveryNthTimeTimeout 1200</b> <br> This directive will timeout previous messages seen if they are older than 20 minutes. In the example above, the count would now be always 1 and consequently no rule would ever be triggered. -<li><a href="omfile.html">$ActionFileDefaultTemplate</a> [templateName] - sets a new default template for file actions</li> -<li><a href="omfile.html">$ActionFileEnableSync</a> [on/<span style="font-weight: bold;">off</span>] - enables file +<li><a href="omfile.html"><b>$ActionFileDefaultTemplate</b></a> [templateName] - sets a new default template for file actions</li> +<li><a href="omfile.html"><b>$ActionFileEnableSync</b></a> [on/<span style="font-weight: bold;">off</span>] - enables file syncing capability of omfile</li> -<li><a href="omfwd.html">$ActionForwardDefaultTemplate</a> [templateName] - sets a new +<li><a href="omfwd.html"><b>$ActionForwardDefaultTemplate</b></a> [templateName] - sets a new default template for UDP and plain TCP forwarding action</li> -<li>$ActionGSSForwardDefaultTemplate [templateName] - sets a +<li><b>$ActionGSSForwardDefaultTemplate</b> [templateName] - sets a new default template for GSS-API forwarding action</li> -<li>$ActionQueueCheckpointInterval <number></li> -<li>$ActionQueueDequeueBatchSize <number> [default 16]</li> -<li>$ActionQueueDequeueSlowdown <number> [number +<li><b>$ActionQueueCheckpointInterval</b> <number></li> +<li><b>$ActionQueueDequeueBatchSize</b> <number> [default 16]</li> +<li><b>$ActionQueueDequeueSlowdown</b> <number> [number is timeout in <i> micro</i>seconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!]</li> -<li>$ActionQueueDiscardMark <number> [default +<li><b>$ActionQueueDiscardMark</b> <number> [default 9750]</li> -<li>$ActionQueueDiscardSeverity <number> -[*numerical* severity! default 4 (warning)]</li> -<li>$ActionQueueFileName <name></li> -<li>$ActionQueueHighWaterMark <number> [default +<li><b>$ActionQueueDiscardSeverity</b> <number> +[*numerical* severity! default 8 (nothing discarded)]</li> +<li><b>$ActionQueueFileName</b> <name></li> +<li><b>$ActionQueueHighWaterMark</b> <number> [default 8000]</li> -<li>$ActionQueueImmediateShutdown [on/<b>off</b>]</li> -<li>$ActionQueueSize <number></li> -<li>$ActionQueueLowWaterMark <number> [default +<li><b>$ActionQueueImmediateShutdown</b> [on/<b>off</b>]</li> +<li><b>$ActionQueueSize</b> <number></li> +<li><b>$ActionQueueLowWaterMark</b> <number> [default 2000]</li> -<li>$ActionQueueMaxFileSize <size_nbr>, default 1m</li> -<li>$ActionQueueTimeoutActionCompletion <number> +<li><b>$ActionQueueMaxFileSize</b> <size_nbr>, default 1m</li> +<li><b>$ActionQueueTimeoutActionCompletion</b> <number> [number is timeout in ms (1000ms is 1sec!), default 1000, 0 means immediate!]</li> -<li>$ActionQueueTimeoutEnqueue <number> [number +<li><b>$ActionQueueTimeoutEnqueue</b> <number> [number is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> -<li>$ActionQueueTimeoutShutdown <number> [number +<li><b>$ActionQueueTimeoutShutdown</b> <number> [number is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> -<li>$ActionQueueWorkerTimeoutThreadShutdown +<li><b>$ActionQueueWorkerTimeoutThreadShutdown</b> <number> [number is timeout in ms (1000ms is 1sec!), default 60000 (1 minute)]</li> -<li>$ActionQueueType [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> -<li>$ActionQueueSaveOnShutdown [on/<b>off</b>] +<li><b>$ActionQueueType</b> [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> +<li><b>$ActionQueueSaveOnShutdown </b> [on/<b>off</b>] </li> -<li>$ActionQueueWorkerThreads <number>, num worker threads, default 1, recommended 1</li> -<li>$ActionQueueWorkerThreadMinumumMessages <number>, default 100</li> -<li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li> -<li>$ActionResumeRetryCount <number> [default 0, -1 means eternal]</li> -<li><a href="omfwd.html">$ActionSendResendLastMsgOnReconnect</a> <[on/<b>off</b>]> specifies if the last message is to be resend when a connecition breaks and has been reconnected. May increase reliability, but comes at the risk of message duplication. -<li><a href="omfwd.html">$ActionSendStreamDriver</a> <driver basename> just like $DefaultNetstreamDriver, but for the specific action</li> -<li><a href="omfwd.html">$ActionSendStreamDriverMode</a> <mode>, default 0, mode to use with the stream driver (driver-specific)</li> -<li><a href="omfwd.html">$ActionSendStreamDriverAuthMode</a> <mode>, authentication mode to use with the stream driver. Note that this directive requires TLS +<li><b>$ActionQueueWorkerThreads</b> <number>, num worker threads, default 1, recommended 1</li> +<li><b>$ActionQueueWorkerThreadMinumumMessages</b> <number>, default 100</li> +<li><a href="rsconf1_actionresumeinterval.html"><b>$ActionResumeInterval</b></a></li> +<li><b>$ActionResumeRetryCount</b> <number> [default 0, -1 means eternal]</li> +<li><a href="omfwd.html"><b>$ActionSendResendLastMsgOnReconnect</b></a> <[on/<b>off</b>]> specifies if the last message is to be resend when a connecition breaks and has been reconnected. May increase reliability, but comes at the risk of message duplication. +<li><a href="omfwd.html"><b>$ActionSendStreamDriver</b></a> <driver basename> just like $DefaultNetstreamDriver, but for the specific action</li> +<li><a href="omfwd.html"><b>$ActionSendStreamDriverMode</b></a> <mode>, default 0, mode to use with the stream driver (driver-specific)</li> +<li><a href="omfwd.html"><b>$ActionSendStreamDriverAuthMode</b></a> <mode>, authentication mode to use with the stream driver. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific)</li> -<li><a href="omfwd.html">$ActionSendStreamDriverPermittedPeer</a> <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS +<li><a href="omfwd.html"><b>$ActionSendStreamDriverPermittedPeer</b></a> <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific) -<span style="font-weight: bold;"> directive may go away</span>!</li> <li><a href="omfwd.html"><b>$ActionSendTCPRebindInterval</b> nbr</a>- [available since 4.5.1] - instructs the TCP send @@ -120,40 +120,40 @@ heartbeat. Note that this option auto-resets to "off", so if you inten actions, it must be specified in front off <b>all</b> selector lines that should provide this functionality. </li> -<li><a href="rsconf1_allowedsender.html">$AllowedSender</a></li> -<li><a href="rsconf1_controlcharacterescapeprefix.html">$ControlCharacterEscapePrefix</a></li> -<li><a href="rsconf1_debugprintcfsyslinehandlerlist.html">$DebugPrintCFSyslineHandlerList</a></li> +<li><a href="rsconf1_allowedsender.html"><b>$AllowedSender</b></a></li> +<li><a href="rsconf1_controlcharacterescapeprefix.html"><b>$ControlCharacterEscapePrefix</b></a></li> +<li><a href="rsconf1_debugprintcfsyslinehandlerlist.html"><b>$DebugPrintCFSyslineHandlerList</b></a></li> -<li><a href="rsconf1_debugprintmodulelist.html">$DebugPrintModuleList</a></li> -<li><a href="rsconf1_debugprinttemplatelist.html">$DebugPrintTemplateList</a></li> -<li>$DefaultNetstreamDriver <drivername>, the default <a href="netstream.html">network stream driver</a> to use. Defaults to ptcp.$DefaultNetstreamDriverCAFile </path/to/cafile.pem></li> -<li>$DefaultNetstreamDriverCertFile </path/to/certfile.pem></li> -<li>$DefaultNetstreamDriverKeyFile </path/to/keyfile.pem></li> +<li><a href="rsconf1_debugprintmodulelist.html"><b>$DebugPrintModuleList</b></a></li> +<li><a href="rsconf1_debugprinttemplatelist.html"><b>$DebugPrintTemplateList</b></a></li> +<li><b>$DefaultNetstreamDriver</b> <drivername>, the default <a href="netstream.html">network stream driver</a> to use. Defaults to ptcp.$DefaultNetstreamDriverCAFile </path/to/cafile.pem></li> +<li><b>$DefaultNetstreamDriverCertFile</b> </path/to/certfile.pem></li> +<li><b>$DefaultNetstreamDriverKeyFile</b> </path/to/keyfile.pem></li> <li><b>$DefaultRuleset</b> <i>name</i> - changes the default ruleset for unbound inputs to the provided <i>name</i> (the default default ruleset is named "RSYSLOG_DefaultRuleset"). It is advised to also read our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> <li><a href="omfile.html"><b>$CreateDirs</b></a> [<b>on</b>/off] - create directories on an as-needed basis</li> -<li><a href="omfile.html">$DirCreateMode</a></li> -<li><a href="omfile.html">$DirGroup</a></li> -<li><a href="omfile.html">$DirOwner</a></li> -<li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> -<li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> -<li><a href="omfile.html">$DynaFileCacheSize</a></li> -<li><a href="rsconf1_escape8bitcharsonreceive.html">$Escape8BitCharactersOnReceive</a></li> -<li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> +<li><a href="omfile.html"><b>$DirCreateMode</b></a></li> +<li><a href="omfile.html"><b>$DirGroup</b></a></li> +<li><a href="omfile.html"><b>$DirOwner</b></a></li> +<li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html"><b>$DropMsgsWithMaliciousDnsPTRRecords</b></a></li> +<li><a href="rsconf1_droptrailinglfonreception.html"><b>$DropTrailingLFOnReception</b></a></li> +<li><a href="omfile.html"><b>$DynaFileCacheSize</b></a></li> +<li><a href="rsconf1_escape8bitcharsonreceive.html"><b>$Escape8BitCharactersOnReceive</b></a></li> +<li><a href="rsconf1_escapecontrolcharactersonreceive.html"><b>$EscapeControlCharactersOnReceive</b></a></li> <li><b>$EscapeControlCharactersOnReceive</b> [<b>on</b>|off] - escape USASCII HT character</li> -<li>$SpaceLFOnReceive [on/<b>off</b>] - instructs rsyslogd to replace LF with spaces during message reception (sysklogd compatibility aid)</li> -<li>$ErrorMessagesToStderr [<b>on</b>|off] - direct rsyslogd error message to stderr (in addition to other targets)</li> -<li><a href="omfile.html">$FailOnChownFailure</a></li> -<li><a href="omfile.html">$FileCreateMode</a></li> -<li><a href="omfile.html">$FileGroup</a></li> -<li><a href="omfile.html">$FileOwner</a></li> -<li><a href="rsconf1_generateconfiggraph.html">$GenerateConfigGraph</a></li> -<li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> -<li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> -<li><a href="rsconf1_gssmode.html">$GssMode</a></li> -<li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li><li>MainMsgQueueCheckpointInterval <number></li> +<li><b>$SpaceLFOnReceive</b> [on/<b>off</b>] - instructs rsyslogd to replace LF with spaces during message reception (sysklogd compatibility aid)</li> +<li><b>$ErrorMessagesToStderr</b> [<b>on</b>|off] - direct rsyslogd error message to stderr (in addition to other targets)</li> +<li><a href="omfile.html"><b>$FailOnChownFailure</b></a></li> +<li><a href="omfile.html"><b>$FileCreateMode</b></a></li> +<li><a href="omfile.html"><b>$FileGroup</b></a></li> +<li><a href="omfile.html"><b>$FileOwner</b></a></li> +<li><a href="rsconf1_generateconfiggraph.html"><b>$GenerateConfigGraph</b></a></li> +<li><a href="rsconf1_gssforwardservicename.html"><b>$GssForwardServiceName</b></a></li> +<li><a href="rsconf1_gsslistenservicename.html"><b>$GssListenServiceName</b></a></li> +<li><a href="rsconf1_gssmode.html"><b>$GssMode</b></a></li> +<li><a href="rsconf1_includeconfig.html"><b>$IncludeConfig</b></a></li><li>MainMsgQueueCheckpointInterval <number></li> <li><b>$LocalHostName</b> [name] - this directive permits to overwrite the system hostname with the one specified in the directive. If the directive is given multiple times, all but the last one will be ignored. Please note that startup @@ -166,39 +166,39 @@ This information might be needed by some log analyzers. If set to off, no such status messages are logged, what may be useful for other scenarios. [available since 4.7.0 and 5.3.0] <li><b>$MainMsgQueueDequeueBatchSize</b> <number> [default 32]</li> -<li>$MainMsgQueueDequeueSlowdown <number> [number +<li><b>$MainMsgQueueDequeueSlowdown</b> <number> [number is timeout in <i> micro</i>seconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!]</li> -<li>$MainMsgQueueDiscardMark <number> [default 9750]</li> -<li>$MainMsgQueueDiscardSeverity <severity> +<li><b>$MainMsgQueueDiscardMark</b> <number> [default 9750]</li> +<li><b>$MainMsgQueueDiscardSeverity</b> <severity> [either a textual or numerical severity! default 4 (warning)]</li> -<li>$MainMsgQueueFileName <name></li> -<li>$MainMsgQueueHighWaterMark <number> [default +<li><b>$MainMsgQueueFileName</b> <name></li> +<li><b>$MainMsgQueueHighWaterMark</b> <number> [default 8000]</li> -<li>$MainMsgQueueImmediateShutdown [on/<b>off</b>]</li> -<li><a href="rsconf1_mainmsgqueuesize.html">$MainMsgQueueSize</a></li> -<li>$MainMsgQueueLowWaterMark <number> [default +<li><b>$MainMsgQueueImmediateShutdown</b> [on/<b>off</b>]</li> +<li><a href="rsconf1_mainmsgqueuesize.html"><b>$MainMsgQueueSize</b></a></li> +<li><b>$MainMsgQueueLowWaterMark</b> <number> [default 2000]</li> -<li>$MainMsgQueueMaxFileSize <size_nbr>, default +<li><b>$MainMsgQueueMaxFileSize</b> <size_nbr>, default 1m</li> -<li>$MainMsgQueueTimeoutActionCompletion +<li><b>$MainMsgQueueTimeoutActionCompletion</b> <number> [number is timeout in ms (1000ms is 1sec!), default 1000, 0 means immediate!]</li> -<li>$MainMsgQueueTimeoutEnqueue <number> [number +<li><b>$MainMsgQueueTimeoutEnqueue</b> <number> [number is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> -<li>$MainMsgQueueTimeoutShutdown <number> [number +<li><b>$MainMsgQueueTimeoutShutdown</b> <number> [number is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> -<li>$MainMsgQueueWorkerTimeoutThreadShutdown +<li><b>$MainMsgQueueWorkerTimeoutThreadShutdown</b> <number> [number is timeout in ms (1000ms is 1sec!), default 60000 (1 minute)]</li> -<li>$MainMsgQueueType [<b>FixedArray</b>/LinkedList/Direct/Disk]</li> -<li>$MainMsgQueueSaveOnShutdown [on/<b>off</b>] +<li><b>$MainMsgQueueType</b> [<b>FixedArray</b>/LinkedList/Direct/Disk]</li> +<li><b>$MainMsgQueueSaveOnShutdown </b> [on/<b>off</b>] </li> -<li>$MainMsgQueueWorkerThreads <number>, num +<li><b>$MainMsgQueueWorkerThreads</b> <number>, num worker threads, default 1, recommended 1</li> -<li>$MainMsgQueueWorkerThreadMinumumMessages <number>, default 100</li> -<li><a href="rsconf1_markmessageperiod.html">$MarkMessagePeriod</a> (immark)</li> +<li><b>$MainMsgQueueWorkerThreadMinumumMessages</b> <number>, default 100</li> +<li><a href="rsconf1_markmessageperiod.html"><b>$MarkMessagePeriod</b></a> (immark)</li> <li><b><i>$MaxMessageSize</i></b> <size_nbr>, default 2k - allows to specify maximum supported message size (both for sending and receiving). The default should be sufficient for almost all cases. Do not set this below 1k, as it would cause @@ -221,9 +221,9 @@ instead of UDP (plain TCP syslog, RELP). This resolves the UDP stack size restri <br>Note that 2k, the current default, is the smallest size that must be supported in order to be compliant to the upcoming new syslog RFC series. </li> -<li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li> -<li><a href="rsconf1_moddir.html">$ModDir</a></li> -<li><a href="rsconf1_modload.html">$ModLoad</a></li> +<li><a href="rsconf1_maxopenfiles.html"><b>$MaxOpenFiles</b></a></li> +<li><a href="rsconf1_moddir.html"><b>$ModDir</b></a></li> +<li><a href="rsconf1_modload.html"><b>$ModLoad</b></a></li> <li><a href="omfile.html"><b>$OMFileAsyncWriting</b></a> [on/<b>off</b>], if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order @@ -246,15 +246,15 @@ error recovery thus can handle write errors without data loss. Note that this op severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -on- is primarily an aid to preserve the traditional syslogd behaviour.</li> -<li><a href="omfile.html">$omfileForceChown</a> - force ownership change for all files</li> +<li><a href="omfile.html"><b>$omfileForceChown</b></a> - force ownership change for all files</li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most probably more (this may change from version to version, thus no specific limit is given). The bottom line is that n is large enough to get a good idea which message was repeated but it is not necessarily large enough for the whole message. (Introduced with 4.1.5). Once set, it affects all following actions.</li> -<li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> -<li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> +<li><a href="rsconf1_repeatedmsgreduction.html"><b>$RepeatedMsgReduction</b></a></li> +<li><a href="rsconf1_resetconfigvariables.html"><b>$ResetConfigVariables</b></a></li> <li><b>$Ruleset</b> <i>name</i> - starts a new ruleset or switches back to one already defined. All following actions belong to that new rule set. the <i>name</i> does not yet exist, it is created. To switch back to rsyslog's @@ -268,17 +268,17 @@ a specific (list of) message parsers to be used with the ruleset. <li><b>$OptimizeForUniprocessor</b> [on/<b>off</b>] - turns on optimizatons which lead to better performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The default may change as uniprocessor systems become less common. [available since 4.1.0]</li> -<li>$PreserveFQDN [on/<b>off</b>) - if set to off (legacy default to remain compatible +<li><b>$PreserveFQDN</b> [on/<b>off</b>) - if set to off (legacy default to remain compatible to sysklogd), the domain part from a name that is within the same domain as the receiving system is stripped. If set to on, full names are always used.</li> -<li>$WorkDirectory <name> (directory for spool and other work files. +<li><b>$WorkDirectory</b> <name> (directory for spool and other work files. Do <b>not</b> use trailing slashes)</li> -<li>$UDPServerAddress <IP> (imudp) -- local IP +<li><b>$UDPServerAddress</b> <IP> (imudp) -- local IP address (or name) the UDP listens should bind to</li> -<li>$UDPServerRun <port> (imudp) -- former +<li><b>$UDPServerRun</b> <port> (imudp) -- former -r<port> option, default 514, start UDP server on this port, "*" means all addresses</li> -<li>$UDPServerTimeRequery <nbr-of-times> (imudp) -- this is a performance +<li><b>$UDPServerTimeRequery</b> <nbr-of-times> (imudp) -- this is a performance optimization. Getting the system time is very costly. With this setting, imudp can be instructed to obtain the precise time only once every n-times. This logic is only activated if messages come in at a very fast rate, so doing less frequent @@ -286,10 +286,10 @@ time calls should usually be acceptable. The default value is two, because we ha seen that even without optimization the kernel often returns twice the identical time. You can set this value as high as you like, but do so at your own risk. The higher the value, the less precise the timestamp. -<li><a href="droppriv.html">$PrivDropToGroup</a></li> -<li><a href="droppriv.html">$PrivDropToGroupID</a></li> -<li><a href="droppriv.html">$PrivDropToUser</a></li> -<li><a href="droppriv.html">$PrivDropToUserID</a></li> +<li><a href="droppriv.html"><b>$PrivDropToGroup</b></a></li> +<li><a href="droppriv.html"><b>$PrivDropToGroupID</b></a></li> +<li><a href="droppriv.html"><b>$PrivDropToUser</b></a></li> +<li><a href="droppriv.html"><b>$PrivDropToUserID</b></a></li> <li><b>$Sleep</b> <seconds> - puts the rsyslog main thread to sleep for the specified number of seconds immediately when the directive is encountered. You should have a good reason for using this directive!</li> @@ -306,7 +306,7 @@ rsyslog.conf</b>. Otherwise, if error messages are triggered before this directi is processed, rsyslog will fix the local host IP to "127.0.0.1", what than can not be reset. </li> -<li><a href="rsconf1_umask.html">$UMASK</a></li> +<li><a href="rsconf1_umask.html"><b>$UMASK</b></a></li> </ul> <p><b>Where <size_nbr> or integers are specified above,</b> modifiers can be used after the number part. For example, 1k means diff --git a/doc/rsyslog_conf_lines.html b/doc/rsyslog_conf_lines.html deleted file mode 100644 index 0e6cc0d3..00000000 --- a/doc/rsyslog_conf_lines.html +++ /dev/null @@ -1,23 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Lines - rsyslog.conf</title></head> -<body> -<p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> -<h1>Lines</h1> -<p>Lines can be continued by specifying a backslash ("\") as the last -character of the line. There is a hard-coded maximum line length of 4K.<br> -If you need lines larger than that, you need to change compile-time -settings inside rsyslog and recompile. -</p> - -<p>[<a href="manual.html">manual index</a>] -[<a href="rsyslog_conf.html">rsyslog.conf</a>] -[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 3 or higher.</font></p> -</body> -</html> - diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index cbd60faf..18d6b8a1 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -35,7 +35,7 @@ to message generators. <ul> <li><a href="imfile.html">imfile</a> - input module for text files</li> <li><a href="imrelp.html">imrelp</a> - RELP input module</li> -<li>imudp - udp syslog message input</li> +<li><a href="imudp.html">imudp</a> - udp syslog message input</li> <li><a href="imtcp.html">imtcp</a> - input plugin for tcp syslog</li> <li><a href="imptcp.html">imptcp</a> - input plugin for plain tcp syslog (no TLS but faster)</li> <li><a href="imgssapi.html">imgssapi</a> - input plugin for plain tcp and GSS-enabled syslog</li> @@ -45,6 +45,7 @@ to message generators. <li><a href="imsolaris.html">imsolaris</a> - input for the Sun Solaris system log source</li> <li><a href="im3195.html">im3195</a> - accepts syslog messages via RFC 3195</li> <li><a href="impstats.html">impstats</a> - provides periodic statistics of rsyslog internal counters</li> +<li><a href="imjournal.html">imjournal</a> - Linux journal inuput module</li> </ul> <a name"om"></a><h2>Output Modules</h2> @@ -53,6 +54,7 @@ and messages be transmitted to various different targets. <ul> <li><a href="omfile.html">omfile</a> - file output module</li> <li><a href="omfwd.html">omfwd</a> - syslog forwarding output module</li> +<li><a href="omjournal.html">omjournal</a> - Linux journal output module</li> <li><a href="ompipe.html">ompipe</a> - named pipe output module</li> <li><a href="omusrmsg.html">omusrmsg</a> - user message output module</li> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> @@ -72,6 +74,8 @@ permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> <li><a href="omuxsock.html">omuxsock</a> - output module Unix domain sockets</li> <li><a href="omhdfs.html">omhdfs</a> - output module for Hadoop's HDFS file system</li> +<li><a href="ommongodb.html">ommongodb</a> - output module for MongoDB</li> +<li><a href="omelasticsearch.html">omelasticsearch</a> - output module for ElasticSearch</li> </ul> <a name="pm"></a><h2>Parser Modules</h2> @@ -99,18 +103,18 @@ They can be implemented using either the output module or the parser module inte From the rsyslog core's point of view, they actually are output or parser modules, it is their implementation that makes them special. <p>Currently, there exists only a limited set of such modules, but new ones could be written with -the methods the engine provides. They could be used, for example, to: -<ul> -<li>anonymize message content -<li>add dynamically computed content to message (fields) -</ul> +the methods the engine provides. They could be used, for example, to +add dynamically computed content to message (fields). <p>Message modification modules are usually written for one specific task and thus usually are not generic enough to be reused. However, existing module's code is probably an excellent starting base for writing a new module. Currently, the following modules exist inside the source tree: <ul> +<li><a href="mmanon.html">mmanon</a> - used to anonymize log messages. <li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. Note that this actually is a <b>generic</b> module. +<li><a href="mmjsonparse.html">mmjsonparse</a> - used to interpret CEE/lumberjack +enabled structured log messages. <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. @@ -179,7 +183,7 @@ filter settings. This graphic above is a high-level message flow diagram. [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_sysklogd_compatibility.html b/doc/rsyslog_conf_sysklogd_compatibility.html new file mode 100644 index 00000000..c95d6fda --- /dev/null +++ b/doc/rsyslog_conf_sysklogd_compatibility.html @@ -0,0 +1,31 @@ +<html><head><title>sysklogdcompatibility - rsyslog.conf</title></head> +<body> +<h1>sysklogd compatibility</h1> +<p>This is a part of the rsyslog.conf documentation.</p> +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<p>Rsyslog supports standard sysklogd's configuration file format +and extends it. So in general, you can take a "normal" syslog.conf and +use it together with rsyslogd. It will understand everything. However, +to use most of rsyslogd's unique features, you need to add extended +configuration directives.</p> +<p>Rsyslogd supports the classical, selector-based rule lines. +They are still at the heart of it and all actions are initiated via +rule lines. +However, there are ample new directives, either in rsyslog traditional +format (starting with a dollar sign) or in RainerScript format. These +work together with sysklogd statements. A few select statements are +no longer supported and may generate error messages. They are mentioned +in the compatibility notes. +</p> + +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> + diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index b97f6609..9a6e1619 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -3,7 +3,7 @@ <body> <p>This is a part of the rsyslog.conf - documentation.</p> <a href="rsyslog_conf.html">back</a> -<h2>Templates</h2> +<h1>Templates</h1> <p>Templates are a key feature of rsyslog. They allow to specify any format a user might want. They are also used for dynamic file name @@ -16,67 +16,202 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> -<p>Starting with 5.5.6, there are actually two differnt types of template: +<p>Templates are specified by template() statements. They can also be specified +via $Template legacy statements. Note that these are scheduled for removal in +later versions of rsyslog, so it is probably a good idea to avoid them +for new uses. +<h2>The template() statement</h2> +<p>The template() statement is used to define templates. Note that it is a +<b>static</b> statement, that means all templates are defined when rsyslog +reads the config file. As such, templates are not affected by if-statements +or config nesting. +<p>The basic structure of the template statement is as follows: +<br><br> +<code>template(parameters)</code> +<br><br> +In addition to this simpler syntax, list templates (to be described below) +support an extended syntax: +<br><br> +<code>template(parameters) { list-descriptions }</code> +<p>Each template has a parameter <b>name</b>, which specifies the templates +name, and a parameter <b>type</b>, which specifies the template type. The name +parameter must be unique, and behaviour is unpredictable if it is not. The <b>type</b> +parameter specifies different template types. Different types simply enable +different ways to specify the template content. The template type <b>does not</b> +affect what an (output) plugin can do with it. So use the type that best fits your +needs (from a config writing point of view!). The following types are available: <ul> -<li>string based -<li>string-generator module based +<li>list +<li>subtree +<li>string +<li>plugin </ul> -<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates -have been introduced in 5.5.6. They permit a string generator, actually a C "program", -the generate a format. Obviously, it is more work required to code such a generator, -but the reward is speed improvement. If you do not need the ultimate throughput, you -can forget about string generators (so most people never need to know what they are). -You may just be interested in learning that for the most important default formats, -rsyslog already contains highly optimized string generators and these are called -without any need to configure anything. But if you have written (or purchased) a -string generator module, you need to know how to call it. Each such module has a name, -which you need to know (look it up in the module doc or ask the developer). Let's assume -that "mystrgen" is the module name. Then you can define a template for that strgen -in the following way: +The various types are described below. -<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> -(Of course, you must have first loaded the module via $ModLoad). -<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that -no string follows but a strgen module name. -<p>There are no additional parameters but the module name supported. This is because -there is no way to customize anything inside such a "template" other than by -modifying the code of the string generator. +<h3>list</h3> +<p>In this case, the template is generated by a list of constant and +variable statements. These follow the template spec in curly braces. This type is +also primarily meant for use with structure-aware outputs, like ommongodb. However, +it also works perfectly with text-based outputs. We recommend to use this mode +if more complex property substitutions needs to be done. In that case, the list-based +template syntax is much clearer than the simple string-based one. +<p>The list template contains the template header (with <b>type="list"</b>) and is followed +by <b>constant</b> and <b>property</b> statements, given in curly braces to signify +the template statement they belong to. As the name says, <b>constant</b> statements +describe constant text and <b>property</b> describes property access. There are many options +to <b>property</b>, described further below. Most of these options are used to extract +only partial property contents or to modify the text obtained (like to change its case +to upper or lower case, only). +<p>To grasp the idea, an actual sample is: +<br><pre><code>template(name="tpl1" type="list") { + constant(value="Syslog MSG is: '") + property(name="msg") + constant(value="', ") + property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + constant(value="\n") + } +</code></pre> +<br>This sample is probably primarily targeted at the usual file-based output.</p> -<p>So for most use cases, string-generator module based templates are <b>not</b> -the route to take. Usually, we use <b>string based templates</b> instead. -This is what the rest of the documentation now talks about. -<p>A template consists of a template directive, a name, the -actual template text and optional options. A sample is:</p> -<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template MyTemplateName,"\7Text -%property% some more text\n",<options></code></blockquote> -<p>The "template" (legacy: $template) is the template directive. It tells rsyslog -that this line contains a template. "MyTemplateName" is the template -name. All -other config lines refer to this name. The text within "string" is the -actual template text. The backslash is an escape character, much as it -is in C. It does all these "cool" things. For example, \7 rings the -bell (this is an ASCII value), \n is a new line. C programmers and perl -coders have the advantage of knowing this, but the set in rsyslog is a -bit restricted currently. -</p> -<p>All text in the template is used literally, except for things -within percent signs. These are properties and allow you access to the -contents of the syslog message. Properties are accessed via the -<a href="property_replacer.html">property replacer</a> -(nice name, huh) and it can do cool things, too. For -example, it can pick a substring or do date-specific formatting. More -on this is below, on some lines of the property replacer.<br> -<br> +<h4>constant statement</h4> +<p>This provides a way to specify constant text. The text is used literally. It is +primarily intended for text-based output, so that some constant text can be included. For +example, if a complex template is build for file output, one usually needs to finish it +by a newline, which can be introduced by a constant statement. Here is an actual sample +of that use case from the rsylsog testbench: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +}</code></pre> +The following escape sequences are recogniced inside the constant text: +<ul> +<li>\\ - single backslash +<li>\n - LF +<li>\ooo - (three octal digits) - represents character with this numerical value (e.g. \101 +equals "A"). Note that three +octal digits must be given (in contrast to some languagues, where between one and three are valid). +While we support octal notation, we recommend to use hex notation as this is better known. +<li>\xhh - (where h is a hex digit) - represents character with this numerical value (e.g. \x41 +equals "A"). Note that two hexadecimal digits must be given (in contrast to some languagues +where one or two are valid). +<li>... some others ... list needs to be extended +</ul> +<p>Note: if an unsupported character follows a backslash, this is treated as an error. Behaviour +is unpredictable in this case. +<p>To aid usage of the same template both for text-based outputs and structured ones, constant +text without an "outname" parameter will be ignored when creating the name/value tree +for structured outputs. So if you want to supply some constant text e.g. to mongodb, you must +include an outname, as can be seen here: +<br><pre><code>template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n" <b>outname="IWantThisInMyDB"</b>) +}</code></pre> + +The "constant" statement supports the following parameters: +<ul> +<li>value - the constant value to use +<li>outname - output field name (for structured outputs) +</ul> + + +<h4>property statement</h4> +<p>This statement is used to include property text. It can access all properties. Also, +options permit to specify picking only part of a property or modifying it. +It supports the following parameters: +<ul> +<li>name - the name of the property to access +<li>outname - output field name (for structured outputs) +<li>dateformat - date format to use (only for date-related properties) +<li>caseconversion - permits to convert case of the text. supported values are +"lower" and "upper" +<li>controlcharacters - specifies how to handle control characters. Supported values are +"escape", which escapes them, "space", which replaces them by a single space, and +"drop", which simply removes them from the string. +<li>securepath - used for creating pathnames suitable for use in dynafile templates +<li>format - specifiy format on a field basis. Supported values are "csv", for use when +csv-data is generated, "json", which formats proper json content (but without a field +header) and "jsonf", which formats as a complete json field. +<li>position.from - obtain substring starting from this position (1 is the first position) +<li>position.to - obtain substring up to this position +<li>position.relativeToEnd - the from and to position is relative to the end of the string + instead of the usual start of string. (available since rsyslog v7.3.10) +<li>field.number - obtain this field match +<li>field.delimiter - decimal value of delimiter character for field extraction +<li>regex.expression - expression to use +<li>regex.type - either ERE or BRE +<li>regex.nomatchmode - what to do if we have no match +<li>regex.match - match to use +<li>regex.submatch - submatch to use +<li>droplastlf - drop a trailing LF, if it is present +<li>mandatory - signifies a field as mandatory. If set to "on", this field will always +be present in data passed to structured outputs, even if it is empty. If "off" (the default) +empty fields will not be passed to structured outputs. This is especially useful for outputs +that support dynamic schemas (like ommongodb). +<li>spifno1stsp - expert options for RFC3164 template processing +</ul> + + +<h3>subtree</h3> +<p>Available since rsyslog 7.1.4 +<p> +In this case, the template is generated based on a complete +(CEE) subtree. This type of template is most useful for outputs that know how to +process hierarchical structure, like ommongodb. With that type, the parameter +<b>subtree</b> must be specified, which tells which subtree to use. For example +template(name="tpl1" type="subtree" subtree="$!") includes all CEE data, while +template(name="tpl2" type="subtree" subtree="$!usr!tpl2") includes only the +subtree starting at $!usr!tpl2. The core idea when using this type of template +is that the actual data is prefabricated via set and unset script statements, +and the resulting strucuture is then used inside the template. This method MUST +be used if a complete subtree needs to be placed <i>directly</i> into the +object's root. With all other template types, only subcontainers can be generated. +Note that subtree type can also be used with text-based outputs, like omfile. HOWEVER, +you do not have any capability to specify constant text, and as such cannot include +line breaks. As a consequence, using this template type for text outputs is usually +only useful for debugging or very special cases (e.g. where the text is interpreted +by a JSON parser later on). +<h4>Use case</h4> +<p>A typical use case is to first create a custom subtree and then include it into +the template, like in this small example: +<br><blockquote><code>set $!usr!tpl2!msg = $msg; +<br>set $!usr!tpl2!dataflow = field($msg, 58, 2); +<br>template(name="tpl2" type="subtree" subtree="$!usr!tpl2") +</code></blockquote> +<p>Here, we assume that $msg contains various fields, and the data from a field +is to be extracted and stored - together with the message - as field content. +<h3>string</h3> +<p>This closely resembles the legacy template statement. It +has a mandatory parameter <b>string</b>, which holds the template string to be +applied. A template string is a mix of constant text and replacement variables +(see property replacer). These variables are taken from message or other dynamic +content when the final string to be passed to a plugin is generated. String-based +templates are a great way to specify textual content, especially if no complex +manipulation to properties is necessary. Full details on how to specify template +text can be found below. +<br>Config example: +<br><blockquote><code>template(name="tpl3" type="string" string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") +</code></blockquote> +<h3>plugin</h3> +In this case, the template is generated by a plugin (which +is then called +a "strgen" or "string generator"). The format is fix as it is coded. While this +is inflexible, it provides superior performance, and is often used for that +reason (not that "regular" templates are slow - but in very demanding environments +that "last bit" can make a difference). Refer to the plugin's documentation +for further details. For this type, the paramter <b>plugin</b> must be specified and +must contain the name of the plugin as it identifies itself. Note that the +plugin must be loaded prior to being used inside a template. +<br>Config example: +<br><blockquote><code>template(name="tpl4" type="plugin" plugin="mystrgen") +</code></blockquote> + +<h3>options</h3> The <options> part is optional. It carries options -influencing the template as whole. See details below. Be sure NOT to -mistake template options with property options - the latter ones are -processed by the property replacer and apply to a SINGLE property, only -(and not the whole template).<br> +influencing the template as whole and is part of the template parameters. +See details below. Be sure NOT to mistake template options with property +options - the latter ones are processed by the property replacer and +apply to a SINGLE property, only (and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> <p><b>option.sql</b> - format the string suitable for a SQL @@ -127,50 +262,102 @@ option. Otherwise you will become vulnerable to SQL injection. <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> +template (name="TraditionalFormat" type="string" string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> <br> -Properties can be accessed by the <a href="property_replacer.html">property -replacer</a> (see there for details).</p> -<p>Templates can be used in the form of a <b>list</b> as well. This has been -introduced with <b>6.5.0</b> The list consists of two parts which are either -a <b>constant</b> or a <b>property</b>. The constants -are taking the part of "text" that you usually enter in string-based templates. -The properties stay variable, as they are a substitute for different values of a -certain type. This type of template is extremely useful for complicated cases, -as it helps you to easily keep an overview over the template. Though, it has -the disadvantage of needing more effort to create it.</p> -<br>Config example: -<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { - <br>constant(value="Test: ") - <br>property(name="msg" outname="mymessage") - <br>constant(value=" --!!!-- ") - <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") - <br>constant(value="\n") - <br>} -</code></blockquote> -<p>First, the general template option will be defined. The values of the template -itself get defined in the curly brackets. As it can be seen, we have constants -and properties in exchange. Whereas constants will be filled with a value and probably -some options, properties do direct to a property and the options that could be needed -additional format definitions.</p> -<p>We suggest to use separate lines for all constants and properties. This -helps to keep a good overview over the different parts of the template. -Though, writing it in a single line will work, it is much harder to debug -if anything goes wrong with the template. </p> +<h3>Examples</h3> +<h4>Standard Template for Writing to Files</h4> +<p><pre><code>template(name="FileFormat" type="list") { + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + property(name="msg" droplastlf="on" ) + constant(value="\n") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="FileFormat" type="string" + string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +)</code></pre> +Note that the template string itself must be on a single line. -<p><b>Please note that templates can also be -used to generate selector lines with dynamic file names.</b> For -example, if you would like to split syslog messages from different -hosts to different files (one per host), you can define the following -template:</p> -<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> -<p>Legacy example:</p> -<blockquote><code>$template -DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> -<p>This template can then be used when defining an output -selector line. It will result in something like -"/var/log/system-localhost.log"</p> +<h4>Standard Template for Forwarding to a Remote Host (RFC3164 mode)</h4> +<p><pre><code>template(name="ForwardFormat" type="list") { + constant(value="<") + property(name="PRI") + constant(value="<") + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag" position.from="1" position.to="32") + constant(value=" ") + property(name="msg" spifno1stsp="on" ) + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="forwardFormat" type="string" + string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +)</code></pre> +Note that the template string itself must be on a single line. + +<h4>Standard Template for write to the MySQL database</h4> +<p><pre><code>template(name="StdSQLformat" type="list" option.sql="on") { + constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") + constant(value=" values ('") + property(name="msg") + constant(value="', ") + property(name="syslogfacility") + constant(value=", '") + property(name="hostname") + constant(value="', ") + property(name="syslogpriority") + constant(value=", '") + property(name="timereported" dateFormat="mysql") + constant(value="', '") + property(name="timegenerated" dateFormat="mysql") + constant(value="', ") + property(name="iut") + constant(value=", '") + property(name="syslogtag") + constant(value="')") + } +</code></pre> +<p>The equivalent string template looks like this: +<br><pre><code>template(name="stdSQLformat" type="string" option.sql="on" + string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')" +)</code></pre> +Note that the template string itself must be on a single line. + +<h2>legacy format</h2> +<p>In pre v6-versions of rsyslog, you need to use the <code>$template</code> +statement to configure templates. They provide the equivalent to string- and +plugin-based templates. The legacy syntax continous to work in v7, however +we recommend to avoid legacy format for newly written config files. Legacy and +current config statements can coexist within the same config file. +<p>The general format is +<br><br><code>$template name,param[,options]</code></br></br> +where "name" is the template name and "param" is a single parameter +that specifies template content. The optional "options" part is used to +set template options. +<h3>string</h3> +The parameter is the same string that with the current-style format you +specify in the <b>string</b> parameter, for example: +<br><br><code>$template strtpl,"PRI: %pri%, MSG: %msg%\n"</code> +<p>Note that list templates are not available in legacy format, so you need +to use complex property replacer constructs to do complex things. + +<h3>plugin</h3> +This is equivalent to the "plugin"-type template directive. Here, the +parameter is the plugin name, with an equal sign prepended. An example +is: +<br><br><code>$template plugintpl,=myplugin</code> + +<h2>Reserved Template Names</h2> <p>Template names beginning with "RSYSLOG_" are reserved for rsyslog use. Do NOT use them if, otherwise you may receive a conflict in the future (and @@ -210,12 +397,122 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> + +<h2>The following is legacy documentation soon to be integrated.</h2> + +<!--<table> +<tr><td>param name</td><td>meaning</td></tr> +<tr><td>name</td><td>name of the template</td></tr> +</table> +--> + +<p>Starting with 5.5.6, there are actually two different types of template: +<ul> +<li>string based +<li>string-generator module based +</ul> +<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: + +<blockquote><code>template(name="MyTemplateName" type="plugin" string="mystrgen")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> +(Of course, you must have first loaded the module via $ModLoad). +<p>The important part is the equal sign in the legacy format: it tells the rsyslog config parser that +no string follows but a strgen module name. +<p>There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +<p>So for most use cases, string-generator module based templates are <b>not</b> +the route to take. Usually, we use <b>string based templates</b> instead. +This is what the rest of the documentation now talks about. + +<p>A template consists of a template directive, a name, the +actual template text and optional options. A sample is:</p> +<blockquote><code>template(name="MyTemplateName" type="string" string="Example: Text %property% some more text\n" options)</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template MyTemplateName,"\7Text +%property% some more text\n",<options></code></blockquote> +<p>The "template" (legacy: $template) is the template directive. It tells rsyslog +that this line contains a template. "MyTemplateName" is the template +name. All +other config lines refer to this name. The text within "string" is the +actual template text. The backslash is an escape character, much as it +is in C. It does all these "cool" things. For example, \7 rings the +bell (this is an ASCII value), \n is a new line. C programmers and perl +coders have the advantage of knowing this, but the set in rsyslog is a +bit restricted currently. +</p> +<p>All text in the template is used literally, except for things +within percent signs. These are properties and allow you access to the +contents of the syslog message. Properties are accessed via the +<a href="property_replacer.html">property replacer</a> +(nice name, huh) and it can do cool things, too. For +example, it can pick a substring or do date-specific formatting. More +on this is below, on some lines of the property replacer.<br> +<br> + +<br> +Properties can be accessed by the <a href="property_replacer.html">property +replacer</a> (see there for details).</p> +<p>Templates can be used in the form of a <b>list</b> as well. This has been +introduced with <b>6.5.0</b> The list consists of two parts which are either +a <b>constant</b> or a <b>property</b>. The constants +are taking the part of "text" that you usually enter in string-based templates. +The properties stay variable, as they are a substitute for different values of a +certain type. This type of template is extremely useful for complicated cases, +as it helps you to easily keep an overview over the template. Though, it has +the disadvantage of needing more effort to create it.</p> +<br>Config example: +<br><blockquote><code>template(name="MyTemplate" type="list" option.json="off") { + <br>constant(value="Test: ") + <br>property(name="msg" outname="mymessage") + <br>constant(value=" --!!!-- ") + <br>property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + <br>constant(value="\n") + <br>} +</code></blockquote> +<p>First, the general template option will be defined. The values of the template +itself get defined in the curly brackets. As it can be seen, we have constants +and properties in exchange. Whereas constants will be filled with a value and probably +some options, properties do direct to a property and the options that could be needed +additional format definitions.</p> +<p>We suggest to use separate lines for all constants and properties. This +helps to keep a good overview over the different parts of the template. +Though, writing it in a single line will work, it is much harder to debug +if anything goes wrong with the template. </p> + +<p><b>Please note that templates can also be +used to generate selector lines with dynamic file names.</b> For +example, if you would like to split syslog messages from different +hosts to different files (one per host), you can define the following +template:</p> +<blockquote><code>template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")</code></blockquote> +<p>Legacy example:</p> +<blockquote><code>$template +DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> +<p>This template can then be used when defining an output +selector line. It will result in something like +"/var/log/system-localhost.log"</p> <h3>Legacy String-based Template Samples</h3> -<p>This section provides some sample of what the default formats would -look as a text-based template. Hopefully, their description is self-explanatory. +<p>This section provides some default templates in legacy format, as used in rsyslog +previous to version 6. Note that this format is still supported, so there is no hard need +to upgrade existing configurations. However, it is strongly recommended that the legacy +constructs are not used when crafting new templates. Note that each $Template statement is on a <b>single</b> line, but probably broken accross several lines for display purposes by your browsers. Lines are separated by -empty lines. +empty lines. Keep in mind, that line breaks are important in legacy format. <p><code> $template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" <br><br> @@ -233,7 +530,7 @@ $template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, P [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> diff --git a/doc/sigprov_gt.html b/doc/sigprov_gt.html new file mode 100644 index 00000000..caeee116 --- /dev/null +++ b/doc/sigprov_gt.html @@ -0,0 +1,100 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>GuardTime Log Signature Provider (gt)</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> + +<h1>GuardTime Log Signature Provider (gt)</h1> +<p><b>Signature Provider Name: gt</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Supported Since: </b>since 7.3.9 +<p><b>Description</b>:</p> +<p>Provides the ability to sign syslog messages via the +GuardTime signature services. +</p> + +<p><b>Configuration Parameters</b>:</p> +<p>Signature providers are loaded by omfile, when the +provider is selected in its "sig.providerName" parameter. +Parameters for the provider are given in the omfile action instance +line. +<p>This provider creates a signature file with the same base name but +the extension ".gtsig" for each log file (both for fixed-name files +as well as dynafiles). Both files together form a set. So you need to +archive both in order to prove integrity. +<ul> +<li><b>sig.hashFunction</b> <Hash Algorithm><br> +The following hash algorithms are currently supported: + <ul> + <li>SHA1 + <li>RIPEMD-160 + <li>SHA2-224 + <li>SHA2-256 + <li>SHA2-384 + <li>SHA2-512 + </ul> +</li> +<li><b>sig.timestampService</b> <timestamper URL><br> +This provides the URL of the timestamper service. If not selected, +a default server is selected. This may not necessarily be a good +one for your region. +</li> +<li><b>sig.block.sizeLimit</b> <nbr-records><br> +The maximum number of records inside a single signature block. By +default, there is no size limit, so the signature is only written +on file closure. Note that a signature request typically takes between +one and two seconds. So signing to frequently is probably not a good +idea. +</li> +<li><b>sig.keepRecordHashes</b> <on/<b>off</b>><br> +Controls if record hashes are written to the .gtsig file. This +enhances the ability to spot the location of a signature breach, +but costs considerable disk space (65 bytes for each log record +for SHA2-512 hashes, for example). +</li> +<li><b>sig.keepTreeHashes</b> <on/<b>off</b>><br> +Controls if tree (intermediate) hashes are written to the .gtsig file. This +enhances the ability to spot the location of a signature breach, +but costs considerable disk space (a bit mire than the amount +sig.keepRecordHashes requries). Note that both Tree and Record +hashes can be kept inside the signature file. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>currently none known +</li> +</ul> +<p><b>Samples:</b></p> +<p>This writes a log file with it's associated signature file. Default +parameters are used. +</p> +<textarea rows="3" cols="60"> +action(type="omfile" file="/var/log/somelog" + sig.provider="gt") +</textarea> + +<p>In the next sample, we use the more secure SHA2-512 hash function, +sign every 10,000 records and Tree and Record hashes are kept. +<textarea rows="3" cols="60"> +action(type="omfile" file="/var/log/somelog" + sig.provider="gt" sig.hashfunction="SHA2-512" + sig.block.sizelimit="10000" + sig.keepTreeHashes="on" sig.keepRecordHashes="on") +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2013 by +<a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/v4compatibility.html b/doc/v4compatibility.html index 72b0f5a9..2a51adea 100644 --- a/doc/v4compatibility.html +++ b/doc/v4compatibility.html @@ -60,7 +60,7 @@ restarting rsyslogd by HUPing it. and most other deamons require that a restart command is typed in if a restart is required. <p>Rsyslog will follow this paradigm in the next versions, resulting in many benefits. In v4, we provide some support for the old-style semantics. We introduced a setting $HUPisRestart -which may be set to "on" (tradional, heavy operationg) +which may be set to "on" (tradional, heavy operation) or "off" (new, lightweight "file close only" operation). The initial versions had the default set to traditional behavior, but starting with 4.5.1 we are now using the new behavior as the default. diff --git a/doc/v7compatibility.html b/doc/v7compatibility.html new file mode 100644 index 00000000..da4772fe --- /dev/null +++ b/doc/v7compatibility.html @@ -0,0 +1,138 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v7</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v7</h1> +This document describes things to keep in mind when moving from v6 to v7. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used v6 and want to use v7 without hassle. +<p>Version 7 builds on the new config language introduced in v6 and extends it. +Other than v6, it not just only extends the config language, but provides +considerable changes to core elements as well. The result is much more power and +ease of use as well (this time that is not contradictionary). +</p> +<h2>BSD-Style blocks</h2> +BSD style blocks are no longer supported (for good reason). See the +<a href="http://www.rsyslog.com/g/BSD">rsyslog BSD blocks info</a> +page for more information and how to upgrade your config. +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + +<h2>CEE-Properties</h2> +In rsyslog v6, CEE properties could not be used across disk-based queues. If this was +done, there content was reset. This was a missing feature in v6. In v7, this feature +has been implemented. Consequently, situations where the previous behaviour were +desired need now to be solved differently. We do not think that this will cause any +problems to anyone, especially as in v6 this was announced as a missing feature. + +<h2>omusrmsg: using just a username or "*" is deprecated</h2> +<p>In legacy config format, the asterisk denotes writing the message to all users. +This is usually used for emergency messages and configured like this: +<pre> +*.emerg * +</pre> +<p>Unfortunately, the use of this single character conflicts with other uses, for +example with the multiplication operator. While rsyslog up to versions v7.4 preserves the meaning of +asterisk as an action, it is deprecated and will probably be removed in future versions. +Consequently, a warning message is emitted. To make this warning go away, the action must +be explicitly given, as follows: +<pre> +*.emerg :omusrmsg:* +</pre> +<p>The same holds true for user names. For example +<pre> +*.emerg john +</pre> +<p>at a minimum should be rewritten as +<pre> +*.emerg :omusrmsg:john +</pre> +<p>Of course, for even more clarity the new RainerScript style of action can +also be used: +<pre> +*.emerg action(type="omusrmsg" users="john") +</pre> +<p>In Rainer's blog, there is more +<a href="http://blog.gerhards.net/2011/07/why-omusrmsg-is-evil-and-how-it-is.html">background +information on why omusrmsg needed to be changed</a> available. + +<h2>omruleset and discard (~) action are deprecated</h2> +<p>Both continue to work, but have been replaced by better alternatives. +<p>The discard action (tilde character) has been replaced by the "stop" +RainerScript directive. It is considered more intuitive and offers slightly +better performance. +<p>The omruleset module has been replaced by the "call" RainerScript directive. +Call permits to execute a ruleset like a subroutine, and does so with much +higher performance than omruleset did. Note that omruleset could be run off +an async queue. This was more a side than a desired effect and is not supported +by the call statement. If that effect was needed, it can simply be simulated by +running the called rulesets actions asynchronously (what in any case is the right +way to handle this). +<p>Note that the deprecated modules emit warning messages when being used. +They tell that the construct is deprecated and which statement is to be used +as replacement. This does <b>not</b> affect operations: both modules are still +fully operational and will not be removed in the v7 timeframe. + +<h2>Retries of output plugins that do not do proper replies</h2> +<p>Some output plugins may not be able to detect if their target is capable of +accepting data again after an error (technically, they always return OK when +TryResume is called). Previously, the rsyslog core engine suspended such an action +after 1000 succesive failures. This lead to potentially a large amount of +errors and error messages. Starting with 7.2.1, this has been reduced to 10 +successive failures. This still gives the plugin a chance to recover. In extreme +cases, a plugin may now enter suspend mode where it previously did not do so. +In practice, we do NOT expect that. +<h1>Notes for the 7.3/7.4 branch</h1> +<h2>"last message repeated n times" Processing</h2> +<p>This processing has been optimized and moved to the input side. This results +in usually far better performance and also de-couples different sources +from the same +processing. It is now also integrated in to the more generic rate-limiting +processing. +<h3>User-Noticable Changes</h3> +The code works almost as before, with two exceptions: +<ul> +<li>The supression amount can be different, as the new algorithm + precisely check's a single source, and while that source is being + read. The previous algorithm worked on a set of mixed messages + from multiple sources. +<li>The previous algorithm wrote a "last message repeated n times" message + at least every 60 seconds. For performance reasons, we do no longer do + this but write this message only when a new message arrives or rsyslog + is shut down. +</ul> +<p>Note that the new algorithms needs support from input modules. If old +modules which do not have the necessary support are used, duplicate +messages will most probably not be detected. Upgrading the module code is +simple, and all rsyslog-provided plugins support the new method, so this +should not be a real problem (crafting a solution would result in rather +complex code - for a case that most probably would never happen). +<h3>Performance Implications</h3> +<p>In general, the new method enables far faster output procesing. However, it +needs to be noted that the "last message repeated n" processing needs parsed +messages in order to detect duplicated. Consequently, if it is enabled the +parser step cannot be deferred to the main queue processing thread and +thus must be done during input processing. The changes workload distribution +and may have (good or bad) effect on the overall performance. If you have +a very high performance installation, it is suggested to check the performance +profile before deploying the new version. Note: for high-performance +environments it is highly recommended NOT to use "last message repeated n times" +processing but rather the other (more efficient) rate-limiting methods. These +also do NOT require the parsing step to be done during input processing. + +<h2>Stricter string-template Processing</h2> +<p>Previously, no error message for invalid string template parameters +was generated. +Rather a malformed template was generated, and error information emitted +at runtime. However, this could be quite confusing. Note that the new code +changes user experience: formerly, rsyslog and the affected +actions properly started up, but the actions did not produce proper +data. Now, there are startup error messages and the actions are NOT +executed (due to missing template due to template error). + +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2011-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body></html> diff --git a/grammar/grammar.y b/grammar/grammar.y index cdb19c3d..c5bad689 100644 --- a/grammar/grammar.y +++ b/grammar/grammar.y @@ -6,10 +6,9 @@ * of course, encouraged to use new constructs only. But it needs to be noted * that some of the legacy constructs (specifically the in-front-of-action * PRI filter) are very hard to beat in ease of use, at least for simpler - * cases. So while we hope that cfsysline support can be dropped some time in - * the future, we will probably keep these useful constructs. + * cases. * - * Copyright 2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -37,7 +36,7 @@ #define YYDEBUG 1 extern int yylineno; -/* keep compile rule cleam of errors */ +/* keep compile rule clean of errors */ extern int yylex(void); extern int yyerror(char*); %} @@ -48,26 +47,29 @@ extern int yyerror(char*); es_str_t *estr; enum cnfobjType objType; struct cnfobj *obj; + struct cnfstmt *stmt; struct nvlst *nvlst; struct objlst *objlst; - struct cnfactlst *actlst; struct cnfexpr *expr; - struct cnfrule *rule; + struct cnfarray *arr; struct cnffunc *func; struct cnffparamlst *fparams; } %token <estr> NAME -%token <estr> VALUE %token <estr> FUNC %token <objType> BEGINOBJ %token ENDOBJ -%token <s> CFSYSLINE %token BEGIN_ACTION %token BEGIN_PROPERTY %token BEGIN_CONSTANT %token BEGIN_TPL +%token BEGIN_RULESET %token STOP +%token SET +%token UNSET +%token CONTINUE +%token <cnfstmt> CALL %token <s> LEGACY_ACTION %token <s> LEGACY_RULESET %token <s> PRIFILT @@ -76,6 +78,7 @@ extern int yyerror(char*); %token <s> BSD_HOST_SELECTOR %token IF %token THEN +%token ELSE %token OR %token AND %token NOT @@ -93,33 +96,23 @@ extern int yyerror(char*); %token CMP_STARTSWITH %token CMP_STARTSWITHI -%type <nvlst> nv nvlst +%type <nvlst> nv nvlst value %type <obj> obj property constant %type <objlst> propconst -%type <actlst> actlst -%type <actlst> act -%type <s> cfsysline -%type <actlst> block %type <expr> expr -%type <rule> rule -%type <rule> scriptfilt +%type <stmt> stmt s_act actlst block script %type <fparams> fparams +%type <arr> array arrayelt %left AND OR %left CMP_EQ CMP_NE CMP_LE CMP_GE CMP_LT CMP_GT CMP_CONTAINS CMP_CONTAINSI CMP_STARTSWITH CMP_STARTSWITHI -%left '+' '-' +%left '+' '-' '&' %left '*' '/' '%' %nonassoc UMINUS NOT -%expect 3 -/* these shift/reduce conflicts are created by the CFSYSLINE construct, which we - * unfortunately can not avoid. The problem is that CFSYSLINE can occur both in - * global context as well as within an action. It's not permitted somewhere else, - * but this is suficient for conflicts. The "dangling else" built-in resolution - * works well to solve this issue, so we accept it (it's a wonder that our - * old style grammar doesn't work at all, so we better do not complain...). - * Use "bison -v rscript.y" if more conflicts arise and check rscript.out for - * were exactly these conflicts exits. +%expect 1 /* dangling else */ +/* If more erors show up, Use "bison -v grammar.y" if more conflicts arise and + * check grammar.output for were exactly these conflicts exits. */ %% /* note: we use left recursion below, because that saves stack space AND @@ -128,41 +121,55 @@ extern int yyerror(char*); */ conf: /* empty (to end recursion) */ | conf obj { cnfDoObj($2); } - | conf rule { cnfDoRule($2); } - | conf cfsysline { cnfDoCfsysline($2); } + | conf stmt { cnfDoScript($2); } | conf LEGACY_RULESET { cnfDoCfsysline($2); } | conf BSD_TAG_SELECTOR { cnfDoBSDTag($2); } | conf BSD_HOST_SELECTOR { cnfDoBSDHost($2); } obj: BEGINOBJ nvlst ENDOBJ { $$ = cnfobjNew($1, $2); } - | BEGIN_ACTION nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_ACTION, $2); } | BEGIN_TPL nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_TPL, $2); } | BEGIN_TPL nvlst ENDOBJ '{' propconst '}' { $$ = cnfobjNew(CNFOBJ_TPL, $2); $$->subobjs = $5; } + | BEGIN_RULESET nvlst ENDOBJ '{' script '}' + { $$ = cnfobjNew(CNFOBJ_RULESET, $2); + $$->script = $5; + } propconst: { $$ = NULL; } | propconst property { $$ = objlstAdd($1, $2); } | propconst constant { $$ = objlstAdd($1, $2); } property: BEGIN_PROPERTY nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_PROPERTY, $2); } constant: BEGIN_CONSTANT nvlst ENDOBJ { $$ = cnfobjNew(CNFOBJ_CONSTANT, $2); } -cfsysline: CFSYSLINE { $$ = $1; } nvlst: { $$ = NULL; } | nvlst nv { $2->next = $1; $$ = $2; } -nv: NAME '=' VALUE { $$ = nvlstNew($1, $3); } -rule: PRIFILT actlst { $$ = cnfruleNew(CNFFILT_PRI, $2); $$->filt.s = $1; } - | PROPFILT actlst { $$ = cnfruleNew(CNFFILT_PROP, $2); $$->filt.s = $1; } - | scriptfilt { $$ = $1; } - -scriptfilt: IF expr THEN actlst { $$ = cnfruleNew(CNFFILT_SCRIPT, $4); - $$->filt.expr = $2; } -block: actlst { $$ = $1; } - | block actlst { $2->next = $1; $$ = $2; } -actlst: act { $$=$1; } - | actlst '&' act { $3->next = $1; $$ = $3; } - | actlst cfsysline { $$ = cnfactlstAddSysline($1, $2); } - | '{' block '}' { $$ = $2; } -act: BEGIN_ACTION nvlst ENDOBJ { $$ = cnfactlstNew(CNFACT_V2, $2, NULL); } - | LEGACY_ACTION { $$ = cnfactlstNew(CNFACT_LEGACY, NULL, $1); } +nv: NAME '=' value { $$ = nvlstSetName($3, $1); } +value: STRING { $$ = nvlstNewStr($1); } + | array { $$ = nvlstNewArray($1); } +script: stmt { $$ = $1; } + | script stmt { $$ = scriptAddStmt($1, $2); } +stmt: actlst { $$ = $1; } + | IF expr THEN block { $$ = cnfstmtNew(S_IF); + $$->d.s_if.expr = $2; + $$->d.s_if.t_then = $4; + $$->d.s_if.t_else = NULL; } + | IF expr THEN block ELSE block { $$ = cnfstmtNew(S_IF); + $$->d.s_if.expr = $2; + $$->d.s_if.t_then = $4; + $$->d.s_if.t_else = $6; } + | SET VAR '=' expr ';' { $$ = cnfstmtNewSet($2, $4); } + | UNSET VAR ';' { $$ = cnfstmtNewUnset($2); } + | PRIFILT block { $$ = cnfstmtNewPRIFILT($1, $2); } + | PROPFILT block { $$ = cnfstmtNewPROPFILT($1, $2); } +block: stmt { $$ = $1; } + | '{' script '}' { $$ = $2; } +actlst: s_act { $$ = $1; } + | actlst '&' s_act { $$ = scriptAddStmt($1, $3); } +/* s_act are actions and action-like statements */ +s_act: BEGIN_ACTION nvlst ENDOBJ { $$ = cnfstmtNewAct($2); } + | LEGACY_ACTION { $$ = cnfstmtNewLegaAct($1); } + | STOP { $$ = cnfstmtNew(S_STOP); } + | CALL NAME { $$ = cnfstmtNewCall($2); } + | CONTINUE { $$ = cnfstmtNewContinue(); } expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | expr OR expr { $$ = cnfexprNew(OR, $1, $3); } | NOT expr { $$ = cnfexprNew(NOT, NULL, $2); } @@ -176,6 +183,7 @@ expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | expr CMP_CONTAINSI expr { $$ = cnfexprNew(CMP_CONTAINSI, $1, $3); } | expr CMP_STARTSWITH expr { $$ = cnfexprNew(CMP_STARTSWITH, $1, $3); } | expr CMP_STARTSWITHI expr { $$ = cnfexprNew(CMP_STARTSWITHI, $1, $3); } + | expr '&' expr { $$ = cnfexprNew('&', $1, $3); } | expr '+' expr { $$ = cnfexprNew('+', $1, $3); } | expr '-' expr { $$ = cnfexprNew('-', $1, $3); } | expr '*' expr { $$ = cnfexprNew('*', $1, $3); } @@ -188,8 +196,12 @@ expr: expr AND expr { $$ = cnfexprNew(AND, $1, $3); } | NUMBER { $$ = (struct cnfexpr*) cnfnumvalNew($1); } | STRING { $$ = (struct cnfexpr*) cnfstringvalNew($1); } | VAR { $$ = (struct cnfexpr*) cnfvarNew($1); } + | array { $$ = (struct cnfexpr*) $1; } fparams: expr { $$ = cnffparamlstNew($1, NULL); } | expr ',' fparams { $$ = cnffparamlstNew($1, $3); } +array: '[' arrayelt ']' { $$ = $2; } +arrayelt: STRING { $$ = cnfarrayNew($1); } + | arrayelt ',' STRING { $$ = cnfarrayAdd($1, $3); } %% /* diff --git a/grammar/lexer.l b/grammar/lexer.l index 86fd97b4..ed5d8a80 100644 --- a/grammar/lexer.l +++ b/grammar/lexer.l @@ -45,6 +45,8 @@ /* INCL is in $IncludeConfig processing (skip to include file) */ %x LINENO /* LINENO: support for setting the linenumber */ +%x INCALL + /* INCALL: support for the call statement */ %x EXPR /* EXPR is a bit ugly, but we need it to support pre v6-syntax. The problem * is that cfsysline statement start with $..., the same like variables in @@ -86,8 +88,9 @@ extern int yydebug; /* somehow, I need these prototype even though the headers are * included. I guess that's some autotools magic I don't understand... */ -//char *strdup(char*); +#if !defined(__FreeBSD__) int fileno(FILE *stream); +#endif %} @@ -96,15 +99,20 @@ int fileno(FILE *stream); /* keywords */ "if" { BEGIN EXPR; return IF; } <EXPR>"then" { BEGIN INITIAL; return THEN; } +<EXPR>";" { BEGIN INITIAL; return ';'; } <EXPR>"or" { return OR; } <EXPR>"and" { return AND; } <EXPR>"not" { return NOT; } +<EXPR>"=" | <EXPR>"," | <EXPR>"*" | <EXPR>"/" | <EXPR>"%" | <EXPR>"+" | +<EXPR>"&" | <EXPR>"-" | +<EXPR>"[" | +<EXPR>"]" | <EXPR>"(" | <EXPR>")" { return yytext[0]; } <EXPR>"==" { return CMP_EQ; } @@ -121,7 +129,7 @@ int fileno(FILE *stream); <EXPR>0[0-7]+ | /* octal number */ <EXPR>0x[0-7a-f] | /* hex number, following rule is dec; strtoll handles all! */ <EXPR>([1-9][0-9]*|0) { yylval.n = strtoll(yytext, NULL, 0); return NUMBER; } -<EXPR>\$[$!]{0,1}[a-z][a-z0-9\-_\.]* { yylval.s = strdup(yytext); return VAR; } +<EXPR>\$[$!]{0,1}[a-z][!a-z0-9\-_\.]* { yylval.s = strdup(yytext); return VAR; } <EXPR>\'([^'\\]|\\['"\\$bntr]|\\x[0-9a-f][0-9a-f]|\\[0-7][0-7][0-7])*\' { yytext[yyleng-1] = '\0'; unescapeStr((uchar*)yytext+1, yyleng-2); @@ -135,15 +143,28 @@ int fileno(FILE *stream); <EXPR>[ \t\n] <EXPR>[a-z][a-z0-9_]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); return FUNC; } -<EXPR>. { dbgprintf("invalid char in expr: %s\n", yytext); } +<EXPR>. { parser_errmsg("invalid character '%s' in expression " + "- is there an invalid escape sequence somewhere?", + yytext); } +<INCALL>[ \t\n] +<INCALL>. { parser_errmsg("invalid character '%s' in 'call' statement" + "- is there an invalid escape sequence somewhere?", + yytext); } +<INCALL>[a-zA-Z][a-zA-Z0-9_\.]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); + BEGIN INITIAL; + return NAME; } "&" { return '&'; } "{" { return '{'; } "}" { return '}'; } -"ruleset" { dbgprintf("RULESET\n"); } +"stop" { return STOP; } +"else" { return ELSE; } +"call" { BEGIN INCALL; return CALL; } +"set" { BEGIN EXPR; return SET; } +"unset" { BEGIN EXPR; return UNSET; } +"continue" { return CONTINUE; } /* line number support because the "preprocessor" combines lines and so needs * to tell us the real source line. */ -"stop" { dbgprintf("STOP\n"); return STOP; } "preprocfilelinenumber(" { BEGIN LINENO; } <LINENO>[0-9]+ { yylineno = atoi(yytext) - 1; } <LINENO>")" { BEGIN INITIAL; } @@ -159,6 +180,8 @@ int fileno(FILE *stream); BEGIN INOBJ; return BEGINOBJ; } "template"[ \n\t]*"(" { yylval.objType = CNFOBJ_TPL; BEGIN INOBJ; return BEGIN_TPL; } +"ruleset"[ \n\t]*"(" { yylval.objType = CNFOBJ_RULESET; + BEGIN INOBJ; return BEGIN_RULESET; } "property"[ \n\t]*"(" { yylval.objType = CNFOBJ_PROPERTY; BEGIN INOBJ; return BEGIN_PROPERTY; } "constant"[ \n\t]*"(" { yylval.objType = CNFOBJ_CONSTANT; @@ -168,25 +191,33 @@ int fileno(FILE *stream); "module"[ \n\t]*"(" { yylval.objType = CNFOBJ_MODULE; BEGIN INOBJ; return BEGINOBJ; } "action"[ \n\t]*"(" { BEGIN INOBJ; return BEGIN_ACTION; } -^[ \t]*:\$?[a-z\-]+[ ]*,[ ]*!?[a-z]+[ ]*,[ ]*\".*\" { - yylval.s = strdup(yytext); return PROPFILT; } -^[ \t]*[\*a-z][\*a-z]*[0-7]*[\.,][,!=;\.\*a-z0-7]+ { yylval.s = strdup(yytext); return PRIFILT; } +^[ \t]*:\$?[a-z\-]+[ ]*,[ ]*!?[a-z]+[ ]*,[ ]*\"(\\\"|[^\"])*\" { + yylval.s = strdup(rmLeadingSpace(yytext)); + dbgprintf("lexer: propfilt is '%s'\n", yylval.s); + return PROPFILT; + } +^[ \t]*[\*a-z][\*a-z]*[0-7]*[\.,][,!=;\.\*a-z0-7]+ { yylval.s = strdup(rmLeadingSpace(yytext)); return PRIFILT; } "~" | "*" | \-\/[^*][^\n]* | \/[^*][^\n]* | :[a-z0-9]+:[^\n]* | [\|\.\-\@\^?~>][^\n]+ | -[a-z0-9_][a-z0-9_\-\+,;]* { yylval.s = strdup(yytext); - dbgprintf("lex: LEGA ACT: '%s'\n", yytext); - return LEGACY_ACTION; } +[a-z0-9_][a-z0-9_\-\+,;]* { yylval.s = yytext; return LEGACY_ACTION; } <INOBJ>")" { BEGIN INITIAL; return ENDOBJ; } <INOBJ>[a-z][a-z0-9_\.]* { yylval.estr = es_newStrFromCStr(yytext, yyleng); return NAME; } +<INOBJ>"," | +<INOBJ>"[" | +<INOBJ>"]" | <INOBJ>"=" { return(yytext[0]); } <INOBJ>\"([^"\\]|\\['"?\\abfnrtv]|\\[0-7]{1,3})*\" { - yylval.estr = es_newStrFromBuf(yytext+1, yyleng-2); - return VALUE; } + yytext[yyleng-1] = '\0'; + unescapeStr((uchar*)yytext+1, yyleng-2); + yylval.estr = es_newStrFromBuf(yytext+1, strlen(yytext)-1); + return STRING; } + /*yylval.estr = es_newStrFromBuf(yytext+1, yyleng-2); + return VALUE; }*/ "/*" { preCommentState = YY_START; BEGIN COMMENT; } <INOBJ>"/*" { preCommentState = YY_START; BEGIN COMMENT; } <EXPR>"/*" { preCommentState = YY_START; BEGIN COMMENT; } @@ -194,7 +225,9 @@ int fileno(FILE *stream); <COMMENT>([^*]|\n)+|. <INOBJ>#.*$ /* skip comments in input */ <INOBJ>[ \n\t] -<INOBJ>. { dbgprintf("INOBJ: invalid char '%s'\n", yytext); } +<INOBJ>. { parser_errmsg("invalid character '%s' in object definition " + "- is there an invalid escape sequence somewhere?", + yytext); } \$[a-z]+.*$ { /* see comment on $IncludeConfig above */ if(!strncasecmp(yytext, "$includeconfig ", 14)) { yyless(14); @@ -203,8 +236,7 @@ int fileno(FILE *stream); yylval.s = strdup(yytext); return LEGACY_RULESET; } else { - yylval.s = strdup(yytext); - return CFSYSLINE; + cnfDoCfsysline(strdup(yytext)); } } ![^ \t\n]+[ \t]*$ { yylval.s = strdup(yytext); return BSD_TAG_SELECTOR; } @@ -213,8 +245,9 @@ int fileno(FILE *stream); ^[ \t]*[+-][a-z0-9.:-]+[ \t]*$ { yylval.s = strdup(yytext); return BSD_HOST_SELECTOR; } \#.*\n /* skip comments in input */ [\n\t ] /* drop whitespace */ -. { dbgprintf("invalid char: %s\n", yytext); - } +. { parser_errmsg("invalid character '%s' " + "- is there an invalid escape sequence somewhere?", + yytext); } <<EOF>> { if(popfile() != 0) yyterminate(); } %% @@ -279,6 +312,7 @@ cnfSetLexFile(char *fname) currbs = bs; cnfcurrfn = bs->fn; yylineno = 1; + dbgprintf("config parser: pushed file %s on top of stack\n", fname); done: if(r != 0) { @@ -306,6 +340,7 @@ popfile(void) * necessary, as otherwise we may provide wrong file name information * at the end of include files as well. -- rgerhards, 2011-07-22 */ + dbgprintf("config parser: reached end of file %s\n", bs->fn); yy_delete_buffer(bs->bs); if(bs->prev != NULL) free(bs->fn); @@ -315,12 +350,16 @@ popfile(void) currbs = bs->prev; free(bs); - if(currbs == NULL) + if(currbs == NULL) { + dbgprintf("config parser: parsing completed\n"); return 1; /* all processed */ + } yy_switch_to_buffer(currbs->bs); yylineno = currbs->lineno; cnfcurrfn = currbs->fn; + dbgprintf("config parser: resume parsing of file %s at line %d\n", + cnfcurrfn, yylineno); return 0; } diff --git a/grammar/parserif.h b/grammar/parserif.h index 597cfe40..aa271ec4 100644 --- a/grammar/parserif.h +++ b/grammar/parserif.h @@ -15,7 +15,7 @@ extern int yylineno; * these functions. */ void cnfDoObj(struct cnfobj *o); -void cnfDoRule(struct cnfrule *rule); +void cnfDoScript(struct cnfstmt *script); void cnfDoCfsysline(char *ln); void cnfDoBSDTag(char *ln); void cnfDoBSDHost(char *ln); diff --git a/grammar/rainerscript.c b/grammar/rainerscript.c index 072bdd5d..0f0b469c 100644 --- a/grammar/rainerscript.c +++ b/grammar/rainerscript.c @@ -2,7 +2,7 @@ * * Module begun 2011-07-01 by Rainer Gerhards * - * Copyright 2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -22,7 +22,6 @@ * A copy of the GPL can be found in the file "COPYING" in this distribution. * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. */ - #include "config.h" #include <stdio.h> #include <stdlib.h> @@ -37,16 +36,206 @@ #include <libestr.h> #include "rsyslog.h" #include "rainerscript.h" +#include "conf.h" #include "parserif.h" +#include "rsconf.h" #include "grammar.h" #include "queue.h" #include "srUtils.h" #include "regexp.h" #include "obj.h" +#include "modules.h" +#include "ruleset.h" DEFobjCurrIf(obj) DEFobjCurrIf(regexp) +struct cnfexpr* cnfexprOptimize(struct cnfexpr *expr); +static void cnfstmtOptimizePRIFilt(struct cnfstmt *stmt); +static void cnfarrayPrint(struct cnfarray *ar, int indent); +struct cnffunc * cnffuncNew_prifilt(int fac); + +/* debug support: convert token to a human-readable string. Note that + * this function only supports a single thread due to a static buffer. + * This is deemed a solid solution, as it is intended to be used during + * startup, only. + * NOTE: This function MUST be updated if new tokens are defined in the + * grammar. + */ +char * +tokenToString(int token) +{ + char *tokstr; + static char tokbuf[512]; + + switch(token) { + case NAME: tokstr = "NAME"; break; + case FUNC: tokstr = "FUNC"; break; + case BEGINOBJ: tokstr ="BEGINOBJ"; break; + case ENDOBJ: tokstr ="ENDOBJ"; break; + case BEGIN_ACTION: tokstr ="BEGIN_ACTION"; break; + case BEGIN_PROPERTY: tokstr ="BEGIN_PROPERTY"; break; + case BEGIN_CONSTANT: tokstr ="BEGIN_CONSTANT"; break; + case BEGIN_TPL: tokstr ="BEGIN_TPL"; break; + case BEGIN_RULESET: tokstr ="BEGIN_RULESET"; break; + case STOP: tokstr ="STOP"; break; + case SET: tokstr ="SET"; break; + case UNSET: tokstr ="UNSET"; break; + case CONTINUE: tokstr ="CONTINUE"; break; + case CALL: tokstr ="CALL"; break; + case LEGACY_ACTION: tokstr ="LEGACY_ACTION"; break; + case LEGACY_RULESET: tokstr ="LEGACY_RULESET"; break; + case PRIFILT: tokstr ="PRIFILT"; break; + case PROPFILT: tokstr ="PROPFILT"; break; + case IF: tokstr ="IF"; break; + case THEN: tokstr ="THEN"; break; + case ELSE: tokstr ="ELSE"; break; + case OR: tokstr ="OR"; break; + case AND: tokstr ="AND"; break; + case NOT: tokstr ="NOT"; break; + case VAR: tokstr ="VAR"; break; + case STRING: tokstr ="STRING"; break; + case NUMBER: tokstr ="NUMBER"; break; + case CMP_EQ: tokstr ="CMP_EQ"; break; + case CMP_NE: tokstr ="CMP_NE"; break; + case CMP_LE: tokstr ="CMP_LE"; break; + case CMP_GE: tokstr ="CMP_GE"; break; + case CMP_LT: tokstr ="CMP_LT"; break; + case CMP_GT: tokstr ="CMP_GT"; break; + case CMP_CONTAINS: tokstr ="CMP_CONTAINS"; break; + case CMP_CONTAINSI: tokstr ="CMP_CONTAINSI"; break; + case CMP_STARTSWITH: tokstr ="CMP_STARTSWITH"; break; + case CMP_STARTSWITHI: tokstr ="CMP_STARTSWITHI"; break; + case UMINUS: tokstr ="UMINUS"; break; + default: snprintf(tokbuf, sizeof(tokbuf), "%c[%d]", token, token); + tokstr = tokbuf; break; + } + return tokstr; +} + + +char* +getFIOPName(unsigned iFIOP) +{ + char *pRet; + switch(iFIOP) { + case FIOP_CONTAINS: + pRet = "contains"; + break; + case FIOP_ISEQUAL: + pRet = "isequal"; + break; + case FIOP_STARTSWITH: + pRet = "startswith"; + break; + case FIOP_REGEX: + pRet = "regex"; + break; + case FIOP_EREREGEX: + pRet = "ereregex"; + break; + case FIOP_ISEMPTY: + pRet = "isempty"; + break; + default: + pRet = "NOP"; + break; + } + return pRet; +} + +static void +prifiltInvert(struct funcData_prifilt *prifilt) +{ + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + prifilt->pmask[i] = ~prifilt->pmask[i]; + } +} + +/* set prifilt so that it matches for some severities, sev is its numerical + * value. Mode is one of the compop tokens CMP_EQ, CMP_LT, CMP_LE, CMP_GT, + * CMP_GE, CMP_NE. + */ +static void +prifiltSetSeverity(struct funcData_prifilt *prifilt, int sev, int mode) +{ + static int lessthanmasks[] = { 0x00, 0x01, 0x03, 0x07, 0x0f, 0x1f, 0x3f, 0x7f, 0xff }; + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + if(mode == CMP_EQ || mode == CMP_NE) + prifilt->pmask[i] = 1 << sev; + else if(mode == CMP_LT) + prifilt->pmask[i] = lessthanmasks[sev]; + else if(mode == CMP_LE) + prifilt->pmask[i] = lessthanmasks[sev+1]; + else if(mode == CMP_GT) + prifilt->pmask[i] = ~lessthanmasks[sev+1]; + else if(mode == CMP_GE) + prifilt->pmask[i] = ~lessthanmasks[sev]; + else + DBGPRINTF("prifiltSetSeverity: program error, invalid mode %s\n", + tokenToString(mode)); + } + if(mode == CMP_NE) + prifiltInvert(prifilt); +} + +/* set prifilt so that it matches for some facilities, fac is its numerical + * value. Mode is one of the compop tokens CMP_EQ, CMP_LT, CMP_LE, CMP_GT, + * CMP_GE, CMP_NE. For the given facilities, all severities are enabled. + * NOTE: fac MUST be in the range 0..24 (not multiplied by 8)! + */ +static void +prifiltSetFacility(struct funcData_prifilt *prifilt, int fac, int mode) +{ + int i; + + memset(prifilt->pmask, 0, sizeof(prifilt->pmask)); + switch(mode) { + case CMP_EQ: + prifilt->pmask[fac] = TABLE_ALLPRI; + break; + case CMP_NE: + prifilt->pmask[fac] = TABLE_ALLPRI; + prifiltInvert(prifilt); + break; + case CMP_LT: + for(i = 0 ; i < fac ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_LE: + for(i = 0 ; i < fac+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_GE: + for(i = fac ; i < LOG_NFACILITIES+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + case CMP_GT: + for(i = fac+1 ; i < LOG_NFACILITIES+1 ; ++i) + prifilt->pmask[i] = TABLE_ALLPRI; + break; + default:break; + } +} + +/* combine a prifilt with AND/OR (the respective token values are + * used to keep things simple). + */ +static void +prifiltCombine(struct funcData_prifilt *prifilt, struct funcData_prifilt *prifilt2, int mode) +{ + int i; + for(i = 0 ; i < LOG_NFACILITIES+1 ; ++i) { + if(mode == AND) + prifilt->pmask[i] = prifilt->pmask[i] & prifilt2->pmask[i]; + else + prifilt->pmask[i] = prifilt->pmask[i] | prifilt2->pmask[i]; + } +} + + void readConfFile(FILE *fp, es_str_t **str) { @@ -98,6 +287,14 @@ readConfFile(FILE *fp, es_str_t **str) es_addChar(str, '\0'); } +/* comparison function for qsort() and bsearch() string array compare */ +static int +qs_arrcmp(const void *s1, const void *s2) +{ + return es_strcmp(*((es_str_t**)s1), *((es_str_t**)s2)); +} + + struct objlst* objlstNew(struct cnfobj *o) { @@ -107,7 +304,6 @@ objlstNew(struct cnfobj *o) lst->next = NULL; lst->obj = o; } -dbgprintf("AAAA: creating new objlst\n"); cnfobjPrint(o); return lst; @@ -131,6 +327,22 @@ objlstAdd(struct objlst *root, struct cnfobj *o) return root; } +/* add stmt to current script, always return root stmt pointer */ +struct cnfstmt* +scriptAddStmt(struct cnfstmt *root, struct cnfstmt *s) +{ + struct cnfstmt *l; + + if(root == NULL) { + root = s; + } else { /* find last, linear search ok, as only during config phase */ + for(l = root ; l->next != NULL ; l = l->next) + ; + l->next = s; + } + return root; +} + void objlstDestruct(struct objlst *lst) { @@ -155,13 +367,12 @@ objlstPrint(struct objlst *lst) } struct nvlst* -nvlstNew(es_str_t *name, es_str_t *value) +nvlstNewStr(es_str_t *value) { struct nvlst *lst; if((lst = malloc(sizeof(struct nvlst))) != NULL) { lst->next = NULL; - lst->name = name; lst->val.datatype = 'S'; lst->val.d.estr = value; lst->bUsed = 0; @@ -170,6 +381,28 @@ nvlstNew(es_str_t *name, es_str_t *value) return lst; } +struct nvlst* +nvlstNewArray(struct cnfarray *ar) +{ + struct nvlst *lst; + + if((lst = malloc(sizeof(struct nvlst))) != NULL) { + lst->next = NULL; + lst->val.datatype = 'A'; + lst->val.d.ar = ar; + lst->bUsed = 0; + } + + return lst; +} + +struct nvlst* +nvlstSetName(struct nvlst *lst, es_str_t *name) +{ + lst->name = name; + return lst; +} + void nvlstDestruct(struct nvlst *lst) { @@ -179,8 +412,7 @@ nvlstDestruct(struct nvlst *lst) toDel = lst; lst = lst->next; es_deleteStr(toDel->name); - if(toDel->val.datatype == 'S') - es_deleteStr(toDel->val.d.estr); + varDelete(&toDel->val); free(toDel); } } @@ -192,11 +424,21 @@ nvlstPrint(struct nvlst *lst) dbgprintf("nvlst %p:\n", lst); while(lst != NULL) { name = es_str2cstr(lst->name, NULL); - // TODO: support for non-string types - value = es_str2cstr(lst->val.d.estr, NULL); - dbgprintf("\tname: '%s', value '%s'\n", name, value); + switch(lst->val.datatype) { + case 'A': + dbgprintf("\tname: '%s':\n", name); + cnfarrayPrint(lst->val.d.ar, 5); + break; + case 'S': + value = es_str2cstr(lst->val.d.estr, NULL); + dbgprintf("\tname: '%s', value '%s'\n", name, value); + free(value); + break; + default:dbgprintf("nvlstPrint: unknown type '%s'\n", + tokenToString(lst->val.datatype)); + break; + } free(name); - free(value); lst = lst->next; } } @@ -380,17 +622,17 @@ doGetFileCreateMode(struct nvlst *valnode, struct cnfparamdescr *param, if(es_strlen(valnode->val.d.estr) == 4) { c = es_getBufAddr(valnode->val.d.estr); - if(!( (c[0] == '0') - && (c[1] >= '0' && c[1] <= '7') - && (c[2] >= '0' && c[2] <= '7') - && (c[3] >= '0' && c[3] <= '7') ) ) { + if( (c[0] == '0') + && (c[1] >= '0' && c[1] <= '7') + && (c[2] >= '0' && c[2] <= '7') + && (c[3] >= '0' && c[3] <= '7') ) { fmtOK = 1; } } if(fmtOK) { val->val.datatype = 'N'; - val->val.d.n = (c[1]-'0') * 64 + (c[2]-'0') * 8 + (c[3]-'0');; + val->val.d.n = (c[1]-'0') * 64 + (c[2]-'0') * 8 + (c[3]-'0'); } else { cstr = es_str2cstr(valnode->val.d.estr, NULL); parser_errmsg("file modes need to be specified as " @@ -515,6 +757,7 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, es_size_t i; int r = 1; unsigned char *c; + val->val.datatype = 'S'; val->val.d.estr = es_newStr(32); c = es_getBufAddr(valnode->val.d.estr); @@ -523,7 +766,7 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, } if(i != es_strlen(valnode->val.d.estr)) { parser_errmsg("parameter '%s' contains whitespace, which is not " - "permitted - data after first whitespace ignored", + "permitted", param->name); r = 0; } @@ -531,6 +774,30 @@ doGetWord(struct nvlst *valnode, struct cnfparamdescr *param, } static inline int +doGetArray(struct nvlst *valnode, struct cnfparamdescr *param, + struct cnfparamvals *val) +{ + int r = 1; + + switch(valnode->val.datatype) { + case 'S': + /* a constant string is assumed to be a single-element array */ + val->val.datatype = 'A'; + val->val.d.ar = cnfarrayNew(es_strdup(valnode->val.d.estr)); + break; + case 'A': + val->val.datatype = 'A'; + val->val.d.ar = cnfarrayDup(valnode->val.d.ar); + break; + default:parser_errmsg("parameter '%s' must be an array, but is a " + "different datatype", param->name); + r = 0; + break; + } + return r; +} + +static inline int doGetChar(struct nvlst *valnode, struct cnfparamdescr *param, struct cnfparamvals *val) { @@ -556,8 +823,15 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, uchar *cstr; int r; - dbgprintf("XXXX: in nvlstGetParam, name '%s', type %d, valnode->bUsed %d\n", + DBGPRINTF("nvlstGetParam: name '%s', type %d, valnode->bUsed %d\n", param->name, (int) param->type, valnode->bUsed); + if(valnode->val.datatype != 'S' && param->type != eCmdHdlrArray) { + parser_errmsg("parameter '%s' is not a string, which is not " + "permitted", + param->name); + r = 0; + goto done; + } valnode->bUsed = 1; val->bUsed = 1; switch(param->type) { @@ -613,6 +887,9 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, val->val.d.estr = es_strdup(valnode->val.d.estr); r = 1; break; + case eCmdHdlrArray: + r = doGetArray(valnode, param, val); + break; case eCmdHdlrGoneAway: parser_errmsg("parameter '%s' is no longer supported", param->name); @@ -623,7 +900,7 @@ nvlstGetParam(struct nvlst *valnode, struct cnfparamdescr *param, r = 0; break; } - return r; +done: return r; } @@ -661,8 +938,14 @@ nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, for(i = 0 ; i < params->nParams ; ++i) { param = params->descr + i; - if((valnode = nvlstFindNameCStr(lst, param->name)) == NULL) + if((valnode = nvlstFindNameCStr(lst, param->name)) == NULL) { + if(param->flags & CNFPARAM_REQUIRED) { + parser_errmsg("parameter '%s' required but not specified - " + "fix config", param->name); + bInError = 1; + } continue; + } if(vals[i].bUsed) { parser_errmsg("parameter '%s' specified more than once - " "one instance is ignored. Fix config", param->name); @@ -673,7 +956,6 @@ nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, } } - if(bInError) { if(bValsWasNULL) cnfparamvalsDestruct(vals, params); @@ -684,6 +966,30 @@ nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, } +/* check if at least one cnfparamval is actually set + * returns 1 if so, 0 otherwise + */ +int +cnfparamvalsIsSet(struct cnfparamblk *params, struct cnfparamvals *vals) +{ + int i; + + if(vals == NULL) + return 0; + if(params->version != CNFPARAMBLK_VERSION) { + dbgprintf("nvlstGetParams: invalid param block version " + "%d, expected %d\n", + params->version, CNFPARAMBLK_VERSION); + return 0; + } + for(i = 0 ; i < params->nParams ; ++i) { + if(vals[i].bUsed) + return 1; + } + return 0; +} + + void cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals) { @@ -700,6 +1006,9 @@ cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals) dbgprintf(" '%s'", cstr); free(cstr); break; + case 'A': + cnfarrayPrint(vals[i].val.d.ar, 0); + break; case 'N': dbgprintf("%lld", vals[i].val.d.n); break; @@ -724,6 +1033,7 @@ cnfobjNew(enum cnfobjType objType, struct nvlst *lst) o->objType = objType; o->nvlst = lst; o->subobjs = NULL; + o->script = NULL; } return o; @@ -747,116 +1057,6 @@ cnfobjPrint(struct cnfobj *o) } -struct cnfactlst* -cnfactlstNew(enum cnfactType actType, struct nvlst *lst, char *actLine) -{ - struct cnfactlst *actlst; - - if((actlst = malloc(sizeof(struct cnfactlst))) != NULL) { - actlst->next = NULL; - actlst->syslines = NULL; - actlst->actType = actType; - actlst->lineno = yylineno; - actlst->cnfFile = strdup(cnfcurrfn); - if(actType == CNFACT_V2) - actlst->data.lst = lst; - else - actlst->data.legActLine = actLine; - } - return actlst; -} - -struct cnfactlst* -cnfactlstAddSysline(struct cnfactlst* actlst, char *line) -{ - struct cnfcfsyslinelst *cflst; - - if((cflst = malloc(sizeof(struct cnfcfsyslinelst))) != NULL) { - cflst->line = line; - if(actlst->syslines == NULL) { - cflst->next = NULL; - } else { - cflst->next = actlst->syslines; - } - actlst->syslines = cflst; - } - return actlst; -} - - -void -cnfactlstDestruct(struct cnfactlst *actlst) -{ - struct cnfactlst *toDel; - - while(actlst != NULL) { - toDel = actlst; - actlst = actlst->next; - free(toDel->cnfFile); - cnfcfsyslinelstDestruct(toDel->syslines); - if(toDel->actType == CNFACT_V2) - nvlstDestruct(toDel->data.lst); - else - free(toDel->data.legActLine); - free(toDel); - } - -} - -static inline struct cnfcfsyslinelst* -cnfcfsyslinelstReverse(struct cnfcfsyslinelst *lst) -{ - struct cnfcfsyslinelst *curr, *prev; - if(lst == NULL) - return NULL; - prev = NULL; - while(lst != NULL) { - curr = lst; - lst = lst->next; - curr->next = prev; - prev = curr; - } - return prev; -} - -struct cnfactlst* -cnfactlstReverse(struct cnfactlst *actlst) -{ - struct cnfactlst *curr, *prev; - - prev = NULL; - while(actlst != NULL) { - curr = actlst; - actlst = actlst->next; - curr->syslines = cnfcfsyslinelstReverse(curr->syslines); - curr->next = prev; - prev = curr; - } - return prev; -} - -void -cnfactlstPrint(struct cnfactlst *actlst) -{ - struct cnfcfsyslinelst *cflst; - - while(actlst != NULL) { - dbgprintf("aclst %p: ", actlst); - if(actlst->actType == CNFACT_V2) { - dbgprintf("V2 action type: "); - nvlstPrint(actlst->data.lst); - } else { - dbgprintf("legacy action line: '%s'\n", - actlst->data.legActLine); - } - for( cflst = actlst->syslines - ; cflst != NULL ; cflst = cflst->next) { - dbgprintf("action:cfsysline: '%s'\n", cflst->line); - } - actlst = actlst->next; - } -} - struct cnfexpr* cnfexprNew(unsigned nodetype, struct cnfexpr *l, struct cnfexpr *r) { @@ -883,15 +1083,19 @@ done: * try to convert it to one. The semantics from es_str2num() * are used (bSuccess tells if the conversion went well or not). */ -static inline long long +static long long var2Number(struct var *r, int *bSuccess) { long long n; if(r->datatype == 'S') { n = es_str2num(r->d.estr, bSuccess); } else { - n = r->d.n; - if(bSuccess) + if(r->datatype == 'J') { + n = (r->d.json == NULL) ? 0 : json_object_get_int(r->d.json); + } else { + n = r->d.n; + } + if(bSuccess != NULL) *bSuccess = 1; } return n; @@ -902,14 +1106,223 @@ var2Number(struct var *r, int *bSuccess) static inline es_str_t * var2String(struct var *r, int *bMustFree) { + es_str_t *estr; + char *cstr; + rs_size_t lenstr; if(r->datatype == 'N') { *bMustFree = 1; - return es_newStrFromNumber(r->d.n); + estr = es_newStrFromNumber(r->d.n); + } else if(r->datatype == 'J') { + *bMustFree = 1; + if(r->d.json == NULL) { + cstr = "", + lenstr = 0; + } else { + cstr = (char*)json_object_get_string(r->d.json); + lenstr = strlen(cstr); + } + estr = es_newStrFromCStr(cstr, lenstr); + } else { + *bMustFree = 0; + estr = r->d.estr; + } + return estr; +} + +static uchar* +var2CString(struct var *r, int *bMustFree) +{ + uchar *cstr; + es_str_t *estr; + estr = var2String(r, bMustFree); + cstr = (uchar*) es_str2cstr(estr, NULL); + if(*bMustFree) + es_deleteStr(estr); + *bMustFree = 1; + return cstr; +} + +static rsRetVal +doExtractFieldByChar(uchar *str, uchar delim, int matchnbr, uchar **resstr) +{ + int iCurrFld; + int iLen; + uchar *pBuf; + uchar *pFld; + uchar *pFldEnd; + DEFiRet; + + /* first, skip to the field in question */ + iCurrFld = 1; + pFld = str; + while(*pFld && iCurrFld < matchnbr) { + /* skip fields until the requested field or end of string is found */ + while(*pFld && (uchar) *pFld != delim) + ++pFld; /* skip to field terminator */ + if(*pFld == delim) { + ++pFld; /* eat it */ + ++iCurrFld; + } + } + dbgprintf("field() field requested %d, field found %d\n", matchnbr, iCurrFld); + + if(iCurrFld == matchnbr) { + /* field found, now extract it */ + /* first of all, we need to find the end */ + pFldEnd = pFld; + while(*pFldEnd && *pFldEnd != delim) + ++pFldEnd; + --pFldEnd; /* we are already at the delimiter - so we need to + * step back a little not to copy it as part of the field. */ + /* we got our end pointer, now do the copy */ + iLen = pFldEnd - pFld + 1; /* the +1 is for an actual char, NOT \0! */ + CHKmalloc(pBuf = MALLOC((iLen + 1) * sizeof(char))); + /* now copy */ + memcpy(pBuf, pFld, iLen); + pBuf[iLen] = '\0'; /* terminate it */ + *resstr = pBuf; + } else { + ABORT_FINALIZE(RS_RET_FIELD_NOT_FOUND); } - *bMustFree = 0; - return r->d.estr; +finalize_it: + RETiRet; +} + + +static rsRetVal +doExtractFieldByStr(uchar *str, char *delim, rs_size_t lenDelim, int matchnbr, uchar **resstr) +{ + int iCurrFld; + int iLen; + uchar *pBuf; + uchar *pFld; + uchar *pFldEnd; + DEFiRet; + + /* first, skip to the field in question */ + iCurrFld = 1; + pFld = str; + while(pFld != NULL && iCurrFld < matchnbr) { + if((pFld = (uchar*) strstr((char*)pFld, delim)) != NULL) { + pFld += lenDelim; + ++iCurrFld; + } + } + dbgprintf("field() field requested %d, field found %d\n", matchnbr, iCurrFld); + + if(iCurrFld == matchnbr) { + /* field found, now extract it */ + /* first of all, we need to find the end */ + pFldEnd = (uchar*) strstr((char*)pFld, delim); + if(pFldEnd == NULL) { + iLen = strlen((char*) pFld); + } else { /* found delmiter! Note that pFldEnd *is* already on + * the first delmi char, we don't need that. */ + iLen = pFldEnd - pFld; + } + /* we got our end pointer, now do the copy */ + CHKmalloc(pBuf = MALLOC((iLen + 1) * sizeof(char))); + /* now copy */ + memcpy(pBuf, pFld, iLen); + pBuf[iLen] = '\0'; /* terminate it */ + *resstr = pBuf; + } else { + ABORT_FINALIZE(RS_RET_FIELD_NOT_FOUND); + } +finalize_it: + RETiRet; } +static inline void +doFunc_re_extract(struct cnffunc *func, struct var *ret, void* usrptr) +{ + size_t submatchnbr; + short matchnbr; + regmatch_t pmatch[50]; + int bMustFree; + es_str_t *estr; + char *str; + struct var r[CNFFUNC_MAX_ARGS]; + int iLenBuf; + unsigned iOffs; + short iTry = 0; + uchar bFound = 0; + iOffs = 0; + sbool bHadNoMatch = 0; + + cnfexprEval(func->expr[0], &r[0], usrptr); + /* search string is already part of the compiled regex, so we don't + * need it here! + */ + cnfexprEval(func->expr[2], &r[2], usrptr); + cnfexprEval(func->expr[3], &r[3], usrptr); + str = (char*) var2CString(&r[0], &bMustFree); + matchnbr = (short) var2Number(&r[2], NULL); + submatchnbr = (size_t) var2Number(&r[3], NULL); + if(submatchnbr > sizeof(pmatch)/sizeof(regmatch_t)) { + DBGPRINTF("re_extract() submatch %d is too large\n", submatchnbr); + bHadNoMatch = 1; + goto finalize_it; + } + + /* first see if we find a match, iterating through the series of + * potential matches over the string. + */ + while(!bFound) { + int iREstat; + iREstat = regexp.regexec(func->funcdata, (char*)(str + iOffs), + submatchnbr+1, pmatch, 0); + dbgprintf("re_extract: regexec return is %d\n", iREstat); + if(iREstat == 0) { + if(pmatch[0].rm_so == -1) { + dbgprintf("oops ... start offset of successful regexec is -1\n"); + break; + } + if(iTry == matchnbr) { + bFound = 1; + } else { + dbgprintf("re_extract: regex found at offset %d, new offset %d, tries %d\n", + iOffs, (int) (iOffs + pmatch[0].rm_eo), iTry); + iOffs += pmatch[0].rm_eo; + ++iTry; + } + } else { + break; + } + } + dbgprintf("re_extract: regex: end search, found %d\n", bFound); + if(!bFound) { + bHadNoMatch = 1; + goto finalize_it; + } else { + /* Match- but did it match the one we wanted? */ + /* we got no match! */ + if(pmatch[submatchnbr].rm_so == -1) { + bHadNoMatch = 1; + goto finalize_it; + } + /* OK, we have a usable match - we now need to malloc pB */ + iLenBuf = pmatch[submatchnbr].rm_eo - pmatch[submatchnbr].rm_so; + estr = es_newStrFromBuf(str + iOffs + pmatch[submatchnbr].rm_so, + iLenBuf); + } + + if(bMustFree) free(str); + if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + if(r[2].datatype == 'S') es_deleteStr(r[2].d.estr); + if(r[3].datatype == 'S') es_deleteStr(r[3].d.estr); +finalize_it: + if(bHadNoMatch) { + cnfexprEval(func->expr[4], &r[4], usrptr); + estr = var2String(&r[4], &bMustFree); + if(r[4].datatype == 'S') es_deleteStr(r[4].d.estr); + } + ret->datatype = 'S'; + ret->d.estr = estr; + return; +} + + /* Perform a function call. This has been moved out of cnfExprEval in order * to keep the code small and easier to maintain. */ @@ -921,8 +1334,13 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) int bMustFree; es_str_t *estr; char *str; + uchar *resStr; int retval; struct var r[CNFFUNC_MAX_ARGS]; + int delim; + int matchnbr; + struct funcData_prifilt *pPrifilt; + rsRetVal localRet; dbgprintf("rainerscript: executing function id %d\n", func->fID); switch(func->fID) { @@ -950,8 +1368,12 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) estr = var2String(&r[0], &bMustFree); str = (char*) es_str2cstr(estr, NULL); envvar = getenv(str); + if(envvar == NULL) { + ret->d.estr = es_newStr(0); + } else { + ret->d.estr = es_newStrFromCStr(envvar, strlen(envvar)); + } ret->datatype = 'S'; - ret->d.estr = es_newStrFromCStr(envvar, strlen(envvar)); if(bMustFree) es_deleteStr(estr); if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); free(str); @@ -990,8 +1412,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) break; case CNFFUNC_RE_MATCH: cnfexprEval(func->expr[0], &r[0], usrptr); - estr = var2String(&r[0], &bMustFree); - str = es_str2cstr(estr, NULL); + str = (char*) var2CString(&r[0], &bMustFree); retval = regexp.regexec(func->funcdata, str, 0, NULL, 0); if(retval == 0) ret->d.n = 1; @@ -1002,10 +1423,54 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } } ret->datatype = 'N'; - if(bMustFree) es_deleteStr(estr); - free(str); + if(bMustFree) free(str); if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); break; + case CNFFUNC_RE_EXTRACT: + doFunc_re_extract(func, ret, usrptr); + break; + case CNFFUNC_FIELD: + cnfexprEval(func->expr[0], &r[0], usrptr); + cnfexprEval(func->expr[1], &r[1], usrptr); + cnfexprEval(func->expr[2], &r[2], usrptr); + str = (char*) var2CString(&r[0], &bMustFree); + matchnbr = var2Number(&r[2], NULL); + if(r[1].datatype == 'S') { + char *delimstr; + delimstr = (char*) es_str2cstr(r[1].d.estr, NULL); + localRet = doExtractFieldByStr((uchar*)str, delimstr, es_strlen(r[1].d.estr), + matchnbr, &resStr); + free(delimstr); + } else { + delim = var2Number(&r[1], NULL); + localRet = doExtractFieldByChar((uchar*)str, (char) delim, matchnbr, &resStr); + } + if(localRet == RS_RET_OK) { + ret->d.estr = es_newStrFromCStr((char*)resStr, strlen((char*)resStr)); + free(resStr); + } else if(localRet == RS_RET_FIELD_NOT_FOUND) { + ret->d.estr = es_newStrFromCStr("***FIELD NOT FOUND***", + sizeof("***FIELD NOT FOUND***")-1); + } else { + ret->d.estr = es_newStrFromCStr("***ERROR in field() FUNCTION***", + sizeof("***ERROR in field() FUNCTION***")-1); + } + ret->datatype = 'S'; + if(bMustFree) free(str); + if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + if(r[1].datatype == 'S') es_deleteStr(r[1].d.estr); + if(r[2].datatype == 'S') es_deleteStr(r[2].d.estr); + break; + case CNFFUNC_PRIFILT: + pPrifilt = (struct funcData_prifilt*) func->funcdata; + if( (pPrifilt->pmask[((msg_t*)usrptr)->iFacility] == TABLE_NOPRI) || + ((pPrifilt->pmask[((msg_t*)usrptr)->iFacility] + & (1<<((msg_t*)usrptr)->iSeverity)) == 0) ) + ret->d.n = 0; + else + ret->d.n = 1; + ret->datatype = 'N'; + break; default: if(Debug) { fname = es_str2cstr(func->fname, NULL); @@ -1018,6 +1483,67 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } } +static inline void +evalVar(struct cnfvar *var, void *usrptr, struct var *ret) +{ + rsRetVal localRet; + es_str_t *estr; + struct json_object *json; + + if(var->name[0] == '$' && var->name[1] == '!') { + /* TODO: unify string libs */ + estr = es_newStrFromBuf(var->name+1, strlen(var->name)-1); + localRet = msgGetCEEPropJSON((msg_t*)usrptr, estr, &json); + es_deleteStr(estr); + ret->datatype = 'J'; + ret->d.json = (localRet == RS_RET_OK) ? json : NULL; + } else { + ret->datatype = 'S'; + ret->d.estr = cnfGetVar(var->name, usrptr); + } + +} + +/* perform a string comparision operation against a while array. Semantic is + * that one one comparison is true, the whole construct is true. + * TODO: we can obviously optimize this process. One idea is to + * compile a regex, which should work faster than serial comparison. + * Note: compiling a regex does NOT work at all. I experimented with that + * and it was generally 5 to 10 times SLOWER than what we do here... + */ +static int +evalStrArrayCmp(es_str_t *estr_l, struct cnfarray* ar, int cmpop) +{ + int i; + int r = 0; + es_str_t **res; + if(cmpop == CMP_EQ) { + res = bsearch(&estr_l, ar->arr, ar->nmemb, sizeof(es_str_t*), qs_arrcmp); + r = res != NULL; + } else if(cmpop == CMP_NE) { + res = bsearch(&estr_l, ar->arr, ar->nmemb, sizeof(es_str_t*), qs_arrcmp); + r = res == NULL; + } else { + for(i = 0 ; (r == 0) && (i < ar->nmemb) ; ++i) { + switch(cmpop) { + case CMP_STARTSWITH: + r = es_strncmp(estr_l, ar->arr[i], es_strlen(ar->arr[i])) == 0; + break; + case CMP_STARTSWITHI: + r = es_strncasecmp(estr_l, ar->arr[i], es_strlen(ar->arr[i])) == 0; + break; + case CMP_CONTAINS: + r = es_strContains(estr_l, ar->arr[i]) != -1; + break; + case CMP_CONTAINSI: + r = es_strCaseContains(estr_l, ar->arr[i]) != -1; + break; + } + } + } + return r; +} + #define FREE_BOTH_RET \ if(r.datatype == 'S') es_deleteStr(r.d.estr); \ if(l.datatype == 'S') es_deleteStr(l.d.estr) @@ -1029,16 +1555,27 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) ret->d.n = var2Number(&l, &convok_l) x var2Number(&r, &convok_r); \ FREE_BOTH_RET +/* NOTE: array as right-hand argument MUST be handled by user */ #define PREP_TWO_STRINGS \ cnfexprEval(expr->l, &l, usrptr); \ estr_l = var2String(&l, &bMustFree2); \ - cnfexprEval(expr->r, &r, usrptr); \ - estr_r = var2String(&r, &bMustFree) + if(expr->r->nodetype == 'S') { \ + estr_r = ((struct cnfstringval*)expr->r)->estr;\ + bMustFree = 0; \ + } else if(expr->r->nodetype != 'A') { \ + cnfexprEval(expr->r, &r, usrptr); \ + estr_r = var2String(&r, &bMustFree); \ + } else { \ + /* Note: this is not really necessary, but if we do not */ \ + /* do it, we get a very irritating compiler warning... */ \ + estr_r = NULL; \ + } #define FREE_TWO_STRINGS \ - if(bMustFree) es_deleteStr(estr_r); \ - if(bMustFree2) es_deleteStr(estr_l); \ - FREE_BOTH_RET + if(bMustFree) es_deleteStr(estr_r); \ + if(expr->r->nodetype != 'S' && expr->r->nodetype != 'A' && r.datatype == 'S') es_deleteStr(r.d.estr); \ + if(bMustFree2) es_deleteStr(estr_l); \ + if(l.datatype == 'S') es_deleteStr(l.d.estr) /* evaluate an expression. * Note that we try to avoid malloc whenever possible (because of @@ -1059,29 +1596,63 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) int bMustFree, bMustFree2; long long n_r, n_l; - dbgprintf("eval expr %p, type '%c'(%u)\n", expr, expr->nodetype, expr->nodetype); + dbgprintf("eval expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); switch(expr->nodetype) { /* note: comparison operations are extremely similar. The code can be copyied, only * places flagged with "CMP" need to be changed. */ case CMP_EQ: + /* this is optimized in regard to right param as a PoC for all compOps + * So this is a NOT yet the copy template! + */ cnfexprEval(expr->l, &l, usrptr); - cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; if(l.datatype == 'S') { - if(r.datatype == 'S') { - ret->d.n = !es_strcmp(l.d.estr, r.d.estr); /*CMP*/ + if(expr->r->nodetype == 'S') { + ret->d.n = !es_strcmp(l.d.estr, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(l.d.estr, (struct cnfarray*) expr->r, CMP_EQ); } else { - n_l = var2Number(&l, &convok_l); - if(convok_l) { - ret->d.n = (n_l == r.d.n); /*CMP*/ + cnfexprEval(expr->r, &r, usrptr); + if(r.datatype == 'S') { + ret->d.n = !es_strcmp(l.d.estr, r.d.estr); /*CMP*/ } else { - estr_r = var2String(&r, &bMustFree); - ret->d.n = !es_strcmp(l.d.estr, estr_r); /*CMP*/ - if(bMustFree) es_deleteStr(estr_r); + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l == r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = !es_strcmp(l.d.estr, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } } + if(r.datatype == 'S') es_deleteStr(r.d.estr); } + } else if(l.datatype == 'J') { + estr_l = var2String(&l, &bMustFree); + if(expr->r->nodetype == 'S') { + ret->d.n = !es_strcmp(estr_l, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_EQ); + } else { + cnfexprEval(expr->r, &r, usrptr); + if(r.datatype == 'S') { + ret->d.n = !es_strcmp(estr_l, r.d.estr); /*CMP*/ + } else { + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l == r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = !es_strcmp(estr_l, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } + } + if(r.datatype == 'S') es_deleteStr(r.d.estr); + } + if(bMustFree) es_deleteStr(estr_l); } else { + cnfexprEval(expr->r, &r, usrptr); if(r.datatype == 'S') { n_r = var2Number(&r, &convok_r); if(convok_r) { @@ -1094,24 +1665,31 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) } else { ret->d.n = (l.d.n == r.d.n); /*CMP*/ } + if(r.datatype == 'S') es_deleteStr(r.d.estr); } - FREE_BOTH_RET; + if(l.datatype == 'S') es_deleteStr(l.d.estr); break; case CMP_NE: cnfexprEval(expr->l, &l, usrptr); cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; if(l.datatype == 'S') { - if(r.datatype == 'S') { - ret->d.n = es_strcmp(l.d.estr, r.d.estr); /*CMP*/ + if(expr->r->nodetype == 'S') { + ret->d.n = es_strcmp(l.d.estr, ((struct cnfstringval*)expr->r)->estr); /*CMP*/ + } else if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(l.d.estr, (struct cnfarray*) expr->r, CMP_NE); } else { - n_l = var2Number(&l, &convok_l); - if(convok_l) { - ret->d.n = (n_l != r.d.n); /*CMP*/ + if(r.datatype == 'S') { + ret->d.n = es_strcmp(l.d.estr, r.d.estr); /*CMP*/ } else { - estr_r = var2String(&r, &bMustFree); - ret->d.n = es_strcmp(l.d.estr, estr_r); /*CMP*/ - if(bMustFree) es_deleteStr(estr_r); + n_l = var2Number(&l, &convok_l); + if(convok_l) { + ret->d.n = (n_l != r.d.n); /*CMP*/ + } else { + estr_r = var2String(&r, &bMustFree); + ret->d.n = es_strcmp(l.d.estr, estr_r); /*CMP*/ + if(bMustFree) es_deleteStr(estr_r); + } } } } else { @@ -1265,25 +1843,45 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) case CMP_STARTSWITH: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strncmp(estr_l, estr_r, estr_r->lenStr) == 0; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_STARTSWITH); + bMustFree = 0; + } else { + ret->d.n = es_strncmp(estr_l, estr_r, estr_r->lenStr) == 0; + } FREE_TWO_STRINGS; break; case CMP_STARTSWITHI: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strncasecmp(estr_l, estr_r, estr_r->lenStr) == 0; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_STARTSWITHI); + bMustFree = 0; + } else { + ret->d.n = es_strncasecmp(estr_l, estr_r, estr_r->lenStr) == 0; + } FREE_TWO_STRINGS; break; case CMP_CONTAINS: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strContains(estr_l, estr_r) != -1; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_CONTAINS); + bMustFree = 0; + } else { + ret->d.n = es_strContains(estr_l, estr_r) != -1; + } FREE_TWO_STRINGS; break; case CMP_CONTAINSI: PREP_TWO_STRINGS; ret->datatype = 'N'; - ret->d.n = es_strCaseContains(estr_l, estr_r) != -1; + if(expr->r->nodetype == 'A') { + ret->d.n = evalStrArrayCmp(estr_l, (struct cnfarray*) expr->r, CMP_CONTAINSI); + bMustFree = 0; + } else { + ret->d.n = es_strCaseContains(estr_l, estr_r) != -1; + } FREE_TWO_STRINGS; break; case OR: @@ -1330,9 +1928,27 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) ret->datatype = 'S'; ret->d.estr = es_strdup(((struct cnfstringval*)expr)->estr); break; + case 'A': + /* if an array is used with "normal" operations, it just evaluates + * to its first element. + */ + ret->datatype = 'S'; + ret->d.estr = es_strdup(((struct cnfarray*)expr)->arr[0]); + break; case 'V': + evalVar((struct cnfvar*)expr, usrptr, ret); + break; + case '&': + /* TODO: think about optimization, should be possible ;) */ + PREP_TWO_STRINGS; + if(expr->r->nodetype == 'A') { + estr_r = ((struct cnfarray*)expr->r)->arr[0]; + bMustFree = 0; + } ret->datatype = 'S'; - ret->d.estr = cnfGetVar(((struct cnfvar*)expr)->name, usrptr); + ret->d.estr = es_strdup(estr_l); + es_addStr(&ret->d.estr, estr_r); + FREE_TWO_STRINGS; break; case '+': COMP_NUM_BINOP(+); @@ -1369,6 +1985,16 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) //--------------------------------------------------------- +void +cnfarrayContentDestruct(struct cnfarray *ar) +{ + unsigned short i; + for(i = 0 ; i < ar->nmemb ; ++i) { + es_deleteStr(ar->arr[i]); + } + free(ar->arr); +} + static inline void cnffuncDestruct(struct cnffunc *func) { @@ -1380,12 +2006,14 @@ cnffuncDestruct(struct cnffunc *func) /* some functions require special destruction */ switch(func->fID) { case CNFFUNC_RE_MATCH: - regexp.regfree(func->funcdata); - free(func->funcdata); - free(func->fname); + case CNFFUNC_RE_EXTRACT: + if(func->funcdata != NULL) + regexp.regfree(func->funcdata); break; default:break; } + free(func->funcdata); + free(func->fname); } /* Destruct an expression and all sub-expressions contained in it. @@ -1394,7 +2022,13 @@ void cnfexprDestruct(struct cnfexpr *expr) { - dbgprintf("cnfexprDestruct expr %p, type '%c'(%u)\n", expr, expr->nodetype, expr->nodetype); + if(expr == NULL) { + /* this is valid and can happen during optimizer run! */ + DBGPRINTF("cnfexprDestruct got NULL ptr - valid, so doing nothing\n"); + return; + } + + DBGPRINTF("cnfexprDestruct expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); switch(expr->nodetype) { case CMP_NE: case CMP_EQ: @@ -1408,6 +2042,7 @@ cnfexprDestruct(struct cnfexpr *expr) case CMP_CONTAINSI: case OR: case AND: + case '&': case '+': case '-': case '*': @@ -1431,6 +2066,9 @@ cnfexprDestruct(struct cnfexpr *expr) case 'F': cnffuncDestruct((struct cnffunc*)expr); break; + case 'A': + cnfarrayContentDestruct((struct cnfarray*)expr); + break; default:break; } free(expr); @@ -1459,13 +2097,39 @@ doIndent(int indent) for(i = 0 ; i < indent ; ++i) dbgprintf(" "); } + +static void +pmaskPrint(uchar *pmask, int indent) +{ + int i; + doIndent(indent); + dbgprintf("pmask: "); + for (i = 0; i <= LOG_NFACILITIES; i++) + if (pmask[i] == TABLE_NOPRI) + dbgprintf(" X "); + else + dbgprintf("%2X ", pmask[i]); + dbgprintf("\n"); +} + +static void +cnfarrayPrint(struct cnfarray *ar, int indent) +{ + int i; + doIndent(indent); dbgprintf("ARRAY:\n"); + for(i = 0 ; i < ar->nmemb ; ++i) { + doIndent(indent+1); + cstrPrint("string '", ar->arr[i]); + dbgprintf("'\n"); + } +} + void cnfexprPrint(struct cnfexpr *expr, int indent) { struct cnffunc *func; int i; - //dbgprintf("expr %p, indent %d, type '%c'\n", expr, indent, expr->nodetype); switch(expr->nodetype) { case CMP_EQ: cnfexprPrint(expr->l, indent+1); @@ -1549,6 +2213,9 @@ cnfexprPrint(struct cnfexpr *expr, int indent) cstrPrint("string '", ((struct cnfstringval*)expr)->estr); dbgprintf("'\n"); break; + case 'A': + cnfarrayPrint((struct cnfarray*)expr, indent); + break; case 'N': doIndent(indent); dbgprintf("%lld\n", ((struct cnfnumval*)expr)->val); @@ -1562,10 +2229,16 @@ cnfexprPrint(struct cnfexpr *expr, int indent) func = (struct cnffunc*) expr; cstrPrint("function '", func->fname); dbgprintf("' (id:%d, params:%hu)\n", func->fID, func->nParams); + if(func->fID == CNFFUNC_PRIFILT) { + struct funcData_prifilt *pD; + pD = (struct funcData_prifilt*) func->funcdata; + pmaskPrint(pD->pmask, indent+1); + } for(i = 0 ; i < func->nParams ; ++i) { cnfexprPrint(func->expr[i], indent+1); } break; + case '&': case '+': case '-': case '*': @@ -1579,11 +2252,109 @@ cnfexprPrint(struct cnfexpr *expr, int indent) cnfexprPrint(expr->r, indent+1); break; default: - dbgprintf("error: unknown nodetype %u\n", - (unsigned) expr->nodetype); + dbgprintf("error: unknown nodetype %u['%c']\n", + (unsigned) expr->nodetype, (char) expr->nodetype); + break; + } +} +/* print only the given stmt + * if "subtree" equals 1, the full statement subtree is printed, else + * really only the statement. + */ +void +cnfstmtPrintOnly(struct cnfstmt *stmt, int indent, sbool subtree) +{ + char *cstr; + switch(stmt->nodetype) { + case S_NOP: + doIndent(indent); dbgprintf("NOP\n"); + break; + case S_STOP: + doIndent(indent); dbgprintf("STOP\n"); + break; + case S_CALL: + cstr = es_str2cstr(stmt->d.s_call.name, NULL); + doIndent(indent); dbgprintf("CALL [%s]\n", cstr); + free(cstr); + break; + case S_ACT: + doIndent(indent); dbgprintf("ACTION %p [%s:%s]\n", stmt->d.act, + modGetName(stmt->d.act->pMod), stmt->printable); + break; + case S_IF: + doIndent(indent); dbgprintf("IF\n"); + cnfexprPrint(stmt->d.s_if.expr, indent+1); + if(subtree) { + doIndent(indent); dbgprintf("THEN\n"); + cnfstmtPrint(stmt->d.s_if.t_then, indent+1); + if(stmt->d.s_if.t_else != NULL) { + doIndent(indent); dbgprintf("ELSE\n"); + cnfstmtPrint(stmt->d.s_if.t_else, indent+1); + } + doIndent(indent); dbgprintf("END IF\n"); + } + break; + case S_SET: + doIndent(indent); dbgprintf("SET %s =\n", + stmt->d.s_set.varname); + cnfexprPrint(stmt->d.s_set.expr, indent+1); + doIndent(indent); dbgprintf("END SET\n"); + break; + case S_UNSET: + doIndent(indent); dbgprintf("UNSET %s\n", + stmt->d.s_unset.varname); + break; + case S_PRIFILT: + doIndent(indent); dbgprintf("PRIFILT '%s'\n", stmt->printable); + pmaskPrint(stmt->d.s_prifilt.pmask, indent); + if(subtree) { + cnfstmtPrint(stmt->d.s_prifilt.t_then, indent+1); + if(stmt->d.s_prifilt.t_else != NULL) { + doIndent(indent); dbgprintf("ELSE\n"); + cnfstmtPrint(stmt->d.s_prifilt.t_else, indent+1); + } + doIndent(indent); dbgprintf("END PRIFILT\n"); + } + break; + case S_PROPFILT: + doIndent(indent); dbgprintf("PROPFILT\n"); + doIndent(indent); dbgprintf("\tProperty.: '%s'\n", + propIDToName(stmt->d.s_propfilt.propID)); + if(stmt->d.s_propfilt.propName != NULL) { + cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + doIndent(indent); + dbgprintf("\tCEE-Prop.: '%s'\n", cstr); + free(cstr); + } + doIndent(indent); dbgprintf("\tOperation: "); + if(stmt->d.s_propfilt.isNegated) + dbgprintf("NOT "); + dbgprintf("'%s'\n", getFIOPName(stmt->d.s_propfilt.operation)); + if(stmt->d.s_propfilt.pCSCompValue != NULL) { + doIndent(indent); dbgprintf("\tValue....: '%s'\n", + rsCStrGetSzStrNoNULL(stmt->d.s_propfilt.pCSCompValue)); + } + if(subtree) { + doIndent(indent); dbgprintf("THEN\n"); + cnfstmtPrint(stmt->d.s_propfilt.t_then, indent+1); + doIndent(indent); dbgprintf("END PROPFILT\n"); + } + break; + default: + dbgprintf("error: unknown stmt type %u\n", + (unsigned) stmt->nodetype); break; } } +void +cnfstmtPrint(struct cnfstmt *root, int indent) +{ + struct cnfstmt *stmt; + //dbgprintf("stmt %p, indent %d, type '%c'\n", expr, indent, expr->nodetype); + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + cnfstmtPrintOnly(stmt, indent, 1); + } +} struct cnfnumval* cnfnumvalNew(long long val) @@ -1607,6 +2378,52 @@ cnfstringvalNew(es_str_t *estr) return strval; } +/* creates array AND adds first element to it */ +struct cnfarray* +cnfarrayNew(es_str_t *val) +{ + struct cnfarray *ar; + if((ar = malloc(sizeof(struct cnfarray))) != NULL) { + ar->nodetype = 'A'; + ar->nmemb = 1; + if((ar->arr = malloc(sizeof(es_str_t*))) == NULL) { + free(ar); + ar = NULL; + goto done; + } + ar->arr[0] = val; + } +done: return ar; +} + +struct cnfarray* +cnfarrayAdd(struct cnfarray *ar, es_str_t *val) +{ + es_str_t **newptr; + if((newptr = realloc(ar->arr, (ar->nmemb+1)*sizeof(es_str_t*))) == NULL) { + DBGPRINTF("cnfarrayAdd: realloc failed, item ignored, ar->arr=%p\n", ar->arr); + goto done; + } else { + ar->arr = newptr; + ar->arr[ar->nmemb] = val; + ar->nmemb++; + } +done: return ar; +} + +/* duplicate an array (deep copy) */ +struct cnfarray* +cnfarrayDup(struct cnfarray *old) +{ + int i; + struct cnfarray *ar; + ar = cnfarrayNew(es_strdup(old->arr[0])); + for(i = 1 ; i < old->nmemb ; ++i) { + cnfarrayAdd(ar, es_strdup(old->arr[i])); + } + return ar; +} + struct cnfvar* cnfvarNew(char *name) { @@ -1618,63 +2435,723 @@ cnfvarNew(char *name) return var; } -struct cnfrule * -cnfruleNew(enum cnfFiltType filttype, struct cnfactlst *actlst) +struct cnfstmt * +cnfstmtNew(unsigned s_type) { - struct cnfrule* cnfrule; - if((cnfrule = malloc(sizeof(struct cnfrule))) != NULL) { - cnfrule->nodetype = 'R'; - cnfrule->filttype = filttype; - cnfrule->actlst = cnfactlstReverse(actlst); + struct cnfstmt* cnfstmt; + if((cnfstmt = malloc(sizeof(struct cnfstmt))) != NULL) { + cnfstmt->nodetype = s_type; + cnfstmt->printable = NULL; + cnfstmt->next = NULL; } - return cnfrule; + return cnfstmt; } +void cnfstmtDestructLst(struct cnfstmt *root); + +/* delete a single stmt */ +static void +cnfstmtDestruct(struct cnfstmt *stmt) +{ + switch(stmt->nodetype) { + case S_NOP: + case S_STOP: + break; + case S_CALL: + es_deleteStr(stmt->d.s_call.name); + break; + case S_ACT: + actionDestruct(stmt->d.act); + break; + case S_IF: + cnfexprDestruct(stmt->d.s_if.expr); + if(stmt->d.s_if.t_then != NULL) { + cnfstmtDestructLst(stmt->d.s_if.t_then); + } + if(stmt->d.s_if.t_else != NULL) { + cnfstmtDestructLst(stmt->d.s_if.t_else); + } + break; + case S_SET: + free(stmt->d.s_set.varname); + cnfexprDestruct(stmt->d.s_set.expr); + break; + case S_UNSET: + free(stmt->d.s_set.varname); + break; + case S_PRIFILT: + cnfstmtDestructLst(stmt->d.s_prifilt.t_then); + cnfstmtDestructLst(stmt->d.s_prifilt.t_else); + break; + case S_PROPFILT: + if(stmt->d.s_propfilt.propName != NULL) + es_deleteStr(stmt->d.s_propfilt.propName); + if(stmt->d.s_propfilt.regex_cache != NULL) + rsCStrRegexDestruct(&stmt->d.s_propfilt.regex_cache); + if(stmt->d.s_propfilt.pCSCompValue != NULL) + cstrDestruct(&stmt->d.s_propfilt.pCSCompValue); + cnfstmtDestructLst(stmt->d.s_propfilt.t_then); + break; + default: + dbgprintf("error: unknown stmt type during destruct %u\n", + (unsigned) stmt->nodetype); + break; + } + free(stmt->printable); + free(stmt); +} + +/* delete a stmt and all others following it */ void -cnfrulePrint(struct cnfrule *rule) +cnfstmtDestructLst(struct cnfstmt *root) +{ + struct cnfstmt *stmt, *todel; + for(stmt = root ; stmt != NULL ; ) { + todel = stmt; + stmt = stmt->next; + cnfstmtDestruct(todel); + } +} + +struct cnfstmt * +cnfstmtNewSet(char *var, struct cnfexpr *expr) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_SET)) != NULL) { + cnfstmt->d.s_set.varname = (uchar*) var; + cnfstmt->d.s_set.expr = expr; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewCall(es_str_t *name) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_CALL)) != NULL) { + cnfstmt->d.s_call.name = name; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewUnset(char *var) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_UNSET)) != NULL) { + cnfstmt->d.s_unset.varname = (uchar*) var; + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewContinue(void) +{ + return cnfstmtNew(S_NOP); +} + +struct cnfstmt * +cnfstmtNewPRIFILT(char *prifilt, struct cnfstmt *t_then) +{ + struct cnfstmt* cnfstmt; + if((cnfstmt = cnfstmtNew(S_PRIFILT)) != NULL) { + cnfstmt->printable = (uchar*)prifilt; + cnfstmt->d.s_prifilt.t_then = t_then; + cnfstmt->d.s_prifilt.t_else = NULL; + DecodePRIFilter((uchar*)prifilt, cnfstmt->d.s_prifilt.pmask); + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewPROPFILT(char *propfilt, struct cnfstmt *t_then) +{ + struct cnfstmt* cnfstmt; + rsRetVal lRet; + if((cnfstmt = cnfstmtNew(S_PROPFILT)) != NULL) { + cnfstmt->printable = (uchar*)propfilt; + cnfstmt->d.s_propfilt.t_then = t_then; + cnfstmt->d.s_propfilt.propName = NULL; + cnfstmt->d.s_propfilt.regex_cache = NULL; + cnfstmt->d.s_propfilt.pCSCompValue = NULL; + lRet = DecodePropFilter((uchar*)propfilt, cnfstmt); + } + return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewAct(struct nvlst *lst) +{ + struct cnfstmt* cnfstmt; + char namebuf[256]; + rsRetVal localRet; + if((cnfstmt = cnfstmtNew(S_ACT)) == NULL) + goto done; + localRet = actionNewInst(lst, &cnfstmt->d.act); + if(localRet == RS_RET_OK_WARN) { + parser_errmsg("warnings occured in file '%s' around line %d", + cnfcurrfn, yylineno); + } else if(localRet != RS_RET_OK) { + parser_errmsg("errors occured in file '%s' around line %d", + cnfcurrfn, yylineno); + cnfstmt->nodetype = S_NOP; /* disable action! */ + goto done; + } + snprintf(namebuf, sizeof(namebuf)-1, "action(type=\"%s\" ...)", + modGetName(cnfstmt->d.act->pMod)); + namebuf[255] = '\0'; /* be on safe side */ + cnfstmt->printable = (uchar*)strdup(namebuf); + nvlstChkUnused(lst); + nvlstDestruct(lst); +done: return cnfstmt; +} + +struct cnfstmt * +cnfstmtNewLegaAct(char *actline) +{ + struct cnfstmt* cnfstmt; + rsRetVal localRet; + if((cnfstmt = cnfstmtNew(S_ACT)) == NULL) + goto done; + cnfstmt->printable = (uchar*)strdup((char*)actline); + localRet = cflineDoAction(loadConf, (uchar**)&actline, &cnfstmt->d.act); + if(localRet != RS_RET_OK && localRet != RS_RET_OK_WARN) { + parser_errmsg("%s occured in file '%s' around line %d", + (localRet == RS_RET_OK_WARN) ? "warnings" : "errors", + cnfcurrfn, yylineno); + if(localRet != RS_RET_OK_WARN) { + cnfstmt->nodetype = S_NOP; /* disable action! */ + goto done; + } + } +done: return cnfstmt; +} + + +/* returns 1 if the two expressions are constants, 0 otherwise + * if both are constants, the expression subtrees are destructed + * (this is an aid for constant folding optimizing) + */ +static int +getConstNumber(struct cnfexpr *expr, long long *l, long long *r) +{ + int ret = 0; + cnfexprOptimize(expr->l); + cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'N') { + if(expr->r->nodetype == 'N') { + ret = 1; + *l = ((struct cnfnumval*)expr->l)->val; + *r = ((struct cnfnumval*)expr->r)->val; + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } else if(expr->r->nodetype == 'S') { + ret = 1; + *l = ((struct cnfnumval*)expr->l)->val; + *r = es_str2num(((struct cnfstringval*)expr->r)->estr, NULL); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } + } else if(expr->l->nodetype == 'S') { + if(expr->r->nodetype == 'N') { + ret = 1; + *l = es_str2num(((struct cnfstringval*)expr->l)->estr, NULL); + *r = ((struct cnfnumval*)expr->r)->val; + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } else if(expr->r->nodetype == 'S') { + ret = 1; + *l = es_str2num(((struct cnfstringval*)expr->l)->estr, NULL); + *r = es_str2num(((struct cnfstringval*)expr->r)->estr, NULL); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + } + } + return ret; +} + + +/* constant folding for string concatenation */ +static inline void +constFoldConcat(struct cnfexpr *expr) +{ + es_str_t *estr; + cnfexprOptimize(expr->l); + cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'S') { + if(expr->r->nodetype == 'S') { + estr = ((struct cnfstringval*)expr->l)->estr; + ((struct cnfstringval*)expr->l)->estr = NULL; + es_addStr(&estr, ((struct cnfstringval*)expr->r)->estr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } else if(expr->r->nodetype == 'N') { + es_str_t *numstr; + estr = ((struct cnfstringval*)expr->l)->estr; + ((struct cnfstringval*)expr->l)->estr = NULL; + numstr = es_newStrFromNumber(((struct cnfnumval*)expr->r)->val); + es_addStr(&estr, numstr); + es_deleteStr(numstr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } + } else if(expr->l->nodetype == 'N') { + if(expr->r->nodetype == 'S') { + estr = es_newStrFromNumber(((struct cnfnumval*)expr->l)->val); + es_addStr(&estr, ((struct cnfstringval*)expr->r)->estr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } else if(expr->r->nodetype == 'S') { + es_str_t *numstr; + estr = es_newStrFromNumber(((struct cnfnumval*)expr->l)->val); + numstr = es_newStrFromNumber(((struct cnfnumval*)expr->r)->val); + es_addStr(&estr, numstr); + es_deleteStr(numstr); + cnfexprDestruct(expr->l); + cnfexprDestruct(expr->r); + expr->nodetype = 'S'; + ((struct cnfstringval*)expr)->estr = estr; + } + } +} + + +/* optimize comparisons with syslog severity/facility. This is a special + * handler as the numerical values also support GT, LT, etc ops. + */ +static inline struct cnfexpr* +cnfexprOptimize_CMP_severity_facility(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(!strcmp("$syslogseverity", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'N') { + int sev = (int) ((struct cnfnumval*)expr->r)->val; + if(sev >= 0 && sev <= 7) { + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); /* fac is irrelevant, set below... */ + prifiltSetSeverity(func->funcdata, sev, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } else { + parser_errmsg("invalid syslogseverity %d, expression will always " + "evaluate to FALSE", sev); + } + } + } else if(!strcmp("$syslogfacility", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'N') { + int fac = (int) ((struct cnfnumval*)expr->r)->val; + if(fac >= 0 && fac <= 24) { + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); /* fac is irrelevant, set below... */ + prifiltSetFacility(func->funcdata, fac, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } else { + parser_errmsg("invalid syslogfacility %d, expression will always " + "evaluate to FALSE", fac); + } + } + } + return expr; +} + +/* optimize a comparison with a variable as left-hand operand + * NOTE: Currently support CMP_EQ, CMP_NE only and code NEEDS + * TO BE CHANGED fgr other comparisons! + */ +static inline struct cnfexpr* +cnfexprOptimize_CMP_var(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(!strcmp("$syslogfacility-text", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'S') { + char *cstr = es_str2cstr(((struct cnfstringval*)expr->r)->estr, NULL); + int fac = decodeSyslogName((uchar*)cstr, syslogFacNames); + if(fac == -1) { + parser_errmsg("invalid facility '%s', expression will always " + "evaluate to FALSE", cstr); + } else { + /* we can acutally optimize! */ + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(fac); + if(expr->nodetype == CMP_NE) + prifiltInvert(func->funcdata); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } + free(cstr); + } + } else if(!strcmp("$syslogseverity-text", ((struct cnfvar*)expr->l)->name)) { + if(expr->r->nodetype == 'S') { + char *cstr = es_str2cstr(((struct cnfstringval*)expr->r)->estr, NULL); + int sev = decodeSyslogName((uchar*)cstr, syslogPriNames); + if(sev == -1) { + parser_errmsg("invalid syslogseverity '%s', expression will always " + "evaluate to FALSE", cstr); + } else { + /* we can acutally optimize! */ + DBGPRINTF("optimizer: change comparison OP to FUNC prifilt()\n"); + func = cnffuncNew_prifilt(0); + prifiltSetSeverity(func->funcdata, sev, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) func; + } + free(cstr); + } + } else { + expr = cnfexprOptimize_CMP_severity_facility(expr); + } + return expr; +} + +static inline struct cnfexpr* +cnfexprOptimize_NOT(struct cnfexpr *expr) +{ + struct cnffunc *func; + + if(expr->r->nodetype == 'F') { + func = (struct cnffunc *)expr->r; + if(func->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimize NOT prifilt() to inverted prifilt()\n"); + expr->r = NULL; + cnfexprDestruct(expr); + prifiltInvert(func->funcdata); + expr = (struct cnfexpr*) func; + } + } + return expr; +} + +static inline struct cnfexpr* +cnfexprOptimize_AND_OR(struct cnfexpr *expr) +{ + struct cnffunc *funcl, *funcr; + + if(expr->l->nodetype == 'F') { + if(expr->r->nodetype == 'F') { + funcl = (struct cnffunc *)expr->l; + funcr = (struct cnffunc *)expr->r; + if(funcl->fID == CNFFUNC_PRIFILT && funcr->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimize combine AND/OR prifilt()\n"); + expr->l = NULL; + prifiltCombine(funcl->funcdata, funcr->funcdata, expr->nodetype); + cnfexprDestruct(expr); + expr = (struct cnfexpr*) funcl; + } + } + } + return expr; +} + + +/* optimize array for EQ/NEQ comparisons. We sort the array in + * this case so that we can apply binary search later on. + */ +static inline void +cnfexprOptimize_CMPEQ_arr(struct cnfarray *arr) { - dbgprintf("------ start rule %p:\n", rule); - dbgprintf("%s: ", cnfFiltType2str(rule->filttype)); - switch(rule->filttype) { - case CNFFILT_NONE: + DBGPRINTF("optimizer: sorting array for CMP_EQ/NEQ comparison\n"); + qsort(arr->arr, arr->nmemb, sizeof(es_str_t*), qs_arrcmp); +} + + +/* (recursively) optimize an expression */ +struct cnfexpr* +cnfexprOptimize(struct cnfexpr *expr) +{ + long long ln, rn; + struct cnfexpr *exprswap; + + dbgprintf("optimize expr %p, type '%s'\n", expr, tokenToString(expr->nodetype)); + switch(expr->nodetype) { + case '&': + constFoldConcat(expr); + break; + case '+': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln + rn; + } break; - case CNFFILT_PRI: - case CNFFILT_PROP: - dbgprintf("%s\n", rule->filt.s); + case '-': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln - rn; + } break; - case CNFFILT_SCRIPT: - dbgprintf("\n"); - cnfexprPrint(rule->filt.expr, 0); + case '*': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln * rn; + } + break; + case '/': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln / rn; + } + break; + case '%': + if(getConstNumber(expr, &ln, &rn)) { + expr->nodetype = 'N'; + ((struct cnfnumval*)expr)->val = ln % rn; + } + break; + case CMP_NE: + case CMP_EQ: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + if(expr->l->nodetype == 'A') { + if(expr->r->nodetype == 'A') { + parser_errmsg("warning: '==' or '<>' " + "comparison of two constant string " + "arrays makes no sense"); + } else { /* swap for simpler execution step */ + exprswap = expr->l; + expr->l = expr->r; + expr->r = exprswap; + } + } + if(expr->l->nodetype == 'V') { + expr = cnfexprOptimize_CMP_var(expr); + } else if(expr->r->nodetype == 'A') { + cnfexprOptimize_CMPEQ_arr((struct cnfarray *)expr->r); + } + break; + case CMP_LE: + case CMP_GE: + case CMP_LT: + case CMP_GT: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_CMP_severity_facility(expr); + break; + case CMP_CONTAINS: + case CMP_CONTAINSI: + case CMP_STARTSWITH: + case CMP_STARTSWITHI: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + break; + case AND: + case OR: + expr->l = cnfexprOptimize(expr->l); + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_AND_OR(expr); + break; + case NOT: + expr->r = cnfexprOptimize(expr->r); + expr = cnfexprOptimize_NOT(expr); + break; + default:/* nodetypes we cannot optimize */ break; } - cnfactlstPrint(rule->actlst); - dbgprintf("------ end rule %p\n", rule); + return expr; } -/* note: the sysline itself was already freed during processing - * and as such MUST NOT be freed again! +/* removes NOPs from a statement list and returns the + * first non-NOP entry. */ -void -cnfcfsyslinelstDestruct(struct cnfcfsyslinelst *cfslst) +static inline struct cnfstmt * +removeNOPs(struct cnfstmt *root) { - struct cnfcfsyslinelst *toDel; - while(cfslst != NULL) { - toDel = cfslst; - cfslst = cfslst->next; - free(toDel); + struct cnfstmt *stmt, *toDel, *prevstmt = NULL; + struct cnfstmt *newRoot = NULL; + + if(root == NULL) goto done; + stmt = root; + while(stmt != NULL) { + if(stmt->nodetype == S_NOP) { + if(prevstmt != NULL) + /* end chain, is rebuild if more non-NOPs follow */ + prevstmt->next = NULL; + toDel = stmt; + stmt = stmt->next; + cnfstmtDestruct(toDel); + } else { + if(newRoot == NULL) + newRoot = stmt; + if(prevstmt != NULL) + prevstmt->next = stmt; + prevstmt = stmt; + stmt = stmt->next; + } } +done: return newRoot; } -void -cnfruleDestruct(struct cnfrule *rule) + +static inline void +cnfstmtOptimizeIf(struct cnfstmt *stmt) { - if( rule->filttype == CNFFILT_PRI - || rule->filttype == CNFFILT_PROP) - free(rule->filt.s); - cnfactlstDestruct(rule->actlst); - free(rule); + struct cnfstmt *t_then, *t_else; + struct cnfexpr *expr; + struct cnffunc *func; + struct funcData_prifilt *prifilt; + + expr = stmt->d.s_if.expr = cnfexprOptimize(stmt->d.s_if.expr); + stmt->d.s_if.t_then = removeNOPs(stmt->d.s_if.t_then); + stmt->d.s_if.t_else = removeNOPs(stmt->d.s_if.t_else); + cnfstmtOptimize(stmt->d.s_if.t_then); + cnfstmtOptimize(stmt->d.s_if.t_else); + + if(stmt->d.s_if.expr->nodetype == 'F') { + func = (struct cnffunc*)expr; + if(func->fID == CNFFUNC_PRIFILT) { + DBGPRINTF("optimizer: change IF to PRIFILT\n"); + t_then = stmt->d.s_if.t_then; + t_else = stmt->d.s_if.t_else; + stmt->nodetype = S_PRIFILT; + prifilt = (struct funcData_prifilt*) func->funcdata; + memcpy(stmt->d.s_prifilt.pmask, prifilt->pmask, + sizeof(prifilt->pmask)); + stmt->d.s_prifilt.t_then = t_then; + stmt->d.s_prifilt.t_else = t_else; + if(func->nParams == 0) + stmt->printable = (uchar*)strdup("[Optimizer Result]"); + else + stmt->printable = (uchar*) + es_str2cstr(((struct cnfstringval*)func->expr[0])->estr, NULL); + cnfexprDestruct(expr); + cnfstmtOptimizePRIFilt(stmt); + } + } +} + +static inline void +cnfstmtOptimizeAct(struct cnfstmt *stmt) +{ + action_t *pAct; + + pAct = stmt->d.act; + if(!strcmp((char*)modGetName(pAct->pMod), "builtin:omdiscard")) { + DBGPRINTF("optimizer: replacing omdiscard by STOP\n"); + actionDestruct(stmt->d.act); + stmt->nodetype = S_STOP; + } +} + +static void +cnfstmtOptimizePRIFilt(struct cnfstmt *stmt) +{ + int i; + int isAlways = 1; + struct cnfstmt *subroot, *last; + + stmt->d.s_prifilt.t_then = removeNOPs(stmt->d.s_prifilt.t_then); + cnfstmtOptimize(stmt->d.s_prifilt.t_then); + + for(i = 0; i <= LOG_NFACILITIES; i++) + if(stmt->d.s_prifilt.pmask[i] != 0xff) { + isAlways = 0; + break; + } + if(!isAlways) + goto done; + + DBGPRINTF("optimizer: removing always-true PRIFILT %p\n", stmt); + if(stmt->d.s_prifilt.t_else != NULL) { + parser_errmsg("error: always-true PRI filter has else part!\n"); + cnfstmtDestructLst(stmt->d.s_prifilt.t_else); + } + free(stmt->printable); + stmt->printable = NULL; + subroot = stmt->d.s_prifilt.t_then; + if(subroot == NULL) { + /* very strange, we set it to NOP, best we can do + * This case is NOT expected in practice + */ + stmt->nodetype = S_NOP; + goto done; + } + for(last = subroot ; last->next != NULL ; last = last->next) + /* find last node in subtree */; + last->next = stmt->next; + memcpy(stmt, subroot, sizeof(struct cnfstmt)); + free(subroot); + +done: return; } +/* we abuse "optimize" a bit. Actually, we obtain a ruleset pointer, as + * all rulesets are only known later in the process (now!). + */ +static void +cnfstmtOptimizeCall(struct cnfstmt *stmt) +{ + ruleset_t *pRuleset; + rsRetVal localRet; + uchar *rsName; + + rsName = (uchar*) es_str2cstr(stmt->d.s_call.name, NULL); + localRet = rulesetGetRuleset(loadConf, &pRuleset, rsName); + if(localRet != RS_RET_OK) { + /* in that case, we accept that a NOP will "survive" */ + parser_errmsg("ruleset '%s' cannot be found\n", rsName); + es_deleteStr(stmt->d.s_call.name); + stmt->nodetype = S_NOP; + goto done; + } + DBGPRINTF("CALL obtained ruleset ptr %p for ruleset %s\n", pRuleset, rsName); + stmt->d.s_call.stmt = pRuleset->root; +done: + free(rsName); + return; +} +/* (recursively) optimize a statement */ +void +cnfstmtOptimize(struct cnfstmt *root) +{ + struct cnfstmt *stmt; + if(root == NULL) goto done; + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + switch(stmt->nodetype) { + case S_IF: + cnfstmtOptimizeIf(stmt); + break; + case S_PRIFILT: + cnfstmtOptimizePRIFilt(stmt); + break; + case S_PROPFILT: + stmt->d.s_propfilt.t_then = removeNOPs(stmt->d.s_propfilt.t_then); + cnfstmtOptimize(stmt->d.s_propfilt.t_then); + break; + case S_SET: + stmt->d.s_set.expr = cnfexprOptimize(stmt->d.s_set.expr); + break; + case S_ACT: + cnfstmtOptimizeAct(stmt); + break; + case S_CALL: + cnfstmtOptimizeCall(stmt); + break; + case S_STOP: + if(stmt->next != NULL) + parser_errmsg("STOP is followed by unreachable statements!\n"); + break; + case S_UNSET: /* nothing to do */ + break; + case S_NOP: + DBGPRINTF("optimizer error: we see a NOP, how come?\n"); + break; + default: + dbgprintf("error: unknown stmt type %u during optimizer run\n", + (unsigned) stmt->nodetype); + break; + } + } +done: return; +} + + struct cnffparamlst * cnffparamlstNew(struct cnfexpr *expr, struct cnffparamlst *next) { @@ -1735,6 +3212,27 @@ funcName2ID(es_str_t *fname, unsigned short nParams) return CNFFUNC_INVALID; } return CNFFUNC_RE_MATCH; + } else if(!es_strbufcmp(fname, (unsigned char*)"re_extract", sizeof("re_extract") - 1)) { + if(nParams != 5) { + parser_errmsg("number of parameters for re_extract() must be five " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_RE_EXTRACT; + } else if(!es_strbufcmp(fname, (unsigned char*)"field", sizeof("field") - 1)) { + if(nParams != 3) { + parser_errmsg("number of parameters for field() must be three " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_FIELD; + } else if(!es_strbufcmp(fname, (unsigned char*)"prifilt", sizeof("prifilt") - 1)) { + if(nParams != 1) { + parser_errmsg("number of parameters for prifilt() must be one " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_PRIFILT; } else { return CNFFUNC_INVALID; } @@ -1751,7 +3249,7 @@ initFunc_re_match(struct cnffunc *func) func->funcdata = NULL; if(func->expr[1]->nodetype != 'S') { - parser_errmsg("param 2 of re_match() must be a constant string"); + parser_errmsg("param 2 of re_match/extract() must be a constant string"); FINALIZE; } @@ -1775,6 +3273,30 @@ finalize_it: RETiRet; } + +static inline rsRetVal +initFunc_prifilt(struct cnffunc *func) +{ + struct funcData_prifilt *pData; + uchar *cstr; + DEFiRet; + + func->funcdata = NULL; + if(func->expr[0]->nodetype != 'S') { + parser_errmsg("param 1 of prifilt() must be a constant string"); + FINALIZE; + } + + CHKmalloc(pData = calloc(1, sizeof(struct funcData_prifilt))); + func->funcdata = pData; + cstr = (uchar*)es_str2cstr(((struct cnfstringval*) func->expr[0])->estr, NULL); + CHKiRet(DecodePRIFilter(cstr, pData->pmask)); + free(cstr); +finalize_it: + RETiRet; +} + + struct cnffunc * cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) { @@ -1792,6 +3314,7 @@ cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) func->nodetype = 'F'; func->fname = fname; func->nParams = nParams; + func->funcdata = NULL; func->fID = funcName2ID(fname, nParams); /* shuffle params over to array (access speed!) */ param = paramlst; @@ -1804,25 +3327,54 @@ cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) /* some functions require special initialization */ switch(func->fID) { case CNFFUNC_RE_MATCH: + case CNFFUNC_RE_EXTRACT: /* need to compile the regexp in param 2, so this MUST be a constant */ initFunc_re_match(func); break; + case CNFFUNC_PRIFILT: + initFunc_prifilt(func); + break; default:break; } } return func; } + +/* A special function to create a prifilt() expression during optimization + * phase. + */ +struct cnffunc * +cnffuncNew_prifilt(int fac) +{ + struct cnffunc* func; + + if((func = malloc(sizeof(struct cnffunc))) != NULL) { + func->nodetype = 'F'; + func->fname = es_newStrFromCStr("prifilt", sizeof("prifilt")-1); + func->nParams = 0; + func->fID = CNFFUNC_PRIFILT; + func->funcdata = calloc(1, sizeof(struct funcData_prifilt)); + ((struct funcData_prifilt *)func->funcdata)->pmask[fac >> 3] = TABLE_ALLPRI; + } + return func; +} + + +/* returns 0 if everything is OK and config parsing shall continue, + * and 1 if things are so wrong that config parsing shall be aborted. + */ int cnfDoInclude(char *name) { char *cfgFile; char *finalName; - unsigned i; + int i; int result; glob_t cfgFiles; struct stat fileInfo; char nameBuf[MAXFNAME+1]; + char cwdBuf[MAXFNAME+1]; finalName = name; if(stat(name, &fileInfo) == 0) { @@ -1833,21 +3385,46 @@ cnfDoInclude(char *name) finalName = nameBuf; } } + /* Use GLOB_MARK to append a trailing slash for directories. */ + /* Use GLOB_NOMAGIC to detect wildcards that match nothing. */ +#ifdef HAVE_GLOB_NOMAGIC + /* Silently ignore wildcards that match nothing */ + result = glob(finalName, GLOB_MARK | GLOB_NOMAGIC, NULL, &cfgFiles); + if(result == GLOB_NOMATCH) { +#else result = glob(finalName, GLOB_MARK, NULL, &cfgFiles); - if(result == GLOB_NOSPACE || result == GLOB_ABORTED || cfgFiles.gl_pathc == 0) { + if(result == GLOB_NOMATCH && containsGlobWildcard(finalName)) { +#endif /* HAVE_GLOB_NOMAGIC */ + return 0; + } + + if(result == GLOB_NOSPACE || result == GLOB_ABORTED) { char errStr[1024]; rs_strerror_r(errno, errStr, sizeof(errStr)); - parser_errmsg("error accessing config file or directory '%s': %s", - finalName, errStr); + if(getcwd(cwdBuf, sizeof(cwdBuf)) == NULL) + strcpy(cwdBuf, "??getcwd() failed??"); + parser_errmsg("error accessing config file or directory '%s' [cwd:%s]: %s", + finalName, cwdBuf, errStr); return 1; } - for(i = 0; i < cfgFiles.gl_pathc; i++) { + /* note: bison "stacks" the files, so we need to submit them + * in reverse order to the *stack* in order to get the proper + * parsing order. Also see + * http://bugzilla.adiscon.com/show_bug.cgi?id=411 + */ + for(i = cfgFiles.gl_pathc - 1; i >= 0 ; i--) { cfgFile = cfgFiles.gl_pathv[i]; - - if(stat(cfgFile, &fileInfo) != 0) - continue; /* continue with the next file if we can't stat() the file */ + if(stat(cfgFile, &fileInfo) != 0) { + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + if(getcwd(cwdBuf, sizeof(cwdBuf)) == NULL) + strcpy(cwdBuf, "??getcwd() failed??"); + parser_errmsg("error accessing config file or directory '%s' " + "[cwd: %s]: %s", cfgFile, cwdBuf, errStr); + return 1; + } if(S_ISREG(fileInfo.st_mode)) { /* config file */ dbgprintf("requested to include config file '%s'\n", cfgFile); @@ -1867,16 +3444,28 @@ cnfDoInclude(char *name) void varDelete(struct var *v) { - if(v->datatype == 'S') + switch(v->datatype) { + case 'S': es_deleteStr(v->d.estr); + break; + case 'A': + cnfarrayContentDestruct(v->d.ar); + free(v->d.ar); + break; + default:break; + } } void cnfparamvalsDestruct(struct cnfparamvals *paramvals, struct cnfparamblk *blk) { int i; + if(paramvals == NULL) + return; for(i = 0 ; i < blk->nParams ; ++i) { - varDelete(¶mvals[i].val); + if(paramvals[i].bUsed) { + varDelete(¶mvals[i].val); + } } free(paramvals); } @@ -1909,6 +3498,15 @@ cstrPrint(char *text, es_str_t *estr) free(str); } +char * +rmLeadingSpace(char *s) +{ + char *p; + for(p = s ; *p && isspace(*p) ; ++p) + ; + return(p); +} + /* init must be called once before any parsing of the script files start */ rsRetVal initRainerscript(void) @@ -2056,3 +3654,52 @@ unescapeStr(uchar *s, int len) s[iDst] = '\0'; } } + +char * +tokenval2str(int tok) +{ + if(tok < 256) return ""; + switch(tok) { + case NAME: return "NAME"; + case FUNC: return "FUNC"; + case BEGINOBJ: return "BEGINOBJ"; + case ENDOBJ: return "ENDOBJ"; + case BEGIN_ACTION: return "BEGIN_ACTION"; + case BEGIN_PROPERTY: return "BEGIN_PROPERTY"; + case BEGIN_CONSTANT: return "BEGIN_CONSTANT"; + case BEGIN_TPL: return "BEGIN_TPL"; + case BEGIN_RULESET: return "BEGIN_RULESET"; + case STOP: return "STOP"; + case SET: return "SET"; + case UNSET: return "UNSET"; + case CONTINUE: return "CONTINUE"; + case CALL: return "CALL"; + case LEGACY_ACTION: return "LEGACY_ACTION"; + case LEGACY_RULESET: return "LEGACY_RULESET"; + case PRIFILT: return "PRIFILT"; + case PROPFILT: return "PROPFILT"; + case BSD_TAG_SELECTOR: return "BSD_TAG_SELECTOR"; + case BSD_HOST_SELECTOR: return "BSD_HOST_SELECTOR"; + case IF: return "IF"; + case THEN: return "THEN"; + case ELSE: return "ELSE"; + case OR: return "OR"; + case AND: return "AND"; + case NOT: return "NOT"; + case VAR: return "VAR"; + case STRING: return "STRING"; + case NUMBER: return "NUMBER"; + case CMP_EQ: return "CMP_EQ"; + case CMP_NE: return "CMP_NE"; + case CMP_LE: return "CMP_LE"; + case CMP_GE: return "CMP_GE"; + case CMP_LT: return "CMP_LT"; + case CMP_GT: return "CMP_GT"; + case CMP_CONTAINS: return "CMP_CONTAINS"; + case CMP_CONTAINSI: return "CMP_CONTAINSI"; + case CMP_STARTSWITH: return "CMP_STARTSWITH"; + case CMP_STARTSWITHI: return "CMP_STARTSWITHI"; + case UMINUS: return "UMINUS"; + default: return "UNKNOWN TOKEN"; + } +} diff --git a/grammar/rainerscript.h b/grammar/rainerscript.h index f5627597..d00cc4c3 100644 --- a/grammar/rainerscript.h +++ b/grammar/rainerscript.h @@ -3,7 +3,11 @@ #include <stdio.h> #include <libestr.h> #include <typedefs.h> +#include <sys/types.h> +#include <regex.h> + +#define LOG_NFACILITIES 24 /* current number of syslog facilities */ #define CNFFUNC_MAX_ARGS 32 /**< maximum number of arguments that any function can have (among * others, this is used to size data structures). @@ -13,6 +17,7 @@ extern int Debug; /* 1 if in debug mode, 0 otherwise -- to be enhanced */ enum cnfobjType { CNFOBJ_ACTION, + CNFOBJ_RULESET, CNFOBJ_GLOBAL, CNFOBJ_INPUT, CNFOBJ_MODULE, @@ -29,6 +34,9 @@ cnfobjType2str(enum cnfobjType ot) case CNFOBJ_ACTION: return "action"; break; + case CNFOBJ_RULESET: + return "ruleset"; + break; case CNFOBJ_GLOBAL: return "global"; break; @@ -63,16 +71,20 @@ enum cnfactType { CNFACT_V2, CNFACT_LEGACY }; struct var { union { es_str_t *estr; - struct cnfexpr *expr; + struct cnfarray *ar; long long n; + struct json_object *json; } d; - char datatype; /* 'N' number, 'S' string, 'E' expression */ + char datatype; /* 'N' number, 'S' string, 'J' JSON, 'A' array + * Note: 'A' is only supported during config phase + */ }; struct cnfobj { enum cnfobjType objType; struct nvlst *nvlst; struct objlst *subobjs; + struct cnfstmt *script; }; struct objlst { @@ -91,23 +103,6 @@ struct nvlst { */ }; -struct cnfcfsyslinelst { - struct cnfcfsyslinelst *next; - char *line; -}; - -struct cnfactlst { - struct cnfactlst *next; - struct cnfcfsyslinelst *syslines; - enum cnfactType actType; - union { - struct nvlst *lst; - char *legActLine; - } data; - char *cnfFile; - int lineno; -}; - /* the following structures support expressions, and may (very much later * be the sole foundation for the AST. * @@ -118,7 +113,19 @@ struct cnfactlst { * R - rule * S - string * V - var + * A - (string) array + * ... plus the S_* #define's below: */ +#define S_STOP 4000 +#define S_PRIFILT 4001 +#define S_PROPFILT 4002 +#define S_IF 4003 +#define S_ACT 4004 +#define S_NOP 4005 /* usually used to disable some statement */ +#define S_SET 4006 +#define S_UNSET 4007 +#define S_CALL 4008 + enum cnfFiltType { CNFFILT_NONE, CNFFILT_PRI, CNFFILT_PROP, CNFFILT_SCRIPT }; static inline char* cnfFiltType2str(enum cnfFiltType filttype) @@ -137,14 +144,44 @@ cnfFiltType2str(enum cnfFiltType filttype) } -struct cnfrule { +struct cnfstmt { unsigned nodetype; - enum cnfFiltType filttype; + struct cnfstmt *next; + uchar *printable; /* printable text for debugging */ union { - char *s; - struct cnfexpr *expr; - } filt; - struct cnfactlst *actlst; + struct { + struct cnfexpr *expr; + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_if; + struct { + uchar *varname; + struct cnfexpr *expr; + } s_set; + struct { + uchar *varname; + } s_unset; + struct { + es_str_t *name; + struct cnfstmt *stmt; + } s_call; + struct { + uchar pmask[LOG_NFACILITIES+1]; /* priority mask */ + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_prifilt; + struct { + fiop_t operation; + regex_t *regex_cache;/* cache for compiled REs, if used */ + struct cstr_s *pCSCompValue;/* value to "compare" against */ + sbool isNegated; + uintTiny propID;/* ID of the requested property */ + es_str_t *propName;/* name of property for CEE-based filters */ + struct cnfstmt *t_then; + struct cnfstmt *t_else; + } s_propfilt; + struct action_s *act; + } d; }; struct cnfexpr { @@ -168,6 +205,12 @@ struct cnfvar { char *name; }; +struct cnfarray { + unsigned nodetype; + int nmemb; + es_str_t **arr; +}; + struct cnffparamlst { unsigned nodetype; /* P */ struct cnffparamlst *next; @@ -182,7 +225,10 @@ enum cnffuncid { CNFFUNC_TOLOWER, CNFFUNC_CSTR, CNFFUNC_CNUM, - CNFFUNC_RE_MATCH + CNFFUNC_RE_MATCH, + CNFFUNC_RE_EXTRACT, + CNFFUNC_FIELD, + CNFFUNC_PRIFILT }; struct cnffunc { @@ -233,13 +279,19 @@ struct cnfparamvals { /* the values we obtained for param descr. */ unsigned char bUsed; }; +struct funcData_prifilt { + uchar pmask[LOG_NFACILITIES+1]; /* priority mask */ +}; + int cnfParseBuffer(char *buf, unsigned lenBuf); void readConfFile(FILE *fp, es_str_t **str); struct objlst* objlstNew(struct cnfobj *obj); void objlstDestruct(struct objlst *lst); void objlstPrint(struct objlst *lst); -struct nvlst* nvlstNew(es_str_t *name, es_str_t *value); +struct nvlst* nvlstNewArray(struct cnfarray *ar); +struct nvlst* nvlstNewStr(es_str_t *value); +struct nvlst* nvlstSetName(struct nvlst *lst, es_str_t *name); void nvlstDestruct(struct nvlst *lst); void nvlstPrint(struct nvlst *lst); void nvlstChkUnused(struct nvlst *lst); @@ -247,11 +299,6 @@ struct nvlst* nvlstFindName(struct nvlst *lst, es_str_t *name); struct cnfobj* cnfobjNew(enum cnfobjType objType, struct nvlst *lst); void cnfobjDestruct(struct cnfobj *o); void cnfobjPrint(struct cnfobj *o); -struct cnfactlst* cnfactlstNew(enum cnfactType actType, struct nvlst *lst, char *actLine); -void cnfactlstDestruct(struct cnfactlst *actlst); -void cnfactlstPrint(struct cnfactlst *actlst); -struct cnfactlst* cnfactlstAddSysline(struct cnfactlst* actlst, char *line); -struct cnfactlst* cnfactlstReverse(struct cnfactlst *actlst); struct cnfexpr* cnfexprNew(unsigned nodetype, struct cnfexpr *l, struct cnfexpr *r); void cnfexprPrint(struct cnfexpr *expr, int indent); void cnfexprEval(struct cnfexpr *expr, struct var *ret, void *pusr); @@ -259,9 +306,6 @@ int cnfexprEvalBool(struct cnfexpr *expr, void *usrptr); void cnfexprDestruct(struct cnfexpr *expr); struct cnfnumval* cnfnumvalNew(long long val); struct cnfstringval* cnfstringvalNew(es_str_t *estr); -struct cnfrule * cnfruleNew(enum cnfFiltType filttype, struct cnfactlst *actlst); -void cnfruleDestruct(struct cnfrule *rule); -void cnfrulePrint(struct cnfrule *rule); struct cnfvar* cnfvarNew(char *name); struct cnffunc * cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst); struct cnffparamlst * cnffparamlstNew(struct cnfexpr *expr, struct cnffparamlst *next); @@ -270,11 +314,33 @@ int cnfparamGetIdx(struct cnfparamblk *params, char *name); struct cnfparamvals* nvlstGetParams(struct nvlst *lst, struct cnfparamblk *params, struct cnfparamvals *vals); void cnfparamsPrint(struct cnfparamblk *params, struct cnfparamvals *vals); +int cnfparamvalsIsSet(struct cnfparamblk *params, struct cnfparamvals *vals); void varDelete(struct var *v); void cnfparamvalsDestruct(struct cnfparamvals *paramvals, struct cnfparamblk *blk); -void cnfcfsyslinelstDestruct(struct cnfcfsyslinelst *cfslst); +struct cnfstmt * cnfstmtNew(unsigned s_type); +void cnfstmtPrintOnly(struct cnfstmt *stmt, int indent, sbool subtree); +void cnfstmtPrint(struct cnfstmt *stmt, int indent); +struct cnfstmt* scriptAddStmt(struct cnfstmt *root, struct cnfstmt *s); +struct objlst* objlstAdd(struct objlst *root, struct cnfobj *o); +char *rmLeadingSpace(char *s); +struct cnfstmt * cnfstmtNewPRIFILT(char *prifilt, struct cnfstmt *t_then); +struct cnfstmt * cnfstmtNewPROPFILT(char *propfilt, struct cnfstmt *t_then); +struct cnfstmt * cnfstmtNewAct(struct nvlst *lst); +struct cnfstmt * cnfstmtNewLegaAct(char *actline); +struct cnfstmt * cnfstmtNewSet(char *var, struct cnfexpr *expr); +struct cnfstmt * cnfstmtNewUnset(char *var); +struct cnfstmt * cnfstmtNewCall(es_str_t *name); +struct cnfstmt * cnfstmtNewContinue(void); +void cnfstmtDestructLst(struct cnfstmt *root); +void cnfstmtOptimize(struct cnfstmt *root); +struct cnfarray* cnfarrayNew(es_str_t *val); +struct cnfarray* cnfarrayDup(struct cnfarray *old); +struct cnfarray* cnfarrayAdd(struct cnfarray *ar, es_str_t *val); +void cnfarrayContentDestruct(struct cnfarray *ar); +char* getFIOPName(unsigned iFIOP); rsRetVal initRainerscript(void); void unescapeStr(uchar *s, int len); +char * tokenval2str(int tok); /* debug helper */ void cstrPrint(char *text, es_str_t *estr); @@ -262,7 +262,7 @@ rsRetVal parsDelimCStr(rsParsObj *pThis, cstr_t **ppCStr, char cDelim, int bTrim pC = rsCStrGetBufBeg(pThis->pCStr) + pThis->iCurrPos; - while(pThis->iCurrPos < rsCStrLen(pThis->pCStr) && *pC != cDelim && *pC != ' ') { + while(pThis->iCurrPos < rsCStrLen(pThis->pCStr) && *pC != cDelim) { CHKiRet(cstrAppendChar(pCStr, bConvLower ? tolower(*pC) : *pC)); ++pThis->iCurrPos; ++pC; diff --git a/platform/README b/platform/README new file mode 100644 index 00000000..adbc3013 --- /dev/null +++ b/platform/README @@ -0,0 +1,4 @@ +This subdirectory contains platform-specific files. They are maintained +based on a best effort basis, and are not necessarily the same like the +specific platform ships them. Some files are changed in the way the +rsyslog projects would recommend them; some may even be outdated. diff --git a/freebsd/rsyslogd b/platform/freebsd/rsyslogd index 1cbcb6fe..1cbcb6fe 100755 --- a/freebsd/rsyslogd +++ b/platform/freebsd/rsyslogd diff --git a/platform/redhat/rsyslog.conf b/platform/redhat/rsyslog.conf new file mode 100644 index 00000000..340874d1 --- /dev/null +++ b/platform/redhat/rsyslog.conf @@ -0,0 +1,88 @@ +/* rsyslog configuration file (for Red Hat-based systems) + * note that most of this config file uses old-style format, + * because it is well-known AND quite suitable for simple cases + * like we have with the default config. For more advanced + * things, RainerScript configuration is suggested. + * + * For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html + * or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html + * If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html + */ + +#### MODULES #### + +module(load="imuxsock") # provides support for local system logging (e.g. via logger command) +module(load="imklog") # provides kernel logging support (previously done by rklogd) +#module(load"immark") # provides --MARK-- message capability + +# Provides UDP syslog reception +# for parameters see http://www.rsyslog.com/doc/imudp.html +#module(load="imudp") # needs to be done just once +#input(type="imudp" port="514") + +# Provides TCP syslog reception +# for parameters see http://www.rsyslog.com/doc/imtcp.html +#module(load="imtcp") # needs to be done just once +#input(type="imtcp" port="514") + + +#### GLOBAL DIRECTIVES #### + +# Use default timestamp format +$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + +# File syncing capability is disabled by default. This feature is usually not required, +# not useful and an extreme performance hit +#$ActionFileEnableSync on + +# Include all config files in /etc/rsyslog.d/ +$IncludeConfig /etc/rsyslog.d/*.conf + + +#### RULES #### + +# Log all kernel messages to the console. +# Logging much else clutters up the screen. +#kern.* /dev/console + +# Log anything (except mail) of level info or higher. +# Don't log private authentication messages! +*.info;mail.none;authpriv.none;cron.none /var/log/messages + +# The authpriv file has restricted access. +authpriv.* /var/log/secure + +# Log all the mail messages in one place. +mail.* /var/log/maillog + + +# Log cron stuff +cron.* /var/log/cron + +# Everybody gets emergency messages +*.emerg :omusrmsg:* + +# Save news errors of level crit and higher in a special file. +uucp,news.crit /var/log/spooler + +# Save boot messages also to boot.log +local7.* /var/log/boot.log + + +# ### begin forwarding rule ### +# The statement between the begin ... end define a SINGLE forwarding +# rule. They belong together, do NOT split them. If you create multiple +# forwarding rules, duplicate the whole block! +# Remote Logging (we use TCP for reliable delivery) +# +# An on-disk queue is created for this action. If the remote host is +# down, messages are spooled to disk and sent when it is up again. +#$WorkDirectory /var/lib/rsyslog # where to place spool files +#$ActionQueueFileName fwdRule1 # unique name prefix for spool files +#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) +#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown +#$ActionQueueType LinkedList # run asynchronously +#$ActionResumeRetryCount -1 # infinite retries if host is down +# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional +#*.* @@remote-host:514 +# ### end of the forwarding rule ### diff --git a/slackware/rc.rsyslogd b/platform/slackware/rc.rsyslogd index f5f8f458..f5f8f458 100755 --- a/slackware/rc.rsyslogd +++ b/platform/slackware/rc.rsyslogd diff --git a/plugins/cust1/cust1.c b/plugins/cust1/cust1.c deleted file mode 100644 index e69de29b..00000000 --- a/plugins/cust1/cust1.c +++ /dev/null diff --git a/plugins/im3195/im3195.c b/plugins/im3195/im3195.c index c75e0e34..b8a4a140 100644 --- a/plugins/im3195/im3195.c +++ b/plugins/im3195/im3195.c @@ -48,6 +48,7 @@ #include "cfsysline.h" #include "msg.h" #include "errmsg.h" +#include "unicode-helper.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -56,6 +57,7 @@ MODULE_CNFNAME("im3195") /* Module static data */ DEF_IMOD_STATIC_DATA DEFobjCurrIf(errmsg) +DEFobjCurrIf(prop) /* configuration settings */ @@ -71,6 +73,8 @@ static int listenPort = 601; */ static srAPIObj* pAPI; +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ + /* This method is called when a message has been fully received. * It passes the received message to the rsyslog main message @@ -90,7 +94,7 @@ void OnReceive(srAPIObj __attribute__((unused)) *pMyAPI, srSLMGObj* pSLMG) srSLMGGetRawMSG(pSLMG, &pszRawMsg); parseAndSubmitMessage(fromHost, fromHostIP, pszRawMsg, strlen((char*)pszRawMsg), - PARSE_HOSTNAME, eFLOWCTL_FULL_DELAY, (uchar*)"im3195", NULL, 0, NULL); + PARSE_HOSTNAME, eFLOWCTL_FULL_DELAY, pInputName, NULL, 0, NULL); } @@ -171,8 +175,12 @@ ENDafterRun BEGINmodExit CODESTARTmodExit srAPIExitLib(pAPI); /* terminate liblogging */ + /* global variable cleanup */ + if(pInputName != NULL) + prop.Destruct(&pInputName); /* release objects we used */ objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDmodExit @@ -193,9 +201,15 @@ CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"input3195listenport", 0, eCmdHdlrInt, NULL, &listenPort, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); + + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("im3195"), sizeof("im3195") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + ENDmodInit /* vim:set ai: */ diff --git a/plugins/imdiag/imdiag.c b/plugins/imdiag/imdiag.c index 09742537..5fdc6ef1 100644 --- a/plugins/imdiag/imdiag.c +++ b/plugins/imdiag/imdiag.c @@ -53,11 +53,11 @@ #include "srUtils.h" #include "msg.h" #include "datetime.h" +#include "ratelimit.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP -MODULE_CNFNAME("imdiag") /* static data */ DEF_IMOD_STATIC_DATA @@ -200,7 +200,7 @@ finalize_it: /* actually submit a message to the rsyslog core */ static rsRetVal -doInjectMsg(int iNum) +doInjectMsg(int iNum, ratelimit_t *ratelimiter) { uchar szMsg[1024]; msg_t *pMsg; @@ -220,7 +220,7 @@ doInjectMsg(int iNum) pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; MsgSetRcvFrom(pMsg, pRcvDummy); CHKiRet(MsgSetRcvFromIP(pMsg, pRcvIPDummy)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(ratelimitAddMsg(ratelimiter, NULL, pMsg)); finalize_it: RETiRet; @@ -238,6 +238,7 @@ injectMsg(uchar *pszCmd, tcps_sess_t *pSess) int iFrom; int nMsgs; int i; + ratelimit_t *ratelimit; DEFiRet; /* we do not check errors here! */ @@ -245,13 +246,15 @@ injectMsg(uchar *pszCmd, tcps_sess_t *pSess) iFrom = atoi((char*)wordBuf); getFirstWord(&pszCmd, wordBuf, sizeof(wordBuf)/sizeof(uchar), TO_LOWERCASE); nMsgs = atoi((char*)wordBuf); + ratelimitNew(&ratelimit, "imdiag", "injectmsg"); for(i = 0 ; i < nMsgs ; ++i) { - doInjectMsg(i + iFrom); + doInjectMsg(i + iFrom, ratelimit); } CHKiRet(sendResponse(pSess, "%d messages injected\n", nMsgs)); DBGPRINTF("imdiag: %d messages injected\n", nMsgs); + ratelimitDestruct(ratelimit); finalize_it: RETiRet; diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 453b6b05..2e80ffc8 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -48,6 +48,7 @@ #include "prop.h" #include "stringbuf.h" #include "ruleset.h" +#include "ratelimit.h" MODULE_TYPE_INPUT /* must be present for input modules, do not remove */ MODULE_TYPE_NOKEEP @@ -82,6 +83,7 @@ typedef struct fileInfo_s { strm_t *pStrm; /* its stream (NULL if not assigned) */ int readMode; /* which mode to use in ReadMulteLine call? */ ruleset_t *pRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + ratelimit_t *ratelimiter; multi_submit_t multiSub; } fileInfo_t; @@ -189,9 +191,7 @@ static rsRetVal enqLine(fileInfo_t *pInfo, cstr_t *cstrLine) pMsg->iFacility = LOG_FAC(pInfo->iFacility); pMsg->iSeverity = LOG_PRI(pInfo->iSeverity); MsgSetRuleset(pMsg, pInfo->pRuleset); - pInfo->multiSub.ppMsgs[pInfo->multiSub.nElem++] = pMsg; - if(pInfo->multiSub.nElem == pInfo->multiSub.maxElem) - CHKiRet(multiSubmitMsg(&pInfo->multiSub)); + ratelimitAddMsg(pInfo->ratelimiter, &pInfo->multiSub, pMsg); finalize_it: RETiRet; } @@ -235,6 +235,7 @@ openFile(fileInfo_t *pThis) /* read back in the object */ CHKiRet(obj.Deserialize(&pThis->pStrm, (uchar*) "strm", psSF, NULL, pThis)); + strm.CheckFileChange(pThis->pStrm); CHKiRet(strm.SeekCurrOffs(pThis->pStrm)); /* note: we do not delete the state file, so that the last position remains @@ -246,6 +247,8 @@ finalize_it: strm.Destruct(&psSF); if(iRet != RS_RET_OK) { + if(pThis->pStrm != NULL) + strm.Destruct(&pThis->pStrm); CHKiRet(strm.Construct(&pThis->pStrm)); CHKiRet(strm.SettOperationsMode(pThis->pStrm, STREAMMODE_READ)); CHKiRet(strm.SetsType(pThis->pStrm, STREAMTYPE_FILE_MONITOR)); @@ -304,18 +307,7 @@ static rsRetVal pollFile(fileInfo_t *pThis, int *pbHadFileData) } finalize_it: - if(pThis->multiSub.nElem > 0) { - /* submit everything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&pThis->multiSub)); - } - ; /*EMPTY STATEMENT - needed to keep compiler happy - see below! */ - /* Note: the problem above is that pthread:cleanup_pop() is a macro which - * evaluates to something like "} while(0);". So the code would become - * "finalize_it: }", that is a label without a statement. The C standard does - * not permit this. So we add an empty statement "finalize_it: ; }" and - * everybody is happy. Note that without the ;, an error is reported only - * on some platforms/compiler versions. -- rgerhards, 2008-08-15 - */ + multiSubmitFlush(&pThis->multiSub); pthread_cleanup_pop(0); if(pCStr != NULL) { @@ -423,6 +415,7 @@ addListner(instanceConf_t *inst) pThis->lenTag = ustrlen(pThis->pszTag); pThis->pszStateFile = (uchar*) strdup((char*) inst->pszStateFile); + CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(msg_t*))); pThis->multiSub.maxElem = inst->nMultiSub; pThis->multiSub.nElem = 0; @@ -456,8 +449,6 @@ CODESTARTnewInpInst pvals = nvlstGetParams(lst, &inppblk, NULL); if(pvals == NULL) { - errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, - "imfile: required parameter are missing\n"); ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); } @@ -603,7 +594,7 @@ CODESTARTactivateCnf for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { addListner(inst); } - /* if we could not set up any listners, there is no point in running... */ + /* if we could not set up any listeners, there is no point in running... */ if(iFilPtr == 0) { errmsg.LogError(0, NO_ERRCODE, "imfile: no file monitors could be started, " "input not activated.\n"); @@ -746,12 +737,20 @@ persistStrmState(fileInfo_t *pInfo) CHKiRet(strm.ConstructFinalize(psSF)); CHKiRet(strm.Serialize(pInfo->pStrm, psSF)); + CHKiRet(strm.Flush(psSF)); CHKiRet(strm.Destruct(&psSF)); finalize_it: if(psSF != NULL) strm.Destruct(&psSF); + + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "imfile: could not persist state " + "file %s - data may be repeated on next " + "startup. Is WorkDirectory set?", + pInfo->pszStateFile); + } RETiRet; } @@ -773,6 +772,8 @@ CODESTARTafterRun persistStrmState(&files[i]); strm.Destruct(&(files[i].pStrm)); } + ratelimitDestruct(files[i].ratelimiter); + free(files[i].multiSub.ppMsgs); free(files[i].pszFileName); free(files[i].pszTag); free(files[i].pszStateFile); diff --git a/plugins/imgssapi/imgssapi.c b/plugins/imgssapi/imgssapi.c index 4e3a70ab..15d994cc 100644 --- a/plugins/imgssapi/imgssapi.c +++ b/plugins/imgssapi/imgssapi.c @@ -72,6 +72,7 @@ MODULE_CNFNAME("imgssapi") /* some forward definitions - they may go away when we no longer include imtcp.c */ static rsRetVal addGSSListener(void __attribute__((unused)) *pVal, uchar *pNewVal); +static rsRetVal actGSSListener(uchar *port); static int TCPSessGSSInit(void); static void TCPSessGSSClose(tcps_sess_t* pSess); static rsRetVal TCPSessGSSRecv(tcps_sess_t *pSess, void *buf, size_t buf_len, ssize_t *); @@ -90,6 +91,7 @@ DEFobjCurrIf(glbl) static tcpsrv_t *pOurTcpsrv = NULL; /* our TCP server(listener) TODO: change for multiple instances */ static gss_cred_id_t gss_server_creds = GSS_C_NO_CREDENTIAL; +static uchar *srvPort; /* our usr structure for the tcpsrv object */ typedef struct gsssrv_s { @@ -317,6 +319,16 @@ static rsRetVal addGSSListener(void __attribute__((unused)) *pVal, uchar *pNewVal) { DEFiRet; + + srvPort = pNewVal; + + RETiRet; +} + +static rsRetVal +actGSSListener(uchar *port) +{ + DEFiRet; gsssrv_t *pGSrv; if(pOurTcpsrv == NULL) { @@ -340,7 +352,7 @@ addGSSListener(void __attribute__((unused)) *pVal, uchar *pNewVal) CHKiRet(tcpsrv.SetCBOnRegularClose(pOurTcpsrv, onRegularClose)); CHKiRet(tcpsrv.SetCBOnErrClose(pOurTcpsrv, onErrClose)); CHKiRet(tcpsrv.SetInputName(pOurTcpsrv, UCHAR_CONSTANT("imgssapi"))); - tcpsrv.configureTCPListen(pOurTcpsrv, pNewVal, 1); + tcpsrv.configureTCPListen(pOurTcpsrv, port, 1); CHKiRet(tcpsrv.ConstructFinalize(pOurTcpsrv)); } @@ -676,6 +688,11 @@ ENDfreeCnf */ BEGINrunInput CODESTARTrunInput + /* This will fail if the priviledges are dropped. Should be + * moved to the '*activateCnfPrePrivDrop' section eventually. + */ + actGSSListener(srvPort); + iRet = tcpsrv.Run(pOurTcpsrv); ENDrunInput @@ -683,7 +700,7 @@ ENDrunInput /* initialize and return if will run or not */ BEGINwillRun CODESTARTwillRun - if(pOurTcpsrv == NULL) + if(srvPort == NULL) ABORT_FINALIZE(RS_RET_NO_RUN); net.PrintAllowedSenders(2); /* TCP */ diff --git a/plugins/imjournal/Makefile.am b/plugins/imjournal/Makefile.am new file mode 100644 index 00000000..df088a3a --- /dev/null +++ b/plugins/imjournal/Makefile.am @@ -0,0 +1,7 @@ +pkglib_LTLIBRARIES = imjournal.la +imjournal_la_SOURCES = imjournal.c imjournal.h + +imjournal_la_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) $(LIBSYSTEMD_JOURNAL_CFLAGS) +#imjournal_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) $(LIBSYSTEMD_JOURNAL_CFLAGS) +imjournal_la_LDFLAGS = -module -avoid-version +imjournal_la_LIBADD = $(LIBSYSTEMD_JOURNAL_LIBS) diff --git a/plugins/imjournal/imjournal.c b/plugins/imjournal/imjournal.c new file mode 100755 index 00000000..36c7e046 --- /dev/null +++ b/plugins/imjournal/imjournal.c @@ -0,0 +1,717 @@ +/* The systemd journal import module + * + * To test under Linux: + * emmit log message into systemd journal + * + * Copyright (C) 2008-2013 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <assert.h> +#include <string.h> +#include <stdarg.h> +#include <ctype.h> +#include <stdlib.h> +#include <time.h> +#include <sys/poll.h> +#include <sys/socket.h> +#include <errno.h> +#include <systemd/sd-journal.h> + +#include "dirty.h" +#include "cfsysline.h" +#include "obj.h" +#include "msg.h" +#include "module-template.h" +#include "datetime.h" +#include "imjournal.h" +#include "net.h" +#include "glbl.h" +#include "prop.h" +#include "errmsg.h" +#include "srUtils.h" +#include "unicode-helper.h" +#include "ratelimit.h" + +MODULE_TYPE_INPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("imjournal") + +/* Module static data */ +DEF_IMOD_STATIC_DATA +DEFobjCurrIf(datetime) +DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) +DEFobjCurrIf(net) +DEFobjCurrIf(errmsg) + +static struct configSettings_s { + char *stateFile; + int iPersistStateInterval; + int ratelimitInterval; + int ratelimitBurst; + int bIgnorePrevious; +} cs; + +/* module-global parameters */ +static struct cnfparamdescr modpdescr[] = { + { "statefile", eCmdHdlrGetWord, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 }, + { "persiststateinterval", eCmdHdlrInt, 0 }, + { "ignorepreviousmessages", eCmdHdlrBinary, 0 } +}; +static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr + }; + +#define DFLT_persiststateinterval 10 + +static int bLegacyCnfModGlobalsPermitted = 1;/* are legacy module-global config parameters permitted? */ + +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ +static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ + +static ratelimit_t *ratelimiter = NULL; +static sd_journal *j; + +/* enqueue the the journal message into the message queue. + * The provided msg string is not freed - thus must be done + * by the caller. + */ +static rsRetVal +enqMsg(uchar *msg, uchar *pszTag, int iFacility, int iSeverity, struct timeval *tp, struct json_object *json) +{ + struct syslogTime st; + msg_t *pMsg; + DEFiRet; + + assert(msg != NULL); + assert(pszTag != NULL); + + if(tp == NULL) { + CHKiRet(msgConstruct(&pMsg)); + } else { + datetime.timeval2syslogTime(tp, &st); + CHKiRet(msgConstructWithTime(&pMsg, &st, tp->tv_sec)); + } + MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsgWOSize(pMsg, (char*)msg); + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, pLocalHostIP); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); + pMsg->iFacility = iFacility; + pMsg->iSeverity = iSeverity; + + if(json != NULL) { + msgAddJSON(pMsg, (uchar*)"!", json); + } + + CHKiRet(ratelimitAddMsg(ratelimiter, NULL, pMsg)); + +finalize_it: + RETiRet; +} + + +/* Read journal log while data are available, each read() reads one + * record of printk buffer. + */ +static rsRetVal +readjournal() { + DEFiRet; + + struct timeval tv; + uint64_t timestamp; + + struct json_object *json = NULL; + int r; + + /* Information from messages */ + char *message; + char *sys_pid; + char *sys_iden; + char *sys_iden_help; + + const void *get; + const void *pidget; + char *parse; + char *get2; + size_t length; + size_t pidlength; + + const void *equal_sign; + struct json_object *jval; + char *data; + char *name; + size_t l; + + long prefixlen = 0; + + int priority = 0; + int facility = 0; + + /* Get message text */ + if (sd_journal_get_data(j, "MESSAGE", &get, &length) < 0) { + logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar *)"log message from journal doesn't have MESSAGE", 0); + iRet = RS_RET_OK; + goto ret; + } + message = strndup(get+8, length-8); + if (message == NULL) { + iRet = RS_RET_OUT_OF_MEMORY; + goto ret; + } + + /* Get message priority */ + if (sd_journal_get_data(j, "PRIORITY", &get, &length) >= 0) { + get2 = strndup(get, length); + priority = ((char *)get2)[9] - '0'; + free (get2); + } + + /* Get syslog facility */ + if (sd_journal_get_data(j, "SYSLOG_FACILITY", &get, &length) >= 0) { + get2 = strndup(get, length); + char f = ((char *)get2)[16]; + if (f >= '0' && f <= '9') { + facility += f - '0'; + } + f = ((char *)get2)[17]; + if (f >= '0' && f <= '9') { + facility *= 10; + facility += (f - '0'); + } + free (get2); + } else { + /* message is missing facility -> internal systemd journal msg, drop */ + iRet = RS_RET_OK; + goto free_message; + } + + /* Get message identifier, client pid and add ':' */ + if (sd_journal_get_data(j, "SYSLOG_IDENTIFIER", &get, &length) >= 0) { + sys_iden = strndup(get+18, length-18); + } else { + sys_iden = strdup("journal"); + } + if (sys_iden == NULL) { + iRet = RS_RET_OUT_OF_MEMORY; + goto free_message; + } + + if (sd_journal_get_data(j, "SYSLOG_PID", &pidget, &pidlength) >= 0) { + sys_pid = strndup(pidget+11, pidlength-11); + if (sys_pid == NULL) { + iRet = RS_RET_OUT_OF_MEMORY; + free (sys_iden); + goto free_message; + } + } else { + sys_pid = NULL; + } + + if (sys_pid) { + r = asprintf(&sys_iden_help, "%s[%s]:", sys_iden, sys_pid); + } else { + r = asprintf(&sys_iden_help, "%s:", sys_iden); + } + + free (sys_iden); + free (sys_pid); + + if (-1 == r) { + iRet = RS_RET_OUT_OF_MEMORY; + goto finalize_it; + } + + json = json_object_new_object(); + + SD_JOURNAL_FOREACH_DATA(j, get, l) { + /* locate equal sign, this is always present */ + equal_sign = memchr(get, '=', l); + + /* ... but we know better than to trust the specs */ + if (equal_sign == NULL) { + errmsg.LogError(0, RS_RET_ERR, "SD_JOURNAL_FOREACH_DATA()" + "returned a malformed field (has no '='): '%s'", get); + continue; /* skip the entry */ + } + + /* get length of journal data prefix */ + prefixlen = ((char *)equal_sign - (char *)get); + + /* translate name fields to lumberjack names */ + parse = (char *)get; + + switch (*parse) + { + case '_': + ++parse; + if (*parse == 'P') { + if (!strncmp(parse+1, "ID=", 4)) { + name = strdup("pid"); + } else { + name = strndup(get, prefixlen); + } + } else if (*parse == 'G') { + if (!strncmp(parse+1, "ID=", 4)) { + name = strdup("gid"); + } else { + name = strndup(get, prefixlen); + } + } else if (*parse == 'U') { + if (!strncmp(parse+1, "ID=", 4)) { + name = strdup("uid"); + } else { + name = strndup(get, prefixlen); + } + } else if (*parse == 'E') { + if (!strncmp(parse+1, "XE=", 4)) { + name = strdup("exe"); + } else { + name = strndup(get, prefixlen); + } + } else if (*parse == 'C') { + parse++; + if (*parse == 'O') { + if (!strncmp(parse+1, "MM=", 4)) { + name = strdup("appname"); + } else { + name = strndup(get, prefixlen); + } + } else if (*parse == 'M') { + if (!strncmp(parse+1, "DLINE=", 7)) { + name = strdup("cmd"); + } else { + name = strndup(get, prefixlen); + } + } else { + name = strndup(get, prefixlen); + } + } else { + name = strndup(get, prefixlen); + } + break; + + default: + name = strndup(get, prefixlen); + break; + } + + if (name == NULL) { + iRet = RS_RET_OUT_OF_MEMORY; + goto ret; + } + + prefixlen++; /* remove '=' */ + + data = strndup(get + prefixlen, l - prefixlen); + if (data == NULL) { + iRet = RS_RET_OUT_OF_MEMORY; + free (name); + goto ret; + } + + /* and save them to json object */ + jval = json_object_new_string((char *)data); + json_object_object_add(json, name, jval); + free (data); + free (name); + } + + /* calculate timestamp */ + if (sd_journal_get_realtime_usec(j, ×tamp) >= 0) { + tv.tv_sec = timestamp / 1000000; + tv.tv_usec = timestamp % 1000000; + } + + /* submit message */ + enqMsg((uchar *)message, (uchar *) sys_iden_help, facility, priority, &tv, json); + +finalize_it: + free(sys_iden_help); +free_message: + free(message); +ret: + RETiRet; +} + + +/* This function gets journal cursor and saves it into state file + */ +static rsRetVal +persistJournalState () { + DEFiRet; + FILE *sf; /* state file */ + char *cursor; + int ret = 0; + + /* On success, sd_journal_get_cursor() returns 1 in systemd + 197 or older and 0 in systemd 198 or newer */ + if ((ret = sd_journal_get_cursor(j, &cursor)) >= 0) { + if ((sf = fopen(cs.stateFile, "wb")) != NULL) { + if (fprintf(sf, "%s", cursor) < 0) { + iRet = RS_RET_IO_ERROR; + } + fclose(sf); + free(cursor); + } else { + char errStr[256]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_FOPEN_FAILURE, "fopen() failed: " + "'%s', path: '%s'\n", errStr, cs.stateFile); + iRet = RS_RET_FOPEN_FAILURE; + } + } else { + char errStr[256]; + rs_strerror_r(-(ret), errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, "sd_journal_get_cursor() failed: '%s'\n", errStr); + iRet = RS_RET_ERR; + } + RETiRet; +} + + +/* Polls the journal for new messages. Similar to sd_journal_wait() + * except for the special handling of EINTR. + */ +static rsRetVal +pollJournal() +{ + DEFiRet; + struct pollfd pollfd; + int r; + + pollfd.fd = sd_journal_get_fd(j); + pollfd.events = sd_journal_get_events(j); + r = poll(&pollfd, 1, -1); + if (r == -1) { + if (errno == EINTR) { + /* EINTR is also received during termination + * so return now to check the term state. + */ + ABORT_FINALIZE(RS_RET_OK); + } else { + char errStr[256]; + + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, + "poll() failed: '%s'", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + } + + assert(r == 1); + + r = sd_journal_process(j); + if (r < 0) { + char errStr[256]; + + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, + "sd_journal_process() failed: '%s'", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + +finalize_it: + RETiRet; +} + + +/* This function loads a journal cursor from the state file. + */ +static rsRetVal +loadJournalState() +{ + DEFiRet; + + if (cs.stateFile[0] != '/') { + char *new_stateFile; + + if (-1 == asprintf(&new_stateFile, "%s/%s", (char *)glbl.GetWorkDir(), cs.stateFile)) { + errmsg.LogError(0, RS_RET_OUT_OF_MEMORY, "imjournal: asprintf failed\n"); + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } + free (cs.stateFile); + cs.stateFile = new_stateFile; + } + + /* if state file exists, set cursor to appropriate position */ + if (access(cs.stateFile, F_OK|R_OK) != -1) { + FILE *r_sf; + + if ((r_sf = fopen(cs.stateFile, "rb")) != NULL) { + char readCursor[128 + 1]; + + if (fscanf(r_sf, "%128s\n", readCursor) != EOF) { + if (sd_journal_seek_cursor(j, readCursor) != 0) { + errmsg.LogError(0, RS_RET_ERR, "imjournal: " + "couldn't seek to cursor `%s'\n", readCursor); + iRet = RS_RET_ERR; + goto finalize_it; + } + sd_journal_next(j); + } else { + errmsg.LogError(0, RS_RET_IO_ERROR, "imjournal: " + "fscanf on state file `%s' failed\n", cs.stateFile); + iRet = RS_RET_IO_ERROR; + goto finalize_it; + } + fclose(r_sf); + } else { + errmsg.LogError(0, RS_RET_FOPEN_FAILURE, "imjournal: " + "open on state file `%s' failed\n", cs.stateFile); + } + } else { + /* when IgnorePrevious, seek to the end of journal */ + if (cs.bIgnorePrevious) { + if (sd_journal_seek_tail(j) < 0) { + char errStr[256]; + + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, + "sd_journal_seek_tail() failed: '%s'", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + + if (sd_journal_previous(j) < 0) { + char errStr[256]; + + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, + "sd_journal_previous() failed: '%s'", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + } + } + +finalize_it: + RETiRet; +} + +BEGINrunInput +CODESTARTrunInput + CHKiRet(ratelimitNew(&ratelimiter, "imjournal", NULL)); + dbgprintf("imjournal: ratelimiting burst %d, interval %d\n", cs.ratelimitBurst, + cs.ratelimitInterval); + ratelimitSetLinuxLike(ratelimiter, cs.ratelimitInterval, cs.ratelimitBurst); + ratelimitSetNoTimeCache(ratelimiter); + + if (cs.stateFile) { + CHKiRet(loadJournalState()); + } + + /* this is an endless loop - it is terminated when the thread is + * signalled to do so. This, however, is handled by the framework. + */ + while (glbl.GetGlobalInputTermState() == 0) { + int count = 0, r; + + r = sd_journal_next(j); + if (r < 0) { + char errStr[256]; + + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, + "sd_journal_next() failed: '%s'", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + + if (r == 0) { + /* No new messages, wait for activity. */ + CHKiRet(pollJournal()); + continue; + } + + CHKiRet(readjournal()); + if (cs.stateFile) { /* can't persist without a state file */ + /* TODO: This could use some finer metric. */ + count++; + if (count == cs.iPersistStateInterval) { + count = 0; + persistJournalState(); + } + } + } + +finalize_it: +ENDrunInput + + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + bLegacyCnfModGlobalsPermitted = 1; + + cs.iPersistStateInterval = DFLT_persiststateinterval; + cs.stateFile = NULL; + cs.ratelimitBurst = 20000; + cs.ratelimitInterval = 600; +ENDbeginCnfLoad + + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + + +BEGINactivateCnf +CODESTARTactivateCnf +ENDactivateCnf + + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + +/* open journal */ +BEGINwillRun +CODESTARTwillRun + int ret; + ret = sd_journal_open(&j, SD_JOURNAL_LOCAL_ONLY); + if (ret < 0) { + iRet = RS_RET_IO_ERROR; + } +ENDwillRun + +/* close journal */ +BEGINafterRun +CODESTARTafterRun + if (cs.stateFile) { /* can't persist without a state file */ + persistJournalState(); + } + sd_journal_close(j); + ratelimitDestruct(ratelimiter); +ENDafterRun + + +BEGINmodExit +CODESTARTmodExit + if(pInputName != NULL) + prop.Destruct(&pInputName); + if(pLocalHostIP != NULL) + prop.Destruct(&pLocalHostIP); + + /* release objects we used */ + objRelease(glbl, CORE_COMPONENT); + objRelease(net, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINsetModCnf + struct cnfparamvals *pvals = NULL; + int i; +CODESTARTsetModCnf + pvals = nvlstGetParams(lst, &modpblk, NULL); + if (pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "error processing module " + "config parameters [module(...)]"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if (Debug) { + dbgprintf("module (global) param blk for imjournal:\n"); + cnfparamsPrint(&modpblk, pvals); + } + + for (i = 0 ; i < modpblk.nParams ; ++i) { + if (!pvals[i].bUsed) + continue; + if (!strcmp(modpblk.descr[i].name, "persiststateinterval")) { + cs.iPersistStateInterval = (int) pvals[i].val.d.n; + } else if (!strcmp(modpblk.descr[i].name, "statefile")) { + cs.stateFile = (char *)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(modpblk.descr[i].name, "ratelimit.burst")) { + cs.ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "ratelimit.interval")) { + cs.ratelimitInterval = (int) pvals[i].val.d.n; + } else if (!strcmp(modpblk.descr[i].name, "ignorepreviousmessages")) { + cs.bIgnorePrevious = (int) pvals[i].val.d.n; + } else { + dbgprintf("imjournal: program error, non-handled " + "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); + } + } + + +finalize_it: + if (pvals != NULL) + cnfparamvalsDestruct(pvals, &modpblk); +ENDsetModCnf + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATURENonCancelInputTermination) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_IMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES +CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(net, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.CreateStringProp(&pInputName, UCHAR_CONSTANT("imjournal"), sizeof("imjournal") - 1)); + CHKiRet(prop.CreateStringProp(&pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournalpersiststateinterval", 0, eCmdHdlrInt, + NULL, &cs.iPersistStateInterval, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournalratelimitinterval", 0, eCmdHdlrInt, + NULL, &cs.ratelimitInterval, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournalratelimitburst", 0, eCmdHdlrInt, + NULL, &cs.ratelimitBurst, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournalstatefile", 0, eCmdHdlrGetWord, + NULL, &cs.stateFile, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournalignorepreviousmessages", 0, eCmdHdlrBinary, + NULL, &cs.bIgnorePrevious, STD_LOADABLE_MODULE_ID)); + + +ENDmodInit +/* vim:set ai: + */ diff --git a/plugins/imjournal/imjournal.h b/plugins/imjournal/imjournal.h new file mode 100644 index 00000000..8d2c1a09 --- /dev/null +++ b/plugins/imjournal/imjournal.h @@ -0,0 +1,36 @@ +/* imjournal.h + * These are the definitions for the journal messages import module + * + * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef IMJOURNAL_H_INCLUDED +#define IMJOURNAL_H_INCLUDED 1 + +#include "rsyslog.h" +#include "dirty.h" +#include <systemd/sd-journal.h> + +struct modConfData_s { +}; + +rsRetVal Syslog(int priority, uchar *msg, struct timeval *tp, struct json_object *json); + +#endif /* #ifndef IMJOURNAL_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/plugins/imklog/bsd.c b/plugins/imklog/bsd.c index cddc6737..9c2eebb2 100644 --- a/plugins/imklog/bsd.c +++ b/plugins/imklog/bsd.c @@ -58,9 +58,6 @@ static int fklog = -1; /* kernel log fd */ #ifdef OS_LINUX /* submit a message to imklog Syslog() API. In this function, we check if * a kernel timestamp is present and, if so, extract and strip it. - * Note: this is an extra processing step. We should revisit the whole - * idea in v6 and remove all that old stuff that we do not longer need - * (like symbol resolution). <-- TODO * Note that this is heavily Linux specific and thus is not compiled or * used for BSD. * Special thanks to Lennart Poettering for suggesting on how to convert @@ -175,7 +172,7 @@ klogWillRun(modConfData_t *pModConf) fklog = open((char*)GetPath(pModConf), O_RDONLY, 0); if (fklog < 0) { - imklogLogIntMsg(RS_RET_ERR_OPEN_KLOG, "imklog: cannot open kernel log(%s): %s.", + imklogLogIntMsg(LOG_ERR, "imklog: cannot open kernel log(%s): %s.", GetPath(pModConf), rs_strerror_r(errno, errmsg, sizeof(errmsg))); ABORT_FINALIZE(RS_RET_ERR_OPEN_KLOG); } diff --git a/plugins/imklog/imklog.c b/plugins/imklog/imklog.c index a24fc63b..810ac264 100644 --- a/plugins/imklog/imklog.c +++ b/plugins/imklog/imklog.c @@ -95,7 +95,7 @@ static struct cnfparamdescr modpdescr[] = { { "permitnonkernelfacility", eCmdHdlrBinary, 0 }, { "consoleloglevel", eCmdHdlrInt, 0 }, { "parsekerneltimestamp", eCmdHdlrBinary, 0 }, - { "keepkerneltimestamp", eCmdHdlrBinary, 0 }, + { "keepkerneltimestamp", eCmdHdlrBinary, 0 }, { "internalmsgfacility", eCmdHdlrFacility, 0 } }; static struct cnfparamblk modpblk = @@ -105,7 +105,7 @@ static struct cnfparamblk modpblk = }; static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ -static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ +static prop_t *pLocalHostIP = NULL; static inline void initConfigSettings(void) @@ -150,7 +150,8 @@ enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity, struct timeval * MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); pMsg->iFacility = iFacility; pMsg->iSeverity = iSeverity; - CHKiRet(submitMsg(pMsg)); + /* note: we do NOT use rate-limiting, as the kernel itself does rate-limiting */ + CHKiRet(submitMsg2(pMsg)); finalize_it: RETiRet; @@ -294,6 +295,7 @@ CODESTARTbeginCnfLoad pModConf->bParseKernelStamp = 0; pModConf->bKeepKernelStamp = 0; pModConf->console_log_level = -1; + pModConf->bKeepKernelStamp = 0; pModConf->iFacilIntMsg = klogFacilIntMsg(); loadModConf->configSetViaV2Method = 0; bLegacyCnfModGlobalsPermitted = 1; diff --git a/plugins/imklog/imklog.h b/plugins/imklog/imklog.h index fa517ccc..1cf9b05a 100644 --- a/plugins/imklog/imklog.h +++ b/plugins/imklog/imklog.h @@ -35,9 +35,9 @@ struct modConfData_s { int iFacilIntMsg; uchar *pszPath; int console_log_level; - sbool bPermitNonKernel; sbool bParseKernelStamp; sbool bKeepKernelStamp; + sbool bPermitNonKernel; sbool configSetViaV2Method; }; diff --git a/plugins/imkmsg/Makefile.am b/plugins/imkmsg/Makefile.am new file mode 100644 index 00000000..87c177d2 --- /dev/null +++ b/plugins/imkmsg/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = imkmsg.la +imkmsg_la_SOURCES = imkmsg.c imkmsg.h + +imkmsg_la_SOURCES += kmsg.c + +imkmsg_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +imkmsg_la_LDFLAGS = -module -avoid-version +imkmsg_la_LIBADD = diff --git a/plugins/imkmsg/imkmsg.c b/plugins/imkmsg/imkmsg.c new file mode 100644 index 00000000..2a97f82d --- /dev/null +++ b/plugins/imkmsg/imkmsg.c @@ -0,0 +1,295 @@ +/* The kernel log module. + * + * This is rsyslog Linux only module for reading structured kernel logs. + * Module is based on imklog module so it retains its structure + * and other part is currently in kmsg.c file instead of this (imkmsg.c) + * For more information see that file. + * + * To test under Linux: + * echo test1 > /dev/kmsg + * + * Copyright (C) 2008-2012 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <assert.h> +#include <string.h> +#include <stdarg.h> +#include <ctype.h> +#include <stdlib.h> +#include <sys/socket.h> + +#include "dirty.h" +#include "cfsysline.h" +#include "obj.h" +#include "msg.h" +#include "module-template.h" +#include "datetime.h" +#include "imkmsg.h" +#include "net.h" +#include "glbl.h" +#include "prop.h" +#include "errmsg.h" +#include "unicode-helper.h" + +MODULE_TYPE_INPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("imkmsg") + +/* Module static data */ +DEF_IMOD_STATIC_DATA +DEFobjCurrIf(datetime) +DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) +DEFobjCurrIf(net) +DEFobjCurrIf(errmsg) + +/* config settings */ +typedef struct configSettings_s { + int iFacilIntMsg; /* the facility to use for internal messages (set by driver) */ +} configSettings_t; +static configSettings_t cs; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ +static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ + +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ +static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ + +static inline void +initConfigSettings(void) +{ + cs.iFacilIntMsg = klogFacilIntMsg(); +} + + +/* enqueue the the kernel message into the message queue. + * The provided msg string is not freed - thus must be done + * by the caller. + * rgerhards, 2008-04-12 + */ +static rsRetVal +enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity, struct timeval *tp, struct json_object *json) +{ + struct syslogTime st; + msg_t *pMsg; + DEFiRet; + + assert(msg != NULL); + assert(pszTag != NULL); + + if(tp == NULL) { + CHKiRet(msgConstruct(&pMsg)); + } else { + datetime.timeval2syslogTime(tp, &st); + CHKiRet(msgConstructWithTime(&pMsg, &st, tp->tv_sec)); + } + MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsgWOSize(pMsg, (char*)msg); + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, pLocalHostIP); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); + pMsg->iFacility = iFacility; + pMsg->iSeverity = iSeverity; + pMsg->json = json; + CHKiRet(submitMsg(pMsg)); + +finalize_it: + RETiRet; +} + + +/* log an imkmsg-internal message + * rgerhards, 2008-04-14 + */ +rsRetVal imkmsgLogIntMsg(int priority, char *fmt, ...) +{ + DEFiRet; + va_list ap; + uchar msgBuf[2048]; /* we use the same size as sysklogd to remain compatible */ + + va_start(ap, fmt); + vsnprintf((char*)msgBuf, sizeof(msgBuf) / sizeof(char), fmt, ap); + va_end(ap); + + logmsgInternal(NO_ERRCODE ,priority, msgBuf, 0); + + RETiRet; +} + + +/* log a message from /dev/kmsg + */ +rsRetVal Syslog(int priority, uchar *pMsg, struct timeval *tp, struct json_object *json) +{ + DEFiRet; + iRet = enqMsg((uchar*)pMsg, (uchar*) "kernel:", LOG_FAC(priority), LOG_PRI(priority), tp, json); + RETiRet; +} + + +/* helper for some klog drivers which need to know the MaxLine global setting. They can + * not obtain it themselfs, because they are no modules and can not query the object hander. + * It would probably be a good idea to extend the interface to support it, but so far + * we create a (sufficiently valid) work-around. -- rgerhards, 2008-11-24 + */ +int klog_getMaxLine(void) +{ + return glbl.GetMaxLine(); +} + + +BEGINrunInput +CODESTARTrunInput + /* this is an endless loop - it is terminated when the thread is + * signalled to do so. This, however, is handled by the framework, + * right into the sleep below. + */ + while(!pThrd->bShallStop) { + /* klogLogKMsg() waits for the next kernel message, obtains it + * and then submits it to the rsyslog main queue. + * rgerhards, 2008-04-09 + */ + CHKiRet(klogLogKMsg(runModConf)); + } +finalize_it: +ENDrunInput + + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; + /* init our settings */ + pModConf->iFacilIntMsg = klogFacilIntMsg(); + loadModConf->configSetViaV2Method = 0; + bLegacyCnfModGlobalsPermitted = 1; + /* init legacy config vars */ + initConfigSettings(); +ENDbeginCnfLoad + + +BEGINendCnfLoad +CODESTARTendCnfLoad + if(!loadModConf->configSetViaV2Method) { + /* persist module-specific settings from legacy config system */ + loadModConf->iFacilIntMsg = cs.iFacilIntMsg; + } + + loadModConf = NULL; /* done loading */ +ENDendCnfLoad + + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + + +BEGINactivateCnfPrePrivDrop +CODESTARTactivateCnfPrePrivDrop + runModConf = pModConf; + iRet = klogWillRun(runModConf); +ENDactivateCnfPrePrivDrop + + +BEGINactivateCnf +CODESTARTactivateCnf +ENDactivateCnf + + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINwillRun +CODESTARTwillRun +ENDwillRun + + +BEGINafterRun +CODESTARTafterRun + iRet = klogAfterRun(runModConf); +ENDafterRun + + +BEGINmodExit +CODESTARTmodExit + if(pInputName != NULL) + prop.Destruct(&pInputName); + if(pLocalHostIP != NULL) + prop.Destruct(&pLocalHostIP); + + /* release objects we used */ + objRelease(glbl, CORE_COMPONENT); + objRelease(net, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_IMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES +ENDqueryEtryPt + +static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + cs.iFacilIntMsg = klogFacilIntMsg(); + return RS_RET_OK; +} + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(net, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.CreateStringProp(&pInputName, UCHAR_CONSTANT("imkmsg"), sizeof("imkmsg") - 1)); + CHKiRet(prop.CreateStringProp(&pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + + /* init legacy config settings */ + initConfigSettings(); + + CHKiRet(omsdRegCFSLineHdlr((uchar *)"debugprintkernelsymbols", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbollookup", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbolstwice", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogusesyscallinterface", 0, eCmdHdlrGoneAway, + NULL, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, + resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); +ENDmodInit +/* vim:set ai: + */ diff --git a/plugins/imkmsg/imkmsg.h b/plugins/imkmsg/imkmsg.h new file mode 100644 index 00000000..220a1634 --- /dev/null +++ b/plugins/imkmsg/imkmsg.h @@ -0,0 +1,64 @@ +/* imkmsg.h + * These are the definitions for the kmsg message generation module. + * + * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef IMKLOG_H_INCLUDED +#define IMKLOG_H_INCLUDED 1 + +#include "rsyslog.h" +#include "dirty.h" + +/* we need to have the modConf type present in all submodules */ +struct modConfData_s { + rsconf_t *pConf; + int iFacilIntMsg; + uchar *pszPath; + int console_log_level; + sbool bPermitNonKernel; + sbool configSetViaV2Method; +}; + +/* interface to "drivers" + * the platform specific drivers must implement these entry points. Only one + * driver may be active at any given time, thus we simply rely on the linker + * to resolve the addresses. + * rgerhards, 2008-04-09 + */ +rsRetVal klogLogKMsg(modConfData_t *pModConf); +rsRetVal klogWillRun(modConfData_t *pModConf); +rsRetVal klogAfterRun(modConfData_t *pModConf); +int klogFacilIntMsg(); + +/* the functions below may be called by the drivers */ +rsRetVal imkmsgLogIntMsg(int priority, char *fmt, ...) __attribute__((format(printf,2, 3))); +rsRetVal Syslog(int priority, uchar *msg, struct timeval *tp, struct json_object *json); + +/* prototypes */ +extern int klog_getMaxLine(void); /* work-around for klog drivers to get configured max line size */ +extern int InitKsyms(modConfData_t*); +extern void DeinitKsyms(void); +extern int InitMsyms(void); +extern void DeinitMsyms(void); +extern char * ExpandKadds(char *, char *); +extern void SetParanoiaLevel(int); + +#endif /* #ifndef IMKLOG_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/plugins/imkmsg/kmsg.c b/plugins/imkmsg/kmsg.c new file mode 100644 index 00000000..822d3dbd --- /dev/null +++ b/plugins/imkmsg/kmsg.c @@ -0,0 +1,249 @@ +/* imkmsg driver for Linux /dev/kmsg structured logging + * + * This contains Linux-specific functionality to read /dev/kmsg + * For a general overview, see head comment in imkmsg.c. + * This is heavily based on imklog bsd.c file. + * + * Copyright 2008-2012 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#include <stdlib.h> +#include <time.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <string.h> +#include <ctype.h> +#include <sys/klog.h> +#include <sys/sysinfo.h> +#include <json/json.h> + +#include "rsyslog.h" +#include "srUtils.h" +#include "debug.h" +#include "imkmsg.h" + +/* globals */ +static int fklog = -1; /* kernel log fd */ + +#ifndef _PATH_KLOG +# define _PATH_KLOG "/dev/kmsg" +#endif + +/* submit a message to imkmsg Syslog() API. In this function, we parse + * necessary information from kernel log line, and make json string + * from the rest. + */ +static void +submitSyslog(uchar *buf) +{ + long offs = 0; + struct timeval tv; + struct sysinfo info; + unsigned long int timestamp = 0; + char name[1024]; + char value[1024]; + char msg[1024]; + int priority = 0; + long int sequnum = 0; + struct json_object *json = NULL, *jval; + + /* create new json object */ + json = json_object_new_object(); + + /* get priority */ + for (; isdigit(*buf); buf++) { + priority += (priority * 10) + (*buf - '0'); + } + buf++; + + /* get messages sequence number and add it to json */ + for (; isdigit(*buf); buf++) { + sequnum = (sequnum * 10) + (*buf - '0'); + } + buf++; /* skip , */ + jval = json_object_new_int(sequnum); + json_object_object_add(json, "sequnum", jval); + + /* get timestamp */ + for (; isdigit(*buf); buf++) { + timestamp = (timestamp * 10) + (*buf - '0'); + } + + while (*buf != ';') { + buf++; /* skip everything till the first ; */ + } + buf++; /* skip ; */ + + /* get message */ + offs = 0; + for (; *buf != '\n' && *buf != '\0'; buf++, offs++) { + msg[offs] = *buf; + } + msg[offs] = '\0'; + jval = json_object_new_string((char*)msg); + json_object_object_add(json, "msg", jval); + + if (*buf != '\0') /* message has appended properties, skip \n */ + buf++; + + while (*buf) { + /* get name of the property */ + buf++; /* skip ' ' */ + offs = 0; + for (; *buf != '=' && *buf != ' '; buf++, offs++) { + name[offs] = *buf; + } + name[offs] = '\0'; + buf++; /* skip = or ' ' */; + + offs = 0; + for (; *buf != '\n' && *buf != '\0'; buf++, offs++) { + value[offs] = *buf; + } + value[offs] = '\0'; + if (*buf != '\0') { + buf++; /* another property, skip \n */ + } + + jval = json_object_new_string((char*)value); + json_object_object_add(json, name, jval); + } + + /* calculate timestamp */ + sysinfo(&info); + gettimeofday(&tv, NULL); + + /* get boot time */ + tv.tv_sec -= info.uptime; + + tv.tv_sec += timestamp / 1000000; + tv.tv_usec += timestamp % 1000000; + + while (tv.tv_usec < 0) { + tv.tv_sec--; + tv.tv_usec += 1000000; + } + + while (tv.tv_usec >= 1000000) { + tv.tv_sec++; + tv.tv_usec -= 1000000; + } + + Syslog(priority, (uchar *)msg, &tv, json); +} + + +/* open the kernel log - will be called inside the willRun() imkmsg entry point + */ +rsRetVal +klogWillRun(modConfData_t *pModConf) +{ + char errmsg[2048]; + DEFiRet; + + fklog = open(_PATH_KLOG, O_RDONLY, 0); + if (fklog < 0) { + imkmsgLogIntMsg(RS_RET_ERR_OPEN_KLOG, "imkmsg: cannot open kernel log(%s): %s.", + _PATH_KLOG, rs_strerror_r(errno, errmsg, sizeof(errmsg))); + ABORT_FINALIZE(RS_RET_ERR_OPEN_KLOG); + } + +finalize_it: + RETiRet; +} + +/* Read kernel log while data are available, each read() reads one + * record of printk buffer. + */ +static void +readkmsg(void) +{ + int i; + uchar pRcv[8192+1]; + char errmsg[2048]; + + for (;;) { + dbgprintf("imkmsg waiting for kernel log line\n"); + + /* every read() from the opened device node receives one record of the printk buffer */ + i = read(fklog, pRcv, 8192); + + if (i > 0) { + /* successful read of message of nonzero length */ + pRcv[i] = '\0'; + } else if (i == -EPIPE) { + imkmsgLogIntMsg(LOG_WARNING, + "imkmsg: some messages in circular buffer got overwritten"); + continue; + } else { + /* something went wrong - error or zero length message */ + if (i < 0 && errno != EINTR && errno != EAGAIN) { + /* error occured */ + imkmsgLogIntMsg(LOG_ERR, + "imkmsg: error reading kernel log - shutting down: %s", + rs_strerror_r(errno, errmsg, sizeof(errmsg))); + fklog = -1; + } + break; + } + + submitSyslog(pRcv); + } +} + + +/* to be called in the module's AfterRun entry point + * rgerhards, 2008-04-09 + */ +rsRetVal klogAfterRun(modConfData_t *pModConf) +{ + DEFiRet; + if(fklog != -1) + close(fklog); + /* Turn on logging of messages to console, but only if a log level was speficied */ + if(pModConf->console_log_level != -1) + klogctl(7, NULL, 0); + RETiRet; +} + + +/* to be called in the module's WillRun entry point, this is the main + * "message pull" mechanism. + * rgerhards, 2008-04-09 + */ +rsRetVal klogLogKMsg(modConfData_t __attribute__((unused)) *pModConf) +{ + DEFiRet; + readkmsg(); + RETiRet; +} + + +/* provide the (system-specific) default facility for internal messages + * rgerhards, 2008-04-14 + */ +int +klogFacilIntMsg(void) +{ + return LOG_SYSLOG; +} + diff --git a/plugins/impstats/impstats.c b/plugins/impstats/impstats.c index 62599969..79749e21 100644 --- a/plugins/impstats/impstats.c +++ b/plugins/impstats/impstats.c @@ -27,6 +27,12 @@ #include <signal.h> #include <string.h> #include <pthread.h> +#include <fcntl.h> +#include <sys/uio.h> +#if defined(__FreeBSD__) +#include <sys/stat.h> +#endif + #include "dirty.h" #include "cfsysline.h" #include "module-template.h" @@ -63,11 +69,14 @@ typedef struct configSettings_s { } configSettings_t; struct modConfData_s { - rsconf_t *pConf; /* our overall config object */ + rsconf_t *pConf; /* our overall config object */ int iStatsInterval; int iFacility; int iSeverity; + int logfd; /* fd if logging to file, or -1 if closed */ statsFmtType_t statsFmt; + sbool bLogToSyslog; + char *logfile; sbool configSetViaV2Method; }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ @@ -82,6 +91,8 @@ static struct cnfparamdescr modpdescr[] = { { "interval", eCmdHdlrInt, 0 }, { "facility", eCmdHdlrInt, 0 }, { "severity", eCmdHdlrInt, 0 }, + { "log.syslog", eCmdHdlrBinary, 0 }, + { "log.file", eCmdHdlrGetWord, 0 }, { "format", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk modpblk = @@ -120,7 +131,7 @@ initConfigSettings(void) /* actually submit a message to the rsyslog core */ -static inline rsRetVal +static inline void doSubmitMsg(uchar *line) { msg_t *pMsg; @@ -138,11 +149,55 @@ doSubmitMsg(uchar *line) pMsg->iSeverity = runModConf->iSeverity; pMsg->msgFlags = 0; - submitMsg(pMsg); + /* we do not use rate-limiting, as the stats message always need to be emitted */ + submitMsg2(pMsg); + DBGPRINTF("impstats: submit [%d,%d] msg '%s'\n", runModConf->iFacility, + runModConf->iSeverity, line); finalize_it: - RETiRet; + return; +} + +/* log stats message to file; limited error handling done */ +static inline void +doLogToFile(cstr_t *cstr) +{ + struct iovec iov[4]; + ssize_t nwritten; + ssize_t nexpect; + time_t t; + char timebuf[32]; + + if(cstrLen(cstr) == 0) + goto done; + if(runModConf->logfd == -1) { + runModConf->logfd = open(runModConf->logfile, O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, S_IRUSR|S_IWUSR); + if(runModConf->logfd == -1) { + dbgprintf("error opening stats file %s\n", runModConf->logfile); + goto done; + } + } + + time(&t); + iov[0].iov_base = ctime_r(&t, timebuf); + iov[0].iov_len = nexpect = strlen(iov[0].iov_base) - 1; /* -1: strip \n */ + iov[1].iov_base = ": "; + iov[1].iov_len = 2; + nexpect += 2; + iov[2].iov_base = rsCStrGetSzStrNoNULL(cstr); + iov[2].iov_len = (size_t) cstrLen(cstr); + nexpect += cstrLen(cstr); + iov[3].iov_base = "\n"; + iov[3].iov_len = 1; + nexpect++; + nwritten = writev(runModConf->logfd, iov, 4); + + if(nwritten != nexpect) { + dbgprintf("error writing stats file %s, nwritten %lld, expected %lld\n", + runModConf->logfile, (long long) nwritten, (long long) nexpect); + } +done: return; } @@ -153,7 +208,10 @@ static rsRetVal doStatsLine(void __attribute__((unused)) *usrptr, cstr_t *cstr) { DEFiRet; - doSubmitMsg(rsCStrGetSzStrNoNULL(cstr)); + if(runModConf->bLogToSyslog) + doSubmitMsg(rsCStrGetSzStrNoNULL(cstr)); + if(runModConf->logfile != NULL) + doLogToFile(cstr); RETiRet; } @@ -178,6 +236,9 @@ CODESTARTbeginCnfLoad loadModConf->iFacility = DEFAULT_FACILITY; loadModConf->iSeverity = DEFAULT_SEVERITY; loadModConf->statsFmt = statsFmt_Legacy; + loadModConf->logfd = -1; + loadModConf->logfile = NULL; + loadModConf->bLogToSyslog = 1; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config vars */ initConfigSettings(); @@ -210,6 +271,10 @@ CODESTARTsetModCnf loadModConf->iFacility = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "severity")) { loadModConf->iSeverity = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "log.syslog")) { + loadModConf->bLogToSyslog = (sbool) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "log.file")) { + loadModConf->logfile = es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "format")) { mode = es_str2cstr(pvals[i].val.d.estr, NULL); if(!strcasecmp(mode, "json")) { @@ -270,7 +335,9 @@ BEGINactivateCnf rsRetVal localRet; CODESTARTactivateCnf runModConf = pModConf; - DBGPRINTF("impstats: stats interval %d seconds\n", runModConf->iStatsInterval); + DBGPRINTF("impstats: stats interval %d seconds, logToSyslog %d, logFile %s\n", + runModConf->iStatsInterval, runModConf->bLogToSyslog, + runModConf->logfile == NULL ? "deactivated" : (char*)runModConf->logfile); localRet = statsobj.EnableStats(); if(localRet != RS_RET_OK) { errmsg.LogError(0, localRet, "impstats: error enabling statistics gathering"); @@ -282,6 +349,9 @@ ENDactivateCnf BEGINfreeCnf CODESTARTfreeCnf + if(runModConf->logfd != -1) + close(runModConf->logfd); + free(runModConf->logfile); ENDfreeCnf @@ -297,6 +367,7 @@ CODESTARTrunInput if(glbl.GetGlobalInputTermState() == 1) break; /* terminate input! */ + DBGPRINTF("impstats: woke up, generating messages\n"); generateStatsMsgs(); } ENDrunInput diff --git a/plugins/imptcp/imptcp.c b/plugins/imptcp/imptcp.c index a13fd990..906521dd 100644 --- a/plugins/imptcp/imptcp.c +++ b/plugins/imptcp/imptcp.c @@ -67,6 +67,7 @@ #include "ruleset.h" #include "msg.h" #include "statsobj.h" +#include "ratelimit.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ /* the define is from tcpsrv.h, we need to find a new (but easier!!!) abstraction layer some time ... */ @@ -121,6 +122,8 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + int ratelimitInterval; + int ratelimitBurst; struct instanceConf_s *next; }; @@ -158,6 +161,8 @@ static struct cnfparamdescr inppdescr[] = { { "keepalive.time", eCmdHdlrInt, 0 }, { "keepalive.interval", eCmdHdlrInt, 0 }, { "addtlframedelimiter", eCmdHdlrInt, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -195,6 +200,7 @@ struct ptcpsrv_s { sbool bKeepAlive; /* support keep-alive packets */ sbool bEmitMsgOnClose; sbool bSuppOctetFram; + ratelimit_t *ratelimiter; }; /* the ptcp session object. Describes a single active session. @@ -295,6 +301,7 @@ destructSess(ptcpsess_t *pSess) static void destructSrv(ptcpsrv_t *pSrv) { + ratelimitDestruct(pSrv->ratelimiter); prop.Destruct(&pSrv->pInputName); pthread_mutex_destroy(&pSrv->mutSessLst); free(pSrv->pszInputName); @@ -414,7 +421,9 @@ startupSrv(ptcpsrv_t *pSrv) #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - DBGPRINTF("error %d while binding tcp socket\n", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; @@ -677,14 +686,7 @@ doSubmitMsg(ptcpsess_t *pThis, struct syslogTime *stTime, time_t ttGenTime, mult MsgSetRuleset(pMsg, pSrv->pRuleset); STATSCOUNTER_INC(pThis->pLstn->ctrSubmit, pThis->pLstn->mutCtrSubmit); - if(pMultiSub == NULL) { - CHKiRet(submitMsg(pMsg)); - } else { - pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; - if(pMultiSub->nElem == pMultiSub->maxElem) - CHKiRet(multiSubmitMsg(pMultiSub)); - } - + ratelimitAddMsg(pSrv->ratelimiter, pMultiSub, pMsg); finalize_it: /* reset status variables */ @@ -803,12 +805,11 @@ processDataRcvd(ptcpsess_t *pThis, char c, struct syslogTime *stTime, time_t ttG * we have just received a bunch of data! -- rgerhards, 2009-06-16 * EXTRACT from tcps_sess.c */ -#define NUM_MULTISUB 1024 static rsRetVal DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) { multi_submit_t multiSub; - msg_t *pMsgs[NUM_MULTISUB]; + msg_t *pMsgs[CONF_NUM_MULTISUB]; struct syslogTime stTime; time_t ttGenTime; char *pEnd; @@ -819,7 +820,7 @@ DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) datetime.getCurrTime(&stTime, &ttGenTime); multiSub.ppMsgs = pMsgs; - multiSub.maxElem = NUM_MULTISUB; + multiSub.maxElem = CONF_NUM_MULTISUB; multiSub.nElem = 0; /* We now copy the message to the session buffer. */ @@ -829,15 +830,11 @@ DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); } - if(multiSub.nElem > 0) { - /* submit anything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&multiSub)); - } + iRet = multiSubmitFlush(&multiSub); finalize_it: RETiRet; } -#undef NUM_MULTISUB /****************************************** --END-- TCP SUPPORT FUNCTIONS ***********************************/ @@ -1049,6 +1046,8 @@ createInstance(instanceConf_t **pinst) inst->bEmitMsgOnClose = 0; inst->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; inst->pBindRuleset = NULL; + inst->ratelimitBurst = 10000; /* arbitrary high limit */ + inst->ratelimitInterval = 0; /* off */ /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -1128,6 +1127,9 @@ addListner(modConfData_t __attribute__((unused)) *modConf, instanceConf_t *inst) pSrv->iKeepAliveProbes = inst->iKeepAliveProbes; pSrv->iKeepAliveTime = inst->iKeepAliveTime; pSrv->bEmitMsgOnClose = inst->bEmitMsgOnClose; + CHKiRet(ratelimitNew(&pSrv->ratelimiter, "imtcp", (char*)inst->pszBindPort)); + ratelimitSetLinuxLike(pSrv->ratelimiter, inst->ratelimitInterval, inst->ratelimitBurst); + ratelimitSetThreadSafe(pSrv->ratelimiter); CHKmalloc(pSrv->port = ustrdup(inst->pszBindPort)); pSrv->iAddtlFrameDelim = inst->iAddtlFrameDelim; if(inst->pszBindAddr == NULL) @@ -1417,10 +1419,7 @@ BEGINnewInpInst CODESTARTnewInpInst DBGPRINTF("newInpInst (imptcp)\n"); - pvals = nvlstGetParams(lst, &inppblk, NULL); - if(pvals == NULL) { - errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, - "imptcp: required parameter are missing\n"); + if((pvals = nvlstGetParams(lst, &inppblk, NULL)) == NULL) { ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); } @@ -1442,7 +1441,7 @@ CODESTARTnewInpInst inst->pszInputName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(inppblk.descr[i].name, "supportOctetCountedFraming")) { + } else if(!strcmp(inppblk.descr[i].name, "supportoctetcountedframing")) { inst->bSuppOctetFram = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "keepalive")) { inst->bKeepAlive = (int) pvals[i].val.d.n; @@ -1456,6 +1455,10 @@ CODESTARTnewInpInst inst->iAddtlFrameDelim = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "notifyonconnectionclose")) { inst->bEmitMsgOnClose = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; } else { dbgprintf("imptcp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); diff --git a/plugins/imrelp/imrelp.c b/plugins/imrelp/imrelp.c index fe987a50..5e0ae552 100644 --- a/plugins/imrelp/imrelp.c +++ b/plugins/imrelp/imrelp.c @@ -113,11 +113,29 @@ static struct cnfparamblk inppblk = * we will only see the hostname (twice). -- rgerhards, 2009-10-14 */ static relpRetVal -onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *pMsg, size_t lenMsg) +onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *msg, size_t lenMsg) { + prop_t *pProp = NULL; + msg_t *pMsg; DEFiRet; - parseAndSubmitMessage(pHostname, pIP, pMsg, lenMsg, PARSE_HOSTNAME, - eFLOWCTL_LIGHT_DELAY, pInputName, NULL, 0, runModConf->pBindRuleset); + + CHKiRet(msgConstruct(&pMsg)); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsg(pMsg, (char*)msg, lenMsg); + MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + MsgSetRuleset(pMsg, runModConf->pBindRuleset); + pMsg->msgFlags = PARSE_HOSTNAME | NEEDS_PARSING; + + /* TODO: optimize this, we can store it inside the session, requires + * changes to librelp --> next librelp iteration?. rgerhards, 2012-10-29 + */ + MsgSetRcvFromStr(pMsg, pHostname, ustrlen(pHostname), &pProp); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(MsgSetRcvFromIPStr(pMsg, pIP, ustrlen(pIP), &pProp)); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(submitMsg2(pMsg)); + +finalize_it: RETiRet; } @@ -190,6 +208,7 @@ addListner(modConfData_t __attribute__((unused)) *modConf, instanceConf_t *inst) if(pRelpEngine == NULL) { CHKiRet(relpEngineConstruct(&pRelpEngine)); CHKiRet(relpEngineSetDbgprint(pRelpEngine, dbgprintf)); + CHKiRet(relpEngineSetFamily(pRelpEngine, glbl.GetDefPFFamily())); CHKiRet(relpEngineSetEnableCmd(pRelpEngine, (uchar*) "syslog", eRelpCmdState_Required)); CHKiRet(relpEngineSetSyslogRcv(pRelpEngine, onSyslogRcv)); if (!glbl.GetDisableDNS()) { @@ -303,7 +322,14 @@ ENDactivateCnf BEGINfreeCnf + instanceConf_t *inst, *del; CODESTARTfreeCnf + for(inst = pModConf->root ; inst != NULL ; ) { + free(inst->pszBindPort); + del = inst; + inst = inst->next; + free(del); + } ENDfreeCnf /* This is used to terminate the plugin. Note that the signal handler blocks diff --git a/plugins/imtcp/imtcp.c b/plugins/imtcp/imtcp.c index 3ad03615..2d9761cb 100644 --- a/plugins/imtcp/imtcp.c +++ b/plugins/imtcp/imtcp.c @@ -36,7 +36,6 @@ * * rgerhards, 2008-05-19 */ - #include "config.h" #include <stdlib.h> #include <assert.h> @@ -62,6 +61,7 @@ #include "errmsg.h" #include "tcpsrv.h" #include "ruleset.h" +#include "rainerscript.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ MODULE_TYPE_INPUT @@ -105,6 +105,8 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ + int ratelimitInterval; + int ratelimitBurst; int bSuppOctetFram; struct instanceConf_s *next; }; @@ -123,6 +125,7 @@ struct modConfData_s { sbool bKeepAlive; sbool bEmitMsgOnClose; /* emit an informational message on close by remote peer */ uchar *pszStrmDrvrAuthMode; /* authentication mode to use */ + struct cnfarray *permittedPeers; sbool configSetViaV2Method; }; @@ -138,8 +141,10 @@ static struct cnfparamdescr modpdescr[] = { { "addtlframedelimiter", eCmdHdlrPositiveInt, 0 }, { "maxsessions", eCmdHdlrPositiveInt, 0 }, { "maxlistners", eCmdHdlrPositiveInt, 0 }, + { "maxlisteners", eCmdHdlrPositiveInt, 0 }, { "streamdriver.mode", eCmdHdlrPositiveInt, 0 }, { "streamdriver.authmode", eCmdHdlrString, 0 }, + { "permittedpeer", eCmdHdlrArray, 0 }, { "keepalive", eCmdHdlrBinary, 0 } }; static struct cnfparamblk modpblk = @@ -153,7 +158,9 @@ static struct cnfparamdescr inppdescr[] = { { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ { "name", eCmdHdlrString, 0 }, { "ruleset", eCmdHdlrString, 0 }, - { "supportOctetCountedFraming", eCmdHdlrBinary, 0 } + { "supportOctetCountedFraming", eCmdHdlrBinary, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -249,6 +256,8 @@ createInstance(instanceConf_t **pinst) inst->pszBindRuleset = NULL; inst->pszInputName = NULL; inst->bSuppOctetFram = 1; + inst->ratelimitInterval = 0; + inst->ratelimitBurst = 10000; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -332,6 +341,7 @@ addListner(modConfData_t *modConf, instanceConf_t *inst) CHKiRet(tcpsrv.SetRuleset(pOurTcpsrv, inst->pBindRuleset)); CHKiRet(tcpsrv.SetInputName(pOurTcpsrv, inst->pszInputName == NULL ? UCHAR_CONSTANT("imtcp") : inst->pszInputName)); + CHKiRet(tcpsrv.SetLinuxLikeRatelimiters(pOurTcpsrv, inst->ratelimitInterval, inst->ratelimitBurst)); tcpsrv.configureTCPListen(pOurTcpsrv, inst->pszBindPort, inst->bSuppOctetFram); finalize_it: @@ -374,6 +384,10 @@ CODESTARTnewInpInst inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "supportOctetCountedFraming")) { inst->bSuppOctetFram = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; } else { dbgprintf("imtcp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); @@ -400,6 +414,7 @@ CODESTARTbeginCnfLoad loadModConf->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; loadModConf->bDisableLFDelim = 0; loadModConf->pszStrmDrvrAuthMode = NULL; + loadModConf->permittedPeers = NULL; loadModConf->configSetViaV2Method = 0; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config variables */ @@ -439,14 +454,17 @@ CODESTARTsetModCnf loadModConf->iAddtlFrameDelim = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "maxsessions")) { loadModConf->iTCPSessMax = (int) pvals[i].val.d.n; - } else if(!strcmp(modpblk.descr[i].name, "maxlistners")) { + } else if(!strcmp(modpblk.descr[i].name, "maxlisteners") || + !strcmp(modpblk.descr[i].name, "maxlistners")) { /* keep old name for a while */ loadModConf->iTCPLstnMax = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "keepalive")) { loadModConf->bKeepAlive = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "streamdriver.mode")) { loadModConf->iStrmDrvrMode = (int) pvals[i].val.d.n; - } else if(!strcmp(modpblk.descr[i].name, "streamdriver.mode")) { + } else if(!strcmp(modpblk.descr[i].name, "streamdriver.authmode")) { loadModConf->pszStrmDrvrAuthMode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(modpblk.descr[i].name, "permittedpeer")) { + loadModConf->permittedPeers = cnfarrayDup(pvals[i].val.d.ar); } else { dbgprintf("imtcp: program error, non-handled " "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); @@ -482,10 +500,10 @@ CODESTARTendCnfLoad loadModConf->pszStrmDrvrAuthMode = NULL; } else { loadModConf->pszStrmDrvrAuthMode = cs.pszStrmDrvrAuthMode; + cs.pszStrmDrvrAuthMode = NULL; } } - if((cs.pszStrmDrvrAuthMode == NULL) || (cs.pszStrmDrvrAuthMode[0] == '\0')) - free(cs.pszStrmDrvrAuthMode); + free(cs.pszStrmDrvrAuthMode); cs.pszStrmDrvrAuthMode = NULL; loadModConf = NULL; /* done loading */ @@ -517,8 +535,15 @@ ENDcheckCnf BEGINactivateCnfPrePrivDrop instanceConf_t *inst; + int i; CODESTARTactivateCnfPrePrivDrop runModConf = pModConf; + if(runModConf->permittedPeers != NULL) { + for(i = 0 ; i < runModConf->permittedPeers->nmemb ; ++i) { + setPermittedPeer(NULL, (uchar*) + es_str2cstr(runModConf->permittedPeers->arr[i], NULL)); + } + } for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { addListner(pModConf, inst); } @@ -538,6 +563,11 @@ ENDactivateCnf BEGINfreeCnf instanceConf_t *inst, *del; CODESTARTfreeCnf + free(pModConf->pszStrmDrvrAuthMode); + if(pModConf->permittedPeers != NULL) { + cnfarrayContentDestruct(pModConf->permittedPeers); + free(pModConf->permittedPeers); + } for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindPort); free(inst->pszInputName); @@ -564,7 +594,9 @@ ENDwillRun BEGINafterRun CODESTARTafterRun - /* do cleanup here */ + if(pOurTcpsrv != NULL) + iRet = tcpsrv.Destruct(&pOurTcpsrv); + net.clearAllowedSenders(UCHAR_CONSTANT("TCP")); ENDafterRun @@ -578,9 +610,6 @@ ENDisCompatibleWithFeature BEGINmodExit CODESTARTmodExit - if(pOurTcpsrv != NULL) - iRet = tcpsrv.Destruct(&pOurTcpsrv); - if(pPermPeersRoot != NULL) { net.DestructPermittedPeers(&pPermPeersRoot); } @@ -643,8 +672,6 @@ CODEmodInit_QueryRegCFSLineHdlr /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverrun"), 0, eCmdHdlrGetWord, addInstance, NULL, STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverstreamdriverpermittedpeer"), 0, eCmdHdlrGetWord, - setPermittedPeer, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverinputname"), 0, eCmdHdlrGetWord, NULL, &cs.pszInputName, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverbindruleset"), 0, eCmdHdlrGetWord, @@ -652,6 +679,8 @@ CODEmodInit_QueryRegCFSLineHdlr /* module-global config params - will be disabled in configs that are loaded * via module(...). */ + CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverstreamdriverpermittedpeer"), 0, eCmdHdlrGetWord, + setPermittedPeer, NULL, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverstreamdriverauthmode"), 0, eCmdHdlrGetWord, NULL, &cs.pszStrmDrvrAuthMode, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(regCfSysLineHdlr2(UCHAR_CONSTANT("inputtcpserverkeepalive"), 0, eCmdHdlrBinary, diff --git a/plugins/imttcp/imttcp.c b/plugins/imttcp/imttcp.c index c72886b3..9bd11f77 100644 --- a/plugins/imttcp/imttcp.c +++ b/plugins/imttcp/imttcp.c @@ -365,7 +365,9 @@ createSrv(ttcpsrv_t *pSrv) #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - DBGPRINTF("error %d while binding tcp socket", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index ea0a8282..312645bd 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -4,8 +4,6 @@ * NOTE: read comments in module-template.h to understand how this file * works! * - * File begun on 2007-12-21 by RGerhards (extracted from syslogd.c) - * * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. @@ -53,6 +51,7 @@ #include "prop.h" #include "ruleset.h" #include "statsobj.h" +#include "ratelimit.h" #include "unicode-helper.h" MODULE_TYPE_INPUT @@ -76,7 +75,9 @@ static struct lstn_s { struct lstn_s *next; int sock; /* socket */ ruleset_t *pRuleset; /* bound ruleset */ + prop_t *pInputName; statsobj_t *stats; /* listener stats */ + ratelimit_t *ratelimiter; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) } *lcnfRoot = NULL, *lcnfLast = NULL; @@ -91,7 +92,6 @@ static uchar *pRcvBuf = NULL; /* receive buffer (for a single packet). We use a * it so that we can check available memory in willRun() and request * termination if we can not get it. -- rgerhards, 2007-12-27 */ -static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ #define TIME_REQUERY_DFLT 2 #define SCHED_PRIO_UNSET -12345678 /* a value that indicates that the scheduling priority has not been set */ @@ -108,8 +108,12 @@ struct instanceConf_s { uchar *pszBindAddr; /* IP to bind socket to */ uchar *pszBindPort; /* Port to bind socket to */ uchar *pszBindRuleset; /* name of ruleset to bind to */ + uchar *inputname; ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + int ratelimitInterval; + int ratelimitBurst; struct instanceConf_s *next; + sbool bAppendPortToInpname; }; struct modConfData_s { @@ -138,9 +142,13 @@ static struct cnfparamblk modpblk = /* input instance parameters */ static struct cnfparamdescr inppdescr[] = { - { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ + { "port", eCmdHdlrArray, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ + { "inputname", eCmdHdlrGetWord, 0 }, + { "inputname.appendport", eCmdHdlrBinary, 0 }, { "address", eCmdHdlrString, 0 }, - { "ruleset", eCmdHdlrString, 0 } + { "ruleset", eCmdHdlrString, 0 }, + { "ratelimit.interval", eCmdHdlrInt, 0 }, + { "ratelimit.burst", eCmdHdlrInt, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -165,6 +173,10 @@ createInstance(instanceConf_t **pinst) inst->pszBindPort = NULL; inst->pszBindAddr = NULL; inst->pszBindRuleset = NULL; + inst->inputname = NULL; + inst->bAppendPortToInpname = 0; + inst->ratelimitBurst = 10000; /* arbitrary high limit */ + inst->ratelimitInterval = 0; /* off */ /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -223,7 +235,8 @@ addListner(instanceConf_t *inst) struct lstn_s *newlcnfinfo; uchar *bindName; uchar *port; - uchar statname[64]; + uchar dispname[64], inpnameBuf[128]; + uchar *inputname; /* check which address to bind to. We could do this more compact, but have not * done so in order to make the code more readable. -- rgerhards, 2007-12-27 @@ -248,11 +261,29 @@ addListner(instanceConf_t *inst) newlcnfinfo->next = NULL; newlcnfinfo->sock = newSocks[iSrc]; newlcnfinfo->pRuleset = inst->pBindRuleset; + snprintf((char*)dispname, sizeof(dispname), "imudp(%s:%s)", bindName, port); + dispname[sizeof(dispname)-1] = '\0'; /* just to be on the save side... */ + CHKiRet(ratelimitNew(&newlcnfinfo->ratelimiter, (char*)dispname, NULL)); + if(inst->inputname == NULL) { + inputname = (uchar*)"imudp"; + } else { + inputname = inst->inputname; + } + if(inst->bAppendPortToInpname) { + snprintf((char*)inpnameBuf, sizeof(inpnameBuf), "%s%s", + inputname, port); + inpnameBuf[sizeof(inpnameBuf)-1] = '\0'; + inputname = inpnameBuf; + } + CHKiRet(prop.Construct(&newlcnfinfo->pInputName)); + CHKiRet(prop.SetString(newlcnfinfo->pInputName, + inputname, ustrlen(inputname))); + CHKiRet(prop.ConstructFinalize(newlcnfinfo->pInputName)); + ratelimitSetLinuxLike(newlcnfinfo->ratelimiter, inst->ratelimitInterval, + inst->ratelimitBurst); /* support statistics gathering */ CHKiRet(statsobj.Construct(&(newlcnfinfo->stats))); - snprintf((char*)statname, sizeof(statname), "imudp(%s:%s)", bindName, port); - statname[sizeof(statname)-1] = '\0'; /* just to be on the save side... */ - CHKiRet(statsobj.SetName(newlcnfinfo->stats, statname)); + CHKiRet(statsobj.SetName(newlcnfinfo->stats, dispname)); STATSCOUNTER_INIT(newlcnfinfo->ctrSubmit, newlcnfinfo->mutCtrSubmit); CHKiRet(statsobj.AddCounter(newlcnfinfo->stats, UCHAR_CONSTANT("submitted"), ctrType_IntCtr, &(newlcnfinfo->ctrSubmit))); @@ -304,7 +335,6 @@ std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, insta static inline rsRetVal processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted) { - DEFiRet; int iNbrTimeUsed; time_t ttGenTime; struct syslogTime stTime; @@ -314,9 +344,15 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f msg_t *pMsg; prop_t *propFromHost = NULL; prop_t *propFromHostIP = NULL; + multi_submit_t multiSub; + msg_t *pMsgs[CONF_NUM_MULTISUB]; char errStr[1024]; + DEFiRet; assert(pThrd != NULL); + multiSub.ppMsgs = pMsgs; + multiSub.maxElem = CONF_NUM_MULTISUB; + multiSub.nElem = 0; iNbrTimeUsed = 0; while(1) { /* loop is terminated if we have a bad receive, done below in the body */ if(pThrd->bShallStop == RSTRUE) @@ -367,7 +403,7 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f *pbIsPermitted = 1; /* no check -> everything permitted */ } - DBGPRINTF("recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, pRcvBuf); + DBGPRINTF("imudp:recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, pRcvBuf); if(*pbIsPermitted != 0) { if((runModConf->iTimeRequery == 0) || (iNbrTimeUsed++ % runModConf->iTimeRequery) == 0) { @@ -376,19 +412,22 @@ processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *f /* we now create our own message object and submit it to the queue */ CHKiRet(msgConstructWithTime(&pMsg, &stTime, ttGenTime)); MsgSetRawMsg(pMsg, (char*)pRcvBuf, lenRcvBuf); - MsgSetInputName(pMsg, pInputName); + MsgSetInputName(pMsg, lstn->pInputName); MsgSetRuleset(pMsg, lstn->pRuleset); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME | NEEDS_DNSRESOL; if(*pbIsPermitted == 2) pMsg->msgFlags |= NEEDS_ACLCHK_U; /* request ACL check after resolution */ CHKiRet(msgSetFromSockinfo(pMsg, &frominet)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(ratelimitAddMsg(lstn->ratelimiter, &multiSub, pMsg)); STATSCOUNTER_INC(lstn->ctrSubmit, lstn->mutCtrSubmit); } } + finalize_it: + multiSubmitFlush(&multiSub); + if(propFromHost != NULL) prop.Destruct(&propFromHost); if(propFromHostIP != NULL) @@ -664,42 +703,64 @@ rsRetVal rcvMainLoop(thrdInfo_t *pThrd) #endif /* #if HAVE_EPOLL_CREATE1 */ -BEGINnewInpInst - struct cnfparamvals *pvals; +static inline rsRetVal +createListner(es_str_t *port, struct cnfparamvals *pvals) +{ instanceConf_t *inst; int i; -CODESTARTnewInpInst - DBGPRINTF("newInpInst (imudp)\n"); - - pvals = nvlstGetParams(lst, &inppblk, NULL); - if(pvals == NULL) { - errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, - "imudp: required parameter are missing\n"); - ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); - } - - if(Debug) { - dbgprintf("input param blk in imudp:\n"); - cnfparamsPrint(&inppblk, pvals); - } + DEFiRet; CHKiRet(createInstance(&inst)); - + inst->pszBindPort = (uchar*)es_str2cstr(port, NULL); for(i = 0 ; i < inppblk.nParams ; ++i) { if(!pvals[i].bUsed) continue; if(!strcmp(inppblk.descr[i].name, "port")) { - inst->pszBindPort = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + continue; /* array, handled by caller */ + } else if(!strcmp(inppblk.descr[i].name, "inputname")) { + inst->inputname = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "inputname.appendport")) { + inst->bAppendPortToInpname = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "address")) { inst->pszBindAddr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { + inst->ratelimitBurst = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { + inst->ratelimitInterval = (int) pvals[i].val.d.n; } else { dbgprintf("imudp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); } } finalize_it: + RETiRet; +} + + +BEGINnewInpInst + struct cnfparamvals *pvals; + int i; + int portIdx; +CODESTARTnewInpInst + DBGPRINTF("newInpInst (imudp)\n"); + + if((pvals = nvlstGetParams(lst, &inppblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + if(Debug) { + dbgprintf("input param blk in imudp:\n"); + cnfparamsPrint(&inppblk, pvals); + } + + portIdx = cnfparamGetIdx(&inppblk, "port"); + assert(portIdx != -1); + for(i = 0 ; i < pvals[portIdx].val.d.ar->nmemb ; ++i) { + createListner(pvals[portIdx].val.d.ar->arr[i], pvals); + } + +finalize_it: CODE_STD_FINALIZERnewInpInst cnfparamvalsDestruct(pvals, &inppblk); ENDnewInpInst @@ -808,14 +869,13 @@ CODESTARTactivateCnfPrePrivDrop for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { addListner(inst); } - /* if we could not set up any listners, there is no point in running... */ + /* if we could not set up any listeners, there is no point in running... */ if(lcnfRoot == NULL) { errmsg.LogError(0, NO_ERRCODE, "imudp: no listeners could be started, " "input not activated.\n"); ABORT_FINALIZE(RS_RET_NO_RUN); } - setSchedParams(pModConf); finalize_it: ENDactivateCnfPrePrivDrop @@ -835,7 +895,7 @@ CODESTARTfreeCnf for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindPort); free(inst->pszBindAddr); - free(inst->pBindRuleset); + free(inst->inputname); del = inst; inst = inst->next; free(del); @@ -848,6 +908,15 @@ ENDfreeCnf */ BEGINrunInput CODESTARTrunInput + /* Note well: the setting of scheduling parameters will not work + * when we dropped privileges (if the user is not sufficently + * privileged, of course). Howerver, we can't change the + * scheduling params in PrePrivDrop(), as at that point our thread + * is not yet created. So at least as an interim solution, we do + * NOT support both setting sched parameters and dropping + * privileges within the same instance. + */ + setSchedParams(runModConf); iRet = rcvMainLoop(pThrd); ENDrunInput @@ -867,7 +936,9 @@ CODESTARTafterRun net.clearAllowedSenders((uchar*)"UDP"); for(lstn = lcnfRoot ; lstn != NULL ; ) { statsobj.Destruct(&(lstn->stats)); + ratelimitDestruct(lstn->ratelimiter); close(lstn->sock); + prop.Destruct(&lstn->pInputName); lstnDel = lstn; lstn = lstn->next; free(lstnDel); @@ -882,9 +953,6 @@ ENDafterRun BEGINmodExit CODESTARTmodExit - if(pInputName != NULL) - prop.Destruct(&pInputName); - /* release what we no longer need */ objRelease(errmsg, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); @@ -939,11 +1007,6 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(ruleset, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); - /* we need to create the inputName property (only once during our lifetime) */ - CHKiRet(prop.Construct(&pInputName)); - CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imudp"), sizeof("imudp") - 1)); - CHKiRet(prop.ConstructFinalize(pInputName)); - /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputudpserverbindruleset", 0, eCmdHdlrGetWord, NULL, &cs.pszBindRuleset, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/imuxsock/Makefile.am b/plugins/imuxsock/Makefile.am index 34a0ad9a..28f9f9e3 100644 --- a/plugins/imuxsock/Makefile.am +++ b/plugins/imuxsock/Makefile.am @@ -1,6 +1,6 @@ pkglib_LTLIBRARIES = imuxsock.la imuxsock_la_SOURCES = imuxsock.c -imuxsock_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -I../../runtime/hashtable -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +imuxsock_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) imuxsock_la_LDFLAGS = -module -avoid-version imuxsock_la_LIBADD = $(RSRT_LIBS) diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index 79c6b734..c503852c 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -6,7 +6,7 @@ * * File begun on 2007-12-20 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -55,6 +55,7 @@ #include "statsobj.h" #include "datetime.h" #include "hashtable.h" +#include "ratelimit.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -105,15 +106,6 @@ STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) STATSCOUNTER_DEF(ctrLostRatelimit, mutCtrLostRatelimit) STATSCOUNTER_DEF(ctrNumRatelimiters, mutCtrNumRatelimiters) -struct rs_ratelimit_state { - unsigned short interval; - unsigned short burst; - unsigned done; - unsigned missed; - time_t begin; -}; -typedef struct rs_ratelimit_state rs_ratelimit_state_t; - /* a very simple "hash function" for process IDs - we simply use the * pid itself: it is quite expected that all pids may log some time, but @@ -143,6 +135,7 @@ typedef struct lstn_s { int flowCtl; /* flow control settings for this socket */ int ratelimitInterval; int ratelimitBurst; + ratelimit_t *dflt_ratelimiter;/*ratelimiter to apply if none else is to be used */ intTiny ratelimitSev; /* severity level (and below) for which rate-limiting shall apply */ struct hashtable *ht; /* our hashtable for rate-limiting */ sbool bParseHost; /* should parser parse host name? read-only after startup */ @@ -151,7 +144,9 @@ typedef struct lstn_s { sbool bAnnotate; /* annotate events with trusted properties */ sbool bParseTrusted; /* parse trusted properties */ sbool bWritePid; /* write original PID into tag */ + sbool bDiscardOwnMsgs; /* discard messages that originated from ourselves */ sbool bUseSysTimeStamp; /* use timestamp from system (instead of from message) */ + sbool bUnlink; /* unlink&re-create socket at start and end of processing */ } lstn_t; static lstn_t listeners[MAXFUNIX]; @@ -164,8 +159,6 @@ static int startIndexUxLocalSockets; /* process fd from that index on (used to static int nfd = 1; /* number of Unix sockets open / read-only after startup */ static int sd_fds = 0; /* number of systemd activated sockets */ -static ee_ctx ctxee = NULL; /* library context */ - /* config vars for legacy config system */ #define DFLT_bCreatePath 0 #define DFLT_ratelimitInterval 0 @@ -208,6 +201,8 @@ struct instanceConf_s { int ratelimitSeverity; int bAnnotate; /* annotate trusted properties */ int bParseTrusted; /* parse trusted properties */ + sbool bDiscardOwnMsgs; /* discard messages that originated from our own pid? */ + sbool bUnlink; struct instanceConf_s *next; }; @@ -225,7 +220,9 @@ struct modConfData_s { sbool bOmitLocalLogging; sbool bWritePidSysSock; sbool bUseSysTimeStamp; + sbool bDiscardOwnMsgs; sbool configSetViaV2Method; + sbool bUnlink; }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ @@ -234,10 +231,13 @@ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current lo static struct cnfparamdescr modpdescr[] = { { "syssock.use", eCmdHdlrBinary, 0 }, { "syssock.name", eCmdHdlrGetWord, 0 }, + { "syssock.unlink", eCmdHdlrBinary, 0 }, { "syssock.ignoretimestamp", eCmdHdlrBinary, 0 }, + { "syssock.ignoreownmessages", eCmdHdlrBinary, 0 }, { "syssock.flowcontrol", eCmdHdlrBinary, 0 }, { "syssock.usesystimestamp", eCmdHdlrBinary, 0 }, { "syssock.annotate", eCmdHdlrBinary, 0 }, + { "syssock.parsetrusted", eCmdHdlrBinary, 0 }, { "syssock.usepidfromsystem", eCmdHdlrBinary, 0 }, { "syssock.ratelimit.interval", eCmdHdlrInt, 0 }, { "syssock.ratelimit.burst", eCmdHdlrInt, 0 }, @@ -252,8 +252,10 @@ static struct cnfparamblk modpblk = /* input instance parameters */ static struct cnfparamdescr inppdescr[] = { { "socket", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: addunixlistensocket */ + { "unlink", eCmdHdlrBinary, 0 }, { "createpath", eCmdHdlrBinary, 0 }, { "parsetrusted", eCmdHdlrBinary, 0 }, + { "ignoreownmessages", eCmdHdlrBinary, 0 }, { "hostname", eCmdHdlrString, 0 }, { "ignoretimestamp", eCmdHdlrBinary, 0 }, { "flowcontrol", eCmdHdlrBinary, 0 }, @@ -273,74 +275,9 @@ static struct cnfparamblk inppblk = /* we do not use this, because we do not bind to a ruleset so far * enable when this is changed: #include "im-helper.h" */ /* must be included AFTER the type definitions! */ -static void -initRatelimitState(struct rs_ratelimit_state *rs, unsigned short interval, unsigned short burst) -{ - rs->interval = interval; - rs->burst = burst; - rs->done = 0; - rs->missed = 0; - rs->begin = 0; -} - static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ -/* ratelimiting support, modelled after the linux kernel - * returns 1 if message is within rate limit and shall be - * processed, 0 otherwise. - * This implementation is NOT THREAD-SAFE and must not - * be called concurrently. - */ -static inline int -withinRatelimit(struct rs_ratelimit_state *rs, time_t tt, pid_t pid) -{ - int ret; - uchar msgbuf[1024]; - - if(rs->interval == 0) { - ret = 1; - goto finalize_it; - } - - assert(rs->burst != 0); - - if(rs->begin == 0) - rs->begin = tt; - - /* resume if we go out of out time window */ - if(tt > rs->begin + rs->interval) { - if(rs->missed) { - snprintf((char*)msgbuf, sizeof(msgbuf), - "imuxsock lost %u messages from pid %lu due to rate-limiting", - rs->missed, (unsigned long) pid); - logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); - rs->missed = 0; - } - rs->begin = 0; - rs->done = 0; - } - - /* do actual limit check */ - if(rs->burst > rs->done) { - rs->done++; - ret = 1; - } else { - if(rs->missed == 0) { - snprintf((char*)msgbuf, sizeof(msgbuf), - "imuxsock begins to drop messages from pid %lu due to rate-limiting", - (unsigned long) pid); - logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); - } - rs->missed++; - ret = 0; - } - -finalize_it: - return ret; -} - - /* create input instance, set default paramters, and * add it to the list of instances. */ @@ -353,7 +290,7 @@ createInstance(instanceConf_t **pinst) inst->sockName = NULL; inst->pLogHostName = NULL; inst->ratelimitInterval = DFLT_ratelimitInterval; - inst->ratelimitBurst = DFLT_ratelimitSeverity; + inst->ratelimitBurst = DFLT_ratelimitBurst; inst->ratelimitSeverity = DFLT_ratelimitSeverity; inst->bUseFlowCtl = 0; inst->bIgnoreTimestamp = 1; @@ -362,6 +299,8 @@ createInstance(instanceConf_t **pinst) inst->bWritePid = 0; inst->bAnnotate = 0; inst->bParseTrusted = 0; + inst->bDiscardOwnMsgs = 1; + inst->bUnlink = 1; inst->next = NULL; /* node created, let's add to config */ @@ -447,7 +386,8 @@ addListner(instanceConf_t *inst) CHKiRet(prop.ConstructFinalize(listeners[nfd].hostName)); } if(inst->ratelimitInterval > 0) { - if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, NULL)) == NULL) { + if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))ratelimitDestruct)) == NULL) { /* in this case, we simply turn off rate-limiting */ DBGPRINTF("imuxsock: turning off rate limiting because we could not " "create hash table\n"); @@ -461,11 +401,19 @@ addListner(instanceConf_t *inst) listeners[nfd].flags = inst->bIgnoreTimestamp ? IGNDATE : NOFLAG; listeners[nfd].bCreatePath = inst->bCreatePath; listeners[nfd].sockName = ustrdup(inst->sockName); - listeners[nfd].bUseCreds = (inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate) ? 1 : 0; + listeners[nfd].bUseCreds = (inst->bDiscardOwnMsgs || inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate) ? 1 : 0; listeners[nfd].bAnnotate = inst->bAnnotate; listeners[nfd].bParseTrusted = inst->bParseTrusted; + listeners[nfd].bDiscardOwnMsgs = inst->bDiscardOwnMsgs; + listeners[nfd].bUnlink = inst->bUnlink; listeners[nfd].bWritePid = inst->bWritePid; listeners[nfd].bUseSysTimeStamp = inst->bUseSysTimeStamp; + CHKiRet(ratelimitNew(&listeners[nfd].dflt_ratelimiter, "imuxsock", NULL)); + ratelimitSetLinuxLike(listeners[nfd].dflt_ratelimiter, + listeners[nfd].ratelimitInterval, + listeners[nfd].ratelimitBurst); + ratelimitSetSeverity(listeners[nfd].dflt_ratelimiter, + listeners[nfd].ratelimitSev); nfd++; } else { errmsg.LogError(0, NO_ERRCODE, "Out of unix socket name descriptors, ignoring %s\n", @@ -477,7 +425,7 @@ finalize_it: } -/* discard all log sockets except for "socket" 0. Data for it comes from +/* discard/Destruct all log sockets except for "socket" 0. Data for it comes from * the constant memory pool - and if not, it is freeed via some other pointer. */ static rsRetVal discardLogSockets(void) @@ -495,6 +443,7 @@ static rsRetVal discardLogSockets(void) if(listeners[i].ht != NULL) { hashtable_destroy(listeners[i].ht, 1); /* 1 => free all values automatically */ } + ratelimitDestruct(listeners[i].dflt_ratelimiter); } return RS_RET_OK; @@ -509,7 +458,8 @@ createLogSocket(lstn_t *pLstn) struct sockaddr_un sunx; DEFiRet; - unlink((char*)pLstn->sockName); + if(pLstn->bUnlink) + unlink((char*)pLstn->sockName); memset(&sunx, 0, sizeof(sunx)); sunx.sun_family = AF_UNIX; if(pLstn->bCreatePath) { @@ -606,19 +556,26 @@ finalize_it: * listener (the latter being a performance enhancement). */ static inline rsRetVal -findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) +findRatelimiter(lstn_t *pLstn, struct ucred *cred, ratelimit_t **prl) { - rs_ratelimit_state_t *rl; + ratelimit_t *rl; int r; pid_t *keybuf; + char pidbuf[256]; DEFiRet; if(cred == NULL) FINALIZE; +#if 0 // TODO: check deactivated? if(pLstn->ratelimitInterval == 0) { *prl = NULL; FINALIZE; } +#endif + if(pLstn->ht == NULL) { + *prl = NULL; + FINALIZE; + } rl = hashtable_search(pLstn->ht, &cred->pid); if(rl == NULL) { @@ -626,10 +583,14 @@ findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) DBGPRINTF("imuxsock: no ratelimiter for pid %lu, creating one\n", (unsigned long) cred->pid); STATSCOUNTER_INC(ctrNumRatelimiters, mutCtrNumRatelimiters); - CHKmalloc(rl = malloc(sizeof(rs_ratelimit_state_t))); + snprintf(pidbuf, sizeof(pidbuf), "pid %lu", + (unsigned long) cred->pid); + pidbuf[sizeof(pidbuf)-1] = '\0'; /* to be on safe side */ + CHKiRet(ratelimitNew(&rl, "imuxsock", pidbuf)); + ratelimitSetLinuxLike(rl, pLstn->ratelimitInterval, pLstn->ratelimitBurst); + ratelimitSetSeverity(rl, pLstn->ratelimitSev); CHKmalloc(keybuf = malloc(sizeof(pid_t))); *keybuf = cred->pid; - initRatelimitState(rl, pLstn->ratelimitInterval, pLstn->ratelimitBurst); r = hashtable_insert(pLstn->ht, keybuf, rl); if(r == 0) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); @@ -638,6 +599,8 @@ findRatelimiter(lstn_t *pLstn, struct ucred *cred, rs_ratelimit_state_t **prl) *prl = rl; finalize_it: + if(*prl == NULL) + *prl = pLstn->dflt_ratelimiter; RETiRet; } @@ -690,14 +653,12 @@ getTrustedProp(struct ucred *cred, char *propName, uchar *buf, size_t lenBuf, in if((fd = open(namebuf, O_RDONLY)) == -1) { DBGPRINTF("error reading '%s'\n", namebuf); - *lenProp = 0; - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } if((lenRead = read(fd, buf, lenBuf - 1)) == -1) { DBGPRINTF("error reading file data for '%s'\n", namebuf); - *lenProp = 0; close(fd); - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } /* we strip after the first \n */ @@ -733,8 +694,7 @@ getTrustedExe(struct ucred *cred, uchar *buf, size_t lenBuf, int* lenProp) if((lenRead = readlink(namebuf, (char*)buf, lenBuf - 1)) == -1) { DBGPRINTF("error reading link '%s'\n", namebuf); - *lenProp = 0; - FINALIZE; + ABORT_FINALIZE(RS_RET_ERR); } buf[lenRead] = '\0'; @@ -767,26 +727,6 @@ copyescaped(uchar *dstbuf, uchar *inbuf, int inlen) } -/* Creates new field to be added to event - * used for SystemLogParseTrusted parsing - */ -struct ee_field * -createNewField(char *fieldname, char *value, int lenValue) { - es_str_t *newStr; - struct ee_value *newVal; - struct ee_field *newField; - - newStr = es_newStrFromBuf(value, (es_size_t) lenValue); - - newVal = ee_newValue(ctxee); - ee_setStrValue(newVal, newStr); - - newField = ee_newFieldFromNV(ctxee, fieldname, newVal); - - return newField; -} - - /* submit received message to the queue engine * We now parse the message according to expected format so that we * can also mangle it if necessary. @@ -805,16 +745,21 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim uchar bufParseTAG[CONF_TAG_MAXSIZE]; struct syslogTime st; time_t tt; - rs_ratelimit_state_t *ratelimiter = NULL; int lenProp; + ratelimit_t *ratelimiter = NULL; uchar propBuf[1024]; uchar msgbuf[8192]; uchar *pmsgbuf; int toffs; /* offset for trusted properties */ struct syslogTime dummyTS; - struct ee_event *event = NULL; + struct json_object *json = NULL, *jval; DEFiRet; + if(pLstn->bDiscardOwnMsgs && cred != NULL && cred->pid == glblGetOurPid()) { + DBGPRINTF("imuxsock: discarding message from our own pid\n"); + FINALIZE; + } + /* TODO: handle format errors?? */ /* we need to parse the pri first, because we need the severity for * rate-limiting as well. @@ -833,10 +778,7 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim facil = LOG_FAC(pri); sever = LOG_PRI(pri); - if(sever >= pLstn->ratelimitSev) { - /* note: if cred == NULL, then ratelimiter == NULL as well! */ - findRatelimiter(pLstn, cred, &ratelimiter); /* ignore error, better so than others... */ - } + findRatelimiter(pLstn, cred, &ratelimiter); /* ignore error, better so than others... */ if(ts == NULL) { datetime.getCurrTime(&st, &tt); @@ -845,10 +787,12 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim tt = ts->tv_sec; } +#if 0 // TODO: think about stats counters (or wait for request...?) if(ratelimiter != NULL && !withinRatelimit(ratelimiter, tt, cred->pid)) { STATSCOUNTER_INC(ctrLostRatelimit, mutCtrLostRatelimit); FINALIZE; } +#endif /* created trusted properties */ if(cred != NULL && pLstn->bAnnotate) { @@ -859,45 +803,27 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim } if (pLstn->bParseTrusted) { - struct ee_field *newField; - - if(ctxee == NULL) { - if((ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } + json = json_object_new_object(); + /* create value string, create field, and add it */ + jval = json_object_new_int(cred->pid); + json_object_object_add(json, "pid", jval); + jval = json_object_new_int(cred->uid); + json_object_object_add(json, "uid", jval); + jval = json_object_new_int(cred->gid); + json_object_object_add(json, "gid", jval); + if(getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "appname", jval); + } + if(getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "exe", jval); + } + if(getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + jval = json_object_new_string((char*)propBuf); + json_object_object_add(json, "cmd", jval); } - - event = ee_newEvent(ctxee); - - /* create value string, create field, and add it to event */ - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->pid); - newField = createNewField("pid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->uid); - newField = createNewField("uid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - lenProp = snprintf((char *)propBuf, sizeof(propBuf), "%lu", (long unsigned) cred->gid); - newField = createNewField("gid", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("appname", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("exe", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - - getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp); - newField = createNewField("cmd", (char *)propBuf, lenProp); - ee_addFieldToEvent(event, newField); - } else { - memcpy(pmsgbuf, pRcv, lenRcv); memcpy(pmsgbuf+lenRcv, " @[", 3); toffs = lenRcv + 3; /* next free location */ @@ -907,23 +833,20 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim memcpy(pmsgbuf+toffs, propBuf, lenProp); toffs = toffs + lenProp; - getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { + if(getTrustedProp(cred, "comm", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { memcpy(pmsgbuf+toffs, " _COMM=", 7); memcpy(pmsgbuf+toffs+7, propBuf, lenProp); toffs = toffs + 7 + lenProp; } - getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { + if(getTrustedExe(cred, propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { memcpy(pmsgbuf+toffs, " _EXE=", 6); memcpy(pmsgbuf+toffs+6, propBuf, lenProp); toffs = toffs + 6 + lenProp; } - getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp); - if(lenProp) { - memcpy(pmsgbuf+toffs, " _CMDLINE=", 9); - toffs = toffs + 9 + - copyescaped(pmsgbuf+toffs+9, propBuf, lenProp); + if(getTrustedProp(cred, "cmdline", propBuf, sizeof(propBuf), &lenProp) == RS_RET_OK) { + memcpy(pmsgbuf+toffs, " _CMDLINE=", 10); + toffs = toffs + 10 + + copyescaped(pmsgbuf+toffs+10, propBuf, lenProp); } /* finalize string */ @@ -949,12 +872,11 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim parse++; lenMsg--; /* '>' */ - /* event is saved to pMsg */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - if (event != NULL) { - pMsg->event = event; + if(json != NULL) { + /* as per lumberjack spec, these properties need to go into + * the CEE root. + */ + msgAddJSON(pMsg, (uchar*)"!", json); } if(ts == NULL) { @@ -962,15 +884,19 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim /* in this case, we still need to find out if we have a valid * datestamp or not .. and advance the parse pointer accordingly. */ - datetime.ParseTIMESTAMP3164(&dummyTS, &parse, &lenMsg); + if (datetime.ParseTIMESTAMP3339(&dummyTS, &parse, &lenMsg) != RS_RET_OK) { + datetime.ParseTIMESTAMP3164(&dummyTS, &parse, &lenMsg); + } } else { - if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK) { + if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK && + datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &parse, &lenMsg) != RS_RET_OK) { DBGPRINTF("we have a problem, invalid timestamp in msg!\n"); } } } else { /* if we pulled the time from the system, we need to update the message text */ uchar *tmpParse = parse; /* just to check correctness of TS */ - if(datetime.ParseTIMESTAMP3164(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK) { + if(datetime.ParseTIMESTAMP3339(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK || + datetime.ParseTIMESTAMP3164(&dummyTS, &tmpParse, &lenMsg) == RS_RET_OK) { /* We modify the message only if it contained a valid timestamp, * otherwise we do not touch it at all. */ datetime.formatTimestamp3164(&st, (char*)parse, 0); @@ -1001,8 +927,7 @@ SubmitMsg(uchar *pRcv, int lenRcv, lstn_t *pLstn, struct ucred *cred, struct tim MsgSetRcvFrom(pMsg, pLstn->hostName == NULL ? glbl.GetLocalHostNameProp() : pLstn->hostName); CHKiRet(MsgSetRcvFromIP(pMsg, pLocalHostIP)); - CHKiRet(submitMsg(pMsg)); - + ratelimitAddMsg(ratelimiter, NULL, pMsg); STATSCOUNTER_INC(ctrSubmit, mutCtrSubmit); finalize_it: RETiRet; @@ -1137,13 +1062,20 @@ activateListeners() listeners[0].ratelimitInterval = runModConf->ratelimitIntervalSysSock; listeners[0].ratelimitBurst = runModConf->ratelimitBurstSysSock; listeners[0].ratelimitSev = runModConf->ratelimitSeveritySysSock; - listeners[0].bUseCreds = (runModConf->bWritePidSysSock || runModConf->ratelimitIntervalSysSock || runModConf->bAnnotateSysSock) ? 1 : 0; + listeners[0].bUseCreds = (runModConf->bWritePidSysSock || runModConf->ratelimitIntervalSysSock || runModConf->bAnnotateSysSock || runModConf->bDiscardOwnMsgs) ? 1 : 0; listeners[0].bWritePid = runModConf->bWritePidSysSock; listeners[0].bAnnotate = runModConf->bAnnotateSysSock; listeners[0].bParseTrusted = runModConf->bParseTrusted; + listeners[0].bDiscardOwnMsgs = runModConf->bDiscardOwnMsgs; + listeners[0].bUnlink = runModConf->bUnlink; listeners[0].bUseSysTimeStamp = runModConf->bUseSysTimeStamp; listeners[0].flags = runModConf->bIgnoreTimestamp ? IGNDATE : NOFLAG; listeners[0].flowCtl = runModConf->bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; + CHKiRet(ratelimitNew(&listeners[0].dflt_ratelimiter, "imuxsock", NULL)); + ratelimitSetLinuxLike(listeners[0].dflt_ratelimiter, + listeners[0].ratelimitInterval, + listeners[0].ratelimitBurst); + ratelimitSetSeverity(listeners[0].dflt_ratelimiter,listeners[0].ratelimitSev); sd_fds = sd_listen_fds(0); if(sd_fds < 0) { @@ -1185,6 +1117,8 @@ CODESTARTbeginCnfLoad pModConf->bWritePidSysSock = 0; pModConf->bAnnotateSysSock = 0; pModConf->bParseTrusted = 0; + pModConf->bDiscardOwnMsgs = 1; + pModConf->bUnlink = 1; pModConf->ratelimitIntervalSysSock = DFLT_ratelimitInterval; pModConf->ratelimitBurstSysSock = DFLT_ratelimitBurst; pModConf->ratelimitSeveritySysSock = DFLT_ratelimitSeverity; @@ -1219,12 +1153,18 @@ CODESTARTsetModCnf loadModConf->pLogSockName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "syssock.ignoretimestamp")) { loadModConf->bIgnoreTimestamp = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "syssock.ignoreownmessages")) { + loadModConf->bDiscardOwnMsgs = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "syssock.unlink")) { + loadModConf->bUnlink = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.flowcontrol")) { loadModConf->bUseFlowCtl = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.usesystimestamp")) { loadModConf->bUseSysTimeStamp = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.annotate")) { loadModConf->bAnnotateSysSock = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "syssock.parsetrusted")) { + loadModConf->bParseTrusted = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.usepidfromsystem")) { loadModConf->bWritePidSysSock = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "syssock.ratelimit.interval")) { @@ -1279,6 +1219,10 @@ CODESTARTnewInpInst inst->bCreatePath = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "parsetrusted")) { inst->bParseTrusted = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "ignoreownmessages")) { + inst->bDiscardOwnMsgs = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "unlink")) { + inst->bUnlink = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "hostname")) { inst->pLogHostName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "ignoretimestamp")) { @@ -1318,6 +1262,9 @@ CODESTARTendCnfLoad loadModConf->bUseFlowCtl = cs.bUseFlowCtlSysSock; loadModConf->bAnnotateSysSock = cs.bAnnotateSysSock; loadModConf->bParseTrusted = cs.bParseTrusted; + loadModConf->ratelimitIntervalSysSock = cs.ratelimitIntervalSysSock; + loadModConf->ratelimitBurstSysSock = cs.ratelimitBurstSysSock; + loadModConf->ratelimitSeveritySysSock = cs.ratelimitSeveritySysSock; } loadModConf = NULL; /* done loading */ @@ -1455,16 +1402,14 @@ CODESTARTafterRun listeners[i].fd < SD_LISTEN_FDS_START + sd_fds) continue; - DBGPRINTF("imuxsock: unlinking unix socket file[%d] %s\n", i, listeners[i].sockName); - unlink((char*) listeners[i].sockName); + if(listeners[i].bUnlink) { + DBGPRINTF("imuxsock: unlinking unix socket file[%d] %s\n", i, listeners[i].sockName); + unlink((char*) listeners[i].sockName); + } } discardLogSockets(); nfd = 1; - if(ctxee != NULL) { - ee_exitCtx(ctxee); - ctxee = NULL; - } ENDafterRun @@ -1572,8 +1517,17 @@ CODEmodInit_QueryRegCFSLineHdlr listeners[0].bUseCreds = 0; listeners[0].bAnnotate = 0; listeners[0].bParseTrusted = 0; + listeners[0].bDiscardOwnMsgs = 1; + listeners[0].bUnlink = 1; listeners[0].bCreatePath = 0; listeners[0].bUseSysTimeStamp = 1; + if((listeners[0].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))ratelimitDestruct)) == NULL) { + /* in this case, we simply turn off rate-limiting */ + DBGPRINTF("imuxsock: turning off rate limiting for system socket " + "because we could not create hash table\n"); + listeners[0].ratelimitInterval = 0; + } /* initialize socket names */ for(i = 1 ; i < MAXFUNIX ; ++i) { diff --git a/plugins/imzmq3/README b/plugins/imzmq3/README index 88653b83..9a108a01 100644 --- a/plugins/imzmq3/README +++ b/plugins/imzmq3/README @@ -1,24 +1,59 @@ ZeroMQ 3.x Input Plugin Building this plugin: -Requires libzmq and libczmq. First, install libzmq from the HEAD on github: -http://github.com/zeromq/libzmq. You can clone the repository, build, then -install it. The directions for doing so are there in the readme. Then, do -the same for libczmq: http://github.com/zeromq/czmq. At some point, the 3.1 -version of libzmq will be released, and a supporting version of libczmq. -At that time, you could simply download and install the tarballs instead of -using git to clone the repositories. Those tarballs (when available) can -be found at http://download.zeromq.org. As of this writing (5/31/2012), the -most recent version of czmq (1.1.0) and libzmq (3.1.0-beta) will not compile -properly. +Requires libzmq and libczmq. First, download the tarballs of both libzmq +and its supporting libczmq from http://download.zeromq.org. As of this +writing (04/23/2013), the most recent versions of libzmq and czmq are +3.2.2 and 1.3.2 respectively. Configure, build, and then install both libs. Imzmq3 allows you to push data into rsyslog from a zeromq socket. The example below binds a SUB socket to port 7172, and then any messages with the topic "foo" will be pushed into rsyslog. +Please note: +This plugin only supports the newer (v7) config format. Legacy config support +was removed. + Example Rsyslog.conf snippet: ------------------------------------------------------------------------------- - -$InputZmq3ServerRun action=BIND,type=SUB,description=tcp://*:7172,subscribe=foo +module(load="imzmq3" ioThreads="1") +input(type="imzmq3" action="CONNECT" socktype="SUB" description="tcp://*:7172" subscribe="foo,bar") ------------------------------------------------------------------------------- +Note you can specify multiple subscriptions with a comma-delimited list, with +no spaces between values. + +The only global parameter for this plugin is ioThreads, which is optional and +probably best left to the zmq default unless you know exactly what you are +doing. + +The instance-level parameters are: + +Required +description +subscribe (required if the sockType is SUB) + +Optional +sockType (defaults to SUB) +action (defaults to BIND +sndHWM +rcvHWM +identity +sndBuf +rcvBuf +linger +backlog +sndTimeout +rcvTimeout +maxMsgSize +rate +recoveryIVL +multicastHops +reconnectIVL +reconnectIVLMax +ipv4Only +affinity + +These all correspond to zmq optional settings. Except where noted, the defaults +are the zmq defaults if not set. See http://api.zeromq.org/3-2:zmq-setsockopt +for info on these. diff --git a/plugins/imzmq3/imzmq3.c b/plugins/imzmq3/imzmq3.c index dc1d64d3..08b1dbe4 100644 --- a/plugins/imzmq3/imzmq3.c +++ b/plugins/imzmq3/imzmq3.c @@ -19,20 +19,21 @@ * License along with this program. If not, see * <http://www.gnu.org/licenses/>. * - * Author: David Kelly - * <davidk@talksum.com> + * Authors: + * David Kelly <davidk@talksum.com> + * Hongfei Cheng <hongfeic@talksum.com> */ + +#include "config.h" +#include "rsyslog.h" + #include <assert.h> #include <errno.h> #include <stdlib.h> #include <string.h> #include <unistd.h> - -#include "rsyslog.h" - #include "cfsysline.h" -#include "config.h" #include "dirty.h" #include "errmsg.h" #include "glbl.h" @@ -49,6 +50,7 @@ MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP +MODULE_CNFNAME("imzmq3"); /* convienent symbols to denote a socket we want to bind * vs one we want to just connect to @@ -83,47 +85,67 @@ typedef struct _poller_data { thrdInfo_t* thread; } poller_data; -typedef struct _socket_info { - int type; - int action; - char* description; - int sndHWM; /* if you want more than 2^32 messages, */ - int rcvHWM; /* then pass in 0 (the default). */ - char* identity; - char** subscriptions; - ruleset_t* ruleset; - int sndBuf; - int rcvBuf; - int linger; - int backlog; - int sndTimeout; - int rcvTimeout; - int maxMsgSize; - int rate; - int recoveryIVL; - int multicastHops; - int reconnectIVL; - int reconnectIVLMax; - int ipv4Only; - int affinity; - -} socket_info; +/* a linked-list of subscription topics */ +typedef struct sublist_t { + char* subscribe; + struct sublist_t* next; +} sublist; + +struct instanceConf_s { + int type; + int action; + char* description; + int sndHWM; /* if you want more than 2^32 messages, */ + int rcvHWM; /* then pass in 0 (the default). */ + char* identity; + sublist* subscriptions; + int sndBuf; + int rcvBuf; + int linger; + int backlog; + int sndTimeout; + int rcvTimeout; + int maxMsgSize; + int rate; + int recoveryIVL; + int multicastHops; + int reconnectIVL; + int reconnectIVLMax; + int ipv4Only; + int affinity; + uchar* pszBindRuleset; + ruleset_t* pBindRuleset; + struct instanceConf_s* next; + +}; + +struct modConfData_s { + rsconf_t* pConf; + instanceConf_t* root; + instanceConf_t* tail; + int io_threads; +}; +struct lstn_s { + struct lstn_s* next; + void* sock; + ruleset_t* pRuleset; +}; /* ---------------------------------------------------------------------------- * Static definitions/initializations. */ -static socket_info* s_socketInfo = NULL; -static size_t s_nitems = 0; -static prop_t * s_namep = NULL; +static modConfData_t* runModConf = NULL; +static struct lstn_s* lcnfRoot = NULL; +static struct lstn_s* lcnfLast = NULL; +static prop_t* s_namep = NULL; static zloop_t* s_zloop = NULL; -static int s_io_threads = 1; static zctx_t* s_context = NULL; -static ruleset_t* s_ruleset = NULL; static socket_type socketTypes[] = { - {"SUB", ZMQ_SUB }, - {"PULL", ZMQ_PULL }, - {"XSUB", ZMQ_XSUB } + {"SUB", ZMQ_SUB }, + {"PULL", ZMQ_PULL }, + {"ROUTER", ZMQ_ROUTER }, + {"XSUB", ZMQ_XSUB } }; static socket_action socketActions[] = { @@ -131,6 +153,48 @@ static socket_action socketActions[] = { {"CONNECT", ACTION_CONNECT}, }; +static struct cnfparamdescr modpdescr[] = { + { "ioThreads", eCmdHdlrInt, 0 }, +}; + +static struct cnfparamblk modpblk = { + CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr +}; + +static struct cnfparamdescr inppdescr[] = { + { "description", eCmdHdlrGetWord, 0 }, + { "sockType", eCmdHdlrGetWord, 0 }, + { "subscribe", eCmdHdlrGetWord, 0 }, + { "ruleset", eCmdHdlrGetWord, 0 }, + { "action", eCmdHdlrGetWord, 0 }, + { "sndHWM", eCmdHdlrInt, 0 }, + { "rcvHWM", eCmdHdlrInt, 0 }, + { "identity", eCmdHdlrGetWord, 0 }, + { "sndBuf", eCmdHdlrInt, 0 }, + { "rcvBuf", eCmdHdlrInt, 0 }, + { "linger", eCmdHdlrInt, 0 }, + { "backlog", eCmdHdlrInt, 0 }, + { "sndTimeout", eCmdHdlrInt, 0 }, + { "rcvTimeout", eCmdHdlrInt, 0 }, + { "maxMsgSize", eCmdHdlrInt, 0 }, + { "rate", eCmdHdlrInt, 0 }, + { "recoveryIVL", eCmdHdlrInt, 0 }, + { "multicastHops", eCmdHdlrInt, 0 }, + { "reconnectIVL", eCmdHdlrInt, 0 }, + { "reconnectIVLMax", eCmdHdlrInt, 0 }, + { "ipv4Only", eCmdHdlrInt, 0 }, + { "affinity", eCmdHdlrInt, 0 } +}; + +static struct cnfparamblk inppblk = { + CNFPARAMBLK_VERSION, + sizeof(inppdescr)/sizeof(struct cnfparamdescr), + inppdescr +}; + +#include "im-helper.h" /* must be included AFTER the type definitions! */ /* ---------------------------------------------------------------------------- * Helper functions @@ -179,15 +243,16 @@ static int getSocketAction(char* name) { } -static void setDefaults(socket_info* info) { - info->type = ZMQ_SUB; - info->action = ACTION_BIND; +static void setDefaults(instanceConf_t* info) { + info->type = -1; + info->action = -1; info->description = NULL; - info->sndHWM = 0; - info->rcvHWM = 0; + info->sndHWM = -1; + info->rcvHWM = -1; info->identity = NULL; info->subscriptions = NULL; - info->ruleset = NULL; + info->pszBindRuleset = NULL; + info->pBindRuleset = NULL; info->sndBuf = -1; info->rcvBuf = -1; info->linger = -1; @@ -202,93 +267,49 @@ static void setDefaults(socket_info* info) { info->reconnectIVLMax = -1; info->ipv4Only = -1; info->affinity = -1; - + info->next = NULL; }; - -/* The config string should look like: - * "action=AAA,type=TTT,description=DDD,sndHWM=SSS,rcvHWM=RRR,subscribe='xxx',subscribe='yyy'" - * +/* given a comma separated list of subscriptions, create a char* array of them + * to set later */ -static rsRetVal parseConfig(char* config, socket_info* info) { - int nsubs = 0; - - char* binding; - char* ptr1; - for (binding = strtok_r(config, ",", &ptr1); - binding != NULL; - binding = strtok_r(NULL, ",", &ptr1)) { - - /* Each binding looks like foo=bar */ - char * sep = strchr(binding, '='); - if (sep == NULL) - { - errmsg.LogError(0, NO_ERRCODE, - "Invalid argument format %s, ignoring ...", - binding); - continue; - } +static rsRetVal parseSubscriptions(char* subscribes, sublist** subList){ + char* tok = strtok(subscribes, ","); + sublist* currentSub; + sublist* head; + DEFiRet; - /* Replace '=' with '\0'. */ - *sep = '\0'; - - char * val = sep + 1; - - if (strcmp(binding, "action") == 0) { - info->action = getSocketAction(val); - } else if (strcmp(binding, "type") == 0) { - info->type = getSocketType(val); - } else if (strcmp(binding, "description") == 0) { - info->description = strdup(val); - } else if (strcmp(binding, "sndHWM") == 0) { - info->sndHWM = atoi(val); - } else if (strcmp(binding, "rcvHWM") == 0) { - info->sndHWM = atoi(val); - } else if (strcmp(binding, "subscribe") == 0) { - /* Add the subscription value to the list.*/ - char * substr = NULL; - substr = strdup(val); - info->subscriptions = realloc(info->subscriptions, sizeof(char *) * nsubs + 1); - info->subscriptions[nsubs] = substr; - ++nsubs; - } else if (strcmp(binding, "sndBuf") == 0) { - info->sndBuf = atoi(val); - } else if (strcmp(binding, "rcvBuf") == 0) { - info->rcvBuf = atoi(val); - } else if (strcmp(binding, "linger") == 0) { - info->linger = atoi(val); - } else if (strcmp(binding, "backlog") == 0) { - info->backlog = atoi(val); - } else if (strcmp(binding, "sndTimeout") == 0) { - info->sndTimeout = atoi(val); - } else if (strcmp(binding, "rcvTimeout") == 0) { - info->rcvTimeout = atoi(val); - } else if (strcmp(binding, "maxMsgSize") == 0) { - info->maxMsgSize = atoi(val); - } else if (strcmp(binding, "rate") == 0) { - info->rate = atoi(val); - } else if (strcmp(binding, "recoveryIVL") == 0) { - info->recoveryIVL = atoi(val); - } else if (strcmp(binding, "multicastHops") == 0) { - info->multicastHops = atoi(val); - } else if (strcmp(binding, "reconnectIVL") == 0) { - info->reconnectIVL = atoi(val); - } else if (strcmp(binding, "reconnectIVLMax") == 0) { - info->reconnectIVLMax = atoi(val); - } else if (strcmp(binding, "ipv4Only") == 0) { - info->ipv4Only = atoi(val); - } else if (strcmp(binding, "affinity") == 0) { - info->affinity = atoi(val); - } else { - errmsg.LogError(0, NO_ERRCODE, "Unknown argument %s", binding); - return RS_RET_INVALID_PARAMS; + /* create empty list */ + CHKmalloc(*subList = (sublist*)MALLOC(sizeof(sublist))); + head = *subList; + head->next = NULL; + head->subscribe=NULL; + currentSub=head; + + if(tok) { + head->subscribe=strdup(tok); + for(tok=strtok(NULL, ","); tok!=NULL;tok=strtok(NULL, ",")) { + CHKmalloc(currentSub->next = (sublist*)MALLOC(sizeof(sublist))); + currentSub=currentSub->next; + currentSub->subscribe=strdup(tok); + currentSub->next=NULL; } + } else { + /* make empty subscription ie subscribe="" */ + head->subscribe=strdup(""); } - - return RS_RET_OK; + /* TODO: temporary logging */ + currentSub = head; + DBGPRINTF("imzmq3: Subscriptions:"); + for(currentSub = head; currentSub != NULL; currentSub=currentSub->next) { + DBGPRINTF("'%s'", currentSub->subscribe); + } + DBGPRINTF("\n"); +finalize_it: + RETiRet; } -static rsRetVal validateConfig(socket_info* info) { +static rsRetVal validateConfig(instanceConf_t* info) { if (info->type == -1) { errmsg.LogError(0, RS_RET_INVALID_PARAMS, @@ -307,7 +328,7 @@ static rsRetVal validateConfig(socket_info* info) { } if(info->type == ZMQ_SUB && info->subscriptions == NULL) { errmsg.LogError(0, RS_RET_INVALID_PARAMS, - "SUB sockets need at least one subscription"); + "SUB sockets need a subscription"); return RS_RET_INVALID_PARAMS; } if(info->type != ZMQ_SUB && info->subscriptions != NULL) { @@ -320,39 +341,40 @@ static rsRetVal validateConfig(socket_info* info) { static rsRetVal createContext() { if (s_context == NULL) { - errmsg.LogError(0, NO_ERRCODE, "creating zctx."); + DBGPRINTF("imzmq3: creating zctx..."); + zsys_handler_set(NULL); s_context = zctx_new(); if (s_context == NULL) { errmsg.LogError(0, RS_RET_INVALID_PARAMS, "zctx_new failed: %s", - strerror(errno)); + zmq_strerror(errno)); /* DK: really should do better than invalid params...*/ return RS_RET_INVALID_PARAMS; } - - if (s_io_threads > 1) { - errmsg.LogError(0, NO_ERRCODE, "setting io worker threads to %d", s_io_threads); - zctx_set_iothreads(s_context, s_io_threads); + DBGPRINTF("success!\n"); + if (runModConf->io_threads > 1) { + DBGPRINTF("setting io worker threads to %d\n", runModConf->io_threads); + zctx_set_iothreads(s_context, runModConf->io_threads); } } return RS_RET_OK; } -static rsRetVal createSocket(socket_info* info, void** sock) { - size_t ii; +static rsRetVal createSocket(instanceConf_t* info, void** sock) { int rv; + sublist* sub; *sock = zsocket_new(s_context, info->type); if (!sock) { - errmsg.LogError(0, + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "zsocket_new failed: %s, for type %d", - strerror(errno),info->type); - /* DK: invalid params seems right here */ + zmq_strerror(errno),info->type); + /* DK: invalid params seems right here */ return RS_RET_INVALID_PARAMS; } - + DBGPRINTF("imzmq3: socket of type %d created successfully\n", info->type) /* Set options *before* the connect/bind. */ if (info->identity) zsocket_set_identity(*sock, info->identity); if (info->sndBuf > -1) zsocket_set_sndbuf(*sock, info->sndBuf); @@ -369,36 +391,36 @@ static rsRetVal createSocket(socket_info* info, void** sock) { if (info->reconnectIVLMax > -1) zsocket_set_reconnect_ivl_max(*sock, info->reconnectIVLMax); if (info->ipv4Only > -1) zsocket_set_ipv4only(*sock, info->ipv4Only); if (info->affinity > -1) zsocket_set_affinity(*sock, info->affinity); - - /* since HWM have defaults, we always set them. No return codes to check, either.*/ - zsocket_set_sndhwm(*sock, info->sndHWM); - zsocket_set_rcvhwm(*sock, info->rcvHWM); - + if (info->sndHWM > -1 ) zsocket_set_sndhwm(*sock, info->sndHWM); + if (info->rcvHWM > -1 ) zsocket_set_rcvhwm(*sock, info->rcvHWM); /* Set subscriptions.*/ - for (ii = 0; ii < sizeof(info->subscriptions)/sizeof(char*); ++ii) - zsocket_set_subscribe(*sock, info->subscriptions[ii]); - - + if (info->type == ZMQ_SUB) { + for(sub = info->subscriptions; sub!=NULL; sub=sub->next) { + zsocket_set_subscribe(*sock, sub->subscribe); + } + } /* Do the bind/connect... */ if (info->action==ACTION_CONNECT) { rv = zsocket_connect(*sock, info->description); - if (rv < 0) { + if (rv == -1) { errmsg.LogError(0, RS_RET_INVALID_PARAMS, "zmq_connect using %s failed: %s", - info->description, strerror(errno)); + info->description, zmq_strerror(errno)); return RS_RET_INVALID_PARAMS; } + DBGPRINTF("imzmq3: connect for %s successful\n",info->description); } else { rv = zsocket_bind(*sock, info->description); - if (rv <= 0) { + if (rv == -1) { errmsg.LogError(0, RS_RET_INVALID_PARAMS, "zmq_bind using %s failed: %s", - info->description, strerror(errno)); + info->description, zmq_strerror(errno)); return RS_RET_INVALID_PARAMS; } + DBGPRINTF("imzmq3: bind for %s successful\n",info->description); } return RS_RET_OK; } @@ -407,89 +429,138 @@ static rsRetVal createSocket(socket_info* info, void** sock) { * Module endpoints */ -/* accept a new ruleset to bind. Checks if it exists and complains, if not. Note - * that this makes the assumption that after the bind ruleset is called in the config, - * another call will be made to add an endpoint. -*/ -static rsRetVal -set_ruleset(void __attribute__((unused)) *pVal, uchar *pszName) { - ruleset_t* ruleset_ptr; - rsRetVal localRet; - DEFiRet; - - localRet = ruleset.GetRuleset(ourConf, &ruleset_ptr, pszName); - if(localRet == RS_RET_NOT_FOUND) { - errmsg.LogError(0, NO_ERRCODE, "error: " - "ruleset '%s' not found - ignored", pszName); - } - CHKiRet(localRet); - s_ruleset = ruleset_ptr; - DBGPRINTF("imzmq3 current bind ruleset '%s'\n", pszName); - -finalize_it: - free(pszName); /* no longer needed */ - RETiRet; -} /* add an actual endpoint */ -static rsRetVal add_endpoint(void __attribute__((unused)) * oldp, uchar * valp) { +static rsRetVal createInstance(instanceConf_t** pinst) { DEFiRet; + instanceConf_t* inst; + CHKmalloc(inst = MALLOC(sizeof(instanceConf_t))); - /* increment number of items and store old num items, as it will be handy.*/ - size_t idx = s_nitems++; - - /* allocate a new socket_info array to accomidate this new endpoint*/ - socket_info* tmpSocketInfo; - CHKmalloc(tmpSocketInfo = (socket_info*)MALLOC(sizeof(socket_info) * s_nitems)); + /* set defaults into new instance config struct */ + setDefaults(inst); - /* copy existing socket_info across into new array, if any, and free old storage*/ - if(idx) { - memcpy(tmpSocketInfo, s_socketInfo, sizeof(socket_info) * idx); - free(s_socketInfo); + /* add this to the config */ + if (runModConf->root == NULL || runModConf->tail == NULL) { + runModConf->tail = runModConf->root = inst; + } else { + runModConf->tail->next = inst; + runModConf->tail = inst; } + *pinst = inst; +finalize_it: + RETiRet; +} - /* set the static to hold the new array */ - s_socketInfo = tmpSocketInfo; - - /* point to the new one */ - socket_info* sockInfo = &s_socketInfo[idx]; - - /* set defaults for the new socket info */ - setDefaults(sockInfo); - - /* Make a writeable copy of the string so we can use strtok - in the parseConfig call */ - char * copy = NULL; - CHKmalloc(copy = strdup((char *) valp)); +static rsRetVal createListener(struct cnfparamvals* pvals) { + instanceConf_t* inst; + int i; + DEFiRet; - /* parse the config string */ - CHKiRet(parseConfig(copy, sockInfo)); + CHKiRet(createInstance(&inst)); + for(i = 0 ; i < inppblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(inppblk.descr[i].name, "ruleset")) { + inst->pszBindRuleset = (uchar *)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "description")) { + inst->description = es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "sockType")){ + inst->type = getSocketType(es_str2cstr(pvals[i].val.d.estr, NULL)); + } else if(!strcmp(inppblk.descr[i].name, "action")){ + inst->action = getSocketAction(es_str2cstr(pvals[i].val.d.estr, NULL)); + } else if(!strcmp(inppblk.descr[i].name, "sndHWM")) { + inst->sndHWM = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "rcvHWM")) { + inst->rcvHWM = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "subscribe")) { + CHKiRet(parseSubscriptions(es_str2cstr(pvals[i].val.d.estr, NULL), + &inst->subscriptions)); + } else if(!strcmp(inppblk.descr[i].name, "identity")){ + inst->identity = es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "sndBuf")) { + inst->sndBuf = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "rcvBuf")) { + inst->rcvBuf = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "linger")) { + inst->linger = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "backlog")) { + inst->backlog = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "sndTimeout")) { + inst->sndTimeout = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "rcvTimeout")) { + inst->rcvTimeout = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "maxMsgSize")) { + inst->maxMsgSize = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "rate")) { + inst->rate = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "recoveryIVL")) { + inst->recoveryIVL = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "multicastHops")) { + inst->multicastHops = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "reconnectIVL")) { + inst->reconnectIVL = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "reconnectIVLMax")) { + inst->reconnectIVLMax = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "ipv4Only")) { + inst->ipv4Only = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "affinity")) { + inst->affinity = (int) pvals[i].val.d.n; + } else { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: program error, non-handled " + "param '%s'\n", inppblk.descr[i].name); + } - /* validate it */ - CHKiRet(validateConfig(sockInfo)); + } +finalize_it: + RETiRet; +} + +static rsRetVal addListener(instanceConf_t* inst){ + /* create the socket */ + void* sock; + struct lstn_s* newcnfinfo; + DEFiRet; - /* bind to the current ruleset (if any)*/ - sockInfo->ruleset = s_ruleset; + CHKiRet(createSocket(inst, &sock)); + + /* now create new lstn_s struct */ + CHKmalloc(newcnfinfo=(struct lstn_s*)MALLOC(sizeof(struct lstn_s))); + newcnfinfo->next = NULL; + newcnfinfo->sock = sock; + newcnfinfo->pRuleset = inst->pBindRuleset; + /* add this struct to the global */ + if(lcnfRoot == NULL) { + lcnfRoot = newcnfinfo; + } + if(lcnfLast == NULL) { + lcnfLast = newcnfinfo; + } else { + lcnfLast->next = newcnfinfo; + lcnfLast = newcnfinfo; + } + finalize_it: - free(valp); /* in any case, this is no longer needed */ - RETiRet; + RETiRet; } - static int handlePoll(zloop_t __attribute__((unused)) * loop, zmq_pollitem_t *poller, void* pd) { - msg_t* logmsg; + msg_t* pMsg; poller_data* pollerData = (poller_data*)pd; char* buf = zstr_recv(poller->socket); - if (msgConstruct(&logmsg) == RS_RET_OK) { - MsgSetRawMsg(logmsg, buf, strlen(buf)); - MsgSetInputName(logmsg, s_namep); - MsgSetFlowControlType(logmsg, eFLOWCTL_NO_DELAY); - MsgSetRuleset(logmsg, pollerData->ruleset); - logmsg->msgFlags = NEEDS_PARSING; - submitMsg(logmsg); + if (msgConstruct(&pMsg) == RS_RET_OK) { + MsgSetRawMsg(pMsg, buf, strlen(buf)); + MsgSetInputName(pMsg, s_namep); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, glbl.GetLocalHostIP()); + MsgSetMSGoffs(pMsg, 0); + MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); + MsgSetRuleset(pMsg, pollerData->ruleset); + pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; + submitMsg2(pMsg); } /* gotta free the string returned from zstr_recv() */ @@ -508,51 +579,65 @@ static int handlePoll(zloop_t __attribute__((unused)) * loop, zmq_pollitem_t *po /* called when runInput is called by rsyslog */ static rsRetVal rcv_loop(thrdInfo_t* pThrd){ + size_t n_items = 0; size_t i; int rv; - zmq_pollitem_t* items; - poller_data* pollerData; - + zmq_pollitem_t* items = NULL; + poller_data* pollerData = NULL; + struct lstn_s* current; + instanceConf_t* inst; DEFiRet; - - /* create the context*/ - CHKiRet(createContext()); + /* now add listeners. This actually creates the sockets, etc... */ + for (inst = runModConf->root; inst != NULL; inst=inst->next) { + addListener(inst); + } + if (lcnfRoot == NULL) { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: no listeners were " + "started, input not activated.\n"); + ABORT_FINALIZE(RS_RET_NO_RUN); + } + + /* count the # of items first */ + for(current=lcnfRoot;current!=NULL;current=current->next) + n_items++; + + /* make arrays of pollitems, pollerdata so they are easy to delete later */ + /* create the poll items*/ - CHKmalloc(items = (zmq_pollitem_t*)MALLOC(sizeof(zmq_pollitem_t)*s_nitems)); + CHKmalloc(items = (zmq_pollitem_t*)MALLOC(sizeof(zmq_pollitem_t)*n_items)); /* create poller data (stuff to pass into the zmq closure called when we get a message)*/ - CHKmalloc(pollerData = (poller_data*)MALLOC(sizeof(poller_data)*s_nitems)); + CHKmalloc(pollerData = (poller_data*)MALLOC(sizeof(poller_data)*n_items)); /* loop through and initialize the poll items and poller_data arrays...*/ - for(i=0; i<s_nitems;++i) { + for(i=0, current = lcnfRoot; current != NULL; current = current->next, i++) { /* create the socket, update items.*/ - createSocket(&s_socketInfo[i], &items[i].socket); + items[i].socket=current->sock; items[i].events = ZMQ_POLLIN; /* now update the poller_data for this item */ pollerData[i].thread = pThrd; - pollerData[i].ruleset = s_socketInfo[i].ruleset; + pollerData[i].ruleset = current->pRuleset; } - + s_zloop = zloop_new(); - for(i=0; i<s_nitems; ++i) { + for(i=0; i<n_items; ++i) { rv = zloop_poller(s_zloop, &items[i], handlePoll, &pollerData[i]); if (rv) { - errmsg.LogError(0, NO_ERRCODE, "imzmq3: zloop_poller failed for item %zu", i); + errmsg.LogError(0, NO_ERRCODE, "imzmq3: zloop_poller failed for item %zu: %s", i, zmq_strerror(errno)); } } + DBGPRINTF("imzmq3: zloop_poller starting..."); zloop_start(s_zloop); zloop_destroy(&s_zloop); - finalize_it: - for(i=0; i< s_nitems; ++i) { - zsocket_destroy(s_context, items[i].socket); - } - + DBGPRINTF("imzmq3: zloop_poller stopped."); +finalize_it: zctx_destroy(&s_context); free(items); + free(pollerData); RETiRet; } @@ -562,7 +647,8 @@ static rsRetVal rcv_loop(thrdInfo_t* pThrd){ BEGINrunInput CODESTARTrunInput - iRet = rcv_loop(pThrd); + CHKiRet(rcv_loop(pThrd)); +finalize_it: RETiRet; ENDrunInput @@ -570,17 +656,13 @@ ENDrunInput /* initialize and return if will run or not */ BEGINwillRun CODESTARTwillRun - /* we need to create the inputName property (only once during our + /* we need to create the inputName property (only once during our lifetime) */ - CHKiRet(prop.Construct(&s_namep)); - CHKiRet(prop.SetString(s_namep, + CHKiRet(prop.Construct(&s_namep)); + CHKiRet(prop.SetString(s_namep, UCHAR_CONSTANT("imzmq3"), sizeof("imzmq3") - 1)); - CHKiRet(prop.ConstructFinalize(s_namep)); - -/* If there are no endpoints this is pointless ...*/ - if (s_nitems == 0) - ABORT_FINALIZE(RS_RET_NO_RUN); + CHKiRet(prop.ConstructFinalize(s_namep)); finalize_it: ENDwillRun @@ -588,70 +670,207 @@ ENDwillRun BEGINafterRun CODESTARTafterRun - /* do cleanup here */ - if(s_namep != NULL) - prop.Destruct(&s_namep); + /* do cleanup here */ + if (s_namep != NULL) + prop.Destruct(&s_namep); ENDafterRun BEGINmodExit CODESTARTmodExit - /* release what we no longer need */ - objRelease(errmsg, CORE_COMPONENT); - objRelease(glbl, CORE_COMPONENT); - objRelease(prop, CORE_COMPONENT); + /* release what we no longer need */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); ENDmodExit BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature - if(eFeat == sFEATURENonCancelInputTermination) - iRet = RS_RET_OK; + if (eFeat == sFEATURENonCancelInputTermination) + iRet = RS_RET_OK; ENDisCompatibleWithFeature +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + /* After endCnfLoad() (BEGINendCnfLoad...ENDendCnfLoad) is called, + * the pModConf pointer must not be used to change the in-memory + * config object. It's safe to use the same pointer for accessing + * the config object until freeCnf() (BEGINfreeCnf...ENDfreeCnf). */ + runModConf = pModConf; + runModConf->pConf = pConf; + /* init module config */ + runModConf->io_threads = 0; /* 0 means don't set it */ +ENDbeginCnfLoad + + +BEGINsetModCnf + struct cnfparamvals* pvals = NULL; + int i; +CODESTARTsetModCnf + pvals = nvlstGetParams(lst, &modpblk, NULL); + if (NULL == pvals) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "imzmq3: error processing module " + " config parameters ['module(...)']"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + for (i=0; i < modpblk.nParams; ++i) { + if (!pvals[i].bUsed) + continue; + if (!strcmp(modpblk.descr[i].name, "ioThreads")) { + runModConf->io_threads = (int)pvals[i].val.d.n; + } else { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, + "imzmq3: config error, unknown " + "param %s in setModCnf\n", + modpblk.descr[i].name); + } + } + +finalize_it: + if (pvals != NULL) + cnfparamvalsDestruct(pvals, &modpblk); +ENDsetModCnf + + +BEGINendCnfLoad +CODESTARTendCnfLoad + /* Last chance to make changes to the in-memory config object for this + * input module. After this call, the config object must no longer be + * changed. */ + if (pModConf != runModConf) { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: pointer of in-memory config object has " + "changed - pModConf=%p, runModConf=%p", pModConf, runModConf); + } + assert(pModConf == runModConf); +ENDendCnfLoad + + +/* function to generate error message if framework does not find requested ruleset */ +static inline void +std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, instanceConf_t *inst) +{ + errmsg.LogError(0, NO_ERRCODE, "imzmq3: ruleset '%s' for socket %s not found - " + "using default ruleset instead", inst->pszBindRuleset, + inst->description); +} + + +BEGINcheckCnf +instanceConf_t* inst; +CODESTARTcheckCnf + for(inst = pModConf->root; inst!=NULL; inst=inst->next) { + std_checkRuleset(pModConf, inst); + /* now, validate the instanceConf */ + CHKiRet(validateConfig(inst)); + } +finalize_it: + RETiRet; +ENDcheckCnf + + +BEGINactivateCnfPrePrivDrop +CODESTARTactivateCnfPrePrivDrop + if (pModConf != runModConf) { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: pointer of in-memory config object has " + "changed - pModConf=%p, runModConf=%p", pModConf, runModConf); + } + assert(pModConf == runModConf); + + /* first create the context */ + createContext(); + + /* could setup context here, and set the global worker threads + and so on... */ +ENDactivateCnfPrePrivDrop + + +BEGINactivateCnf +CODESTARTactivateCnf + if (pModConf != runModConf) { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: pointer of in-memory config object has " + "changed - pModConf=%p, runModConf=%p", pModConf, runModConf); + } + assert(pModConf == runModConf); +ENDactivateCnf + + +BEGINfreeCnf + struct lstn_s *lstn, *lstn_r; + instanceConf_t *inst, *inst_r; + sublist *sub, *sub_r; +CODESTARTfreeCnf + DBGPRINTF("imzmq3: BEGINfreeCnf ...\n"); + if (pModConf != runModConf) { + errmsg.LogError(0, NO_ERRCODE, "imzmq3: pointer of in-memory config object has " + "changed - pModConf=%p, runModConf=%p", pModConf, runModConf); + } + for (lstn = lcnfRoot; lstn != NULL; ) { + lstn_r = lstn; + lstn = lstn_r->next; + free(lstn_r); + } + for (inst = pModConf->root ; inst != NULL ; ) { + for (sub = inst->subscriptions; sub != NULL; ) { + free(sub->subscribe); + sub_r = sub; + sub = sub_r->next; + free(sub_r); + } + free(inst->pszBindRuleset); + inst_r = inst; + inst = inst->next; + free(inst_r); + } +ENDfreeCnf + + +BEGINnewInpInst + struct cnfparamvals* pvals; +CODESTARTnewInpInst + + DBGPRINTF("newInpInst (imzmq3)\n"); + pvals = nvlstGetParams(lst, &inppblk, NULL); + if(NULL==pvals) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, + "imzmq3: required parameters are missing\n"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + DBGPRINTF("imzmq3: input param blk:\n"); + cnfparamsPrint(&inppblk, pvals); + + /* now, parse the config params and so on... */ + CHKiRet(createListener(pvals)); + +finalize_it: +CODE_STD_FINALIZERnewInpInst + cnfparamvalsDestruct(pvals, &inppblk); +ENDnewInpInst + + BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_IMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES +CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES +CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES ENDqueryEtryPt -static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, - void __attribute__((unused)) *pVal) { - return RS_RET_OK; -} -static rsRetVal setGlobalWorkerThreads(uchar __attribute__((unused)) *pp, int val) { - errmsg.LogError(0, NO_ERRCODE, "setGlobalWorkerThreads called with %d",val); - s_io_threads = val; - return RS_RET_OK; -} BEGINmodInit() CODESTARTmodInit /* we only support the current interface specification */ - *ipIFVersProvided = CURR_MOD_IF_VERSION; + *ipIFVersProvided = CURR_MOD_IF_VERSION; CODEmodInit_QueryRegCFSLineHdlr - CHKiRet(objUse(errmsg, CORE_COMPONENT)); - CHKiRet(objUse(glbl, CORE_COMPONENT)); - CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(ruleset, CORE_COMPONENT)); - - /* register config file handlers */ - CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputzmq3serverbindruleset", - 0, eCmdHdlrGetWord, - set_ruleset, NULL, - STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputzmq3serverrun", - 0, eCmdHdlrGetWord, - add_endpoint, NULL, - STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", - 1, eCmdHdlrCustomHandler, - resetConfigVariables, NULL, - STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputzmq3globalWorkerThreads", - 1, eCmdHdlrInt, - setGlobalWorkerThreads, NULL, - STD_LOADABLE_MODULE_ID)); ENDmodInit + + diff --git a/plugins/mmanon/Makefile.am b/plugins/mmanon/Makefile.am new file mode 100644 index 00000000..98f0da24 --- /dev/null +++ b/plugins/mmanon/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmanon.la + +mmanon_la_SOURCES = mmanon.c +mmanon_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmanon_la_LDFLAGS = -module -avoid-version +mmanon_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmanon/mmanon.c b/plugins/mmanon/mmanon.c new file mode 100644 index 00000000..a1c99d09 --- /dev/null +++ b/plugins/mmanon/mmanon.c @@ -0,0 +1,401 @@ +/* mmanon.c + * anonnymize IP addresses inside the syslog message part + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmanon") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +/* precomputed table of IPv4 anonymization masks */ +static const uint32_t ipv4masks[33] = { + 0xffffffff, 0xfffffffe, 0xfffffffc, 0xfffffff8, + 0xfffffff0, 0xffffffe0, 0xffffffc0, 0xffffff80, + 0xffffff00, 0xfffffe00, 0xfffffc00, 0xfffff800, + 0xfffff000, 0xffffe000, 0xffffc000, 0xffff8000, + 0xffff0000, 0xfffe0000, 0xfffc0000, 0xfff80000, + 0xfff00000, 0xffe00000, 0xffc00000, 0xff800000, + 0xff000000, 0xfe000000, 0xfc000000, 0xf8000000, + 0xf0000000, 0xe0000000, 0xc0000000, 0x80000000, + 0x00000000 + }; + +/* define operation modes we have */ +#define SIMPLE_MODE 0 /* just overwrite */ +#define REWRITE_MODE 1 /* rewrite IP address, canoninized */ +typedef struct _instanceData { + char replChar; + int8_t mode; + struct { + int8_t bits; + } ipv4; +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "mode", eCmdHdlrGetWord, 0 }, + { "replacementchar", eCmdHdlrGetChar, 0 }, + { "ipv4.bits", eCmdHdlrInt, 0 }, +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->mode = REWRITE_MODE; + pData->replChar = 'x'; + pData->ipv4.bits = 16; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; + sbool bHadBitsErr; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmanon)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "mode")) { + if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"simple", + sizeof("simple")-1)) { + pData->mode = SIMPLE_MODE; + } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"rewrite", + sizeof("rewrite")-1)) { + pData->mode = REWRITE_MODE; + } else { + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_INVLD_MODE, + "mmanon: invalid anonymization mode '%s' - ignored", + cstr); + free(cstr); + } + pData->replChar = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else if(!strcmp(actpblk.descr[i].name, "replacementchar")) { + pData->replChar = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else if(!strcmp(actpblk.descr[i].name, "ipv4.bits")) { + pData->ipv4.bits = (int8_t) pvals[i].val.d.n; + } else { + dbgprintf("mmanon: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + + if(pData->mode == SIMPLE_MODE) { + bHadBitsErr = 0; + if(pData->ipv4.bits < 8) { + pData->ipv4.bits = 8; + bHadBitsErr = 1; + } else if(pData->ipv4.bits < 16) { + pData->ipv4.bits = 16; + bHadBitsErr = 1; + } else if(pData->ipv4.bits < 24) { + pData->ipv4.bits = 24; + bHadBitsErr = 1; + } else if(pData->ipv4.bits != 32) { + pData->ipv4.bits = 32; + bHadBitsErr = 1; + } + if(bHadBitsErr) + errmsg.LogError(0, RS_RET_INVLD_ANON_BITS, + "mmanon: invalid number of ipv4 bits " + "in simple mode, corrected to %d", + pData->ipv4.bits); + } else { /* REWRITE_MODE */ + if(pData->ipv4.bits < 1 || pData->ipv4.bits > 32) { + pData->ipv4.bits = 32; + errmsg.LogError(0, RS_RET_INVLD_ANON_BITS, + "mmanon: invalid number of ipv4 bits " + "in rewrite mode, corrected to %d", + pData->ipv4.bits); + } + if(pData->replChar != 'x') { + errmsg.LogError(0, RS_RET_REPLCHAR_IGNORED, + "mmanon: replacementChar parameter is ignored " + "in rewrite mode"); + } + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +static int +getnum(uchar *msg, int lenMsg, int *idx) +{ + int num = 0; + int i = *idx; + + while(i < lenMsg && msg[i] >= '0' && msg[i] <= '9') { + num = num * 10 + msg[i] - '0'; + ++i; + } + + *idx = i; + return num; +} + + +/* write an IP address octet to the output position */ +static int +writeOctet(uchar *msg, int idx, int *nxtidx, uint8_t octet) +{ + if(octet > 99) { + msg[idx++] = '0' + octet / 100; + octet = octet % 100; + } + if(octet > 9) { + msg[idx++] = '0' + octet / 10; + octet = octet % 10; + } + msg[idx++] = '0' + octet; + + if(nxtidx != NULL) { + if(idx + 1 != *nxtidx) { + /* we got shorter, fix it! */ + msg[idx] = '.'; + *nxtidx = idx + 1; + } + } + return idx; +} + +/* currently works for IPv4 only! */ +void +anonip(instanceData *pData, uchar *msg, int *pLenMsg, int *idx) +{ + int i = *idx; + int octet; + uint32_t ipv4addr; + int ipstart[4]; + int j; + int endpos; + int lenMsg = *pLenMsg; + + while(i < lenMsg && (msg[i] <= '0' || msg[i] >= '9')) { + ++i; /* skip to first number */ + } + if(i >= lenMsg) + goto done; + + /* got digit, let's see if ip */ + ipstart[0] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr = octet << 24; + ++i; + ipstart[1] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr |= octet << 16; + ++i; + ipstart[2] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || msg[i] != '.') goto done; + ipv4addr |= octet << 8; + ++i; + ipstart[3] = i; + octet = getnum(msg, lenMsg, &i); + if(octet > 255 || !(msg[i] == ' ' || msg[i] == ':')) goto done; + ipv4addr |= octet; + + /* OK, we now found an ip address */ + if(pData->mode == SIMPLE_MODE) { + if(pData->ipv4.bits == 8) + j = ipstart[3]; + else if(pData->ipv4.bits == 16) + j = ipstart[2]; + else if(pData->ipv4.bits == 24) + j = ipstart[1]; + else /* due to our checks, this *must* be 32 */ + j = ipstart[0]; + while(j < i) { + if(msg[j] != '.') + msg[j] = pData->replChar; + ++j; + } + } else { /* REWRITE_MODE */ + ipv4addr &= ipv4masks[pData->ipv4.bits]; + if(pData->ipv4.bits > 24) + writeOctet(msg, ipstart[0], &(ipstart[1]), ipv4addr >> 24); + if(pData->ipv4.bits > 16) + writeOctet(msg, ipstart[1], &(ipstart[2]), (ipv4addr >> 16) & 0xff); + if(pData->ipv4.bits > 8) + writeOctet(msg, ipstart[2], &(ipstart[3]), (ipv4addr >> 8) & 0xff); + endpos = writeOctet(msg, ipstart[3], NULL, ipv4addr & 0xff); + /* if we had truncation, we need to shrink the msg */ + dbgprintf("existing i %d, endpos %d\n", i, endpos); + if(i - endpos > 0) { + *pLenMsg = lenMsg - (i - endpos); + memmove(msg+endpos, msg+i, lenMsg - i + 1); + } + } + +done: *idx = i; + return; +} + + +BEGINdoAction + msg_t *pMsg; + uchar *msg; + int lenMsg; + int i; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + lenMsg = getMSGLen(pMsg); + msg = getMSG(pMsg); + for(i = 0 ; i < lenMsg ; ++i) { + anonip(pData, msg, &lenMsg, &i); + } + if(lenMsg != getMSGLen(pMsg)) + setMSGLen(pMsg, lenMsg); +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmanon:", sizeof(":mmanon:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmanon supports only v6+ config format, use: " + "action(type=\"mmanon\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmanon: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmaudit/Makefile.am b/plugins/mmaudit/Makefile.am index c64d0822..77b2e85f 100644 --- a/plugins/mmaudit/Makefile.am +++ b/plugins/mmaudit/Makefile.am @@ -1,8 +1,8 @@ pkglib_LTLIBRARIES = mmaudit.la mmaudit_la_SOURCES = mmaudit.c -mmaudit_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) $(LIBLOGNORM_CFLAGS) $(LIBEE_CFLAGS) -mmaudit_la_LDFLAGS = -module -avoid-version $(LIBLOGNORM_LIBS) $(LIBEE_LIBS) +mmaudit_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmaudit_la_LDFLAGS = -module -avoid-version mmaudit_la_LIBADD = EXTRA_DIST = diff --git a/plugins/mmaudit/mmaudit.c b/plugins/mmaudit/mmaudit.c index fcefd013..6b6b804c 100644 --- a/plugins/mmaudit/mmaudit.c +++ b/plugins/mmaudit/mmaudit.c @@ -43,8 +43,7 @@ #include <errno.h> #include <unistd.h> #include <ctype.h> -#include <libestr.h> -#include <libee/libee.h> +#include <json/json.h> #include "conf.h" #include "syslogd-types.h" #include "template.h" @@ -67,13 +66,8 @@ DEFobjCurrIf(errmsg); DEF_OMOD_STATIC_DATA typedef struct _instanceData { - ee_ctx ctxee; /**< context to be used for libee */ -} instanceData; - -typedef struct configSettings_s { int dummy; /* remove when the first real parameter is needed */ -} configSettings_t; -static configSettings_t cs; +} instanceData; BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars @@ -93,7 +87,6 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - ee_exitCtx(pData->ctxee); ENDfreeInstance @@ -169,17 +162,20 @@ finalize_it: /* parse the audit record and create libee structure */ static rsRetVal -audit_parse(instanceData *pData, uchar *buf, struct ee_event **event) +audit_parse(uchar *buf, struct json_object **jsonRoot) { - es_str_t *estr; + struct json_object *json; + struct json_object *jval; char name[1024]; char val[1024]; DEFiRet; - *event = ee_newEvent(pData->ctxee); - if(event == NULL) { + *jsonRoot = json_object_new_object(); + if(*jsonRoot == NULL) { ABORT_FINALIZE(RS_RET_ERR); } + json = json_object_new_object(); + json_object_object_add(*jsonRoot, "data", json); while(*buf) { //dbgprintf("audit_parse, buf: '%s'\n", buf); @@ -189,10 +185,8 @@ audit_parse(instanceData *pData, uchar *buf, struct ee_event **event) } ++buf; CHKiRet(parseValue(&buf, val, sizeof(val))); - - estr = es_newStrFromCStr(val, strlen(val)); - ee_addStrFieldToEvent(*event, name, estr); - es_deleteStr(estr); + jval = json_object_new_string(val); + json_object_object_add(json, name, jval); dbgprintf("mmaudit: parsed %s=%s\n", name, val); } @@ -206,9 +200,10 @@ BEGINdoAction msg_t *pMsg; uchar *buf; int typeID; - struct ee_event *event; + struct json_object *jsonRoot; + struct json_object *json; + struct json_object *jval; int i; - es_str_t *estr; char auditID[1024]; int bSuccess = 0; CODESTARTdoAction @@ -252,48 +247,24 @@ dbgprintf("mmaudit: msg is '%s'\n", buf); } buf += 2; -dbgprintf("mmaudit: cookie found, type %d, auditID '%s', rest of message: '%s'\n", typeID, auditID, buf); - audit_parse(pData, buf, &event); - if(event == NULL) { + audit_parse(buf, &jsonRoot); + if(jsonRoot == NULL) { DBGPRINTF("mmaudit: audit parse error, assuming no " "audit message: '%s'\n", buf); FINALIZE; } /* we now need to shuffle the "outer" properties into that stream */ - estr = es_newStrFromCStr(auditID, strlen(auditID)); - ee_addStrFieldToEvent(event, "audithdr.auditid", estr); - es_deleteStr(estr); - - /* we abuse auditID a bit to save space... (TODO: change!) */ - snprintf(auditID, sizeof(auditID), "%d", typeID); - estr = es_newStrFromCStr(auditID, strlen(auditID)); - ee_addStrFieldToEvent(event, "audithdr.type", estr); - es_deleteStr(estr); - - /* TODO: in the long term, we need to think about merging & different - name spaces (probably best to add the newly-obtained event as a child to - the existing event...) - */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - pMsg->event = event; + json = json_object_new_object(); + json_object_object_add(jsonRoot, "hdr", json); + jval = json_object_new_string(auditID); + json_object_object_add(json, "auditid", jval); + jval = json_object_new_int(typeID); + json_object_object_add(json, "type", jval); + + msgAddJSON(pMsg, (uchar*)"!audit", jsonRoot); bSuccess = 1; -#if 1 - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - es_str_t *str; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmaudit generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ -#endif finalize_it: MsgSetParseSuccess(pMsg, bSuccess); ENDdoAction @@ -318,13 +289,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct diff --git a/plugins/mmjsonparse/Makefile.am b/plugins/mmjsonparse/Makefile.am index 5175fe81..ef39163e 100644 --- a/plugins/mmjsonparse/Makefile.am +++ b/plugins/mmjsonparse/Makefile.am @@ -1,8 +1,8 @@ pkglib_LTLIBRARIES = mmjsonparse.la mmjsonparse_la_SOURCES = mmjsonparse.c -mmjsonparse_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) $(LIBLOGNORM_CFLAGS) $(LIBEE_CFLAGS) -mmjsonparse_la_LDFLAGS = -module -avoid-version $(LIBLOGNORM_LIBS) $(LIBEE_LIBS) +mmjsonparse_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmjsonparse_la_LDFLAGS = -module -avoid-version mmjsonparse_la_LIBADD = EXTRA_DIST = diff --git a/plugins/mmjsonparse/mmjsonparse.c b/plugins/mmjsonparse/mmjsonparse.c index 111ecc2f..35f69aab 100644 --- a/plugins/mmjsonparse/mmjsonparse.c +++ b/plugins/mmjsonparse/mmjsonparse.c @@ -35,8 +35,7 @@ #include <errno.h> #include <unistd.h> #include <ctype.h> -#include <libestr.h> -#include <libee/libee.h> +#include <json/json.h> #include "conf.h" #include "syslogd-types.h" #include "template.h" @@ -59,22 +58,49 @@ DEFobjCurrIf(errmsg); DEF_OMOD_STATIC_DATA typedef struct _instanceData { - ee_ctx ctxee; /**< context to be used for libee */ + struct json_tokener *tokener; } instanceData; -typedef struct configSettings_s { - int dummy; /* remove when the first real parameter is needed */ -} configSettings_t; -static configSettings_t cs; +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ -BEGINinitConfVars /* (re)set config variables to default values */ -CODESTARTinitConfVars - resetConfigVariables(NULL, NULL); -ENDinitConfVars + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance + pData->tokener = json_tokener_new(); + if(pData->tokener == NULL) { + errmsg.LogError(0, RS_RET_ERR, "error: could not create json " + "tokener, cannot activate action"); + ABORT_FINALIZE(RS_RET_ERR); + } +finalize_it: ENDcreateInstance @@ -85,13 +111,14 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - ee_exitCtx(pData->ctxee); + if(pData->tokener != NULL) + json_tokener_free(pData->tokener); ENDfreeInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo - dbgprintf("mmjsonparse\n"); + DBGPRINTF("mmjsonparse\n"); ENDdbgPrintInstInfo @@ -99,13 +126,57 @@ BEGINtryResume CODESTARTtryResume ENDtryResume + +static rsRetVal +processJSON(instanceData *pData, msg_t *pMsg, char *buf, size_t lenBuf) +{ + struct json_object *json; + const char *errMsg; + DEFiRet; + + assert(pData->tokener != NULL); + DBGPRINTF("mmjsonparse: toParse: '%s'\n", buf); + json_tokener_reset(pData->tokener); + + json = json_tokener_parse_ex(pData->tokener, buf, lenBuf); + if(Debug) { + errMsg = NULL; + if(json == NULL) { + enum json_tokener_error err; + + err = pData->tokener->err; + if(err != json_tokener_continue) + errMsg = json_tokener_errors[err]; + else + errMsg = "Unterminated input"; + } else if((size_t)pData->tokener->char_offset < lenBuf) + errMsg = "Extra characters after JSON object"; + else if(!json_object_is_type(json, json_type_object)) + errMsg = "JSON value is not an object"; + if(errMsg != NULL) { + DBGPRINTF("mmjsonparse: Error parsing JSON '%s': %s\n", + buf, errMsg); + } + } + if(json == NULL + || ((size_t)pData->tokener->char_offset < lenBuf) + || (!json_object_is_type(json, json_type_object))) { + ABORT_FINALIZE(RS_RET_NO_CEE_MSG); + } + + msgAddJSON(pMsg, (uchar*)"!", json); +finalize_it: + RETiRet; +} + #define COOKIE "@cee:" #define LEN_COOKIE (sizeof(COOKIE)-1) BEGINdoAction msg_t *pMsg; uchar *buf; - struct ee_event *event; int bSuccess = 0; + struct json_object *jval; + struct json_object *json; CODESTARTdoAction pMsg = (msg_t*) ppString[0]; /* note that we can performance-optimize the interface, but this also @@ -114,50 +185,45 @@ CODESTARTdoAction */ buf = getMSG(pMsg); -dbgprintf("mmjsonparse: msg is '%s'\n", buf); while(*buf && isspace(*buf)) { ++buf; } if(*buf == '\0' || strncmp((char*)buf, COOKIE, LEN_COOKIE)) { DBGPRINTF("mmjsonparse: no JSON cookie: '%s'\n", buf); - FINALIZE; + ABORT_FINALIZE(RS_RET_NO_CEE_MSG); } buf += LEN_COOKIE; -dbgprintf("mmjsonparse: cookie found, rest of message: '%s'\n", buf); - event = ee_newEventFromJSON(pData->ctxee, (char*)buf); - if(event == NULL) { - DBGPRINTF("mmjsonparse: JSON parse error, assuming no " - "JSON-enhanced message: '%s'\n", buf); - FINALIZE; - } - /* TODO: in the long term, we need to think about merging & different - name spaces (probably best to add the newly-obtained event as a child to - the existing event...) - */ - if(pMsg->event != NULL) { - ee_deleteEvent(pMsg->event); - } - pMsg->event = event; + CHKiRet(processJSON(pData, pMsg, (char*) buf, strlen((char*)buf))); bSuccess = 1; - -#if 1 - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - es_str_t *str; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmjsonparse generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ -#endif finalize_it: + if(iRet == RS_RET_NO_CEE_MSG) { + /* add buf as msg */ + json = json_object_new_object(); + jval = json_object_new_string((char*)buf); + json_object_object_add(json, "msg", jval); + msgAddJSON(pMsg, (uchar*)"!", json); + iRet = RS_RET_OK; + } MsgSetParseSuccess(pMsg, bSuccess); ENDdoAction +BEGINnewActInst +CODESTARTnewActInst + /* Note: we currently do not have any parameters, so we do not need + * the lst ptr. However, we will most probably need params in the + * future. + */ + DBGPRINTF("newActInst (mmjsonparse)\n"); + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + /*setInstParamDefaults(pData);*/ + +CODE_STD_FINALIZERnewActInst +/* cnfparamvalsDestruct(pvals, &actpblk);*/ +ENDnewActInst BEGINparseSelectorAct CODESTARTparseSelectorAct @@ -178,13 +244,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -198,6 +257,8 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES ENDqueryEtryPt @@ -217,10 +278,10 @@ BEGINmodInit() unsigned long opts; int bMsgPassingSupported; CODESTARTmodInit -INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmjsonparse: module compiled with rsyslog version %s.\n", VERSION); /* check if the rsyslog core supports parameter passing code */ bMsgPassingSupported = 0; localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", diff --git a/plugins/mmnormalize/mmnormalize.c b/plugins/mmnormalize/mmnormalize.c index 2dacb80b..fcadc328 100644 --- a/plugins/mmnormalize/mmnormalize.c +++ b/plugins/mmnormalize/mmnormalize.c @@ -4,9 +4,12 @@ * * NOTE: read comments in module-template.h for details on the calling interface! * + * TODO: check if we can replace libee via JSON system - currently that part + * is pretty inefficient... rgerhards, 2012-08-27 + * * File begun on 2010-01-01 by RGerhards * - * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * Copyright 2010-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -37,6 +40,7 @@ #include <unistd.h> #include <libestr.h> #include <libee/libee.h> +#include <json/json.h> #include <liblognorm.h> #include "conf.h" #include "syslogd-types.h" @@ -61,6 +65,7 @@ DEF_OMOD_STATIC_DATA typedef struct _instanceData { sbool bUseRawMsg; /**< use %rawmsg% instead of %msg% */ + uchar *rulebase; /**< name of rulebase to use */ ln_ctx ctxln; /**< context to be used for liblognorm */ ee_ctx ctxee; /**< context to be used for libee */ } instanceData; @@ -71,6 +76,58 @@ typedef struct configSettings_s { } configSettings_t; static configSettings_t cs; +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "rulebase", eCmdHdlrGetWord, 1 }, + { "userawmsg", eCmdHdlrBinary, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* to be called to build the libee part of the instance ONCE ALL PARAMETERS ARE CORRECT + * (and set within pData!). + */ +static rsRetVal +buildInstance(instanceData *pData) +{ + DEFiRet; + if((pData->ctxee = ee_initCtx()) == NULL) { + errmsg.LogError(0, RS_RET_ERR_LIBEE_INIT, "error: could not initialize libee " + "ctx, cannot activate action"); + ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); + } + + if((pData->ctxln = ln_initCtx()) == NULL) { + errmsg.LogError(0, RS_RET_ERR_LIBLOGNORM_INIT, "error: could not initialize " + "liblognorm ctx, cannot activate action"); + ee_exitCtx(pData->ctxee); + ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_INIT); + } + ln_setEECtx(pData->ctxln, pData->ctxee); + if(ln_loadSamples(pData->ctxln, (char*) pData->rulebase) != 0) { + errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rulebase '%s' " + "could not be loaded cannot activate action", cs.rulebase); + ee_exitCtx(pData->ctxee); + ln_exitCtx(pData->ctxln); + ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); + } +finalize_it: + RETiRet; +} + + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars resetConfigVariables(NULL, NULL); @@ -82,6 +139,35 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + + +BEGINendCnfLoad +CODESTARTendCnfLoad + loadModConf = NULL; /* done loading */ + /* free legacy config vars */ + free(cs.rulebase); + cs.rulebase = NULL; +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature ENDisCompatibleWithFeature @@ -89,6 +175,7 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance + free(pData->rulebase); ee_exitCtx(pData->ctxee); ln_exitCtx(pData->ctxln); ENDfreeInstance @@ -108,8 +195,12 @@ BEGINdoAction msg_t *pMsg; es_str_t *str; uchar *buf; + char *cstrJSON; int len; int r; + struct ee_event *event = NULL; + struct json_tokener *tokener; + struct json_object *json; CODESTARTdoAction pMsg = (msg_t*) ppString[0]; /* note that we can performance-optimize the interface, but this also @@ -123,7 +214,7 @@ CODESTARTdoAction len = getMSGLen(pMsg); } str = es_newStrFromCStr((char*)buf, len); - r = ln_normalize(pData->ctxln, str, &pMsg->event); + r = ln_normalize(pData->ctxln, str, &event); if(r != 0) { DBGPRINTF("error %d during ln_normalize\n", r); MsgSetParseSuccess(pMsg, 0); @@ -131,19 +222,76 @@ CODESTARTdoAction MsgSetParseSuccess(pMsg, 1); } es_deleteStr(str); - /***DEBUG***/ // TODO: remove after initial testing - 2010-12-01 - { - char *cstr; - ee_fmtEventToJSON(pMsg->event, &str); - cstr = es_str2cstr(str, NULL); - dbgprintf("mmnormalize generated: %s\n", cstr); - free(cstr); - es_deleteStr(str); - } - /***END DEBUG***/ + + /* reformat to our json data struct */ + /* TODO: this is all extremly ineffcient! */ + ee_fmtEventToJSON(event, &str); + cstrJSON = es_str2cstr(str, NULL); + dbgprintf("mmnormalize generated: %s\n", cstrJSON); + + tokener = json_tokener_new(); + json = json_tokener_parse_ex(tokener, cstrJSON, strlen((char*)cstrJSON)); + json_tokener_free(tokener); + msgAddJSON(pMsg, (uchar*)"!", json); + + free(cstrJSON); + es_deleteStr(str); ENDdoAction +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->rulebase = NULL; + pData->bUseRawMsg = 0; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; + int bDestructPValsOnExit; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmnormalize)\n"); + + bDestructPValsOnExit = 0; + pvals = nvlstGetParams(lst, &actpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "mmnormalize: error reading " + "config parameters"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + bDestructPValsOnExit = 1; + + if(Debug) { + dbgprintf("action param blk in mmnormalize:\n"); + cnfparamsPrint(&actpblk, pvals); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "rulebase")) { + pData->rulebase = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "userawmsg")) { + pData->bUseRawMsg = (int) pvals[i].val.d.n; + } else { + DBGPRINTF("mmnormalize: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + + iRet = buildInstance(pData); +CODE_STD_FINALIZERnewActInst + if(bDestructPValsOnExit) + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + BEGINparseSelectorAct CODESTARTparseSelectorAct CODE_STD_STRING_REQUESTparseSelectorAct(1) @@ -153,15 +301,21 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) } if(cs.rulebase == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: no normalization rulebase was specified, use " + errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: no normalization rulebase was specified, use " "$MMNormalizeSampleDB directive first!"); - ABORT_FINALIZE(RS_RET_NO_RULESET); + ABORT_FINALIZE(RS_RET_NO_RULEBASE); } /* ok, if we reach this point, we have something for us */ p += sizeof(":mmnormalize:") - 1; /* eat indicator sequence (-1 because of '\0'!) */ CHKiRet(createInstance(&pData)); + pData->rulebase = cs.rulebase; + pData->bUseRawMsg = cs.bUseRawMsg; + /* all config vars auto-reset! */ + cs.bUseRawMsg = 0; + cs.rulebase = NULL; /* we used it up! */ + /* check if a non-standard template is to be applied */ if(*(p-1) == ';') --p; @@ -169,34 +323,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * the format specified (if any) is always ignored. */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_TPL_AS_MSG, (uchar*) "RSYSLOG_FileFormat")); - - /* finally build the instance */ - if((pData->ctxee = ee_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize libee ctx, cannot " - "activate action"); - ABORT_FINALIZE(RS_RET_ERR_LIBEE_INIT); - } - - if((pData->ctxln = ln_initCtx()) == NULL) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: could not initialize liblognorm ctx, cannot " - "activate action"); - ee_exitCtx(pData->ctxee); - ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_INIT); - } - ln_setEECtx(pData->ctxln, pData->ctxee); - if(ln_loadSamples(pData->ctxln, (char*) cs.rulebase) != 0) { - errmsg.LogError(0, RS_RET_NO_RULESET, "error: normalization rulebase '%s' could not be loaded " - "cannot activate action", cs.rulebase); - ee_exitCtx(pData->ctxee); - ln_exitCtx(pData->ctxln); - ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); - } - pData->bUseRawMsg = cs.bUseRawMsg; - - /* all config vars auto-reset! */ - cs.bUseRawMsg = 0; - free(cs.rulebase); - cs.rulebase = NULL; + CHKiRet(buildInstance(pData)); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -210,6 +337,8 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt @@ -244,6 +373,7 @@ INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmnormalize: module compiled with rsyslog version %s.\n", VERSION); /* check if the rsyslog core supports parameter passing code */ bMsgPassingSupported = 0; localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", @@ -267,7 +397,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"mmnormalizerulebase", 0, eCmdHdlrGetWord, setRuleBase, NULL, STD_LOADABLE_MODULE_ID)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"mmnormalizeuserawmsg", 0, eCmdHdlrInt, + CHKiRet(omsdRegCFSLineHdlr((uchar *)"mmnormalizeuserawmsg", 0, eCmdHdlrBinary, NULL, &cs.bUseRawMsg, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/mmsnmptrapd/mmsnmptrapd.c b/plugins/mmsnmptrapd/mmsnmptrapd.c index b1ac2f64..b79a311b 100644 --- a/plugins/mmsnmptrapd/mmsnmptrapd.c +++ b/plugins/mmsnmptrapd/mmsnmptrapd.c @@ -362,6 +362,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omelasticsearch/Makefile.am b/plugins/omelasticsearch/Makefile.am index a574c72f..ba85a896 100644 --- a/plugins/omelasticsearch/Makefile.am +++ b/plugins/omelasticsearch/Makefile.am @@ -1,8 +1,9 @@ pkglib_LTLIBRARIES = omelasticsearch.la -omelasticsearch_la_SOURCES = omelasticsearch.c +# TODO: replace cJSON +omelasticsearch_la_SOURCES = omelasticsearch.c cJSON/cjson.c cJSON/cjson.h omelasticsearch_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) omelasticsearch_la_LDFLAGS = -module -avoid-version -omelasticsearch_la_LIBADD = $(CURL_LIBS) +omelasticsearch_la_LIBADD = $(CURL_LIBS) $(LIBM) EXTRA_DIST = diff --git a/plugins/omelasticsearch/README b/plugins/omelasticsearch/README new file mode 100644 index 00000000..9021bc0e --- /dev/null +++ b/plugins/omelasticsearch/README @@ -0,0 +1,17 @@ +How to produce an error: +======================== +It's quite easy to get 400, if you put a wrong mapping to your +index. That would be easy to reproduce in "normal" omelasticsearch usage +conditions, by only altering the ES configuration: + +1. Make your index first. Let's call it "testindex": +$ curl -XPUT localhost:9200/testindex/ + +2. Put your mapping for a search type called "mytype", where you specify +that date property should be an integer: +$ curl -XPUT localhost:9200/testindex/mytype/_mapping -d '{"mytype":{"properties": {"timegenerated":{"type":"integer"}}}}' + +3. Now try to insert something where date is not an integer: +$ curl -XPOST localhost:9200/testindex/mytype/ -d '{"timegenerated":"bla"}' +{"error":"MapperParsingException[Failed to parse [date]]; nested: NumberFormatException[For input string: \"bla\"]; ","status":400} + diff --git a/plugins/omelasticsearch/cJSON/README b/plugins/omelasticsearch/cJSON/README new file mode 100644 index 00000000..7531c049 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/README @@ -0,0 +1,247 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +Welcome to cJSON. + +cJSON aims to be the dumbest possible parser that you can get your job done with. +It's a single file of C, and a single header file. + +JSON is described best here: http://www.json.org/ +It's like XML, but fat-free. You use it to move data around, store things, or just +generally represent your program's state. + + +First up, how do I build? +Add cJSON.c to your project, and put cJSON.h somewhere in the header search path. +For example, to build the test app: + +gcc cJSON.c test.c -o test -lm +./test + + +As a library, cJSON exists to take away as much legwork as it can, but not get in your way. +As a point of pragmatism (i.e. ignoring the truth), I'm going to say that you can use it +in one of two modes: Auto and Manual. Let's have a quick run-through. + + +I lifted some JSON from this page: http://www.json.org/fatfree.html +That page inspired me to write cJSON, which is a parser that tries to share the same +philosophy as JSON itself. Simple, dumb, out of the way. + +Some JSON: +{ + "name": "Jack (\"Bee\") Nimble", + "format": { + "type": "rect", + "width": 1920, + "height": 1080, + "interlace": false, + "frame rate": 24 + } +} + +Assume that you got this from a file, a webserver, or magic JSON elves, whatever, +you have a char * to it. Everything is a cJSON struct. +Get it parsed: + cJSON *root = cJSON_Parse(my_json_string); + +This is an object. We're in C. We don't have objects. But we do have structs. +What's the framerate? + + cJSON *format = cJSON_GetObjectItem(root,"format"); + int framerate = cJSON_GetObjectItem(format,"frame rate")->valueint; + + +Want to change the framerate? + cJSON_GetObjectItem(format,"frame rate")->valueint=25; + +Back to disk? + char *rendered=cJSON_Print(root); + +Finished? Delete the root (this takes care of everything else). + cJSON_Delete(root); + +That's AUTO mode. If you're going to use Auto mode, you really ought to check pointers +before you dereference them. If you want to see how you'd build this struct in code? + cJSON *root,*fmt; + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "name", cJSON_CreateString("Jack (\"Bee\") Nimble")); + cJSON_AddItemToObject(root, "format", fmt=cJSON_CreateObject()); + cJSON_AddStringToObject(fmt,"type", "rect"); + cJSON_AddNumberToObject(fmt,"width", 1920); + cJSON_AddNumberToObject(fmt,"height", 1080); + cJSON_AddFalseToObject (fmt,"interlace"); + cJSON_AddNumberToObject(fmt,"frame rate", 24); + +Hopefully we can agree that's not a lot of code? There's no overhead, no unnecessary setup. +Look at test.c for a bunch of nice examples, mostly all ripped off the json.org site, and +a few from elsewhere. + +What about manual mode? First up you need some detail. +Let's cover how the cJSON objects represent the JSON data. +cJSON doesn't distinguish arrays from objects in handling; just type. +Each cJSON has, potentially, a child, siblings, value, a name. + +The root object has: Object Type and a Child +The Child has name "name", with value "Jack ("Bee") Nimble", and a sibling: +Sibling has type Object, name "format", and a child. +That child has type String, name "type", value "rect", and a sibling: +Sibling has type Number, name "width", value 1920, and a sibling: +Sibling has type Number, name "height", value 1080, and a sibling: +Sibling hs type False, name "interlace", and a sibling: +Sibling has type Number, name "frame rate", value 24 + +Here's the structure: +typedef struct cJSON { + struct cJSON *next,*prev; + struct cJSON *child; + + int type; + + char *valuestring; + int valueint; + double valuedouble; + + char *string; +} cJSON; + +By default all values are 0 unless set by virtue of being meaningful. + +next/prev is a doubly linked list of siblings. next takes you to your sibling, +prev takes you back from your sibling to you. +Only objects and arrays have a "child", and it's the head of the doubly linked list. +A "child" entry will have prev==0, but next potentially points on. The last sibling has next=0. +The type expresses Null/True/False/Number/String/Array/Object, all of which are #defined in +cJSON.h + +A Number has valueint and valuedouble. If you're expecting an int, read valueint, if not read +valuedouble. + +Any entry which is in the linked list which is the child of an object will have a "string" +which is the "name" of the entry. When I said "name" in the above example, that's "string". +"string" is the JSON name for the 'variable name' if you will. + +Now you can trivially walk the lists, recursively, and parse as you please. +You can invoke cJSON_Parse to get cJSON to parse for you, and then you can take +the root object, and traverse the structure (which is, formally, an N-tree), +and tokenise as you please. If you wanted to build a callback style parser, this is how +you'd do it (just an example, since these things are very specific): + +void parse_and_callback(cJSON *item,const char *prefix) +{ + while (item) + { + char *newprefix=malloc(strlen(prefix)+strlen(item->name)+2); + sprintf(newprefix,"%s/%s",prefix,item->name); + int dorecurse=callback(newprefix, item->type, item); + if (item->child && dorecurse) parse_and_callback(item->child,newprefix); + item=item->next; + free(newprefix); + } +} + +The prefix process will build you a separated list, to simplify your callback handling. +The 'dorecurse' flag would let the callback decide to handle sub-arrays on it's own, or +let you invoke it per-item. For the item above, your callback might look like this: + +int callback(const char *name,int type,cJSON *item) +{ + if (!strcmp(name,"name")) { /* populate name */ } + else if (!strcmp(name,"format/type") { /* handle "rect" */ } + else if (!strcmp(name,"format/width") { /* 800 */ } + else if (!strcmp(name,"format/height") { /* 600 */ } + else if (!strcmp(name,"format/interlace") { /* false */ } + else if (!strcmp(name,"format/frame rate") { /* 24 */ } + return 1; +} + +Alternatively, you might like to parse iteratively. +You'd use: + +void parse_object(cJSON *item) +{ + int i; for (i=0;i<cJSON_GetArraySize(item);i++) + { + cJSON *subitem=cJSON_GetArrayItem(item,i); + // handle subitem. + } +} + +Or, for PROPER manual mode: + +void parse_object(cJSON *item) +{ + cJSON *subitem=item->child; + while (subitem) + { + // handle subitem + if (subitem->child) parse_object(subitem->child); + + subitem=subitem->next; + } +} + +Of course, this should look familiar, since this is just a stripped-down version +of the callback-parser. + +This should cover most uses you'll find for parsing. The rest should be possible +to infer.. and if in doubt, read the source! There's not a lot of it! ;) + + +In terms of constructing JSON data, the example code above is the right way to do it. +You can, of course, hand your sub-objects to other functions to populate. +Also, if you find a use for it, you can manually build the objects. +For instance, suppose you wanted to build an array of objects? + +cJSON *objects[24]; + +cJSON *Create_array_of_anything(cJSON **items,int num) +{ + int i;cJSON *prev, *root=cJSON_CreateArray(); + for (i=0;i<24;i++) + { + if (!i) root->child=objects[i]; + else prev->next=objects[i], objects[i]->prev=prev; + prev=objects[i]; + } + return root; +} + +and simply: Create_array_of_anything(objects,24); + +cJSON doesn't make any assumptions about what order you create things in. +You can attach the objects, as above, and later add children to each +of those objects. + +As soon as you call cJSON_Print, it renders the structure to text. + + + +The test.c code shows how to handle a bunch of typical cases. If you uncomment +the code, it'll load, parse and print a bunch of test files, also from json.org, +which are more complex than I'd care to try and stash into a const char array[]. + + +Enjoy cJSON! + + +- Dave Gamble, Aug 2009 diff --git a/plugins/omelasticsearch/cJSON/cjson.c b/plugins/omelasticsearch/cJSON/cjson.c new file mode 100644 index 00000000..99a831e9 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/cjson.c @@ -0,0 +1,514 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +/* cJSON */ +/* JSON parser in C. */ + +#include <string.h> +#include <stdio.h> +#include <math.h> +#include <stdlib.h> +#include <float.h> +#include <limits.h> +#include <ctype.h> +#include "cjson.h" + +static const char *ep; + +const char *cJSON_GetErrorPtr() {return ep;} + +static int cJSON_strcasecmp(const char *s1,const char *s2) +{ + if (!s1) return (s1==s2)?0:1;if (!s2) return 1; + for(; tolower(*s1) == tolower(*s2); ++s1, ++s2) if(*s1 == 0) return 0; + return tolower(*(const unsigned char *)s1) - tolower(*(const unsigned char *)s2); +} + +static void *(*cJSON_malloc)(size_t sz) = malloc; +static void (*cJSON_free)(void *ptr) = free; + +static char* cJSON_strdup(const char* str) +{ + size_t len; + char* copy; + + len = strlen(str) + 1; + if (!(copy = (char*)cJSON_malloc(len))) return 0; + memcpy(copy,str,len); + return copy; +} + +void cJSON_InitHooks(cJSON_Hooks* hooks) +{ + if (!hooks) { /* Reset hooks */ + cJSON_malloc = malloc; + cJSON_free = free; + return; + } + + cJSON_malloc = (hooks->malloc_fn)?hooks->malloc_fn:malloc; + cJSON_free = (hooks->free_fn)?hooks->free_fn:free; +} + +/* Internal constructor. */ +static cJSON *cJSON_New_Item() +{ + cJSON* node = (cJSON*)cJSON_malloc(sizeof(cJSON)); + if (node) memset(node,0,sizeof(cJSON)); + return node; +} + +/* Delete a cJSON structure. */ +void cJSON_Delete(cJSON *c) +{ + cJSON *next; + while (c) + { + next=c->next; + if (!(c->type&cJSON_IsReference) && c->child) cJSON_Delete(c->child); + if (!(c->type&cJSON_IsReference) && c->valuestring) cJSON_free(c->valuestring); + if (c->string) cJSON_free(c->string); + cJSON_free(c); + c=next; + } +} + +/* Parse the input text to generate a number, and populate the result into item. */ +static const char *parse_number(cJSON *item,const char *num) +{ + double n=0,sign=1,scale=0;int subscale=0,signsubscale=1; + + /* Could use sscanf for this? */ + if (*num=='-') sign=-1,num++; /* Has sign? */ + if (*num=='0') num++; /* is zero */ + if (*num>='1' && *num<='9') do n=(n*10.0)+(*num++ -'0'); while (*num>='0' && *num<='9'); /* Number? */ + if (*num=='.' && num[1]>='0' && num[1]<='9') {num++; do n=(n*10.0)+(*num++ -'0'),scale--; while (*num>='0' && *num<='9');} /* Fractional part? */ + if (*num=='e' || *num=='E') /* Exponent? */ + { num++;if (*num=='+') num++; else if (*num=='-') signsubscale=-1,num++; /* With sign? */ + while (*num>='0' && *num<='9') subscale=(subscale*10)+(*num++ - '0'); /* Number? */ + } + + n=sign*n*pow(10.0,(scale+subscale*signsubscale)); /* number = +/- number.fraction * 10^+/- exponent */ + + item->valuedouble=n; + item->valueint=(int)n; + item->type=cJSON_Number; + return num; +} + +/* Render the number nicely from the given item into a string. */ +char *cJSON_print_number(cJSON *item) +{ + char *str; + double d=item->valuedouble; + if (fabs(((double)item->valueint)-d)<=DBL_EPSILON && d<=INT_MAX && d>=INT_MIN) + { + str=(char*)cJSON_malloc(21); /* 2^64+1 can be represented in 21 chars. */ + if (str) sprintf(str,"%d",item->valueint); + } + else + { + str=(char*)cJSON_malloc(64); /* This is a nice tradeoff. */ + if (str) + { + if (fabs(floor(d)-d)<=DBL_EPSILON) sprintf(str,"%.0f",d); + else if (fabs(d)<1.0e-6 || fabs(d)>1.0e9) sprintf(str,"%e",d); + else sprintf(str,"%f",d); + } + } + return str; +} + +/* Parse the input text into an unescaped cstring, and populate item. */ +static const unsigned char firstByteMark[7] = { 0x00, 0x00, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC }; +static const char *parse_string(cJSON *item,const char *str) +{ + const char *ptr=str+1;char *ptr2;char *out;int len=0;unsigned uc,uc2; + if (*str!='\"') {ep=str;return 0;} /* not a string! */ + + while (*ptr!='\"' && *ptr && ++len) if (*ptr++ == '\\') ptr++; /* Skip escaped quotes. */ + + out=(char*)cJSON_malloc(len+1); /* This is how long we need for the string, roughly. */ + if (!out) return 0; + + ptr=str+1;ptr2=out; + while (*ptr!='\"' && *ptr) + { + if (*ptr!='\\') *ptr2++=*ptr++; + else + { + ptr++; + switch (*ptr) + { + case 'b': *ptr2++='\b'; break; + case 'f': *ptr2++='\f'; break; + case 'n': *ptr2++='\n'; break; + case 'r': *ptr2++='\r'; break; + case 't': *ptr2++='\t'; break; + case 'u': /* transcode utf16 to utf8. */ + sscanf(ptr+1,"%4x",&uc);ptr+=4; /* get the unicode char. */ + + if ((uc>=0xDC00 && uc<=0xDFFF) || uc==0) break; // check for invalid. + + if (uc>=0xD800 && uc<=0xDBFF) // UTF16 surrogate pairs. + { + if (ptr[1]!='\\' || ptr[2]!='u') break; // missing second-half of surrogate. + sscanf(ptr+3,"%4x",&uc2);ptr+=6; + if (uc2<0xDC00 || uc2>0xDFFF) break; // invalid second-half of surrogate. + uc=0x10000 | ((uc&0x3FF)<<10) | (uc2&0x3FF); + } + + len=4;if (uc<0x80) len=1;else if (uc<0x800) len=2;else if (uc<0x10000) len=3; ptr2+=len; + + switch (len) { + case 4: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 3: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 2: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; + case 1: *--ptr2 =(uc | firstByteMark[len]); + } + ptr2+=len; + break; + default: *ptr2++=*ptr; break; + } + ptr++; + } + } + *ptr2=0; + if (*ptr=='\"') ptr++; + item->valuestring=out; + item->type=cJSON_String; + return ptr; +} + +/* Render the cstring provided to an escaped version that can be printed. */ +static char *print_string_ptr(const char *str) +{ + const char *ptr;char *ptr2,*out;int len=0;unsigned char token; + + if (!str) return cJSON_strdup(""); + ptr=str;while ((token=*ptr) && ++len) {if (strchr("\"\\\b\f\n\r\t",token)) len++; else if (token<32) len+=5;ptr++;} + + out=(char*)cJSON_malloc(len+3); + if (!out) return 0; + + ptr2=out;ptr=str; + *ptr2++='\"'; + while (*ptr) + { + if ((unsigned char)*ptr>31 && *ptr!='\"' && *ptr!='\\') *ptr2++=*ptr++; + else + { + *ptr2++='\\'; + switch (token=*ptr++) + { + case '\\': *ptr2++='\\'; break; + case '\"': *ptr2++='\"'; break; + case '\b': *ptr2++='b'; break; + case '\f': *ptr2++='f'; break; + case '\n': *ptr2++='n'; break; + case '\r': *ptr2++='r'; break; + case '\t': *ptr2++='t'; break; + default: sprintf(ptr2,"u%04x",token);ptr2+=5; break; /* escape and print */ + } + } + } + *ptr2++='\"';*ptr2++=0; + return out; +} +/* Invote print_string_ptr (which is useful) on an item. */ +static char *print_string(cJSON *item) {return print_string_ptr(item->valuestring);} + +/* Predeclare these prototypes. */ +static const char *parse_value(cJSON *item,const char *value); +static char *print_value(cJSON *item,int depth,int fmt); +static const char *parse_array(cJSON *item,const char *value); +static char *print_array(cJSON *item,int depth,int fmt); +static const char *parse_object(cJSON *item,const char *value); +static char *print_object(cJSON *item,int depth,int fmt); + +/* Utility to jump whitespace and cr/lf */ +static const char *skip(const char *in) {while (in && *in && (unsigned char)*in<=32) in++; return in;} + +/* Parse an object - create a new root, and populate. */ +cJSON *cJSON_Parse(const char *value) +{ + cJSON *c=cJSON_New_Item(); + ep=0; + if (!c) return 0; /* memory fail */ + + if (!parse_value(c,skip(value))) {cJSON_Delete(c);return 0;} + return c; +} + +/* Render a cJSON item/entity/structure to text. */ +char *cJSON_Print(cJSON *item) {return print_value(item,0,1);} +char *cJSON_PrintUnformatted(cJSON *item) {return print_value(item,0,0);} + +/* Parser core - when encountering text, process appropriately. */ +static const char *parse_value(cJSON *item,const char *value) +{ + if (!value) return 0; /* Fail on null. */ + if (!strncmp(value,"null",4)) { item->type=cJSON_NULL; return value+4; } + if (!strncmp(value,"false",5)) { item->type=cJSON_False; return value+5; } + if (!strncmp(value,"true",4)) { item->type=cJSON_True; item->valueint=1; return value+4; } + if (*value=='\"') { return parse_string(item,value); } + if (*value=='-' || (*value>='0' && *value<='9')) { return parse_number(item,value); } + if (*value=='[') { return parse_array(item,value); } + if (*value=='{') { return parse_object(item,value); } + + ep=value;return 0; /* failure. */ +} + +/* Render a value to text. */ +static char *print_value(cJSON *item,int depth,int fmt) +{ + char *out=0; + if (!item) return 0; + switch ((item->type)&255) + { + case cJSON_NULL: out=cJSON_strdup("null"); break; + case cJSON_False: out=cJSON_strdup("false");break; + case cJSON_True: out=cJSON_strdup("true"); break; + case cJSON_Number: out=cJSON_print_number(item);break; + case cJSON_String: out=print_string(item);break; + case cJSON_Array: out=print_array(item,depth,fmt);break; + case cJSON_Object: out=print_object(item,depth,fmt);break; + } + return out; +} + +/* Build an array from input text. */ +static const char *parse_array(cJSON *item,const char *value) +{ + cJSON *child; + if (*value!='[') {ep=value;return 0;} /* not an array! */ + + item->type=cJSON_Array; + value=skip(value+1); + if (*value==']') return value+1; /* empty array. */ + + item->child=child=cJSON_New_Item(); + if (!item->child) return 0; /* memory fail */ + value=skip(parse_value(child,skip(value))); /* skip any spacing, get the value. */ + if (!value) return 0; + + while (*value==',') + { + cJSON *new_item; + if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ + child->next=new_item;new_item->prev=child;child=new_item; + value=skip(parse_value(child,skip(value+1))); + if (!value) return 0; /* memory fail */ + } + + if (*value==']') return value+1; /* end of array */ + ep=value;return 0; /* malformed. */ +} + +/* Render an array to text */ +static char *print_array(cJSON *item,int depth,int fmt) +{ + char **entries; + char *out=0,*ptr,*ret;int len=5; + cJSON *child=item->child; + int numentries=0,i=0,fail=0; + + /* How many entries in the array? */ + while (child) numentries++,child=child->next; + /* Allocate an array to hold the values for each */ + entries=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!entries) return 0; + memset(entries,0,numentries*sizeof(char*)); + /* Retrieve all the results: */ + child=item->child; + while (child && !fail) + { + ret=print_value(child,depth+1,fmt); + entries[i++]=ret; + if (ret) len+=strlen(ret)+2+(fmt?1:0); else fail=1; + child=child->next; + } + + /* If we didn't fail, try to malloc the output string */ + if (!fail) out=(char*)cJSON_malloc(len); + /* If that fails, we fail. */ + if (!out) fail=1; + + /* Handle failure. */ + if (fail) + { + for (i=0;i<numentries;i++) if (entries[i]) cJSON_free(entries[i]); + cJSON_free(entries); + return 0; + } + + /* Compose the output array. */ + *out='['; + ptr=out+1;*ptr=0; + for (i=0;i<numentries;i++) + { + strcpy(ptr,entries[i]);ptr+=strlen(entries[i]); + if (i!=numentries-1) {*ptr++=',';if(fmt)*ptr++=' ';*ptr=0;} + cJSON_free(entries[i]); + } + cJSON_free(entries); + *ptr++=']';*ptr++=0; + return out; +} + +/* Build an object from the text. */ +static const char *parse_object(cJSON *item,const char *value) +{ + cJSON *child; + if (*value!='{') {ep=value;return 0;} /* not an object! */ + + item->type=cJSON_Object; + value=skip(value+1); + if (*value=='}') return value+1; /* empty array. */ + + item->child=child=cJSON_New_Item(); + if (!item->child) return 0; + value=skip(parse_string(child,skip(value))); + if (!value) return 0; + child->string=child->valuestring;child->valuestring=0; + if (*value!=':') {ep=value;return 0;} /* fail! */ + value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ + if (!value) return 0; + + while (*value==',') + { + cJSON *new_item; + if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ + child->next=new_item;new_item->prev=child;child=new_item; + value=skip(parse_string(child,skip(value+1))); + if (!value) return 0; + child->string=child->valuestring;child->valuestring=0; + if (*value!=':') {ep=value;return 0;} /* fail! */ + value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ + if (!value) return 0; + } + + if (*value=='}') return value+1; /* end of array */ + ep=value;return 0; /* malformed. */ +} + +/* Render an object to text. */ +static char *print_object(cJSON *item,int depth,int fmt) +{ + char **entries=0,**names=0; + char *out=0,*ptr,*ret,*str;int len=7,i=0,j; + cJSON *child=item->child; + int numentries=0,fail=0; + /* Count the number of entries. */ + while (child) numentries++,child=child->next; + /* Allocate space for the names and the objects */ + entries=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!entries) return 0; + names=(char**)cJSON_malloc(numentries*sizeof(char*)); + if (!names) {cJSON_free(entries);return 0;} + memset(entries,0,sizeof(char*)*numentries); + memset(names,0,sizeof(char*)*numentries); + + /* Collect all the results into our arrays: */ + child=item->child;depth++;if (fmt) len+=depth; + while (child) + { + names[i]=str=print_string_ptr(child->string); + entries[i++]=ret=print_value(child,depth,fmt); + if (str && ret) len+=strlen(ret)+strlen(str)+2+(fmt?2+depth:0); else fail=1; + child=child->next; + } + + /* Try to allocate the output string */ + if (!fail) out=(char*)cJSON_malloc(len); + if (!out) fail=1; + + /* Handle failure */ + if (fail) + { + for (i=0;i<numentries;i++) {if (names[i]) cJSON_free(names[i]);if (entries[i]) cJSON_free(entries[i]);} + cJSON_free(names);cJSON_free(entries); + return 0; + } + + /* Compose the output: */ + *out='{';ptr=out+1;if (fmt)*ptr++='\n';*ptr=0; + for (i=0;i<numentries;i++) + { + if (fmt) for (j=0;j<depth;j++) *ptr++='\t'; + strcpy(ptr,names[i]);ptr+=strlen(names[i]); + *ptr++=':';if (fmt) *ptr++='\t'; + strcpy(ptr,entries[i]);ptr+=strlen(entries[i]); + if (i!=numentries-1) *ptr++=','; + if (fmt) *ptr++='\n';*ptr=0; + cJSON_free(names[i]);cJSON_free(entries[i]); + } + + cJSON_free(names);cJSON_free(entries); + if (fmt) for (i=0;i<depth-1;i++) *ptr++='\t'; + *ptr++='}';*ptr++=0; + return out; +} + +/* Get Array size/item / object item. */ +int cJSON_GetArraySize(cJSON *array) {cJSON *c=array->child;int i=0;while(c)i++,c=c->next;return i;} +cJSON *cJSON_GetArrayItem(cJSON *array,int item) {cJSON *c=array->child; while (c && item>0) item--,c=c->next; return c;} +cJSON *cJSON_GetObjectItem(cJSON *object,const char *string) {cJSON *c=object->child; while (c && cJSON_strcasecmp(c->string,string)) c=c->next; return c;} + +/* Utility for array list handling. */ +static void suffix_object(cJSON *prev,cJSON *item) {prev->next=item;item->prev=prev;} +/* Utility for handling references. */ +static cJSON *create_reference(cJSON *item) {cJSON *ref=cJSON_New_Item();if (!ref) return 0;memcpy(ref,item,sizeof(cJSON));ref->string=0;ref->type|=cJSON_IsReference;ref->next=ref->prev=0;return ref;} + +/* Add item to array/object. */ +void cJSON_AddItemToArray(cJSON *array, cJSON *item) {cJSON *c=array->child;if (!item) return; if (!c) {array->child=item;} else {while (c && c->next) c=c->next; suffix_object(c,item);}} +void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item) {if (!item) return; if (item->string) cJSON_free(item->string);item->string=cJSON_strdup(string);cJSON_AddItemToArray(object,item);} +void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item) {cJSON_AddItemToArray(array,create_reference(item));} +void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item) {cJSON_AddItemToObject(object,string,create_reference(item));} + +cJSON *cJSON_DetachItemFromArray(cJSON *array,int which) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return 0; + if (c->prev) c->prev->next=c->next;if (c->next) c->next->prev=c->prev;if (c==array->child) array->child=c->next;c->prev=c->next=0;return c;} +void cJSON_DeleteItemFromArray(cJSON *array,int which) {cJSON_Delete(cJSON_DetachItemFromArray(array,which));} +cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string) {int i=0;cJSON *c=object->child;while (c && cJSON_strcasecmp(c->string,string)) i++,c=c->next;if (c) return cJSON_DetachItemFromArray(object,i);return 0;} +void cJSON_DeleteItemFromObject(cJSON *object,const char *string) {cJSON_Delete(cJSON_DetachItemFromObject(object,string));} + +/* Replace array/object items with new ones. */ +void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return; + newitem->next=c->next;newitem->prev=c->prev;if (newitem->next) newitem->next->prev=newitem; + if (c==array->child) array->child=newitem; else newitem->prev->next=newitem;c->next=c->prev=0;cJSON_Delete(c);} +void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem){int i=0;cJSON *c=object->child;while(c && cJSON_strcasecmp(c->string,string))i++,c=c->next;if(c){newitem->string=cJSON_strdup(string);cJSON_ReplaceItemInArray(object,i,newitem);}} + +/* Create basic types: */ +cJSON *cJSON_CreateNull() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_NULL;return item;} +cJSON *cJSON_CreateTrue() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_True;return item;} +cJSON *cJSON_CreateFalse() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_False;return item;} +cJSON *cJSON_CreateBool(int b) {cJSON *item=cJSON_New_Item();if(item)item->type=b?cJSON_True:cJSON_False;return item;} +cJSON *cJSON_CreateNumber(double num) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_Number;item->valuedouble=num;item->valueint=(int)num;}return item;} +cJSON *cJSON_CreateString(const char *string) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_String;item->valuestring=cJSON_strdup(string);}return item;} +cJSON *cJSON_CreateArray() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Array;return item;} +cJSON *cJSON_CreateObject() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Object;return item;} + +/* Create Arrays: */ +cJSON *cJSON_CreateIntArray(int *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateFloatArray(float *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateDoubleArray(double *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateNumber(numbers[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} +cJSON *cJSON_CreateStringArray(const char **strings,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && i<count;i++){n=cJSON_CreateString(strings[i]);if(!i)a->child=n;else suffix_object(p,n);p=n;}return a;} diff --git a/plugins/omelasticsearch/cJSON/cjson.h b/plugins/omelasticsearch/cJSON/cjson.h new file mode 100644 index 00000000..a621720c --- /dev/null +++ b/plugins/omelasticsearch/cJSON/cjson.h @@ -0,0 +1,130 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +#ifndef cJSON__h +#define cJSON__h + +#ifdef __cplusplus +extern "C" +{ +#endif + +/* cJSON Types: */ +#define cJSON_False 0 +#define cJSON_True 1 +#define cJSON_NULL 2 +#define cJSON_Number 3 +#define cJSON_String 4 +#define cJSON_Array 5 +#define cJSON_Object 6 + +#define cJSON_IsReference 256 + +/* The cJSON structure: */ +typedef struct cJSON { + struct cJSON *next,*prev; /* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */ + struct cJSON *child; /* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */ + + int type; /* The type of the item, as above. */ + + char *valuestring; /* The item's string, if type==cJSON_String */ + int valueint; /* The item's number, if type==cJSON_Number */ + double valuedouble; /* The item's number, if type==cJSON_Number */ + + char *string; /* The item's name string, if this item is the child of, or is in the list of subitems of an object. */ +} cJSON; + +typedef struct cJSON_Hooks { + void *(*malloc_fn)(size_t sz); + void (*free_fn)(void *ptr); +} cJSON_Hooks; + +/* Supply malloc, realloc and free functions to cJSON */ +extern void cJSON_InitHooks(cJSON_Hooks* hooks); + + +/* Supply a block of JSON, and this returns a cJSON object you can interrogate. Call cJSON_Delete when finished. */ +extern cJSON *cJSON_Parse(const char *value); +/* Render a cJSON entity to text for transfer/storage. Free the char* when finished. */ +extern char *cJSON_Print(cJSON *item); +/* Render a cJSON entity to text for transfer/storage without any formatting. Free the char* when finished. */ +extern char *cJSON_PrintUnformatted(cJSON *item); +/* Delete a cJSON entity and all subentities. */ +extern void cJSON_Delete(cJSON *c); + +/* Returns the number of items in an array (or object). */ +extern int cJSON_GetArraySize(cJSON *array); +/* Retrieve item number "item" from array "array". Returns NULL if unsuccessful. */ +extern cJSON *cJSON_GetArrayItem(cJSON *array,int item); +/* Get item "string" from object. Case insensitive. */ +extern cJSON *cJSON_GetObjectItem(cJSON *object,const char *string); + +/* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */ +extern const char *cJSON_GetErrorPtr(); + +/* These calls create a cJSON item of the appropriate type. */ +extern cJSON *cJSON_CreateNull(); +extern cJSON *cJSON_CreateTrue(); +extern cJSON *cJSON_CreateFalse(); +extern cJSON *cJSON_CreateBool(int b); +extern cJSON *cJSON_CreateNumber(double num); +extern cJSON *cJSON_CreateString(const char *string); +extern cJSON *cJSON_CreateArray(); +extern cJSON *cJSON_CreateObject(); + +/* These utilities create an Array of count items. */ +extern cJSON *cJSON_CreateIntArray(int *numbers,int count); +extern cJSON *cJSON_CreateFloatArray(float *numbers,int count); +extern cJSON *cJSON_CreateDoubleArray(double *numbers,int count); +extern cJSON *cJSON_CreateStringArray(const char **strings,int count); + +/* Append item to the specified array/object. */ +extern void cJSON_AddItemToArray(cJSON *array, cJSON *item); +extern void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item); +/* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */ +extern void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item); +extern void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item); + +/* Remove/Detatch items from Arrays/Objects. */ +extern cJSON *cJSON_DetachItemFromArray(cJSON *array,int which); +extern void cJSON_DeleteItemFromArray(cJSON *array,int which); +extern cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string); +extern void cJSON_DeleteItemFromObject(cJSON *object,const char *string); + +/* Update array items. */ +extern void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem); +extern void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem); + +/* rger: added helpers */ + +char *cJSON_print_number(cJSON *item); +#define cJSON_AddNullToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateNull()) +#define cJSON_AddTrueToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateTrue()) +#define cJSON_AddFalseToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateFalse()) +#define cJSON_AddNumberToObject(object,name,n) cJSON_AddItemToObject(object, name, cJSON_CreateNumber(n)) +#define cJSON_AddStringToObject(object,name,s) cJSON_AddItemToObject(object, name, cJSON_CreateString(s)) + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/plugins/omelasticsearch/cJSON/test.c b/plugins/omelasticsearch/cJSON/test.c new file mode 100644 index 00000000..2cab632a --- /dev/null +++ b/plugins/omelasticsearch/cJSON/test.c @@ -0,0 +1,156 @@ +/* + Copyright (c) 2009 Dave Gamble + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. +*/ + +#include <stdio.h> +#include <stdlib.h> +#include "cJSON.h" + +/* Parse text to JSON, then render back to text, and print! */ +void doit(char *text) +{ + char *out;cJSON *json; + + json=cJSON_Parse(text); + if (!json) {printf("Error before: [%s]\n",cJSON_GetErrorPtr());} + else + { + out=cJSON_Print(json); + cJSON_Delete(json); + printf("%s\n",out); + free(out); + } +} + +/* Read a file, parse, render back, etc. */ +void dofile(char *filename) +{ + FILE *f=fopen(filename,"rb");fseek(f,0,SEEK_END);long len=ftell(f);fseek(f,0,SEEK_SET); + char *data=malloc(len+1);fread(data,1,len,f);fclose(f); + doit(data); + free(data); +} + +/* Used by some code below as an example datatype. */ +struct record {const char *precision;double lat,lon;const char *address,*city,*state,*zip,*country; }; + +/* Create a bunch of objects as demonstration. */ +void create_objects() +{ + cJSON *root,*fmt,*img,*thm,*fld;char *out;int i; /* declare a few. */ + + /* Here we construct some JSON standards, from the JSON site. */ + + /* Our "Video" datatype: */ + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "name", cJSON_CreateString("Jack (\"Bee\") Nimble")); + cJSON_AddItemToObject(root, "format", fmt=cJSON_CreateObject()); + cJSON_AddStringToObject(fmt,"type", "rect"); + cJSON_AddNumberToObject(fmt,"width", 1920); + cJSON_AddNumberToObject(fmt,"height", 1080); + cJSON_AddFalseToObject (fmt,"interlace"); + cJSON_AddNumberToObject(fmt,"frame rate", 24); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); /* Print to text, Delete the cJSON, print it, release the string. + + /* Our "days of the week" array: */ + const char *strings[7]={"Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"}; + root=cJSON_CreateStringArray(strings,7); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + /* Our matrix: */ + int numbers[3][3]={{0,-1,0},{1,0,0},{0,0,1}}; + root=cJSON_CreateArray(); + for (i=0;i<3;i++) cJSON_AddItemToArray(root,cJSON_CreateIntArray(numbers[i],3)); + +/* cJSON_ReplaceItemInArray(root,1,cJSON_CreateString("Replacement")); */ + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + + /* Our "gallery" item: */ + int ids[4]={116,943,234,38793}; + root=cJSON_CreateObject(); + cJSON_AddItemToObject(root, "Image", img=cJSON_CreateObject()); + cJSON_AddNumberToObject(img,"Width",800); + cJSON_AddNumberToObject(img,"Height",600); + cJSON_AddStringToObject(img,"Title","View from 15th Floor"); + cJSON_AddItemToObject(img, "Thumbnail", thm=cJSON_CreateObject()); + cJSON_AddStringToObject(thm, "Url", "http:/*www.example.com/image/481989943"); + cJSON_AddNumberToObject(thm,"Height",125); + cJSON_AddStringToObject(thm,"Width","100"); + cJSON_AddItemToObject(img,"IDs", cJSON_CreateIntArray(ids,4)); + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + + /* Our array of "records": */ + struct record fields[2]={ + {"zip",37.7668,-1.223959e+2,"","SAN FRANCISCO","CA","94107","US"}, + {"zip",37.371991,-1.22026e+2,"","SUNNYVALE","CA","94085","US"}}; + + root=cJSON_CreateArray(); + for (i=0;i<2;i++) + { + cJSON_AddItemToArray(root,fld=cJSON_CreateObject()); + cJSON_AddStringToObject(fld, "precision", fields[i].precision); + cJSON_AddNumberToObject(fld, "Latitude", fields[i].lat); + cJSON_AddNumberToObject(fld, "Longitude", fields[i].lon); + cJSON_AddStringToObject(fld, "Address", fields[i].address); + cJSON_AddStringToObject(fld, "City", fields[i].city); + cJSON_AddStringToObject(fld, "State", fields[i].state); + cJSON_AddStringToObject(fld, "Zip", fields[i].zip); + cJSON_AddStringToObject(fld, "Country", fields[i].country); + } + +/* cJSON_ReplaceItemInObject(cJSON_GetArrayItem(root,1),"City",cJSON_CreateIntArray(ids,4)); */ + + out=cJSON_Print(root); cJSON_Delete(root); printf("%s\n",out); free(out); + +} + +int main (int argc, const char * argv[]) { + /* a bunch of json: */ + char text1[]="{\n\"name\": \"Jack (\\\"Bee\\\") Nimble\", \n\"format\": {\"type\": \"rect\", \n\"width\": 1920, \n\"height\": 1080, \n\"interlace\": false,\"frame rate\": 24\n}\n}"; + char text2[]="[\"Sunday\", \"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\", \"Friday\", \"Saturday\"]"; + char text3[]="[\n [0, -1, 0],\n [1, 0, 0],\n [0, 0, 1]\n ]\n"; + char text4[]="{\n \"Image\": {\n \"Width\": 800,\n \"Height\": 600,\n \"Title\": \"View from 15th Floor\",\n \"Thumbnail\": {\n \"Url\": \"http:/*www.example.com/image/481989943\",\n \"Height\": 125,\n \"Width\": \"100\"\n },\n \"IDs\": [116, 943, 234, 38793]\n }\n }"; + char text5[]="[\n {\n \"precision\": \"zip\",\n \"Latitude\": 37.7668,\n \"Longitude\": -122.3959,\n \"Address\": \"\",\n \"City\": \"SAN FRANCISCO\",\n \"State\": \"CA\",\n \"Zip\": \"94107\",\n \"Country\": \"US\"\n },\n {\n \"precision\": \"zip\",\n \"Latitude\": 37.371991,\n \"Longitude\": -122.026020,\n \"Address\": \"\",\n \"City\": \"SUNNYVALE\",\n \"State\": \"CA\",\n \"Zip\": \"94085\",\n \"Country\": \"US\"\n }\n ]"; + + /* Process each json textblock by parsing, then rebuilding: */ + doit(text1); + doit(text2); + doit(text3); + doit(text4); + doit(text5); + + /* Parse standard testfiles: +/* dofile("../../tests/test1"); */ +/* dofile("../../tests/test2"); */ +/* dofile("../../tests/test3"); */ +/* dofile("../../tests/test4"); */ +/* dofile("../../tests/test5"); */ + + /* Now some samplecode for building objects concisely: */ + create_objects(); + + return 0; +} diff --git a/plugins/omelasticsearch/cJSON/tests/test1 b/plugins/omelasticsearch/cJSON/tests/test1 new file mode 100644 index 00000000..eacfbf5e --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test1 @@ -0,0 +1,22 @@ +{ + "glossary": { + "title": "example glossary", + "GlossDiv": { + "title": "S", + "GlossList": { + "GlossEntry": { + "ID": "SGML", + "SortAs": "SGML", + "GlossTerm": "Standard Generalized Markup Language", + "Acronym": "SGML", + "Abbrev": "ISO 8879:1986", + "GlossDef": { + "para": "A meta-markup language, used to create markup languages such as DocBook.", + "GlossSeeAlso": ["GML", "XML"] + }, + "GlossSee": "markup" + } + } + } + } +} diff --git a/plugins/omelasticsearch/cJSON/tests/test2 b/plugins/omelasticsearch/cJSON/tests/test2 new file mode 100644 index 00000000..5600991a --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test2 @@ -0,0 +1,11 @@ +{"menu": { + "id": "file", + "value": "File", + "popup": { + "menuitem": [ + {"value": "New", "onclick": "CreateNewDoc()"}, + {"value": "Open", "onclick": "OpenDoc()"}, + {"value": "Close", "onclick": "CloseDoc()"} + ] + } +}} diff --git a/plugins/omelasticsearch/cJSON/tests/test3 b/plugins/omelasticsearch/cJSON/tests/test3 new file mode 100644 index 00000000..5662b377 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test3 @@ -0,0 +1,26 @@ +{"widget": { + "debug": "on", + "window": { + "title": "Sample Konfabulator Widget", + "name": "main_window", + "width": 500, + "height": 500 + }, + "image": { + "src": "Images/Sun.png", + "name": "sun1", + "hOffset": 250, + "vOffset": 250, + "alignment": "center" + }, + "text": { + "data": "Click Here", + "size": 36, + "style": "bold", + "name": "text1", + "hOffset": 250, + "vOffset": 100, + "alignment": "center", + "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;" + } +}}
\ No newline at end of file diff --git a/plugins/omelasticsearch/cJSON/tests/test4 b/plugins/omelasticsearch/cJSON/tests/test4 new file mode 100644 index 00000000..d540b57f --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test4 @@ -0,0 +1,88 @@ +{"web-app": { + "servlet": [ + { + "servlet-name": "cofaxCDS", + "servlet-class": "org.cofax.cds.CDSServlet", + "init-param": { + "configGlossary:installationAt": "Philadelphia, PA", + "configGlossary:adminEmail": "ksm@pobox.com", + "configGlossary:poweredBy": "Cofax", + "configGlossary:poweredByIcon": "/images/cofax.gif", + "configGlossary:staticPath": "/content/static", + "templateProcessorClass": "org.cofax.WysiwygTemplate", + "templateLoaderClass": "org.cofax.FilesTemplateLoader", + "templatePath": "templates", + "templateOverridePath": "", + "defaultListTemplate": "listTemplate.htm", + "defaultFileTemplate": "articleTemplate.htm", + "useJSP": false, + "jspListTemplate": "listTemplate.jsp", + "jspFileTemplate": "articleTemplate.jsp", + "cachePackageTagsTrack": 200, + "cachePackageTagsStore": 200, + "cachePackageTagsRefresh": 60, + "cacheTemplatesTrack": 100, + "cacheTemplatesStore": 50, + "cacheTemplatesRefresh": 15, + "cachePagesTrack": 200, + "cachePagesStore": 100, + "cachePagesRefresh": 10, + "cachePagesDirtyRead": 10, + "searchEngineListTemplate": "forSearchEnginesList.htm", + "searchEngineFileTemplate": "forSearchEngines.htm", + "searchEngineRobotsDb": "WEB-INF/robots.db", + "useDataStore": true, + "dataStoreClass": "org.cofax.SqlDataStore", + "redirectionClass": "org.cofax.SqlRedirection", + "dataStoreName": "cofax", + "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver", + "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon", + "dataStoreUser": "sa", + "dataStorePassword": "dataStoreTestQuery", + "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';", + "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log", + "dataStoreInitConns": 10, + "dataStoreMaxConns": 100, + "dataStoreConnUsageLimit": 100, + "dataStoreLogLevel": "debug", + "maxUrlLength": 500}}, + { + "servlet-name": "cofaxEmail", + "servlet-class": "org.cofax.cds.EmailServlet", + "init-param": { + "mailHost": "mail1", + "mailHostOverride": "mail2"}}, + { + "servlet-name": "cofaxAdmin", + "servlet-class": "org.cofax.cds.AdminServlet"}, + + { + "servlet-name": "fileServlet", + "servlet-class": "org.cofax.cds.FileServlet"}, + { + "servlet-name": "cofaxTools", + "servlet-class": "org.cofax.cms.CofaxToolsServlet", + "init-param": { + "templatePath": "toolstemplates/", + "log": 1, + "logLocation": "/usr/local/tomcat/logs/CofaxTools.log", + "logMaxSize": "", + "dataLog": 1, + "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log", + "dataLogMaxSize": "", + "removePageCache": "/content/admin/remove?cache=pages&id=", + "removeTemplateCache": "/content/admin/remove?cache=templates&id=", + "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder", + "lookInContext": 1, + "adminGroupID": 4, + "betaServer": true}}], + "servlet-mapping": { + "cofaxCDS": "/", + "cofaxEmail": "/cofaxutil/aemail/*", + "cofaxAdmin": "/admin/*", + "fileServlet": "/static/*", + "cofaxTools": "/tools/*"}, + + "taglib": { + "taglib-uri": "cofax.tld", + "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}
\ No newline at end of file diff --git a/plugins/omelasticsearch/cJSON/tests/test5 b/plugins/omelasticsearch/cJSON/tests/test5 new file mode 100644 index 00000000..49980ca2 --- /dev/null +++ b/plugins/omelasticsearch/cJSON/tests/test5 @@ -0,0 +1,27 @@ +{"menu": { + "header": "SVG Viewer", + "items": [ + {"id": "Open"}, + {"id": "OpenNew", "label": "Open New"}, + null, + {"id": "ZoomIn", "label": "Zoom In"}, + {"id": "ZoomOut", "label": "Zoom Out"}, + {"id": "OriginalView", "label": "Original View"}, + null, + {"id": "Quality"}, + {"id": "Pause"}, + {"id": "Mute"}, + null, + {"id": "Find", "label": "Find..."}, + {"id": "FindAgain", "label": "Find Again"}, + {"id": "Copy"}, + {"id": "CopyAgain", "label": "Copy Again"}, + {"id": "CopySVG", "label": "Copy SVG"}, + {"id": "ViewSVG", "label": "View SVG"}, + {"id": "ViewSource", "label": "View Source"}, + {"id": "SaveAs", "label": "Save As"}, + null, + {"id": "Help"}, + {"id": "About", "label": "About Adobe CVG Viewer..."} + ] +}} diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c index 50acdf11..cb96fb6b 100644 --- a/plugins/omelasticsearch/omelasticsearch.c +++ b/plugins/omelasticsearch/omelasticsearch.c @@ -11,11 +11,11 @@ * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 * -or- * see COPYING.ASL20 in the source distribution - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -34,6 +34,10 @@ #include <signal.h> #include <errno.h> #include <time.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include "cJSON/cjson.h" #include "conf.h" #include "syslogd-types.h" #include "srUtils.h" @@ -64,8 +68,10 @@ STATSCOUNTER_DEF(indexSuccess, mutIndexSuccess) */ typedef struct curl_slist HEADER; typedef struct _instanceData { - uchar *server; int port; + int replyLen; + int fdErrFile; /* error file fd or -1 if not open */ + uchar *server; uchar *uid; uchar *pwd; uchar *searchIndex; @@ -73,9 +79,14 @@ typedef struct _instanceData { uchar *parent; uchar *tplName; uchar *timeout; + uchar *bulkId; + uchar *restURL; /* last used URL for error reporting */ + uchar *errorFile; + char *reply; sbool dynSrchIdx; sbool dynSrchType; sbool dynParent; + sbool dynBulkId; sbool bulkmode; sbool asyncRepl; struct { @@ -104,7 +115,10 @@ static struct cnfparamdescr actpdescr[] = { { "bulkmode", eCmdHdlrBinary, 0 }, { "asyncrepl", eCmdHdlrBinary, 0 }, { "timeout", eCmdHdlrGetWord, 0 }, - { "template", eCmdHdlrGetWord, 1 } + { "errorfile", eCmdHdlrGetWord, 0 }, + { "template", eCmdHdlrGetWord, 1 }, + { "dynbulkid", eCmdHdlrBinary, 0 }, + { "bulkid", eCmdHdlrGetWord, 0 }, }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -114,6 +128,8 @@ static struct cnfparamblk actpblk = BEGINcreateInstance CODESTARTcreateInstance + pData->restURL = NULL; + pData->fdErrFile = -1; ENDcreateInstance BEGINisCompatibleWithFeature @@ -132,6 +148,8 @@ CODESTARTfreeInstance curl_easy_cleanup(pData->curlHandle); pData->curlHandle = NULL; } + if(pData->fdErrFile != -1) + close(pData->fdErrFile); free(pData->server); free(pData->uid); free(pData->pwd); @@ -139,6 +157,10 @@ CODESTARTfreeInstance free(pData->searchType); free(pData->parent); free(pData->tplName); + free(pData->timeout); + free(pData->restURL); + free(pData->errorFile); + free(pData->bulkId); ENDfreeInstance BEGINdbgPrintInstInfo @@ -158,6 +180,10 @@ CODESTARTdbgPrintInstInfo dbgprintf("\tdynamic parent=%d\n", pData->dynParent); dbgprintf("\tasync replication=%d\n", pData->asyncRepl); dbgprintf("\tbulkmode=%d\n", pData->bulkmode); + dbgprintf("\terrorfile='%s'\n", pData->errorFile == NULL ? + (uchar*)"(not configured)" : pData->errorFile); + dbgprintf("\tdynbulkid=%d\n", pData->dynBulkId); + dbgprintf("\tbulkid='%s'\n", pData->bulkId); ENDdbgPrintInstInfo @@ -201,13 +227,17 @@ checkConn(instanceData *pData) cstr = es_str2cstr(url, NULL); curl_easy_setopt(curl, CURLOPT_URL, cstr); free(cstr); - + + pData->reply = NULL; + pData->replyLen = 0; + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); res = curl_easy_perform(curl); if(res != CURLE_OK) { DBGPRINTF("omelasticsearch: checkConn() curl_easy_perform() " "failed: %s\n", curl_easy_strerror(res)); ABORT_FINALIZE(RS_RET_SUSPENDED); } + free(pData->reply); DBGPRINTF("omelasticsearch: checkConn() completed with success\n"); finalize_it: @@ -227,7 +257,8 @@ ENDtryResume /* get the current index and type for this message */ static inline void getIndexTypeAndParent(instanceData *pData, uchar **tpls, - uchar **srchIndex, uchar **srchType, uchar **parent) + uchar **srchIndex, uchar **srchType, uchar **parent, + uchar **bulkId) { if(pData->dynSrchIdx) { *srchIndex = tpls[1]; @@ -235,15 +266,27 @@ getIndexTypeAndParent(instanceData *pData, uchar **tpls, *srchType = tpls[2]; if(pData->dynParent) { *parent = tpls[3]; + if(pData->dynBulkId) { + *bulkId = tpls[4]; + } } else { *parent = pData->parent; + if(pData->dynBulkId) { + *bulkId = tpls[3]; + } } } else { *srchType = pData->searchType; if(pData->dynParent) { *parent = tpls[2]; + if(pData->dynBulkId) { + *bulkId = tpls[3]; + } } else { *parent = pData->parent; + if(pData->dynBulkId) { + *bulkId = tpls[2]; + } } } } else { @@ -252,15 +295,27 @@ getIndexTypeAndParent(instanceData *pData, uchar **tpls, *srchType = tpls[1]; if(pData->dynParent) { *parent = tpls[2]; + if(pData->dynBulkId) { + *bulkId = tpls[3]; + } } else { *parent = pData->parent; + if(pData->dynBulkId) { + *bulkId = tpls[2]; + } } } else { *srchType = pData->searchType; if(pData->dynParent) { *parent = tpls[1]; + if(pData->dynBulkId) { + *bulkId = tpls[2]; + } } else { *parent = pData->parent; + if(pData->dynBulkId) { + *bulkId = tpls[1]; + } } } } @@ -271,10 +326,10 @@ static rsRetVal setCurlURL(instanceData *pData, uchar **tpls) { char authBuf[1024]; - char *restURL; uchar *searchIndex; uchar *searchType; uchar *parent; + uchar *bulkId; es_str_t *url; int rLocal; int r; @@ -286,7 +341,7 @@ setCurlURL(instanceData *pData, uchar **tpls) r = es_addBuf(&url, "_bulk", sizeof("_bulk")-1); parent = NULL; } else { - getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent); + getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent, &bulkId); r = es_addBuf(&url, (char*)searchIndex, ustrlen(searchIndex)); if(r == 0) r = es_addChar(&url, '/'); if(r == 0) r = es_addBuf(&url, (char*)searchType, ustrlen(searchType)); @@ -305,22 +360,23 @@ setCurlURL(instanceData *pData, uchar **tpls) if(r == 0) r = es_addBuf(&url, "parent=", sizeof("parent=")-1); if(r == 0) r = es_addBuf(&url, (char*)parent, ustrlen(parent)); } - restURL = es_str2cstr(url, NULL); - curl_easy_setopt(pData->curlHandle, CURLOPT_URL, restURL); + + free(pData->restURL); + pData->restURL = (uchar*)es_str2cstr(url, NULL); + curl_easy_setopt(pData->curlHandle, CURLOPT_URL, pData->restURL); es_deleteStr(url); - DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", restURL); - free(restURL); + DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", pData->restURL); if(pData->uid != NULL) { rLocal = snprintf(authBuf, sizeof(authBuf), "%s:%s", pData->uid, (pData->pwd == NULL) ? "" : (char*)pData->pwd); - if(rLocal != (int) es_strlen(url)) { + if(rLocal < 1) { errmsg.LogError(0, RS_RET_ERR, "omelasticsearch: snprintf failed " "when trying to build auth string (return %d)\n", rLocal); ABORT_FINALIZE(RS_RET_ERR); } - curl_easy_setopt(pData->curlHandle, CURLOPT_USERPWD, authBuf); + curl_easy_setopt(pData->curlHandle, CURLOPT_USERPWD, authBuf); curl_easy_setopt(pData->curlHandle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); } finalize_it: @@ -340,16 +396,15 @@ buildBatch(instanceData *pData, uchar *message, uchar **tpls) uchar *searchIndex; uchar *searchType; uchar *parent; + uchar *bulkId = NULL; DEFiRet; # define META_STRT "{\"index\":{\"_index\": \"" # define META_TYPE "\",\"_type\":\"" # define META_PARENT "\",\"_parent\":\"" +# define META_ID "\", \"_id\":\"" # define META_END "\"}}\n" - getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent); -dbgprintf("AAA: searchIndex: '%s'\n", searchIndex); -dbgprintf("AAA: searchType: '%s'\n", searchType); -dbgprintf("AAA: parent: '%s'\n", parent); + getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent, &bulkId); r = es_addBuf(&pData->batch.data, META_STRT, sizeof(META_STRT)-1); if(r == 0) r = es_addBuf(&pData->batch.data, (char*)searchIndex, ustrlen(searchIndex)); @@ -360,6 +415,10 @@ dbgprintf("AAA: parent: '%s'\n", parent); if(r == 0) r = es_addBuf(&pData->batch.data, META_PARENT, sizeof(META_PARENT)-1); if(r == 0) r = es_addBuf(&pData->batch.data, (char*)parent, ustrlen(parent)); } + if(bulkId != NULL) { + if(r == 0) r = es_addBuf(&pData->batch.data, META_ID, sizeof(META_ID)-1); + if(r == 0) r = es_addBuf(&pData->batch.data, (char*)bulkId, ustrlen(bulkId)); + } if(r == 0) r = es_addBuf(&pData->batch.data, META_END, sizeof(META_END)-1); if(r == 0) r = es_addBuf(&pData->batch.data, (char*)message, length); if(r == 0) r = es_addBuf(&pData->batch.data, "\n", sizeof("\n")-1); @@ -373,22 +432,166 @@ finalize_it: RETiRet; } + +/* write data error request/replies to separate error file + * Note: we open the file but never close it before exit. If it + * needs to be closed, HUP must be sent. + */ +static inline rsRetVal +writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) +{ + char *rendered = NULL; + cJSON *errRoot; + cJSON *req; + cJSON *replyRoot = *pReplyRoot; + size_t toWrite; + ssize_t wrRet; + char errStr[1024]; + DEFiRet; + + if(pData->errorFile == NULL) { + DBGPRINTF("omelasticsearch: no local error logger defined - " + "ignoring ES error information\n"); + FINALIZE; + } + + if(pData->fdErrFile == -1) { + pData->fdErrFile = open((char*)pData->errorFile, + O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, + S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP); + if(pData->fdErrFile == -1) { + rs_strerror_r(errno, errStr, sizeof(errStr)); + DBGPRINTF("omelasticsearch: error opening error file: %s\n", errStr); + ABORT_FINALIZE(RS_RET_ERR); + } + } + if((req=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); + cJSON_AddItemToObject(req, "url", cJSON_CreateString((char*)pData->restURL)); + cJSON_AddItemToObject(req, "postdata", cJSON_CreateString((char*)reqmsg)); + + if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); + cJSON_AddItemToObject(errRoot, "request", req); + cJSON_AddItemToObject(errRoot, "reply", replyRoot); + rendered = cJSON_Print(errRoot); + /* we do not do real error-handling on the err file, as this finally complicates + * things way to much. + */ + DBGPRINTF("omelasticsearch: error record: '%s'\n", rendered); + toWrite = strlen(rendered); + wrRet = write(pData->fdErrFile, rendered, toWrite); + if(wrRet != (ssize_t) toWrite) { + DBGPRINTF("omelasticsearch: error %d writing error file, write returns %lld\n", + errno, (long long) wrRet); + } + cJSON_Delete(errRoot); + *pReplyRoot = NULL; /* tell caller not to delete once again! */ + +finalize_it: + free(rendered); + RETiRet; +} + + +static inline rsRetVal +checkResultBulkmode(instanceData *pData, cJSON *root) +{ + int i; + int numitems; + cJSON *items; + cJSON *item; + cJSON *create; + cJSON *ok; + DEFiRet; + + items = cJSON_GetObjectItem(root, "items"); + if(items == NULL || items->type != cJSON_Array) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "bulkmode insert does not return array, reply is: %s\n", + pData->reply); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + numitems = cJSON_GetArraySize(items); +DBGPRINTF("omelasticsearch: %d items in reply\n", numitems); + for(i = 0 ; i < numitems ; ++i) { + item = cJSON_GetArrayItem(items, i); + if(item == NULL) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain reply array item %d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + create = cJSON_GetObjectItem(item, "create"); + if(create == NULL || create->type != cJSON_Object) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain 'create' item for #%d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + ok = cJSON_GetObjectItem(create, "ok"); + if(ok == NULL || ok->type != cJSON_True) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "item %d, prop ok (%p) not ok\n", i, ok); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + } + +finalize_it: + RETiRet; +} + + +static inline rsRetVal +checkResult(instanceData *pData, uchar *reqmsg) +{ + cJSON *root; + cJSON *ok; + DEFiRet; + + root = cJSON_Parse(pData->reply); + if(root == NULL) { + DBGPRINTF("omelasticsearch: could not parse JSON result \n"); + ABORT_FINALIZE(RS_RET_ERR); + } + + if(pData->bulkmode) { + iRet = checkResultBulkmode(pData, root); + } else { + ok = cJSON_GetObjectItem(root, "ok"); + if(ok == NULL || ok->type != cJSON_True) { + iRet = RS_RET_DATAFAIL; + } + } + + /* Note: we ignore errors writing the error file, as we cannot handle + * these in any case. + */ + if(iRet == RS_RET_DATAFAIL) { + writeDataError(pData, &root, reqmsg); + iRet = RS_RET_OK; /* we have handled the problem! */ + } + +finalize_it: + if(root != NULL) + cJSON_Delete(root); + RETiRet; +} + + static rsRetVal -curlPost(instanceData *instance, uchar *message, int msglen, uchar **tpls) +curlPost(instanceData *pData, uchar *message, int msglen, uchar **tpls) { CURLcode code; - CURL *curl = instance->curlHandle; + CURL *curl = pData->curlHandle; DEFiRet; - if(instance->dynSrchIdx || instance->dynSrchType || instance->dynParent) - CHKiRet(setCurlURL(instance, tpls)); + pData->reply = NULL; + pData->replyLen = 0; - curl_easy_setopt(curl, CURLOPT_WRITEDATA, (char *)message); - curl_easy_setopt(curl, CURLOPT_POSTFIELDS, (char *)message); - curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, msglen); -dbgprintf("omelasticsearch: do curl_easy_perform()\n"); + if(pData->dynSrchIdx || pData->dynSrchType || pData->dynParent) + CHKiRet(setCurlURL(pData, tpls)); + + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); + curl_easy_setopt(curl, CURLOPT_POSTFIELDS, (char *)message); + curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, msglen); code = curl_easy_perform(curl); -DBGPRINTF("omelasticsearch: curl_easy_perform() returned %lld\n", (long long) code); switch (code) { case CURLE_COULDNT_RESOLVE_HOST: case CURLE_COULDNT_RESOLVE_PROXY: @@ -398,12 +601,18 @@ DBGPRINTF("omelasticsearch: curl_easy_perform() returned %lld\n", (long long) co DBGPRINTF("omelasticsearch: we are suspending ourselfs due " "to failure %lld of curl_easy_perform()\n", (long long) code); - return RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); default: STATSCOUNTER_INC(indexSubmit, mutIndexSubmit); - return RS_RET_OK; + break; } + + pData->reply[pData->replyLen] = '\0'; /* byte has been reserved in malloc */ + DBGPRINTF("omelasticsearch: es reply: '%s'\n", pData->reply); + + CHKiRet(checkResult(pData, message)); finalize_it: + free(pData->reply); RETiRet; } @@ -424,7 +633,6 @@ CODESTARTdoAction if(pData->bulkmode) { CHKiRet(buildBatch(pData, ppString[0], ppString)); } else { -dbgprintf("omelasticsearch: doAction calling curlPost\n"); CHKiRet(curlPost(pData, ppString[0], strlen((char*)ppString[0]), ppString)); } @@ -449,35 +657,20 @@ ENDendTransaction size_t curlResult(void *ptr, size_t size, size_t nmemb, void *userdata) { - unsigned int i; char *p = (char *)ptr; - char *jsonData = (char *)userdata; - static char ok[] = "{\"ok\":true,"; - - ASSERT(size == 1); -DBGPRINTF("omelasticsearch request: %s\n", jsonData); -DBGPRINTF("omelasticsearch result: "); -for (i = 0; i < nmemb; i++) - DBGPRINTF("%c", p[i]); -DBGPRINTF("\n"); - - if (size == 1 && - nmemb > sizeof(ok)-1 && - strncmp(p, ok, sizeof(ok)-1) == 0) { - STATSCOUNTER_INC(indexSuccess, mutIndexSuccess); -dbgprintf("omelasticsearch ok\n"); - } else { -dbgprintf("omelasticsearch fail\n"); - STATSCOUNTER_INC(indexFailed, mutIndexFailed); - if (Debug) { - DBGPRINTF("omelasticsearch (fail) request: %s\n", jsonData); - DBGPRINTF("omelasticsearch (fail) result: "); - for (i = 0; i < nmemb; i++) - DBGPRINTF("%c", p[i]); - DBGPRINTF("\n"); - } + instanceData *pData = (instanceData*) userdata; + char *buf; + size_t newlen; + + newlen = pData->replyLen + size*nmemb; + if((buf = realloc(pData->reply, newlen + 1)) == NULL) { + DBGPRINTF("omelasticsearch: realloc failed in curlResult\n"); + return 0; /* abort due to failure */ } - return size * nmemb; + memcpy(buf+pData->replyLen, p, size*nmemb); + pData->replyLen = newlen; + pData->reply = buf; + return size*nmemb; } @@ -493,10 +686,10 @@ curlSetup(instanceData *pData) } header = curl_slist_append(NULL, "Content-Type: text/json; charset=utf-8"); - curl_easy_setopt(handle, CURLOPT_HTTPHEADER, header); + curl_easy_setopt(handle, CURLOPT_HTTPHEADER, header); curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, curlResult); - curl_easy_setopt(handle, CURLOPT_POST, 1); + curl_easy_setopt(handle, CURLOPT_POST, 1); pData->curlHandle = handle; pData->postHeader = header; @@ -533,6 +726,9 @@ setInstParamDefaults(instanceData *pData) pData->asyncRepl = 0; pData->bulkmode = 0; pData->tplName = NULL; + pData->errorFile = NULL; + pData->dynBulkId= 0; + pData->bulkId = NULL; } BEGINnewActInst @@ -552,6 +748,8 @@ CODESTARTnewActInst continue; if(!strcmp(actpblk.descr[i].name, "server")) { pData->server = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "errorfile")) { + pData->errorFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "serverport")) { pData->port = (int) pvals[i].val.d.n, NULL; } else if(!strcmp(actpblk.descr[i].name, "uid")) { @@ -578,12 +776,16 @@ CODESTARTnewActInst pData->asyncRepl = pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "dynbulkid")) { + pData->dynBulkId = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "bulkid")) { + pData->bulkId = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { dbgprintf("omelasticsearch: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); } } - + if(pData->pwd != NULL && pData->uid == NULL) { errmsg.LogError(0, RS_RET_UID_MISSING, "omelasticsearch: password is provided, but no uid " @@ -608,6 +810,12 @@ CODESTARTnewActInst "name for parent template given - action definition invalid"); ABORT_FINALIZE(RS_RET_CONFIG_ERROR); } + if(pData->dynBulkId && pData->bulkId == NULL) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, + "omelasticsearch: requested dynamic bulkid, but no " + "name for bulkid template given - action definition invalid"); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } if(pData->bulkmode) { pData->batch.currTpl1 = NULL; @@ -623,6 +831,7 @@ CODESTARTnewActInst if(pData->dynSrchIdx) ++iNumTpls; if(pData->dynSrchType) ++iNumTpls; if(pData->dynParent) ++iNumTpls; + if(pData->dynBulkId) ++iNumTpls; DBGPRINTF("omelasticsearch: requesting %d templates\n", iNumTpls); CODE_STD_STRING_REQUESTnewActInst(iNumTpls) @@ -644,11 +853,29 @@ CODESTARTnewActInst if(pData->dynParent) { CHKiRet(OMSRsetEntry(*ppOMSR, 3, ustrdup(pData->parent), OMSR_NO_RQD_TPL_OPTS)); + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 4, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } + } else { + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 3, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } } } else { if(pData->dynParent) { CHKiRet(OMSRsetEntry(*ppOMSR, 2, ustrdup(pData->parent), OMSR_NO_RQD_TPL_OPTS)); + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 3, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } + } else { + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 2, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } } } } else { @@ -658,12 +885,30 @@ CODESTARTnewActInst if(pData->dynParent) { CHKiRet(OMSRsetEntry(*ppOMSR, 2, ustrdup(pData->parent), OMSR_NO_RQD_TPL_OPTS)); + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 3, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } + } else { + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 2, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } } } else { if(pData->dynParent) { CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->parent), OMSR_NO_RQD_TPL_OPTS)); - } + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 2, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } + } else { + if(pData->dynBulkId) { + CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->bulkId), + OMSR_NO_RQD_TPL_OPTS)); + } + } } } @@ -693,6 +938,14 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct +BEGINdoHUP +CODESTARTdoHUP + if(pData->fdErrFile != -1) { + close(pData->fdErrFile); + pData->fdErrFile = -1; + } +ENDdoHUP + BEGINmodExit CODESTARTmodExit @@ -707,6 +960,7 @@ CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_doHUP CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt diff --git a/plugins/omhiredis/README b/plugins/omhiredis/README index 5ca31373..3b2bf9de 100644 --- a/plugins/omhiredis/README +++ b/plugins/omhiredis/README @@ -1,29 +1,22 @@ Redis Outplug Plugin using hiredis library -tested in Centos 6.2 +tested in Centos 6.2 and Archlinux BUILDING THIS PLUGIN -Requires the hiredis C client library: https://github.com/antirez/hiredis/ +Requires the hiredis C client library: https://github.com/redis/hiredis/ in your /etc/rsyslog.conf, together with other modules: -TODO - -* Error handling for redis calls -* Integrating with impstats -* Clean up code -* Make it work with rsyslog batch mode -* Fix bugs - -Brian Knox <bknox@talksum.com> +Brian Knox <briank@talksum.com> --------------------------------------------------------------------------------------------- -$ModLoad omhiredis.so # provides redis output +module(load="omhiredis") -$template TestRedis, "hincrby progcount %programname% 1" +template(name="simple_count" type="string" string="HINCRBY progcount %programname% 1") -if $msg then { - action(type="omhiredis", template="TestRedis") -} +action(name="simple_count_redis" type="omhiredis" queue.type="FixedArray" queue.size="10000" queue.dequeuebatchsize="100" template="simple_count") --------------------------------------------------------------------------------------------- +Note: dequeuebatchsize now sets the pipeline size for hiredis, allowing pipelining commands. +Note: this plugin will NOT handle full rsyslog messages properly yet. spaces in a property will + cause the redis command to be constructed improperly. a fix for this is in the works! diff --git a/plugins/omhiredis/omhiredis.c b/plugins/omhiredis/omhiredis.c index 7c454436..051ac0bf 100644 --- a/plugins/omhiredis/omhiredis.c +++ b/plugins/omhiredis/omhiredis.c @@ -20,7 +20,6 @@ */ - #include "config.h" #include <stdio.h> #include <string.h> @@ -49,11 +48,16 @@ MODULE_CNFNAME("omhiredis") DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) +/* our instance data. + * this will be accessable + * via pData */ typedef struct _instanceData { - redisContext *conn; - uchar *server; - int port; - uchar *tplName; + redisContext *conn; /* redis connection */ + uchar *server; /* redis server address */ + int port; /* redis port */ + uchar *tplName; /* template name */ + redisReply **replies; /* array to hold replies from redis */ + int count; /* count of command sent for current batch */ } instanceData; @@ -78,6 +82,7 @@ CODESTARTisCompatibleWithFeature iRet = RS_RET_OK; ENDisCompatibleWithFeature +/* called when closing */ static void closeHiredis(instanceData *pData) { if(pData->conn != NULL) { @@ -86,7 +91,8 @@ static void closeHiredis(instanceData *pData) } } - +/* Free our instance data. + * TODO: free **replies */ BEGINfreeInstance CODESTARTfreeInstance closeHiredis(pData); @@ -100,7 +106,7 @@ CODESTARTdbgPrintInstInfo /* nothing special here */ ENDdbgPrintInstInfo - +/* establish our connection to redis */ static rsRetVal initHiredis(instanceData *pData, int bSilent) { char *server; @@ -115,55 +121,104 @@ static rsRetVal initHiredis(instanceData *pData, int bSilent) if(!bSilent) errmsg.LogError(0, RS_RET_SUSPENDED, "can not initialize redis handle"); - ABORT_FINALIZE(RS_RET_SUSPENDED); + ABORT_FINALIZE(RS_RET_SUSPENDED); } - finalize_it: RETiRet; } rsRetVal writeHiredis(uchar *message, instanceData *pData) { - redisReply *reply; DEFiRet; + /* if we do not have a redis connection, call + * initHiredis and try to establish one */ if(pData->conn == NULL) CHKiRet(initHiredis(pData, 0)); - reply = redisCommand(pData->conn, (char*)message); - if (reply->type == REDIS_REPLY_ERROR) { - errmsg.LogError(0, NO_ERRCODE, "omhiredis: %s", reply->str); - dbgprintf("omhiredis: %s\n", reply->str); - freeReplyObject(reply); + /* try to append the command to the pipeline. + * REDIS_ERR reply indicates something bad + * happened, in which case abort. otherwise + * increase our current pipeline count + * by 1 and continue. */ + int rc; + rc = redisAppendCommand(pData->conn, (char*)message); + if (rc == REDIS_ERR) { + errmsg.LogError(0, NO_ERRCODE, "omhiredis: %s", pData->conn->errstr); + dbgprintf("omhiredis: %s\n", pData->conn->errstr); ABORT_FINALIZE(RS_RET_ERR); } else { - freeReplyObject(reply); - } + pData->count++; + } finalize_it: RETiRet; } +/* called when resuming from suspended state. + * try to restablish our connection to redis */ BEGINtryResume CODESTARTtryResume if(pData->conn == NULL) iRet = initHiredis(pData, 0); ENDtryResume +/* begin a transaction. for now does nothing. + * if I decide to use MULTI ... EXEC in the + * fture, this block should send the + * MULTI command to redis. */ +BEGINbeginTransaction +CODESTARTbeginTransaction + dbgprintf("omhiredis: beginTransaction called\n"); +ENDbeginTransaction + +/* call writeHiredis for this log line, + * which appends it as a command to the + * current pipeline */ BEGINdoAction CODESTARTdoAction - iRet = writeHiredis(ppString[0], pData); + CHKiRet(writeHiredis(ppString[0], pData)); + iRet = RS_RET_DEFER_COMMIT; +finalize_it: ENDdoAction - +/* called when we have reached the end of a + * batch (queue.dequeuebatchsize). this + * iterates over the replies, putting them + * into the pData->replies buffer. we currently + * don't really bother to check for errors + * which should be fixed */ +BEGINendTransaction +CODESTARTendTransaction + dbgprintf("omhiredis: endTransaction called\n"); + int i; + pData->replies = malloc ( sizeof ( redisReply* ) * pData->count ); + for ( i = 0; i < pData->count; i++ ) { + redisGetReply ( pData->conn, (void *)&pData->replies[i] ); + /* TODO: add error checking here! */ + free ( pData->replies[i] ); + } + free ( pData->replies ); + pData->count = 0; +ENDendTransaction + +/* set defaults. note server is set to NULL + * and is set to a default in initHiredis if + * it is still null when it's called - I should + * probable just set the default here instead */ static inline void setInstParamDefaults(instanceData *pData) { pData->server = NULL; pData->port = 6379; pData->tplName = NULL; + pData->count = 0; } +/* here is where the work to set up a new instance + * is done. this reads the config options from + * the rsyslog conf and takes appropriate setup + * actions. */ BEGINnewActInst struct cnfparamvals *pvals; int i; @@ -222,18 +277,25 @@ BEGINmodExit CODESTARTmodExit ENDmodExit - +/* register our plugin entry points + * with the rsyslog core engine */ BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES /* supports transaction interface */ ENDqueryEtryPt +/* note we do not support rsyslog v5 syntax */ BEGINmodInit() CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* only supports rsyslog 6 configs */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); + if (!bCoreSupportsBatching) { + errmsg.LogError(0, NO_ERRCODE, "omhiredis: rsyslog core does not support batching - abort"); + ABORT_FINALIZE(RS_RET_ERR); + } DBGPRINTF("omhiredis: module compiled with rsyslog version %s.\n", VERSION); ENDmodInit diff --git a/plugins/omjournal/Makefile.am b/plugins/omjournal/Makefile.am new file mode 100644 index 00000000..4cfbbd96 --- /dev/null +++ b/plugins/omjournal/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = omjournal.la + +omjournal_la_SOURCES = omjournal.c +omjournal_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) $(LIBSYSTEMD_JOURNAL_CFLAGS) +omjournal_la_LDFLAGS = -module -avoid-version +omjournal_la_LIBADD = $(LIBSYSTEMD_JOURNAL_LIBS) + +EXTRA_DIST = diff --git a/plugins/omjournal/omjournal.c b/plugins/omjournal/omjournal.c new file mode 100644 index 00000000..160c369d --- /dev/null +++ b/plugins/omjournal/omjournal.c @@ -0,0 +1,187 @@ +/* omjournal.c + * send messages to the Linux Journal. This is meant to be used + * in cases where journal serves as the whole system log database. + * Note that we may get into a loop if journald re-injects messages + * into the syslog stream and we read that via imuxsock. Thus there + * is an option in imuxsock to ignore messages from ourselves + * (actually from our pid). So there are some module-interdependencies. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include <systemd/sd-journal.h> + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("omjournal") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + + +typedef struct _instanceData { +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +BEGINnewActInst +CODESTARTnewActInst + /* Note: we currently do not have any parameters, so we do not need + * the lst ptr. However, we will most probably need params in the + * future. + */ + (void) lst; /* prevent compiler warning */ + DBGPRINTF("newActInst (mmjournal)\n"); + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + /*setInstParamDefaults(pData);*/ +CODE_STD_FINALIZERnewActInst +/* cnfparamvalsDestruct(pvals, &actpblk);*/ +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +BEGINdoAction + msg_t *pMsg; + uchar *tag; + int lenTag; + int sev; + int r; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + MsgGetSeverity(pMsg, &sev); + getTAG(pMsg, &tag, &lenTag); + /* we can use more properties here, but let's see if there + * is some real user interest. We can always add later... + */ + r = sd_journal_send("MESSAGE=%s", getMSG(pMsg), + "PRIORITY=%d", sev, + "SYSLOG_FACILITY=%d", pMsg->iFacility, + "SYSLOG_IDENTIFIER=%s", tag, + NULL); + /* FIXME: think about what to do with errors ;) */ + (void) r; /* prevent compiler warning */ +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":omjournal:", sizeof(":omjournal:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "omjournal supports only v6+ config format, use: " + "action(type=\"omjournal\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("omjournal: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/omlibdbi/omlibdbi.c b/plugins/omlibdbi/omlibdbi.c index 99bcaf9d..3beba4f0 100644 --- a/plugins/omlibdbi/omlibdbi.c +++ b/plugins/omlibdbi/omlibdbi.c @@ -10,7 +10,7 @@ * * File begun on 2008-02-14 by RGerhards (extracted from syslogd.c) * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -70,6 +70,7 @@ typedef struct _instanceData { uchar *dbName; /* database to use */ unsigned uLastDBErrno; /* last errno returned by libdbi or 0 if all is well */ uchar *tplName; /* format template to use */ + int txSupport; /* transaction support */ } instanceData; typedef struct configSettings_s { @@ -81,15 +82,36 @@ typedef struct configSettings_s { uchar *dbName; /* database to use */ } configSettings_t; static configSettings_t cs; +uchar *pszFileDfltTplName; /* name of the default template to use */ + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ + uchar *dbiDrvrDir; /* where do the dbi drivers reside? */ + uchar *tplName; /* default template */ +}; + +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ +static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ + /* tables for interfacing with the v6 config system */ +/* module-global parameters */ +static struct cnfparamdescr modpdescr[] = { + { "template", eCmdHdlrGetWord, 0 }, + { "driverdirectory", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr + }; /* action (instance) parameters */ static struct cnfparamdescr actpdescr[] = { { "server", eCmdHdlrGetWord, 1 }, { "db", eCmdHdlrGetWord, 1 }, { "uid", eCmdHdlrGetWord, 1 }, { "pwd", eCmdHdlrGetWord, 1 }, - { "driverdirectory", eCmdHdlrGetWord, 0 }, { "driver", eCmdHdlrGetWord, 1 }, { "template", eCmdHdlrGetWord, 0 } }; @@ -99,6 +121,20 @@ static struct cnfparamblk actpblk = actpdescr }; +/* this function gets the default template. It coordinates action between + * old-style and new-style configuration parts. + */ +static inline uchar* +getDfltTpl(void) +{ + if(loadModConf != NULL && loadModConf->tplName != NULL) + return loadModConf->tplName; + else if(pszFileDfltTplName == NULL) + return (uchar*)" StdDBFmt"; + else + return pszFileDfltTplName; +} + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars @@ -144,7 +180,6 @@ static void closeConn(instanceData *pData) BEGINfreeInstance CODESTARTfreeInstance closeConn(pData); - free(pData->dbiDrvrDir); free(pData->drvrName); free(pData->host); free(pData->usrName); @@ -227,7 +262,7 @@ static rsRetVal initConn(instanceData *pData, int bSilent) # endif if(pData->conn == NULL) { errmsg.LogError(0, RS_RET_SUSPENDED, "can not initialize libdbi connection"); - iRet = RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); } else { /* we could get the handle, now on with work... */ /* Connect to database */ dbi_conn_set_option(pData->conn, "host", (char*) pData->host); @@ -238,8 +273,9 @@ static rsRetVal initConn(instanceData *pData, int bSilent) if(dbi_conn_connect(pData->conn) < 0) { reportDBError(pData, bSilent); closeConn(pData); /* ignore any error we may get */ - iRet = RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); } + pData->txSupport = dbi_conn_cap_get(pData->conn, "transaction_support"); } finalize_it: @@ -295,12 +331,127 @@ CODESTARTtryResume } ENDtryResume +/* transaction support 2013-03 */ +BEGINbeginTransaction +CODESTARTbeginTransaction + if(pData->conn == NULL) { + CHKiRet(initConn(pData, 0)); + } +# if HAVE_DBI_TXSUPP + if (pData->txSupport == 1) { + if (dbi_conn_transaction_begin(pData->conn) != 0) { + const char *emsg; + dbi_conn_error(pData->conn, &emsg); + dbgprintf("libdbi server error: begin transaction " + "not successful: %s\n", emsg); + closeConn(pData); + ABORT_FINALIZE(RS_RET_SUSPENDED); + } + } +# endif +finalize_it: +ENDbeginTransaction +/* end transaction */ + BEGINdoAction CODESTARTdoAction - dbgprintf("\n"); - iRet = writeDB(ppString[0], pData); + CHKiRet(writeDB(ppString[0], pData)); +# if HAVE_DBI_TXSUPP + if (pData->txSupport == 1) { + iRet = RS_RET_DEFER_COMMIT; + } +# endif +finalize_it: ENDdoAction +/* transaction support 2013-03 */ +BEGINendTransaction +CODESTARTendTransaction +# if HAVE_DBI_TXSUPP + if (dbi_conn_transaction_commit(pData->conn) != 0) { + const char *emsg; + dbi_conn_error(pData->conn, &emsg); + dbgprintf("libdbi server error: transaction not committed: %s\n", + emsg); + closeConn(pData); + iRet = RS_RET_SUSPENDED; + } +# endif +ENDendTransaction +/* end transaction */ + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; + pModConf->tplName = NULL; + bLegacyCnfModGlobalsPermitted = 1; +ENDbeginCnfLoad + +BEGINsetModCnf + struct cnfparamvals *pvals = NULL; + int i; +CODESTARTsetModCnf + pvals = nvlstGetParams(lst, &modpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "omlibdbi: error processing " + "module config parameters [module(...)]"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(Debug) { + dbgprintf("module (global) param blk for omlibdbi:\n"); + cnfparamsPrint(&modpblk, pvals); + } + + for(i = 0 ; i < modpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(modpblk.descr[i].name, "template")) { + loadModConf->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + if(pszFileDfltTplName != NULL) { + errmsg.LogError(0, RS_RET_DUP_PARAM, "omlibdbi: warning: default template " + "was already set via legacy directive - may lead to inconsistent " + "results."); + } + } else if(!strcmp(modpblk.descr[i].name, "driverdirectory")) { + loadModConf->dbiDrvrDir = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("omlibdbi: program error, non-handled " + "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); + } + } + bLegacyCnfModGlobalsPermitted = 0; +finalize_it: + if(pvals != NULL) + cnfparamvalsDestruct(pvals, &modpblk); +ENDsetModCnf + +BEGINendCnfLoad +CODESTARTendCnfLoad + loadModConf = NULL; /* done loading */ + /* free legacy config vars */ + free(pszFileDfltTplName); + pszFileDfltTplName = NULL; +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf + free(pModConf->tplName); + free(pModConf->dbiDrvrDir); +ENDfreeCnf + + + static inline void setInstParamDefaults(instanceData *pData) @@ -311,6 +462,7 @@ setInstParamDefaults(instanceData *pData) BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; CODESTARTnewActInst if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { @@ -319,7 +471,6 @@ CODESTARTnewActInst CHKiRet(createInstance(&pData)); setInstParamDefaults(pData); - CODE_STD_STRING_REQUESTnewActInst(1) for(i = 0 ; i < actpblk.nParams ; ++i) { if(!pvals[i].bUsed) @@ -332,28 +483,19 @@ CODESTARTnewActInst pData->usrName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "pwd")) { pData->pwd = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(actpblk.descr[i].name, "driverdirectory")) { - pData->dbiDrvrDir = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "driver")) { pData->drvrName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { - dbgprintf("ommysql: program error, non-handled " + dbgprintf("omlibdbi: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); } } - if(pData->tplName == NULL) { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*) strdup(" StdDBFmt"), - OMSR_RQD_TPL_OPT_SQL)); - } else { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, - (uchar*) strdup((char*) pData->tplName), - OMSR_RQD_TPL_OPT_SQL)); - } + tplToUse = (pData->tplName == NULL) ? (uchar*)strdup((char*)getDfltTpl()) : pData->tplName; + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_RQD_TPL_OPT_SQL)); CODE_STD_FINALIZERnewActInst -dbgprintf("XXXX: added param, iRet %d\n", iRet); cnfparamvalsDestruct(pvals, &actpblk); ENDnewActInst @@ -369,7 +511,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* ok, if we reach this point, we have something for us */ CHKiRet(createInstance(&pData)); - /* no create the instance based on what we currently have */ if(cs.drvrName == NULL) { errmsg.LogError(0, RS_RET_NO_DRIVERNAME, "omlibdbi: no db driver name given - action can not be created"); @@ -380,19 +521,17 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* NULL values are supported because drivers have different needs. * They will err out on connect. -- rgerhards, 2008-02-15 */ - if(cs.host != NULL) - if((pData->host = (uchar*) strdup((char*)cs.host)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + if(cs.host != NULL) + CHKmalloc(pData->host = (uchar*) strdup((char*)cs.host)); if(cs.usrName != NULL) - if((pData->usrName = (uchar*) strdup((char*)cs.usrName)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - if(cs.dbName != NULL) - if((pData->dbName = (uchar*) strdup((char*)cs.dbName)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - if(cs.pwd != NULL) - if((pData->pwd = (uchar*) strdup((char*)cs.pwd)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pData->usrName = (uchar*) strdup((char*)cs.usrName)); + if(cs.dbName != NULL) + CHKmalloc(pData->dbName = (uchar*) strdup((char*)cs.dbName)); + if(cs.pwd != NULL) + CHKmalloc(pData->pwd = (uchar*) strdup((char*)cs.pwd)); if(cs.dbiDrvrDir != NULL) - if((pData->dbiDrvrDir = (uchar*) strdup((char*)cs.dbiDrvrDir)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - - CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_RQD_TPL_OPT_SQL, (uchar*) " StdDBFmt")); - + CHKmalloc(loadModConf->dbiDrvrDir = (uchar*) strdup((char*)cs.dbiDrvrDir)); + CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_RQD_TPL_OPT_SQL, getDfltTpl())); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -413,7 +552,10 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt @@ -443,8 +585,12 @@ CODESTARTmodInit INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr +# ifndef HAVE_DBI_TXSUPP + DBGPRINTF("omlibdbi: no transaction support in libdbi\n"); +# warning libdbi too old - transactions are not enabled (use 0.9 or later) +# endif CHKiRet(objUse(errmsg, CORE_COMPONENT)); - CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbidriverdirectory", 0, eCmdHdlrGetWord, NULL, &cs.dbiDrvrDir, STD_LOADABLE_MODULE_ID)); + CHKiRet(regCfSysLineHdlr2((uchar *)"actionlibdbidriverdirectory", 0, eCmdHdlrGetWord, NULL, &cs.dbiDrvrDir, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbidriver", 0, eCmdHdlrGetWord, NULL, &cs.drvrName, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbihost", 0, eCmdHdlrGetWord, NULL, &cs.host, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionlibdbiusername", 0, eCmdHdlrGetWord, NULL, &cs.usrName, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/ommail/ommail.c b/plugins/ommail/ommail.c index d70fa30a..6044d2e9 100644 --- a/plugins/ommail/ommail.c +++ b/plugins/ommail/ommail.c @@ -689,6 +689,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/ommongodb/ommongodb.c b/plugins/ommongodb/ommongodb.c index d246fab4..dd997410 100644 --- a/plugins/ommongodb/ommongodb.c +++ b/plugins/ommongodb/ommongodb.c @@ -68,6 +68,7 @@ typedef struct _instanceData { uchar *pwd; uchar *dbNcoll; uchar *tplName; + int bErrMsgPermitted; /* only one errmsg permitted per connection */ } instanceData; @@ -139,19 +140,21 @@ static void reportMongoError(instanceData *pData) { char errStr[1024]; - errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", - rs_strerror_r(errno, errStr, sizeof(errStr))); -#if 0 gchar *err; - if(mongo_sync_cmd_get_last_error(pData->conn, (gchar*)pData->db, &err) == TRUE) { - errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", err); - } else { - errmsg.LogError(0, RS_RET_ERR, "ommongodb: we had an error, but can " - "not obtain specifics"); + int eno; + + if(pData->bErrMsgPermitted) { + eno = errno; + if(mongo_sync_cmd_get_last_error(pData->conn, (gchar*)pData->db, &err) == TRUE) { + errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", err); + } else { + DBGPRINTF("ommongodb: we had an error, but can not obtain specifics, " + "using plain old errno error message generator\n"); + errmsg.LogError(0, RS_RET_ERR, "ommongodb: error: %s", + rs_strerror_r(eno, errStr, sizeof(errStr))); + } + pData->bErrMsgPermitted = 0; } -#else - (void)pData; -#endif } @@ -224,20 +227,20 @@ static bson * getDefaultBSON(msg_t *pMsg) { bson *doc = NULL; - uchar *procid; short unsigned procid_free; size_t procid_len; - uchar *tag; short unsigned tag_free; size_t tag_len; - uchar *pid; short unsigned pid_free; size_t pid_len; - uchar *sys; short unsigned sys_free; size_t sys_len; - uchar *msg; short unsigned msg_free; size_t msg_len; + uchar *procid; short unsigned procid_free; rs_size_t procid_len; + uchar *tag; short unsigned tag_free; rs_size_t tag_len; + uchar *pid; short unsigned pid_free; rs_size_t pid_len; + uchar *sys; short unsigned sys_free; rs_size_t sys_len; + uchar *msg; short unsigned msg_free; rs_size_t msg_len; int severity, facil; gint64 ts_gen, ts_rcv; /* timestamps: generated, received */ int secfrac; - procid = MsgGetProp(pMsg, NULL, PROP_PROGRAMNAME, NULL, &procid_len, &procid_free); - tag = MsgGetProp(pMsg, NULL, PROP_SYSLOGTAG, NULL, &tag_len, &tag_free); - pid = MsgGetProp(pMsg, NULL, PROP_PROCID, NULL, &pid_len, &pid_free); - sys = MsgGetProp(pMsg, NULL, PROP_HOSTNAME, NULL, &sys_len, &sys_free); - msg = MsgGetProp(pMsg, NULL, PROP_MSG, NULL, &msg_len, &msg_free); + procid = MsgGetProp(pMsg, NULL, PROP_PROGRAMNAME, NULL, &procid_len, &procid_free, NULL); + tag = MsgGetProp(pMsg, NULL, PROP_SYSLOGTAG, NULL, &tag_len, &tag_free, NULL); + pid = MsgGetProp(pMsg, NULL, PROP_PROCID, NULL, &pid_len, &pid_free, NULL); + sys = MsgGetProp(pMsg, NULL, PROP_HOSTNAME, NULL, &sys_len, &sys_free, NULL); + msg = MsgGetProp(pMsg, NULL, PROP_MSG, NULL, &msg_len, &msg_free, NULL); // TODO: move to datetime? Refactor in any case! rgerhards, 2012-03-30 ts_gen = (gint64) datetime.syslogTime2time_t(&pMsg->tTIMESTAMP) * 1000; /* ms! */ @@ -296,7 +299,9 @@ static bson *BSONFromJSONObject(struct json_object *json); static gboolean BSONAppendJSONObject(bson *doc, const gchar *name, struct json_object *json) { - switch(json_object_get_type(json)) { + switch(json != NULL ? json_object_get_type(json) : json_type_null) { + case json_type_null: + return bson_append_null(doc, name); case json_type_boolean: return bson_append_boolean(doc, name, json_object_get_boolean(json)); @@ -431,9 +436,11 @@ CODESTARTdoAction /* FIXME: is this a correct return code? */ ABORT_FINALIZE(RS_RET_ERR); } - if(!mongo_sync_cmd_insert(pData->conn, (char*)pData->dbNcoll, doc, NULL)) { - reportMongoError(pData); + if(mongo_sync_cmd_insert(pData->conn, (char*)pData->dbNcoll, doc, NULL)) { + pData->bErrMsgPermitted = 1; + } else { dbgprintf("ommongodb: insert error\n"); + reportMongoError(pData); ABORT_FINALIZE(RS_RET_SUSPENDED); } diff --git a/plugins/ommysql/ommysql.c b/plugins/ommysql/ommysql.c index 69ffb9ac..2dfa29de 100644 --- a/plugins/ommysql/ommysql.c +++ b/plugins/ommysql/ommysql.c @@ -189,7 +189,6 @@ static rsRetVal initMySQL(instanceData *pData, int bSilent) ASSERT(pData != NULL); ASSERT(pData->f_hmysql == NULL); - pData->f_hmysql = mysql_init(NULL); if(pData->f_hmysql == NULL) { errmsg.LogError(0, RS_RET_SUSPENDED, "can not initialize MySQL handle"); @@ -219,10 +218,12 @@ static rsRetVal initMySQL(instanceData *pData, int bSilent) pData->f_dbpwd, pData->f_dbname, pData->f_dbsrvPort, NULL, 0) == NULL) { reportDBError(pData, bSilent); closeMySQL(pData); /* ignore any error we may get */ - iRet = RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); } + mysql_autocommit(pData->f_hmysql, 0); } +finalize_it: RETiRet; } @@ -241,6 +242,7 @@ rsRetVal writeMySQL(uchar *psz, instanceData *pData) /* see if we are ready to proceed */ if(pData->f_hmysql == NULL) { CHKiRet(initMySQL(pData, 0)); + } /* try insert */ @@ -272,12 +274,28 @@ CODESTARTtryResume } ENDtryResume +BEGINbeginTransaction +CODESTARTbeginTransaction + CHKiRet(writeMySQL((uchar*)"START TRANSACTION", pData)); +finalize_it: +ENDbeginTransaction + BEGINdoAction CODESTARTdoAction dbgprintf("\n"); - iRet = writeMySQL(ppString[0], pData); + CHKiRet(writeMySQL(ppString[0], pData)); + iRet = RS_RET_DEFER_COMMIT; +finalize_it: ENDdoAction +BEGINendTransaction +CODESTARTendTransaction + if (mysql_commit(pData->f_hmysql) != 0) { + dbgprintf("mysql server error: transaction not committed\n"); + iRet = RS_RET_SUSPENDED; + } +ENDendTransaction + static inline void setInstParamDefaults(instanceData *pData) @@ -305,7 +323,7 @@ CODESTARTnewActInst CHKiRet(createInstance(&pData)); setInstParamDefaults(pData); - CODE_STD_STRING_REQUESTnewActInst(1) + CODE_STD_STRING_REQUESTparseSelectorAct(1) for(i = 0 ; i < actpblk.nParams ; ++i) { if(!pvals[i].bUsed) continue; @@ -437,6 +455,7 @@ BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt @@ -459,6 +478,11 @@ INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); + INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); + if(!bCoreSupportsBatching) { + errmsg.LogError(0, NO_ERRCODE, "ommysql: rsyslog core too old"); + ABORT_FINALIZE(RS_RET_ERR); + } /* we need to init the MySQL library. If that fails, we cannot run */ if( diff --git a/plugins/omprog/omprog.c b/plugins/omprog/omprog.c index 6978a9d0..e425b428 100644 --- a/plugins/omprog/omprog.c +++ b/plugins/omprog/omprog.c @@ -128,7 +128,12 @@ static void execBinary(instanceData *pData, int fdStdin) assert(pData != NULL); fclose(stdin); - dup(fdStdin); + if(dup(fdStdin) == -1) { + DBGPRINTF("omprog: dup() failed\n"); + /* do some more error handling here? Maybe if the module + * gets some more widespread use... + */ + } //fclose(stdout); /* we close all file handles as we fork soon diff --git a/plugins/omrabbitmq/Makefile.am b/plugins/omrabbitmq/Makefile.am new file mode 100644 index 00000000..de374081 --- /dev/null +++ b/plugins/omrabbitmq/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = omrabbitmq.la + +omrabbitmq_la_SOURCES = omrabbitmq.c +omrabbitmq_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +omrabbitmq_la_LDFLAGS = -module -avoid-version +omrabbitmq_la_LIBADD = $(RABBITMQ_LIBS) + +EXTRA_DIST = diff --git a/plugins/omrabbitmq/README.md b/plugins/omrabbitmq/README.md new file mode 100644 index 00000000..7aa60206 --- /dev/null +++ b/plugins/omrabbitmq/README.md @@ -0,0 +1,56 @@ + +# rsyslog output module for RabbitMQ + +This module sends syslog messages into RabbitMQ server. + +Only v6 configuration syntax is supported. + +**omrabbitmq is tested only with 6.6.0 version of rsyslog.** + + +## Compile +To successfully compile omrabbitmq module you need [rabbitmq-c](https://github.com/alanxz/rabbitmq-c) library. + + ./configure --enable-omrabbitmq ... + + +---- +## Configure + +omrabbitmq output module supports only v6 configuration syntax. + +Parameters: + +* host=<hostname> – server +* virtual_host=<virtual\_host> – virtual message broker +* user=<user> – user name +* password=<password> – password +* exchange=<name> – exchange name +* routing_key=<name> – name of routing key + + +Example: + + $ModLoad omrabbitmq + + *.* action(type="omrabbitmq" + host="localhost" + virtual_host="/" + user="guest" + password="guest" + exchange="syslog" + routing_key="syslog.all" + template="RSYSLOG_ForwardFormat" + queue.type="linkedlist" + queue.timeoutenqueue="0" + queue.filename="rabbitmq" + queue.highwatermark="500000" + queue.lowwatermark="400000" + queue.discardmark="5000000" + queue.timeoutenqueue="0" + queue.maxdiskspace="5g" + queue.size="2000000" + queue.saveonshutdown="on" + action.resumeretrycount="-1") + + diff --git a/plugins/omrabbitmq/omrabbitmq.c b/plugins/omrabbitmq/omrabbitmq.c new file mode 100644 index 00000000..7ea7793d --- /dev/null +++ b/plugins/omrabbitmq/omrabbitmq.c @@ -0,0 +1,466 @@ +/* omrabbitmq.c + * + * This output plugin enables rsyslog to send messages to the RabbitMQ. + * + * Copyright 2012-2013 Vaclav Tomec + * + * This program is free software: you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public License + * as published by the Free Software Foundation, either version 3 of + * the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this program. If not, see + * <http://www.gnu.org/licenses/>. + * + * Author: Vaclav Tomec + * <vaclav.tomec@gmail.com> + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <time.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include "cfsysline.h" + +#include <amqp.h> + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("omrabbitmq") + + +/* + * internal structures + */ +DEF_OMOD_STATIC_DATA +DEFobjCurrIf(errmsg) + + +typedef struct _instanceData { + /* here you need to define all action-specific data. A record of type + * instanceData will be handed over to each instance of the action. Keep + * in mind that there may be several invocations of the same type of action + * inside rsyslog.conf, and this is what keeps them apart. Do NOT use + * static data for this! + */ + amqp_connection_state_t conn; + amqp_basic_properties_t props; + uchar *host; + int port; + uchar *vhost; + uchar *user; + uchar *password; + uchar *exchange; + uchar *routing_key; + uchar *tplName; +} instanceData; + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "host", eCmdHdlrGetWord, 0 }, + { "port", eCmdHdlrInt, 0 }, + { "virtual_host", eCmdHdlrGetWord, 0 }, + { "user", eCmdHdlrGetWord, 0 }, + { "password", eCmdHdlrGetWord, 0 }, + { "exchange", eCmdHdlrGetWord, 0 }, + { "routing_key", eCmdHdlrGetWord, 0 }, + { "template", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk actpblk = + { + CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + + +/* + * Report general error + */ +static int +die_on_error(int x, char const *context) +{ + int retVal = 0; // false + + if (x < 0) { + char *errstr = amqp_error_string(-x); + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: %s", context, errstr); + free(errstr); + retVal = 1; // true + } + + return retVal; +} + + +/* + * Report AMQP specific error + */ +static int +die_on_amqp_error(amqp_rpc_reply_t x, char const *context) +{ + int retVal = 1; // true + + switch (x.reply_type) { + case AMQP_RESPONSE_NORMAL: + retVal = 0; // false + break; + + case AMQP_RESPONSE_NONE: + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: missing RPC reply type!", context); + break; + + case AMQP_RESPONSE_LIBRARY_EXCEPTION: + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: %s", context, amqp_error_string(x.library_error)); + break; + + case AMQP_RESPONSE_SERVER_EXCEPTION: + switch (x.reply.id) { + case AMQP_CONNECTION_CLOSE_METHOD: { + amqp_connection_close_t *m = (amqp_connection_close_t *) x.reply.decoded; + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: server connection error %d, message: %.*s", + context, + m->reply_code, + (int) m->reply_text.len, (char *) m->reply_text.bytes); + break; + } + case AMQP_CHANNEL_CLOSE_METHOD: { + amqp_channel_close_t *m = (amqp_channel_close_t *) x.reply.decoded; + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: server channel error %d, message: %.*s", + context, + m->reply_code, + (int) m->reply_text.len, (char *) m->reply_text.bytes); + break; + } + default: + errmsg.LogError(0, RS_RET_ERR, "omrabbitmq: %s: unknown server error, method id 0x%08X\n", context, x.reply.id); + break; + } + break; + + } + + return retVal; +} + + +static amqp_bytes_t +cstring_bytes(const char *str) +{ + return str ? amqp_cstring_bytes(str) : amqp_empty_bytes; +} + + +static void +closeAMQPConnection(instanceData *pData) +{ + if (pData->conn != NULL) { + die_on_amqp_error(amqp_channel_close(pData->conn, 1, AMQP_REPLY_SUCCESS), "amqp_channel_close"); + die_on_amqp_error(amqp_connection_close(pData->conn, AMQP_REPLY_SUCCESS), "amqp_connection_close"); + die_on_error(amqp_destroy_connection(pData->conn), "amqp_destroy_connection"); + + pData->conn = NULL; + } +} + + +/* + * Initialize RabbitMQ connection + */ +static rsRetVal +initRabbitMQ(instanceData *pData) +{ + int sockfd; + DEFiRet; + + DBGPRINTF("omrabbitmq: trying connect to '%s' at port %d\n", pData->host, pData->port); + + pData->conn = amqp_new_connection(); + + if (die_on_error(sockfd = amqp_open_socket((char*) pData->host, pData->port), "Opening socket")) { + pData->conn = NULL; + ABORT_FINALIZE(RS_RET_SUSPENDED); + } + + amqp_set_sockfd(pData->conn, sockfd); + + if (die_on_amqp_error(amqp_login(pData->conn, (char*) pData->vhost, 0, 131072, 0, AMQP_SASL_METHOD_PLAIN, pData->user, pData->password), + "Logging in")) { + pData->conn = NULL; + ABORT_FINALIZE(RS_RET_SUSPENDED); + } + + amqp_channel_open(pData->conn, 1); + + if (die_on_amqp_error(amqp_get_rpc_reply(pData->conn), "Opening channel")) { + pData->conn = NULL; + ABORT_FINALIZE(RS_RET_SUSPENDED); + } + +finalize_it: + RETiRet; +} + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + /* use this to specify if select features are supported by this + * plugin. If not, the framework will handle that. Currently, only + * RepeatedMsgReduction ("last message repeated n times") is optional. + */ + if(eFeat == sFEATURERepeatedMsgReduction) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance + /* this is a cleanup callback. All dynamically-allocated resources + * in instance data must be cleaned up here. Prime examples are + * malloc()ed memory, file & database handles and the like. + */ + closeAMQPConnection(pData); + free(pData->host); + free(pData->vhost); + free(pData->user); + free(pData->password); + free(pData->exchange); + free(pData->routing_key); + free(pData->tplName); +ENDfreeInstance + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo + /* permits to spit out some debug info */ + dbgprintf("omrabbitmq\n"); + dbgprintf("\thost='%s'\n", pData->host); + dbgprintf("\tport=%d\n", pData->port); + dbgprintf("\tvirtual_host='%s'\n", pData->vhost); + dbgprintf("\tuser='%s'\n", pData->user == NULL ? (uchar*)"(not configured)" : pData->user); + dbgprintf("\tpassword=(%sconfigured)\n", pData->password == NULL ? "not " : ""); + dbgprintf("\texchange='%s'\n", pData->exchange); + dbgprintf("\trouting_key='%s'\n", pData->routing_key); + dbgprintf("\ttemplate='%s'\n", pData->tplName); +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume + /* this is called when an action has been suspended and the + * rsyslog core tries to resume it. The action must then + * retry (if possible) and report RS_RET_OK if it succeeded + * or RS_RET_SUSPENDED otherwise. + * Note that no data can be written in this callback, as it is + * not present. Prime examples of what can be retried are + * reconnects to remote hosts, reconnects to database, + * opening of files and the like. + * If there is no retry-type of operation, the action may + * return RS_RET_OK, so that it will get called on its doAction + * entry point (where it receives data), retries there, and + * immediately returns RS_RET_SUSPENDED if that does not work + * out. This disables some optimizations in the core's retry logic, + * but is a valid and expected behaviour. Note that it is also OK + * for the retry entry point to return OK but the immediately following + * doAction call to fail. In real life, for example, a buggy com line + * may cause such behaviour. + * Note that there is no guarantee that the core will very quickly + * call doAction after the retry succeeded. Today, it does, but that may + * not always be the case. + */ + + if (pData->conn == NULL) { + iRet = initRabbitMQ(pData); + } + +ENDtryResume + + +BEGINdoAction +CODESTARTdoAction + /* this is where you receive the message and need to carry out the + * action. Data is provided in ppString[i] where 0 <= i <= num of strings + * requested. + * Return RS_RET_OK if all goes well, RS_RET_SUSPENDED if the action can + * currently not complete, or an error code or RS_RET_DISABLED. The later + * two should only be returned if there is no hope that the action can be + * restored unless an rsyslog restart (prime example is an invalid config). + * Error code or RS_RET_DISABLED permanently disables the action, up to + * the next restart. + */ + + amqp_bytes_t body_bytes; + + if (pData->conn == NULL) { + CHKiRet(initRabbitMQ(pData)); + } + + body_bytes = amqp_cstring_bytes((char *)ppString[0]); + + if (die_on_error(amqp_basic_publish(pData->conn, 1, + cstring_bytes((char *) pData->exchange), + cstring_bytes((char *) pData->routing_key), + 0, 0, &pData->props, body_bytes), "amqp_basic_publish")) { + closeAMQPConnection(pData); + ABORT_FINALIZE(RS_RET_SUSPENDED); + } + +finalize_it: + +ENDdoAction + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->host = NULL; + pData->port = 5672; + pData->vhost = NULL; + pData->user = NULL; + pData->password = NULL; + pData->exchange = NULL; + pData->routing_key = NULL; + pData->tplName = NULL; +} + + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + CODE_STD_STRING_REQUESTparseSelectorAct(1) + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if (!pvals[i].bUsed) + continue; + if (!strcmp(actpblk.descr[i].name, "host")) { + pData->host = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "port")) { + pData->port = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "virtual_host")) { + pData->vhost = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "user")) { + pData->user = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "password")) { + pData->password = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "exchange")) { + pData->exchange = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "routing_key")) { + pData->routing_key = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "template")) { + pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("omrabbitmq: program error, non-handled param '%s'\n", actpblk.descr[i].name); + } + } + + if (pData->host == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter host must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + if (pData->vhost == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter virtual_host must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + if (pData->user == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter user must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + if (pData->password == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter password must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + if (pData->exchange == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter exchange must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + if (pData->routing_key == NULL) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "omrabbitmq module disabled: parameter routing_key must be specified"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + // RabbitMQ properties initialization + memset(&pData->props, 0, sizeof pData->props); + pData->props._flags = AMQP_BASIC_DELIVERY_MODE_FLAG; + pData->props.delivery_mode = 2; /* persistent delivery mode */ + pData->props._flags |= AMQP_BASIC_CONTENT_TYPE_FLAG; + pData->props.content_type = amqp_cstring_bytes("application/json"); + + CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*)strdup((pData->tplName == NULL) ? + " StdJSONFmt" : (char*)pData->tplName), + OMSR_NO_RQD_TPL_OPTS)); + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct + CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(!strncmp((char*) p, ":omrabbitmq:", sizeof(":omrabbitmq:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "omrabbitmq supports only v6 config format, use: " + "action(type=\"omrabbitmq\" host=...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt + CODEqueryEtryPt_STD_OMOD_QUERIES + CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c index 39ffe7fb..3e355464 100644 --- a/plugins/omrelp/omrelp.c +++ b/plugins/omrelp/omrelp.c @@ -7,7 +7,7 @@ * * File begun on 2008-03-13 by RGerhards * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -43,6 +43,7 @@ #include "glbl.h" #include "errmsg.h" #include "debug.h" +#include "unicode-helper.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -57,12 +58,13 @@ DEFobjCurrIf(glbl) static relpEngine_t *pRelpEngine; /* our relp engine */ typedef struct _instanceData { - char *f_hname; - int compressionLevel; /* 0 - no compression, else level for zlib */ - char *port; + uchar *target; + uchar *port; int bInitialConnect; /* is this the initial connection request of our module? (0-no, 1-yes) */ int bIsConnected; /* currently connected to server? 0 - no, 1 - yes */ - relpClt_t *pRelpClt; /* relp client for this instance */ + unsigned timeout; + relpClt_t *pRelpClt; /* relp client for this instance */ + uchar *tplName; } instanceData; typedef struct configSettings_s { @@ -70,30 +72,116 @@ typedef struct configSettings_s { } configSettings_t; static configSettings_t __attribute__((unused)) cs; + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "target", eCmdHdlrGetWord, 1 }, + { "port", eCmdHdlrGetWord, 0 }, + { "timeout", eCmdHdlrInt, 0 }, + { "template", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars ENDinitConfVars -/* get the syslog forward port from selector_t. The passed in - * struct must be one that is setup for forwarding. - * rgerhards, 2007-06-28 - * We may change the implementation to try to lookup the port - * if it is unspecified. So far, we use the IANA default auf 514. +/* We may change the implementation to try to lookup the port + * if it is unspecified. So far, we use 514 as default (what probably + * is not a really bright idea, but kept for backward compatibility). */ -static char *getRelpPt(instanceData *pData) +static uchar *getRelpPt(instanceData *pData) { assert(pData != NULL); if(pData->port == NULL) - return("514"); + return((uchar*)"514"); else return(pData->port); } +static inline rsRetVal +doCreateRelpClient(instanceData *pData) +{ + DEFiRet; + if(relpEngineCltConstruct(pRelpEngine, &pData->pRelpClt) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetTimeout(pData->pRelpClt, pData->timeout) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); +finalize_it: + RETiRet; +} + + BEGINcreateInstance CODESTARTcreateInstance pData->bInitialConnect = 1; + pData->timeout = 90; ENDcreateInstance +BEGINfreeInstance +CODESTARTfreeInstance + if(pData->pRelpClt != NULL) + relpEngineCltDestruct(pRelpEngine, &pData->pRelpClt); + free(pData->target); + free(pData->port); + free(pData->tplName); +ENDfreeInstance + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->target = NULL; + pData->port = NULL; + pData->tplName = NULL; + pData->timeout = 90; +} + + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "target")) { + pData->target = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "port")) { + pData->port = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "template")) { + pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "timeout")) { + pData->timeout = (unsigned) pvals[i].val.d.n; + } else { + dbgprintf("omrelp: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + + CODE_STD_STRING_REQUESTnewActInst(1) + + CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*)strdup((pData->tplName == NULL) ? + "RSYSLOG_ForwardFormat" : (char*)pData->tplName), + OMSR_NO_RQD_TPL_OPTS)); + + CHKiRet(doCreateRelpClient(pData)); + +CODE_STD_FINALIZERnewActInst + if(pvals != NULL) + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -101,25 +189,16 @@ CODESTARTisCompatibleWithFeature iRet = RS_RET_OK; ENDisCompatibleWithFeature - -BEGINfreeInstance -CODESTARTfreeInstance - if(pData->port != NULL) - free(pData->port); - - /* final cleanup */ - if(pData->pRelpClt != NULL) - relpEngineCltDestruct(pRelpEngine, &pData->pRelpClt); - - if(pData->f_hname != NULL) - free(pData->f_hname); - -ENDfreeInstance +BEGINSetShutdownImmdtPtr +CODESTARTSetShutdownImmdtPtr + relpEngineSetShutdownImmdtPtr(pRelpEngine, pPtr); + DBGPRINTF("omrelp: shutdownImmediate ptr now is %p\n", pPtr); +ENDSetShutdownImmdtPtr BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo - printf("RELP/%s", pData->f_hname); + dbgprintf("RELP/%s", pData->target); ENDdbgPrintInstInfo @@ -131,7 +210,7 @@ static rsRetVal doConnect(instanceData *pData) DEFiRet; if(pData->bInitialConnect) { - iRet = relpCltConnect(pData->pRelpClt, glbl.GetDefPFFamily(), (uchar*) pData->port, (uchar*) pData->f_hname); + iRet = relpCltConnect(pData->pRelpClt, glbl.GetDefPFFamily(), pData->port, pData->target); if(iRet == RELP_RET_OK) pData->bInitialConnect = 0; } else { @@ -160,7 +239,7 @@ BEGINdoAction size_t lenMsg; relpRetVal ret; CODESTARTdoAction - dbgprintf(" %s:%s/RELP\n", pData->f_hname, getRelpPt(pData)); + dbgprintf(" %s:%s/RELP\n", pData->target, getRelpPt(pData)); if(!pData->bIsConnected) { CHKiRet(doConnect(pData)); @@ -201,62 +280,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) if((iRet = createInstance(&pData)) != RS_RET_OK) FINALIZE; - /* we are now after the protocol indicator. Now check if we should - * use compression. We begin to use a new option format for this: - * @(option,option)host:port - * The first option defined is "z[0..9]" where the digit indicates - * the compression level. If it is not given, 9 (best compression) is - * assumed. An example action statement might be: - * :omrelp:(z5,o)127.0.0.1:1400 - * Which means send via TCP with medium (5) compresion (z) to the local - * host on port 1400. The '0' option means that octet-couting (as in - * IETF I-D syslog-transport-tls) is to be used for framing (this option - * applies to TCP-based syslog only and is ignored when specified with UDP). - * That is not yet implemented. - * rgerhards, 2006-12-07 - * TODO: think of all this in spite of RELP -- rgerhards, 2008-03-13 - */ - if(*p == '(') { - /* at this position, it *must* be an option indicator */ - do { - ++p; /* eat '(' or ',' (depending on when called) */ - /* check options */ - if(*p == 'z') { /* compression */ -# ifdef USE_NETZIP - ++p; /* eat */ - if(isdigit((int) *p)) { - int iLevel; - iLevel = *p - '0'; - ++p; /* eat */ - pData->compressionLevel = iLevel; - } else { - errmsg.LogError(0, NO_ERRCODE, "Invalid compression level '%c' specified in " - "forwardig action - NOT turning on compression.", - *p); - } -# else - errmsg.LogError(0, NO_ERRCODE, "Compression requested, but rsyslogd is not compiled " - "with compression support - request ignored."); -# endif /* #ifdef USE_NETZIP */ - } else { /* invalid option! Just skip it... */ - errmsg.LogError(0, NO_ERRCODE, "Invalid option %c in forwarding action - ignoring.", *p); - ++p; /* eat invalid option */ - } - /* the option processing is done. We now do a generic skip - * to either the next option or the end of the option - * block. - */ - while(*p && *p != ')' && *p != ',') - ++p; /* just skip it */ - } while(*p && *p == ','); /* Attention: do.. while() */ - if(*p == ')') - ++p; /* eat terminator, on to next */ - else - /* we probably have end of string - leave it for the rest - * of the code to handle it (but warn the user) - */ - errmsg.LogError(0, NO_ERRCODE, "Option block not terminated in forwarding action."); - } /* extract the host first (we do a trick - we replace the ';' or ':' with a '\0') * now skip to port and then template name. rgerhards 2005-07-06 */ @@ -309,21 +332,17 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* TODO: make this if go away! */ if(*p == ';') { *p = '\0'; /* trick to obtain hostname (later)! */ - CHKmalloc(pData->f_hname = strdup((char*) q)); + CHKmalloc(pData->target = ustrdup(q)); *p = ';'; } else { - CHKmalloc(pData->f_hname = strdup((char*) q)); + CHKmalloc(pData->target = ustrdup(q)); } /* process template */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) "RSYSLOG_ForwardFormat")); - /* create our relp client */ - CHKiRet(relpEngineCltConstruct(pRelpEngine, &pData->pRelpClt)); /* we use CHKiRet as librelp has a similar return value range */ + CHKiRet(doCreateRelpClient(pData)); - /* TODO: do we need to call freeInstance if we failed - this is a general question for - * all output modules. I'll address it later as the interface evolves. rgerhards, 2007-07-25 - */ CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -341,6 +360,9 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_SetShutdownImmdtPtr ENDqueryEtryPt diff --git a/plugins/omruleset/omruleset.c b/plugins/omruleset/omruleset.c index 67aee97e..11765507 100644 --- a/plugins/omruleset/omruleset.c +++ b/plugins/omruleset/omruleset.c @@ -120,7 +120,11 @@ CODESTARTdoAction (char*) pData->pszRulesetName, pData->pRuleset); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); MsgSetRuleset(pMsg, pData->pRuleset); - submitMsg(pMsg); + /* Note: we intentionally use submitMsg2() here, as we process messages + * that were already run through the rate-limiter. So it is (at least) + * questionable if they were rate-limited again. + */ + submitMsg2(pMsg); finalize_it: ENDdoAction @@ -165,6 +169,9 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) p += sizeof(":omruleset:") - 1; /* eat indicator sequence (-1 because of '\0'!) */ CHKiRet(createInstance(&pData)); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: omruleset is deprecated, consider " + "using the 'call' statement instead"); + /* check if a non-standard template is to be applied */ if(*(p-1) == ';') --p; @@ -192,6 +199,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt @@ -237,6 +245,9 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(ruleset, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: omruleset is deprecated, consider " + "using the 'call' statement instead"); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionomrulesetrulesetname", 0, eCmdHdlrGetWord, setRuleset, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, diff --git a/plugins/omstdout/omstdout.c b/plugins/omstdout/omstdout.c index fb95e951..a84a7593 100644 --- a/plugins/omstdout/omstdout.c +++ b/plugins/omstdout/omstdout.c @@ -105,6 +105,7 @@ BEGINdoAction int iBuf; char szBuf[65564]; size_t len; + int r; CODESTARTdoAction if(pData->bUseArrayInterface) { /* if we use array passing, we need to put together a string @@ -136,9 +137,19 @@ CODESTARTdoAction toWrite = (char*) ppString[0]; } len = strlen(toWrite); - write(1, toWrite, len); /* 1 is stdout! */ + /* the following if's are just to silence compiler warnings. If someone + * actually intends to use this module in production (why???), this code + * needs to be more solid. -- rgerhards, 2012-11-28 + */ + if((r = write(1, toWrite, len)) != (int) len) { /* 1 is stdout! */ + DBGPRINTF("omstdout: error %d writing to stdout[%d]: %s\n", + r, len, toWrite); + } if(pData->bEnsureLFEnding && toWrite[len-1] != '\n') { - write(1, "\n", 1); /* write missing LF */ + if((r = write(1, "\n", 1)) != 1) { /* write missing LF */ + DBGPRINTF("omstdout: error %d writing \\n to stdout\n", + r); + } } ENDdoAction @@ -175,6 +186,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omtesting/omtesting.c b/plugins/omtesting/omtesting.c index ff290c94..c9f1e06b 100644 --- a/plugins/omtesting/omtesting.c +++ b/plugins/omtesting/omtesting.c @@ -313,6 +313,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omudpspoof/omudpspoof.c b/plugins/omudpspoof/omudpspoof.c index d2c5364c..c80f0e57 100644 --- a/plugins/omudpspoof/omudpspoof.c +++ b/plugins/omudpspoof/omudpspoof.c @@ -5,7 +5,7 @@ * This file builds on UDP spoofing code contributed by * David Lang <david@lang.hm>. I then created a "real" rsyslog module * out of that code and omfwd. I decided to make it a separate module because - * omfwd already mixes up too many things (TCP & UDP & a differnt modes, + * omfwd already mixes up too many things (TCP & UDP & a different modes, * this has historic reasons), it would not be a good idea to also add * spoofing to it. And, looking at the requirements, there is little in * common between omfwd and this module. @@ -93,14 +93,19 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(net) typedef struct _instanceData { + uchar *tplName; /* name of assigned template */ uchar *host; uchar *port; + uchar *sourceTpl; + int mtu; int *pSockArray; /* sockets to use for UDP */ - int compressionLevel; /* 0 - no compression, else level for zlib */ struct addrinfo *f_addr; u_short sourcePort; u_short sourcePortStart; /* for sorce port iteration */ u_short sourcePortEnd; + int bReportLibnetInitErr; /* help prevent multiple error messages on init err */ + libnet_t *libnet_handle; + char errbuf[LIBNET_ERRBUF_SIZE]; } instanceData; #define DFLT_SOURCE_PORT_START 32000 @@ -116,6 +121,22 @@ typedef struct configSettings_s { } configSettings_t; static configSettings_t cs; +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "target", eCmdHdlrGetWord, 1 }, + { "port", eCmdHdlrGetWord, 0 }, + { "sourcetemplate", eCmdHdlrGetWord, 0 }, + { "sourceport.start", eCmdHdlrInt, 0 }, + { "sourceport.end", eCmdHdlrInt, 0 }, + { "mtu", eCmdHdlrInt, 0 }, + { "template", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + /* module-global parameters */ static struct cnfparamdescr modpdescr[] = { { "template", eCmdHdlrGetWord, 0 }, @@ -148,8 +169,6 @@ ENDinitConfVars /* add some variables needed for libnet */ -libnet_t *libnet_handle; -char errbuf[LIBNET_ERRBUF_SIZE]; pthread_mutex_t mutLibnet; /* forward definitions */ @@ -291,6 +310,9 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance + pData->libnet_handle = NULL; + pData->mtu = 1500; + pData->bReportLibnetInitErr = 1; ENDcreateInstance @@ -305,8 +327,12 @@ BEGINfreeInstance CODESTARTfreeInstance /* final cleanup */ closeUDPSockets(pData); + free(pData->tplName); free(pData->port); free(pData->host); + free(pData->sourceTpl); + if(pData->libnet_handle != NULL) + libnet_destroy(pData->libnet_handle); ENDfreeInstance @@ -331,16 +357,20 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) libnet_ptag_t ip, ipo; libnet_ptag_t udp; sbool bNeedUnlock = 0; + /* hdrOffs = fragmentation flags + offset (in bytes) + * divided by 8 */ + unsigned msgOffs, hdrOffs; + unsigned maxPktLen, pktLen; DEFiRet; if(pData->pSockArray == NULL) { CHKiRet(doTryResume(pData)); } - if(len > 1472) { - DBGPRINTF("omudpspoof: msg with length %d truncated to 1472 bytes: '%.768s'\n", + if(len > 65528) { + DBGPRINTF("omudpspoof: msg with length %d truncated to 64k: '%.768s'\n", len, msg); - len = 1472; + len = 65528; } ip = ipo = udp = 0; @@ -353,28 +383,43 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) bSendSuccess = RSFALSE; d_pthread_mutex_lock(&mutLibnet); bNeedUnlock = 1; - for (r = pData->f_addr; r; r = r->ai_next) { + for (r = pData->f_addr; r && bSendSuccess == RSFALSE ; r = r->ai_next) { tempaddr = (struct sockaddr_in *)r->ai_addr; - libnet_clear_packet(libnet_handle); + /* Getting max payload size (must be multiple of 8) */ + maxPktLen = (pData->mtu - LIBNET_IPV4_H) & ~0x07; + msgOffs = 0; + /* We're doing (payload size - UDP header size) and not + * checking if it's a multiple of 8 because we know the + * header is 8 bytes long */ + if(len > (maxPktLen - LIBNET_UDP_H) ) { + hdrOffs = IP_MF; + pktLen = maxPktLen - LIBNET_UDP_H; + } else { + hdrOffs = 0; + pktLen = len; + } + DBGPRINTF("omudpspoof: stage 1: MF:%d, hdrOffs %d, pktLen %d\n", + (hdrOffs & IP_MF) >> 13, (hdrOffs & 0x1FFF) << 3, pktLen); + libnet_clear_packet(pData->libnet_handle); /* note: libnet does need ports in host order NOT in network byte order! -- rgerhards, 2009-11-12 */ udp = libnet_build_udp( ntohs(pData->sourcePort),/* source port */ ntohs(tempaddr->sin_port),/* destination port */ - LIBNET_UDP_H + len, /* packet length */ + pktLen+LIBNET_UDP_H, /* packet length */ 0, /* checksum */ (u_char*)msg, /* payload */ - len, /* payload size */ - libnet_handle, /* libnet handle */ + pktLen, /* payload size */ + pData->libnet_handle, /* libnet handle */ udp); /* libnet id */ if (udp == -1) { - DBGPRINTF("Can't build UDP header: %s\n", libnet_geterror(libnet_handle)); + DBGPRINTF("omudpspoof: can't build UDP header: %s\n", libnet_geterror(pData->libnet_handle)); } ip = libnet_build_ipv4( - LIBNET_IPV4_H + len + LIBNET_UDP_H, /* length */ + LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, /* length */ 0, /* TOS */ 242, /* IP ID */ - 0, /* IP Frag */ + hdrOffs, /* IP Frag */ 64, /* TTL */ IPPROTO_UDP, /* protocol */ 0, /* checksum */ @@ -382,28 +427,84 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) tempaddr->sin_addr.s_addr, NULL, /* payload */ 0, /* payload size */ - libnet_handle, /* libnet handle */ + pData->libnet_handle, /* libnet handle */ ip); /* libnet id */ if (ip == -1) { - DBGPRINTF("Can't build IP header: %s\n", libnet_geterror(libnet_handle)); + DBGPRINTF("omudpspoof: can't build IP header: %s\n", libnet_geterror(pData->libnet_handle)); } /* Write it to the wire. */ - lsent = libnet_write(libnet_handle); - if (lsent == -1) { - DBGPRINTF("omudpspoof: write error: %s\n", libnet_geterror(libnet_handle)); + lsent = libnet_write(pData->libnet_handle); + if(lsent != (int) (LIBNET_IPV4_H+LIBNET_UDP_H+pktLen)) { + /* note: access to fd is a libnet internal. If a newer version of libnet does + * not expose that member, we should simply remove it. However, while it is there + * it is useful for consolidating with strace output. + */ + DBGPRINTF("omudpspoof: write error (total len %d): pktLen %d, sent %d, fd %d: %s\n", + len, LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, lsent, pData->libnet_handle->fd, + libnet_geterror(pData->libnet_handle)); + if(lsent != -1) { + bSendSuccess = RSTRUE; + } } else { bSendSuccess = RSTRUE; - break; } - } - /* finished looping */ - if (bSendSuccess == RSFALSE) { - DBGPRINTF("error forwarding via udp, suspending\n"); - iRet = RS_RET_SUSPENDED; + msgOffs += pktLen; + + /* We need to get rid of the UDP header to build the other fragments */ + libnet_clear_packet(pData->libnet_handle); + ip = LIBNET_PTAG_INITIALIZER; + while(len > msgOffs ) { /* loop until all payload is sent */ + /* check if there will be more fragments */ + if((len - msgOffs) > maxPktLen) { + /* In IP's eyes, the UDP header in the first packet + * needs to be in the offset, so we add its size to + * the payload offset here */ + hdrOffs = IP_MF + (msgOffs + LIBNET_UDP_H)/8; + pktLen = maxPktLen; + } else { + /* See above */ + hdrOffs = (msgOffs + LIBNET_UDP_H)/8; + pktLen = len - msgOffs; + } + DBGPRINTF("omudpspoof: stage 2: MF:%d, hdrOffs %d, pktLen %d\n", + (hdrOffs & IP_MF) >> 13, (hdrOffs & 0x1FFF) << 3, pktLen); + ip = libnet_build_ipv4( + LIBNET_IPV4_H + pktLen, /* length */ + 0, /* TOS */ + 242, /* IP ID */ + hdrOffs, /* IP Frag */ + 64, /* TTL */ + IPPROTO_UDP, /* protocol */ + 0, /* checksum */ + source_ip.sin_addr.s_addr, + tempaddr->sin_addr.s_addr, + (uint8_t*)(msg+msgOffs), /* payload */ + pktLen, /* payload size */ + pData->libnet_handle, /* libnet handle */ + ip); /* libnet id */ + if (ip == -1) { + DBGPRINTF("omudpspoof: can't build IP fragment header: %s\n", libnet_geterror(pData->libnet_handle)); + } + /* Write it to the wire. */ + lsent = libnet_write(pData->libnet_handle); + if(lsent != (int) (LIBNET_IPV4_H+pktLen)) { + DBGPRINTF("omudpspoof: fragment write error len %d, sent %d: %s\n", + LIBNET_IPV4_H+LIBNET_UDP_H+len, lsent, libnet_geterror(pData->libnet_handle)); + bSendSuccess = RSFALSE; + continue; + } + msgOffs += pktLen; + } } finalize_it: + if(iRet != RS_RET_OK) { + if(pData->libnet_handle != NULL) { + libnet_destroy(pData->libnet_handle); + pData->libnet_handle = NULL; + } + } if(bNeedUnlock) { d_pthread_mutex_unlock(&mutLibnet); } @@ -424,8 +525,32 @@ static rsRetVal doTryResume(instanceData *pData) if(pData->pSockArray != NULL) FINALIZE; + if(pData->host == NULL) + ABORT_FINALIZE(RS_RET_DISABLE_ACTION); + + if(pData->libnet_handle == NULL) { + /* Initialize the libnet library. Root priviledges are required. + * this initializes a IPv4 socket to use for forging UDP packets. + */ + pData->libnet_handle = libnet_init( + LIBNET_RAW4, /* injection type */ + NULL, /* network interface */ + pData->errbuf); /* errbuf */ + + if(pData->libnet_handle == NULL) { + if(pData->bReportLibnetInitErr) { + errmsg.LogError(0, RS_RET_ERR_LIBNET_INIT, "omudpsoof: error " + "initializing libnet - are you running as root?"); + pData->bReportLibnetInitErr = 0; + } + ABORT_FINALIZE(RS_RET_ERR_LIBNET_INIT); + } + } + DBGPRINTF("omudpspoof: libnit_init() ok\n"); + pData->bReportLibnetInitErr = 1; + /* The remote address is not yet known and needs to be obtained */ - DBGPRINTF(" %s\n", pData->host); + DBGPRINTF("omudpspoof trying resume for '%s'\n", pData->host); memset(&hints, 0, sizeof(hints)); /* port must be numeric, because config file syntax requires this */ hints.ai_flags = AI_NUMERICSERV; @@ -446,7 +571,8 @@ finalize_it: freeaddrinfo(pData->f_addr); pData->f_addr = NULL; } - iRet = RS_RET_SUSPENDED; + if(iRet != RS_RET_DISABLE_ACTION) + iRet = RS_RET_SUSPENDED; } RETiRet; @@ -460,67 +586,95 @@ ENDtryResume BEGINdoAction char *psz; /* temporary buffering */ - register unsigned l; + unsigned l; int iMaxLine; CODESTARTdoAction CHKiRet(doTryResume(pData)); - iMaxLine = glbl.GetMaxLine(); - - DBGPRINTF(" %s:%s/udpspoofs\n", pData->host, getFwdPt(pData)); + DBGPRINTF(" %s:%s/omudpspoof, src '%s', msg strt '%.256s'\n", pData->host, + getFwdPt(pData), ppString[1], ppString[0]); + iMaxLine = glbl.GetMaxLine(); psz = (char*) ppString[0]; l = strlen((char*) psz); if((int) l > iMaxLine) l = iMaxLine; -# ifdef USE_NETZIP - /* Check if we should compress and, if so, do it. We also - * check if the message is large enough to justify compression. - * The smaller the message, the less likely is a gain in compression. - * To save CPU cycles, we do not try to compress very small messages. - * What "very small" means needs to be configured. Currently, it is - * hard-coded but this may be changed to a config parameter. - * rgerhards, 2006-11-30 - */ - if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { - Bytef *out; - uLongf destLen = iMaxLine + iMaxLine/100 +12; /* recommended value from zlib doc */ - uLong srcLen = l; - int ret; - /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ - CHKmalloc(out = (Bytef*) MALLOC(destLen)); - out[0] = 'z'; - out[1] = '\0'; - ret = compress2((Bytef*) out+1, &destLen, (Bytef*) psz, - srcLen, pData->compressionLevel); - DBGPRINTF("Compressing message, length was %d now %d, return state %d.\n", - l, (int) destLen, ret); - if(ret != Z_OK) { - /* if we fail, we complain, but only in debug mode - * Otherwise, we are silent. In any case, we ignore the - * failed compression and just sent the uncompressed - * data, which is still valid. So this is probably the - * best course of action. - * rgerhards, 2006-11-30 - */ - DBGPRINTF("Compression failed, sending uncompressed message\n"); - } else if(destLen+1 < l) { - /* only use compression if there is a gain in using it! */ - DBGPRINTF("there is gain in compression, so we do it\n"); - psz = (char*) out; - l = destLen + 1; /* take care for the "z" at message start! */ - } - ++destLen; - } -# endif - CHKiRet(UDPSend(pData, ppString[1], psz, l)); finalize_it: ENDdoAction +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->tplName = NULL; + pData->sourcePortStart = DFLT_SOURCE_PORT_START; + pData->sourcePortEnd = DFLT_SOURCE_PORT_END; + pData->host = NULL; + pData->port = NULL; + pData->sourceTpl = (uchar*) strdup("RSYSLOG_omudpspoofDfltSourceTpl"); + pData->mtu = 1500; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + uchar *tplToUse; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (omudpspoof)\n"); + + pvals = nvlstGetParams(lst, &actpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "omudpspoof: mandatory " + "parameters missing"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(Debug) { + dbgprintf("action param blk in omudpspoof:\n"); + cnfparamsPrint(&actpblk, pvals); + } + + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "target")) { + pData->host = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "port")) { + pData->port = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "sourcetemplate")) { + free(pData->sourceTpl); + pData->sourceTpl = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "sourceport.start")) { + pData->sourcePortStart = (int) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "sourceport.end")) { + pData->sourcePortEnd = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "mtu")) { + pData->mtu = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "template")) { + pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + DBGPRINTF("omudpspoof: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + CODE_STD_STRING_REQUESTnewActInst(2) + pData->sourcePort = pData->sourcePortStart; + + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); + CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->sourceTpl), OMSR_NO_RQD_TPL_OPTS)); + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + BEGINparseSelectorAct uchar *sourceTpl; CODESTARTparseSelectorAct @@ -578,7 +732,6 @@ freeConfigVars(void) BEGINmodExit CODESTARTmodExit /* destroy the libnet state needed for forged UDP sources */ - libnet_destroy(libnet_handle); pthread_mutex_destroy(&mutLibnet); /* release what we no longer need */ objRelease(errmsg, CORE_COMPONENT); @@ -591,6 +744,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES ENDqueryEtryPt @@ -618,18 +772,6 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net,LM_NET_FILENAME)); - /* Initialize the libnet library. Root priviledges are required. - * this initializes a IPv4 socket to use for forging UDP packets. - */ - libnet_handle = libnet_init( - LIBNET_RAW4, /* injection type */ - NULL, /* network interface */ - errbuf); /* errbuf */ - - if(libnet_handle == NULL) { - errmsg.LogError(0, NO_ERRCODE, "Error initializing libnet, can not continue "); - ABORT_FINALIZE(RS_RET_ERR_LIBNET_INIT); - } pthread_mutex_init(&mutLibnet, NULL); CHKiRet(regCfSysLineHdlr((uchar *)"actionomudpspoofdefaulttemplate", 0, eCmdHdlrGetWord, setLegacyDfltTpl, NULL, NULL)); diff --git a/plugins/omzmq3/README b/plugins/omzmq3/README index ccc96c74..c2a33555 100644 --- a/plugins/omzmq3/README +++ b/plugins/omzmq3/README @@ -1,16 +1,10 @@ ZeroMQ 3.x Output Plugin Building this plugin: -Requires libzmq and libczmq. First, install libzmq from the HEAD on github: -http://github.com/zeromq/libzmq. You can clone the repository, build, then -install it. The directions for doing so are there in the readme. Then, do -the same for libczmq: http://github.com/zeromq/czmq. At some point, the 3.1 -version of libzmq will be released, and a supporting version of libczmq. -At that time, you could simply download and install the tarballs instead of -using git to clone the repositories. Those tarballs (when available) can -be found at http://download.zeromq.org. As of this writing (5/31/2012), the -most recent version of czmq (1.1.0) and libzmq (3.1.0-beta) will not compile -properly. +Requires libzmq and libczmq. First, download the tarballs of both libzmq +and its supporting libczmq from http://download.zeromq.org. As of this +writing (04/23/2013), the most recent versions of libzmq and czmq are +3.2.2 and 1.3.2 respectively. Configure, build, and then install both libs. Omzmq3 allows you to push data out of rsyslog from a zeromq socket. The example below binds a PUB socket to port 7171, and any message fitting the criteria will diff --git a/plugins/omzmq3/omzmq3.c b/plugins/omzmq3/omzmq3.c index ee6756b9..c8552f11 100644 --- a/plugins/omzmq3/omzmq3.c +++ b/plugins/omzmq3/omzmq3.c @@ -110,9 +110,10 @@ static zctx_t* s_context = NULL; static int s_workerThreads = -1; static struct socket_type types[] = { - {"PUB", ZMQ_PUB }, - {"PUSH", ZMQ_PUSH }, - {"XPUB", ZMQ_XPUB } + {"PUB", ZMQ_PUB }, + {"PUSH", ZMQ_PUSH }, + {"DEALER", ZMQ_DEALER }, + {"XPUB", ZMQ_XPUB } }; static struct socket_action actions[] = { @@ -201,17 +202,18 @@ static rsRetVal initZMQ(instanceData* pData) { /* create the context if necessary. */ if (NULL == s_context) { + zsys_handler_set(NULL); s_context = zctx_new(); if (s_workerThreads > 0) zctx_set_iothreads(s_context, s_workerThreads); } pData->socket = zsocket_new(s_context, pData->type); - - /* ALWAYS set the HWM as the zmq3 default is 1000 and we default - to 0 (infinity) */ - zsocket_set_rcvhwm(pData->socket, pData->rcvHWM); - zsocket_set_sndhwm(pData->socket, pData->sndHWM); - + if (NULL == pData->socket) { + errmsg.LogError(0, RS_RET_NO_ERRCODE, + "omzmq3: zsocket_new failed for %s: %s", + pData->description, zmq_strerror(errno)); + ABORT_FINALIZE(RS_RET_NO_ERRCODE); + } /* use czmq defaults for these, unless set to non-default values */ if(pData->identity) zsocket_set_identity(pData->socket, (char*)pData->identity); if(pData->sndBuf > -1) zsocket_set_sndbuf(pData->socket, pData->sndBuf); @@ -228,17 +230,26 @@ static rsRetVal initZMQ(instanceData* pData) { if(pData->reconnectIVLMax > -1) zsocket_set_reconnect_ivl_max(pData->socket, pData->reconnectIVLMax); if(pData->ipv4Only > -1) zsocket_set_ipv4only(pData->socket, pData->ipv4Only); if(pData->affinity != 1) zsocket_set_affinity(pData->socket, pData->affinity); - + if(pData->rcvHWM > -1) zsocket_set_rcvhwm(pData->socket, pData->rcvHWM); + if(pData->sndHWM > -1) zsocket_set_sndhwm(pData->socket, pData->sndHWM); + /* bind or connect to it */ if (pData->action == ACTION_BIND) { /* bind asserts, so no need to test return val here which isn't the greatest api -- oh well */ - zsocket_bind(pData->socket, (char*)pData->description); + if(-1 == zsocket_bind(pData->socket, (char*)pData->description)) { + errmsg.LogError(0, RS_RET_NO_ERRCODE, "omzmq3: bind failed for %s: %s", + pData->description, zmq_strerror(errno)); + ABORT_FINALIZE(RS_RET_NO_ERRCODE); + } + DBGPRINTF("omzmq3: bind to %s successful\n",pData->description); } else { - if(zsocket_connect(pData->socket, (char*)pData->description) == -1) { - errmsg.LogError(0, RS_RET_SUSPENDED, "omzmq3: connect failed!"); - ABORT_FINALIZE(RS_RET_SUSPENDED); + if(-1 == zsocket_connect(pData->socket, (char*)pData->description)) { + errmsg.LogError(0, RS_RET_NO_ERRCODE, "omzmq3: connect failed for %s: %s", + pData->description, zmq_strerror(errno)); + ABORT_FINALIZE(RS_RET_NO_ERRCODE); } + DBGPRINTF("omzmq3: connect to %s successful", pData->description); } finalize_it: RETiRet; @@ -256,7 +267,7 @@ rsRetVal writeZMQ(uchar* msg, instanceData* pData) { /* whine if things went wrong */ if (result == -1) { - errmsg.LogError(0, NO_ERRCODE, "omzmq3: send of %s failed with return %d", msg, result); + errmsg.LogError(0, NO_ERRCODE, "omzmq3: send of %s failed: %s", msg, zmq_strerror(errno)); ABORT_FINALIZE(RS_RET_ERR); } finalize_it: @@ -265,13 +276,13 @@ rsRetVal writeZMQ(uchar* msg, instanceData* pData) { static inline void setInstParamDefaults(instanceData* pData) { - pData->description = (uchar*)"tcp://*:7171"; + pData->description = NULL; pData->socket = NULL; pData->tplName = NULL; pData->type = ZMQ_PUB; pData->action = ACTION_BIND; - pData->sndHWM = 0; /*unlimited*/ - pData->rcvHWM = 0; /*unlimited*/ + pData->sndHWM = -1; + pData->rcvHWM = -1; pData->identity = NULL; pData->sndBuf = -1; pData->rcvBuf = -1; @@ -314,6 +325,7 @@ CODESTARTfreeInstance closeZMQ(pData); free(pData->description); free(pData->tplName); + free(pData->identity); ENDfreeInstance BEGINtryResume @@ -329,88 +341,90 @@ ENDdoAction BEGINnewActInst - struct cnfparamvals *pvals; - int i; + struct cnfparamvals *pvals; + int i; CODESTARTnewActInst -if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { - ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); - } + if ((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } CHKiRet(createInstance(&pData)); setInstParamDefaults(pData); CODE_STD_STRING_REQUESTnewActInst(1) -for(i = 0 ; i < actpblk.nParams ; ++i) { - if(!pvals[i].bUsed) - continue; - if(!strcmp(actpblk.descr[i].name, "description")) { - pData->description = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(actpblk.descr[i].name, "template")) { - pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(actpblk.descr[i].name, "sockType")){ - pData->type = getSocketType(es_str2cstr(pvals[i].val.d.estr, NULL)); - } else if(!strcmp(actpblk.descr[i].name, "action")){ - pData->action = getSocketAction(es_str2cstr(pvals[i].val.d.estr, NULL)); - } else if(!strcmp(actpblk.descr[i].name, "sndHWM")) { - pData->sndHWM = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "rcvHWM")) { - pData->rcvHWM = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "identity")){ - pData->identity = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(actpblk.descr[i].name, "sndBuf")) { - pData->sndBuf = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "rcvBuf")) { - pData->rcvBuf = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "linger")) { - pData->linger = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "backlog")) { - pData->backlog = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "sndTimeout")) { - pData->sndTimeout = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "rcvTimeout")) { - pData->rcvTimeout = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "maxMsgSize")) { - pData->maxMsgSize = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "rate")) { - pData->rate = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "recoveryIVL")) { - pData->recoveryIVL = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "multicastHops")) { - pData->multicastHops = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "reconnectIVL")) { - pData->reconnectIVL = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "reconnectIVLMax")) { - pData->reconnectIVLMax = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "ipv4Only")) { - pData->ipv4Only = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "affinity")) { - pData->affinity = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "globalWorkerThreads")) { - s_workerThreads = (int) pvals[i].val.d.n; - } else { - errmsg.LogError(0, NO_ERRCODE, "omzmq3: program error, non-handled " - "param '%s'\n", actpblk.descr[i].name); + for (i = 0; i < actpblk.nParams; ++i) { + if (!pvals[i].bUsed) + continue; + if (!strcmp(actpblk.descr[i].name, "description")) { + pData->description = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "template")) { + pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "sockType")){ + pData->type = getSocketType(es_str2cstr(pvals[i].val.d.estr, NULL)); + } else if (!strcmp(actpblk.descr[i].name, "action")){ + pData->action = getSocketAction(es_str2cstr(pvals[i].val.d.estr, NULL)); + } else if (!strcmp(actpblk.descr[i].name, "sndHWM")) { + pData->sndHWM = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "rcvHWM")) { + pData->rcvHWM = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "identity")){ + pData->identity = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if (!strcmp(actpblk.descr[i].name, "sndBuf")) { + pData->sndBuf = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "rcvBuf")) { + pData->rcvBuf = (int) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "linger")) { + pData->linger = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "backlog")) { + pData->backlog = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "sndTimeout")) { + pData->sndTimeout = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "rcvTimeout")) { + pData->rcvTimeout = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "maxMsgSize")) { + pData->maxMsgSize = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "rate")) { + pData->rate = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "recoveryIVL")) { + pData->recoveryIVL = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "multicastHops")) { + pData->multicastHops = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "reconnectIVL")) { + pData->reconnectIVL = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "reconnectIVLMax")) { + pData->reconnectIVLMax = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "ipv4Only")) { + pData->ipv4Only = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "affinity")) { + pData->affinity = (int) pvals[i].val.d.n; + } else if (!strcmp(actpblk.descr[i].name, "globalWorkerThreads")) { + s_workerThreads = (int) pvals[i].val.d.n; + } else { + errmsg.LogError(0, NO_ERRCODE, "omzmq3: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } } - } - -if(pData->tplName == NULL) { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); - } else { - CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*)pData->tplName, OMSR_NO_RQD_TPL_OPTS)); - } - -if(pData->type == -1) { - errmsg.LogError(0, RS_RET_CONFIG_ERROR, "omzmq3: unknown socket type."); - ABORT_FINALIZE(RS_RET_CONFIG_ERROR); - } -if(pData->action == -1) { - errmsg.LogError(0, RS_RET_CONFIG_ERROR, "omzmq3: unknown socket action"); - ABORT_FINALIZE(RS_RET_CONFIG_ERROR); - } + if (pData->tplName == NULL) { + CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*)strdup("RSYSLOG_ForwardFormat"), OMSR_NO_RQD_TPL_OPTS)); + } else { + CHKiRet(OMSRsetEntry(*ppOMSR, 0, (uchar*)pData->tplName, OMSR_NO_RQD_TPL_OPTS)); + } + if (NULL == pData->description) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, "omzmq3: you didn't enter a description"); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + if (pData->type == -1) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, "omzmq3: unknown socket type."); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + if (pData->action == -1) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, "omzmq3: unknown socket action"); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } CODE_STD_FINALIZERnewActInst - cnfparamvalsDestruct(pvals, &actpblk); + cnfparamvalsDestruct(pvals, &actpblk); ENDnewActInst BEGINparseSelectorAct @@ -433,10 +447,10 @@ ENDinitConfVars BEGINmodExit CODESTARTmodExit -if(NULL != s_context) { - zctx_destroy(&s_context); - s_context=NULL; - } + if (NULL != s_context) { + zctx_destroy(&s_context); + s_context=NULL; + } ENDmodExit diff --git a/rsyslog.conf b/rsyslog.conf deleted file mode 100644 index 47fc4402..00000000 --- a/rsyslog.conf +++ /dev/null @@ -1,60 +0,0 @@ -# if you experience problems, check -# http://www.rsyslog.com/troubleshoot for assistance - -# rsyslog v3: load input modules -# If you do not load inputs, nothing happens! -# You may need to set the module load path if modules are not found. - -$ModLoad immark # provides --MARK-- message capability -$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) -$ModLoad imklog # kernel logging (formerly provided by rklogd) - -# Log all kernel messages to the console. -# Logging much else clutters up the screen. -#kern.* /dev/console - -# Log anything (except mail) of level info or higher. -# Don't log private authentication messages! -*.info;mail.none;authpriv.none;cron.none -/var/log/messages - -# The authpriv file has restricted access. -authpriv.* /var/log/secure - -# Log all the mail messages in one place. -mail.* -/var/log/maillog - - -# Log cron stuff -cron.* -/var/log/cron - -# Everybody gets emergency messages -*.emerg * - -# Save news errors of level crit and higher in a special file. -uucp,news.crit -/var/log/spooler - -# Save boot messages also to boot.log -local7.* /var/log/boot.log - -# Remote Logging (we use TCP for reliable delivery) -# An on-disk queue is created for this action. If the remote host is -# down, messages are spooled to disk and sent when it is up again. -#$WorkDirectory /rsyslog/spool # where to place spool files -#$ActionQueueFileName uniqName # unique name prefix for spool files -#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) -#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown -#$ActionQueueType LinkedList # run asynchronously -#$ActionResumeRetryCount -1 # infinite retries if host is down -# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional -#*.* @@remote-host:514 - - -# ######### Receiving Messages from Remote Hosts ########## -# TCP Syslog Server: -# provides TCP syslog reception and GSS-API (if compiled to support it) -#$ModLoad imtcp.so # load module -#$InputTCPServerRun 514 # start up TCP listener at port 514 - -# UDP Syslog Server: -#$ModLoad imudp.so # provides UDP syslog reception -#$UDPServerRun 514 # start a UDP syslog server at standard port 514 diff --git a/rsyslog.service.in b/rsyslog.service.in index 2265a491..8e2d64c2 100644 --- a/rsyslog.service.in +++ b/rsyslog.service.in @@ -1,10 +1,10 @@ [Unit] Description=System Logging Service +Requires=syslog.socket [Service] -ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service +Type=notify ExecStart=@sbindir@/rsyslogd -n -Sockets=syslog.socket StandardOutput=null [Install] diff --git a/runtime/Makefile.am b/runtime/Makefile.am index f49aac91..dea06fe0 100644 --- a/runtime/Makefile.am +++ b/runtime/Makefile.am @@ -17,6 +17,8 @@ librsyslog_la_SOURCES = \ module-template.h \ im-helper.h \ obj-types.h \ + sigprov.h \ + cryprov.h \ nsd.h \ glbl.h \ glbl.c \ @@ -63,16 +65,15 @@ librsyslog_la_SOURCES = \ queue.h \ ruleset.c \ ruleset.h \ - rule.c \ - rule.h \ prop.c \ prop.h \ + ratelimit.c \ + ratelimit.h \ cfsysline.c \ cfsysline.h \ sd-daemon.c \ sd-daemon.h \ \ - \ ../action.h \ ../action.c \ ../threads.c \ @@ -93,14 +94,15 @@ librsyslog_la_SOURCES = \ ../template.h # the files with ../ we need to work on - so that they either become part of the # runtime or will no longer be needed. -- rgerhards, 2008-06-13 +# if WITH_MODDIRS -librsyslog_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -D_PATH_MODDIR=\"$(pkglibdir)/:$(moddirs)\" $(PTHREADS_CFLAGS) $(LIBEE_CFLAGS) -I\$(top_srcdir)/tools +librsyslog_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -D_PATH_MODDIR=\"$(pkglibdir)/:$(moddirs)\" $(PTHREADS_CFLAGS) -I\$(top_srcdir)/tools else -librsyslog_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -D_PATH_MODDIR=\"$(pkglibdir)/\" -I$(top_srcdir) $(PTHREADS_CFLAGS) $(LIBEE_CFLAGS) -I\$(top_srcdir)/tools -I\$(top_srcdir)/grammar +librsyslog_la_CPPFLAGS = -DSD_EXPORT_SYMBOLS -D_PATH_MODDIR=\"$(pkglibdir)/\" -I$(top_srcdir) $(PTHREADS_CFLAGS) -I\$(top_srcdir)/tools -I\$(top_srcdir)/grammar endif #librsyslog_la_LDFLAGS = -module -avoid-version -librsyslog_la_LIBADD = $(DL_LIBS) $(RT_LIBS) $(LIBEE_LIBS) +librsyslog_la_LIBADD = $(DL_LIBS) $(RT_LIBS) # # regular expression support @@ -131,7 +133,7 @@ pkglib_LTLIBRARIES += lmnet.la lmnetstrms.la # lmnet_la_SOURCES = net.c net.h lmnet_la_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) -lmnet_la_LDFLAGS = -module -avoid-version +lmnet_la_LDFLAGS = -module -avoid-version ../compat/compat_la-getifaddrs.lo lmnet_la_LIBADD = # network stream master class and stream factory @@ -173,6 +175,35 @@ lmnsd_gtls_la_LDFLAGS = -module -avoid-version lmnsd_gtls_la_LIBADD = $(GNUTLS_LIBS) endif +# +# support library for libgcrypt +# +if ENABLE_LIBGCRYPT + noinst_LTLIBRARIES += libgcry.la + libgcry_la_SOURCES = libgcry.c libgcry_common.c libgcry.h + libgcry_la_CPPFLAGS = $(RSRT_CFLAGS) $(LIBGCRYPT_CFLAGS) + pkglib_LTLIBRARIES += lmcry_gcry.la + lmcry_gcry_la_SOURCES = lmcry_gcry.c lmcry_gcry.h + lmcry_gcry_la_CPPFLAGS = $(RSRT_CFLAGS) $(LIBGCRYPT_CFLAGS) + lmcry_gcry_la_LDFLAGS = -module -avoid-version + lmcry_gcry_la_LIBADD = libgcry.la $(LIBGCRYPT_LIBS) +endif + + +# +# support library for guardtime +# +if ENABLE_GUARDTIME + noinst_LTLIBRARIES += librsgt.la + librsgt_la_SOURCES = librsgt.c librsgt_read.c librsgt.h + pkglib_LTLIBRARIES += lmsig_gt.la + lmsig_gt_la_SOURCES = lmsig_gt.c lmsig_gt.h + lmsig_gt_la_CPPFLAGS = $(RSRT_CFLAGS) $(GUARDTIME_CFLAGS) + lmsig_gt_la_LDFLAGS = -module -avoid-version + lmsig_gt_la_LIBADD = librsgt.la $(GUARDTIME_LIBS) +endif + + update-systemd: curl http://cgit.freedesktop.org/systemd/systemd/plain/src/libsystemd-daemon/sd-daemon.c > sd-daemon.c curl http://cgit.freedesktop.org/systemd/systemd/plain/src/systemd/sd-daemon.h > sd-daemon.h diff --git a/runtime/batch.h b/runtime/batch.h index 944889bd..2ec07670 100644 --- a/runtime/batch.h +++ b/runtime/batch.h @@ -34,24 +34,21 @@ * main message queue. But over time, it could potentially be useful to split the two. * rgerhad, 2009-05-12 */ -typedef enum { - BATCH_STATE_RDY = 0, /* object ready for processing */ - BATCH_STATE_BAD = 1, /* unrecoverable failure while processing, do NOT resubmit to same action */ - BATCH_STATE_SUB = 2, /* message submitted for processing, outcome yet unknown */ - BATCH_STATE_COMM = 3, /* message successfully commited */ - BATCH_STATE_DISC = 4, /* discarded - processed OK, but do not submit to any other action */ -} batch_state_t; +#define BATCH_STATE_RDY 0 /* object ready for processing */ +#define BATCH_STATE_BAD 1 /* unrecoverable failure while processing, do NOT resubmit to same action */ +#define BATCH_STATE_SUB 2 /* message submitted for processing, outcome yet unknown */ +#define BATCH_STATE_COMM 3 /* message successfully commited */ +#define BATCH_STATE_DISC 4 /* discarded - processed OK, but do not submit to any other action */ +typedef unsigned char batch_state_t; /* an object inside a batch, including any information (state!) needed for it to "life". */ struct batch_obj_s { - obj_t *pUsrp; /* pointer to user object (most often message) */ - batch_state_t state; /* associated state */ + msg_t *pMsg; /* work variables for action processing; these are reused for each action (or block of * actions) */ - sbool bFilterOK; /* work area for filter processing (per action, reused!) */ sbool bPrevWasSuspended; /* following are caches to save allocs if not absolutely necessary */ uchar *staticActStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /**< for strings */ @@ -83,8 +80,16 @@ struct batch_s { int iDoneUpTo; /* all messages below this index have state other than RDY */ qDeqID deqID; /* ID of dequeue operation that generated this batch */ int *pbShutdownImmediate;/* end processing of this batch immediately if set to 1 */ + sbool *active; /* which messages are active for processing, NULL=all */ sbool bSingleRuleset; /* do all msgs of this batch use a single ruleset? */ batch_obj_t *pElem; /* batch elements */ + batch_state_t *eltState;/* state (array!) for individual objects. + NOTE: we have moved this out of batch_obj_t because we + get a *much* better cache hit ratio this way. So do not + move it back into this structure! Note that this is really + a HUGE saving, even if it doesn't look so (both profiler + data as well as practical tests indicate that!). + */ }; @@ -97,13 +102,13 @@ batchSetSingleRuleset(batch_t *pBatch, sbool val) { /* get the batches ruleset (if we have a single ruleset) */ static inline ruleset_t* batchGetRuleset(batch_t *pBatch) { - return (pBatch->nElem > 0) ? ((msg_t*) pBatch->pElem[0].pUsrp)->pRuleset : NULL; + return (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; } /* get the ruleset of a specifc element of the batch (index not verified!) */ static inline ruleset_t* batchElemGetRuleset(batch_t *pBatch, int i) { - return ((msg_t*) pBatch->pElem[i].pUsrp)->pRuleset; + return pBatch->pElem[i].pMsg->pRuleset; } /* get number of msgs for this batch */ @@ -119,8 +124,8 @@ batchNumMsgs(batch_t *pBatch) { */ static inline void batchSetElemState(batch_t *pBatch, int i, batch_state_t newState) { - if(pBatch->pElem[i].state != BATCH_STATE_DISC) - pBatch->pElem[i].state = newState; + if(pBatch->eltState[i] != BATCH_STATE_DISC) + pBatch->eltState[i] = newState; } @@ -129,23 +134,8 @@ batchSetElemState(batch_t *pBatch, int i, batch_state_t newState) { */ static inline int batchIsValidElem(batch_t *pBatch, int i) { - return(pBatch->pElem[i].bFilterOK && pBatch->pElem[i].state != BATCH_STATE_DISC); -} - - -/* copy one batch element to another. - * This creates a complete duplicate in those cases where - * it is needed. Use duplication only when absolutely necessary! - * Note that all working fields are reset to zeros. If that were - * not done, we would have potential problems with invalid - * or double pointer frees. - * rgerhards, 2010-06-10 - */ -static inline void -batchCopyElem(batch_obj_t *pDest, batch_obj_t *pSrc) { - memset(pDest, 0, sizeof(batch_obj_t)); - pDest->pUsrp = pSrc->pUsrp; - pDest->state = pSrc->state; + return( (pBatch->eltState[i] != BATCH_STATE_DISC) + && (pBatch->active == NULL || pBatch->active[i])); } @@ -166,6 +156,7 @@ batchFree(batch_t *pBatch) { } } free(pBatch->pElem); + free(pBatch->eltState); } @@ -179,6 +170,7 @@ batchInit(batch_t *pBatch, int maxElem) { pBatch->iDoneUpTo = 0; pBatch->maxElem = maxElem; CHKmalloc(pBatch->pElem = calloc((size_t)maxElem, sizeof(batch_obj_t))); + CHKmalloc(pBatch->eltState = calloc((size_t)maxElem, sizeof(batch_state_t))); // TODO: replace calloc by inidividual writes? finalize_it: RETiRet; diff --git a/runtime/conf.c b/runtime/conf.c index 488d1b86..c3c7e447 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -61,17 +61,16 @@ #include "srUtils.h" #include "errmsg.h" #include "net.h" -#include "rule.h" #include "ruleset.h" #include "rsconf.h" #include "unicode-helper.h" +#include "rainerscript.h" #ifdef OS_SOLARIS # define NAME_MAX MAXNAMELEN #endif /* forward definitions */ -//static rsRetVal cfline(rsconf_t *conf, uchar *line, rule_t **pfCurr); /* static data */ @@ -79,7 +78,6 @@ DEFobjStaticHelpers DEFobjCurrIf(module) DEFobjCurrIf(errmsg) DEFobjCurrIf(net) -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) int bConfStrictScoping = 0; /* force strict scoping during config processing? */ @@ -131,6 +129,23 @@ finalize_it: } +/* remove leading spaces from name; this "fixes" some anomalies in + * getSubString(), but I was not brave enough to fix the former as + * it has many other callers... -- rgerhards, 2013-05-27 + */ +static inline void +ltrim(char *src) +{ + char *dst = src; + while(isspace(*src)) + ++src; /*SKIP*/; + if(dst != src) { + while(*src != '\0') + *dst++ = *src++; + *dst = '\0'; + } +} + /* parse and interpret a $-config line that starts with * a name (this is common code). It is parsed to the name * and then the proper sub-function is called to handle @@ -157,6 +172,7 @@ doNameLine(uchar **pp, void* pVal) errmsg.LogError(0, RS_RET_NOT_FOUND, "Invalid config line: could not extract name - line ignored"); ABORT_FINALIZE(RS_RET_NOT_FOUND); } + ltrim(szName); if(*p == ',') ++p; /* comma was eaten */ @@ -326,14 +342,9 @@ cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int } -/* Helper to cfline(). This function takes the filter part of a traditional, PRI - * based line and decodes the PRIs given in the selector line. It processed the - * line up to the beginning of the action part. A pointer to that beginnig is - * passed back to the caller. - * rgerhards 2005-09-15 - */ +/* Decode a traditional PRI filter */ /* GPLv3 - stems back to sysklogd */ -rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) +rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]) { uchar *p; register uchar *q; @@ -347,22 +358,15 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) DEFiRet; ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ISOBJ_TYPE_assert(pRule, rule); - dbgprintf(" - traditional PRI filter '%s'\n", *pline); - errno = 0; /* keep strerror_r() stuff out of logerror messages */ + dbgprintf("Decoding traditional PRI filter '%s'\n", pline); - pRule->f_filter_type = FILTER_PRI; - /* Note: file structure is pre-initialized to zero because it was - * created with calloc()! - */ for (i = 0; i <= LOG_NFACILITIES; i++) { - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; } /* scan through the list of selectors */ - for (p = *pline; *p && *p != '\t' && *p != ' ';) { + for (p = pline; *p && *p != '\t' && *p != ' ';) { /* find the end of this facility name list */ for (q = p; *q && *q != '\t' && *q++ != '.'; ) continue; @@ -411,28 +415,28 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) for (i = 0; i <= LOG_NFACILITIES; i++) { if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pmask[i] = TABLE_ALLPRI; else - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] &= ~(1<<pri); + pmask[i] &= ~(1<<pri); else - pRule->f_filterData.f_pmask[i] |= (1<<pri); + pmask[i] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; + pmask[i] = TABLE_NOPRI; else - pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pmask[i] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i] &= ~(1<<i2); + pmask[i] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i] |= (1<<i2); + pmask[i] |= (1<<i2); } } } @@ -447,27 +451,27 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pmask[i >> 3] = TABLE_ALLPRI; else - pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pmask[i >> 3] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<pri); + pmask[i >> 3] &= ~(1<<pri); else - pRule->f_filterData.f_pmask[i >> 3] |= (1<<pri); + pmask[i >> 3] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pmask[i >> 3] = TABLE_NOPRI; else - pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pmask[i >> 3] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<i2); + pmask[i >> 3] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - pRule->f_filterData.f_pmask[i >> 3] |= (1<<i2); + pmask[i >> 3] |= (1<<i2); } } } @@ -478,11 +482,6 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) p = q; } - /* skip to action part */ - while (*p == '\t' || *p == ' ') - p++; - - *pline = p; RETiRet; } @@ -492,7 +491,7 @@ rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) * of the action part. A pointer to that beginnig is passed back to the caller. * rgerhards 2005-09-15 */ -rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) +rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt) { rsParsObj *pPars; cstr_t *pCSCompOp; @@ -501,16 +500,11 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) int iOffset; /* for compare operations */ ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(f != NULL); - dbgprintf(" - property-based filter '%s'\n", *pline); - errno = 0; /* keep strerror_r() stuff out of logerror messages */ - - f->f_filter_type = FILTER_PROP; + dbgprintf("Decoding property-based filter '%s'\n", pline); /* create parser object starting with line string without leading colon */ - if((iRet = rsParsConstructFromSz(&pPars, (*pline)+1)) != RS_RET_OK) { + if((iRet = rsParsConstructFromSz(&pPars, pline+1)) != RS_RET_OK) { errmsg.LogError(0, iRet, "Error %d constructing parser object - ignoring selector", iRet); return(iRet); } @@ -522,15 +516,15 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) rsParsDestruct(pPars); return(iRet); } - iRet = propNameToID(pCSPropName, &f->f_filterData.prop.propID); + iRet = propNameToID(pCSPropName, &stmt->d.s_propfilt.propID); if(iRet != RS_RET_OK) { errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); rsParsDestruct(pPars); return(iRet); } - if(f->f_filterData.prop.propID == PROP_CEE) { + if(stmt->d.s_propfilt.propID == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - if((f->f_filterData.prop.propName = + if((stmt->d.s_propfilt.propName = es_newStrFromBuf((char*)cstrGetSzStrNoNULL(pCSPropName)+2, cstrLen(pCSPropName)-2)) == NULL) { cstrDestruct(&pCSPropName); return(RS_RET_ERR); @@ -553,38 +547,38 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) */ if(rsCStrLen(pCSCompOp) > 0) { if(*rsCStrGetBufBeg(pCSCompOp) == '!') { - f->f_filterData.prop.isNegated = 1; + stmt->d.s_propfilt.isNegated = 1; iOffset = 1; /* ignore '!' */ } else { - f->f_filterData.prop.isNegated = 0; + stmt->d.s_propfilt.isNegated = 0; iOffset = 0; } } else { - f->f_filterData.prop.isNegated = 0; + stmt->d.s_propfilt.isNegated = 0; iOffset = 0; } if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "contains", 8)) { - f->f_filterData.prop.operation = FIOP_CONTAINS; + stmt->d.s_propfilt.operation = FIOP_CONTAINS; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isequal", 7)) { - f->f_filterData.prop.operation = FIOP_ISEQUAL; + stmt->d.s_propfilt.operation = FIOP_ISEQUAL; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isempty", 7)) { - f->f_filterData.prop.operation = FIOP_ISEMPTY; + stmt->d.s_propfilt.operation = FIOP_ISEMPTY; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "startswith", 10)) { - f->f_filterData.prop.operation = FIOP_STARTSWITH; + stmt->d.s_propfilt.operation = FIOP_STARTSWITH; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "regex", 5)) { - f->f_filterData.prop.operation = FIOP_REGEX; + stmt->d.s_propfilt.operation = FIOP_REGEX; } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "ereregex", 8)) { - f->f_filterData.prop.operation = FIOP_EREREGEX; + stmt->d.s_propfilt.operation = FIOP_EREREGEX; } else { errmsg.LogError(0, NO_ERRCODE, "error: invalid compare operation '%s' - ignoring selector", (char*) rsCStrGetSzStrNoNULL(pCSCompOp)); } rsCStrDestruct(&pCSCompOp); /* no longer needed */ - if(f->f_filterData.prop.operation != FIOP_ISEMPTY) { + if(stmt->d.s_propfilt.operation != FIOP_ISEMPTY) { /* read compare value */ - iRet = parsQuotedCStr(pPars, &f->f_filterData.prop.pCSCompValue); + iRet = parsQuotedCStr(pPars, &stmt->d.s_propfilt.pCSCompValue); if(iRet != RS_RET_OK) { errmsg.LogError(0, iRet, "error %d compare value property - ignoring selector", iRet); rsParsDestruct(pPars); @@ -592,114 +586,10 @@ rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) } } - /* skip to action part */ - if((iRet = parsSkipWhitespace(pPars)) != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d skipping to action part - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - - /* cleanup */ - *pline = *pline + rsParsGetParsePointer(pPars) + 1; - /* we are adding one for the skipped initial ":" */ - return rsParsDestruct(pPars); } -/* - * Helper to cfline(). This function interprets a BSD host selector line - * from the config file ("+/-hostname"). It stores it for further reference. - * rgerhards 2005-10-19 - */ -rsRetVal cflineProcessHostSelector(uchar **pline) -{ - DEFiRet; - - ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(**pline == '-' || **pline == '+'); - - dbgprintf(" - host selector line\n"); - - /* check include/exclude setting */ - if(**pline == '+') { - eDfltHostnameCmpMode = HN_COMP_MATCH; - } else { /* we do not check for '-', it must be, else we wouldn't be here */ - eDfltHostnameCmpMode = HN_COMP_NOMATCH; - } - (*pline)++; /* eat + or - */ - - /* the below is somewhat of a quick hack, but it is efficient (this is - * why it is in here. "+*" resets the tag selector with BSD syslog. We mimic - * this, too. As it is easy to check that condition, we do not fire up a - * parser process, just make sure we do not address beyond our space. - * Order of conditions in the if-statement is vital! rgerhards 2005-10-18 - */ - if(**pline != '\0' && **pline == '*' && *(*pline+1) == '\0') { - dbgprintf("resetting BSD-like hostname filter\n"); - eDfltHostnameCmpMode = HN_NO_COMP; - if(pDfltHostnameCmp != NULL) { - CHKiRet(rsCStrSetSzStr(pDfltHostnameCmp, NULL)); - } - } else { - dbgprintf("setting BSD-like hostname filter to '%s'\n", *pline); - if(pDfltHostnameCmp == NULL) { - /* create string for parser */ - CHKiRet(rsCStrConstructFromszStr(&pDfltHostnameCmp, *pline)); - } else { /* string objects exists, just update... */ - CHKiRet(rsCStrSetSzStr(pDfltHostnameCmp, *pline)); - } - } - -finalize_it: - RETiRet; -} - - -/* - * Helper to cfline(). This function interprets a BSD tag selector line - * from the config file ("!tagname"). It stores it for further reference. - * rgerhards 2005-10-18 - */ -rsRetVal cflineProcessTagSelector(uchar **pline) -{ - DEFiRet; - - ASSERT(pline != NULL); - ASSERT(*pline != NULL); - ASSERT(**pline == '!'); - - dbgprintf(" - programname selector line\n"); - - (*pline)++; /* eat '!' */ - - /* the below is somewhat of a quick hack, but it is efficient (this is - * why it is in here. "!*" resets the tag selector with BSD syslog. We mimic - * this, too. As it is easy to check that condition, we do not fire up a - * parser process, just make sure we do not address beyond our space. - * Order of conditions in the if-statement is vital! rgerhards 2005-10-18 - */ - if(**pline != '\0' && **pline == '*' && *(*pline+1) == '\0') { - dbgprintf("resetting programname filter\n"); - if(pDfltProgNameCmp != NULL) { - rsCStrDestruct(&pDfltProgNameCmp); - } - } else { - dbgprintf("setting programname filter to '%s'\n", *pline); - if(pDfltProgNameCmp == NULL) { - /* create string for parser */ - CHKiRet(rsCStrConstructFromszStr(&pDfltProgNameCmp, *pline)); - } else { /* string objects exists, just update... */ - CHKiRet(rsCStrSetSzStr(pDfltProgNameCmp, *pline)); - } - } - -finalize_it: - RETiRet; -} - - /* process the action part of a selector line * rgerhards, 2007-08-01 */ @@ -735,13 +625,8 @@ rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction) if(iRet == RS_RET_OK || iRet == RS_RET_SUSPENDED) { if((iRet = addAction(&pAction, pMod, pModData, pOMSR, NULL, NULL, (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { - /* now check if the module is compatible with select features */ - if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) - pAction->f_ReduceRepeated = loadConf->globals.bReduceRepeatMsgs; - else { - dbgprintf("module is incompatible with RepeatedMsgReduction - turned off\n"); - pAction->f_ReduceRepeated = 0; - } + /* here check if the module is compatible with select features + * (currently, we have no such features!) */ pAction->eState = ACT_STATE_RDY; /* action is enabled */ conf->actions.nbrActions++; /* one more active action! */ } @@ -831,7 +716,6 @@ CODESTARTObjClassExit(conf) objRelease(module, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); - objRelease(rule, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); ENDObjClassExit(conf) @@ -845,7 +729,6 @@ BEGINAbstractObjClassInit(conf, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANG CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* TODO: make this dependcy go away! */ - CHKiRet(objUse(rule, CORE_COMPONENT)); CHKiRet(objUse(ruleset, CORE_COMPONENT)); /* These commands will NOT be supported -- the new v6.3 config system provides diff --git a/runtime/conf.h b/runtime/conf.h index 018d9111..a1bb51ad 100644 --- a/runtime/conf.h +++ b/runtime/conf.h @@ -62,11 +62,8 @@ PROTOTYPEObj(conf); rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName); rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); -/* more dirt to cover the new config interface (will go away...) */ -rsRetVal cflineProcessTagSelector(uchar **pline); -rsRetVal cflineProcessHostSelector(uchar **pline); -rsRetVal cflineProcessTradPRIFilter(uchar **pline, rule_t *pRule); -rsRetVal cflineProcessPropFilter(uchar **pline, rule_t *f); +rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]); +rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt); rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction); extern EHostnameCmpMode eDfltHostnameCmpMode; extern cstr_t *pDfltHostnameCmp; diff --git a/runtime/cryprov.h b/runtime/cryprov.h new file mode 100644 index 00000000..8496b745 --- /dev/null +++ b/runtime/cryprov.h @@ -0,0 +1,39 @@ +/* The interface definition for (file) crypto providers. + * + * This is just an abstract driver interface, which needs to be + * implemented by concrete classes. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_CRYPROV_H +#define INCLUDED_CRYPROV_H + +#include <gcrypt.h> + +/* interface */ +BEGINinterface(cryprov) /* name must also be changed in ENDinterface macro! */ + rsRetVal (*Construct)(void *ppThis); + rsRetVal (*SetCnfParam)(void *ppThis, struct nvlst *lst); + rsRetVal (*Destruct)(void *ppThis); + rsRetVal (*OnFileOpen)(void *pThis, uchar *fn, void *pFileInstData); + rsRetVal (*Encrypt)(void *pFileInstData, uchar *buf, size_t *lenBuf); + rsRetVal (*OnFileClose)(void *pFileInstData, off64_t offsLogfile); +ENDinterface(cryprov) +#define cryprovCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#endif /* #ifndef INCLUDED_CRYPROV_H */ diff --git a/runtime/datetime.c b/runtime/datetime.c index 0b9b1ae2..841ff625 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -61,8 +61,10 @@ timeval2syslogTime(struct timeval *tp, struct syslogTime *t) struct tm *tm; struct tm tmBuf; long lBias; + time_t secs; - tm = localtime_r((time_t*) &(tp->tv_sec), &tmBuf); + secs = tp->tv_sec; + tm = localtime_r(&secs, &tmBuf); t->year = tm->tm_year + 1900; t->month = tm->tm_mon + 1; @@ -180,12 +182,13 @@ getTime(time_t *ttSeconds) * the method always returns zero. * \retval The number parsed. */ -static int srSLMGParseInt32(uchar** ppsz, int *pLenStr) +static inline int +srSLMGParseInt32(uchar** ppsz, int *pLenStr) { register int i; i = 0; - while(*pLenStr > 0 && isdigit((int) **ppsz)) { + while(*pLenStr > 0 && **ppsz >= '0' && **ppsz <= '9') { i = i * 10 + **ppsz - '0'; ++(*ppsz); --(*pLenStr); @@ -902,6 +905,11 @@ time_t syslogTime2time_t(struct syslogTime *ts) case 12: MonthInDays = 334; //until 01 of December break; + default: /* this cannot happen (and would be a program error) + * but we need the code to keep the compiler silent. + */ + MonthInDays = 0; /* any value fits ;) */ + break; } diff --git a/runtime/debug.c b/runtime/debug.c index 307a8bb8..68474989 100644 --- a/runtime/debug.c +++ b/runtime/debug.c @@ -44,6 +44,9 @@ #include <unistd.h> #include <fcntl.h> #include <sys/stat.h> +#ifdef HAVE_SYS_SYSCALL_H +# include <sys/syscall.h> +#endif #if _POSIX_TIMERS <= 0 #include <sys/time.h> #endif @@ -66,6 +69,7 @@ static int bPrintMutexAction = 0; /* shall mutex calls be printed to the debug l static int bPrintTime = 1; /* print a timestamp together with debug message */ static int bPrintAllDebugOnExit = 0; static int bAbortTrace = 1; /* print a trace after SIGABRT or SIGSEGV */ +static int bOutputTidToStderr = 0;/* output TID to stderr on thread creation */ static char *pszAltDbgFileName = NULL; /* if set, debug output is *also* sent to here */ static int altdbg = -1; /* and the handle for alternate debug output */ int stddbg = 1; /* the handle for regular debug output, set to stdout if not forking, -1 otherwise */ @@ -293,6 +297,21 @@ static inline void dbgFuncDBRemoveMutexLock(dbgFuncDB_t *pFuncDB, pthread_mutex_ /* ------------------------- END FuncDB utility functions ------------------------- */ +/* output the current thread ID to "relevant" places + * (what "relevant" means is determinded by various ways) + */ +void +dbgOutputTID(char* name) +{ +# if defined(HAVE_SYSCALL) && defined(HAVE_SYS_gettid) + if(bOutputTidToStderr) + fprintf(stderr, "thread tid %u, name '%s'\n", + (unsigned)syscall(SYS_gettid), name); + DBGPRINTF("thread created, tid %u, name '%s'\n", + (unsigned)syscall(SYS_gettid), name); +# endif +} + /* ########################################################################### * IMPORTANT NOTE * Mutex instrumentation reduces the code's concurrency and thus affects its @@ -902,8 +921,12 @@ do_dbgprint(uchar *pszObjName, char *pszMsg, size_t lenMsg) lenCopy = lenMsg; memcpy(pszWriteBuf + offsWriteBuf, pszMsg, lenCopy); offsWriteBuf += lenCopy; - if(stddbg != -1) write(stddbg, pszWriteBuf, offsWriteBuf); - if(altdbg != -1) write(altdbg, pszWriteBuf, offsWriteBuf); + /* the write is included in an "if" just to silence compiler + * warnings. Here, we really don't care if the write fails, we + * have no good response to that in any case... -- rgerhards, 2012-11-28 + */ + if(stddbg != -1) if(write(stddbg, pszWriteBuf, offsWriteBuf)){}; + if(altdbg != -1) if(write(altdbg, pszWriteBuf, offsWriteBuf)){}; bWasNL = (pszMsg[lenMsg - 1] == '\n') ? 1 : 0; } @@ -1292,6 +1315,15 @@ dbgmalloc(size_t size) } +/* report fd used for debug log. This is needed in case of + * auto-backgrounding, where the debug log shall not be closed. + */ +int +dbgGetDbglogFd(void) +{ + return altdbg; +} + /* read in the runtime options * rgerhards, 2008-02-28 */ @@ -1321,6 +1353,7 @@ dbgGetRuntimeOptions(void) "PrintAllDebugInfoOnExit (not yet implemented)\n" "NoLogTimestamp\n" "Nostdoout\n" + "OutputTidToStderr\n" "filetrace=file (may be provided multiple times)\n" "DebugOnDemand - enables debugging on USR1, but does not turn on output\n" "\nSee debug.html in your doc set or http://www.rsyslog.com for details\n"); @@ -1354,6 +1387,8 @@ dbgGetRuntimeOptions(void) stddbg = -1; } else if(!strcasecmp((char*)optname, "noaborttrace")) { bAbortTrace = 0; + } else if(!strcasecmp((char*)optname, "outputtidtostderr")) { + bOutputTidToStderr = 1; } else if(!strcasecmp((char*)optname, "filetrace")) { if(*optval == '\0') { fprintf(stderr, "rsyslogd " VERSION " error: logfile debug option requires filename, " @@ -1372,10 +1407,30 @@ dbgGetRuntimeOptions(void) } +void +dbgSetDebugLevel(int level) +{ + Debug = level; + debugging_on = (level == DEBUG_FULL) ? 1 : 0; +} + +void +dbgSetDebugFile(uchar *fn) +{ + if(altdbg != -1) { + dbgprintf("switching to debug file %s\n", fn); + close(altdbg); + } + if((altdbg = open((char*)fn, O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY|O_CLOEXEC, S_IRUSR|S_IWUSR)) == -1) { + fprintf(stderr, "alternate debug file could not be opened, ignoring. Error: %s\n", strerror(errno)); + } +} + /* end support system to set debug options at runtime */ rsRetVal dbgClassInit(void) { + pthread_mutexattr_t mutAttr; rsRetVal iRet; /* do not use DEFiRet, as this makes calls into the debug system! */ struct sigaction sigAct; @@ -1383,14 +1438,16 @@ rsRetVal dbgClassInit(void) (void) pthread_key_create(&keyCallStack, dbgCallStackDestruct); /* MUST be the first action done! */ - /* we initialize all Mutexes with code, as some platforms seem to have - * bugs in the static initializer macros. So better be on the safe side... - * rgerhards, 2008-03-06 + /* the mutexes must be recursive, because it may be called from within + * signal handlers, which can lead to a hang if the signal interrupted dbgprintf + * (yes, we have really seen that situation in practice!). -- rgerhards, 2013-05-17 */ - pthread_mutex_init(&mutFuncDBList, NULL); - pthread_mutex_init(&mutMutLog, NULL); - pthread_mutex_init(&mutCallStack, NULL); - pthread_mutex_init(&mutdbgprint, NULL); + pthread_mutexattr_init(&mutAttr); + pthread_mutexattr_settype(&mutAttr, PTHREAD_MUTEX_RECURSIVE); + pthread_mutex_init(&mutFuncDBList, &mutAttr); + pthread_mutex_init(&mutMutLog, &mutAttr); + pthread_mutex_init(&mutCallStack, &mutAttr); + pthread_mutex_init(&mutdbgprint, &mutAttr); /* while we try not to use any of the real rsyslog code (to avoid infinite loops), we * need to have the ability to query object names. Thus, we need to obtain a pointer to diff --git a/runtime/debug.h b/runtime/debug.h index 5bd26bd8..f3226098 100644 --- a/runtime/debug.h +++ b/runtime/debug.h @@ -89,6 +89,8 @@ typedef struct dbgCallStack_s { /* prototypes */ rsRetVal dbgClassInit(void); rsRetVal dbgClassExit(void); +void dbgSetDebugFile(uchar *fn); +void dbgSetDebugLevel(int level); void sigsegvHdlr(int signum); void dbgoprint(obj_t *pObj, char *fmt, ...) __attribute__((format(printf, 2, 3))); void dbgprintf(char *fmt, ...) __attribute__((format(printf, 1, 2))); @@ -104,6 +106,8 @@ void dbgSetExecLocation(int iStackPtr, int line); void dbgSetThrdName(uchar *pszName); void dbgPrintAllDebugInfo(void); void *dbgmalloc(size_t size); +void dbgOutputTID(char* name); +int dbgGetDbglogFd(void); /* macros */ #ifdef DEBUGLESS diff --git a/runtime/dnscache.c b/runtime/dnscache.c index 32d6e425..2096aa36 100644 --- a/runtime/dnscache.c +++ b/runtime/dnscache.c @@ -7,7 +7,7 @@ * In any case, even the initial implementaton is far faster than what we had * before. -- rgerhards, 2011-06-06 * - * Copyright 2011 by Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2013 by Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -33,6 +33,7 @@ #include <signal.h> #include <netdb.h> #include <unistd.h> +#include <ctype.h> #include "syslogd-types.h" #include "glbl.h" @@ -40,46 +41,99 @@ #include "obj.h" #include "unicode-helper.h" #include "net.h" +#include "hashtable.h" +#include "prop.h" +#include "dnscache.h" -/* in this initial implementation, we use a simple, non-optimized at all - * linear list. - */ /* module data structures */ struct dnscache_entry_s { struct sockaddr_storage addr; - uchar *pszHostFQDN; - uchar *ip; + prop_t *fqdn; + prop_t *fqdnLowerCase; + prop_t *localName; /* only local name, without domain part (if configured so) */ + prop_t *ip; struct dnscache_entry_s *next; unsigned nUsed; }; typedef struct dnscache_entry_s dnscache_entry_t; struct dnscache_s { pthread_rwlock_t rwlock; - dnscache_entry_t *root; + struct hashtable *ht; unsigned nEntries; }; typedef struct dnscache_s dnscache_t; -#define MAX_CACHE_ENTRIES 1000 /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) DEFobjCurrIf(errmsg) +DEFobjCurrIf(prop) static dnscache_t dnsCache; +static prop_t *staticErrValue; + + +/* Our hash function. + * TODO: check how well it performs on socket addresses! + */ +unsigned int +hash_from_key_fn(void *k) +{ + int len; + uchar *rkey = (uchar*) k; /* we treat this as opaque bytes */ + unsigned hashval = 1; + len = SALEN((struct sockaddr*)k); + while(len--) + hashval = hashval * 33 + *rkey++; + + return hashval; +} + +static int +key_equals_fn(void *key1, void *key2) +{ + return (SALEN((struct sockaddr*)key1) == SALEN((struct sockaddr*) key2) + && !memcmp(key1, key2, SALEN((struct sockaddr*) key1))); +} + +/* destruct a cache entry. + * Precondition: entry must already be unlinked from list + */ +static void +entryDestruct(dnscache_entry_t *etry) +{ + if(etry->fqdn != NULL) + prop.Destruct(&etry->fqdn); + if(etry->fqdnLowerCase != NULL) + prop.Destruct(&etry->fqdnLowerCase); + if(etry->localName != NULL) + prop.Destruct(&etry->localName); + if(etry->ip != NULL) + prop.Destruct(&etry->ip); + free(etry); +} /* init function (must be called once) */ rsRetVal dnscacheInit(void) { DEFiRet; - dnsCache.root = NULL; + if((dnsCache.ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))entryDestruct)) == NULL) { + DBGPRINTF("dnscache: error creating hash table!\n"); + ABORT_FINALIZE(RS_RET_ERR); // TODO: make this degrade, but run! + } dnsCache.nEntries = 0; pthread_rwlock_init(&dnsCache.rwlock, NULL); CHKiRet(objGetObjInterface(&obj)); /* this provides the root pointer for all other queries */ CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + + prop.Construct(&staticErrValue); + prop.SetString(staticErrValue, (uchar*)"???", 3); + prop.ConstructFinalize(staticErrValue); finalize_it: RETiRet; } @@ -89,38 +143,20 @@ rsRetVal dnscacheDeinit(void) { DEFiRet; - //TODO: free cache elements dnsCache.root = NULL; + prop.Destruct(&staticErrValue); + hashtable_destroy(dnsCache.ht, 1); /* 1 => free all values automatically */ pthread_rwlock_destroy(&dnsCache.rwlock); objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); RETiRet; } -/* destruct a cache entry. - * Precondition: entry must already be unlinked from list - */ -static inline void -entryDestruct(dnscache_entry_t *etry) -{ - free(etry->pszHostFQDN); - free(etry->ip); - free(etry); -} - - static inline dnscache_entry_t* findEntry(struct sockaddr_storage *addr) { - dnscache_entry_t *etry; - for(etry = dnsCache.root ; etry != NULL ; etry = etry->next) { - if(SALEN((struct sockaddr*)addr) == SALEN((struct sockaddr*) &etry->addr) - && !memcmp(addr, &etry->addr, SALEN((struct sockaddr*) addr))) - break; /* in this case, we found our entry */ - } - if(etry != NULL) - ++etry->nUsed; /* this is *not* atomic, but we can live with an occasional loss! */ - return etry; + return((dnscache_entry_t*) hashtable_search(dnsCache.ht, addr)); } @@ -144,6 +180,73 @@ mygetnameinfo(const struct sockaddr *sa, socklen_t salen, } +/* get only the local part of the hostname and set it in cache entry */ +static inline void +setLocalHostName(dnscache_entry_t *etry) +{ + uchar *fqdnLower; + uchar *p; + int count; + int i; + uchar hostbuf[NI_MAXHOST]; + + if(glbl.GetPreserveFQDN()) { + prop.AddRef(etry->fqdnLowerCase); + etry->localName = etry->fqdnLowerCase; + goto done; + } + + /* strip domain, if configured for this entry */ + fqdnLower = propGetSzStr(etry->fqdnLowerCase); + p = (uchar*)strchr((char*)fqdnLower, '.'); /* find start of domain name "machine.example.com" */ + if(p == NULL) { /* do we have a domain part? */ + prop.AddRef(etry->fqdnLowerCase); /* no! */ + etry->localName = etry->fqdnLowerCase; + goto done; + } + + i = p - fqdnLower; /* length of hostname */ + memcpy(hostbuf, fqdnLower, i); + /* now check if we belong to any of the domain names that were specified + * in the -s command line option. If so, remove and we are done. + */ + if(glbl.GetStripDomains() != NULL) { + count=0; + while(glbl.GetStripDomains()[count]) { + if(strcmp((char*)(p + 1), glbl.GetStripDomains()[count]) == 0) { + prop.CreateStringProp(&etry->localName, hostbuf, i); + goto done; + } + count++; + } + } + /* if we reach this point, we have not found any domain we should strip. Now + * we try and see if the host itself is listed in the -l command line option + * and so should be stripped also. If so, we do it and return. Please note that + * -l list FQDNs, not just the hostname part. If it did just list the hostname, the + * door would be wide-open for all kinds of mixing up of hosts. Because of this, + * you'll see comparison against the full string (pszHostFQDN) below. + */ + if(glbl.GetLocalHosts() != NULL) { + count=0; + while(glbl.GetLocalHosts()[count]) { + if(!strcmp((char*)fqdnLower, (char*)glbl.GetLocalHosts()[count])) { + prop.CreateStringProp(&etry->localName, hostbuf, i); + goto done; + } + count++; + } + } + + /* at this point, we have not found anything, so we again use the + * already-created complete full name property. + */ + prop.AddRef(etry->fqdnLowerCase); + etry->localName = etry->fqdnLowerCase; +done: return; +} + + /* resolve an address. * * Please see http://www.hmug.org/man/3/getnameinfo.php (under Caveats) @@ -154,18 +257,19 @@ mygetnameinfo(const struct sockaddr *sa, socklen_t salen, * message should be processed (1) or discarded (0). */ static rsRetVal -resolveAddr(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) +resolveAddr(struct sockaddr_storage *addr, dnscache_entry_t *etry) { DEFiRet; int error; sigset_t omask, nmask; struct addrinfo hints, *res; + char szIP[80]; /* large enough for IPv6 */ + char fqdnBuf[NI_MAXHOST]; + rs_size_t fqdnLen; + rs_size_t i; - assert(addr != NULL); - assert(pszHostFQDN != NULL); - error = mygetnameinfo((struct sockaddr *)addr, SALEN((struct sockaddr *)addr), - (char*) ip, NI_MAXHOST, NULL, 0, NI_NUMERICHOST); + (char*) szIP, sizeof(szIP), NULL, 0, NI_NUMERICHOST); if(error) { dbgprintf("Malformed from address %s\n", gai_strerror(error)); ABORT_FINALIZE(RS_RET_INVALID_SOURCE); @@ -177,9 +281,8 @@ resolveAddr(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) pthread_sigmask(SIG_BLOCK, &nmask, &omask); error = mygetnameinfo((struct sockaddr *)addr, SALEN((struct sockaddr *) addr), - (char*)pszHostFQDN, NI_MAXHOST, NULL, 0, NI_NAMEREQD); + fqdnBuf, NI_MAXHOST, NULL, 0, NI_NAMEREQD); -dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); if(error == 0) { memset (&hints, 0, sizeof (struct addrinfo)); hints.ai_flags = AI_NUMERICHOST; @@ -188,7 +291,7 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); * because we should not have obtained a non-numeric address. If * we got a numeric one, someone messed with DNS! */ - if(getaddrinfo ((char*)pszHostFQDN, NULL, &hints, &res) == 0) { + if(getaddrinfo (fqdnBuf, NULL, &hints, &res) == 0) { uchar szErrMsg[1024]; freeaddrinfo (res); /* OK, we know we have evil. The question now is what to do about @@ -204,7 +307,7 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); snprintf((char*)szErrMsg, sizeof(szErrMsg) / sizeof(uchar), "Malicious PTR record, message dropped " "IP = \"%s\" HOST = \"%s\"", - ip, pszHostFQDN); + szIP, fqdnBuf); errmsg.LogError(0, RS_RET_MALICIOUS_ENTITY, "%s", szErrMsg); pthread_sigmask(SIG_SETMASK, &omask, NULL); ABORT_FINALIZE(RS_RET_MALICIOUS_ENTITY); @@ -219,92 +322,76 @@ dbgprintf("dnscache: error %d after 2nd mygetnameinfo\n", error); snprintf((char*)szErrMsg, sizeof(szErrMsg) / sizeof(uchar), "Malicious PTR record (message accepted, but used IP " "instead of PTR name: IP = \"%s\" HOST = \"%s\"", - ip, pszHostFQDN); + szIP, fqdnBuf); errmsg.LogError(0, NO_ERRCODE, "%s", szErrMsg); error = 1; /* that will trigger using IP address below. */ + } else {/* we have a valid entry, so let's create the respective properties */ + fqdnLen = strlen(fqdnBuf); + prop.CreateStringProp(&etry->fqdn, (uchar*)fqdnBuf, fqdnLen); + for(i = 0 ; i < fqdnLen ; ++i) + fqdnBuf[i] = tolower(fqdnBuf[i]); + prop.CreateStringProp(&etry->fqdnLowerCase, (uchar*)fqdnBuf, fqdnLen); } } pthread_sigmask(SIG_SETMASK, &omask, NULL); } -dbgprintf("dnscache: error %d, DisableDNS %d\n", error, glbl.GetDisableDNS()); - if(error || glbl.GetDisableDNS()) { - dbgprintf("Host name for your address (%s) unknown\n", ip); - strcpy((char*) pszHostFQDN, (char*)ip); - } finalize_it: - RETiRet; -} + if(iRet != RS_RET_OK) { + strcpy(szIP, "?error.obtaining.ip?"); + error = 1; /* trigger hostname copies below! */ + } + /* we need to create the inputName property (only once during our lifetime) */ + prop.CreateStringProp(&etry->ip, (uchar*)szIP, strlen(szIP)); -/* evict an entry from the cache. We should try to evict one that does - * not decrease the hit rate that much, but we do not try to hard currently - * (as the base cache data structure may change). - * This MUST NOT be called when the cache is empty! - * rgerhards, 2011-06-06 - */ -static inline void -evictEntry(void) -{ - dnscache_entry_t *prev, *evict, *prevEvict, *etry; - unsigned lowest; - - prev = prevEvict = NULL; - evict = dnsCache.root; - lowest = evict->nUsed; - for(etry = dnsCache.root->next ; etry != NULL ; etry = etry->next) { - if(etry->nUsed < lowest) { - evict = etry; - lowest = etry->nUsed; - prevEvict = prev; - } - prev = etry; - } + if(error || glbl.GetDisableDNS()) { + dbgprintf("Host name for your address (%s) unknown\n", szIP); + prop.AddRef(etry->ip); + etry->fqdn = etry->ip; + prop.AddRef(etry->ip); + etry->fqdnLowerCase = etry->ip; + } - /* found lowest, unlink */ - if(prevEvict == NULL) { /* remove root? */ - dnsCache.root = evict->next; - } else { - prevEvict = evict->next; - } - entryDestruct(evict); + setLocalHostName(etry); + + RETiRet; } -/* add a new entry to the cache. This means the address is resolved and - * then added to the cache. - */ static inline rsRetVal addEntry(struct sockaddr_storage *addr, dnscache_entry_t **pEtry) { - uchar pszHostFQDN[NI_MAXHOST]; - uchar ip[80]; /* 80 is safe for larges IPv6 addr */ - dnscache_entry_t *etry; + int r; + struct sockaddr_storage *keybuf; + dnscache_entry_t *etry = NULL; DEFiRet; - CHKiRet(resolveAddr(addr, pszHostFQDN, ip)); + CHKmalloc(etry = MALLOC(sizeof(dnscache_entry_t))); - CHKmalloc(etry->pszHostFQDN = ustrdup(pszHostFQDN)); - CHKmalloc(etry->ip = ustrdup(ip)); + CHKiRet(resolveAddr(addr, etry)); memcpy(&etry->addr, addr, SALEN((struct sockaddr*) addr)); etry->nUsed = 0; *pEtry = etry; - /* add to list. Currently, we place the new element always at - * the root node. This needs to be optimized later. 2011-06-06 - */ + CHKmalloc(keybuf = malloc(sizeof(struct sockaddr_storage))); + memcpy(keybuf, addr, sizeof(struct sockaddr_storage)); + pthread_rwlock_unlock(&dnsCache.rwlock); /* release read lock */ pthread_rwlock_wrlock(&dnsCache.rwlock); /* and re-aquire for writing */ - if(dnsCache.nEntries >= MAX_CACHE_ENTRIES) { - evictEntry(); + r = hashtable_insert(dnsCache.ht, keybuf, *pEtry); + if(r == 0) { + DBGPRINTF("dnscache: inserting element failed\n"); } - etry->next = dnsCache.root; - dnsCache.root = etry; pthread_rwlock_unlock(&dnsCache.rwlock); - pthread_rwlock_rdlock(&dnsCache.rwlock); /* TODO: optimize this! */ + pthread_rwlock_rdlock(&dnsCache.rwlock); /* we need this again */ finalize_it: + if(iRet != RS_RET_OK && etry != NULL) { + /* Note: sub-fields cannot be populated in this case */ + free(etry); + } RETiRet; } @@ -314,7 +401,7 @@ finalize_it: * TODO: implement! */ static inline rsRetVal -validateEntry(dnscache_entry_t *etry, struct sockaddr_storage *addr) +validateEntry(dnscache_entry_t __attribute__((unused)) *etry, struct sockaddr_storage __attribute__((unused)) *addr) { return RS_RET_OK; } @@ -322,10 +409,12 @@ validateEntry(dnscache_entry_t *etry, struct sockaddr_storage *addr) /* This is the main function: it looks up an entry and returns it's name * and IP address. If the entry is not yet inside the cache, it is added. - * If the entry can not be resolved, an error is reported back. + * If the entry can not be resolved, an error is reported back. If fqdn + * or fqdnLowerCase are NULL, they are not set. */ rsRetVal -dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) +dnscacheLookup(struct sockaddr_storage *addr, prop_t **fqdn, prop_t **fqdnLowerCase, + prop_t **localName, prop_t **ip) { dnscache_entry_t *etry; DEFiRet; @@ -338,18 +427,39 @@ dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip) } else { CHKiRet(validateEntry(etry, addr)); } - // TODO/QUESTION: can we get rid of the strcpy? -dbgprintf("XXXX: hostn '%s', ip '%s'\n", etry->pszHostFQDN, etry->ip); - strcpy((char*)pszHostFQDN, (char*)etry->pszHostFQDN); - strcpy((char*)ip, (char*)etry->ip); + prop.AddRef(etry->ip); + *ip = etry->ip; + if(fqdn != NULL) { + prop.AddRef(etry->fqdn); + *fqdn = etry->fqdn; + } + if(fqdnLowerCase != NULL) { + prop.AddRef(etry->fqdnLowerCase); + *fqdnLowerCase = etry->fqdnLowerCase; + } + if(localName != NULL) { + prop.AddRef(etry->localName); + *localName = etry->localName; + } finalize_it: pthread_rwlock_unlock(&dnsCache.rwlock); -dbgprintf("XXXX: dnscacheLookup finished, iRet=%d\n", iRet); if(iRet != RS_RET_OK && iRet != RS_RET_ADDRESS_UNKNOWN) { DBGPRINTF("dnscacheLookup failed with iRet %d\n", iRet); - strcpy((char*) pszHostFQDN, "???"); - strcpy((char*) ip, "???"); + prop.AddRef(staticErrValue); + *ip = staticErrValue; + if(fqdn != NULL) { + prop.AddRef(staticErrValue); + *fqdn = staticErrValue; + } + if(fqdnLowerCase != NULL) { + prop.AddRef(staticErrValue); + *fqdnLowerCase = staticErrValue; + } + if(localName != NULL) { + prop.AddRef(staticErrValue); + *localName = staticErrValue; + } } RETiRet; } diff --git a/runtime/dnscache.h b/runtime/dnscache.h index 69f038ee..9c21a645 100644 --- a/runtime/dnscache.h +++ b/runtime/dnscache.h @@ -1,6 +1,6 @@ /* Definitions for dnscache module. * - * Copyright 2011-2012 Adiscon GmbH. + * Copyright 2011-2013 Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -24,6 +24,6 @@ rsRetVal dnscacheInit(void); rsRetVal dnscacheDeinit(void); -rsRetVal dnscacheLookup(struct sockaddr_storage *addr, uchar *pszHostFQDN, uchar *ip); +rsRetVal dnscacheLookup(struct sockaddr_storage *addr, prop_t **fqdn, prop_t **fqdnLowerCase, prop_t **localName, prop_t **ip); #endif /* #ifndef INCLUDED_DNSCACHE_H */ diff --git a/runtime/glbl.c b/runtime/glbl.c index a0997829..b3fe3a1d 100644 --- a/runtime/glbl.c +++ b/runtime/glbl.c @@ -7,7 +7,7 @@ * * Module begun 2008-04-16 by Rainer Gerhards * - * Copyright 2008-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -82,6 +82,7 @@ static uchar *pszDfltNetstrmDrvrCAF = NULL; /* default CA file for the netstrm d static uchar *pszDfltNetstrmDrvrKeyFile = NULL; /* default key file for the netstrm driver (server) */ static uchar *pszDfltNetstrmDrvrCertFile = NULL; /* default cert file for the netstrm driver (server) */ static int bTerminateInputs = 0; /* global switch that inputs shall terminate ASAP (1=> terminate) */ +pid_t glbl_ourpid; #ifndef HAVE_ATOMIC_BUILTINS static DEF_ATOMIC_HELPER_MUT(mutTerminateInputs); #endif @@ -210,7 +211,7 @@ setLocalHostIPIF(void __attribute__((unused)) *pVal, uchar *pNewVal) if(propLocalIPIF != NULL) { errmsg.LogError(0, RS_RET_ERR, "$LocalHostIPIF is already set " "and cannot be reset; place it at TOP OF rsyslog.conf!"); - ABORT_FINALIZE(RS_RET_ERR_WRKDIR); + ABORT_FINALIZE(RS_RET_ERR); } localRet = net.GetIFIPAddr(pNewVal, AF_UNSPEC, myIP, (int) sizeof(myIP)); @@ -278,6 +279,28 @@ finalize_it: RETiRet; } + +static rsRetVal +setDebugFile(void __attribute__((unused)) *pVal, uchar *pNewVal) +{ + DEFiRet; + dbgSetDebugFile(pNewVal); + free(pNewVal); + RETiRet; +} + + +static rsRetVal +setDebugLevel(void __attribute__((unused)) *pVal, int level) +{ + DEFiRet; + dbgSetDebugLevel(level); + dbgprintf("debug level %d set via config file\n", level); + dbgprintf("This is rsyslog version " VERSION "\n"); + RETiRet; +} + + /* return our local IP. * If no local IP is set, "127.0.0.1" is selected *and* set. This * is an intensional side effect that we do in order to keep things @@ -610,6 +633,8 @@ BEGINAbstractObjClassInit(glbl, 1, OBJ_IS_CORE_MODULE) /* class, version */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); /* config handlers are never unregistered and need not be - we are always loaded ;) */ + CHKiRet(regCfSysLineHdlr((uchar *)"debugfile", 0, eCmdHdlrGetWord, setDebugFile, NULL, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"debuglevel", 0, eCmdHdlrInt, setDebugLevel, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"workdirectory", 0, eCmdHdlrGetWord, setWorkDir, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"dropmsgswithmaliciousdnsptrrecords", 0, eCmdHdlrBinary, NULL, &bDropMalPTRMsgs, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"defaultnetstreamdriver", 0, eCmdHdlrGetWord, NULL, &pszDfltNetstrmDrvr, NULL)); diff --git a/runtime/glbl.h b/runtime/glbl.h index d2d1e66a..e95e48f7 100644 --- a/runtime/glbl.h +++ b/runtime/glbl.h @@ -30,11 +30,14 @@ #ifndef GLBL_H_INCLUDED #define GLBL_H_INCLUDED +#include <sys/types.h> #include "rainerscript.h" #include "prop.h" #define glblGetIOBufSize() 4096 /* size of the IO buffer, e.g. for strm class */ +extern pid_t glbl_ourpid; + /* interfaces */ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ uchar* (*GetWorkDir)(void); @@ -86,6 +89,9 @@ ENDinterface(glbl) /* the remaining prototypes */ PROTOTYPEObj(glbl); +static inline pid_t glblGetOurPid(void) { return glbl_ourpid; } +static inline void glblSetOurPid(pid_t pid) { glbl_ourpid = pid; } + void glblPrepCnf(void); void glblProcessCnf(struct cnfobj *o); void glblDoneLoadCnf(void); diff --git a/runtime/hashtable.c b/runtime/hashtable.c index a01fa7d9..f718bd43 100644 --- a/runtime/hashtable.c +++ b/runtime/hashtable.c @@ -263,7 +263,7 @@ hashtable_destroy(struct hashtable *h, int free_values) /* some generic hash functions */ -/* one provided by Aaaron Wiebe based on perl's hashng algorithm +/* one provided by Aaaron Wiebe based on perl's hashing algorithm * (so probably pretty generic). Not for excessively large strings! */ unsigned int diff --git a/runtime/libgcry.c b/runtime/libgcry.c new file mode 100644 index 00000000..51c10af4 --- /dev/null +++ b/runtime/libgcry.c @@ -0,0 +1,426 @@ +/* gcry.c - rsyslog's libgcrypt based crypto provider + * + * Copyright 2013 Adiscon GmbH. + * + * We need to store some additional information in support of encryption. + * For this, we create a side-file, which is named like the actual log + * file, but with the suffix ".encinfo" appended. It contains the following + * records: + * IV:<hex> The initial vector used at block start. Also indicates start + * start of block. + * END:<int> The end offset of the block, as uint64_t in decimal notation. + * This is used during encryption to know when the current + * encryption block ends. + * For the current implementation, there must always be an IV record + * followed by an END record. Each records is LF-terminated. Record + * types can simply be extended in the future by specifying new + * types (like "IV") before the colon. + * To identify a file as rsyslog encryption info file, it must start with + * the line "FILETYPE:rsyslog-enrcyption-info" + * There are some size constraints: the recordtype must be 31 bytes at + * most and the actual value (between : and LF) must be 1023 bytes at most. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#if HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdio.h> +#include <gcrypt.h> +#include <sys/stat.h> +#include <sys/uio.h> +#include <sys/types.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> + +#include "rsyslog.h" +#include "libgcry.h" + + +static rsRetVal +eiWriteRec(gcryfile gf, char *recHdr, size_t lenRecHdr, char *buf, size_t lenBuf) +{ + struct iovec iov[3]; + ssize_t nwritten, towrite; + DEFiRet; + + iov[0].iov_base = recHdr; + iov[0].iov_len = lenRecHdr; + iov[1].iov_base = buf; + iov[1].iov_len = lenBuf; + iov[2].iov_base = "\n"; + iov[2].iov_len = 1; + towrite = iov[0].iov_len + iov[1].iov_len + iov[2].iov_len; + nwritten = writev(gf->fd, iov, sizeof(iov)/sizeof(struct iovec)); + if(nwritten != towrite) { + DBGPRINTF("eiWrite%s: error writing file, towrite %d, " + "nwritten %d\n", recHdr, (int) towrite, (int) nwritten); + ABORT_FINALIZE(RS_RET_EI_WR_ERR); + } + DBGPRINTF("encryption info file %s: written %s, len %d\n", + recHdr, gf->eiName, (int) nwritten); +finalize_it: + RETiRet; +} + +static rsRetVal +eiOpenRead(gcryfile gf) +{ + DEFiRet; + gf->fd = open((char*)gf->eiName, O_RDONLY|O_NOCTTY|O_CLOEXEC); + if(gf->fd == -1) { + ABORT_FINALIZE(errno == ENOENT ? RS_RET_EI_NO_EXISTS : RS_RET_EI_OPN_ERR); + } +finalize_it: + RETiRet; +} + + +static rsRetVal +eiCheckFiletype(gcryfile gf) +{ + char hdrBuf[128]; + size_t toRead, didRead; + DEFiRet; + + CHKiRet(eiOpenRead(gf)); + if(Debug) memset(hdrBuf, 0, sizeof(hdrBuf)); /* for dbgprintf below! */ + toRead = sizeof("FILETYPE:")-1 + sizeof(RSGCRY_FILETYPE_NAME)-1 + 1; + didRead = read(gf->fd, hdrBuf, toRead); + close(gf->fd); + DBGPRINTF("eiCheckFiletype read %d bytes: '%s'\n", didRead, hdrBuf); + if( didRead != toRead + || strncmp(hdrBuf, "FILETYPE:" RSGCRY_FILETYPE_NAME "\n", toRead)) + iRet = RS_RET_EI_INVLD_FILE; +finalize_it: + RETiRet; +} + +static rsRetVal +eiOpenAppend(gcryfile gf) +{ + rsRetVal localRet; + DEFiRet; + localRet = eiCheckFiletype(gf); + if(localRet == RS_RET_OK) { + gf->fd = open((char*)gf->eiName, + O_WRONLY|O_APPEND|O_NOCTTY|O_CLOEXEC, 0600); + if(gf->fd == -1) { + ABORT_FINALIZE(RS_RET_EI_OPN_ERR); + } + } else if(localRet == RS_RET_EI_NO_EXISTS) { + /* looks like we need to create a new file */ + gf->fd = open((char*)gf->eiName, + O_WRONLY|O_CREAT|O_NOCTTY|O_CLOEXEC, 0600); + if(gf->fd == -1) { + ABORT_FINALIZE(RS_RET_EI_OPN_ERR); + } + CHKiRet(eiWriteRec(gf, "FILETYPE:", 9, RSGCRY_FILETYPE_NAME, + sizeof(RSGCRY_FILETYPE_NAME)-1)); + } else { + gf->fd = -1; + ABORT_FINALIZE(localRet); + } + DBGPRINTF("encryption info file %s: opened as #%d\n", + gf->eiName, gf->fd); +finalize_it: + RETiRet; +} + +static rsRetVal +eiWriteIV(gcryfile gf, uchar *iv) +{ + static const char hexchars[16] = + {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'}; + unsigned iSrc, iDst; + char hex[4096]; + DEFiRet; + + if(gf->blkLength > sizeof(hex)/2) { + DBGPRINTF("eiWriteIV: crypto block len way too large, aborting " + "write"); + ABORT_FINALIZE(RS_RET_ERR); + } + + for(iSrc = iDst = 0 ; iSrc < gf->blkLength ; ++iSrc) { + hex[iDst++] = hexchars[iv[iSrc]>>4]; + hex[iDst++] = hexchars[iv[iSrc]&0x0f]; + } + + iRet = eiWriteRec(gf, "IV:", 3, hex, gf->blkLength*2); +finalize_it: + RETiRet; +} + +/* we do not return an error state, as we MUST close the file, + * no matter what happens. + */ +static void +eiClose(gcryfile gf, off64_t offsLogfile) +{ + char offs[21]; + size_t len; + if(gf->fd == -1) + return; + /* 2^64 is 20 digits, so the snprintf buffer is large enough */ + len = snprintf(offs, sizeof(offs), "%lld", offsLogfile); + eiWriteRec(gf, "END:", 4, offs, len); + close(gf->fd); + DBGPRINTF("encryption info file %s: closed\n", gf->eiName); +} + +static rsRetVal +gcryfileConstruct(gcryctx ctx, gcryfile *pgf, uchar *logfn) +{ + char fn[MAXFNAME+1]; + gcryfile gf; + DEFiRet; + + CHKmalloc(gf = calloc(1, sizeof(struct gcryfile_s))); + gf->ctx = ctx; + snprintf(fn, sizeof(fn), "%s%s", logfn, ENCINFO_SUFFIX); + fn[MAXFNAME] = '\0'; /* be on save side */ + gf->eiName = (uchar*) strdup(fn); + *pgf = gf; +finalize_it: + RETiRet; +} + + +gcryctx +gcryCtxNew(void) +{ + gcryctx ctx; + ctx = calloc(1, sizeof(struct gcryctx_s)); + ctx->algo = GCRY_CIPHER_AES128; + ctx->mode = GCRY_CIPHER_MODE_CBC; + return ctx; +} + +int +gcryfileDestruct(gcryfile gf, off64_t offsLogfile) +{ + int r = 0; + if(gf == NULL) + goto done; + + eiClose(gf, offsLogfile); + free(gf->eiName); + free(gf); +done: return r; +} +void +rsgcryCtxDel(gcryctx ctx) +{ + if(ctx != NULL) { + free(ctx); + } +} + +static inline void +addPadding(gcryfile pF, uchar *buf, size_t *plen) +{ + unsigned i; + size_t nPad; + nPad = (pF->blkLength - *plen % pF->blkLength) % pF->blkLength; + DBGPRINTF("libgcry: addPadding %d chars, blkLength %d, mod %d, pad %d\n", + *plen, pF->blkLength, *plen % pF->blkLength, nPad); + for(i = 0 ; i < nPad ; ++i) + buf[(*plen)+i] = 0x00; + (*plen)+= nPad; +} + +static inline void +removePadding(char *buf, size_t *plen) +{ + unsigned len = (unsigned) *plen; + unsigned iSrc, iDst; + char *frstNUL; + + frstNUL = strchr(buf, 0x00); + if(frstNUL == NULL) + goto done; + iDst = iSrc = frstNUL - buf; + + while(iSrc < len) { + if(buf[iSrc] != 0x00) + buf[iDst++] = buf[iSrc]; + ++iSrc; + } + + *plen = iDst; +done: return; +} + +/* returns 0 on succes, positive if key length does not match and key + * of return value size is required. + */ +int +rsgcrySetKey(gcryctx ctx, unsigned char *key, uint16_t keyLen) +{ + uint16_t reqKeyLen; + int r; + + reqKeyLen = gcry_cipher_get_algo_keylen(ctx->algo); + if(keyLen != reqKeyLen) { + r = reqKeyLen; + goto done; + } + ctx->keyLen = keyLen; + ctx->key = malloc(keyLen); + memcpy(ctx->key, key, keyLen); + r = 0; +done: return r; +} + +rsRetVal +rsgcrySetMode(gcryctx ctx, uchar *modename) +{ + int mode; + DEFiRet; + + mode = rsgcryModename2Mode((char *)modename); + if(mode == GCRY_CIPHER_MODE_NONE) { + ABORT_FINALIZE(RS_RET_CRY_INVLD_MODE); + } + ctx->mode = mode; +finalize_it: + RETiRet; +} + +rsRetVal +rsgcrySetAlgo(gcryctx ctx, uchar *algoname) +{ + int algo; + DEFiRet; + + algo = rsgcryAlgoname2Algo((char *)algoname); + if(algo == GCRY_CIPHER_NONE) { + ABORT_FINALIZE(RS_RET_CRY_INVLD_ALGO); + } + ctx->algo = algo; +finalize_it: + RETiRet; +} + +/* As of some Linux and security expert I spoke to, /dev/urandom + * provides very strong random numbers, even if it runs out of + * entropy. As far as he knew, this is save for all applications + * (and he had good proof that I currently am not permitted to + * reproduce). -- rgerhards, 2013-03-04 + */ +void +seedIV(gcryfile gf, uchar **iv) +{ + int fd; + + *iv = malloc(gf->blkLength); /* do NOT zero-out! */ + /* if we cannot obtain data from /dev/urandom, we use whatever + * is present at the current memory location as random data. Of + * course, this is very weak and we should consider a different + * option, especially when not running under Linux (for Linux, + * unavailability of /dev/urandom is just a theoretic thing, it + * will always work...). -- TODO -- rgerhards, 2013-03-06 + */ + if((fd = open("/dev/urandom", O_RDONLY)) > 0) { + if(read(fd, *iv, gf->blkLength)) {}; /* keep compiler happy */ + close(fd); + } +} + +rsRetVal +rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname) +{ + gcry_error_t gcryError; + gcryfile gf = NULL; + uchar *iv = NULL; + DEFiRet; + + CHKiRet(gcryfileConstruct(ctx, &gf, fname)); + + gf->blkLength = gcry_cipher_get_algo_blklen(ctx->algo); + + gcryError = gcry_cipher_open(&gf->chd, ctx->algo, ctx->mode, 0); + if (gcryError) { + dbgprintf("gcry_cipher_open failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + ABORT_FINALIZE(RS_RET_ERR); + } + + gcryError = gcry_cipher_setkey(gf->chd, gf->ctx->key, gf->ctx->keyLen); + if (gcryError) { + dbgprintf("gcry_cipher_setkey failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + ABORT_FINALIZE(RS_RET_ERR); + } + + seedIV(gf, &iv); + gcryError = gcry_cipher_setiv(gf->chd, iv, gf->blkLength); + if (gcryError) { + dbgprintf("gcry_cipher_setiv failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + ABORT_FINALIZE(RS_RET_ERR); + } + CHKiRet(eiOpenAppend(gf)); + CHKiRet(eiWriteIV(gf, iv)); + *pgf = gf; +finalize_it: + free(iv); + if(iRet != RS_RET_OK && gf != NULL) + gcryfileDestruct(gf, -1); + RETiRet; +} + +int +rsgcryEncrypt(gcryfile pF, uchar *buf, size_t *len) +{ + int gcryError; + DEFiRet; + + if(*len == 0) + FINALIZE; + + addPadding(pF, buf, len); + gcryError = gcry_cipher_encrypt(pF->chd, buf, *len, NULL, 0); + if(gcryError) { + dbgprintf("gcry_cipher_encrypt failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + ABORT_FINALIZE(RS_RET_ERR); + } +finalize_it: + RETiRet; +} + + +/* module-init dummy for potential later use */ +int +rsgcryInit(void) +{ + return 0; +} + +/* module-deinit dummy for potential later use */ +void +rsgcryExit(void) +{ + return; +} diff --git a/runtime/libgcry.h b/runtime/libgcry.h new file mode 100644 index 00000000..b77b0f9e --- /dev/null +++ b/runtime/libgcry.h @@ -0,0 +1,101 @@ +/* libgcry.h - rsyslog's guardtime support library + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_LIBGCRY_H +#define INCLUDED_LIBGCRY_H +#include <stdint.h> + + +struct gcryctx_s { + uchar *key; + size_t keyLen; + int algo; + int mode; +}; +typedef struct gcryctx_s *gcryctx; +typedef struct gcryfile_s *gcryfile; + +/* this describes a file, as far as libgcry is concerned */ +struct gcryfile_s { + gcry_cipher_hd_t chd; /* cypher handle */ + size_t blkLength; /* size of low-level crypto block */ + uchar *eiName; /* name of .encinfo file */ + int fd; /* descriptor of .encinfo file (-1 if not open) */ + gcryctx ctx; +}; + +int gcryGetKeyFromFile(char *fn, char **key, unsigned *keylen); +int rsgcryInit(void); +void rsgcryExit(void); +int rsgcrySetKey(gcryctx ctx, unsigned char *key, uint16_t keyLen); +rsRetVal rsgcrySetMode(gcryctx ctx, uchar *algoname); +rsRetVal rsgcrySetAlgo(gcryctx ctx, uchar *modename); +gcryctx gcryCtxNew(void); +void rsgcryCtxDel(gcryctx ctx); +int gcryfileDestruct(gcryfile gf, off64_t offsLogfile); +rsRetVal rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname); +int rsgcryEncrypt(gcryfile pF, uchar *buf, size_t *len); + +/* error states */ +#define RSGCRYE_EI_OPEN 1 /* error opening .encinfo file */ +#define RSGCRYE_OOM 4 /* ran out of memory */ + +#define EIF_MAX_RECTYPE_LEN 31 /* max length of record types */ +#define EIF_MAX_VALUE_LEN 1023 /* max length of value types */ +#define RSGCRY_FILETYPE_NAME "rsyslog-enrcyption-info" +#define ENCINFO_SUFFIX ".encinfo" + +static inline int +rsgcryAlgoname2Algo(char *algoname) { + if(!strcmp((char*)algoname, "3DES")) return GCRY_CIPHER_3DES; + if(!strcmp((char*)algoname, "CAST5")) return GCRY_CIPHER_CAST5; + if(!strcmp((char*)algoname, "BLOWFISH")) return GCRY_CIPHER_BLOWFISH; + if(!strcmp((char*)algoname, "AES128")) return GCRY_CIPHER_AES128; + if(!strcmp((char*)algoname, "AES192")) return GCRY_CIPHER_AES192; + if(!strcmp((char*)algoname, "AES256")) return GCRY_CIPHER_AES256; + if(!strcmp((char*)algoname, "TWOFISH")) return GCRY_CIPHER_TWOFISH; + if(!strcmp((char*)algoname, "TWOFISH128")) return GCRY_CIPHER_TWOFISH128; + if(!strcmp((char*)algoname, "ARCFOUR")) return GCRY_CIPHER_ARCFOUR; + if(!strcmp((char*)algoname, "DES")) return GCRY_CIPHER_DES; + if(!strcmp((char*)algoname, "SERPENT128")) return GCRY_CIPHER_SERPENT128; + if(!strcmp((char*)algoname, "SERPENT192")) return GCRY_CIPHER_SERPENT192; + if(!strcmp((char*)algoname, "SERPENT256")) return GCRY_CIPHER_SERPENT256; + if(!strcmp((char*)algoname, "RFC2268_40")) return GCRY_CIPHER_RFC2268_40; + if(!strcmp((char*)algoname, "SEED")) return GCRY_CIPHER_SEED; + if(!strcmp((char*)algoname, "CAMELLIA128")) return GCRY_CIPHER_CAMELLIA128; + if(!strcmp((char*)algoname, "CAMELLIA192")) return GCRY_CIPHER_CAMELLIA192; + if(!strcmp((char*)algoname, "CAMELLIA256")) return GCRY_CIPHER_CAMELLIA256; + return GCRY_CIPHER_NONE; +} + +static inline int +rsgcryModename2Mode(char *modename) { + if(!strcmp((char*)modename, "ECB")) return GCRY_CIPHER_MODE_ECB; + if(!strcmp((char*)modename, "CFB")) return GCRY_CIPHER_MODE_CFB; + if(!strcmp((char*)modename, "CBC")) return GCRY_CIPHER_MODE_CBC; + if(!strcmp((char*)modename, "STREAM")) return GCRY_CIPHER_MODE_STREAM; + if(!strcmp((char*)modename, "OFB")) return GCRY_CIPHER_MODE_OFB; + if(!strcmp((char*)modename, "CTR")) return GCRY_CIPHER_MODE_CTR; +# ifdef GCRY_CIPHER_MODE_AESWRAP + if(!strcmp((char*)modename, "AESWRAP")) return GCRY_CIPHER_MODE_AESWRAP; +# endif + return GCRY_CIPHER_MODE_NONE; +} +#endif /* #ifndef INCLUDED_LIBGCRY_H */ diff --git a/runtime/libgcry_common.c b/runtime/libgcry_common.c new file mode 100644 index 00000000..07a524dc --- /dev/null +++ b/runtime/libgcry_common.c @@ -0,0 +1,206 @@ +/* libgcry_common.c + * This file hosts functions both being used by the rsyslog runtime as + * well as tools who do not use the runtime (so we can maintain the + * code at a single place). + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#if HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdio.h> +#include <gcrypt.h> +#include <sys/stat.h> +#include <sys/uio.h> +#include <sys/types.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> + +#include "rsyslog.h" /* we need data typedefs */ +#include "libgcry.h" + + +/* read a key from a key file + * @param[out] key - key buffer, must be freed by caller + * @param[out] keylen - length of buffer + * @returns 0 if OK, something else otherwise (we do not use + * iRet as this is also called from non-rsyslog w/o runtime) + * The key length is limited to 64KiB to prevent DoS. + * Note well: key is a blob, not a C string (NUL may be present!) + */ +int +gcryGetKeyFromFile(char *fn, char **key, unsigned *keylen) +{ + struct stat sb; + int fd; + int r; + + if(stat(fn, &sb) == -1) { + r = 1; goto done; + } + if((sb.st_mode & S_IFMT) != S_IFREG) { + r = 2; goto done; + } + if(sb.st_size > 64*1024) { + r = 3; goto done; + } + if((*key = malloc(sb.st_size)) == NULL) { + r = -1; goto done; + } + if((fd = open(fn, O_RDONLY)) < 0) { + r = 4; goto done; + } + if(read(fd, *key, sb.st_size) != sb.st_size) { + r = 5; goto done; + } + *keylen = sb.st_size; + close(fd); + r = 0; +done: return r; +} + + +/* execute the child process (must be called in child context + * after fork). + */ + +static void +execKeyScript(char *cmd, int pipefd[]) +{ + char *newargv[] = { NULL }; + char *newenviron[] = { NULL }; + + dup2(pipefd[0], STDIN_FILENO); + dup2(pipefd[1], STDOUT_FILENO); + + /* finally exec child */ +fprintf(stderr, "pre execve: %s\n", cmd); + execve(cmd, newargv, newenviron); + /* switch to? + execlp((char*)program, (char*) program, (char*)arg, NULL); + */ + + /* we should never reach this point, but if we do, we terminate */ + return; +} + + +static int +openPipe(char *cmd, int *fd) +{ + int pipefd[2]; + pid_t cpid; + int r; + + if(pipe(pipefd) == -1) { + r = 1; goto done; + } + + cpid = fork(); + if(cpid == -1) { + r = 1; goto done; + } + + if(cpid == 0) { + /* we are the child */ + execKeyScript(cmd, pipefd); + exit(1); + } + + close(pipefd[1]); + *fd = pipefd[0]; + r = 0; +done: return r; +} + + +/* Read a character from the program's output. */ +// TODO: highly unoptimized version, should be used in buffered +// mode +static int +readProgChar(int fd, char *c) +{ + int r; + if(read(fd, c, 1) != 1) { + r = 1; goto done; + } + r = 0; +done: return r; +} + +/* Read a line from the script. Line is terminated by LF, which + * is NOT put into the buffer. + * buf must be 64KiB + */ +static int +readProgLine(int fd, char *buf) +{ + char c; + int r; + unsigned i; + + for(i = 0 ; i < 64*1024 ; ++i) { + if((r = readProgChar(fd, &c)) != 0) goto done; + if(c == '\n') + break; + buf[i] = c; + }; + if(i >= 64*1024) { + r = 1; goto done; + } + buf[i] = '\0'; + r = 0; +done: return r; +} +static int +readProgKey(int fd, char *buf, unsigned keylen) +{ + char c; + int r; + unsigned i; + + for(i = 0 ; i < keylen ; ++i) { + if((r = readProgChar(fd, &c)) != 0) goto done; + buf[i] = c; + }; + r = 0; +done: return r; +} + +int +gcryGetKeyFromProg(char *cmd, char **key, unsigned *keylen) +{ + int r; + int fd; + char rcvBuf[64*1024]; + + if((r = openPipe(cmd, &fd)) != 0) goto done; + if((r = readProgLine(fd, rcvBuf)) != 0) goto done; + if(strcmp(rcvBuf, "RSYSLOG-KEY-PROVIDER:0")) { + r = 2; goto done; + } + if((r = readProgLine(fd, rcvBuf)) != 0) goto done; + *keylen = atoi(rcvBuf); + if((*key = malloc(*keylen)) == NULL) { + r = -1; goto done; + } + if((r = readProgKey(fd, *key, *keylen)) != 0) goto done; +done: return r; +} diff --git a/runtime/librsgt.c b/runtime/librsgt.c new file mode 100644 index 00000000..85fc7742 --- /dev/null +++ b/runtime/librsgt.c @@ -0,0 +1,845 @@ +/* librsgt.c - rsyslog's guardtime support library + * + * Regarding the online algorithm for Merkle tree signing. Expected + * calling sequence is: + * + * sigblkConstruct + * for each signature block: + * sigblkInit + * for each record: + * sigblkAddRecord + * sigblkFinish + * sigblkDestruct + * + * Obviously, the next call after sigblkFinsh must either be to + * sigblkInit or sigblkDestruct (if no more signature blocks are + * to be emitted, e.g. on file close). sigblkDestruct saves state + * information (most importantly last block hash) and sigblkConstruct + * reads (or initilizes if not present) it. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> +#include <assert.h> +#include <unistd.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#define MAXFNAME 1024 + +#include <gt_http.h> + +#include "librsgt.h" + +typedef unsigned char uchar; +#ifndef VERSION +#define VERSION "no-version" +#endif + + +static void +reportErr(gtctx ctx, char *errmsg) +{ + if(ctx->errFunc == NULL) + goto done; + ctx->errFunc(ctx->usrptr, (uchar*)errmsg); +done: return; +} + +static void +reportGTAPIErr(gtctx ctx, gtfile gf, char *apiname, int ecode) +{ + char errbuf[4096]; + snprintf(errbuf, sizeof(errbuf), "%s[%s:%d]: %s", + (gf == NULL) ? (uchar*)"" : gf->sigfilename, + apiname, ecode, GT_getErrorString(ecode)); + errbuf[sizeof(errbuf)-1] = '\0'; + reportErr(ctx, errbuf); +} + +void +rsgtsetErrFunc(gtctx ctx, void (*func)(void*, uchar *), void *usrptr) +{ + ctx->usrptr = usrptr; + ctx->errFunc = func; +} + +imprint_t * +rsgtImprintFromGTDataHash(GTDataHash *hash) +{ + imprint_t *imp; + + if((imp = calloc(1, sizeof(imprint_t))) == NULL) { + goto done; + } + imp->hashID = hashIdentifier(hash->algorithm), + imp->len = hash->digest_length; + if((imp->data = (uint8_t*)malloc(imp->len)) == NULL) { + free(imp); imp = NULL; goto done; + } + memcpy(imp->data, hash->digest, imp->len); +done: return imp; +} + +void +rsgtimprintDel(imprint_t *imp) +{ + if(imp != NULL) { + free(imp->data), + free(imp); + } +} + +int +rsgtInit(char *usragent) +{ + int r = 0; + int ret = GT_OK; + + ret = GT_init(); + if(ret != GT_OK) { + r = 1; + goto done; + } + ret = GTHTTP_init(usragent, 1); + if(ret != GT_OK) { + r = 1; + goto done; + } +done: return r; +} + +void +rsgtExit(void) +{ + GTHTTP_finalize(); + GT_finalize(); +} + + +static inline gtfile +rsgtfileConstruct(gtctx ctx) +{ + gtfile gf; + if((gf = calloc(1, sizeof(struct gtfile_s))) == NULL) + goto done; + gf->ctx = ctx; + gf->hashAlg = ctx->hashAlg; + gf->blockSizeLimit = ctx->blockSizeLimit; + gf->bKeepRecordHashes = ctx->bKeepRecordHashes; + gf->bKeepTreeHashes = ctx->bKeepTreeHashes; + gf->x_prev = NULL; + +done: return gf; +} + +static inline int +tlvbufPhysWrite(gtfile gf) +{ + ssize_t lenBuf; + ssize_t iTotalWritten; + ssize_t iWritten; + char *pWriteBuf; + int r = 0; + + lenBuf = gf->tlvIdx; + pWriteBuf = gf->tlvBuf; + iTotalWritten = 0; + do { + iWritten = write(gf->fd, pWriteBuf, lenBuf); + if(iWritten < 0) { + iWritten = 0; /* we have written NO bytes! */ + if(errno == EINTR) { + /*NO ERROR, just continue */; + } else { + reportErr(gf->ctx, "signature file write error"); + r = RSGTE_IO; + goto finalize_it; + } + } + /* advance buffer to next write position */ + iTotalWritten += iWritten; + lenBuf -= iWritten; + pWriteBuf += iWritten; + } while(lenBuf > 0); /* Warning: do..while()! */ + +finalize_it: + gf->tlvIdx = 0; + return r; +} + +static inline int +tlvbufChkWrite(gtfile gf) +{ + if(gf->tlvIdx == sizeof(gf->tlvBuf)) { + return tlvbufPhysWrite(gf); + } + return 0; +} + + +/* write to TLV file buffer. If buffer is full, an actual call occurs. Else + * output is written only on flush or close. + */ +static inline int +tlvbufAddOctet(gtfile gf, int8_t octet) +{ + int r; + r = tlvbufChkWrite(gf); + if(r != 0) goto done; + gf->tlvBuf[gf->tlvIdx++] = octet; +done: return r; +} +static inline int +tlvbufAddOctetString(gtfile gf, uint8_t *octet, int size) +{ + int i, r = 0; + for(i = 0 ; i < size ; ++i) { + r = tlvbufAddOctet(gf, octet[i]); + if(r != 0) goto done; + } +done: return r; +} +/* return the actual length in to-be-written octets of an integer */ +static inline uint8_t +tlvbufGetInt64OctetSize(uint64_t val) +{ + if(val >> 56) + return 8; + if((val >> 48) & 0xff) + return 7; + if((val >> 40) & 0xff) + return 6; + if((val >> 32) & 0xff) + return 5; + if((val >> 24) & 0xff) + return 4; + if((val >> 16) & 0xff) + return 3; + if((val >> 8) & 0xff) + return 2; + return 1; +} +static inline int +tlvbufAddInt64(gtfile gf, uint64_t val) +{ + uint8_t doWrite = 0; + int r; + if(val >> 56) { + r = tlvbufAddOctet(gf, (val >> 56) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 48) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 48) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 40) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 40) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 32) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 32) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 24) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 24) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 16) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 16) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + if(doWrite || ((val >> 8) & 0xff)) { + r = tlvbufAddOctet(gf, (val >> 8) & 0xff), doWrite = 1; + if(r != 0) goto done; + } + r = tlvbufAddOctet(gf, val & 0xff); +done: return r; +} + + +int +tlv8Write(gtfile gf, int flags, int tlvtype, int len) +{ + int r; + r = tlvbufAddOctet(gf, (flags << 5)|tlvtype); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, len & 0xff); +done: return r; +} + +int +tlv16Write(gtfile gf, int flags, int tlvtype, uint16_t len) +{ + uint16_t typ; + int r; + typ = ((flags|1) << 15)|tlvtype; + r = tlvbufAddOctet(gf, typ >> 8); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, typ & 0xff); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, (len >> 8) & 0xff); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, len & 0xff); +done: return r; +} + +int +tlvFlush(gtfile gf) +{ + return (gf->tlvIdx == 0) ? 0 : tlvbufPhysWrite(gf); +} + +int +tlvWriteHash(gtfile gf, uint16_t tlvtype, GTDataHash *rec) +{ + unsigned tlvlen; + int r; + tlvlen = 1 + rec->digest_length; + r = tlv16Write(gf, 0x00, tlvtype, tlvlen); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, hashIdentifier(gf->hashAlg)); + if(r != 0) goto done; + r = tlvbufAddOctetString(gf, rec->digest, rec->digest_length); +done: return r; +} + +int +tlvWriteBlockSig(gtfile gf, uchar *der, uint16_t lenDer) +{ + unsigned tlvlen; + uint8_t tlvlenRecords; + int r; + + tlvlenRecords = tlvbufGetInt64OctetSize(gf->nRecords); + tlvlen = 2 + 1 /* hash algo TLV */ + + 2 + hashOutputLengthOctets(gf->hashAlg) /* iv */ + + 2 + 1 + gf->lenBlkStrtHash /* last hash */ + + 2 + tlvlenRecords /* rec-count */ + + 4 + lenDer /* rfc-3161 */; + /* write top-level TLV object (block-sig */ + r = tlv16Write(gf, 0x00, 0x0902, tlvlen); + if(r != 0) goto done; + /* and now write the children */ + //FIXME: flags??? + /* hash-algo */ + r = tlv8Write(gf, 0x00, 0x00, 1); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, hashIdentifier(gf->hashAlg)); + if(r != 0) goto done; + /* block-iv */ + r = tlv8Write(gf, 0x00, 0x01, hashOutputLengthOctets(gf->hashAlg)); + if(r != 0) goto done; + r = tlvbufAddOctetString(gf, gf->IV, hashOutputLengthOctets(gf->hashAlg)); + if(r != 0) goto done; + /* last-hash */ + r = tlv8Write(gf, 0x00, 0x02, gf->lenBlkStrtHash+1); + if(r != 0) goto done; + r = tlvbufAddOctet(gf, hashIdentifier(gf->hashAlg)); + if(r != 0) goto done; + r = tlvbufAddOctetString(gf, gf->blkStrtHash, gf->lenBlkStrtHash); + if(r != 0) goto done; + /* rec-count */ + r = tlv8Write(gf, 0x00, 0x03, tlvlenRecords); + if(r != 0) goto done; + r = tlvbufAddInt64(gf, gf->nRecords); + if(r != 0) goto done; + /* rfc-3161 */ + r = tlv16Write(gf, 0x00, 0x906, lenDer); + if(r != 0) goto done; + r = tlvbufAddOctetString(gf, der, lenDer); +done: return r; +} + +/* support for old platforms - graceful degrade */ +#ifndef O_CLOEXEC +#define O_CLOEXEC 0 +#endif +/* read rsyslog log state file; if we cannot access it or the + * contents looks invalid, we flag it as non-present (and thus + * begin a new hash chain). + * The context is initialized accordingly. + */ +static void +readStateFile(gtfile gf) +{ + int fd; + struct rsgtstatefile sf; + + fd = open((char*)gf->statefilename, O_RDONLY|O_NOCTTY|O_CLOEXEC, 0600); + if(fd == -1) goto err; + + if(read(fd, &sf, sizeof(sf)) != sizeof(sf)) goto err; + if(strncmp(sf.hdr, "GTSTAT10", 8)) goto err; + + gf->lenBlkStrtHash = sf.lenHash; + gf->blkStrtHash = calloc(1, gf->lenBlkStrtHash); + if(read(fd, gf->blkStrtHash, gf->lenBlkStrtHash) + != gf->lenBlkStrtHash) { + free(gf->blkStrtHash); + goto err; + } +return; + +err: + gf->lenBlkStrtHash = hashOutputLengthOctets(gf->hashAlg); + gf->blkStrtHash = calloc(1, gf->lenBlkStrtHash); +} + +/* persist all information that we need to re-open and append + * to a log signature file. + */ +static void +writeStateFile(gtfile gf) +{ + int fd; + struct rsgtstatefile sf; + + fd = open((char*)gf->statefilename, + O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY|O_CLOEXEC, 0600); + if(fd == -1) + goto done; + + memcpy(sf.hdr, "GTSTAT10", 8); + sf.hashID = hashIdentifier(gf->hashAlg); + sf.lenHash = gf->x_prev->len; + /* if the write fails, we cannot do anything against that. We check + * the condition just to keep the compiler happy. + */ + if(write(fd, &sf, sizeof(sf))){}; + if(write(fd, gf->x_prev->data, gf->x_prev->len)){}; + close(fd); +done: return; +} + + +int +tlvClose(gtfile gf) +{ + int r; + r = tlvFlush(gf); + close(gf->fd); + gf->fd = -1; + writeStateFile(gf); + return r; +} + + +/* note: if file exists, the last hash for chaining must + * be read from file. + */ +int +tlvOpen(gtfile gf, char *hdr, unsigned lenHdr) +{ + int r = 0; + gf->fd = open((char*)gf->sigfilename, + O_WRONLY|O_APPEND|O_NOCTTY|O_CLOEXEC, 0600); + if(gf->fd == -1) { + /* looks like we need to create a new file */ + gf->fd = open((char*)gf->sigfilename, + O_WRONLY|O_CREAT|O_NOCTTY|O_CLOEXEC, 0600); + if(gf->fd == -1) { + r = RSGTE_IO; + goto done; + } + memcpy(gf->tlvBuf, hdr, lenHdr); + gf->tlvIdx = lenHdr; + } else { + gf->tlvIdx = 0; /* header already present! */ + } + /* we now need to obtain the last previous hash, so that + * we can continue the hash chain. We do not check for error + * as a state file error can be recovered by graceful degredation. + */ + readStateFile(gf); +done: return r; +} + +/* + * As of some Linux and security expert I spoke to, /dev/urandom + * provides very strong random numbers, even if it runs out of + * entropy. As far as he knew, this is save for all applications + * (and he had good proof that I currently am not permitted to + * reproduce). -- rgerhards, 2013-03-04 + */ +void +seedIV(gtfile gf) +{ + int hashlen; + int fd; + + hashlen = hashOutputLengthOctets(gf->hashAlg); + gf->IV = malloc(hashlen); /* do NOT zero-out! */ + /* if we cannot obtain data from /dev/urandom, we use whatever + * is present at the current memory location as random data. Of + * course, this is very weak and we should consider a different + * option, especially when not running under Linux (for Linux, + * unavailability of /dev/urandom is just a theoretic thing, it + * will always work...). -- TODO -- rgerhards, 2013-03-06 + */ + if((fd = open("/dev/urandom", O_RDONLY)) > 0) { + if(read(fd, gf->IV, hashlen)) {}; /* keep compiler happy */ + close(fd); + } +} + +gtctx +rsgtCtxNew(void) +{ + gtctx ctx; + ctx = calloc(1, sizeof(struct gtctx_s)); + ctx->hashAlg = GT_HASHALG_SHA256; + ctx->errFunc = NULL; + ctx->usrptr = NULL; + ctx->timestamper = strdup( + "http://stamper.guardtime.net/gt-signingservice"); + return ctx; +} + +/* either returns gtfile object or NULL if something went wrong */ +gtfile +rsgtCtxOpenFile(gtctx ctx, unsigned char *logfn) +{ + gtfile gf; + char fn[MAXFNAME+1]; + + if((gf = rsgtfileConstruct(ctx)) == NULL) + goto done; + + snprintf(fn, sizeof(fn), "%s.gtsig", logfn); + fn[MAXFNAME] = '\0'; /* be on save side */ + gf->sigfilename = (uchar*) strdup(fn); + snprintf(fn, sizeof(fn), "%s.gtstate", logfn); + fn[MAXFNAME] = '\0'; /* be on save side */ + gf->statefilename = (uchar*) strdup(fn); + if(tlvOpen(gf, LOGSIGHDR, sizeof(LOGSIGHDR)-1) != 0) { + reportErr(ctx, "signature file open failed"); + gf = NULL; + } +done: return gf; +} + + +/* returns 0 on succes, 1 if algo is unknown */ +int +rsgtSetHashFunction(gtctx ctx, char *algName) +{ + int r = 0; + if(!strcmp(algName, "SHA2-256")) + ctx->hashAlg = GT_HASHALG_SHA256; + else if(!strcmp(algName, "SHA2-384")) + ctx->hashAlg = GT_HASHALG_SHA384; + else if(!strcmp(algName, "SHA2-512")) + ctx->hashAlg = GT_HASHALG_SHA512; + else if(!strcmp(algName, "SHA1")) + ctx->hashAlg = GT_HASHALG_SHA1; + else if(!strcmp(algName, "RIPEMD-160")) + ctx->hashAlg = GT_HASHALG_RIPEMD160; + else if(!strcmp(algName, "SHA2-224")) + ctx->hashAlg = GT_HASHALG_SHA224; + else + r = 1; + return r; +} + +int +rsgtfileDestruct(gtfile gf) +{ + int r = 0; + if(gf == NULL) + goto done; + + if(!gf->disabled && gf->bInBlk) { + r = sigblkFinish(gf); + if(r != 0) gf->disabled = 1; + } + if(!gf->disabled) + r = tlvClose(gf); + free(gf->sigfilename); + free(gf->statefilename); + free(gf->IV); + free(gf->blkStrtHash); + rsgtimprintDel(gf->x_prev); + free(gf); +done: return r; +} + +void +rsgtCtxDel(gtctx ctx) +{ + if(ctx != NULL) { + free(ctx->timestamper); + free(ctx); + } +} + +/* new sigblk is initialized, but maybe in existing ctx */ +void +sigblkInit(gtfile gf) +{ + if(gf == NULL) goto done; + seedIV(gf); + memset(gf->roots_valid, 0, sizeof(gf->roots_valid)/sizeof(char)); + gf->nRoots = 0; + gf->nRecords = 0; + gf->bInBlk = 1; +done: return; +} + + +/* concat: add IV to buffer */ +static inline void +bufAddIV(gtfile gf, uchar *buf, size_t *len) +{ + memcpy(buf+*len, gf->IV, hashOutputLengthOctets(gf->hashAlg)); + *len += sizeof(gf->IV); +} + + +/* concat: add imprint to buffer */ +static inline void +bufAddImprint(gtfile gf, uchar *buf, size_t *len, imprint_t *imp) +{ + if(imp == NULL) { + /* TODO: how to get the REAL HASH ID? --> add field? */ + buf[*len] = hashIdentifier(gf->hashAlg); + ++(*len); + memcpy(buf+*len, gf->blkStrtHash, gf->lenBlkStrtHash); + *len += gf->lenBlkStrtHash; + } else { + buf[*len] = imp->hashID; + ++(*len); + memcpy(buf+*len, imp->data, imp->len); + *len += imp->len; + } +} +/* concat: add hash to buffer */ +static inline void +bufAddHash(gtfile gf, uchar *buf, size_t *len, GTDataHash *hash) +{ + buf[*len] = hashIdentifier(gf->hashAlg); + ++(*len); + memcpy(buf+*len, hash->digest, hash->digest_length); + *len += hash->digest_length; +} +/* concat: add tree level to buffer */ +static inline void +bufAddLevel(uchar *buf, size_t *len, uint8_t level) +{ + memcpy(buf+*len, &level, sizeof(level)); + *len += sizeof(level); +} + + +int +hash_m(gtfile gf, GTDataHash **m) +{ + int rgt; + uchar concatBuf[16*1024]; + size_t len = 0; + int r = 0; + + bufAddImprint(gf, concatBuf, &len, gf->x_prev); + bufAddIV(gf, concatBuf, &len); + rgt = GTDataHash_create(gf->hashAlg, concatBuf, len, m); + if(rgt != GT_OK) { + reportGTAPIErr(gf->ctx, gf, "GTDataHash_create", rgt); + r = RSGTE_HASH_CREATE; + goto done; + } +done: return r; +} + +int +hash_r(gtfile gf, GTDataHash **r, const uchar *rec, const size_t len) +{ + int ret = 0, rgt; + rgt = GTDataHash_create(gf->hashAlg, rec, len, r); + if(rgt != GT_OK) { + reportGTAPIErr(gf->ctx, gf, "GTDataHash_create", rgt); + ret = RSGTE_HASH_CREATE; + goto done; + } +done: return ret; +} + + +int +hash_node(gtfile gf, GTDataHash **node, GTDataHash *m, GTDataHash *rec, + uint8_t level) +{ + int r = 0, rgt; + uchar concatBuf[16*1024]; + size_t len = 0; + + bufAddHash(gf, concatBuf, &len, m); + bufAddHash(gf, concatBuf, &len, rec); + bufAddLevel(concatBuf, &len, level); + rgt = GTDataHash_create(gf->hashAlg, concatBuf, len, node); + if(rgt != GT_OK) { + reportGTAPIErr(gf->ctx, gf, "GTDataHash_create", rgt); + r = RSGTE_HASH_CREATE; + goto done; + } +done: return r; +} + + +int +sigblkAddRecord(gtfile gf, const uchar *rec, const size_t len) +{ + GTDataHash *x; /* current hash */ + GTDataHash *m, *r, *t, *t_del; + uint8_t j; + int ret = 0; + + if(gf == NULL || gf->disabled) goto done; + if((ret = hash_m(gf, &m)) != 0) goto done; + if((ret = hash_r(gf, &r, rec, len)) != 0) goto done; + if(gf->bKeepRecordHashes) + tlvWriteHash(gf, 0x0900, r); + if((ret = hash_node(gf, &x, m, r, 1)) != 0) goto done; /* hash leaf */ + /* persists x here if Merkle tree needs to be persisted! */ + if(gf->bKeepTreeHashes) + tlvWriteHash(gf, 0x0901, x); + rsgtimprintDel(gf->x_prev); + gf->x_prev = rsgtImprintFromGTDataHash(x); + /* add x to the forest as new leaf, update roots list */ + t = x; + for(j = 0 ; j < gf->nRoots ; ++j) { + if(gf->roots_valid[j] == 0) { + gf->roots_hash[j] = t; + gf->roots_valid[j] = 1; + t = NULL; + break; + } else if(t != NULL) { + /* hash interim node */ + t_del = t; + ret = hash_node(gf, &t, gf->roots_hash[j], t_del, j+2); + gf->roots_valid[j] = 0; + GTDataHash_free(gf->roots_hash[j]); + GTDataHash_free(t_del); + if(ret != 0) goto done; + if(gf->bKeepTreeHashes) + tlvWriteHash(gf, 0x0901, t); + } + } + if(t != NULL) { + /* new level, append "at the top" */ + gf->roots_hash[gf->nRoots] = t; + gf->roots_valid[gf->nRoots] = 1; + ++gf->nRoots; + assert(gf->nRoots < MAX_ROOTS); + t = NULL; + } + ++gf->nRecords; + + /* cleanup (x is cleared as part of the roots array) */ + GTDataHash_free(m); + GTDataHash_free(r); + + if(gf->nRecords == gf->blockSizeLimit) { + ret = sigblkFinish(gf); + if(ret != 0) goto done; + sigblkInit(gf); + } +done: + if(ret != 0) { + gf->disabled = 1; + } + return ret; +} + +static int +timestampIt(gtfile gf, GTDataHash *hash) +{ + unsigned char *der = NULL; + size_t lenDer; + int r = GT_OK; + int ret = 0; + GTTimestamp *timestamp = NULL; + + /* Get the timestamp. */ + r = GTHTTP_createTimestampHash(hash, gf->ctx->timestamper, ×tamp); + + if(r != GT_OK) { + reportGTAPIErr(gf->ctx, gf, "GTHTTP_createTimestampHash", r); + ret = 1; + goto done; + } + + /* Encode timestamp. */ + r = GTTimestamp_getDEREncoded(timestamp, &der, &lenDer); + if(r != GT_OK) { + reportGTAPIErr(gf->ctx, gf, "GTTimestamp_getDEREncoded", r); + ret = 1; + goto done; + } + + tlvWriteBlockSig(gf, der, lenDer); + +done: + GT_free(der); + GTTimestamp_free(timestamp); + return ret; +} + + +int +sigblkFinish(gtfile gf) +{ + GTDataHash *root, *rootDel; + int8_t j; + int ret = 0; + + if(gf->nRecords == 0) + goto done; + + root = NULL; + for(j = 0 ; j < gf->nRoots ; ++j) { + if(root == NULL) { + root = gf->roots_valid[j] ? gf->roots_hash[j] : NULL; + gf->roots_valid[j] = 0; + } else if(gf->roots_valid[j]) { + rootDel = root; + ret = hash_node(gf, &root, gf->roots_hash[j], rootDel, j+2); + gf->roots_valid[j] = 0; + GTDataHash_free(gf->roots_hash[j]); + GTDataHash_free(rootDel); + if(ret != 0) goto done; /* checks hash_node() result! */ + } + } + if((ret = timestampIt(gf, root)) != 0) goto done; + + GTDataHash_free(root); + free(gf->blkStrtHash); + gf->lenBlkStrtHash = gf->x_prev->len; + gf->blkStrtHash = malloc(gf->lenBlkStrtHash); + memcpy(gf->blkStrtHash, gf->x_prev->data, gf->x_prev->len); +done: + gf->bInBlk = 0; + return ret; +} diff --git a/runtime/librsgt.h b/runtime/librsgt.h new file mode 100644 index 00000000..bfcc4628 --- /dev/null +++ b/runtime/librsgt.h @@ -0,0 +1,388 @@ +/* librsgt.h - rsyslog's guardtime support library + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_LIBRSGT_H +#define INCLUDED_LIBRSGT_H +#include <gt_base.h> + +/* Max number of roots inside the forest. This permits blocks of up to + * 2^MAX_ROOTS records. We assume that 64 is sufficient for all use + * cases ;) [and 64 is not really a waste of memory, so we do not even + * try to work with reallocs and such...] + */ +#define MAX_ROOTS 64 +#define LOGSIGHDR "LOGSIG10" + +/* context for gt calls. This primarily serves as a container for the + * config settings. The actual file-specific data is kept in gtfile. + */ +struct gtctx_s { + enum GTHashAlgorithm hashAlg; + uint8_t bKeepRecordHashes; + uint8_t bKeepTreeHashes; + uint64_t blockSizeLimit; + char *timestamper; + void (*errFunc)(void *, unsigned char*); + void *usrptr; /* for error function */ +}; +typedef struct gtctx_s *gtctx; +typedef struct gtfile_s *gtfile; +typedef struct gterrctx_s gterrctx_t; +typedef struct imprint_s imprint_t; +typedef struct block_sig_s block_sig_t; +typedef struct tlvrecord_s tlvrecord_t; + +/* this describes a file, as far as librsgt is concerned */ +struct gtfile_s { + /* the following data items are mirrored from gtctx to + * increase cache hit ratio (they are frequently accesed). + */ + enum GTHashAlgorithm hashAlg; + uint8_t bKeepRecordHashes; + uint8_t bKeepTreeHashes; + /* end mirrored properties */ + uint8_t disabled; /* permits to disable this file --> set to 1 */ + uint64_t blockSizeLimit; + uint8_t *IV; /* initial value for blinding masks */ + imprint_t *x_prev; /* last leaf hash (maybe of previous block) --> preserve on term */ + unsigned char *sigfilename; + unsigned char *statefilename; + int fd; + unsigned char *blkStrtHash; /* last hash from previous block */ + uint16_t lenBlkStrtHash; + uint64_t nRecords; /* current number of records in current block */ + uint64_t bInBlk; /* are we currently inside a blk --> need to finish on close */ + int8_t nRoots; + /* algo engineering: roots structure is split into two arrays + * in order to improve cache hits. + */ + int8_t roots_valid[MAX_ROOTS]; + GTDataHash *roots_hash[MAX_ROOTS]; + /* data members for the associated TLV file */ + char tlvBuf[4096]; + int tlvIdx; /* current index into tlvBuf */ + gtctx ctx; +}; + +struct tlvrecord_s { + uint16_t tlvtype; + uint16_t tlvlen; + uint8_t hdr[4]; /* the raw header (as persisted to file) */ + uint8_t lenHdr; /* length of raw header */ + uint8_t data[64*1024]; /* the actual data part (of length tlvlen) */ +}; + +/* The following structure describes the "error context" to be used + * for verification and similiar reader functions. While verifying, + * we need some information (like filenames or block numbers) that + * is not readily available from the other objects (or not even known + * to librsgt). In order to provide meaningful error messages, this + * information must be passed in from the external callers. In order + * to centralize information (and make it more manageable), we use + * ths error context here, which contains everything needed to + * generate good error messages. Members of this structure are + * maintained both by library users (the callers) as well as + * the library itself. Who does what simply depends on who has + * the relevant information. + */ +struct gterrctx_s { + FILE *fp; /**< file for error messages */ + char *filename; + uint8_t verbose; + uint64_t recNumInFile; + uint64_t recNum; + uint64_t blkNum; + uint8_t treeLevel; + GTDataHash *computedHash; + GTDataHash *lefthash, *righthash; /* hashes to display if tree hash fails */ + imprint_t *fileHash; + int gtstate; /* status from last relevant GT.*() function call */ + char *errRec; + char *frstRecInBlk; /* This holds the first message seen inside the current block */ +}; + +struct imprint_s { + uint8_t hashID; + int len; + uint8_t *data; +}; + +#define SIGID_RFC3161 0 +struct block_sig_s { + uint8_t hashID; + uint8_t sigID; /* what type of *signature*? */ + uint8_t *iv; + imprint_t lastHash; + uint64_t recCount; + struct { + struct { + uint8_t *data; + size_t len; /* must be size_t due to GT API! */ + } der; + } sig; +}; + + +/* the following defines the gtstate file record. Currently, this record + * is fixed, we may change that over time. + */ +struct rsgtstatefile { + char hdr[8]; /* must be "GTSTAT10" */ + uint8_t hashID; + uint8_t lenHash; + /* after that, the hash value is contained within the file */ +}; + +/* Flags and record types for TLV handling */ +#define RSGT_FLAG_TLV16 0x20 + +/* error states */ +#define RSGTE_IO 1 /* any kind of io error */ +#define RSGTE_FMT 2 /* data fromat error */ +#define RSGTE_INVLTYP 3 /* invalid TLV type record (unexcpected at this point) */ +#define RSGTE_OOM 4 /* ran out of memory */ +#define RSGTE_LEN 5 /* error related to length records */ +#define RSGTE_TS_EXTEND 6/* error extending timestamp */ +#define RSGTE_INVLD_RECCNT 7/* mismatch between actual records and records + given in block-sig record */ +#define RSGTE_INVLHDR 8/* invalid file header */ +#define RSGTE_EOF 9 /* specific EOF */ +#define RSGTE_MISS_REC_HASH 10 /* record hash missing when expected */ +#define RSGTE_MISS_TREE_HASH 11 /* tree hash missing when expected */ +#define RSGTE_INVLD_REC_HASH 12 /* invalid record hash (failed verification) */ +#define RSGTE_INVLD_TREE_HASH 13 /* invalid tree hash (failed verification) */ +#define RSGTE_INVLD_REC_HASHID 14 /* invalid record hash ID (failed verification) */ +#define RSGTE_INVLD_TREE_HASHID 15 /* invalid tree hash ID (failed verification) */ +#define RSGTE_MISS_BLOCKSIG 16 /* block signature record missing when expected */ +#define RSGTE_INVLD_TIMESTAMP 17 /* RFC3161 timestamp is invalid */ +#define RSGTE_TS_DERDECODE 18 /* error DER-Decoding a timestamp */ +#define RSGTE_TS_DERENCODE 19 /* error DER-Encoding a timestamp */ +#define RSGTE_HASH_CREATE 20 /* error creating a hash */ + +/* the following function maps RSGTE_* state to a string - must be updated + * whenever a new state is added. + * Note: it is thread-safe to call this function, as it returns a pointer + * into constant memory pool. + */ +static inline char * +RSGTE2String(int err) +{ + switch(err) { + case 0: + return "success"; + case RSGTE_IO: + return "i/o error"; + case RSGTE_FMT: + return "data format error"; + case RSGTE_INVLTYP: + return "invalid/unexpected tlv record type"; + case RSGTE_OOM: + return "out of memory"; + case RSGTE_LEN: + return "length record problem"; + case RSGTE_TS_EXTEND: + return "error extending timestamp"; + case RSGTE_INVLD_RECCNT: + return "mismatch between actual record count and number in block signature record"; + case RSGTE_INVLHDR: + return "invalid file header"; + case RSGTE_EOF: + return "EOF"; + case RSGTE_MISS_REC_HASH: + return "record hash missing"; + case RSGTE_MISS_TREE_HASH: + return "tree hash missing"; + case RSGTE_INVLD_REC_HASH: + return "record hash mismatch"; + case RSGTE_INVLD_TREE_HASH: + return "tree hash mismatch"; + case RSGTE_INVLD_REC_HASHID: + return "invalid record hash ID"; + case RSGTE_INVLD_TREE_HASHID: + return "invalid tree hash ID"; + case RSGTE_MISS_BLOCKSIG: + return "missing block signature record"; + case RSGTE_INVLD_TIMESTAMP: + return "RFC3161 timestamp invalid"; + case RSGTE_TS_DERDECODE: + return "error DER-decoding RFC3161 timestamp"; + case RSGTE_TS_DERENCODE: + return "error DER-encoding RFC3161 timestamp"; + case RSGTE_HASH_CREATE: + return "error creating hash"; + default: + return "unknown error"; + } +} + + +static inline uint16_t +hashOutputLengthOctets(uint8_t hashID) +{ + switch(hashID) { + case GT_HASHALG_SHA1: /* paper: SHA1 */ + return 20; + case GT_HASHALG_RIPEMD160: /* paper: RIPEMD-160 */ + return 20; + case GT_HASHALG_SHA224: /* paper: SHA2-224 */ + return 28; + case GT_HASHALG_SHA256: /* paper: SHA2-256 */ + return 32; + case GT_HASHALG_SHA384: /* paper: SHA2-384 */ + return 48; + case GT_HASHALG_SHA512: /* paper: SHA2-512 */ + return 64; + default:return 32; + } +} + +static inline uint8_t +hashIdentifier(enum GTHashAlgorithm hashID) +{ + switch(hashID) { + case GT_HASHALG_SHA1: /* paper: SHA1 */ + return 0x00; + case GT_HASHALG_RIPEMD160: /* paper: RIPEMD-160 */ + return 0x02; + case GT_HASHALG_SHA224: /* paper: SHA2-224 */ + return 0x03; + case GT_HASHALG_SHA256: /* paper: SHA2-256 */ + return 0x01; + case GT_HASHALG_SHA384: /* paper: SHA2-384 */ + return 0x04; + case GT_HASHALG_SHA512: /* paper: SHA2-512 */ + return 0x05; + default:return 0xff; + } +} +static inline char * +hashAlgName(uint8_t hashID) +{ + switch(hashID) { + case GT_HASHALG_SHA1: + return "SHA1"; + case GT_HASHALG_RIPEMD160: + return "RIPEMD-160"; + case GT_HASHALG_SHA224: + return "SHA2-224"; + case GT_HASHALG_SHA256: + return "SHA2-256"; + case GT_HASHALG_SHA384: + return "SHA2-384"; + case GT_HASHALG_SHA512: + return "SHA2-512"; + default:return "[unknown]"; + } +} +static inline enum GTHashAlgorithm +hashID2Alg(uint8_t hashID) +{ + switch(hashID) { + case 0x00: + return GT_HASHALG_SHA1; + case 0x02: + return GT_HASHALG_RIPEMD160; + case 0x03: + return GT_HASHALG_SHA224; + case 0x01: + return GT_HASHALG_SHA256; + case 0x04: + return GT_HASHALG_SHA384; + case 0x05: + return GT_HASHALG_SHA512; + default: + return 0xff; + } +} +static inline char * +sigTypeName(uint8_t sigID) +{ + switch(sigID) { + case SIGID_RFC3161: + return "RFC3161"; + default:return "[unknown]"; + } +} +static inline uint16_t +getIVLen(block_sig_t *bs) +{ + return hashOutputLengthOctets(bs->hashID); +} +static inline void +rsgtSetTimestamper(gtctx ctx, char *timestamper) +{ + free(ctx->timestamper); + ctx->timestamper = strdup(timestamper); +} +static inline void +rsgtSetBlockSizeLimit(gtctx ctx, uint64_t limit) +{ + ctx->blockSizeLimit = limit; +} +static inline void +rsgtSetKeepRecordHashes(gtctx ctx, int val) +{ + ctx->bKeepRecordHashes = val; +} +static inline void +rsgtSetKeepTreeHashes(gtctx ctx, int val) +{ + ctx->bKeepTreeHashes = val; +} + +int rsgtSetHashFunction(gtctx ctx, char *algName); +int rsgtInit(char *usragent); +void rsgtExit(void); +gtctx rsgtCtxNew(void); +void rsgtsetErrFunc(gtctx ctx, void (*func)(void*, unsigned char *), void *usrptr); +gtfile rsgtCtxOpenFile(gtctx ctx, unsigned char *logfn); +int rsgtfileDestruct(gtfile gf); +void rsgtCtxDel(gtctx ctx); +void sigblkInit(gtfile gf); +int sigblkAddRecord(gtfile gf, const unsigned char *rec, const size_t len); +int sigblkFinish(gtfile gf); +imprint_t * rsgtImprintFromGTDataHash(GTDataHash *hash); +void rsgtimprintDel(imprint_t *imp); +/* reader functions */ +int rsgt_tlvrdHeader(FILE *fp, unsigned char *hdr); +int rsgt_tlvrd(FILE *fp, tlvrecord_t *rec, void *obj); +void rsgt_tlvprint(FILE *fp, uint16_t tlvtype, void *obj, uint8_t verbose); +void rsgt_printBLOCK_SIG(FILE *fp, block_sig_t *bs, uint8_t verbose); +int rsgt_getBlockParams(FILE *fp, uint8_t bRewind, block_sig_t **bs, uint8_t *bHasRecHashes, uint8_t *bHasIntermedHashes); +int rsgt_chkFileHdr(FILE *fp, char *expect); +gtfile rsgt_vrfyConstruct_gf(void); +void rsgt_vrfyBlkInit(gtfile gf, block_sig_t *bs, uint8_t bHasRecHashes, uint8_t bHasIntermedHashes); +int rsgt_vrfy_nextRec(block_sig_t *bs, gtfile gf, FILE *sigfp, FILE *nsigfp, unsigned char *rec, size_t len, gterrctx_t *ectx); +int verifyBLOCK_SIG(block_sig_t *bs, gtfile gf, FILE *sigfp, FILE *nsigfp, uint8_t bExtend, gterrctx_t *ectx); +void rsgt_errctxInit(gterrctx_t *ectx); +void rsgt_errctxExit(gterrctx_t *ectx); +void rsgt_errctxSetErrRec(gterrctx_t *ectx, char *rec); +void rsgt_errctxFrstRecInBlk(gterrctx_t *ectx, char *rec); +void rsgt_objfree(uint16_t tlvtype, void *obj); + + +/* TODO: replace these? */ +int hash_m(gtfile gf, GTDataHash **m); +int hash_r(gtfile gf, GTDataHash **r, const unsigned char *rec, const size_t len); +int hash_node(gtfile gf, GTDataHash **node, GTDataHash *m, GTDataHash *r, uint8_t level); +extern char *rsgt_read_puburl; /**< url of publication server */ +extern uint8_t rsgt_read_showVerified; + +#endif /* #ifndef INCLUDED_LIBRSGT_H */ diff --git a/runtime/librsgt_read.c b/runtime/librsgt_read.c new file mode 100644 index 00000000..a6e33160 --- /dev/null +++ b/runtime/librsgt_read.c @@ -0,0 +1,1092 @@ +/* librsgt_read.c - rsyslog's guardtime support library + * This includes functions used for reading signature (and + * other related) files. Well, actually it also contains + * some writing functionality, but only as far as rsyslog + * itself is not concerned, but "just" the utility programs. + * + * This part of the library uses C stdio and expects that the + * caller will open and close the file to be read itself. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdio.h> +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> +#include <assert.h> +#include <unistd.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <gt_http.h> + +#include "librsgt.h" + +typedef unsigned char uchar; +#ifndef VERSION +#define VERSION "no-version" +#endif +#define MAXFNAME 1024 + +static int rsgt_read_debug = 0; +char *rsgt_read_puburl = "http://verify.guardtime.com/gt-controlpublications.bin"; +char *rsgt_extend_puburl = "http://verifier.guardtime.net/gt-extendingservice"; +uint8_t rsgt_read_showVerified = 0; + +/* macro to obtain next char from file including error tracking */ +#define NEXTC if((c = fgetc(fp)) == EOF) { \ + r = feof(fp) ? RSGTE_EOF : RSGTE_IO; \ + goto done; \ + } + +/* check return state of operation and abort, if non-OK */ +#define CHKr(code) if((r = code) != 0) goto done + + +/* if verbose==0, only the first and last two octets are shown, + * otherwise everything. + */ +static void +outputHexBlob(FILE *fp, uint8_t *blob, uint16_t len, uint8_t verbose) +{ + unsigned i; + if(verbose || len <= 8) { + for(i = 0 ; i < len ; ++i) + fprintf(fp, "%2.2x", blob[i]); + } else { + fprintf(fp, "%2.2x%2.2x%2.2x[...]%2.2x%2.2x%2.2x", + blob[0], blob[1], blob[2], + blob[len-3], blob[len-2], blob[len-1]); + } +} + +static inline void +outputHash(FILE *fp, char *hdr, uint8_t *data, uint16_t len, uint8_t verbose) +{ + fprintf(fp, "%s", hdr); + outputHexBlob(fp, data, len, verbose); + fputc('\n', fp); +} + +void +rsgt_errctxInit(gterrctx_t *ectx) +{ + ectx->fp = NULL; + ectx->filename = NULL; + ectx->recNum = 0; + ectx->gtstate = 0; + ectx->recNumInFile = 0; + ectx->blkNum = 0; + ectx->verbose = 0; + ectx->errRec = NULL; + ectx->frstRecInBlk = NULL; + ectx->fileHash = NULL; + ectx->lefthash = ectx->righthash = ectx->computedHash = NULL; +} +void +rsgt_errctxExit(gterrctx_t *ectx) +{ + free(ectx->filename); + free(ectx->frstRecInBlk); +} + +/* note: we do not copy the record, so the caller MUST not destruct + * it before processing of the record is completed. To remove the + * current record without setting a new one, call this function + * with rec==NULL. + */ +void +rsgt_errctxSetErrRec(gterrctx_t *ectx, char *rec) +{ + ectx->errRec = strdup(rec); +} +/* This stores the block's first record. Here we copy the data, + * as the caller will usually not preserve it long enough. + */ +void +rsgt_errctxFrstRecInBlk(gterrctx_t *ectx, char *rec) +{ + free(ectx->frstRecInBlk); + ectx->frstRecInBlk = strdup(rec); +} + +static void +reportError(int errcode, gterrctx_t *ectx) +{ + if(ectx->fp != NULL) { + fprintf(ectx->fp, "%s[%llu:%llu:%llu]: error[%u]: %s\n", + ectx->filename, + (long long unsigned) ectx->blkNum, (long long unsigned) ectx->recNum, + (long long unsigned) ectx->recNumInFile, + errcode, RSGTE2String(errcode)); + if(ectx->frstRecInBlk != NULL) + fprintf(ectx->fp, "\tBlock Start Record.: '%s'\n", ectx->frstRecInBlk); + if(ectx->errRec != NULL) + fprintf(ectx->fp, "\tRecord in Question.: '%s'\n", ectx->errRec); + if(ectx->computedHash != NULL) { + outputHash(ectx->fp, "\tComputed Hash......: ", ectx->computedHash->digest, + ectx->computedHash->digest_length, ectx->verbose); + } + if(ectx->fileHash != NULL) { + outputHash(ectx->fp, "\tSignature File Hash: ", ectx->fileHash->data, + ectx->fileHash->len, ectx->verbose); + } + if(errcode == RSGTE_INVLD_TREE_HASH || + errcode == RSGTE_INVLD_TREE_HASHID) { + fprintf(ectx->fp, "\tTree Level.........: %d\n", (int) ectx->treeLevel); + outputHash(ectx->fp, "\tTree Left Hash.....: ", ectx->lefthash->digest, + ectx->lefthash->digest_length, ectx->verbose); + outputHash(ectx->fp, "\tTree Right Hash....: ", ectx->righthash->digest, + ectx->righthash->digest_length, ectx->verbose); + } + if(errcode == RSGTE_INVLD_TIMESTAMP || + errcode == RSGTE_TS_DERDECODE) { + fprintf(ectx->fp, "\tPublication Server.: %s\n", rsgt_read_puburl); + fprintf(ectx->fp, "\tGT Verify Timestamp: [%u]%s\n", + ectx->gtstate, GTHTTP_getErrorString(ectx->gtstate)); + } + if(errcode == RSGTE_TS_EXTEND || + errcode == RSGTE_TS_DERDECODE) { + fprintf(ectx->fp, "\tExtending Server...: %s\n", rsgt_extend_puburl); + fprintf(ectx->fp, "\tGT Extend Timestamp: [%u]%s\n", + ectx->gtstate, GTHTTP_getErrorString(ectx->gtstate)); + } + if(errcode == RSGTE_TS_DERENCODE) { + fprintf(ectx->fp, "\tAPI return state...: [%u]%s\n", + ectx->gtstate, GTHTTP_getErrorString(ectx->gtstate)); + } + } +} + +/* obviously, this is not an error-reporting function. We still use + * ectx, as it has most information we need. + */ +static void +reportVerifySuccess(gterrctx_t *ectx, GTVerificationInfo *vrfyInf) +{ + if(ectx->fp != NULL) { + fprintf(ectx->fp, "%s[%llu:%llu:%llu]: block signature successfully verified\n", + ectx->filename, + (long long unsigned) ectx->blkNum, (long long unsigned) ectx->recNum, + (long long unsigned) ectx->recNumInFile); + if(ectx->frstRecInBlk != NULL) + fprintf(ectx->fp, "\tBlock Start Record.: '%s'\n", ectx->frstRecInBlk); + if(ectx->errRec != NULL) + fprintf(ectx->fp, "\tBlock End Record...: '%s'\n", ectx->errRec); + fprintf(ectx->fp, "\tGT Verify Timestamp: [%u]%s\n", + ectx->gtstate, GTHTTP_getErrorString(ectx->gtstate)); + GTVerificationInfo_print(ectx->fp, 0, vrfyInf); + } +} + +/** + * Write the provided record to the current file position. + * + * @param[in] fp file pointer for writing + * @param[out] rec tlvrecord to write + * + * @returns 0 if ok, something else otherwise + */ +static int +rsgt_tlvwrite(FILE *fp, tlvrecord_t *rec) +{ + int r = RSGTE_IO; + if(fwrite(rec->hdr, (size_t) rec->lenHdr, 1, fp) != 1) goto done; + if(fwrite(rec->data, (size_t) rec->tlvlen, 1, fp) != 1) goto done; + r = 0; +done: return r; +} + +/** + * Read a header from a binary file. + * @param[in] fp file pointer for processing + * @param[in] hdr buffer for the header. Must be 9 bytes + * (8 for header + NUL byte) + * @returns 0 if ok, something else otherwise + */ +int +rsgt_tlvrdHeader(FILE *fp, uchar *hdr) +{ + int r; + if(fread(hdr, 8, 1, fp) != 1) { + r = RSGTE_IO; + goto done; + } + hdr[8] = '\0'; + r = 0; +done: return r; +} + +/* read type a complete tlv record + */ +static int +rsgt_tlvRecRead(FILE *fp, tlvrecord_t *rec) +{ + int r = 1; + int c; + + NEXTC; + rec->hdr[0] = c; + rec->tlvtype = c & 0x1f; + if(c & 0x80) { /* tlv16? */ + rec->lenHdr = 4; + NEXTC; + rec->hdr[1] = c; + rec->tlvtype = (rec->tlvtype << 8) | c; + NEXTC; + rec->hdr[2] = c; + rec->tlvlen = c << 8; + NEXTC; + rec->hdr[3] = c; + rec->tlvlen |= c; + } else { + NEXTC; + rec->lenHdr = 2; + rec->hdr[1] = c; + rec->tlvlen = c; + } + if(fread(rec->data, (size_t) rec->tlvlen, 1, fp) != 1) { + r = RSGTE_IO; + goto done; + } + + if(rsgt_read_debug) + printf("read tlvtype %4.4x, len %u\n", (unsigned) rec->tlvtype, + (unsigned) rec->tlvlen); + r = 0; +done: return r; +} + +/* decode a sub-tlv record from an existing record's memory buffer + */ +static int +rsgt_tlvDecodeSUBREC(tlvrecord_t *rec, uint16_t *stridx, tlvrecord_t *newrec) +{ + int r = 1; + int c; + + if(rec->tlvlen == *stridx) {r=RSGTE_LEN; goto done;} + c = rec->data[(*stridx)++]; + newrec->hdr[0] = c; + newrec->tlvtype = c & 0x1f; + if(c & 0x80) { /* tlv16? */ + newrec->lenHdr = 4; + if(rec->tlvlen == *stridx) {r=RSGTE_LEN; goto done;} + c = rec->data[(*stridx)++]; + newrec->hdr[1] = c; + newrec->tlvtype = (newrec->tlvtype << 8) | c; + if(rec->tlvlen == *stridx) {r=RSGTE_LEN; goto done;} + c = rec->data[(*stridx)++]; + newrec->hdr[2] = c; + newrec->tlvlen = c << 8; + if(rec->tlvlen == *stridx) {r=RSGTE_LEN; goto done;} + c = rec->data[(*stridx)++]; + newrec->hdr[3] = c; + newrec->tlvlen |= c; + } else { + if(rec->tlvlen == *stridx) {r=RSGTE_LEN; goto done;} + c = rec->data[(*stridx)++]; + newrec->lenHdr = 2; + newrec->hdr[1] = c; + newrec->tlvlen = c; + } + if(rec->tlvlen < *stridx + newrec->tlvlen) {r=RSGTE_LEN; goto done;} + memcpy(newrec->data, (rec->data)+(*stridx), newrec->tlvlen); + *stridx += newrec->tlvlen; + + if(rsgt_read_debug) + printf("read sub-tlv: tlvtype %4.4x, len %u\n", + (unsigned) newrec->tlvtype, + (unsigned) newrec->tlvlen); + r = 0; +done: return r; +} + + +static int +rsgt_tlvDecodeIMPRINT(tlvrecord_t *rec, imprint_t **imprint) +{ + int r = 1; + imprint_t *imp; + + if((imp = calloc(1, sizeof(imprint_t))) == NULL) { + r = RSGTE_OOM; + goto done; + } + + imp->hashID = rec->data[0]; + if(rec->tlvlen != 1 + hashOutputLengthOctets(imp->hashID)) { + r = RSGTE_LEN; + goto done; + } + imp->len = rec->tlvlen - 1; + if((imp->data = (uint8_t*)malloc(imp->len)) == NULL) {r=RSGTE_OOM;goto done;} + memcpy(imp->data, rec->data+1, imp->len); + *imprint = imp; + r = 0; +done: return r; +} + +static int +rsgt_tlvDecodeHASH_ALGO(tlvrecord_t *rec, uint16_t *strtidx, uint8_t *hashAlg) +{ + int r = 1; + tlvrecord_t subrec; + + CHKr(rsgt_tlvDecodeSUBREC(rec, strtidx, &subrec)); + if(!(subrec.tlvtype == 0x00 && subrec.tlvlen == 1)) { + r = RSGTE_FMT; + goto done; + } + *hashAlg = subrec.data[0]; + r = 0; +done: return r; +} +static int +rsgt_tlvDecodeBLOCK_IV(tlvrecord_t *rec, uint16_t *strtidx, uint8_t **iv) +{ + int r = 1; + tlvrecord_t subrec; + + CHKr(rsgt_tlvDecodeSUBREC(rec, strtidx, &subrec)); + if(!(subrec.tlvtype == 0x01)) { + r = RSGTE_INVLTYP; + goto done; + } + if((*iv = (uint8_t*)malloc(subrec.tlvlen)) == NULL) {r=RSGTE_OOM;goto done;} + memcpy(*iv, subrec.data, subrec.tlvlen); + r = 0; +done: return r; +} +static int +rsgt_tlvDecodeLAST_HASH(tlvrecord_t *rec, uint16_t *strtidx, imprint_t *imp) +{ + int r = 1; + tlvrecord_t subrec; + + CHKr(rsgt_tlvDecodeSUBREC(rec, strtidx, &subrec)); + if(!(subrec.tlvtype == 0x02)) { r = RSGTE_INVLTYP; goto done; } + imp->hashID = subrec.data[0]; + if(subrec.tlvlen != 1 + hashOutputLengthOctets(imp->hashID)) { + r = RSGTE_LEN; + goto done; + } + imp->len = subrec.tlvlen - 1; + if((imp->data = (uint8_t*)malloc(imp->len)) == NULL) {r=RSGTE_OOM;goto done;} + memcpy(imp->data, subrec.data+1, subrec.tlvlen-1); + r = 0; +done: return r; +} +static int +rsgt_tlvDecodeREC_COUNT(tlvrecord_t *rec, uint16_t *strtidx, uint64_t *cnt) +{ + int r = 1; + int i; + uint64_t val; + tlvrecord_t subrec; + + CHKr(rsgt_tlvDecodeSUBREC(rec, strtidx, &subrec)); + if(!(subrec.tlvtype == 0x03 && subrec.tlvlen <= 8)) { r = RSGTE_INVLTYP; goto done; } + val = 0; + for(i = 0 ; i < subrec.tlvlen ; ++i) { + val = (val << 8) + subrec.data[i]; + } + *cnt = val; + r = 0; +done: return r; +} +static int +rsgt_tlvDecodeSIG(tlvrecord_t *rec, uint16_t *strtidx, block_sig_t *bs) +{ + int r = 1; + tlvrecord_t subrec; + + CHKr(rsgt_tlvDecodeSUBREC(rec, strtidx, &subrec)); + if(!(subrec.tlvtype == 0x0906)) { r = RSGTE_INVLTYP; goto done; } + bs->sig.der.len = subrec.tlvlen; + bs->sigID = SIGID_RFC3161; + if((bs->sig.der.data = (uint8_t*)malloc(bs->sig.der.len)) == NULL) {r=RSGTE_OOM;goto done;} + memcpy(bs->sig.der.data, subrec.data, bs->sig.der.len); + r = 0; +done: return r; +} + +static int +rsgt_tlvDecodeBLOCK_SIG(tlvrecord_t *rec, block_sig_t **blocksig) +{ + int r = 1; + uint16_t strtidx = 0; + block_sig_t *bs; + if((bs = calloc(1, sizeof(block_sig_t))) == NULL) { + r = RSGTE_OOM; + goto done; + } + CHKr(rsgt_tlvDecodeHASH_ALGO(rec, &strtidx, &(bs->hashID))); + CHKr(rsgt_tlvDecodeBLOCK_IV(rec, &strtidx, &(bs->iv))); + CHKr(rsgt_tlvDecodeLAST_HASH(rec, &strtidx, &(bs->lastHash))); + CHKr(rsgt_tlvDecodeREC_COUNT(rec, &strtidx, &(bs->recCount))); + CHKr(rsgt_tlvDecodeSIG(rec, &strtidx, bs)); + if(strtidx != rec->tlvlen) { + r = RSGTE_LEN; + goto done; + } + *blocksig = bs; + r = 0; +done: return r; +} +static int +rsgt_tlvRecDecode(tlvrecord_t *rec, void *obj) +{ + int r = 1; + switch(rec->tlvtype) { + case 0x0900: + case 0x0901: + r = rsgt_tlvDecodeIMPRINT(rec, obj); + if(r != 0) goto done; + break; + case 0x0902: + r = rsgt_tlvDecodeBLOCK_SIG(rec, obj); + if(r != 0) goto done; + break; + } +done: + return r; +} + +static int +rsgt_tlvrdRecHash(FILE *fp, FILE *outfp, imprint_t **imp) +{ + int r; + tlvrecord_t rec; + + if((r = rsgt_tlvrd(fp, &rec, imp)) != 0) goto done; + if(rec.tlvtype != 0x0900) { + r = RSGTE_MISS_REC_HASH; + rsgt_objfree(rec.tlvtype, *imp); + goto done; + } + if(outfp != NULL) + if((r = rsgt_tlvwrite(outfp, &rec)) != 0) goto done; + r = 0; +done: return r; +} + +static int +rsgt_tlvrdTreeHash(FILE *fp, FILE *outfp, imprint_t **imp) +{ + int r; + tlvrecord_t rec; + + if((r = rsgt_tlvrd(fp, &rec, imp)) != 0) goto done; + if(rec.tlvtype != 0x0901) { + r = RSGTE_MISS_TREE_HASH; + rsgt_objfree(rec.tlvtype, *imp); + goto done; + } + if(outfp != NULL) + if((r = rsgt_tlvwrite(outfp, &rec)) != 0) goto done; + r = 0; +done: return r; +} + +/* read BLOCK_SIG during verification phase */ +static int +rsgt_tlvrdVrfyBlockSig(FILE *fp, block_sig_t **bs, tlvrecord_t *rec) +{ + int r; + + if((r = rsgt_tlvrd(fp, rec, bs)) != 0) goto done; + if(rec->tlvtype != 0x0902) { + r = RSGTE_MISS_BLOCKSIG; + rsgt_objfree(rec->tlvtype, *bs); + goto done; + } + r = 0; +done: return r; +} + +/** + * Read the next "object" from file. This usually is + * a single TLV, but may be something larger, for + * example in case of a block-sig TLV record. + * Unknown type records are ignored (or run aborted + * if we are not permitted to skip). + * + * @param[in] fp file pointer for processing + * @param[out] tlvtype type of tlv record (top-level for + * structured objects. + * @param[out] tlvlen length of the tlv record value + * @param[out] obj pointer to object; This is a proper + * tlv record structure, which must be casted + * by the caller according to the reported type. + * The object must be freed by the caller (TODO: better way?) + * + * @returns 0 if ok, something else otherwise + */ +int +rsgt_tlvrd(FILE *fp, tlvrecord_t *rec, void *obj) +{ + int r; + if((r = rsgt_tlvRecRead(fp, rec)) != 0) goto done; + r = rsgt_tlvRecDecode(rec, obj); +done: return r; +} + + +/* return if a blob is all zero */ +static inline int +blobIsZero(uint8_t *blob, uint16_t len) +{ + int i; + for(i = 0 ; i < len ; ++i) + if(blob[i] != 0) + return 0; + return 1; +} + +static void +rsgt_printIMPRINT(FILE *fp, char *name, imprint_t *imp, uint8_t verbose) +{ + fprintf(fp, "%s", name); + outputHexBlob(fp, imp->data, imp->len, verbose); + fputc('\n', fp); +} + +static void +rsgt_printREC_HASH(FILE *fp, imprint_t *imp, uint8_t verbose) +{ + rsgt_printIMPRINT(fp, "[0x0900]Record hash: ", + imp, verbose); +} + +static void +rsgt_printINT_HASH(FILE *fp, imprint_t *imp, uint8_t verbose) +{ + rsgt_printIMPRINT(fp, "[0x0901]Tree hash..: ", + imp, verbose); +} + +/** + * Output a human-readable representation of a block_sig_t + * to proviced file pointer. This function is mainly inteded for + * debugging purposes or dumping tlv files. + * + * @param[in] fp file pointer to send output to + * @param[in] bsig ponter to block_sig_t to output + * @param[in] verbose if 0, abbreviate blob hexdump, else complete + */ +void +rsgt_printBLOCK_SIG(FILE *fp, block_sig_t *bs, uint8_t verbose) +{ + fprintf(fp, "[0x0902]Block Signature Record:\n"); + fprintf(fp, "\tPrevious Block Hash:\n"); + fprintf(fp, "\t Algorithm..: %s\n", hashAlgName(bs->lastHash.hashID)); + fprintf(fp, "\t Hash.......: "); + outputHexBlob(fp, bs->lastHash.data, bs->lastHash.len, verbose); + fputc('\n', fp); + if(blobIsZero(bs->lastHash.data, bs->lastHash.len)) + fprintf(fp, "\t NOTE: New Hash Chain Start!\n"); + fprintf(fp, "\tHash Algorithm: %s\n", hashAlgName(bs->hashID)); + fprintf(fp, "\tIV............: "); + outputHexBlob(fp, bs->iv, getIVLen(bs), verbose); + fputc('\n', fp); + fprintf(fp, "\tRecord Count..: %llu\n", bs->recCount); + fprintf(fp, "\tSignature Type: %s\n", sigTypeName(bs->sigID)); + fprintf(fp, "\tSignature Len.: %u\n", bs->sig.der.len); + fprintf(fp, "\tSignature.....: "); + outputHexBlob(fp, bs->sig.der.data, bs->sig.der.len, verbose); + fputc('\n', fp); +} + + +/** + * Output a human-readable representation of a tlv object. + * + * @param[in] fp file pointer to send output to + * @param[in] tlvtype type of tlv object (record) + * @param[in] verbose if 0, abbreviate blob hexdump, else complete + */ +void +rsgt_tlvprint(FILE *fp, uint16_t tlvtype, void *obj, uint8_t verbose) +{ + switch(tlvtype) { + case 0x0900: + rsgt_printREC_HASH(fp, obj, verbose); + break; + case 0x0901: + rsgt_printINT_HASH(fp, obj, verbose); + break; + case 0x0902: + rsgt_printBLOCK_SIG(fp, obj, verbose); + break; + default:fprintf(fp, "unknown tlv record %4.4x\n", tlvtype); + break; + } +} + +/** + * Free the provided object. + * + * @param[in] tlvtype type of tlv object (record) + * @param[in] obj the object to be destructed + */ +void +rsgt_objfree(uint16_t tlvtype, void *obj) +{ + switch(tlvtype) { + case 0x0900: + case 0x0901: + free(((imprint_t*)obj)->data); + break; + case 0x0902: + free(((block_sig_t*)obj)->iv); + free(((block_sig_t*)obj)->lastHash.data); + free(((block_sig_t*)obj)->sig.der.data); + break; + default:fprintf(stderr, "rsgt_objfree: unknown tlv record %4.4x\n", + tlvtype); + break; + } + free(obj); +} + +/** + * Read block parameters. This detects if the block contains the + * individual log hashes, the intermediate hashes and the overall + * block paramters (from the signature block). As we do not have any + * begin of block record, we do not know e.g. the hash algorithm or IV + * until reading the block signature record. And because the file is + * purely sequential and variable size, we need to read all records up to + * the next signature record. + * If a caller intends to verify a log file based on the parameters, + * he must re-read the file from the begining (we could keep things + * in memory, but this is impractical for large blocks). In order + * to facitate this, the function permits to rewind to the original + * read location when it is done. + * + * @param[in] fp file pointer of tlv file + * @param[in] bRewind 0 - do not rewind at end of procesing, 1 - do so + * @param[out] bs block signature record + * @param[out] bHasRecHashes 0 if record hashes are present, 1 otherwise + * @param[out] bHasIntermedHashes 0 if intermediate hashes are present, + * 1 otherwise + * + * @returns 0 if ok, something else otherwise + */ +int +rsgt_getBlockParams(FILE *fp, uint8_t bRewind, block_sig_t **bs, + uint8_t *bHasRecHashes, uint8_t *bHasIntermedHashes) +{ + int r; + uint64_t nRecs = 0; + uint8_t bDone = 0; + off_t rewindPos = 0; + void *obj; + tlvrecord_t rec; + + if(bRewind) + rewindPos = ftello(fp); + *bHasRecHashes = 0; + *bHasIntermedHashes = 0; + *bs = NULL; + + while(!bDone) { /* we will err out on EOF */ + if((r = rsgt_tlvrd(fp, &rec, &obj)) != 0) goto done; + switch(rec.tlvtype) { + case 0x0900: + ++nRecs; + *bHasRecHashes = 1; + break; + case 0x0901: + *bHasIntermedHashes = 1; + break; + case 0x0902: + *bs = (block_sig_t*) obj; + bDone = 1; + break; + default:fprintf(fp, "unknown tlv record %4.4x\n", rec.tlvtype); + break; + } + if(!bDone) + rsgt_objfree(rec.tlvtype, obj); + } + + if(*bHasRecHashes && (nRecs != (*bs)->recCount)) { + r = RSGTE_INVLD_RECCNT; + goto done; + } + + if(bRewind) { + if(fseeko(fp, rewindPos, SEEK_SET) != 0) { + r = RSGTE_IO; + goto done; + } + } +done: + return r; +} + + +/** + * Read the file header and compare it to the expected value. + * The file pointer is placed right after the header. + * @param[in] fp file pointer of tlv file + * @param[in] excpect expected header (e.g. "LOGSIG10") + * @returns 0 if ok, something else otherwise + */ +int +rsgt_chkFileHdr(FILE *fp, char *expect) +{ + int r; + char hdr[9]; + + if((r = rsgt_tlvrdHeader(fp, (uchar*)hdr)) != 0) goto done; + if(strcmp(hdr, expect)) + r = RSGTE_INVLHDR; + else + r = 0; +done: + return r; +} + +gtfile +rsgt_vrfyConstruct_gf(void) +{ + gtfile gf; + if((gf = calloc(1, sizeof(struct gtfile_s))) == NULL) + goto done; + gf->x_prev = NULL; + +done: return gf; +} + +void +rsgt_vrfyBlkInit(gtfile gf, block_sig_t *bs, uint8_t bHasRecHashes, uint8_t bHasIntermedHashes) +{ + gf->hashAlg = hashID2Alg(bs->hashID); + gf->bKeepRecordHashes = bHasRecHashes; + gf->bKeepTreeHashes = bHasIntermedHashes; + free(gf->IV); + gf->IV = malloc(getIVLen(bs)); + memcpy(gf->IV, bs->iv, getIVLen(bs)); + free(gf->blkStrtHash); + gf->lenBlkStrtHash = bs->lastHash.len; + gf->blkStrtHash = malloc(gf->lenBlkStrtHash); + memcpy(gf->blkStrtHash, bs->lastHash.data, gf->lenBlkStrtHash); +} + +static int +rsgt_vrfy_chkRecHash(gtfile gf, FILE *sigfp, FILE *nsigfp, + GTDataHash *recHash, gterrctx_t *ectx) +{ + int r = 0; + imprint_t *imp = NULL; + + if((r = rsgt_tlvrdRecHash(sigfp, nsigfp, &imp)) != 0) + reportError(r, ectx); + goto done; + if(imp->hashID != hashIdentifier(gf->hashAlg)) { + reportError(r, ectx); + r = RSGTE_INVLD_REC_HASHID; + goto done; + } + if(memcmp(imp->data, recHash->digest, + hashOutputLengthOctets(imp->hashID))) { + r = RSGTE_INVLD_REC_HASH; + ectx->computedHash = recHash; + ectx->fileHash = imp; + reportError(r, ectx); + ectx->computedHash = NULL, ectx->fileHash = NULL; + goto done; + } + r = 0; +done: + if(imp != NULL) + rsgt_objfree(0x0900, imp); + return r; +} + +static int +rsgt_vrfy_chkTreeHash(gtfile gf, FILE *sigfp, FILE *nsigfp, + GTDataHash *hash, gterrctx_t *ectx) +{ + int r = 0; + imprint_t *imp = NULL; + + if((r = rsgt_tlvrdTreeHash(sigfp, nsigfp, &imp)) != 0) { + reportError(r, ectx); + goto done; + } + if(imp->hashID != hashIdentifier(gf->hashAlg)) { + reportError(r, ectx); + r = RSGTE_INVLD_TREE_HASHID; + goto done; + } + if(memcmp(imp->data, hash->digest, + hashOutputLengthOctets(imp->hashID))) { + r = RSGTE_INVLD_TREE_HASH; + ectx->computedHash = hash; + ectx->fileHash = imp; + reportError(r, ectx); + ectx->computedHash = NULL, ectx->fileHash = NULL; + goto done; + } + r = 0; +done: + if(imp != NULL) + rsgt_objfree(0x0901, imp); + return r; +} + +int +rsgt_vrfy_nextRec(block_sig_t *bs, gtfile gf, FILE *sigfp, FILE *nsigfp, + unsigned char *rec, size_t len, gterrctx_t *ectx) +{ + int r = 0; + GTDataHash *x; /* current hash */ + GTDataHash *m, *recHash = NULL, *t, *t_del; + uint8_t j; + + hash_m(gf, &m); + hash_r(gf, &recHash, rec, len); + if(gf->bKeepRecordHashes) { + r = rsgt_vrfy_chkRecHash(gf, sigfp, nsigfp, recHash, ectx); + if(r != 0) goto done; + } + hash_node(gf, &x, m, recHash, 1); /* hash leaf */ + if(gf->bKeepTreeHashes) { + ectx->treeLevel = 0; + ectx->lefthash = m; + ectx->righthash = recHash; + r = rsgt_vrfy_chkTreeHash(gf, sigfp, nsigfp, x, ectx); + if(r != 0) goto done; + } + rsgtimprintDel(gf->x_prev); + gf->x_prev = rsgtImprintFromGTDataHash(x); + /* add x to the forest as new leaf, update roots list */ + t = x; + for(j = 0 ; j < gf->nRoots ; ++j) { + if(gf->roots_valid[j] == 0) { + gf->roots_hash[j] = t; + gf->roots_valid[j] = 1; + t = NULL; + break; + } else if(t != NULL) { + /* hash interim node */ + ectx->treeLevel = j+1; + ectx->righthash = t; + t_del = t; + hash_node(gf, &t, gf->roots_hash[j], t_del, j+2); + gf->roots_valid[j] = 0; + if(gf->bKeepTreeHashes) { + ectx->lefthash = gf->roots_hash[j]; + r = rsgt_vrfy_chkTreeHash(gf, sigfp, nsigfp, t, ectx); + if(r != 0) goto done; /* mem leak ok, we terminate! */ + } + GTDataHash_free(gf->roots_hash[j]); + GTDataHash_free(t_del); + } + } + if(t != NULL) { + /* new level, append "at the top" */ + gf->roots_hash[gf->nRoots] = t; + gf->roots_valid[gf->nRoots] = 1; + ++gf->nRoots; + assert(gf->nRoots < MAX_ROOTS); + t = NULL; + } + ++gf->nRecords; + + /* cleanup */ + GTDataHash_free(m); +done: + if(recHash != NULL) + GTDataHash_free(recHash); + return r; +} + + +/* TODO: think about merging this with the writer. The + * same applies to the other computation algos. + */ +static int +verifySigblkFinish(gtfile gf, GTDataHash **pRoot) +{ + GTDataHash *root, *rootDel; + int8_t j; + int r; + + if(gf->nRecords == 0) + goto done; + + root = NULL; + for(j = 0 ; j < gf->nRoots ; ++j) { + if(root == NULL) { + root = gf->roots_valid[j] ? gf->roots_hash[j] : NULL; + gf->roots_valid[j] = 0; /* guess this is redundant with init, maybe del */ + } else if(gf->roots_valid[j]) { + rootDel = root; + hash_node(gf, &root, gf->roots_hash[j], root, j+2); + gf->roots_valid[j] = 0; /* guess this is redundant with init, maybe del */ + GTDataHash_free(rootDel); + } + } + + free(gf->blkStrtHash); + gf->blkStrtHash = NULL; + *pRoot = root; + r = 0; +done: + gf->bInBlk = 0; + return r; +} + + +/* helper for rsgt_extendSig: */ +#define COPY_SUBREC_TO_NEWREC \ + memcpy(newrec.data+iWr, subrec.hdr, subrec.lenHdr); \ + iWr += subrec.lenHdr; \ + memcpy(newrec.data+iWr, subrec.data, subrec.tlvlen); \ + iWr += subrec.tlvlen; +static inline int +rsgt_extendSig(GTTimestamp *timestamp, tlvrecord_t *rec, gterrctx_t *ectx) +{ + GTTimestamp *out_timestamp; + uint8_t *der; + size_t lenDer; + int r, rgt; + tlvrecord_t newrec, subrec; + uint16_t iRd, iWr; + + rgt = GTHTTP_extendTimestamp(timestamp, rsgt_extend_puburl, &out_timestamp); + if(rgt != GT_OK) { + ectx->gtstate = rgt; + r = RSGTE_TS_EXTEND; + goto done; + } + r = GTTimestamp_getDEREncoded(out_timestamp, &der, &lenDer); + if(r != GT_OK) { + r = RSGTE_TS_DERENCODE; + ectx->gtstate = rgt; + goto done; + } + /* update block_sig tlv record with new extended timestamp */ + /* we now need to copy all tlv records before the actual der + * encoded part. + */ + iRd = iWr = 0; + // TODO; check tlvtypes at comment places below! + if ((r = rsgt_tlvDecodeSUBREC(rec, &iRd, &subrec)) != 0) goto done; + /* HASH_ALGO */ + COPY_SUBREC_TO_NEWREC + if ((r = rsgt_tlvDecodeSUBREC(rec, &iRd, &subrec)) != 0) goto done; + /* BLOCK_IV */ + COPY_SUBREC_TO_NEWREC + if ((r = rsgt_tlvDecodeSUBREC(rec, &iRd, &subrec)) != 0) goto done; + /* LAST_HASH */ + COPY_SUBREC_TO_NEWREC + if ((r = rsgt_tlvDecodeSUBREC(rec, &iRd, &subrec)) != 0) goto done; + /* REC_COUNT */ + COPY_SUBREC_TO_NEWREC + if ((r = rsgt_tlvDecodeSUBREC(rec, &iRd, &subrec)) != 0) goto done; + /* actual sig! */ + newrec.data[iWr++] = 0x09 | RSGT_FLAG_TLV16; + newrec.data[iWr++] = 0x06; + newrec.data[iWr++] = (lenDer >> 8) & 0xff; + newrec.data[iWr++] = lenDer & 0xff; + /* now we know how large the new main record is */ + newrec.tlvlen = (uint16_t) iWr+lenDer; + newrec.tlvtype = rec->tlvtype; + newrec.hdr[0] = rec->hdr[0]; + newrec.hdr[1] = rec->hdr[1]; + newrec.hdr[2] = (newrec.tlvlen >> 8) & 0xff; + newrec.hdr[3] = newrec.tlvlen & 0xff; + newrec.lenHdr = 4; + memcpy(newrec.data+iWr, der, lenDer); + /* and finally copy back new record to existing one */ + memcpy(rec, &newrec, sizeof(newrec)-sizeof(newrec.data)+newrec.tlvlen+4); + r = 0; +done: + return r; +} + + +/* verify the root hash. This also means we need to compute the + * Merkle tree root for the current block. + */ +int +verifyBLOCK_SIG(block_sig_t *bs, gtfile gf, FILE *sigfp, FILE *nsigfp, + uint8_t bExtend, gterrctx_t *ectx) +{ + int r; + int gtstate; + block_sig_t *file_bs = NULL; + GTTimestamp *timestamp = NULL; + GTVerificationInfo *vrfyInf; + GTDataHash *root = NULL; + tlvrecord_t rec; + + if((r = verifySigblkFinish(gf, &root)) != 0) + goto done; + if((r = rsgt_tlvrdVrfyBlockSig(sigfp, &file_bs, &rec)) != 0) + goto done; + if(ectx->recNum != bs->recCount) { + r = RSGTE_INVLD_RECCNT; + goto done; + } + + gtstate = GTTimestamp_DERDecode(file_bs->sig.der.data, + file_bs->sig.der.len, ×tamp); + if(gtstate != GT_OK) { + r = RSGTE_TS_DERDECODE; + ectx->gtstate = gtstate; + goto done; + } + + gtstate = GTHTTP_verifyTimestampHash(timestamp, root, NULL, + NULL, NULL, rsgt_read_puburl, 0, &vrfyInf); + if(! (gtstate == GT_OK + && vrfyInf->verification_errors == GT_NO_FAILURES) ) { + r = RSGTE_INVLD_TIMESTAMP; + ectx->gtstate = gtstate; + goto done; + } + + if(rsgt_read_showVerified) + reportVerifySuccess(ectx, vrfyInf); + if(bExtend) + if((r = rsgt_extendSig(timestamp, &rec, ectx)) != 0) goto done; + + if(nsigfp != NULL) + if((r = rsgt_tlvwrite(nsigfp, &rec)) != 0) goto done; + r = 0; +done: + if(file_bs != NULL) + rsgt_objfree(0x0902, file_bs); + if(r != 0) + reportError(r, ectx); + if(timestamp != NULL) + GTTimestamp_free(timestamp); + return r; +} diff --git a/runtime/lmcry_gcry.c b/runtime/lmcry_gcry.c new file mode 100644 index 00000000..0a9b94bc --- /dev/null +++ b/runtime/lmcry_gcry.c @@ -0,0 +1,285 @@ +/* lmcry_gcry.c + * + * An implementation of the cryprov interface for libgcrypt. + * + * Copyright 2013 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" + +#include "rsyslog.h" +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "module-template.h" +#include "glbl.h" +#include "errmsg.h" +#include "cryprov.h" +#include "libgcry.h" +#include "lmcry_gcry.h" + +MODULE_TYPE_LIB +MODULE_TYPE_NOKEEP + +/* static data */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) + +/* tables for interfacing with the v6 config system */ +static struct cnfparamdescr cnfpdescr[] = { + { "cry.key", eCmdHdlrGetWord, 0 }, + { "cry.keyfile", eCmdHdlrGetWord, 0 }, + { "cry.keyprogram", eCmdHdlrGetWord, 0 }, + { "cry.mode", eCmdHdlrGetWord, 0 }, /* CBC, ECB, etc */ + { "cry.algo", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk pblk = + { CNFPARAMBLK_VERSION, + sizeof(cnfpdescr)/sizeof(struct cnfparamdescr), + cnfpdescr + }; + + +#if 0 +static void +errfunc(__attribute__((unused)) void *usrptr, uchar *emsg) +{ + errmsg.LogError(0, RS_RET_CRYPROV_ERR, "Crypto Provider" + "Error: %s - disabling encryption", emsg); +} +#endif + +/* Standard-Constructor + */ +BEGINobjConstruct(lmcry_gcry) + pThis->ctx = gcryCtxNew(); +ENDobjConstruct(lmcry_gcry) + + +/* destructor for the lmcry_gcry object */ +BEGINobjDestruct(lmcry_gcry) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(lmcry_gcry) + rsgcryCtxDel(pThis->ctx); +ENDobjDestruct(lmcry_gcry) + + +/* apply all params from param block to us. This must be called + * after construction, but before the OnFileOpen() entry point. + * Defaults are expected to have been set during construction. + */ +static rsRetVal +SetCnfParam(void *pT, struct nvlst *lst) +{ + lmcry_gcry_t *pThis = (lmcry_gcry_t*) pT; + int i, r; + unsigned keylen; + uchar *key = NULL; + uchar *keyfile = NULL; + uchar *keyprogram = NULL; + uchar *algo = NULL; + uchar *mode = NULL; + int nKeys; /* number of keys (actually methods) specified */ + struct cnfparamvals *pvals; + DEFiRet; + + nKeys = 0; + pvals = nvlstGetParams(lst, &pblk, NULL); + if(Debug) { + dbgprintf("param blk in lmcry_gcry:\n"); + cnfparamsPrint(&pblk, pvals); + } + + for(i = 0 ; i < pblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(pblk.descr[i].name, "cry.key")) { + key = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + ++nKeys; + } else if(!strcmp(pblk.descr[i].name, "cry.keyfile")) { + keyfile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + ++nKeys; + } else if(!strcmp(pblk.descr[i].name, "cry.keyprogram")) { + keyprogram = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + ++nKeys; + } else if(!strcmp(pblk.descr[i].name, "cry.mode")) { + mode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(pblk.descr[i].name, "cry.algo")) { + algo = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + DBGPRINTF("lmcry_gcry: program error, non-handled " + "param '%s'\n", pblk.descr[i].name); + } + } + if(algo != NULL) { + iRet = rsgcrySetAlgo(pThis->ctx, algo); + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "cry.algo '%s' is not know/supported", algo); + FINALIZE; + } + } + if(mode != NULL) { + iRet = rsgcrySetMode(pThis->ctx, mode); + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "cry.mode '%s' is not know/supported", mode); + FINALIZE; + } + } + /* note: key must be set AFTER algo/mode is set (as it depends on them) */ + if(nKeys != 1) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "excactly one of the following " + "parameters can be specified: cry.key, cry.keyfile, cry.keyprogram\n"); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + if(key != NULL) { + errmsg.LogError(0, RS_RET_ERR, "Note: specifying an actual key directly from the " + "config file is highly insecure - DO NOT USE FOR PRODUCTION"); + keylen = strlen((char*)key); + } + if(keyfile != NULL) { + r = gcryGetKeyFromFile((char*)keyfile, (char**)&key, &keylen); + if(r != 0) { + errmsg.LogError(0, RS_RET_ERR, "error %d reading keyfile %s\n", + r, keyfile); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + } + if(keyprogram != NULL) { + r = gcryGetKeyFromProg((char*)keyprogram, (char**)&key, &keylen); + if(r != 0) { + errmsg.LogError(0, RS_RET_ERR, "error %d obtaining key from program %s\n", + r, keyprogram); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + } + + /* if we reach this point, we have a valid key */ + r = rsgcrySetKey(pThis->ctx, key, keylen); + if(r > 0) { + errmsg.LogError(0, RS_RET_INVALID_PARAMS, "Key length %d expected, but " + "key of length %d given", r, keylen); + ABORT_FINALIZE(RS_RET_INVALID_PARAMS); + } + + cnfparamvalsDestruct(pvals, &pblk); + if(key != NULL) { + memset(key, 0, strlen((char*)key)); + free(key); + } + free(keyfile); + free(algo); + free(mode); +finalize_it: + RETiRet; +} + + +static rsRetVal +OnFileOpen(void *pT, uchar *fn, void *pGF) +{ + lmcry_gcry_t *pThis = (lmcry_gcry_t*) pT; + gcryfile *pgf = (gcryfile*) pGF; + DEFiRet; + + CHKiRet(rsgcryInitCrypt(pThis->ctx, pgf, fn)); +finalize_it: + /* TODO: enable this error message (need to cleanup loop first ;)) + errmsg.LogError(0, iRet, "Encryption Provider" + "Error: cannot open .encinfo file - disabling log file"); + */ + RETiRet; +} + +static rsRetVal +Encrypt(void *pF, uchar *rec, size_t *lenRec) +{ + DEFiRet; + iRet = rsgcryEncrypt(pF, rec, lenRec); + + RETiRet; +} + +static rsRetVal +OnFileClose(void *pF, off64_t offsLogfile) +{ + DEFiRet; + gcryfileDestruct(pF, offsLogfile); + + RETiRet; +} + +BEGINobjQueryInterface(lmcry_gcry) +CODESTARTobjQueryInterface(lmcry_gcry) + if(pIf->ifVersion != cryprovCURR_IF_VERSION) {/* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + pIf->Construct = (rsRetVal(*)(void*)) lmcry_gcryConstruct; + pIf->SetCnfParam = SetCnfParam; + pIf->Destruct = (rsRetVal(*)(void*)) lmcry_gcryDestruct; + pIf->OnFileOpen = OnFileOpen; + pIf->Encrypt = Encrypt; + pIf->OnFileClose = OnFileClose; +finalize_it: +ENDobjQueryInterface(lmcry_gcry) + + +BEGINObjClassExit(lmcry_gcry, OBJ_IS_LOADABLE_MODULE) /* CHANGE class also in END MACRO! */ +CODESTARTObjClassExit(lmcry_gcry) + /* release objects we no longer need */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + + rsgcryExit(); +ENDObjClassExit(lmcry_gcry) + + +BEGINObjClassInit(lmcry_gcry, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ + /* request objects we use */ + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + + if(rsgcryInit() != 0) { + errmsg.LogError(0, RS_RET_CRYPROV_ERR, "error initializing " + "crypto provider - cannot encrypt"); + ABORT_FINALIZE(RS_RET_CRYPROV_ERR); + } +ENDObjClassInit(lmcry_gcry) + + +/* --------------- here now comes the plumbing that makes as a library module --------------- */ + + +BEGINmodExit +CODESTARTmodExit + lmcry_gcryClassExit(); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_LIB_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ + /* Initialize all classes that are in our module - this includes ourselfs */ + CHKiRet(lmcry_gcryClassInit(pModInfo)); /* must be done after tcps_sess, as we use it */ +ENDmodInit diff --git a/runtime/lmcry_gcry.h b/runtime/lmcry_gcry.h new file mode 100644 index 00000000..c0205ab9 --- /dev/null +++ b/runtime/lmcry_gcry.h @@ -0,0 +1,39 @@ +/* An implementation of the cryprov interface for libgcrypt. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_LMCRY_GCRY_H +#define INCLUDED_LMCRY_GCRY_H +#include "cryprov.h" + +/* interface is defined in cryprov.h, we just implement it! */ +#define lmcry_gcryCURR_IF_VERSION cryprovCURR_IF_VERSION +typedef cryprov_if_t lmcry_gcry_if_t; + +/* the lmcry_gcry object */ +struct lmcry_gcry_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + gcryctx ctx; +}; +typedef struct lmcry_gcry_s lmcry_gcry_t; + +/* prototypes */ +PROTOTYPEObj(lmcry_gcry); + +#endif /* #ifndef INCLUDED_LMCRY_GCRY_H */ diff --git a/runtime/lmsig_gt.c b/runtime/lmsig_gt.c new file mode 100644 index 00000000..116a48d5 --- /dev/null +++ b/runtime/lmsig_gt.c @@ -0,0 +1,233 @@ +/* lmsig_gt.c + * + * An implementation of the sigprov interface for GuardTime. + * + * Copyright 2013 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" + +#include "rsyslog.h" +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "module-template.h" +#include "glbl.h" +#include "errmsg.h" +#include "sigprov.h" +#include "lmsig_gt.h" + +MODULE_TYPE_LIB +MODULE_TYPE_NOKEEP + +/* static data */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) + +/* tables for interfacing with the v6 config system */ +static struct cnfparamdescr cnfpdescr[] = { + { "sig.hashfunction", eCmdHdlrGetWord, 0 }, + { "sig.timestampservice", eCmdHdlrGetWord, 0 }, + { "sig.block.sizelimit", eCmdHdlrSize, 0 }, + { "sig.keeprecordhashes", eCmdHdlrBinary, 0 }, + { "sig.keeptreehashes", eCmdHdlrBinary, 0 } +}; +static struct cnfparamblk pblk = + { CNFPARAMBLK_VERSION, + sizeof(cnfpdescr)/sizeof(struct cnfparamdescr), + cnfpdescr + }; + + +static void +errfunc(__attribute__((unused)) void *usrptr, uchar *emsg) +{ + errmsg.LogError(0, RS_RET_SIGPROV_ERR, "Signature Provider" + "Error: %s - disabling signatures", emsg); +} + +/* Standard-Constructor + */ +BEGINobjConstruct(lmsig_gt) + pThis->ctx = rsgtCtxNew(); + rsgtsetErrFunc(pThis->ctx, errfunc, NULL); +ENDobjConstruct(lmsig_gt) + + +/* destructor for the lmsig_gt object */ +BEGINobjDestruct(lmsig_gt) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(lmsig_gt) + rsgtCtxDel(pThis->ctx); +ENDobjDestruct(lmsig_gt) + + +/* apply all params from param block to us. This must be called + * after construction, but before the OnFileOpen() entry point. + * Defaults are expected to have been set during construction. + */ +rsRetVal +SetCnfParam(void *pT, struct nvlst *lst) +{ + lmsig_gt_t *pThis = (lmsig_gt_t*) pT; + int i; + uchar *cstr; + struct cnfparamvals *pvals; + DEFiRet; + pvals = nvlstGetParams(lst, &pblk, NULL); + if(pvals == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + if(Debug) { + dbgprintf("sig param blk in lmsig_gt:\n"); + cnfparamsPrint(&pblk, pvals); + } + + for(i = 0 ; i < pblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(pblk.descr[i].name, "sig.hashfunction")) { + cstr = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + if(rsgtSetHashFunction(pThis->ctx, (char*)cstr) != 0) { + errmsg.LogError(0, RS_RET_ERR, "Hash function " + "'%s' unknown - using default", cstr); + } + free(cstr); + } else if(!strcmp(pblk.descr[i].name, "sig.timestampservice")) { + cstr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + rsgtSetTimestamper(pThis->ctx, (char*) cstr); + free(cstr); + } else if(!strcmp(pblk.descr[i].name, "sig.block.sizelimit")) { + rsgtSetBlockSizeLimit(pThis->ctx, pvals[i].val.d.n); + } else if(!strcmp(pblk.descr[i].name, "sig.keeprecordhashes")) { + rsgtSetKeepRecordHashes(pThis->ctx, pvals[i].val.d.n); + } else if(!strcmp(pblk.descr[i].name, "sig.keeptreehashes")) { + rsgtSetKeepTreeHashes(pThis->ctx, pvals[i].val.d.n); + } else { + DBGPRINTF("lmsig_gt: program error, non-handled " + "param '%s'\n", pblk.descr[i].name); + } + } +finalize_it: + if(pvals != NULL) + cnfparamvalsDestruct(pvals, &pblk); + RETiRet; +} + + +static rsRetVal +OnFileOpen(void *pT, uchar *fn, void *pGF) +{ + lmsig_gt_t *pThis = (lmsig_gt_t*) pT; + gtfile *pgf = (gtfile*) pGF; + DEFiRet; + DBGPRINTF("lmsig_gt: onFileOpen: %s\n", fn); + /* note: if *pgf is set to NULL, this auto-disables GT functions */ + *pgf = rsgtCtxOpenFile(pThis->ctx, fn); + sigblkInit(*pgf); + RETiRet; +} + +/* Note: we assume that the record is terminated by a \n. + * As of the GuardTime paper, \n is not part of the signed + * message, so we subtract one from the record size. This + * may cause issues with non-standard formats, but let's + * see how things evolve (the verifier will not work in + * any case when the records are not \n delimited...). + * rgerhards, 2013-03-17 + */ +static rsRetVal +OnRecordWrite(void *pF, uchar *rec, rs_size_t lenRec) +{ + DEFiRet; + DBGPRINTF("lmsig_gt: onRecordWrite (%d): %s\n", lenRec-1, rec); + sigblkAddRecord(pF, rec, lenRec-1); + + RETiRet; +} + +static rsRetVal +OnFileClose(void *pF) +{ + DEFiRet; + DBGPRINTF("lmsig_gt: onFileClose\n"); + rsgtfileDestruct(pF); + + RETiRet; +} + +BEGINobjQueryInterface(lmsig_gt) +CODESTARTobjQueryInterface(lmsig_gt) + if(pIf->ifVersion != sigprovCURR_IF_VERSION) {/* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + pIf->Construct = (rsRetVal(*)(void*)) lmsig_gtConstruct; + pIf->SetCnfParam = SetCnfParam; + pIf->Destruct = (rsRetVal(*)(void*)) lmsig_gtDestruct; + pIf->OnFileOpen = OnFileOpen; + pIf->OnRecordWrite = OnRecordWrite; + pIf->OnFileClose = OnFileClose; +finalize_it: +ENDobjQueryInterface(lmsig_gt) + + +BEGINObjClassExit(lmsig_gt, OBJ_IS_LOADABLE_MODULE) /* CHANGE class also in END MACRO! */ +CODESTARTObjClassExit(lmsig_gt) + /* release objects we no longer need */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + + rsgtExit(); +ENDObjClassExit(lmsig_gt) + + +BEGINObjClassInit(lmsig_gt, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ + /* request objects we use */ + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + + if(rsgtInit("rsyslogd " VERSION) != 0) { + errmsg.LogError(0, RS_RET_SIGPROV_ERR, "error initializing " + "signature provider - cannot sign"); + ABORT_FINALIZE(RS_RET_SIGPROV_ERR); + } +ENDObjClassInit(lmsig_gt) + + +/* --------------- here now comes the plumbing that makes as a library module --------------- */ + + +BEGINmodExit +CODESTARTmodExit + lmsig_gtClassExit(); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_LIB_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ + /* Initialize all classes that are in our module - this includes ourselfs */ + CHKiRet(lmsig_gtClassInit(pModInfo)); /* must be done after tcps_sess, as we use it */ +ENDmodInit diff --git a/runtime/lmsig_gt.h b/runtime/lmsig_gt.h new file mode 100644 index 00000000..665e6a8e --- /dev/null +++ b/runtime/lmsig_gt.h @@ -0,0 +1,40 @@ +/* An implementation of the sigprov interface for GuardTime. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_LMSIG_GT_H +#define INCLUDED_LMSIG_GT_H +#include "sigprov.h" +#include "librsgt.h" + +/* interface is defined in sigprov.h, we just implement it! */ +#define lmsig_gtCURR_IF_VERSION sigprovCURR_IF_VERSION +typedef sigprov_if_t lmsig_gt_if_t; + +/* the lmsig_gt object */ +struct lmsig_gt_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + gtctx ctx; /* librsgt context - contains all we need */ +}; +typedef struct lmsig_gt_s lmsig_gt_t; + +/* prototypes */ +PROTOTYPEObj(lmsig_gt); + +#endif /* #ifndef INCLUDED_LMSIG_GT_H */ diff --git a/runtime/module-template.h b/runtime/module-template.h index 9dd759a5..8a958f90 100644 --- a/runtime/module-template.h +++ b/runtime/module-template.h @@ -113,7 +113,7 @@ static rsRetVal modGetID(void **pID) \ /* macro to provide the v6 config system module name */ #define MODULE_CNFNAME(name) \ -static __attribute__((unused)) rsRetVal modGetCnfName(uchar **cnfName) \ +static rsRetVal modGetCnfName(uchar **cnfName) \ { \ *cnfName = (uchar*) name; \ return RS_RET_OK;\ @@ -246,7 +246,8 @@ static rsRetVal dbgPrintInstInfo(void *pModData)\ instanceData *pData = NULL; #define CODESTARTdbgPrintInstInfo \ - pData = (instanceData*) pModData; + pData = (instanceData*) pModData; \ + (void)pData; /* prevent compiler warning if unused! */ #define ENDdbgPrintInstInfo \ RETiRet;\ @@ -937,6 +938,28 @@ static rsRetVal doHUP(instanceData __attribute__((unused)) *pData)\ } +/* SetShutdownImmdtPtr() + * This function is optional. If defined by an output plugin, it is called + * each time the action is invoked to set the "ShutdownImmediate" pointer, + * which is used during termination to indicate the action should shutdown + * as quickly as possible. + */ +#define CODEqueryEtryPt_SetShutdownImmdtPtr \ + else if(!strcmp((char*) name, "SetShutdownImmdtPtr")) {\ + *pEtryPoint = SetShutdownImmdtPtr;\ + } +#define BEGINSetShutdownImmdtPtr \ +static rsRetVal SetShutdownImmdtPtr(instanceData __attribute__((unused)) *pData, int *pPtr)\ +{\ + DEFiRet; + +#define CODESTARTSetShutdownImmdtPtr + +#define ENDSetShutdownImmdtPtr \ + RETiRet;\ +} + + /* parse() - main entry point of parser modules */ #define BEGINparse \ diff --git a/runtime/modules.c b/runtime/modules.c index 5706685f..56606306 100644 --- a/runtime/modules.c +++ b/runtime/modules.c @@ -313,7 +313,8 @@ finalize_it: /* get the name of a module */ -static uchar *modGetName(modInfo_t *pThis) +uchar * +modGetName(modInfo_t *pThis) { return((pThis->pszName == NULL) ? (uchar*) "" : pThis->pszName); } @@ -656,6 +657,10 @@ doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_ if(localRet != RS_RET_OK && localRet != RS_RET_MODULE_ENTRY_POINT_NOT_FOUND) ABORT_FINALIZE(localRet); + localRet = (*pNew->modQueryEtryPt)((uchar*)"SetShutdownImmdtPtr", &pNew->mod.om.SetShutdownImmdtPtr); + if(localRet != RS_RET_OK && localRet != RS_RET_MODULE_ENTRY_POINT_NOT_FOUND) + ABORT_FINALIZE(localRet); + localRet = (*pNew->modQueryEtryPt)((uchar*)"beginTransaction", &pNew->mod.om.beginTransaction); if(localRet == RS_RET_MODULE_ENTRY_POINT_NOT_FOUND) pNew->mod.om.beginTransaction = dummyBeginTransaction; @@ -1040,7 +1045,6 @@ Load(uchar *pModName, sbool bConfLoad, struct nvlst *lst) if(bConfLoad) { localRet = readyModForCnf(pModInfo, &pNew, &pLast); if(pModInfo->setModCnf != NULL && localRet == RS_RET_OK) { - addModToCnfList(pNew, pLast); if(!strncmp((char*)pModName, "builtin:", sizeof("builtin:")-1)) { if(pModInfo->bSetModCnfCalled) { errmsg.LogError(0, RS_RET_DUP_PARAM, @@ -1056,6 +1060,11 @@ Load(uchar *pModName, sbool bConfLoad, struct nvlst *lst) pModInfo->setModCnf(lst); pModInfo->bSetModCnfCalled = 1; } + } else { + /* regular modules need to be added to conf list (for + * builtins, this happend during initial load). + */ + addModToCnfList(pNew, pLast); } } } diff --git a/runtime/modules.h b/runtime/modules.h index 02e4a699..64644be2 100644 --- a/runtime/modules.h +++ b/runtime/modules.h @@ -142,6 +142,7 @@ struct modInfo_s { rsRetVal (*endTransaction)(void*); rsRetVal (*parseSelectorAct)(uchar**, void**,omodStringRequest_t**); rsRetVal (*newActInst)(uchar *modName, struct nvlst *lst, void **, omodStringRequest_t **); + rsRetVal (*SetShutdownImmdtPtr)(void *pData, void *pPtr); } om; struct { /* data for library modules */ char dummy; @@ -190,8 +191,11 @@ ENDinterface(module) PROTOTYPEObj(module); /* in v6, we go back to in-core static link for core objects, at least those * that are not called from plugins. + * ... and we need to know that none of the module functions are called from plugins! + * rgerhards, 2012-09-24 */ rsRetVal modulesProcessCnf(struct cnfobj *o); +uchar *modGetName(modInfo_t *pThis); rsRetVal addModToCnfList(cfgmodules_etry_t *pNew, cfgmodules_etry_t *pLast); rsRetVal readyModForCnf(modInfo_t *pThis, cfgmodules_etry_t **ppNew, cfgmodules_etry_t **ppLast); #endif /* #ifndef MODULES_H_INCLUDED */ diff --git a/runtime/msg.c b/runtime/msg.c index 45ebf5f9..67d957d1 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -7,7 +7,7 @@ * of the "old" message code without any modifications. However, it * helps to have things at the right place one we go to the meat of it. * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -36,14 +36,20 @@ #include <assert.h> #include <ctype.h> #include <sys/socket.h> +#if HAVE_SYSINFO_UPTIME #include <sys/sysinfo.h> +#endif #include <netdb.h> #include <libestr.h> -#include <libee/libee.h> +#include <json/json.h> +/* For struct json_object_iter, should not be necessary in future versions */ +#include <json/json_object_private.h> #if HAVE_MALLOC_H # include <malloc.h> #endif -#include <uuid/uuid.h> +#ifdef USE_LIBUUID + #include <uuid/uuid.h> +#endif #include "rsyslog.h" #include "srUtils.h" #include "stringbuf.h" @@ -57,6 +63,7 @@ #include "ruleset.h" #include "prop.h" #include "net.h" +#include "var.h" #include "rsconf.h" /* static data */ @@ -66,6 +73,19 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(regexp) DEFobjCurrIf(prop) DEFobjCurrIf(net) +DEFobjCurrIf(var) + +static char *two_digits[100] = { + "00", "01", "02", "03", "04", "05", "06", "07", "08", "09", + "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", + "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", + "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", + "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", + "50", "51", "52", "53", "54", "55", "56", "57", "58", "59", + "60", "61", "62", "63", "64", "65", "66", "67", "68", "69", + "70", "71", "72", "73", "74", "75", "76", "77", "78", "79", + "80", "81", "82", "83", "84", "85", "86", "87", "88", "89", + "90", "91", "92", "93", "94", "95", "96", "97", "98", "99"}; static struct { uchar *pszName; @@ -273,9 +293,15 @@ static char *syslog_fac_names[24] = { "kern", "user", "mail", "daemon", "auth", "news", "uucp", "cron", "authpriv", "ftp", "ntp", "audit", "alert", "clock", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7" }; +/* length of the facility names string (for optimizatiions) */ +static short len_syslog_fac_names[24] = { 4, 4, 4, 6, 4, 6, 3, + 4, 4, 4, 8, 3, 3, 5, + 5, 5, 6, 6, 6, 6, + 6, 6, 6, 6 }; /* table of severity names (in numerical order)*/ static char *syslog_severity_names[8] = { "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug" }; +static short len_syslog_severity_names[8] = { 5, 5, 4, 3, 7, 6, 4, 5 }; /* numerical values as string - this is the most efficient approach to convert severity * and facility values to a numerical string... -- rgerhars, 2009-06-17 @@ -291,6 +317,9 @@ static pthread_mutex_t mutTrimCtr; /* mutex to handle malloc trim */ /* some forward declarations */ static int getAPPNAMELen(msg_t *pM, sbool bLockMutex); +static rsRetVal jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate); +static uchar * jsonPathGetLeaf(uchar *name, int lenName); +static struct json_object *jsonDeepCopy(struct json_object *src); /* the locking and unlocking implementations: */ @@ -308,6 +337,47 @@ MsgUnlock(msg_t *pThis) } +/* set RcvFromIP name in msg object WITHOUT calling AddRef. + * rgerhards, 2013-01-22 + */ +static inline void +MsgSetRcvFromIPWithoutAddRef(msg_t *pThis, prop_t *new) +{ + if(pThis->pRcvFromIP != NULL) + prop.Destruct(&pThis->pRcvFromIP); + pThis->pRcvFromIP = new; +} + + +/* set RcvFrom name in msg object WITHOUT calling AddRef. + * rgerhards, 2013-01-22 + */ +void MsgSetRcvFromWithoutAddRef(msg_t *pThis, prop_t *new) +{ + assert(pThis != NULL); + + if(pThis->msgFlags & NEEDS_DNSRESOL) { + if(pThis->rcvFrom.pfrominet != NULL) + free(pThis->rcvFrom.pfrominet); + pThis->msgFlags &= ~NEEDS_DNSRESOL; + } else { + if(pThis->rcvFrom.pRcvFrom != NULL) + prop.Destruct(&pThis->rcvFrom.pRcvFrom); + } + pThis->rcvFrom.pRcvFrom = new; +} + + +/* rgerhards 2012-04-18: set associated ruleset (by ruleset name) + * If ruleset cannot be found, no update is done. + */ +static void +MsgSetRulesetByName(msg_t *pMsg, cstr_t *rulesetName) +{ + rulesetGetRuleset(runConf, &(pMsg->pRuleset), rsCStrGetSzStrNoNULL(rulesetName)); +} + + static inline int getProtocolVersion(msg_t *pM) { return(pM->iProtocolVersion); @@ -321,19 +391,18 @@ static inline rsRetVal resolveDNS(msg_t *pMsg) { rsRetVal localRet; prop_t *propFromHost = NULL; - prop_t *propFromHostIP = NULL; - uchar fromHost[NI_MAXHOST]; - uchar fromHostIP[NI_MAXHOST]; - uchar fromHostFQDN[NI_MAXHOST]; + prop_t *ip; + prop_t *localName; DEFiRet; MsgLock(pMsg); CHKiRet(objUse(net, CORE_COMPONENT)); if(pMsg->msgFlags & NEEDS_DNSRESOL) { - localRet = net.cvthname(pMsg->rcvFrom.pfrominet, fromHost, fromHostFQDN, fromHostIP); + localRet = net.cvthname(pMsg->rcvFrom.pfrominet, &localName, NULL, &ip); if(localRet == RS_RET_OK) { - MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); - CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); + /* we pass down the props, so no need for AddRef */ + MsgSetRcvFromWithoutAddRef(pMsg, localName); + MsgSetRcvFromIPWithoutAddRef(pMsg, ip); } } finalize_it: @@ -345,8 +414,6 @@ finalize_it: MsgUnlock(pMsg); if(propFromHost != NULL) prop.Destruct(&propFromHost); - if(propFromHostIP != NULL) - prop.Destruct(&propFromHostIP); RETiRet; } @@ -442,8 +509,10 @@ propNameStrToID(uchar *pName, propid_t *pPropID) *pPropID = PROP_MSGID; } else if(!strcmp((char*) pName, "parsesuccess")) { *pPropID = PROP_PARSESUCCESS; +#ifdef USE_LIBUUID } else if(!strcmp((char*) pName, "uuid")) { *pPropID = PROP_UUID; +#endif /* here start system properties (those, that do not relate to the message itself */ } else if(!strcmp((char*) pName, "$now")) { *pPropID = PROP_SYS_NOW; @@ -611,6 +680,7 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->iRefCount = 1; pM->iSeverity = -1; pM->iFacility = -1; + pM->iLenPROGNAME = -1; pM->offAfterPRI = 0; pM->offMSG = -1; pM->iProtocolVersion = 0; @@ -629,7 +699,6 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->pszTIMESTAMP3339 = NULL; pM->pszTIMESTAMP_MySQL = NULL; pM->pszTIMESTAMP_PgSQL = NULL; - pM->pCSProgName = NULL; pM->pCSStrucData = NULL; pM->pCSAPPNAME = NULL; pM->pCSPROCID = NULL; @@ -638,7 +707,7 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->pRcvFromIP = NULL; pM->rcvFrom.pRcvFrom = NULL; pM->pRuleset = NULL; - pM->event = NULL; + pM->json = NULL; memset(&pM->tRcvdAt, 0, sizeof(pM->tRcvdAt)); memset(&pM->tTIMESTAMP, 0, sizeof(pM->tTIMESTAMP)); pM->TAG.pszTAG = NULL; @@ -708,6 +777,19 @@ finalize_it: } +/* Special msg constructor, to be used when an object is deserialized. + * we do only the base init as we know the properties will be set in + * any case by the deserializer. We still do the "inexpensive" inits + * just to be on the safe side. The whole process needs to be + * refactored together with the msg serialization subsystem. + */ +rsRetVal +msgConstructForDeserializer(msg_t **ppThis) +{ + return msgBaseConstruct(ppThis); +} + + /* some free handlers for (slightly) complicated cases... All of them may be called * with an empty element. */ @@ -759,8 +841,8 @@ CODESTARTobjDestruct(msg) free(pThis->pszRcvdAt_PgSQL); free(pThis->pszTIMESTAMP_MySQL); free(pThis->pszTIMESTAMP_PgSQL); - if(pThis->pCSProgName != NULL) - rsCStrDestruct(&pThis->pCSProgName); + if(pThis->iLenPROGNAME >= CONF_PROGNAME_BUFSIZE) + free(pThis->PROGNAME.ptr); if(pThis->pCSStrucData != NULL) rsCStrDestruct(&pThis->pCSStrucData); if(pThis->pCSAPPNAME != NULL) @@ -769,8 +851,8 @@ CODESTARTobjDestruct(msg) rsCStrDestruct(&pThis->pCSPROCID); if(pThis->pCSMSGID != NULL) rsCStrDestruct(&pThis->pCSMSGID); - if(pThis->event != NULL) - ee_deleteEvent(pThis->event); + if(pThis->json != NULL) + json_object_put(pThis->json); if(pThis->pszUUID != NULL) free(pThis->pszUUID); # ifndef HAVE_ATOMIC_BUILTINS @@ -913,12 +995,14 @@ msg_t* MsgDup(msg_t* pOld) } } - tmpCOPYCSTR(ProgName); tmpCOPYCSTR(StrucData); tmpCOPYCSTR(APPNAME); tmpCOPYCSTR(PROCID); tmpCOPYCSTR(MSGID); + if(pOld->json != NULL) + pNew->json = jsonDeepCopy(pOld->json); + /* we do not copy all other cache properties, as we do not even know * if they are needed once again. So we let them re-create if needed. */ @@ -972,6 +1056,10 @@ static rsRetVal MsgSerialize(msg_t *pThis, strm_t *pStrm) CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFrom"), PROPTYPE_PSZ, (void*) psz)); psz = getRcvFromIP(pThis); CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFromIP"), PROPTYPE_PSZ, (void*) psz)); + if(pThis->json != NULL) { + psz = (uchar*) json_object_get_string(pThis->json); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("json"), PROPTYPE_PSZ, (void*) psz)); + } objSerializePTR(pStrm, pCSStrucData, CSTR); objSerializePTR(pStrm, pCSAPPNAME, CSTR); @@ -998,6 +1086,168 @@ finalize_it: } +/* This is a helper for MsgDeserialize that re-inits the var object. This + * whole construct should be replaced, var is really ready to be retired. + * But as an interim help during refactoring let's introduce this function + * here (and thus NOT as method of var object!). -- rgerhads, 2012-11-06 + */ +static inline void +reinitVar(var_t *pVar) +{ + rsCStrDestruct(&pVar->pcsName); /* no longer needed */ + if(pVar->varType == VARTYPE_STR) { + if(pVar->val.pStr != NULL) + rsCStrDestruct(&pVar->val.pStr); + } +} +/* deserialize the message again + * we deserialize the properties in the same order that we serialized them. Except + * for some checks to cover downlevel version, we do not need to do all these + * CPU intense name checkings. + */ +#define isProp(name) !rsCStrSzStrCmp(pVar->pcsName, (uchar*) name, sizeof(name) - 1) +rsRetVal +MsgDeserialize(msg_t *pMsg, strm_t *pStrm) +{ + prop_t *myProp; + prop_t *propRcvFrom = NULL; + prop_t *propRcvFromIP = NULL; + struct json_tokener *tokener; + var_t *pVar = NULL; + DEFiRet; + + ISOBJ_TYPE_assert(pStrm, strm); + + CHKiRet(var.Construct(&pVar)); + CHKiRet(var.ConstructFinalize(pVar)); + + CHKiRet(objDeserializeProperty(pVar, pStrm)); + if(isProp("iProtocolVersion")) { + setProtocolVersion(pMsg, pVar->val.num); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("iSeverity")) { + pMsg->iSeverity = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("iFacility")) { + pMsg->iFacility = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("msgFlags")) { + pMsg->msgFlags = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("ttGenTime")) { + pMsg->ttGenTime = pVar->val.num; + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("tRcvdAt")) { + memcpy(&pMsg->tRcvdAt, &pVar->val.vSyslogTime, sizeof(struct syslogTime)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("tTIMESTAMP")) { + memcpy(&pMsg->tTIMESTAMP, &pVar->val.vSyslogTime, sizeof(struct syslogTime)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszTAG")) { + MsgSetTAG(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), cstrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRawMsg")) { + MsgSetRawMsg(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr), cstrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszHOSTNAME")) { + MsgSetHOSTNAME(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszInputName")) { + /* we need to create a property */ + CHKiRet(prop.Construct(&myProp)); + CHKiRet(prop.SetString(myProp, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr))); + CHKiRet(prop.ConstructFinalize(myProp)); + MsgSetInputName(pMsg, myProp); + prop.Destruct(&myProp); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRcvFrom")) { + MsgSetRcvFromStr(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr), &propRcvFrom); + prop.Destruct(&propRcvFrom); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRcvFromIP")) { + MsgSetRcvFromIPStr(pMsg, rsCStrGetSzStrNoNULL(pVar->val.pStr), rsCStrLen(pVar->val.pStr), &propRcvFromIP); + prop.Destruct(&propRcvFromIP); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("json")) { + tokener = json_tokener_new(); + pMsg->json = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pVar->val.pStr), + cstrLen(pVar->val.pStr)); + json_tokener_free(tokener); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSStrucData")) { + MsgSetStructuredData(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSAPPNAME")) { + MsgSetAPPNAME(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSPROCID")) { + MsgSetPROCID(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pCSMSGID")) { + MsgSetMSGID(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszUUID")) { + pMsg->pszUUID = ustrdup(rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszRuleset")) { + MsgSetRulesetByName(pMsg, pVar->val.pStr); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + /* "offMSG" must always be our last field, so we use this as an + * indicator if the sequence is correct. This is a bit questionable, + * but on the other hand it works decently AND we will probably replace + * the whole persisted format soon in any case. -- rgerhards, 2012-11-06 + */ + if(!isProp("offMSG")) + ABORT_FINALIZE(RS_RET_DS_PROP_SEQ_ERR); + MsgSetMSGoffs(pMsg, pVar->val.num); +finalize_it: + if(pVar != NULL) + var.Destruct(&pVar); + RETiRet; +} +#undef isProp + + /* Increment reference count - see description of the "msg" * structure for details. As a convenience to developers, * this method returns the msg pointer that is passed to it. @@ -1092,32 +1342,33 @@ finalize_it: * The above definition has been taken from the FreeBSD syslogd sources. * * The program name is not parsed by default, because it is infrequently-used. - * If it is needed, this function should be called first. It checks if it is - * already set and extracts it, if not. - * * IMPORTANT: A locked message object must be provided, else a crash will occur. * rgerhards, 2005-10-19 */ -static rsRetVal aquireProgramName(msg_t *pM) +static inline rsRetVal +aquireProgramName(msg_t *pM) { - register int i; - uchar *pszTag; + int i; + uchar *pszTag, *pszProgName; DEFiRet; assert(pM != NULL); - if(pM->pCSProgName == NULL) { - /* ok, we do not yet have it. So let's parse the TAG to obtain it. */ - pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); - CHKiRet(cstrConstruct(&pM->pCSProgName)); - for( i = 0 - ; (i < pM->iLenTAG) && isprint((int) pszTag[i]) - && (pszTag[i] != '\0') && (pszTag[i] != ':') - && (pszTag[i] != '[') && (pszTag[i] != '/') - ; ++i) { - CHKiRet(cstrAppendChar(pM->pCSProgName, pszTag[i])); - } - CHKiRet(cstrFinalize(pM->pCSProgName)); + pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); + for( i = 0 + ; (i < pM->iLenTAG) && isprint((int) pszTag[i]) + && (pszTag[i] != '\0') && (pszTag[i] != ':') + && (pszTag[i] != '[') && (pszTag[i] != '/') + ; ++i) + ; /* just search end of PROGNAME */ + if(i < CONF_PROGNAME_BUFSIZE) { + pszProgName = pM->PROGNAME.szBuf; + } else { + CHKmalloc(pM->PROGNAME.ptr = malloc(i+1)); + pszProgName = pM->PROGNAME.ptr; } + memcpy((char*)pszProgName, (char*)pszTag, i); + pszProgName[i] = '\0'; + pM->iLenPROGNAME = i; finalize_it: RETiRet; } @@ -1142,6 +1393,7 @@ char *getProtocolVersionString(msg_t *pM) return(pM->iProtocolVersion ? "1" : "0"); } +#ifdef USE_LIBUUID /* note: libuuid seems not to be thread-safe, so we need * to get some safeguards in place. */ @@ -1196,6 +1448,7 @@ void getUUID(msg_t *pM, uchar **pBuf, int *piLen) } dbgprintf("[getUUID] END\n"); } +#endif void getRawMsg(msg_t *pM, uchar **pBuf, int *piLen) @@ -1215,6 +1468,14 @@ getRawMsg(msg_t *pM, uchar **pBuf, int *piLen) } +/* note: setMSGLen() is only for friends who really know what they + * do. Setting an invalid length can be desasterous! + */ +void setMSGLen(msg_t *pM, int lenMsg) +{ + pM->iLenMSG = lenMsg; +} + int getMSGLen(msg_t *pM) { return((pM == NULL) ? 0 : pM->iLenMSG); @@ -1664,16 +1925,6 @@ void MsgSetRuleset(msg_t *pMsg, ruleset_t *pRuleset) } -/* rgerhards 2012-04-18: set associated ruleset (by ruleset name) - * If ruleset cannot be found, no update is done. - */ -static void -MsgSetRulesetByName(msg_t *pMsg, cstr_t *rulesetName) -{ - rulesetGetRuleset(runConf, &(pMsg->pRuleset), rsCStrGetSzStrNoNULL(rulesetName)); -} - - /* set TAG in msg object * (rewritten 2009-06-18 rgerhards) */ @@ -1862,53 +2113,24 @@ static inline char *getStructuredData(msg_t *pM) return (char*) pszRet; } -/* check if we have a ProgramName, and, if not, try to aquire/emulate it. - * rgerhards, 2009-06-26 - */ -static inline void prepareProgramName(msg_t *pM, sbool bLockMutex) -{ - if(pM->pCSProgName == NULL) { - if(bLockMutex == LOCK_MUTEX) - MsgLock(pM); - - /* re-query as things might have changed during locking */ - if(pM->pCSProgName == NULL) - aquireProgramName(pM); - - if(bLockMutex == LOCK_MUTEX) - MsgUnlock(pM); - } -} - - -/* get the length of the "programname" sz string - * rgerhards, 2005-10-19 - */ -int getProgramNameLen(msg_t *pM, sbool bLockMutex) -{ - assert(pM != NULL); - prepareProgramName(pM, bLockMutex); - return (pM->pCSProgName == NULL) ? 0 : rsCStrLen(pM->pCSProgName); -} - - /* get the "programname" as sz string * rgerhards, 2005-10-19 */ uchar *getProgramName(msg_t *pM, sbool bLockMutex) { - uchar *pszRet; - - if(bLockMutex == LOCK_MUTEX) - MsgLock(pM); - prepareProgramName(pM, MUTEX_ALREADY_LOCKED); - if(pM->pCSProgName == NULL) - pszRet = UCHAR_CONSTANT(""); - else - pszRet = rsCStrGetSzStrNoNULL(pM->pCSProgName); - if(bLockMutex == LOCK_MUTEX) - MsgUnlock(pM); - return pszRet; + if(pM->iLenPROGNAME == -1) { + if(bLockMutex == LOCK_MUTEX) { + MsgLock(pM); + /* need to re-check, things may have change in between! */ + if(pM->iLenPROGNAME == -1) + aquireProgramName(pM); + MsgUnlock(pM); + } else { + aquireProgramName(pM); + } + } + return (pM->iLenPROGNAME < CONF_PROGNAME_BUFSIZE) ? pM->PROGNAME.szBuf + : pM->PROGNAME.ptr; } @@ -2023,18 +2245,8 @@ finalize_it: */ void MsgSetRcvFrom(msg_t *pThis, prop_t *new) { - assert(pThis != NULL); - prop.AddRef(new); - if(pThis->msgFlags & NEEDS_DNSRESOL) { - if(pThis->rcvFrom.pfrominet != NULL) - free(pThis->rcvFrom.pfrominet); - pThis->msgFlags &= ~NEEDS_DNSRESOL; - } else { - if(pThis->rcvFrom.pRcvFrom != NULL) - prop.Destruct(&pThis->rcvFrom.pRcvFrom); - } - pThis->rcvFrom.pRcvFrom = new; + MsgSetRcvFromWithoutAddRef(pThis, new); } @@ -2067,9 +2279,7 @@ rsRetVal MsgSetRcvFromIP(msg_t *pThis, prop_t *new) BEGINfunc prop.AddRef(new); - if(pThis->pRcvFromIP != NULL) - prop.Destruct(&pThis->pRcvFromIP); - pThis->pRcvFromIP = new; + MsgSetRcvFromIPWithoutAddRef(pThis, new); ENDfunc return RS_RET_OK; } @@ -2223,21 +2433,20 @@ void MsgSetRawMsgWOSize(msg_t *pMsg, char* pszRawMsg) /* Decode a priority into textual information like auth.emerg. - * The variable pRes must point to a user-supplied buffer and - * pResLen must contain its size. The pointer to the buffer + * The variable pRes must point to a user-supplied buffer. + * The pointer to the buffer * is also returned, what makes this functiona suitable for * use in printf-like functions. * Note: a buffer size of 20 characters is always sufficient. - * Interface to this function changed 2007-06-15 by RGerhards */ -char *textpri(char *pRes, size_t pResLen, int pri) +char *textpri(char *pRes, int pri) { assert(pRes != NULL); - assert(pResLen > 0); - - snprintf(pRes, pResLen, "%s.%s", syslog_fac_names[LOG_FAC(pri)], - syslog_severity_names[LOG_PRI(pri)]); - + memcpy(pRes, syslog_fac_names[LOG_FAC(pri)], len_syslog_fac_names[LOG_FAC(pri)]); + pRes[len_syslog_fac_names[LOG_FAC(pri)]] = '.'; + memcpy(pRes+len_syslog_fac_names[LOG_FAC(pri)]+1, + syslog_severity_names[LOG_PRI(pri)], + len_syslog_severity_names[LOG_PRI(pri)]+1 /* for \0! */); return pRes; } @@ -2251,40 +2460,48 @@ char *textpri(char *pRes, size_t pResLen, int pri) */ typedef enum ENOWType { NOW_NOW, NOW_YEAR, NOW_MONTH, NOW_DAY, NOW_HOUR, NOW_HHOUR, NOW_QHOUR, NOW_MINUTE } eNOWType; #define tmpBUFSIZE 16 /* size of formatting buffer */ -static uchar *getNOW(eNOWType eNow) +static uchar *getNOW(eNOWType eNow, struct syslogTime *t) { uchar *pBuf; - struct syslogTime t; if((pBuf = (uchar*) MALLOC(sizeof(uchar) * tmpBUFSIZE)) == NULL) { return NULL; } - datetime.getCurrTime(&t, NULL); + if(t->year == 0) { /* not yet set! */ + datetime.getCurrTime(t, NULL); + } + switch(eNow) { case NOW_NOW: - snprintf((char*) pBuf, tmpBUFSIZE, "%4.4d-%2.2d-%2.2d", t.year, t.month, t.day); + memcpy(pBuf, two_digits[t->year/100], 2); + memcpy(pBuf+2, two_digits[t->year%100], 2); + pBuf[4] = '-'; + memcpy(pBuf+5, two_digits[(int)t->month], 2); + pBuf[7] = '-'; + memcpy(pBuf+8, two_digits[(int)t->day], 3); break; case NOW_YEAR: - snprintf((char*) pBuf, tmpBUFSIZE, "%4.4d", t.year); + memcpy(pBuf, two_digits[t->year/100], 2); + memcpy(pBuf+2, two_digits[t->year%100], 3); break; case NOW_MONTH: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.month); + memcpy(pBuf, two_digits[(int)t->month], 3); break; case NOW_DAY: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.day); + memcpy(pBuf, two_digits[(int)t->day], 3); break; case NOW_HOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.hour); + memcpy(pBuf, two_digits[(int)t->hour], 3); break; case NOW_HHOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute / 30); + memcpy(pBuf, two_digits[t->minute/30], 3); break; case NOW_QHOUR: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute / 15); + memcpy(pBuf, two_digits[t->minute/15], 3); break; case NOW_MINUTE: - snprintf((char*) pBuf, tmpBUFSIZE, "%2.2d", t.minute); + memcpy(pBuf, two_digits[(int)t->minute], 3); break; } @@ -2293,39 +2510,77 @@ static uchar *getNOW(eNOWType eNow) #undef tmpBUFSIZE /* clean up */ -/* Get a CEE-Property from libee. This function probably should be - * placed somewhere else, but this smells like a big restructuring - * useful in any case. So for the time being, I'll simply leave the - * function here, as the context seems good enough. -- rgerhards, 2010-12-01 - */ -static inline void -getCEEPropVal(msg_t *pMsg, es_str_t *propName, uchar **pRes, int *buflen, unsigned short *pbMustBeFreed) +/* Get a CEE-Property as string value*/ +rsRetVal +getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed) { - es_str_t *str = NULL; - int r; + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + struct json_object *field; + DEFiRet; if(*pbMustBeFreed) free(*pRes); *pRes = NULL; + // TODO: mutex? + if(pM->json == NULL) goto finalize_it; - if(pMsg->event == NULL) goto finalize_it; - r = ee_getEventFieldAsString(pMsg->event, propName, &str); - - if(r != EE_OK) { - DBGPRINTF("msgGtCEEVar: libee error %d during ee_getEventFieldAsString\n", r); - FINALIZE; + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + field = pM->json; + } else { + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + field = json_object_object_get(parent, (char*)leaf); + } + if(field != NULL) { + *pRes = (uchar*) strdup(json_object_get_string(field)); + *buflen = (int) ustrlen(*pRes); + *pbMustBeFreed = 1; } - *pRes = (unsigned char*) es_str2cstr(str, "#000"); - es_deleteStr(str); - *buflen = (int) ustrlen(*pRes); - *pbMustBeFreed = 1; finalize_it: + free(name); if(*pRes == NULL) { /* could not find any value, so set it to empty */ *pRes = (unsigned char*)""; *pbMustBeFreed = 0; } + RETiRet; +} + + +/* Get a CEE-Property as native json object + */ +rsRetVal +msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson) +{ + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + DEFiRet; + + // TODO: mutex? + if(pM->json == NULL) { + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + *pjson = pM->json; + FINALIZE; + } + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + *pjson = json_object_object_get(parent, (char*)leaf); + if(*pjson == NULL) { + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + +finalize_it: + free(name); + RETiRet; } @@ -2513,7 +2768,7 @@ finalize_it: * Parameter "bMustBeFreed" is set by this function. It tells the * caller whether or not the string returned must be freed by the * caller itself. It is is 0, the caller MUST NOT free it. If it is - * 1, the caller MUST free 1. Handling this wrongly leads to either + * 1, the caller MUST free it. Handling this wrongly leads to either * a memory leak of a program abort (do to double-frees or frees on * the constant memory pool). So be careful to do it right. * rgerhards 2004-11-23 @@ -2529,16 +2784,16 @@ finalize_it: *pPropLen = sizeof("**OUT OF MEMORY**") - 1; \ return(UCHAR_CONSTANT("**OUT OF MEMORY**"));} uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - propid_t propid, es_str_t *propName, size_t *pPropLen, - unsigned short *pbMustBeFreed) + propid_t propid, es_str_t *propName, rs_size_t *pPropLen, + unsigned short *pbMustBeFreed, struct syslogTime *ttNow) { uchar *pRes; /* result pointer */ - int bufLen = -1; /* length of string or -1, if not known */ + rs_size_t bufLen = -1; /* length of string or -1, if not known */ uchar *pBufStart; uchar *pBuf; int iLen; short iOffs; - es_str_t *str; /* for CEE handling, temp. string */ + enum tplFormatTypes datefmt; BEGINfunc assert(pMsg != NULL); @@ -2558,7 +2813,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, bufLen = getMSGLen(pMsg); break; case PROP_TIMESTAMP: - pRes = (uchar*)getTimeReported(pMsg, pTpe->data.field.eDateFormat); + if (pTpe != NULL) + datefmt = pTpe->data.field.eDateFormat; + else + datefmt = tplFmtDefault; + pRes = (uchar*)getTimeReported(pMsg, datefmt); break; case PROP_HOSTNAME: pRes = (uchar*)getHOSTNAME(pMsg); @@ -2588,7 +2847,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, RET_OUT_OF_MEMORY; } else { *pbMustBeFreed = 1; - pRes = (uchar*)textpri((char*)pBuf, 20, getPRIi(pMsg)); + pRes = (uchar*)textpri((char*)pBuf, getPRIi(pMsg)); } break; case PROP_IUT: @@ -2608,7 +2867,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, pRes = (uchar*)getSeverityStr(pMsg); break; case PROP_TIMEGENERATED: - pRes = (uchar*)getTimeGenerated(pMsg, pTpe->data.field.eDateFormat); + if (pTpe != NULL) + datefmt = pTpe->data.field.eDateFormat; + else + datefmt = tplFmtDefault; + pRes = (uchar*)getTimeGenerated(pMsg, datefmt); break; case PROP_PROGRAMNAME: pRes = getProgramName(pMsg, LOCK_MUTEX); @@ -2628,74 +2891,91 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, case PROP_MSGID: pRes = (uchar*)getMSGID(pMsg); break; +#ifdef USE_LIBUUID case PROP_UUID: getUUID(pMsg, &pRes, &bufLen); break; +#endif case PROP_PARSESUCCESS: pRes = (uchar*)getParseSuccess(pMsg); break; case PROP_SYS_NOW: - if((pRes = getNOW(NOW_NOW)) == NULL) { + if((pRes = getNOW(NOW_NOW, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 10; + } break; case PROP_SYS_YEAR: - if((pRes = getNOW(NOW_YEAR)) == NULL) { + if((pRes = getNOW(NOW_YEAR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 4; + } break; case PROP_SYS_MONTH: - if((pRes = getNOW(NOW_MONTH)) == NULL) { + if((pRes = getNOW(NOW_MONTH, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_DAY: - if((pRes = getNOW(NOW_DAY)) == NULL) { + if((pRes = getNOW(NOW_DAY, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_HOUR: - if((pRes = getNOW(NOW_HOUR)) == NULL) { + if((pRes = getNOW(NOW_HOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_HHOUR: - if((pRes = getNOW(NOW_HHOUR)) == NULL) { + if((pRes = getNOW(NOW_HHOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_QHOUR: - if((pRes = getNOW(NOW_QHOUR)) == NULL) { + if((pRes = getNOW(NOW_QHOUR, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_MINUTE: - if((pRes = getNOW(NOW_MINUTE)) == NULL) { + if((pRes = getNOW(NOW_MINUTE, ttNow)) == NULL) { RET_OUT_OF_MEMORY; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + } else { + *pbMustBeFreed = 1; + bufLen = 2; + } break; case PROP_SYS_MYHOSTNAME: pRes = glbl.GetLocalHostName(); break; case PROP_CEE_ALL_JSON: - if(pMsg->event == NULL) { - if(*pbMustBeFreed == 1) - free(pRes); - pRes = (uchar*) "{}"; - *pbMustBeFreed = 0; + if(pMsg->json == NULL) { + if(*pbMustBeFreed == 1) + free(pRes); + pRes = (uchar*) "{}"; + bufLen = 2; + *pbMustBeFreed = 0; } else { - ee_fmtEventToJSON(pMsg->event, &str); - pRes = (uchar*) es_str2cstr(str, "#000"); - es_deleteStr(str); - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + pRes = (uchar*)strdup(json_object_get_string(pMsg->json)); + *pbMustBeFreed = 1; } break; case PROP_CEE: @@ -2708,13 +2988,40 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, *pbMustBeFreed = 0; break; case PROP_SYS_UPTIME: -# ifdef OS_SOLARIS - pRes = (uchar*) "UPTIME NOT available under Solaris"; +# ifndef HAVE_SYSINFO_UPTIME + /* An alternative on some systems (eg Solaris) is to scan + * /var/adm/utmpx for last boot time. + */ + pRes = (uchar*) "UPTIME NOT available on this system"; *pbMustBeFreed = 0; + +# elif defined(__FreeBSD__) + + { + struct timespec tp; + + if(*pbMustBeFreed == 1) + free(pRes); + if((pRes = (uchar*) MALLOC(sizeof(uchar) * 32)) == NULL) { + RET_OUT_OF_MEMORY; + } + *pbMustBeFreed = 1; + + if(clock_gettime(CLOCK_UPTIME, &tp) == -1) { + *pPropLen = sizeof("**SYSCALL FAILED**") - 1; + return(UCHAR_CONSTANT("**SYSCALL FAILED**")); + } + + snprintf((char*) pRes, sizeof(uchar) * 32, "%ld", tp.tv_sec); + } + # else + { struct sysinfo s_info; + if(*pbMustBeFreed == 1) + free(pRes); if((pRes = (uchar*) MALLOC(sizeof(uchar) * 32)) == NULL) { RET_OUT_OF_MEMORY; } @@ -2740,7 +3047,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } /* If we did not receive a template pointer, we are already done... */ - if(pTpe == NULL) { + if(pTpe == NULL || !pTpe->bComplexProcessing) { *pPropLen = (bufLen == -1) ? ustrlen(pRes) : bufLen; return pRes; } @@ -2947,13 +3254,18 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, uchar *pSb; iFrom = pTpe->data.field.iFromPos; iTo = pTpe->data.field.iToPos; - /* need to zero-base to and from (they are 1-based!) */ - if(iFrom > 0) - --iFrom; - if(iTo > 0) - --iTo; if(bufLen == -1) bufLen = ustrlen(pRes); + if(pTpe->data.field.options.bFromPosEndRelative) { + iFrom = (bufLen < iFrom) ? 0 : bufLen - iFrom; + iTo = (bufLen < iTo)? 0 : bufLen - iTo; + } else { + /* need to zero-base to and from (they are 1-based!) */ + if(iFrom > 0) + --iFrom; + if(iTo > 0) + --iTo; + } if(iFrom == 0 && iTo >= bufLen) { /* in this case, the requested string is a superset of what we already have, * so there is no need to do any processing. This is a frequent case for size-limited @@ -2962,6 +3274,8 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ ; /*DO NOTHING*/ } else { + if(iTo > bufLen) /* iTo is very large, if no to-position is set in the template! */ + iTo = bufLen; iLen = iTo - iFrom + 1; /* the +1 is for an actual char, NOT \0! */ pBufStart = pBuf = MALLOC((iLen + 1) * sizeof(char)); if(pBuf == NULL) { @@ -3327,9 +3641,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, jsonField(pTpe, &pRes, pbMustBeFreed, &bufLen); } - if(bufLen == -1) - bufLen = ustrlen(pRes); - *pPropLen = bufLen; + *pPropLen = (bufLen == -1) ? ustrlen(pRes) : bufLen; ENDfunc return(pRes); @@ -3346,29 +3658,25 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, es_str_t* msgGetCEEVarNew(msg_t *pMsg, char *name) { + uchar *leaf; + char *val; es_str_t *estr = NULL; - es_str_t *epropName = NULL; - struct ee_field *field; + struct json_object *json, *parent; ISOBJ_TYPE_assert(pMsg, msg); - if(pMsg->event == NULL) { + if(pMsg->json == NULL) { estr = es_newStr(1); goto done; } - - epropName = es_newStrFromCStr(name, strlen(name)); // TODO: optimize (in grammar!) - field = ee_getEventField(pMsg->event, epropName); - if(field != NULL) { - ee_getFieldAsString(field, &estr); - } - if(estr == NULL) { - DBGPRINTF("msgGetCEEVar: error obtaining var (field=%p, var='%s')\n", - field, name); - estr = es_newStrFromCStr("*ERROR*", sizeof("*ERROR*") - 1); + leaf = jsonPathGetLeaf((uchar*)name, strlen(name)); + if(jsonPathFindParent(pMsg, (uchar*)name, leaf, &parent, 1) != RS_RET_OK) { + estr = es_newStr(1); + goto done; } - es_deleteStr(epropName); - + json = json_object_object_get(parent, (char*)leaf); + val = (char*)json_object_get_string(json); + estr = es_newStrFromCStr(val, strlen(val)); done: return estr; } @@ -3379,7 +3687,7 @@ done: es_str_t* msgGetMsgVarNew(msg_t *pThis, uchar *name) { - size_t propLen; + rs_size_t propLen; uchar *pszProp = NULL; propid_t propid; unsigned short bMustBeFreed = 0; @@ -3390,7 +3698,7 @@ msgGetMsgVarNew(msg_t *pThis, uchar *name) /* always call MsgGetProp() without a template specifier */ /* TODO: optimize propNameToID() call -- rgerhards, 2009-06-26 */ propNameStrToID(name, &propid); - pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, NULL, &propLen, &bMustBeFreed); + pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, NULL, &propLen, &bMustBeFreed, NULL); estr = es_newStrFromCStr((char*)pszProp, propLen); if(bMustBeFreed) @@ -3412,6 +3720,8 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) prop_t *myProp; prop_t *propRcvFrom = NULL; prop_t *propRcvFromIP = NULL; + struct json_tokener *tokener; + struct json_object *json; DEFiRet; ISOBJ_TYPE_assert(pThis, msg); @@ -3466,6 +3776,12 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) MsgSetRulesetByName(pThis, pProp->val.pStr); } else if(isProp("pszMSG")) { dbgprintf("no longer supported property pszMSG silently ignored\n"); + } else if(isProp("json")) { + tokener = json_tokener_new(); + json = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pProp->val.pStr), + cstrLen(pProp->val.pStr)); + json_tokener_free(tokener); + msgAddJSON(pThis, (uchar*)"!", json); } else { dbgprintf("unknown supported property '%s' silently ignored\n", rsCStrGetSzStrNoNULL(pProp->pcsName)); @@ -3481,16 +3797,293 @@ finalize_it: * satisfies the base object class getSeverity semantics. * rgerhards, 2008-01-14 */ -static rsRetVal -MsgGetSeverity(obj_t_ptr pThis, int *piSeverity) +rsRetVal +MsgGetSeverity(msg_t *pMsg, int *piSeverity) { - ISOBJ_TYPE_assert(pThis, msg); - assert(piSeverity != NULL); - *piSeverity = ((msg_t*) pThis)->iSeverity; + *piSeverity = pMsg->iSeverity; return RS_RET_OK; } +static uchar * +jsonPathGetLeaf(uchar *name, int lenName) +{ + int i; + for(i = lenName ; name[i] != '!' && i >= 0 ; --i) + /* just skip */; + if(name[i] == '!') + ++i; + return name + i; +} + + +static rsRetVal +jsonPathFindNext(struct json_object *root, uchar **name, uchar *leaf, + struct json_object **found, int bCreate) +{ + uchar namebuf[1024]; + struct json_object *json; + size_t i; + uchar *p = *name; + DEFiRet; + + if(*p == '!') + ++p; + for(i = 0 ; *p && *p != '!' && p != leaf && i < sizeof(namebuf)-1 ; ++i, ++p) + namebuf[i] = *p; + if(i > 0) { + namebuf[i] = '\0'; + dbgprintf("AAAA: next JSONPath elt: '%s'\n", namebuf); + json = json_object_object_get(root, (char*)namebuf); + } else + json = root; + if(json == NULL) { + if(!bCreate) { + ABORT_FINALIZE(RS_RET_JNAME_INVALID); + } else { + json = json_object_new_object(); + json_object_object_add(root, (char*)namebuf, json); + } + } + + *name = p; + *found = json; +finalize_it: + RETiRet; +} + +static rsRetVal +jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate) +{ + DEFiRet; + *parent = pM->json; + while(name < leaf-1) { + jsonPathFindNext(*parent, &name, leaf, parent, bCreate); + } + RETiRet; +} + +static rsRetVal +jsonMerge(struct json_object *existing, struct json_object *json) +{ + /* TODO: check & handle duplicate names */ + DEFiRet; + struct json_object_iter it; + + json_object_object_foreachC(json, it) { +DBGPRINTF("AAAA jsonMerge adds '%s'\n", it.key); + json_object_object_add(existing, it.key, + json_object_get(it.val)); + } + /* note: json-c does ref counting. We added all descandants refcounts + * in the loop above. So when we now free(_put) the root object, only + * root gets freed(). + */ + json_object_put(json); + RETiRet; +} + +/* find a JSON structure element (field or container doesn't matter). */ +rsRetVal +jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres) +{ + uchar *name = NULL; + uchar *leaf; + struct json_object *parent; + struct json_object *field; + DEFiRet; + + if(pM->json == NULL) { + field = NULL; + goto finalize_it; + } + + if(!es_strbufcmp(propName, (uchar*)"!", 1)) { + field = pM->json; + } else { + name = (uchar*)es_str2cstr(propName, NULL); + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 0)); + field = json_object_object_get(parent, (char*)leaf); + } + *jsonres = field; + +finalize_it: + free(name); + RETiRet; +} + +rsRetVal +msgAddJSON(msg_t *pM, uchar *name, struct json_object *json) +{ + /* TODO: error checks! This is a quick&dirty PoC! */ + struct json_object *parent, *leafnode; + uchar *leaf; + DEFiRet; + + MsgLock(pM); + if(name[0] == '!' && name[1] == '\0') { + if(pM->json == NULL) + pM->json = json; + else + CHKiRet(jsonMerge(pM->json, json)); + } else { + if(pM->json == NULL) { + /* now we need a root obj */ + pM->json = json_object_new_object(); + } + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + leafnode = json_object_object_get(parent, (char*)leaf); + if(leafnode == NULL) { + json_object_object_add(parent, (char*)leaf, json); + } else { + if(json_object_get_type(json) == json_type_object) { + CHKiRet(jsonMerge(pM->json, json)); + } else { +//dbgprintf("AAAA: leafnode already exists, type is %d, update with %d\n", (int)json_object_get_type(leafnode), (int)json_object_get_type(json)); + /* TODO: improve the code below, however, the current + * state is not really bad */ + if(json_object_get_type(leafnode) == json_type_object) { + DBGPRINTF("msgAddJSON: trying to update a container " + "node with a leaf, name is '%s' - " + "forbidden\n", name); + json_object_put(json); + ABORT_FINALIZE(RS_RET_INVLD_SETOP); + } + /* json-c code indicates we can simply replace a + * json type. Unfortunaltely, this is not documented + * as part of the interface spec. We still use it, + * because it speeds up processing. If it does not work + * at some point, use + * json_object_object_del(parent, (char*)leaf); + * before adding. rgerhards, 2012-09-17 + */ + json_object_object_add(parent, (char*)leaf, json); + } + } + } + +finalize_it: + MsgUnlock(pM); + RETiRet; +} + +rsRetVal +msgDelJSON(msg_t *pM, uchar *name) +{ + struct json_object *parent, *leafnode; + uchar *leaf; + DEFiRet; + +dbgprintf("AAAA: unset variable '%s'\n", name); + MsgLock(pM); + if(name[0] == '!' && name[1] == '\0') { + /* strange, but I think we should permit this. After all, + * we trust rsyslog.conf to be written by the admin. + */ + DBGPRINTF("unsetting JSON root object\n"); + json_object_put(pM->json); + pM->json = NULL; + } else { + if(pM->json == NULL) { + /* now we need a root obj */ + pM->json = json_object_new_object(); + } + leaf = jsonPathGetLeaf(name, ustrlen(name)); + CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + leafnode = json_object_object_get(parent, (char*)leaf); +DBGPRINTF("AAAA: unset found JSON value path '%s', " "leaf '%s', leafnode %p\n", name, leaf, leafnode); + if(leafnode == NULL) { + DBGPRINTF("unset JSON: could not find '%s'\n", name); + ABORT_FINALIZE(RS_RET_JNAME_NOTFOUND); + } else { + DBGPRINTF("deleting JSON value path '%s', " + "leaf '%s', type %d\n", + name, leaf, json_object_get_type(leafnode)); + json_object_object_del(parent, (char*)leaf); + } + } + +finalize_it: + MsgUnlock(pM); + RETiRet; +} + +static struct json_object * +jsonDeepCopy(struct json_object *src) +{ + struct json_object *dst = NULL, *json; + struct json_object_iter it; + int arrayLen, i; + + if(src == NULL) goto done; + + switch(json_object_get_type(src)) { + case json_type_boolean: + dst = json_object_new_boolean(json_object_get_boolean(src)); + break; + case json_type_double: + dst = json_object_new_double(json_object_get_double(src)); + break; + case json_type_int: + dst = json_object_new_int(json_object_get_int(src)); + break; + case json_type_string: + dst = json_object_new_string(json_object_get_string(src)); + break; + case json_type_object: + dst = json_object_new_object(); + json_object_object_foreachC(src, it) { + json = jsonDeepCopy(it.val); + json_object_object_add(dst, it.key, json); + } + break; + case json_type_array: + arrayLen = json_object_array_length(src); + dst = json_object_new_array(); + for(i = 0 ; i < arrayLen ; ++i) { + json = json_object_array_get_idx(src, i); + json = jsonDeepCopy(json); + json_object_array_add(dst, json); + } + break; + default:DBGPRINTF("jsonDeepCopy(): error unknown type %d\n", + json_object_get_type(src)); + dst = NULL; + break; + } +done: return dst; +} + + +rsRetVal +msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *v) +{ + struct json_object *json = NULL; + char *cstr; + DEFiRet; + switch(v->datatype) { + case 'S':/* string */ + cstr = es_str2cstr(v->d.estr, NULL); + json = json_object_new_string(cstr); + free(cstr); + break; + case 'N':/* number (integer) */ + json = json_object_new_int((int) v->d.n); + break; + case 'J':/* native JSON */ + json = jsonDeepCopy(v->d.json); + break; + default:DBGPRINTF("msgSetJSONFromVar: unsupported datatype %c\n", + v->datatype); + ABORT_FINALIZE(RS_RET_ERR); + } + msgAddJSON(pMsg, varname+1, json); +finalize_it: + RETiRet; +} + /* dummy */ rsRetVal msgQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } @@ -3503,11 +4096,10 @@ BEGINObjClassInit(msg, 1, OBJ_IS_CORE_MODULE) CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(var, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_SERIALIZE, MsgSerialize); - OBJSetMethodHandler(objMethod_SETPROPERTY, MsgSetProperty); - OBJSetMethodHandler(objMethod_GETSEVERITY, MsgGetSeverity); /* some more inits */ # if HAVE_MALLOC_TRIM INIT_ATOMIC_HELPER_MUT(mutTrimCtr); diff --git a/runtime/msg.h b/runtime/msg.h index c0b50709..6faf066a 100644 --- a/runtime/msg.h +++ b/runtime/msg.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-13 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -30,12 +30,11 @@ #include <pthread.h> #include <libestr.h> +#include <json/json.h> #include "obj.h" #include "syslogd-types.h" #include "template.h" #include "atomic.h" -#include "libee/libee.h" - /* rgerhards 2004-11-08: The following structure represents a * syslog message. @@ -75,6 +74,7 @@ struct msg { int iLenMSG; /* Length of the MSG part */ int iLenTAG; /* Length of the TAG part */ int iLenHOSTNAME; /* Length of HOSTNAME */ + int iLenPROGNAME; /* Length of PROGNAME (-1 = not yet set) */ uchar *pszRawMsg; /* message as it was received on the wire. This is important in case we * need to preserve cryptographic verifiers. */ uchar *pszHOSTNAME; /* HOSTNAME from syslog message */ @@ -86,7 +86,6 @@ struct msg { char *pszTIMESTAMP3339; /* TIMESTAMP as RFC3339 formatted string (32 charcters at most) */ char *pszTIMESTAMP_MySQL;/* TIMESTAMP as MySQL formatted string (always 14 charcters) */ char *pszTIMESTAMP_PgSQL;/* TIMESTAMP as PgSQL formatted string (always 21 characters) */ - cstr_t *pCSProgName; /* the (BSD) program name */ cstr_t *pCSStrucData; /* STRUCTURED-DATA */ cstr_t *pCSAPPNAME; /* APP-NAME */ cstr_t *pCSPROCID; /* PROCID */ @@ -108,11 +107,15 @@ struct msg { it obviously is solved in way or another...). */ struct syslogTime tRcvdAt;/* time the message entered this program */ struct syslogTime tTIMESTAMP;/* (parsed) value of the timestamp */ - struct ee_event *event; /**< libee event */ + struct json_object *json; /* some fixed-size buffers to save malloc()/free() for frequently used fields (from the default templates) */ uchar szRawMsg[CONF_RAWMSG_BUFSIZE]; /* most messages are small, and these are stored here (without malloc/free!) */ uchar szHOSTNAME[CONF_HOSTNAME_BUFSIZE]; union { + uchar *ptr; /* pointer to progname value */ + uchar szBuf[CONF_PROGNAME_BUFSIZE]; + } PROGNAME; + union { uchar *pszTAG; /* pointer to tag value */ uchar szBuf[CONF_TAG_BUFSIZE]; } TAG; @@ -145,6 +148,8 @@ struct msg { PROTOTYPEObjClassInit(msg); rsRetVal msgConstruct(msg_t **ppThis); rsRetVal msgConstructWithTime(msg_t **ppThis, struct syslogTime *stTime, time_t ttGenTime); +rsRetVal msgConstructForDeserializer(msg_t **ppThis); +rsRetVal msgConstructFinalizer(msg_t *pThis); rsRetVal msgDestruct(msg_t **ppM); msg_t* MsgDup(msg_t* pOld); msg_t *MsgAddRef(msg_t *pM); @@ -171,8 +176,7 @@ void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg, size_t lenMsg); rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG); uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, propid_t propid, es_str_t *propName, - size_t *pPropLen, unsigned short *pbMustBeFreed); -char *textpri(char *pRes, size_t pResLen, int pri); + rs_size_t *pPropLen, unsigned short *pbMustBeFreed, struct syslogTime *ttNow); rsRetVal msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar); es_str_t* msgGetMsgVarNew(msg_t *pThis, uchar *name); uchar *getRcvFrom(msg_t *pM); @@ -182,21 +186,34 @@ char *getPRI(msg_t *pMsg); void getRawMsg(msg_t *pM, uchar **pBuf, int *piLen); rsRetVal msgGetCEEVar(msg_t *pThis, cstr_t *propName, var_t **ppVar); es_str_t* msgGetCEEVarNew(msg_t *pMsg, char *name); +rsRetVal msgAddJSON(msg_t *pM, uchar *name, struct json_object *json); +rsRetVal getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed); +rsRetVal MsgGetSeverity(msg_t *pThis, int *piSeverity); +rsRetVal MsgDeserialize(msg_t *pMsg, strm_t *pStrm); /* TODO: remove these five (so far used in action.c) */ uchar *getMSG(msg_t *pM); char *getHOSTNAME(msg_t *pM); char *getPROCID(msg_t *pM, sbool bLockMutex); char *getAPPNAME(msg_t *pM, sbool bLockMutex); +void setMSGLen(msg_t *pM, int lenMsg); int getMSGLen(msg_t *pM); char *getHOSTNAME(msg_t *pM); int getHOSTNAMELen(msg_t *pM); uchar *getProgramName(msg_t *pM, sbool bLockMutex); -int getProgramNameLen(msg_t *pM, sbool bLockMutex); uchar *getRcvFrom(msg_t *pM); rsRetVal propNameToID(cstr_t *pCSPropName, propid_t *pPropID); uchar *propIDToName(propid_t propID); +rsRetVal msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson); +rsRetVal msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *var); +rsRetVal msgDelJSON(msg_t *pMsg, uchar *varname); +rsRetVal jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres); + +static inline rsRetVal +msgUnsetJSON(msg_t *pMsg, uchar *varname) { + return msgDelJSON(pMsg, varname+1); +} /* ------------------------------ some inline functions ------------------------------ */ diff --git a/runtime/net.c b/runtime/net.c index dcf9cb52..13391cc0 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -54,7 +54,11 @@ #include <fnmatch.h> #include <fcntl.h> #include <unistd.h> +#if HAVE_GETIFADDRS #include <ifaddrs.h> +#else +#include "compat/ifaddrs.h" +#endif /* HAVE_GETIFADDRS */ #include <sys/types.h> #include <arpa/inet.h> @@ -66,6 +70,7 @@ #include "errmsg.h" #include "net.h" #include "dnscache.h" +#include "prop.h" #ifdef OS_SOLARIS # define s6_addr32 _S6_un._S6_u32 @@ -79,6 +84,7 @@ MODULE_TYPE_NOKEEP DEFobjStaticHelpers DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) /* support for defining allowed TCP and UDP senders. We use the same * structure to implement this (a linked list), but we define two different @@ -226,6 +232,7 @@ finalize_it: /* enqueue the element */ if(pPeer->pWildcardRoot == NULL) { pPeer->pWildcardRoot = pNew; + pPeer->pWildcardLast = pNew; } else { pPeer->pWildcardLast->pNext = pNew; } @@ -575,7 +582,7 @@ static void clearAllowedSenders(uchar *pszType) { struct AllowedSenders *pPrev; - struct AllowedSenders *pCurr; + struct AllowedSenders *pCurr = NULL; if(setAllowRoot(&pCurr, pszType) != RS_RET_OK) return; /* if something went wrong, so let's leave */ @@ -983,7 +990,7 @@ MaskCmp(struct NetAddr *pAllow, uint8_t bits, struct sockaddr *pFrom, const char static int isAllowedSender2(uchar *pszType, struct sockaddr *pFrom, const char *pszFromHost, int bChkDNS) { struct AllowedSenders *pAllow; - struct AllowedSenders *pAllowRoot; + struct AllowedSenders *pAllowRoot = NULL; int bNeededDNS = 0; /* partial check because we could not resolve DNS? */ int ret; @@ -1111,98 +1118,15 @@ void debugListenInfo(int fd, char *type) } -/* Return a printable representation of a host address. - * Now (2007-07-16) also returns the full host name (if it could be obtained) - * in the second param [thanks to mildew@gmail.com for the patch]. - * The caller must provide buffer space for pszHost and pszHostFQDN. These - * buffers must be of size NI_MAXHOST. This is not checked here, because - * there is no way to check it. We use this way of doing things because it - * frees us from using dynamic memory allocation where it really does not - * pay. - * 2005-05-16 rgerhards: added IP representation. Must also be NI_MAXHOST +/* Return a printable representation of a host addresses. If + * a parameter is NULL, it is not set. rgerhards, 2013-01-22 */ -rsRetVal cvthname(struct sockaddr_storage *f, uchar *pszHost, uchar *pszHostFQDN, uchar *pszIP) +rsRetVal +cvthname(struct sockaddr_storage *f, prop_t **localName, prop_t **fqdn, prop_t **ip) { DEFiRet; - register uchar *p; - int count; - assert(f != NULL); - assert(pszHost != NULL); - assert(pszHostFQDN != NULL); - - iRet = dnscacheLookup(f, pszHostFQDN, pszIP); - - if(iRet == RS_RET_INVALID_SOURCE) { - strcpy((char*) pszHost, (char*) pszHostFQDN); /* we use whatever was provided as replacement */ - ABORT_FINALIZE(RS_RET_OK); /* this is handled, we are happy with it */ - } else if(iRet != RS_RET_OK) { - FINALIZE; /* we return whatever error state we have - can not handle it */ - } - - /* if we reach this point, we obtained a non-numeric hostname and can now process it */ - - /* Convert to lower case */ - for(p = pszHostFQDN ; *p ; p++) - if (isupper((int) *p)) - *p = tolower(*p); - - /* OK, the fqdn is now known. Now it is time to extract only the hostname - * part if we were instructed to do so. - */ - /* TODO: quick and dirty right now: we need to optimize that. We simply - * copy over the buffer and then use the old code. In the long term, that should - * be placed in its own function and probably outside of the net module (at least - * if should no longer reley on syslogd.c's global config-setting variables). - * Note that the old code always removes the local domain. We may want to - * make this in option in the long term. (rgerhards, 2007-09-11) - */ - strcpy((char*)pszHost, (char*)pszHostFQDN); - if( (glbl.GetPreserveFQDN() == 0) - && (p = (uchar*) strchr((char*)pszHost, '.'))) { /* find start of domain name "machine.example.com" */ - strcmp((char*)(p + 1), (char*)glbl.GetLocalDomain()); - if(strcmp((char*)(p + 1), (char*)glbl.GetLocalDomain()) == 0) { - *p = '\0'; /* simply terminate the string */ - } else { - /* now check if we belong to any of the domain names that were specified - * in the -s command line option. If so, remove and we are done. - * TODO: this must go away! -- rgerhards, 2008-04-16 - * For proper modularization, this must be done different, e.g. via a - * "to be stripped" property of *this* object itself. - */ - if(glbl.GetStripDomains() != NULL) { - count=0; - while(glbl.GetStripDomains()[count]) { - if (strcmp((char*)(p + 1), glbl.GetStripDomains()[count]) == 0) { - *p = '\0'; - FINALIZE; /* we are done */ - } - count++; - } - } - /* if we reach this point, we have not found any domain we should strip. Now - * we try and see if the host itself is listed in the -l command line option - * and so should be stripped also. If so, we do it and return. Please note that - * -l list FQDNs, not just the hostname part. If it did just list the hostname, the - * door would be wide-open for all kinds of mixing up of hosts. Because of this, - * you'll see comparison against the full string (pszHost) below. The termination - * still occurs at *p, which points at the first dot after the hostname. - * TODO: this must also go away - see comment above -- rgerhards, 2008-04-16 - */ - if(glbl.GetLocalHosts() != NULL) { - count=0; - while (glbl.GetLocalHosts()[count]) { - if (!strcmp((char*)pszHost, (char*)glbl.GetLocalHosts()[count])) { - *p = '\0'; - break; /* we are done */ - } - count++; - } - } - } - } - -finalize_it: + iRet = dnscacheLookup(f, NULL, fqdn, localName, ip); RETiRet; } @@ -1467,7 +1391,7 @@ finalize_it: */ static rsRetVal HasRestrictions(uchar *pszType, int *bHasRestrictions) { - struct AllowedSenders *pAllowRoot; + struct AllowedSenders *pAllowRoot = NULL; DEFiRet; CHKiRet(setAllowRoot(&pAllowRoot, pszType)); @@ -1577,6 +1501,7 @@ BEGINObjClassExit(net, OBJ_IS_LOADABLE_MODULE) /* CHANGE class also in END MACRO CODESTARTObjClassExit(net) /* release objects we no longer need */ objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); ENDObjClassExit(net) @@ -1589,6 +1514,7 @@ BEGINAbstractObjClassInit(net, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ ENDObjClassInit(net) diff --git a/runtime/net.h b/runtime/net.h index 1b41c81c..b196116b 100644 --- a/runtime/net.h +++ b/runtime/net.h @@ -1,6 +1,6 @@ /* Definitions for network-related stuff. * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -131,7 +131,7 @@ struct permittedPeers_s { /* interfaces */ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ - rsRetVal (*cvthname)(struct sockaddr_storage *f, uchar *pszHost, uchar *pszHostFQDN, uchar *pszIP); + rsRetVal (*cvthname)(struct sockaddr_storage *f, prop_t **localName, prop_t **fqdn, prop_t **ip); /* things to go away after proper modularization */ rsRetVal (*addAllowedSenderLine)(char* pName, uchar** ppRestOfConfLine); void (*PrintAllowedSenders)(int iListToPrint); @@ -156,8 +156,9 @@ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ /* data members - these should go away over time... TODO */ int *pACLAddHostnameOnFail; /* add hostname to acl when DNS resolving has failed */ int *pACLDontResolve; /* add hostname to acl instead of resolving it to IP(s) */ + /* v8 cvthname() signature change -- rgerhards, 2013-01-18 */ ENDinterface(net) -#define netCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ +#define netCURR_IF_VERSION 8 /* increment whenever you change the interface structure! */ /* prototypes */ PROTOTYPEObj(net); diff --git a/runtime/netstrm.c b/runtime/netstrm.c index 58f38280..c046cf52 100644 --- a/runtime/netstrm.c +++ b/runtime/netstrm.c @@ -271,11 +271,11 @@ GetRemoteHName(netstrm_t *pThis, uchar **ppsz) /* get remote IP - slim wrapper for NSD driver function */ static rsRetVal -GetRemoteIP(netstrm_t *pThis, uchar **ppsz) +GetRemoteIP(netstrm_t *pThis, prop_t **ip) { DEFiRet; ISOBJ_TYPE_assert(pThis, netstrm); - iRet = pThis->Drvr.GetRemoteIP(pThis->pDrvrData, ppsz); + iRet = pThis->Drvr.GetRemoteIP(pThis->pDrvrData, ip); RETiRet; } diff --git a/runtime/netstrm.h b/runtime/netstrm.h index ee8d9e59..4ef24229 100644 --- a/runtime/netstrm.h +++ b/runtime/netstrm.h @@ -49,7 +49,7 @@ BEGINinterface(netstrm) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Send)(netstrm_t *pThis, uchar *pBuf, ssize_t *pLenBuf); rsRetVal (*Connect)(netstrm_t *pThis, int family, unsigned char *port, unsigned char *host); rsRetVal (*GetRemoteHName)(netstrm_t *pThis, uchar **pszName); - rsRetVal (*GetRemoteIP)(netstrm_t *pThis, uchar **pszIP); + rsRetVal (*GetRemoteIP)(netstrm_t *pThis, prop_t **ip); rsRetVal (*SetDrvrMode)(netstrm_t *pThis, int iMode); rsRetVal (*SetDrvrAuthMode)(netstrm_t *pThis, uchar*); rsRetVal (*SetDrvrPermPeers)(netstrm_t *pThis, permittedPeers_t*); @@ -72,10 +72,11 @@ BEGINinterface(netstrm) /* name must also be changed in ENDinterface macro! */ /* v4 */ rsRetVal (*EnableKeepAlive)(netstrm_t *pThis); ENDinterface(netstrm) -#define netstrmCURR_IF_VERSION 5 /* increment whenever you change the interface structure! */ +#define netstrmCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ /* interface version 3 added GetRemAddr() * interface version 4 added EnableKeepAlive() -- rgerhards, 2009-06-02 * interface version 5 changed return of CheckConnection from void to rsRetVal -- alorbach, 2012-09-06 + * interface version 6 changed signature of GetRemoteIP() -- rgerhards, 2013-01-21 * */ /* prototypes */ diff --git a/runtime/nsd.h b/runtime/nsd.h index d7d6abbd..aa3662a4 100644 --- a/runtime/nsd.h +++ b/runtime/nsd.h @@ -59,7 +59,7 @@ BEGINinterface(nsd) /* name must also be changed in ENDinterface macro! */ uchar *pLstnPort, uchar *pLstnIP, int iSessMax); rsRetVal (*AcceptConnReq)(nsd_t *pThis, nsd_t **ppThis); rsRetVal (*GetRemoteHName)(nsd_t *pThis, uchar **pszName); - rsRetVal (*GetRemoteIP)(nsd_t *pThis, uchar **pszIP); + rsRetVal (*GetRemoteIP)(nsd_t *pThis, prop_t **ip); rsRetVal (*SetMode)(nsd_t *pThis, int mode); /* sets a driver specific mode - see driver doc for details */ rsRetVal (*SetAuthMode)(nsd_t *pThis, uchar*); /* sets a driver specific mode - see driver doc for details */ rsRetVal (*SetPermPeers)(nsd_t *pThis, permittedPeers_t*); /* sets driver permitted peers for auth needs */ @@ -80,10 +80,11 @@ BEGINinterface(nsd) /* name must also be changed in ENDinterface macro! */ /* v5 */ rsRetVal (*EnableKeepAlive)(nsd_t *pThis); ENDinterface(nsd) -#define nsdCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define nsdCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ /* interface version 4 added GetRemAddr() * interface version 5 added EnableKeepAlive() -- rgerhards, 2009-06-02 * interface version 6 changed return of CheckConnection from void to rsRetVal -- alorbach, 2012-09-06 + * interface version 7 changed signature ofGetRempoteIP() -- rgerhards, 2013-01-21 */ /* interface for the select call */ diff --git a/runtime/nsd_gtls.c b/runtime/nsd_gtls.c index 71eafbd2..6ef4feba 100644 --- a/runtime/nsd_gtls.c +++ b/runtime/nsd_gtls.c @@ -259,9 +259,9 @@ gtlsClientCertCallback(gnutls_session session, static rsRetVal gtlsGetCertInfo(nsd_gtls_t *pThis, cstr_t **ppStr) { - char dn[128]; - uchar lnBuf[256]; - size_t size; + uchar szBufA[1024]; + uchar *szBuf = szBufA; + size_t szBufLen = sizeof(szBufA), tmp; unsigned int algo, bits; time_t expiration_time, activation_time; const gnutls_datum *cert_list; @@ -271,8 +271,6 @@ gtlsGetCertInfo(nsd_gtls_t *pThis, cstr_t **ppStr) int gnuRet; DEFiRet; unsigned iAltName; - size_t szAltNameLen; - char szAltName[1024]; /* this is sufficient for the DNSNAME... */ assert(ppStr != NULL); ISOBJ_TYPE_assert(pThis, nsd_gtls); @@ -281,61 +279,62 @@ gtlsGetCertInfo(nsd_gtls_t *pThis, cstr_t **ppStr) return RS_RET_TLS_CERT_ERR; cert_list = gnutls_certificate_get_peers(pThis->sess, &cert_list_size); - - CHKiRet(rsCStrConstruct(&pStr)); - - snprintf((char*)lnBuf, sizeof(lnBuf), "peer provided %d certificate(s). ", cert_list_size); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); + CHKiRet(rsCStrConstructFromszStrf(&pStr, "peer provided %d certificate(s). ", cert_list_size)); if(cert_list_size > 0) { /* we only print information about the first certificate */ CHKgnutls(gnutls_x509_crt_init(&cert)); CHKgnutls(gnutls_x509_crt_import(cert, &cert_list[0], GNUTLS_X509_FMT_DER)); - CHKiRet(rsCStrAppendStr(pStr, (uchar*)"Certificate 1 info: ")); - expiration_time = gnutls_x509_crt_get_expiration_time(cert); activation_time = gnutls_x509_crt_get_activation_time(cert); - ctime_r(&activation_time, dn); - dn[strlen(dn) - 1] = '\0'; /* strip linefeed */ - snprintf((char*)lnBuf, sizeof(lnBuf), "certificate valid from %s ", dn); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); - - ctime_r(&expiration_time, dn); - dn[strlen(dn) - 1] = '\0'; /* strip linefeed */ - snprintf((char*)lnBuf, sizeof(lnBuf), "to %s; ", dn); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); + ctime_r(&activation_time, szBuf); + szBuf[strlen(szBuf) - 1] = '\0'; /* strip linefeed */ + CHKiRet(rsCStrAppendStrf(pStr, (uchar*)"Certificate 1 info: " + "certificate valid from %s ", szBuf)); + ctime_r(&expiration_time, szBuf); + szBuf[strlen(szBuf) - 1] = '\0'; /* strip linefeed */ + CHKiRet(rsCStrAppendStrf(pStr, "to %s; ", szBuf)); /* Extract some of the public key algorithm's parameters */ algo = gnutls_x509_crt_get_pk_algorithm(cert, &bits); - - snprintf((char*)lnBuf, sizeof(lnBuf), "Certificate public key: %s; ", - gnutls_pk_algorithm_get_name(algo)); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); + CHKiRet(rsCStrAppendStrf(pStr, "Certificate public key: %s; ", + gnutls_pk_algorithm_get_name(algo))); /* names */ - size = sizeof(dn); - gnutls_x509_crt_get_dn(cert, dn, &size); - snprintf((char*)lnBuf, sizeof(lnBuf), "DN: %s; ", dn); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); - - size = sizeof(dn); - gnutls_x509_crt_get_issuer_dn(cert, dn, &size); - snprintf((char*)lnBuf, sizeof(lnBuf), "Issuer DN: %s; ", dn); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); + tmp = szBufLen; + if(gnutls_x509_crt_get_dn(cert, szBuf, &tmp) + == GNUTLS_E_SHORT_MEMORY_BUFFER) { + szBufLen = tmp; + szBuf = malloc(tmp); + gnutls_x509_crt_get_dn(cert, szBuf, &tmp); + } + CHKiRet(rsCStrAppendStrf(pStr, "DN: %s; ", szBuf)); + + tmp = szBufLen; + if(gnutls_x509_crt_get_issuer_dn(cert, szBuf, &tmp) + == GNUTLS_E_SHORT_MEMORY_BUFFER) { + szBufLen = tmp; + szBuf = realloc((szBuf == szBufA) ? NULL : szBuf, tmp); + gnutls_x509_crt_get_issuer_dn(cert, szBuf, &tmp); + } + CHKiRet(rsCStrAppendStrf(pStr, "Issuer DN: %s; ", szBuf)); /* dNSName alt name */ iAltName = 0; while(1) { /* loop broken below */ - szAltNameLen = sizeof(szAltName); + tmp = szBufLen; gnuRet = gnutls_x509_crt_get_subject_alt_name(cert, iAltName, - szAltName, &szAltNameLen, NULL); - if(gnuRet < 0) + szBuf, &tmp, NULL); + if(gnuRet == GNUTLS_E_SHORT_MEMORY_BUFFER) { + szBufLen = tmp; + szBuf = realloc((szBuf == szBufA) ? NULL : szBuf, tmp); + continue; + } else if(gnuRet < 0) break; else if(gnuRet == GNUTLS_SAN_DNSNAME) { /* we found it! */ - snprintf((char*)lnBuf, sizeof(lnBuf), "SAN:DNSname: %s; ", szAltName); - CHKiRet(rsCStrAppendStr(pStr, lnBuf)); + CHKiRet(rsCStrAppendStrf(pStr, "SAN:DNSname: %s; ", szBuf)); /* do NOT break, because there may be multiple dNSName's! */ } ++iAltName; @@ -352,6 +351,8 @@ finalize_it: if(pStr != NULL) rsCStrDestruct(&pStr); } + if(szBuf != szBufA) + free(szBuf); RETiRet; } @@ -1357,16 +1358,14 @@ GetRemAddr(nsd_t *pNsd, struct sockaddr_storage **ppAddr) } -/* get the remote host's IP address. The returned string must be freed by the - * caller. -- rgerhards, 2008-04-25 - */ +/* get the remote host's IP address. Caller must Destruct the object. */ static rsRetVal -GetRemoteIP(nsd_t *pNsd, uchar **ppszIP) +GetRemoteIP(nsd_t *pNsd, prop_t **ip) { DEFiRet; nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_gtls); - iRet = nsd_ptcp.GetRemoteIP(pThis->pTcp, ppszIP); + iRet = nsd_ptcp.GetRemoteIP(pThis->pTcp, ip); RETiRet; } diff --git a/runtime/nsd_ptcp.c b/runtime/nsd_ptcp.c index 12f891ea..f889a00e 100644 --- a/runtime/nsd_ptcp.c +++ b/runtime/nsd_ptcp.c @@ -2,7 +2,7 @@ * * An implementation of the nsd interface for plain tcp sockets. * - * Copyright 2007, 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -50,6 +50,7 @@ #include "nsdsel_ptcp.h" #include "nsdpoll_ptcp.h" #include "nsd_ptcp.h" +#include "prop.h" #include "dnscache.h" MODULE_TYPE_LIB @@ -62,6 +63,7 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(net) DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) +DEFobjCurrIf(prop) /* a few deinit helpers */ @@ -87,10 +89,9 @@ ENDobjConstruct(nsd_ptcp) BEGINobjDestruct(nsd_ptcp) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(nsd_ptcp) sockClose(&pThis->sock); - if(pThis->pRemHostIP != NULL) - free(pThis->pRemHostIP); - if(pThis->pRemHostName != NULL) - free(pThis->pRemHostName); + if(pThis->remoteIP != NULL) + prop.Destruct(&pThis->remoteIP); + free(pThis->pRemHostName); ENDobjDestruct(nsd_ptcp) @@ -251,32 +252,22 @@ Abort(nsd_t *pNsd) static rsRetVal FillRemHost(nsd_ptcp_t *pThis, struct sockaddr_storage *pAddr) { - uchar szIP[NI_MAXHOST] = ""; - uchar szHname[NI_MAXHOST] = ""; - size_t len; + prop_t *fqdn; DEFiRet; ISOBJ_TYPE_assert(pThis, nsd_ptcp); assert(pAddr != NULL); - CHKiRet(dnscacheLookup(pAddr, szHname, szIP)); + CHKiRet(dnscacheLookup(pAddr, &fqdn, NULL, NULL, &pThis->remoteIP)); /* We now have the names, so now let's allocate memory and store them permanently. * (side note: we may hold on to these values for quite a while, thus we trim their * memory consumption) */ - len = strlen((char*)szIP) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostIP = MALLOC(len)) == NULL) + if((pThis->pRemHostName = MALLOC(prop.GetStringLen(fqdn)+1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - memcpy(pThis->pRemHostIP, szIP, len); - - len = strlen((char*)szHname) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostName = MALLOC(len)) == NULL) { - free(pThis->pRemHostIP); /* prevent leak */ - pThis->pRemHostIP = NULL; - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - memcpy(pThis->pRemHostName, szHname, len); + memcpy(pThis->pRemHostName, propGetSzStr(fqdn), prop.GetStringLen(fqdn)+1); + prop.Destruct(&fqdn); finalize_it: RETiRet; @@ -460,7 +451,9 @@ LstnInit(netstrms_t *pNS, void *pUsr, rsRetVal(*fAddLstn)(void*,netstrm_t*), #endif ) { /* TODO: check if *we* bound the socket - else we *have* an error! */ - dbgprintf("error %d while binding tcp socket\n", errno); + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("error %d while binding tcp socket: %s\n", errno, errStr); close(sock); sock = -1; continue; @@ -717,21 +710,16 @@ finalize_it: } -/* get the remote host's IP address. The returned string must be freed by the - * caller. - * rgerhards, 2008-04-24 +/* get the remote host's IP address. Caller must Destruct the object. */ static rsRetVal -GetRemoteIP(nsd_t *pNsd, uchar **ppszIP) +GetRemoteIP(nsd_t *pNsd, prop_t **ip) { DEFiRet; nsd_ptcp_t *pThis = (nsd_ptcp_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_ptcp); - assert(ppszIP != NULL); - - CHKmalloc(*ppszIP = (uchar*)strdup(pThis->pRemHostIP == NULL ? "" : (char*) pThis->pRemHostIP)); - -finalize_it: + prop.AddRef(pThis->remoteIP); + *ip = pThis->remoteIP; RETiRet; } @@ -777,6 +765,7 @@ CODESTARTObjClassExit(nsd_ptcp) /* release objects we no longer need */ objRelease(net, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(netstrm, DONT_LOAD_LIB); objRelease(netstrms, LM_NETSTRMS_FILENAME); @@ -791,6 +780,7 @@ BEGINObjClassInit(nsd_ptcp, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(net, CORE_COMPONENT)); CHKiRet(objUse(netstrms, LM_NETSTRMS_FILENAME)); CHKiRet(objUse(netstrm, DONT_LOAD_LIB)); diff --git a/runtime/nsd_ptcp.h b/runtime/nsd_ptcp.h index a1bcd646..ed6b8565 100644 --- a/runtime/nsd_ptcp.h +++ b/runtime/nsd_ptcp.h @@ -30,7 +30,7 @@ typedef nsd_if_t nsd_ptcp_if_t; /* we just *implement* this interface */ /* the nsd_ptcp object */ struct nsd_ptcp_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - uchar *pRemHostIP; /**< IP address of remote peer (currently used in server mode, only) */ + prop_t *remoteIP; /**< IP address of remote peer (currently used in server mode, only) */ uchar *pRemHostName; /**< host name of remote peer (currently used in server mode, only) */ struct sockaddr_storage remAddr; /**< remote addr as sockaddr - used for legacy ACL code */ int sock; /**< the socket we use for regular, single-socket, operations */ diff --git a/runtime/obj-types.h b/runtime/obj-types.h index da27a391..30a6a2c0 100644 --- a/runtime/obj-types.h +++ b/runtime/obj-types.h @@ -282,14 +282,12 @@ rsRetVal objName##ClassExit(void) \ rsRetVal OBJ##Destruct(OBJ##_t __attribute__((unused)) **ppThis) \ { \ DEFiRet; \ - int iCancelStateSave; \ OBJ##_t *pThis; #define CODESTARTobjDestruct(OBJ) \ ASSERT(ppThis != NULL); \ pThis = *ppThis; \ - ISOBJ_TYPE_assert(pThis, OBJ); \ - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + ISOBJ_TYPE_assert(pThis, OBJ); /* note: there was a long-time bug in the macro below that lead to *ppThis = NULL * only when the object was actually destructed. I discovered this issue during @@ -309,7 +307,6 @@ rsRetVal objName##ClassExit(void) \ free(pThis); \ } \ *ppThis = NULL; \ - pthread_setcancelstate(iCancelStateSave, NULL); \ RETiRet; \ } diff --git a/runtime/obj.c b/runtime/obj.c index eb151b67..63f1f38c 100644 --- a/runtime/obj.c +++ b/runtime/obj.c @@ -604,7 +604,7 @@ finalize_it: /* Deserialize a single property. Pointer must be positioned at begin of line. Whole line * up until the \n is read. */ -static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) +rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) { DEFiRet; number_t i; @@ -665,7 +665,7 @@ static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) if(c != '\n') ABORT_FINALIZE(RS_RET_INVALID_PROPFRAME); finalize_it: - if(Debug && iRet != RS_RET_OK) { + if(Debug && iRet != RS_RET_OK && iRet != RS_RET_NO_PROPLINE) { strm.GetCurrOffset(pStrm, &offs); dbgprintf("error %d deserializing property name, offset %lld, step %d\n", iRet, offs, step); @@ -767,21 +767,20 @@ finalize_it: * of the trailer. Header must already have been processed. * rgerhards, 2008-01-11 */ -static rsRetVal objDeserializeProperties(obj_t *pObj, objInfo_t *pObjInfo, strm_t *pStrm) +static rsRetVal objDeserializeProperties(obj_t *pObj, rsRetVal (*objSetProperty)(), strm_t *pStrm) { DEFiRet; var_t *pVar = NULL; ISOBJ_assert(pObj); ISOBJ_TYPE_assert(pStrm, strm); - ASSERT(pObjInfo != NULL); CHKiRet(var.Construct(&pVar)); CHKiRet(var.ConstructFinalize(pVar)); iRet = objDeserializeProperty(pVar, pStrm); while(iRet == RS_RET_OK) { - CHKiRet(pObjInfo->objMethods[objMethod_SETPROPERTY](pObj, pVar)); + CHKiRet(objSetProperty(pObj, pVar)); /* re-init var object - TODO: method of var! */ rsCStrDestruct(&pVar->pcsName); /* no longer needed */ if(pVar->varType == VARTYPE_STR) { @@ -848,7 +847,7 @@ Deserialize(void *ppObj, uchar *pszTypeExpected, strm_t *pStrm, rsRetVal (*fFixu CHKiRet(pObjInfo->objMethods[objMethod_CONSTRUCT](&pObj)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); /* check if we need to call a fixup function that modifies the object * before it is finalized. -- rgerhards, 2008-01-13 @@ -873,6 +872,104 @@ finalize_it: } +/* De-Serialize an object, with known constructur and destructor. Params like Deserialize(). + * rgerhards, 2012-11-03 + */ +rsRetVal +objDeserializeWithMethods(void *ppObj, uchar *pszTypeExpected, int lenTypeExpected, strm_t *pStrm, rsRetVal (*fFixup)(obj_t*,void*), void *pUsr, rsRetVal (*objConstruct)(), rsRetVal (*objConstructFinalize)(), rsRetVal (*objDeserialize)()) +{ + DEFiRet; + rsRetVal iRetLocal; + obj_t *pObj = NULL; + int oVers = 0; /* keep compiler happy, but it is totally useless but takes up some execution time... */ + cstr_t *pstrID = NULL; + + assert(ppObj != NULL); + assert(pszTypeExpected != NULL); + ISOBJ_TYPE_assert(pStrm, strm); + + /* we de-serialize the header. if all goes well, we are happy. However, if + * we experience a problem, we try to recover. We do this by skipping to + * the next object header. This is defined via the line-start cookies. In + * worst case, we exhaust the queue, but then we receive EOF return state, + * from objDeserializeTryRecover(), what will cause us to ultimately give up. + * rgerhards, 2008-07-08 + */ + do { + iRetLocal = objDeserializeHeader((uchar*) "Obj", &pstrID, &oVers, pStrm); + if(iRetLocal != RS_RET_OK) { + dbgprintf("objDeserialize error %d during header processing - " + "trying to recover\n", iRetLocal); + CHKiRet(objDeserializeTryRecover(pStrm)); + } + } while(iRetLocal != RS_RET_OK); + + if(rsCStrSzStrCmp(pstrID, pszTypeExpected, lenTypeExpected)) + ABORT_FINALIZE(RS_RET_INVALID_OID); + + CHKiRet(objConstruct(&pObj)); + + /* we got the object, now we need to fill the properties */ + CHKiRet(objDeserialize(pObj, pStrm)); + CHKiRet(objDeserializeTrailer(pStrm)); /* do trailer checks */ + + /* check if we need to call a fixup function that modifies the object + * before it is finalized. -- rgerhards, 2008-01-13 + */ + if(fFixup != NULL) + CHKiRet(fFixup(pObj, pUsr)); + + /* we have a valid object, let's finalize our work and return */ + if(objConstructFinalize != NULL) { + CHKiRet(objConstructFinalize(pObj)); + } + + *((obj_t**) ppObj) = pObj; + +finalize_it: + if(iRet != RS_RET_OK && pObj != NULL) + free(pObj); /* TODO: check if we can call destructor 2008-01-13 rger */ + + if(pstrID != NULL) + rsCStrDestruct(&pstrID); + + RETiRet; +} + +/* This is a dummy deserializer, to be used for the delete queue reader + * specifically. This is kind of a hack, but also to be replace (hopefully) soon + * by totally different code. So let's make it as simple as possible... + * rgerhards, 2012-11-06 + */ +rsRetVal +objDeserializeDummy(obj_t __attribute__((unused)) *pObj, strm_t *pStrm) +{ + DEFiRet; + var_t *pVar = NULL; + + CHKiRet(var.Construct(&pVar)); + CHKiRet(var.ConstructFinalize(pVar)); + + iRet = objDeserializeProperty(pVar, pStrm); + while(iRet == RS_RET_OK) { + /* this loop does actually NOGHTING but read the file... */ + /* re-init var object - TODO: method of var! */ + rsCStrDestruct(&pVar->pcsName); /* no longer needed */ + if(pVar->varType == VARTYPE_STR) { + if(pVar->val.pStr != NULL) + rsCStrDestruct(&pVar->val.pStr); + } + iRet = objDeserializeProperty(pVar, pStrm); + } +finalize_it: + if(iRet == RS_RET_NO_PROPLINE) + iRet = RS_RET_OK; /* NO_PROPLINE is OK and a kind of EOF! */ + if(pVar != NULL) + var.Destruct(&pVar); + RETiRet; +} + + /* De-Serialize an object, but treat it as property bag. * rgerhards, 2008-01-11 */ @@ -909,7 +1006,7 @@ objDeserializeObjAsPropBag(obj_t *pObj, strm_t *pStrm) CHKiRet(FindObjInfo(pstrID, &pObjInfo)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); finalize_it: if(pstrID != NULL) @@ -961,7 +1058,7 @@ DeserializePropBag(obj_t *pObj, strm_t *pStrm) CHKiRet(FindObjInfo(pstrID, &pObjInfo)); /* we got the object, now we need to fill the properties */ - CHKiRet(objDeserializeProperties(pObj, pObjInfo, pStrm)); + CHKiRet(objDeserializeProperties(pObj, pObjInfo->objMethods[objMethod_SETPROPERTY], pStrm)); finalize_it: if(pstrID != NULL) diff --git a/runtime/obj.h b/runtime/obj.h index 32f7ef09..27d32b7a 100644 --- a/runtime/obj.h +++ b/runtime/obj.h @@ -83,10 +83,7 @@ ((obj_t*) (pThis))->pObjInfo = pObjInfoOBJ; \ ((obj_t*) (pThis))->pszName = NULL #endif -#define objDestruct(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_DESTRUCT])(&pThis) #define objSerialize(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_SERIALIZE]) -#define objGetSeverity(pThis, piSever) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_GETSEVERITY])(pThis, piSever) -#define objDebugPrint(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_DEBUGPRINT])(pThis) #define OBJSetMethodHandler(methodID, pHdlr) \ CHKiRet(obj.InfoSetMethod(pObjInfoOBJ, methodID, (rsRetVal (*)(void*)) pHdlr)) @@ -121,6 +118,9 @@ ENDinterface(obj) rsRetVal objGetObjInterface(obj_if_t *pIf); PROTOTYPEObjClassInit(obj); PROTOTYPEObjClassExit(obj); +rsRetVal objDeserializeWithMethods(void *ppObj, uchar *pszTypeExpected, int lenTypeExpected, strm_t *pStrm, rsRetVal (*fFixup)(obj_t*,void*), void *pUsr, rsRetVal (*objConstruct)(), rsRetVal (*objConstructFinalize)(), rsRetVal (*objDeserialize)()); +rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm); +rsRetVal objDeserializeDummy(obj_t *pObj, strm_t *pStrm); /* the following definition is only for "friends" */ diff --git a/runtime/objomsr.c b/runtime/objomsr.c index 9cf3781b..e63eb681 100644 --- a/runtime/objomsr.c +++ b/runtime/objomsr.c @@ -42,9 +42,7 @@ rsRetVal OMSRdestruct(omodStringRequest_t *pThis) /* free the strings */ if(pThis->ppTplName != NULL) { for(i = 0 ; i < pThis->iNumEntries ; ++i) { - if(pThis->ppTplName[i] != NULL) { - free(pThis->ppTplName[i]); - } + free(pThis->ppTplName[i]); } free(pThis->ppTplName); } diff --git a/runtime/parser.c b/runtime/parser.c index 645ea0f4..74b28f4c 100644 --- a/runtime/parser.c +++ b/runtime/parser.c @@ -143,6 +143,14 @@ finalize_it: RETiRet; } +void +printParserList(parserList_t *pList) +{ + while(pList != NULL) { + dbgprintf("parser: %s\n", pList->pParser->pName); + pList = pList->pNext; + } +} /* find a parser based on the provided name */ static rsRetVal @@ -354,11 +362,10 @@ SanitizeMsg(msg_t *pMsg) */ int bNeedSanitize = 0; for(iSrc = 0 ; iSrc < lenMsg ; iSrc++) { - if(iscntrl(pszMsg[iSrc])) { + if(pszMsg[iSrc] < 32) { if(bSpaceLFOnRcv && pszMsg[iSrc] == '\n') pszMsg[iSrc] = ' '; - else - if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { + else if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { bNeedSanitize = 1; if (!bSpaceLFOnRcv) break; @@ -375,7 +382,9 @@ SanitizeMsg(msg_t *pMsg) FINALIZE; } - /* now copy over the message and sanitize it */ + /* now copy over the message and sanitize it. Note that up to iSrc-1 there was + * obviously no need to sanitize, so we can go over that quickly... + */ iMaxLine = glbl.GetMaxLine(); maxDest = lenMsg * 4; /* message can grow at most four-fold */ if(maxDest > iMaxLine) @@ -384,9 +393,13 @@ SanitizeMsg(msg_t *pMsg) pDst = szSanBuf; else CHKmalloc(pDst = MALLOC(sizeof(uchar) * (iMaxLine + 1))); - iSrc = iDst = 0; + if(iSrc > 0) { + iSrc--; /* go back to where everything is OK */ + memcpy(pDst, pszMsg, iSrc); /* fast copy known good */ + } + iDst = iSrc; while(iSrc < lenMsg && iDst < maxDest - 3) { /* leave some space if last char must be escaped */ - if(iscntrl((int) pszMsg[iSrc]) && (pszMsg[iSrc] != '\t' || bEscapeTab)) { + if((pszMsg[iSrc] < 32) && (pszMsg[iSrc] != '\t' || bEscapeTab)) { /* note: \0 must always be escaped, the rest of the code currently * can not handle it! -- rgerhards, 2009-08-26 */ diff --git a/runtime/parser.h b/runtime/parser.h index f214ba0c..87a6269e 100644 --- a/runtime/parser.h +++ b/runtime/parser.h @@ -62,6 +62,7 @@ BEGINinterface(parser) /* name must also be changed in ENDinterface macro! */ ENDinterface(parser) #define parserCURR_IF_VERSION 1 /* increment whenever you change the interface above! */ +void printParserList(parserList_t *pList); /* prototypes */ PROTOTYPEObj(parser); diff --git a/runtime/prop.c b/runtime/prop.c index 9d5927fd..cb89fac0 100644 --- a/runtime/prop.c +++ b/runtime/prop.c @@ -100,7 +100,7 @@ static int GetStringLen(prop_t *pThis) /* get string */ -static rsRetVal GetString(prop_t *pThis, uchar **ppsz, int *plen) +rsRetVal GetString(prop_t *pThis, uchar **ppsz, int *plen) { BEGINfunc ISOBJ_TYPE_assert(pThis, prop); diff --git a/runtime/prop.h b/runtime/prop.h index 40a35f9b..c7564e6b 100644 --- a/runtime/prop.h +++ b/runtime/prop.h @@ -52,6 +52,13 @@ ENDinterface(prop) #define propCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +/* get classic c-style string */ +static inline uchar * +propGetSzStr(prop_t *pThis) +{ + return(pThis->len < CONF_PROP_BUFSIZE) ? pThis->szVal.sz : pThis->szVal.psz; +} + /* prototypes */ PROTOTYPEObj(prop); diff --git a/runtime/queue.c b/runtime/queue.c index 09ba7e67..abe2be06 100644 --- a/runtime/queue.c +++ b/runtime/queue.c @@ -59,7 +59,6 @@ #include "datetime.h" #include "unicode-helper.h" #include "statsobj.h" -#include "msg.h" /* TODO: remove once we remove MsgAddRef() call */ #ifdef OS_SOLARIS # include <sched.h> @@ -74,7 +73,7 @@ DEFobjCurrIf(datetime) DEFobjCurrIf(statsobj) /* forward-definitions */ -static inline rsRetVal doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr); +static inline rsRetVal doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg); static rsRetVal qqueueChkPersist(qqueue_t *pThis, int nUpdates); static rsRetVal RateLimiter(qqueue_t *pThis); static int qqueueChkStopWrkrDA(qqueue_t *pThis); @@ -83,7 +82,7 @@ static rsRetVal ConsumerDA(qqueue_t *pThis, wti_t *pWti); static rsRetVal batchProcessed(qqueue_t *pThis, wti_t *pWti); static rsRetVal qqueueMultiEnqObjNonDirect(qqueue_t *pThis, multi_submit_t *pMultiSub); static rsRetVal qqueueMultiEnqObjDirect(qqueue_t *pThis, multi_submit_t *pMultiSub); -static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr); +static rsRetVal qAddDirect(qqueue_t *pThis, msg_t *pMsg); static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qConstructDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis); @@ -127,11 +126,11 @@ static struct cnfparamblk pblk = }; /* debug aid */ -static void displayBatchState(batch_t *pBatch) +static inline void displayBatchState(batch_t *pBatch) { int i; for(i = 0 ; i < pBatch->nElem ; ++i) { - DBGPRINTF("displayBatchState %p[%d]: %d\n", pBatch, i, pBatch->pElem[i].state); + DBGPRINTF("displayBatchState %p[%d]: %d\n", pBatch, i, pBatch->eltState[i]); } } @@ -232,12 +231,19 @@ getQueueTypeName(queueType_t t) switch(t) { case QUEUETYPE_FIXED_ARRAY: r = "FixedArray"; + break; case QUEUETYPE_LINKEDLIST: r = "LinkedList"; + break; case QUEUETYPE_DISK: r = "Disk"; + break; case QUEUETYPE_DIRECT: r = "Direct"; + break; + default: + r = "invalid/unknown queue mode"; + break; } return r; } @@ -308,16 +314,16 @@ getLogicalQueueSize(qqueue_t *pThis) */ static inline void queueDrain(qqueue_t *pThis) { - void *pUsr; + msg_t *pMsg; ASSERT(pThis != NULL); BEGINfunc DBGOPRINT((obj_t*) pThis, "queue (type %d) will lose %d messages, destroying...\n", pThis->qType, pThis->iQueueSize); /* iQueueSize is not decremented by qDel(), so we need to do it ourselves */ while(ATOMIC_DEC_AND_FETCH(&pThis->iQueueSize, &pThis->mutQueueSize) > 0) { - pThis->qDeq(pThis, &pUsr); - if(pUsr != NULL) { - objDestruct(pUsr); + pThis->qDeq(pThis, &pMsg); + if(pMsg != NULL) { + msgDestruct(&pMsg); } pThis->qDel(pThis); } @@ -407,7 +413,7 @@ StartDA(qqueue_t *pThis) */ pThis->pqDA->pqParent = pThis; - CHKiRet(qqueueSetpUsr(pThis->pqDA, pThis->pUsr)); + CHKiRet(qqueueSetpAction(pThis->pqDA, pThis->pAction)); CHKiRet(qqueueSetsizeOnDiskMax(pThis->pqDA, pThis->sizeOnDiskMax)); CHKiRet(qqueueSetiDeqSlowdown(pThis->pqDA, pThis->iDeqSlowdown)); CHKiRet(qqueueSetMaxFileSize(pThis->pqDA, pThis->iMaxFileSize)); @@ -542,7 +548,7 @@ static rsRetVal qDestructFixedArray(qqueue_t *pThis) } -static rsRetVal qAddFixedArray(qqueue_t *pThis, void* in) +static rsRetVal qAddFixedArray(qqueue_t *pThis, msg_t* in) { DEFiRet; @@ -556,7 +562,7 @@ static rsRetVal qAddFixedArray(qqueue_t *pThis, void* in) } -static rsRetVal qDeqFixedArray(qqueue_t *pThis, void **out) +static rsRetVal qDeqFixedArray(qqueue_t *pThis, msg_t **out) { DEFiRet; @@ -617,7 +623,7 @@ static rsRetVal qDestructLinkedList(qqueue_t __attribute__((unused)) *pThis) RETiRet; } -static rsRetVal qAddLinkedList(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddLinkedList(qqueue_t *pThis, msg_t* pMsg) { qLinkedList_t *pEntry; DEFiRet; @@ -625,7 +631,7 @@ static rsRetVal qAddLinkedList(qqueue_t *pThis, void* pUsr) CHKmalloc((pEntry = (qLinkedList_t*) MALLOC(sizeof(qLinkedList_t)))); pEntry->pNext = NULL; - pEntry->pUsr = pUsr; + pEntry->pMsg = pMsg; if(pThis->tVars.linklist.pDelRoot == NULL) { pThis->tVars.linklist.pDelRoot = pThis->tVars.linklist.pDeqRoot = pThis->tVars.linklist.pLast = pEntry; @@ -643,14 +649,13 @@ finalize_it: } -static rsRetVal qDeqLinkedList(qqueue_t *pThis, obj_t **ppUsr) +static rsRetVal qDeqLinkedList(qqueue_t *pThis, msg_t **ppMsg) { qLinkedList_t *pEntry; DEFiRet; pEntry = pThis->tVars.linklist.pDeqRoot; - ISOBJ_TYPE_assert(pEntry->pUsr, msg); - *ppUsr = pEntry->pUsr; + *ppMsg = pEntry->pMsg; pThis->tVars.linklist.pDeqRoot = pEntry->pNext; RETiRet; @@ -740,18 +745,12 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) { DEFiRet; strm_t *psQIF = NULL; - uchar pszQIFNam[MAXFNAME]; - size_t lenQIFNam; struct stat stat_buf; ISOBJ_TYPE_assert(pThis, qqueue); - /* Construct file name */ - lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), "%s/%s.qi", - (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); - /* check if the file exists */ - if(stat((char*) pszQIFNam, &stat_buf) == -1) { + if(stat((char*) pThis->pszQIFNam, &stat_buf) == -1) { if(errno == ENOENT) { DBGOPRINT((obj_t*) pThis, "clean startup, no .qi file found\n"); ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); @@ -766,7 +765,7 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) CHKiRet(strm.Construct(&psQIF)); CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_READ)); CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.SetFName(psQIF, pThis->pszQIFNam, pThis->lenQIFNam)); CHKiRet(strm.ConstructFinalize(psQIF)); /* first, we try to read the property bag for ourselfs */ @@ -778,9 +777,7 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) CHKiRet(obj.Deserialize(&pThis->tVars.disk.pReadDel, (uchar*) "strm", psQIF, (rsRetVal(*)(obj_t*,void*))qqueueLoadPersStrmInfoFixup, pThis)); - /* create a duplicate for the read "pointer". - */ - + /* create a duplicate for the read "pointer". */ CHKiRet(strm.Dup(pThis->tVars.disk.pReadDel, &pThis->tVars.disk.pReadDeq)); CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pReadDeq, 0)); /* deq must NOT delete the files! */ CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pReadDeq)); @@ -799,7 +796,7 @@ finalize_it: strm.Destruct(&psQIF); if(iRet != RS_RET_OK) { - DBGOPRINT((obj_t*) pThis, "error %d reading .qi file - can not read persisted info (if any)\n", + DBGOPRINT((obj_t*) pThis, "state %d reading .qi file - can not read persisted info (if any)\n", iRet); } @@ -880,7 +877,8 @@ static rsRetVal qDestructDisk(qqueue_t *pThis) DEFiRet; ASSERT(pThis != NULL); - + + free(pThis->pszQIFNam); if(pThis->tVars.disk.pWrite != NULL) strm.Destruct(&pThis->tVars.disk.pWrite); if(pThis->tVars.disk.pReadDeq != NULL) @@ -891,7 +889,7 @@ static rsRetVal qDestructDisk(qqueue_t *pThis) RETiRet; } -static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddDisk(qqueue_t *pThis, msg_t* pMsg) { DEFiRet; number_t nWriteCount; @@ -899,7 +897,7 @@ static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) ASSERT(pThis != NULL); CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, &nWriteCount)); - CHKiRet((objSerialize(pUsr))(pUsr, pThis->tVars.disk.pWrite)); + CHKiRet((objSerialize(pMsg))(pMsg, pThis->tVars.disk.pWrite)); CHKiRet(strm.Flush(pThis->tVars.disk.pWrite)); CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, NULL)); /* no more counting for now... */ @@ -909,7 +907,7 @@ static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) * the in-memory representation. The instance will be re-created upon * dequeue. -- rgerhards, 2008-07-09 */ - objDestruct(pUsr); + msgDestruct(&pMsg); DBGOPRINT((obj_t*) pThis, "write wrote %lld octets to disk, queue disk size now %lld octets, EnqOnly:%d\n", nWriteCount, pThis->tVars.disk.sizeOnDisk, pThis->bEnqOnly); @@ -919,43 +917,11 @@ finalize_it: } -static rsRetVal qDeqDisk(qqueue_t *pThis, void **ppUsr) +static rsRetVal qDeqDisk(qqueue_t *pThis, msg_t **ppMsg) { DEFiRet; - iRet = obj.Deserialize(ppUsr, (uchar*) "msg", pThis->tVars.disk.pReadDeq, NULL, NULL); - RETiRet; -} - - -static rsRetVal qDelDisk(qqueue_t *pThis) -{ - obj_t *pDummyObj; /* we need to deserialize it... */ - DEFiRet; - - int64 offsIn; - int64 offsOut; - - CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pReadDel, &offsIn)); - CHKiRet(obj.Deserialize(&pDummyObj, (uchar*) "msg", pThis->tVars.disk.pReadDel, NULL, NULL)); - objDestruct(pDummyObj); - CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pReadDel, &offsOut)); - - /* This time it is a bit tricky: we free disk space only upon file deletion. So we need - * to keep track of what we have read until we get an out-offset that is lower than the - * in-offset (which indicates file change). Then, we can subtract the whole thing from - * the on-disk size. -- rgerhards, 2008-01-30 - */ - if(offsIn < offsOut) { - pThis->tVars.disk.bytesRead += offsOut - offsIn; - } else { - pThis->tVars.disk.sizeOnDisk -= pThis->tVars.disk.bytesRead; - pThis->tVars.disk.bytesRead = offsOut; - DBGOPRINT((obj_t*) pThis, "a file has been deleted, now %lld octets disk space used\n", pThis->tVars.disk.sizeOnDisk); - /* awake possibly waiting enq process */ - pthread_cond_signal(&pThis->notFull); /* we hold the mutex while we are in here! */ - } - -finalize_it: + iRet = objDeserializeWithMethods(ppMsg, (uchar*) "msg", 3, pThis->tVars.disk.pReadDeq, NULL, + NULL, msgConstructForDeserializer, NULL, MsgDeserialize); RETiRet; } @@ -972,10 +938,12 @@ static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis) return RS_RET_OK; } -static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr) +static rsRetVal qAddDirect(qqueue_t *pThis, msg_t* pMsg) { batch_t singleBatch; batch_obj_t batchObj; + batch_state_t batchState = BATCH_STATE_RDY; + sbool active = 1; int i; DEFiRet; @@ -992,17 +960,17 @@ static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr) */ memset(&batchObj, 0, sizeof(batch_obj_t)); memset(&singleBatch, 0, sizeof(batch_t)); - batchObj.state = BATCH_STATE_RDY; - batchObj.pUsrp = (obj_t*) pUsr; - batchObj.bFilterOK = 1; + batchObj.pMsg = pMsg; singleBatch.nElem = 1; /* there always is only one in direct mode */ singleBatch.pElem = &batchObj; - iRet = pThis->pConsumer(pThis->pUsr, &singleBatch, &pThis->bShutdownImmediate); + singleBatch.eltState = &batchState; + singleBatch.active = &active; + iRet = pThis->pConsumer(pThis->pAction, &singleBatch, &pThis->bShutdownImmediate); /* delete the batch string params: TODO: create its own "class" for this */ for(i = 0 ; i < CONF_OMOD_NUMSTRINGS_MAXSIZE ; ++i) { free(batchObj.staticActStrings[i]); } - objDestruct(pUsr); + msgDestruct(&pMsg); RETiRet; } @@ -1024,7 +992,7 @@ rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch) * We use our knowledge about the batch_t structure below, but without that, we * pay a too-large performance toll... -- rgerhards, 2009-04-22 */ - iRet = pThis->pConsumer(pThis->pUsr, pBatch, &pThis->bShutdownImmediate); + iRet = pThis->pConsumer(pThis->pAction, pBatch, NULL); RETiRet; } @@ -1045,17 +1013,17 @@ static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis) * things truely different. -- rgerhards, 2008-02-12 */ static rsRetVal -qqueueAdd(qqueue_t *pThis, void *pUsr) +qqueueAdd(qqueue_t *pThis, msg_t *pMsg) { DEFiRet; ASSERT(pThis != NULL); - CHKiRet(pThis->qAdd(pThis, pUsr)); + CHKiRet(pThis->qAdd(pThis, pMsg)); if(pThis->qType != QUEUETYPE_DIRECT) { ATOMIC_INC(&pThis->iQueueSize, &pThis->mutQueueSize); - DBGOPRINT((obj_t*) pThis, "entry added, size now log %d, phys %d entries\n", + DBGOPRINT((obj_t*) pThis, "qqueueAdd: entry added, size now log %d, phys %d entries\n", getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); } @@ -1067,7 +1035,7 @@ finalize_it: /* generic code to dequeue a queue entry */ static rsRetVal -qqueueDeq(qqueue_t *pThis, void **ppUsr) +qqueueDeq(qqueue_t *pThis, msg_t **ppMsg) { DEFiRet; @@ -1078,7 +1046,7 @@ qqueueDeq(qqueue_t *pThis, void **ppUsr) * If we decrement, however, we may lose a message. But that is better than * losing the whole process because it loops... -- rgerhards, 2008-01-03 */ - iRet = pThis->qDeq(pThis, ppUsr); + iRet = pThis->qDeq(pThis, ppMsg); ATOMIC_INC(&pThis->nLogDeq, &pThis->mutLogDeq); // DBGOPRINT((obj_t*) pThis, "entry deleted, size now log %d, phys %d entries\n", @@ -1176,11 +1144,11 @@ tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) rsRetVal iRetLocal; DEFiRet; -RUNLOG_STR("trying to shutdown workers within Action Timeout"); ISOBJ_TYPE_assert(pThis, qqueue); ASSERT(pThis->pqParent == NULL); /* detect invalid calling sequence */ /* instruct workers to finish ASAP, even if still work exists */ + DBGOPRINT((obj_t*) pThis, "trying to shutdown workers within Action Timeout"); DBGOPRINT((obj_t*) pThis, "setting EnqOnly mode\n"); pThis->bEnqOnly = 1; pThis->bShutdownImmediate = 1; @@ -1324,8 +1292,6 @@ finalize_it: RETiRet; } - - /* Constructor for the queue object * This constructs the data structure, but does not yet start the queue. That * is done by queueStart(). The reason is that we want to give the caller a chance @@ -1374,7 +1340,7 @@ finalize_it: } -/* set default inisde queue object suitable for action queues. +/* set default inside queue object suitable for action queues. * This shall be called directly after queue construction. This functions has * been added in support of the new v6 config system. It expect properly pre-initialized * objects, but we need to differentiate between ruleset main and action queues. @@ -1408,6 +1374,36 @@ qqueueSetDefaultsActionQueue(qqueue_t *pThis) } +/* set defaults inside queue object suitable for main/ruleset queues. + * See queueSetDefaultsActionQueue() for more details and background. + */ +void +qqueueSetDefaultsRulesetQueue(qqueue_t *pThis) +{ + pThis->qType = QUEUETYPE_FIXED_ARRAY; /* type of the main message queue above */ + pThis->iMaxQueueSize = 50000; /* size of the main message queue above */ + pThis->iDeqBatchSize = 1024; /* default batch size */ + pThis->iHighWtrMrk = 45000; /* high water mark for disk-assisted queues */ + pThis->iLowWtrMrk = 20000; /* low water mark for disk-assisted queues */ + pThis->iDiscardMrk = 49500; /* begin to discard messages */ + pThis->iDiscardSeverity = 8; /* turn off */ + pThis->iNumWorkerThreads = 1; /* number of worker threads for the mm queue above */ + pThis->iMaxFileSize = 16*1024*1024; + pThis->iPersistUpdCnt = 0; /* persist queue info every n updates */ + pThis->bSyncQueueFiles = 0; + pThis->toQShutdown = 1500; /* queue shutdown */ + pThis->toActShutdown = 1000; /* action shutdown (in phase 2) */ + pThis->toEnq = 2000; /* timeout for queue enque */ + pThis->toWrkShutdown = 60000; /* timeout for worker thread shutdown */ + pThis->iMinMsgsPerWrkr = 1000; /* minimum messages per worker needed to start a new one */ + pThis->bSaveOnShutdown = 1; /* save queue on shutdown (when DA enabled)? */ + pThis->sizeOnDiskMax = 0; /* unlimited */ + pThis->iDeqSlowdown = 0; + pThis->iDeqtWinFromHr = 0; + pThis->iDeqtWinToHr = 25; /* disable time-windowed dequeuing by default */ +} + + /* This function checks if the provided message shall be discarded and does so, if needed. * In DA mode, we do not discard any messages as we assume the disk subsystem is fast enough to * provide real-time creation of spool files. @@ -1421,22 +1417,21 @@ qqueueSetDefaultsActionQueue(qqueue_t *pThis) * the return state! * rgerhards, 2008-01-24 */ -static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, void *pUsr) +static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, msg_t *pMsg) { DEFiRet; rsRetVal iRetLocal; int iSeverity; ISOBJ_TYPE_assert(pThis, qqueue); - ISOBJ_assert(pUsr); if(pThis->iDiscardMrk > 0 && iQueueSize >= pThis->iDiscardMrk) { - iRetLocal = objGetSeverity(pUsr, &iSeverity); + iRetLocal = MsgGetSeverity(pMsg, &iSeverity); if(iRetLocal == RS_RET_OK && iSeverity >= pThis->iDiscardSeverity) { DBGOPRINT((obj_t*) pThis, "queue nearly full (%d entries), discarded severity %d message\n", iQueueSize, iSeverity); STATSCOUNTER_INC(pThis->ctrNFDscrd, pThis->mutCtrNFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } else { DBGOPRINT((obj_t*) pThis, "queue nearly full (%d entries), but could not drop msg " @@ -1455,19 +1450,38 @@ static inline rsRetVal DoDeleteBatchFromQStore(qqueue_t *pThis, int nElem) { int i; + off64_t bytesDel; DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); /* now send delete request to storage driver */ - for(i = 0 ; i < nElem ; ++i) { - pThis->qDel(pThis); + if(pThis->qType == QUEUETYPE_DISK) { + strmMultiFileSeek(pThis->tVars.disk.pReadDel, pThis->tVars.disk.deqFileNumOut, + pThis->tVars.disk.deqOffs, &bytesDel); + /* We need to correct the on-disk file size. This time it is a bit tricky: + * we free disk space only upon file deletion. So we need to keep track of what we + * have read until we get an out-offset that is lower than the in-offset (which + * indicates file change). Then, we can subtract the whole thing from the on-disk + * size. -- rgerhards, 2008-01-30 + */ + if(bytesDel != 0) { + pThis->tVars.disk.sizeOnDisk -= bytesDel; + DBGOPRINT((obj_t*) pThis, "doDeleteBatch: a %lld octet file has been deleted, now %lld octets disk " + "space used\n", bytesDel, pThis->tVars.disk.sizeOnDisk); + /* awake possibly waiting enq process */ + pthread_cond_signal(&pThis->notFull); /* we hold the mutex while we are in here! */ + } + } else { /* memory queue */ + for(i = 0 ; i < nElem ; ++i) { + pThis->qDel(pThis); + } } /* iQueueSize is not decremented by qDel(), so we need to do it ourselves */ ATOMIC_SUB(&pThis->iQueueSize, nElem, &pThis->mutQueueSize); ATOMIC_SUB(&pThis->nLogDeq, nElem, &pThis->mutLogDeq); - DBGPRINTF("delete batch from store, new sizes: log %d, phys %d\n", + DBGPRINTF("doDeleteBatch: delete batch from store, new sizes: log %d, phys %d\n", getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); ++pThis->deqIDDel; /* one more batch dequeued */ @@ -1522,7 +1536,7 @@ static inline rsRetVal DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) { int i; - void *pUsr; + msg_t *pMsg; int nEnqueued = 0; rsRetVal localRet; DEFiRet; @@ -1531,20 +1545,19 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) assert(pBatch != NULL); for(i = 0 ; i < pBatch->nElem ; ++i) { - pUsr = pBatch->pElem[i].pUsrp; - if( pBatch->pElem[i].state == BATCH_STATE_RDY - || pBatch->pElem[i].state == BATCH_STATE_SUB) { - localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, - (obj_t*)MsgAddRef((msg_t*) pUsr)); + pMsg = pBatch->pElem[i].pMsg; + if( pBatch->eltState[i] == BATCH_STATE_RDY + || pBatch->eltState[i] == BATCH_STATE_SUB) { + localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); ++nEnqueued; if(localRet != RS_RET_OK) { - DBGPRINTF("error %d re-enqueuing unprocessed data element - discarded\n", localRet); + DBGPRINTF("DeleteProcessedBatch: error %d re-enqueuing unprocessed data element - discarded\n", localRet); } } - objDestruct(pUsr); + msgDestruct(&pMsg); } - DBGPRINTF("we deleted %d objects and enqueued %d objects\n", i-nEnqueued, nEnqueued); + DBGPRINTF("DeleteProcessedBatch: we deleted %d objects and enqueued %d objects\n", i-nEnqueued, nEnqueued); if(nEnqueued > 0) qqueueChkPersist(pThis, nEnqueued); @@ -1573,7 +1586,7 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz int nDiscarded; int nDeleted; int iQueueSize; - void *pUsr; + msg_t *pMsg; rsRetVal localRet; DEFiRet; @@ -1581,11 +1594,14 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz DeleteProcessedBatch(pThis, &pWti->batch); nDequeued = nDiscarded = 0; + if(pThis->qType == QUEUETYPE_DISK) { + pThis->tVars.disk.deqFileNumIn = strmGetCurrFileNum(pThis->tVars.disk.pReadDeq); + } while((iQueueSize = getLogicalQueueSize(pThis)) > 0 && nDequeued < pThis->iDeqBatchSize) { - CHKiRet(qqueueDeq(pThis, &pUsr)); + CHKiRet(qqueueDeq(pThis, &pMsg)); /* check if we should discard this element */ - localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr); + localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pMsg); if(localRet == RS_RET_QUEUE_FULL) { ++nDiscarded; continue; @@ -1594,12 +1610,16 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz } /* all well, use this element */ - pWti->batch.pElem[nDequeued].pUsrp = pUsr; - pWti->batch.pElem[nDequeued].state = BATCH_STATE_RDY; - pWti->batch.pElem[nDequeued].bFilterOK = 1; // TODO: think again if we can handle that with more performance + pWti->batch.pElem[nDequeued].pMsg = pMsg; + pWti->batch.eltState[nDequeued] = BATCH_STATE_RDY; ++nDequeued; } + if(pThis->qType == QUEUETYPE_DISK) { + strm.GetCurrOffset(pThis->tVars.disk.pReadDeq, &pThis->tVars.disk.deqOffs); + pThis->tVars.disk.deqFileNumOut = strmGetCurrFileNum(pThis->tVars.disk.pReadDeq); + } + /* it is sufficient to persist only when the bulk of work is done */ qqueueChkPersist(pThis, nDequeued+nDiscarded+nDeleted); @@ -1607,7 +1627,6 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz pWti->batch.nElemDeq = nDequeued + nDiscarded; pWti->batch.deqID = getNextDeqID(pThis); *piRemainingQueueSize = iQueueSize; - finalize_it: RETiRet; } @@ -1643,7 +1662,6 @@ DequeueConsumable(qqueue_t *pThis, wti_t *pWti) pthread_cond_broadcast(&pThis->belowLightDlyWtrMrk); } - // TODO: MULTI: check physical queue size? pthread_cond_signal(&pThis->notFull); /* WE ARE NO LONGER PROTECTED BY THE MUTEX */ @@ -1837,7 +1855,8 @@ ConsumerReg(qqueue_t *pThis, wti_t *pWti) /* at this spot, we may be cancelled */ pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &iCancelStateSave); - CHKiRet(pThis->pConsumer(pThis->pUsr, &pWti->batch, &pThis->bShutdownImmediate)); + + CHKiRet(pThis->pConsumer(pThis->pAction, &pWti->batch, &pThis->bShutdownImmediate)); /* we now need to check if we should deliberately delay processing a bit * and, if so, do that. -- rgerhards, 2008-01-30 @@ -1895,23 +1914,55 @@ ConsumerDA(qqueue_t *pThis, wti_t *pWti) /* iterate over returned results and enqueue them in DA queue */ for(i = 0 ; i < pWti->batch.nElem && !pThis->bShutdownImmediate ; i++) { - /* TODO: we must add a generic "addRef" mechanism, because the disk queue enqueue destructs - * the message. So far, we simply assume we always have msg_t, what currently is always the case. - * rgerhards, 2009-05-28 - */ - CHKiRet(qqueueEnqObj(pThis->pqDA, eFLOWCTL_NO_DELAY, - (obj_t*)MsgAddRef((msg_t*)(pWti->batch.pElem[i].pUsrp)))); - pWti->batch.pElem[i].state = BATCH_STATE_COMM; /* commited to other queue! */ + iRet = qqueueEnqMsg(pThis->pqDA, eFLOWCTL_NO_DELAY, MsgAddRef(pWti->batch.pElem[i].pMsg)); + if(iRet != RS_RET_OK) { + if(iRet == RS_RET_ERR_QUEUE_EMERGENCY) { + /* Queue emergency error occured */ + DBGOPRINT((obj_t*) pThis, "ConsumerDA:qqueueEnqMsg caught RS_RET_ERR_QUEUE_EMERGENCY, aborting loop.\n"); + FINALIZE; + } else { + DBGOPRINT((obj_t*) pThis, "ConsumerDA:qqueueEnqMsg item (%d) returned with error state: '%d'\n", i, iRet); + } + } + pWti->batch.eltState[i] = BATCH_STATE_COMM; /* commited to other queue! */ } /* but now cancellation is no longer permitted */ pthread_setcancelstate(iCancelStateSave, NULL); finalize_it: + /* Check the last return state of qqueueEnqMsg. If an error was returned, we acknowledge it only. + * Unless the error code is RS_RET_ERR_QUEUE_EMERGENCY, we reset the return state to RS_RET_OK. + * Otherwise the Caller functions would run into an infinite Loop trying to enqueue the + * same messages over and over again. + * + * However we do NOT overwrite positive return states like + * RS_RET_TERMINATE_NOW, + * RS_RET_NO_RUN, + * RS_RET_IDLE, + * RS_RET_TERMINATE_WHEN_IDLE + * These return states are important for Queue handling of the upper laying functions. + * RGer: Note that checking for iRet < 0 is a bit bold. In theory, positive iRet + * values are "OK" states, and things that the caller shall deal with. However, + * this has not been done so consistently. Andre convinced me that the current + * code is an elegant solution. However, if problems with queue workers and/or + * shutdown come up, this code here should be looked at suspiciously. In those + * cases it may work out to check all status codes explicitely, just to avoid + * a pitfall due to unexpected states being passed on to the caller. + */ + if( iRet != RS_RET_OK && + iRet != RS_RET_ERR_QUEUE_EMERGENCY && + iRet < 0) { + DBGOPRINT((obj_t*) pThis, "ConsumerDA:qqueueEnqMsg Resetting iRet from %d back to RS_RET_OK\n", iRet); + iRet = RS_RET_OK; + } else { + DBGOPRINT((obj_t*) pThis, "ConsumerDA:qqueueEnqMsg returns with iRet %d\n", iRet); + } + /* now we are done, but potentially need to re-aquire the mutex */ if(bNeedReLock) d_pthread_mutex_lock(pThis->mut); - DBGOPRINT((obj_t*) pThis, "DAConsumer returns with iRet %d\n", iRet); + RETiRet; } @@ -1981,6 +2032,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ { DEFiRet; uchar pszBuf[64]; + uchar pszQIFNam[MAXFNAME]; int wrk; uchar *qName; size_t lenBuf; @@ -2003,8 +2055,8 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->qConstruct = qConstructLinkedList; pThis->qDestruct = qDestructLinkedList; pThis->qAdd = qAddLinkedList; - pThis->qDeq = (rsRetVal (*)(qqueue_t*,void**)) qDeqLinkedList; - pThis->qDel = (rsRetVal (*)(qqueue_t*)) qDelLinkedList; + pThis->qDeq = qDeqLinkedList; + pThis->qDel = qDelLinkedList; pThis->MultiEnq = qqueueMultiEnqObjNonDirect; break; case QUEUETYPE_DISK: @@ -2012,10 +2064,16 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->qDestruct = qDestructDisk; pThis->qAdd = qAddDisk; pThis->qDeq = qDeqDisk; - pThis->qDel = qDelDisk; + pThis->qDel = NULL; /* delete for disk handled via special code! */ pThis->MultiEnq = qqueueMultiEnqObjNonDirect; /* special handling */ pThis->iNumWorkerThreads = 1; /* we need exactly one worker */ + /* pre-construct file name for .qi file */ + pThis->lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), + "%s/%s.qi", (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); + pThis->pszQIFNam = ustrdup(pszQIFNam); + DBGOPRINT((obj_t*) pThis, ".qi file name is '%s', len %d\n", pThis->pszQIFNam, + (int) pThis->lenQIFNam); break; case QUEUETYPE_DIRECT: pThis->qConstruct = qConstructDirect; @@ -2075,6 +2133,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pThis->pqParent == NULL ? 0 : 1, pThis->iFullDlyMrk, pThis->iLightDlyMrk, pThis->iDeqBatchSize); + pThis->bQueueStarted = 1; if(pThis->qType == QUEUETYPE_DIRECT) FINALIZE; /* with direct queues, we are already finished... */ @@ -2105,7 +2164,6 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ * the case when a disk queue has been loaded. If we did not start it here, it would never start. */ qqueueAdviseMaxWorkers(pThis); - pThis->bQueueStarted = 1; /* support statistics gathering */ qName = obj.GetName((obj_t*)pThis); @@ -2142,7 +2200,7 @@ finalize_it: } -/* persist the queue to disk. If we have something to persist, we first +/* persist the queue to disk (write the .qi file). If we have something to persist, we first * save the information on the queue properties itself and then we call * the queue-type specific drivers. * Variable bIsCheckpoint is set to 1 if the persist is for a checkpoint, @@ -2153,8 +2211,6 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) { DEFiRet; strm_t *psQIF = NULL; /* Queue Info File */ - uchar pszQIFNam[MAXFNAME]; - size_t lenQIFNam; ASSERT(pThis != NULL); @@ -2172,13 +2228,9 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) DBGOPRINT((obj_t*) pThis, "persisting queue to disk, %d entries...\n", getPhysicalQueueSize(pThis)); - /* Construct file name */ - lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), "%s/%s.qi", - (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); - if((bIsCheckpoint != QUEUE_CHECKPOINT) && (getPhysicalQueueSize(pThis) == 0)) { if(pThis->bNeedDelQIF) { - unlink((char*)pszQIFNam); + unlink((char*)pThis->pszQIFNam); pThis->bNeedDelQIF = 0; } /* indicate spool file needs to be deleted */ @@ -2191,7 +2243,7 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_WRITE_TRUNC)); CHKiRet(strm.SetbSync(psQIF, pThis->bSyncQueueFiles)); CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.SetFName(psQIF, pThis->pszQIFNam, pThis->lenQIFNam)); CHKiRet(strm.ConstructFinalize(psQIF)); /* first, write the property bag for ourselfs @@ -2203,7 +2255,6 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) CHKiRet(obj.BeginSerializePropBag(psQIF, (obj_t*) pThis)); objSerializeSCALAR(psQIF, iQueueSize, INT); objSerializeSCALAR(psQIF, tVars.disk.sizeOnDisk, INT64); - objSerializeSCALAR(psQIF, tVars.disk.bytesRead, INT64); CHKiRet(obj.EndSerialize(psQIF)); /* now persist the stream info */ @@ -2303,73 +2354,75 @@ DoSaveOnShutdown(qqueue_t *pThis) /* destructor for the queue object */ BEGINobjDestruct(qqueue) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(qqueue) - /* shut down all workers - * We do not need to shutdown workers when we are in enqueue-only mode or we are a - * direct queue - because in both cases we have none... ;) - * with a child! -- rgerhards, 2008-01-28 - */ - if(pThis->qType != QUEUETYPE_DIRECT && !pThis->bEnqOnly && pThis->pqParent == NULL - && pThis->pWtpReg != NULL) - ShutdownWorkers(pThis); + if(pThis->bQueueStarted) { + /* shut down all workers + * We do not need to shutdown workers when we are in enqueue-only mode or we are a + * direct queue - because in both cases we have none... ;) + * with a child! -- rgerhards, 2008-01-28 + */ + if(pThis->qType != QUEUETYPE_DIRECT && !pThis->bEnqOnly && pThis->pqParent == NULL + && pThis->pWtpReg != NULL) + ShutdownWorkers(pThis); - if(pThis->bIsDA && getPhysicalQueueSize(pThis) > 0 && pThis->bSaveOnShutdown) { - CHKiRet(DoSaveOnShutdown(pThis)); - } + if(pThis->bIsDA && getPhysicalQueueSize(pThis) > 0 && pThis->bSaveOnShutdown) { + CHKiRet(DoSaveOnShutdown(pThis)); + } - /* finally destruct our (regular) worker thread pool - * Note: currently pWtpReg is never NULL, but if we optimize our logic, this may happen, - * e.g. when they are not created in enqueue-only mode. We already check the condition - * as this may otherwise be very hard to find once we optimize (and have long forgotten - * about this condition here ;) - * rgerhards, 2008-01-25 - */ - if(pThis->qType != QUEUETYPE_DIRECT && pThis->pWtpReg != NULL) { - wtpDestruct(&pThis->pWtpReg); - } + /* finally destruct our (regular) worker thread pool + * Note: currently pWtpReg is never NULL, but if we optimize our logic, this may happen, + * e.g. when they are not created in enqueue-only mode. We already check the condition + * as this may otherwise be very hard to find once we optimize (and have long forgotten + * about this condition here ;) + * rgerhards, 2008-01-25 + */ + if(pThis->qType != QUEUETYPE_DIRECT && pThis->pWtpReg != NULL) { + wtpDestruct(&pThis->pWtpReg); + } - /* Now check if we actually have a DA queue and, if so, destruct it. - * Note that the wtp must be destructed first, it may be in cancel cleanup handler - * *right now* and actually *need* to access the queue object to persist some final - * data (re-queueing case). So we need to destruct the wtp first, which will make - * sure all workers have terminated. Please note that this also generates a situation - * where it is possible that the DA queue has a parent pointer but the parent has - * no WtpDA associated with it - which is perfectly legal thanks to this code here. - */ - if(pThis->pWtpDA != NULL) { - wtpDestruct(&pThis->pWtpDA); - } - if(pThis->pqDA != NULL) { - qqueueDestruct(&pThis->pqDA); - } + /* Now check if we actually have a DA queue and, if so, destruct it. + * Note that the wtp must be destructed first, it may be in cancel cleanup handler + * *right now* and actually *need* to access the queue object to persist some final + * data (re-queueing case). So we need to destruct the wtp first, which will make + * sure all workers have terminated. Please note that this also generates a situation + * where it is possible that the DA queue has a parent pointer but the parent has + * no WtpDA associated with it - which is perfectly legal thanks to this code here. + */ + if(pThis->pWtpDA != NULL) { + wtpDestruct(&pThis->pWtpDA); + } + if(pThis->pqDA != NULL) { + qqueueDestruct(&pThis->pqDA); + } - /* persist the queue (we always do that - queuePersits() does cleanup if the queue is empty) - * This handler is most important for disk queues, it will finally persist the necessary - * on-disk structures. In theory, other queueing modes may implement their other (non-DA) - * methods of persisting a queue between runs, but in practice all of this is done via - * disk queues and DA mode. Anyhow, it doesn't hurt to know that we could extend it here - * if need arises (what I doubt...) -- rgerhards, 2008-01-25 - */ - CHKiRet_Hdlr(qqueuePersist(pThis, QUEUE_NO_CHECKPOINT)) { - DBGOPRINT((obj_t*) pThis, "error %d persisting queue - data lost!\n", iRet); - } + /* persist the queue (we always do that - queuePersits() does cleanup if the queue is empty) + * This handler is most important for disk queues, it will finally persist the necessary + * on-disk structures. In theory, other queueing modes may implement their other (non-DA) + * methods of persisting a queue between runs, but in practice all of this is done via + * disk queues and DA mode. Anyhow, it doesn't hurt to know that we could extend it here + * if need arises (what I doubt...) -- rgerhards, 2008-01-25 + */ + CHKiRet_Hdlr(qqueuePersist(pThis, QUEUE_NO_CHECKPOINT)) { + DBGOPRINT((obj_t*) pThis, "error %d persisting queue - data lost!\n", iRet); + } - /* finally, clean up some simple things... */ - if(pThis->pqParent == NULL) { - /* if we are not a child, we allocated our own mutex, which we now need to destroy */ - pthread_mutex_destroy(pThis->mut); - free(pThis->mut); - } - pthread_mutex_destroy(&pThis->mutThrdMgmt); - pthread_cond_destroy(&pThis->notFull); - pthread_cond_destroy(&pThis->notEmpty); - pthread_cond_destroy(&pThis->belowFullDlyWtrMrk); - pthread_cond_destroy(&pThis->belowLightDlyWtrMrk); + /* finally, clean up some simple things... */ + if(pThis->pqParent == NULL) { + /* if we are not a child, we allocated our own mutex, which we now need to destroy */ + pthread_mutex_destroy(pThis->mut); + free(pThis->mut); + } + pthread_mutex_destroy(&pThis->mutThrdMgmt); + pthread_cond_destroy(&pThis->notFull); + pthread_cond_destroy(&pThis->notEmpty); + pthread_cond_destroy(&pThis->belowFullDlyWtrMrk); + pthread_cond_destroy(&pThis->belowLightDlyWtrMrk); - DESTROY_ATOMIC_HELPER_MUT(pThis->mutQueueSize); - DESTROY_ATOMIC_HELPER_MUT(pThis->mutLogDeq); + DESTROY_ATOMIC_HELPER_MUT(pThis->mutQueueSize); + DESTROY_ATOMIC_HELPER_MUT(pThis->mutLogDeq); - /* type-specific destructor */ - iRet = pThis->qDestruct(pThis); + /* type-specific destructor */ + iRet = pThis->qDestruct(pThis); + } free(pThis->pszFilePrefix); free(pThis->pszSpoolDir); @@ -2431,7 +2484,7 @@ finalize_it: * rgerhards, 2009-06-16 */ static inline rsRetVal -doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) +doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg) { DEFiRet; int err; @@ -2440,7 +2493,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) STATSCOUNTER_INC(pThis->ctrEnqueued, pThis->mutCtrEnqueued); /* first check if we need to discard this message (which will cause CHKiRet() to exit) */ - CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr)); + CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pMsg)); /* handle flow control * There are two different flow control mechanisms: basic and advanced flow control. @@ -2475,7 +2528,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) * In any case, this was the old code (if we do the TODO): * pthread_cond_wait(&pThis->belowFullDlyWtrMrk, pThis->mut); */ - DBGOPRINT((obj_t*) pThis, "enqueueMsg: FullDelay mark reached for full delayable message " + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: FullDelay mark reached for full delayable message " "- blocking, queue size is %d.\n", pThis->iQueueSize); timeoutComp(&t, 1000); err = pthread_cond_timedwait(&pThis->belowLightDlyWtrMrk, pThis->mut, &t); @@ -2492,7 +2545,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) } } else if(flowCtlType == eFLOWCTL_LIGHT_DELAY && !glbl.GetGlobalInputTermState()) { if(pThis->iQueueSize >= pThis->iLightDlyMrk) { - DBGOPRINT((obj_t*) pThis, "enqueueMsg: LightDelay mark reached for light " + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: LightDelay mark reached for light " "delayable message - blocking a bit.\n"); timeoutComp(&t, 1000); /* 1000 millisconds = 1 second TODO: make configurable */ err = pthread_cond_timedwait(&pThis->belowLightDlyWtrMrk, pThis->mut, &t); @@ -2515,29 +2568,31 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) && pThis->tVars.disk.sizeOnDisk > pThis->sizeOnDiskMax)) { STATSCOUNTER_INC(pThis->ctrFull, pThis->mutCtrFull); if(pThis->toEnq == 0 || pThis->bEnqOnly) { - DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL - configured for immediate discarding.\n"); + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: queue FULL - configured for immediate discarding QueueSize=%d " + "MaxQueueSize=%d sizeOnDisk=%lld sizeOnDiskMax=%lld\n", pThis->iQueueSize, pThis->iMaxQueueSize, + pThis->tVars.disk.sizeOnDisk, pThis->sizeOnDiskMax); STATSCOUNTER_INC(pThis->ctrFDscrd, pThis->mutCtrFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } else { - DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL - waiting %dms to drain.\n", pThis->toEnq); + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: queue FULL - waiting %dms to drain.\n", pThis->toEnq); if(glbl.GetGlobalInputTermState()) { - DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL, discard due to FORCE_TERM.\n"); + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: queue FULL, discard due to FORCE_TERM.\n"); ABORT_FINALIZE(RS_RET_FORCE_TERM); } timeoutComp(&t, pThis->toEnq); if(pthread_cond_timedwait(&pThis->notFull, pThis->mut, &t) != 0) { - DBGOPRINT((obj_t*) pThis, "enqueueMsg: cond timeout, dropping message!\n"); + DBGOPRINT((obj_t*) pThis, "doEnqSingleObject: cond timeout, dropping message!\n"); STATSCOUNTER_INC(pThis->ctrFDscrd, pThis->mutCtrFDscrd); - objDestruct(pUsr); + msgDestruct(&pMsg); ABORT_FINALIZE(RS_RET_QUEUE_FULL); } - dbgoprint((obj_t*) pThis, "enqueueMsg: wait solved queue full condition, enqueing\n"); + dbgoprint((obj_t*) pThis, "doEnqSingleObject: wait solved queue full condition, enqueing\n"); } } /* and finally enqueue the message */ - CHKiRet(qqueueAdd(pThis, pUsr)); + CHKiRet(qqueueAdd(pThis, pMsg)); STATSCOUNTER_SETMAX_NOMUT(pThis->ctrMaxqsize, pThis->iQueueSize); finalize_it: @@ -2613,11 +2668,11 @@ finalize_it: * Enqueues the new element and awakes worker thread. */ rsRetVal -qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr) +qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg) { DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); - iRet = qAddDirect(pThis, pUsr); + iRet = qAddDirect(pThis, pMsg); RETiRet; } @@ -2626,7 +2681,7 @@ qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr) * Enqueues the new element and awakes worker thread. */ rsRetVal -qqueueEnqObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) +qqueueEnqMsg(qqueue_t *pThis, flowControl_t flowCtlType, msg_t *pMsg) { DEFiRet; int iCancelStateSave; @@ -2638,7 +2693,7 @@ qqueueEnqObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) d_pthread_mutex_lock(pThis->mut); } - CHKiRet(doEnqSingleObj(pThis, flowCtlType, pUsr)); + CHKiRet(doEnqSingleObj(pThis, flowCtlType, pMsg)); qqueueChkPersist(pThis, 1); @@ -2669,6 +2724,15 @@ qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals) return RS_RET_OK; } + +/* are any queue params set at all? 1 - yes, 0 - no */ +int +queueCnfParamsSet(struct cnfparamvals *pvals) +{ + return cnfparamvalsIsSet(&pblk, pvals); +} + + /* apply all params from param block to queue. Must be called before * finalizing. This supports the v6 config system. Defaults were already * set during queue creation. The pvals object is destructed by this @@ -2735,6 +2799,12 @@ qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals) "param '%s'\n", pblk.descr[i].name); } } + if(pThis->qType == QUEUETYPE_DISK && pThis->pszFilePrefix == NULL) { + errmsg.LogError(0, RS_RET_QUEUE_DISK_NO_FN, "error on queue '%s', disk mode selected, but " + "no queue file name given; queue type changed to 'linkedList'", + obj.GetName((obj_t*) pThis)); + pThis->qType = QUEUETYPE_LINKEDLIST; + } cnfparamvalsDestruct(pvals, &pblk); return RS_RET_OK; } @@ -2758,7 +2828,7 @@ DEFpropSetMeth(qqueue, iLightDlyMrk, int) DEFpropSetMeth(qqueue, bIsDA, int) DEFpropSetMeth(qqueue, iMinMsgsPerWrkr, int) DEFpropSetMeth(qqueue, bSaveOnShutdown, int) -DEFpropSetMeth(qqueue, pUsr, void*) +DEFpropSetMeth(qqueue, pAction, action_t*) DEFpropSetMeth(qqueue, iDeqSlowdown, int) DEFpropSetMeth(qqueue, iDeqBatchSize, int) DEFpropSetMeth(qqueue, sizeOnDiskMax, int64) @@ -2781,8 +2851,6 @@ static rsRetVal qqueueSetProperty(qqueue_t *pThis, var_t *pProp) pThis->iQueueSize = pProp->val.num; } else if(isProp("tVars.disk.sizeOnDisk")) { pThis->tVars.disk.sizeOnDisk = pProp->val.num; - } else if(isProp("tVars.disk.bytesRead")) { - pThis->tVars.disk.bytesRead = pProp->val.num; } else if(isProp("qType")) { if(pThis->qType != pProp->val.num) ABORT_FINALIZE(RS_RET_QTYPE_MISMATCH); diff --git a/runtime/queue.h b/runtime/queue.h index edb770c6..886fac8d 100644 --- a/runtime/queue.h +++ b/runtime/queue.h @@ -51,7 +51,7 @@ typedef enum { /* list member definition for linked list types of queues: */ typedef struct qLinkedList_S { struct qLinkedList_S *pNext; - void *pUsr; + msg_t *pMsg; } qLinkedList_t; @@ -71,7 +71,7 @@ struct queue_s { int iMinMsgsPerWrkr;/* minimum nbr of msgs per worker thread, if more, a new worker is started until max wrkrs */ wtp_t *pWtpDA; wtp_t *pWtpReg; - void *pUsr; /* a global, user-supplied pointer. Is passed back to consumer. */ + action_t *pAction; /* for action queues, ptr to action object; for main queues unused */ int iUpdsSincePersist;/* nbr of queue updates since the last persist call */ int iPersistUpdCnt; /* persits queue info after this nbr of updates - 0 -> persist only on shutdown */ sbool bSyncQueueFiles;/* if working with files, sync them after each write? */ @@ -111,8 +111,8 @@ struct queue_s { /* type-specific handlers (set during construction) */ rsRetVal (*qConstruct)(struct queue_s *pThis); rsRetVal (*qDestruct)(struct queue_s *pThis); - rsRetVal (*qAdd)(struct queue_s *pThis, void *pUsr); - rsRetVal (*qDeq)(struct queue_s *pThis, void **ppUsr); + rsRetVal (*qAdd)(struct queue_s *pThis, msg_t *pMsg); + rsRetVal (*qDeq)(struct queue_s *pThis, msg_t **ppMsg); rsRetVal (*qDel)(struct queue_s *pThis); /* end type-specific handler */ /* public entry points (set during construction, permit to set best algorithm for params selected) */ @@ -135,6 +135,8 @@ struct queue_s { size_t lenSpoolDir; uchar *pszFilePrefix; size_t lenFilePrefix; + uchar *pszQIFNam; /* full .qi file name, based on parts above */ + size_t lenQIFNam; int iNumberFiles; /* how many files make up the queue? */ int64 iMaxFileSize; /* max size for a single queue file */ int64 sizeOnDiskMax; /* maximum size on disk allowed */ @@ -145,7 +147,8 @@ struct queue_s { struct queue_s *pqParent;/* pointer to the parent (if this is a child queue) */ int bDAEnqOnly; /* EnqOnly setting for DA queue */ /* now follow queueing mode specific data elements */ - union { /* different data elements based on queue type (qType) */ + //union { /* different data elements based on queue type (qType) */ + struct { /* different data elements based on queue type (qType) */ struct { long deqhead, head, tail; void** pBuf; /* the queued user data structure */ @@ -157,7 +160,9 @@ struct queue_s { } linklist; struct { int64 sizeOnDisk; /* current amount of disk space used */ - int64 bytesRead; /* number of bytes read from current (undeleted!) file */ + int64 deqOffs; /* offset after dequeue batch - used for file deleter */ + int deqFileNumIn; /* same for the circular file numbers, mainly for */ + int deqFileNumOut;/* deleting finished files */ strm_t *pWrite; /* current file to be written */ strm_t *pReadDeq; /* current file for dequeueing */ strm_t *pReadDel; /* current file for deleting */ @@ -184,8 +189,8 @@ struct queue_s { /* prototypes */ rsRetVal qqueueDestruct(qqueue_t **ppThis); -rsRetVal qqueueEnqObjDirect(qqueue_t *pThis, void *pUsr); -rsRetVal qqueueEnqObj(qqueue_t *pThis, flowControl_t flwCtlType, void *pUsr); +rsRetVal qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg); +rsRetVal qqueueEnqMsg(qqueue_t *pThis, flowControl_t flwCtlType, msg_t *pMsg); rsRetVal qqueueStart(qqueue_t *pThis); rsRetVal qqueueSetMaxFileSize(qqueue_t *pThis, size_t iMaxFileSize); rsRetVal qqueueSetFilePrefix(qqueue_t *pThis, uchar *pszPrefix, size_t iLenPrefix); @@ -193,7 +198,9 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*, int*)); rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch); rsRetVal qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals); +int queueCnfParamsSet(struct cnfparamvals *pvals); rsRetVal qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals); +void qqueueSetDefaultsRulesetQueue(qqueue_t *pThis); void qqueueSetDefaultsActionQueue(qqueue_t *pThis); void qqueueDbgPrint(qqueue_t *pThis); @@ -213,7 +220,7 @@ PROTOTYPEpropSetMeth(qqueue, iDiscardMrk, int); PROTOTYPEpropSetMeth(qqueue, iDiscardSeverity, int); PROTOTYPEpropSetMeth(qqueue, iMinMsgsPerWrkr, int); PROTOTYPEpropSetMeth(qqueue, bSaveOnShutdown, int); -PROTOTYPEpropSetMeth(qqueue, pUsr, void*); +PROTOTYPEpropSetMeth(qqueue, pAction, action_t*); PROTOTYPEpropSetMeth(qqueue, iDeqSlowdown, int); PROTOTYPEpropSetMeth(qqueue, sizeOnDiskMax, int64); PROTOTYPEpropSetMeth(qqueue, iDeqBatchSize, int); diff --git a/runtime/ratelimit.c b/runtime/ratelimit.c new file mode 100644 index 00000000..a808e04a --- /dev/null +++ b/runtime/ratelimit.c @@ -0,0 +1,385 @@ +/* ratelimit.c + * support for rate-limiting sources, including "last message + * repeated n times" processing. + * + * Copyright 2012 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> + +#include "rsyslog.h" +#include "errmsg.h" +#include "ratelimit.h" +#include "datetime.h" +#include "parser.h" +#include "unicode-helper.h" +#include "msg.h" +#include "rsconf.h" +#include "dirty.h" + +/* definitions for objects we access */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) +DEFobjCurrIf(datetime) +DEFobjCurrIf(parser) + +/* static data */ + +/* generate a "repeated n times" message */ +static inline msg_t * +ratelimitGenRepMsg(ratelimit_t *ratelimit) +{ + msg_t *repMsg; + size_t lenRepMsg; + uchar szRepMsg[1024]; + + if(ratelimit->nsupp == 1) { /* we simply use the original message! */ + repMsg = MsgAddRef(ratelimit->pMsg); + } else {/* we need to duplicate, original message may still be in use in other + * parts of the system! */ + if((repMsg = MsgDup(ratelimit->pMsg)) == NULL) { + DBGPRINTF("Message duplication failed, dropping repeat message.\n"); + goto done; + } + lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), + " message repeated %d times: [%.800s]", + ratelimit->nsupp, getMSG(ratelimit->pMsg)); + MsgReplaceMSG(repMsg, szRepMsg, lenRepMsg); + } + +done: return repMsg; +} + +static inline rsRetVal +doLastMessageRepeatedNTimes(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRepMsg) +{ + int bNeedUnlockMutex = 0; + rsRetVal localRet; + DEFiRet; + + if((pMsg->msgFlags & NEEDS_PARSING) != 0) { + if((localRet = parser.ParseMsg(pMsg)) != RS_RET_OK) { + DBGPRINTF("Message discarded, parsing error %d\n", localRet); + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } + } + + if(ratelimit->bThreadSafe) { + pthread_mutex_lock(&ratelimit->mut); + bNeedUnlockMutex = 1; + } + + if( ratelimit->pMsg != NULL && + getMSGLen(pMsg) == getMSGLen(ratelimit->pMsg) && + !ustrcmp(getMSG(pMsg), getMSG(ratelimit->pMsg)) && + !strcmp(getHOSTNAME(pMsg), getHOSTNAME(ratelimit->pMsg)) && + !strcmp(getPROCID(pMsg, LOCK_MUTEX), getPROCID(ratelimit->pMsg, LOCK_MUTEX)) && + !strcmp(getAPPNAME(pMsg, LOCK_MUTEX), getAPPNAME(ratelimit->pMsg, LOCK_MUTEX))) { + ratelimit->nsupp++; + DBGPRINTF("msg repeated %d times\n", ratelimit->nsupp); + /* use current message, so we have the new timestamp + * (means we need to discard previous one) */ + msgDestruct(&ratelimit->pMsg); + ratelimit->pMsg = pMsg; + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } else {/* new message, do "repeat processing" & save it */ + if(ratelimit->pMsg != NULL) { + if(ratelimit->nsupp > 0) { + *ppRepMsg = ratelimitGenRepMsg(ratelimit); + ratelimit->nsupp = 0; + } + msgDestruct(&ratelimit->pMsg); + } + ratelimit->pMsg = MsgAddRef(pMsg); + } + +finalize_it: + if(bNeedUnlockMutex) + pthread_mutex_unlock(&ratelimit->mut); + RETiRet; +} + + +/* helper: tell how many messages we lost due to linux-like ratelimiting */ +static inline void +tellLostCnt(ratelimit_t *ratelimit) +{ + uchar msgbuf[1024]; + if(ratelimit->missed) { + snprintf((char*)msgbuf, sizeof(msgbuf), + "%s: %u messages lost due to rate-limiting", + ratelimit->name, ratelimit->missed); + ratelimit->missed = 0; + logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); + } +} + +/* Linux-like ratelimiting, modelled after the linux kernel + * returns 1 if message is within rate limit and shall be + * processed, 0 otherwise. + * This implementation is NOT THREAD-SAFE and must not + * be called concurrently. + */ +static inline int +withinRatelimit(ratelimit_t *ratelimit, time_t tt) +{ + int ret; + uchar msgbuf[1024]; + + if(ratelimit->interval == 0) { + ret = 1; + goto finalize_it; + } + + /* we primarily need "NoTimeCache" mode for imjournal, as it + * sets the message generation time to the journal timestamp. + * As such, we do not get a proper indication of the actual + * message rate. To prevent this, we need to query local + * system time ourselvs. + */ + if(ratelimit->bNoTimeCache) + tt = time(NULL); + + assert(ratelimit->burst != 0); + + if(ratelimit->begin == 0) + ratelimit->begin = tt; + + /* resume if we go out of time window */ + if(tt > ratelimit->begin + ratelimit->interval) { + ratelimit->begin = 0; + ratelimit->done = 0; + tellLostCnt(ratelimit); + } + + /* do actual limit check */ + if(ratelimit->burst > ratelimit->done) { + ratelimit->done++; + ret = 1; + } else { + ratelimit->missed++; + if(ratelimit->missed == 1) { + snprintf((char*)msgbuf, sizeof(msgbuf), + "%s: begin to drop messages due to rate-limiting", + ratelimit->name); + logmsgInternal(RS_RET_RATE_LIMITED, LOG_SYSLOG|LOG_INFO, msgbuf, 0); + } + ret = 0; + } + +finalize_it: + return ret; +} + + +/* ratelimit a message, that means: + * - handle "last message repeated n times" logic + * - handle actual (discarding) rate-limiting + * This function returns RS_RET_OK, if the caller shall process + * the message regularly and RS_RET_DISCARD if the caller must + * discard the message. The caller should also discard the message + * if another return status occurs. This places some burden on the + * caller logic, but provides best performance. Demanding this + * cooperative mode can enable a faulty caller to thrash up part + * of the system, but we accept that risk (a faulty caller can + * always do all sorts of evil, so...) + * If *ppRepMsg != NULL on return, the caller must enqueue that + * message before the original message. + */ +rsRetVal +ratelimitMsg(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRepMsg) +{ + DEFiRet; + + *ppRepMsg = NULL; + /* Only the messages having severity level at or below the + * treshold (the value is >=) are subject to ratelimiting. */ + if(ratelimit->interval && (pMsg->iSeverity >= ratelimit->severity)) { + if(withinRatelimit(ratelimit, pMsg->ttGenTime) == 0) { + msgDestruct(&pMsg); + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } + } + if(ratelimit->bReduceRepeatMsgs) { + CHKiRet(doLastMessageRepeatedNTimes(ratelimit, pMsg, ppRepMsg)); + } +finalize_it: + RETiRet; +} + +/* returns 1, if the ratelimiter performs any checks and 0 otherwise */ +int +ratelimitChecked(ratelimit_t *ratelimit) +{ + return ratelimit->interval || ratelimit->bReduceRepeatMsgs; +} + + +/* add a message to a ratelimiter/multisubmit structure. + * ratelimiting is automatically handled according to the ratelimit + * settings. + * if pMultiSub == NULL, a single-message enqueue happens (under reconsideration) + */ +rsRetVal +ratelimitAddMsg(ratelimit_t *ratelimit, multi_submit_t *pMultiSub, msg_t *pMsg) +{ + rsRetVal localRet; + msg_t *repMsg; + DEFiRet; + + if(pMultiSub == NULL) { + localRet = ratelimitMsg(ratelimit, pMsg, &repMsg); + if(repMsg != NULL) + CHKiRet(submitMsg2(repMsg)); + if(localRet == RS_RET_OK) + CHKiRet(submitMsg2(pMsg)); + } else { + localRet = ratelimitMsg(ratelimit, pMsg, &repMsg); + if(repMsg != NULL) { + pMultiSub->ppMsgs[pMultiSub->nElem++] = repMsg; + if(pMultiSub->nElem == pMultiSub->maxElem) + CHKiRet(multiSubmitMsg2(pMultiSub)); + } + if(localRet == RS_RET_OK) { + pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; + if(pMultiSub->nElem == pMultiSub->maxElem) + CHKiRet(multiSubmitMsg2(pMultiSub)); + } + } + +finalize_it: + RETiRet; +} + + +/* modname must be a static name (usually expected to be the module + * name and MUST be present. dynname may be NULL and can be used for + * dynamic information, e.g. PID or listener IP, ... + * Both values should be kept brief. + */ +rsRetVal +ratelimitNew(ratelimit_t **ppThis, char *modname, char *dynname) +{ + ratelimit_t *pThis; + char namebuf[256]; + DEFiRet; + + CHKmalloc(pThis = calloc(1, sizeof(ratelimit_t))); + if(modname == NULL) + modname ="*ERROR:MODULE NAME MISSING*"; + + if(dynname == NULL) { + pThis->name = strdup(modname); + } else { + snprintf(namebuf, sizeof(namebuf), "%s[%s]", + modname, dynname); + namebuf[sizeof(namebuf)-1] = '\0'; /* to be on safe side */ + pThis->name = strdup(namebuf); + } + /* pThis->severity == 0 - all messages are ratelimited */ + pThis->bReduceRepeatMsgs = loadConf->globals.bReduceRepeatMsgs; + *ppThis = pThis; +finalize_it: + RETiRet; +} + + +/* enable linux-like ratelimiting */ +void +ratelimitSetLinuxLike(ratelimit_t *ratelimit, unsigned short interval, unsigned short burst) +{ + ratelimit->interval = interval; + ratelimit->burst = burst; + ratelimit->done = 0; + ratelimit->missed = 0; + ratelimit->begin = 0; +} + + +/* enable thread-safe operations mode. This make sure that + * a single ratelimiter can be called from multiple threads. As + * this causes some overhead and is not always required, it needs + * to be explicitely enabled. This operation cannot be undone + * (think: why should one do that???) + */ +void +ratelimitSetThreadSafe(ratelimit_t *ratelimit) +{ + ratelimit->bThreadSafe = 1; + pthread_mutex_init(&ratelimit->mut, NULL); +} +void +ratelimitSetNoTimeCache(ratelimit_t *ratelimit) +{ + ratelimit->bNoTimeCache = 1; + pthread_mutex_init(&ratelimit->mut, NULL); +} + +/* Severity level determines which messages are subject to + * ratelimiting. Default (no value set) is all messages. + */ +void +ratelimitSetSeverity(ratelimit_t *ratelimit, intTiny severity) +{ + ratelimit->severity = severity; +} + +void +ratelimitDestruct(ratelimit_t *ratelimit) +{ + msg_t *pMsg; + if(ratelimit->pMsg != NULL) { + if(ratelimit->nsupp > 0) { + pMsg = ratelimitGenRepMsg(ratelimit); + if(pMsg != NULL) + submitMsg2(pMsg); + } + msgDestruct(&ratelimit->pMsg); + } + tellLostCnt(ratelimit); + if(ratelimit->bThreadSafe) + pthread_mutex_destroy(&ratelimit->mut); + free(ratelimit->name); + free(ratelimit); +} + +void +ratelimitModExit(void) +{ + objRelease(datetime, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); +} + +rsRetVal +ratelimitModInit(void) +{ + DEFiRet; + CHKiRet(objGetObjInterface(&obj)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(parser, CORE_COMPONENT)); +finalize_it: + RETiRet; +} diff --git a/runtime/ratelimit.h b/runtime/ratelimit.h new file mode 100644 index 00000000..563777fd --- /dev/null +++ b/runtime/ratelimit.h @@ -0,0 +1,55 @@ +/* header for ratelimit.c + * + * Copyright 2012 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_RATELIMIT_H +#define INCLUDED_RATELIMIT_H + +struct ratelimit_s { + char *name; /**< rate limiter name, e.g. for user messages */ + /* support for Linux kernel-type ratelimiting */ + unsigned short interval; + unsigned short burst; + intTiny severity; /**< ratelimit only equal or lower severity levels (eq or higher values) */ + unsigned done; + unsigned missed; + time_t begin; + /* support for "last message repeated n times */ + int bReduceRepeatMsgs; /**< shall we do "last message repeated n times" processing? */ + unsigned nsupp; /**< nbr of msgs suppressed */ + msg_t *pMsg; + sbool bThreadSafe; /**< do we need to operate in Thread-Safe mode? */ + sbool bNoTimeCache; /**< if we shall not used cached reception time */ + pthread_mutex_t mut; /**< mutex if thread-safe operation desired */ +}; + +/* prototypes */ +rsRetVal ratelimitNew(ratelimit_t **ppThis, char *modname, char *dynname); +void ratelimitSetThreadSafe(ratelimit_t *ratelimit); +void ratelimitSetLinuxLike(ratelimit_t *ratelimit, unsigned short interval, unsigned short burst); +void ratelimitSetNoTimeCache(ratelimit_t *ratelimit); +void ratelimitSetSeverity(ratelimit_t *ratelimit, intTiny severity); +rsRetVal ratelimitMsg(ratelimit_t *ratelimit, msg_t *pMsg, msg_t **ppRep); +rsRetVal ratelimitAddMsg(ratelimit_t *ratelimit, multi_submit_t *pMultiSub, msg_t *pMsg); +void ratelimitDestruct(ratelimit_t *pThis); +int ratelimitChecked(ratelimit_t *ratelimit); +rsRetVal ratelimitModInit(void); +void ratelimitModExit(void); + +#endif /* #ifndef INCLUDED_RATELIMIT_H */ diff --git a/runtime/rsconf.c b/runtime/rsconf.c index 118e9c11..d8b81f1b 100644 --- a/runtime/rsconf.c +++ b/runtime/rsconf.c @@ -36,7 +36,6 @@ #include "rsyslog.h" #include "obj.h" #include "srUtils.h" -#include "rule.h" #include "ruleset.h" #include "modules.h" #include "conf.h" @@ -68,9 +67,9 @@ #include "dirty.h" #include "template.h" +extern char* yytext; /* static data */ DEFobjStaticHelpers -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) DEFobjCurrIf(module) DEFobjCurrIf(conf) @@ -116,8 +115,8 @@ void cnfDoCfsysline(char *ln); */ BEGINobjConstruct(rsconf) /* be sure to specify the object type also in END macro! */ pThis->globals.bDebugPrintTemplateList = 1; - pThis->globals.bDebugPrintModuleList = 1; - pThis->globals.bDebugPrintCfSysLineHandlerList = 1; + pThis->globals.bDebugPrintModuleList = 0; + pThis->globals.bDebugPrintCfSysLineHandlerList = 0; pThis->globals.bLogStatusMsgs = DFLT_bLogStatusMsgs; pThis->globals.bErrMsgToStderr = 1; pThis->globals.umask = -1; @@ -254,54 +253,6 @@ CODESTARTobjDebugPrint(rsconf) ENDobjDebugPrint(rsconf) -rsRetVal -cnfDoActlst(struct cnfactlst *actlst, rule_t *pRule) -{ - struct cnfcfsyslinelst *cflst; - action_t *pAction; - uchar *str; - rsRetVal localRet; - DEFiRet; - - while(actlst != NULL) { - dbgprintf("aclst %p: ", actlst); - if(actlst->actType == CNFACT_V2) { - dbgprintf("v6+ action object\n"); - if(actionNewInst(actlst->data.lst, &pAction) == RS_RET_OK) { - iRet = llAppend(&(pRule)->llActList, NULL, (void*) pAction); - } else { - errmsg.LogError(0, RS_RET_ERR, "errors occured in file '%s' " - "around line %d", actlst->cnfFile, actlst->lineno); - } - } else { - DBGPRINTF("legacy action line:%s\n", actlst->data.legActLine); - str = (uchar*) actlst->data.legActLine; - if((localRet = cflineDoAction(loadConf, &str, &pAction)) != RS_RET_OK) { - uchar szErrLoc[MAXFNAME + 64]; - if(localRet != RS_RET_OK_WARN) { - DBGPRINTF("legacy action line NOT successfully processed\n"); - } - snprintf((char*)szErrLoc, sizeof(szErrLoc) / sizeof(uchar), - "%s, line %d", actlst->cnfFile, actlst->lineno); - errmsg.LogError(0, NO_ERRCODE, "the last %s occured in %s:\"%s\"", - (localRet == RS_RET_OK_WARN) ? "warning" : "error", - (char*)szErrLoc, (char*)actlst->data.legActLine); - if(localRet != RS_RET_OK_WARN) { - ABORT_FINALIZE(localRet); - } - } - iRet = llAppend(&(pRule)->llActList, NULL, (void*) pAction); - } - for( cflst = actlst->syslines - ; cflst != NULL ; cflst = cflst->next) { - cnfDoCfsysline(cflst->line); - } - actlst = actlst->next; - } -finalize_it: - RETiRet; -} - /* This function returns the current date in different * variants. It is used to construct the $NOW series of * system properties. The returned buffer must be freed @@ -341,6 +292,9 @@ getNOW(eNOWType eNow, es_str_t **estr) case NOW_MINUTE: len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.minute); break; + default: + len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "*invld eNow*"); + break; } /* now create a string object out of it and hand that over to the var */ @@ -397,15 +351,11 @@ inputProcessCnf(struct cnfobj *o) pvals = nvlstGetParams(o->nvlst, &inppblk, NULL); if(pvals == NULL) { - ABORT_FINALIZE(RS_RET_ERR); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); } DBGPRINTF("input param blk after inputProcessCnf:\n"); cnfparamsPrint(&inppblk, pvals); typeIdx = cnfparamGetIdx(&inppblk, "type"); - if(pvals[typeIdx].bUsed == 0) { - errmsg.LogError(0, RS_RET_CONF_RQRD_PARAM_MISSING, "input type missing"); - ABORT_FINALIZE(RS_RET_CONF_RQRD_PARAM_MISSING); // TODO: move this into rainerscript handlers - } cnfModName = (uchar*)es_str2cstr(pvals[typeIdx].val.d.estr, NULL); if((pMod = module.FindWithCnfName(loadConf, cnfModName, eMOD_IN)) == NULL) { errmsg.LogError(0, RS_RET_MOD_UNKNOWN, "input module name '%s' is unknown", cnfModName); @@ -435,9 +385,6 @@ parser_errmsg(char *fmt, ...) va_start(ap, fmt); if(vsnprintf(errBuf, sizeof(errBuf), fmt, ap) == sizeof(errBuf)) errBuf[sizeof(errBuf)-1] = '\0'; -dbgprintf("XXXX: msg: %s\n", errBuf); -dbgprintf("XXXX: cnfcurrfn: %s\n", cnfcurrfn); -dbgprintf("XXXX: yylineno: %d\n", yylineno); errmsg.LogError(0, RS_RET_CONF_PARSE_ERROR, "error during parsing file %s, on or before line %d: %s", cnfcurrfn, yylineno, errBuf); @@ -447,7 +394,7 @@ dbgprintf("XXXX: yylineno: %d\n", yylineno); int yyerror(char *s) { - parser_errmsg("%s", s); + parser_errmsg("%s on token '%s'", s, yytext); return 0; } void cnfDoObj(struct cnfobj *o) @@ -463,75 +410,35 @@ void cnfDoObj(struct cnfobj *o) case CNFOBJ_MODULE: modulesProcessCnf(o); break; - case CNFOBJ_ACTION: - actionProcessCnf(o); - break; case CNFOBJ_INPUT: inputProcessCnf(o); break; case CNFOBJ_TPL: - tplProcessCnf(o); + if(tplProcessCnf(o) != RS_RET_OK) + parser_errmsg("error processing template object"); + break; + case CNFOBJ_RULESET: + rulesetProcessCnf(o); break; case CNFOBJ_PROPERTY: case CNFOBJ_CONSTANT: /* these types are processed at a later stage */ bChkUnuse = 0; break; + default: + dbgprintf("cnfDoObj program error: unexpected object type %u\n", + o->objType); + break; } if(bChkUnuse) nvlstChkUnused(o->nvlst); cnfobjDestruct(o); } -void cnfDoRule(struct cnfrule *cnfrule) +void cnfDoScript(struct cnfstmt *script) { - rule_t *pRule; - uchar *str; - rsRetVal iRet = RS_RET_OK; //DEFiRet; - - dbgprintf("cnf:global:rule\n"); - cnfrulePrint(cnfrule); - - CHKiRet(rule.Construct(&pRule)); /* create "fresh" selector */ - CHKiRet(rule.SetAssRuleset(pRule, ruleset.GetCurrent(loadConf))); - CHKiRet(rule.ConstructFinalize(pRule)); - - switch(cnfrule->filttype) { - case CNFFILT_NONE: - break; - case CNFFILT_PRI: - str = (uchar*) cnfrule->filt.s; - iRet = cflineProcessTradPRIFilter(&str, pRule); - break; - case CNFFILT_PROP: - dbgprintf("%s\n", cnfrule->filt.s); - str = (uchar*) cnfrule->filt.s; - iRet = cflineProcessPropFilter(&str, pRule); - break; - case CNFFILT_SCRIPT: - pRule->f_filter_type = FILTER_EXPR; - pRule->f_filterData.expr = cnfrule->filt.expr; - break; - } - /* we now check if there are some global (BSD-style) filter conditions - * and, if so, we copy them over. rgerhards, 2005-10-18 - */ - if(pDfltProgNameCmp != NULL) { - CHKiRet(rsCStrConstructFromCStr(&(pRule->pCSProgNameComp), pDfltProgNameCmp)); - } - - if(eDfltHostnameCmpMode != HN_NO_COMP) { - pRule->eHostnameCmpMode = eDfltHostnameCmpMode; - CHKiRet(rsCStrConstructFromCStr(&(pRule->pCSHostnameComp), pDfltHostnameCmp)); - } - - cnfDoActlst(cnfrule->actlst, pRule); - - CHKiRet(ruleset.AddRule(rule.GetAssRuleset(pRule), &pRule)); - -finalize_it: - //TODO: do something with error states - cnfruleDestruct(cnfrule); + dbgprintf("cnf:global:script\n"); + ruleset.AddScript(ruleset.GetCurrent(loadConf), script); } void cnfDoCfsysline(char *ln) @@ -545,13 +452,21 @@ void cnfDoCfsysline(char *ln) void cnfDoBSDTag(char *ln) { DBGPRINTF("cnf:global:BSD tag: %s\n", ln); - cflineProcessTagSelector((uchar**)&ln); + errmsg.LogError(0, RS_RET_BSD_BLOCKS_UNSUPPORTED, + "BSD-style blocks are no longer supported in rsyslog, " + "see http://www.rsyslog.com/g/BSD for details and a " + "solution (Block '%s')", ln); + free(ln); } void cnfDoBSDHost(char *ln) { DBGPRINTF("cnf:global:BSD host: %s\n", ln); - cflineProcessHostSelector((uchar**)&ln); + errmsg.LogError(0, RS_RET_BSD_BLOCKS_UNSUPPORTED, + "BSD-style blocks are no longer supported in rsyslog, " + "see http://www.rsyslog.com/g/BSD for details and a " + "solution (Block '%s')", ln); + free(ln); } es_str_t* @@ -565,6 +480,9 @@ cnfGetVar(char *name, void *usrptr) estr = msgGetCEEVarNew((msg_t*) usrptr, name+2); else estr = msgGetMsgVarNew((msg_t*) usrptr, (uchar*)name+1); + } else { /* if this happens, we have a program logic error */ + estr = es_newStrFromCStr("err: var must start with $", + strlen("err: var must start with $")); } if(Debug) { char *s; @@ -841,7 +759,7 @@ activateMainQueue() { DEFiRet; /* create message queue */ - CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"))) { + CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"), NULL)) { /* no queue is fatal, we need to give up in that case... */ fprintf(stderr, "fatal error %d: could not create message queue - rsyslogd can not run!\n", iRet); FINALIZE; @@ -961,6 +879,7 @@ setCurrRuleset(void __attribute__((unused)) *pVal, uchar *pszName) CHKiRet(ruleset.Construct(&pRuleset)); CHKiRet(ruleset.SetName(pRuleset, pszName)); CHKiRet(ruleset.ConstructFinalize(ourConf, pRuleset)); + rulesetSetCurrRulesetPtr(pRuleset); } else { ABORT_FINALIZE(localRet); } @@ -1162,6 +1081,7 @@ initLegacyConf(void) ruleset.Construct(&pRuleset); ruleset.SetName(pRuleset, UCHAR_CONSTANT("RSYSLOG_DefaultRuleset")); ruleset.ConstructFinalize(loadConf, pRuleset); + rulesetSetCurrRulesetPtr(pRuleset); /* now register config handlers */ CHKiRet(regCfSysLineHdlr((uchar *)"sleep", 0, eCmdHdlrGoneAway, @@ -1378,6 +1298,7 @@ ourConf = loadConf; // TODO: remove, once ourConf is gone! ABORT_FINALIZE(RS_RET_NO_ACTIONS); } tellLexEndParsing(); + rulesetOptimizeAll(loadConf); tellCoreConfigLoadDone(); tellModulesConfigLoadDone(); @@ -1436,7 +1357,6 @@ ENDobjQueryInterface(rsconf) BEGINObjClassInit(rsconf, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(ruleset, CORE_COMPONENT)); - CHKiRet(objUse(rule, CORE_COMPONENT)); CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(conf, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); @@ -1453,7 +1373,6 @@ ENDObjClassInit(rsconf) /* De-initialize the rsconf class. */ BEGINObjClassExit(rsconf, OBJ_IS_CORE_MODULE) /* class, version */ - objRelease(rule, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); objRelease(module, CORE_COMPONENT); objRelease(conf, CORE_COMPONENT); diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index cbab06b7..047dfa9b 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -72,7 +72,6 @@ #include "glbl.h" #include "errmsg.h" #include "prop.h" -#include "rule.h" #include "ruleset.h" #include "parser.h" #include "strgen.h" @@ -171,8 +170,6 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(glblClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "msg"; CHKiRet(msgClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "rule"; - CHKiRet(ruleClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "ruleset"; CHKiRet(rulesetClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "wti"; @@ -220,7 +217,6 @@ rsrtExit(void) confClassExit(); glblClassExit(); rulesetClassExit(); - ruleClassExit(); objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */ } diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index a6e4b100..47b34783 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -46,6 +46,7 @@ #define CONF_HOSTNAME_MAXSIZE 512 /* a value that is deemed far too large for any valid HOSTNAME */ #define CONF_RAWMSG_BUFSIZE 101 #define CONF_TAG_BUFSIZE 32 +#define CONF_PROGNAME_BUFSIZE 16 #define CONF_HOSTNAME_BUFSIZE 32 #define CONF_PROP_BUFSIZE 16 /* should be close to sizeof(ptr) or lighly above it */ #define CONF_MIN_SIZE_FOR_COMPRESS 60 /* config param: minimum message size to try compression. The smaller @@ -60,15 +61,16 @@ * rgerhards, 2006-11-30 */ -#define CONF_OMOD_NUMSTRINGS_MAXSIZE 3 /* cache for pointers to output module buffer pointers. All - * rsyslog-provided plugins do NOT need more than three buffers. If - * more are needed (future developments, third-parties), rsyslog +#define CONF_OMOD_NUMSTRINGS_MAXSIZE 5 /* cache for pointers to output module buffer pointers. All + * rsyslog-provided plugins do NOT need more than five buffers. If + * more are needed (future developments, third-parties), rsyslog * must be recompiled with a larger parameter. Hardcoding this * saves us some overhead, both in runtime in code complexity. As * it is doubtful if ever more than 3 parameters are needed, the * approach taken here is considered appropriate. * rgerhards, 2010-06-24 */ +#define CONF_NUM_MULTISUB 1024 /* default number of messages per multisub structure */ /* ############################################################# * * # End Config Settings # * @@ -89,7 +91,7 @@ /* the rsyslog core provides information about present feature to plugins - * asking it. Below are feature-test macros which must be used to query + * asking it. Below are feature-test macros which must be used to query * features. Note that this must be powers of two, so that multiple queries * can be combined. -- rgerhards, 2009-04-27 */ @@ -151,7 +153,7 @@ typedef uintTiny propid_t; */ enum rsRetVal_ /** return value. All methods return this if not specified otherwise */ { - /* the first two define are for errmsg.logError(), so that we can use the rsRetVal + /* the first two define are for errmsg.logError(), so that we can use the rsRetVal * as an rsyslog error code. -- rgerhards, 20080-06-27 */ RS_RET_NO_ERRCODE = -1, /**< RESERVED for NO_ERRCODE errmsg.logError status name */ @@ -322,7 +324,7 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_ERR_DOOR = -2147, /**< some problems with handling the Solaris door functionality */ RS_RET_NO_SRCNAME_TPL = -2150, /**< sourcename template was not specified where one was needed (omudpspoof spoof addr) */ RS_RET_HOST_NOT_SPECIFIED = -2151, /**< (target) host was not specified where it was needed */ - RS_RET_ERR_LIBNET_INIT = -2152, /**< error initializing libnet */ + RS_RET_ERR_LIBNET_INIT = -2152, /**< error initializing libnet, e.g. because not running as root */ RS_RET_FORCE_TERM = -2153, /**< thread was forced to terminate by bShallShutdown, a state, not an error */ RS_RET_RULES_QUEUE_EXISTS = -2154,/**< we were instructed to create a new ruleset queue, but one already exists */ RS_RET_NO_CURR_RULESET = -2155,/**< no current ruleset exists (but one is required) */ @@ -376,15 +378,45 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_LEGA_ACT_NOT_SUPPORTED = -2215, /**< the module (no longer) supports legacy action syntax */ RS_RET_MAX_OMSR_REACHED = -2216, /**< max nbr of string requests reached, not supported by core */ RS_RET_UID_MISSING = -2217, /**< a user id is missing (but e.g. a password provided) */ + RS_RET_DATAFAIL = -2218, /**< data passed to action caused failure */ /* reserved for pre-v6.5 */ RS_RET_DUP_PARAM = -2220, /**< config parameter is given more than once */ RS_RET_MODULE_ALREADY_IN_CONF = -2221, /**< module already in current configuration */ RS_RET_PARAM_NOT_PERMITTED = -2222, /**< legacy parameter no longer permitted (usally already set by v2) */ RS_RET_NO_JSON_PASSING = -2223, /**< rsyslog core does not support JSON-passing plugin API */ RS_RET_MOD_NO_INPUT_STMT = -2224, /**< (input) module does not support input() statement */ + RS_RET_NO_CEE_MSG = -2225, /**< the message being processed is NOT CEE-enhanced */ + + /**** up to 2290 is reserved for v6 use ****/ + RS_RET_RELP_ERR = -2291, /**<< error in RELP processing */ + /**** up to 3000 is reserved for c7 use ****/ + RS_RET_JNAME_NO_ROOT = -2301, /**< root element is missing in JSON path */ + RS_RET_JNAME_INVALID = -2302, /**< JSON path is invalid */ + RS_RET_JSON_PARSE_ERR = -2303, /**< we had a problem parsing JSON (or extra data) */ + RS_RET_BSD_BLOCKS_UNSUPPORTED = -2304, /**< BSD-style config blocks are no longer supported */ + RS_RET_JNAME_NOTFOUND = -2305, /**< JSON name not found (does not exist) */ + RS_RET_INVLD_SETOP = -2305, /**< invalid variable set operation, incompatible type */ + RS_RET_RULESET_EXISTS = -2306,/**< ruleset already exists */ + RS_RET_DEPRECATED = -2307,/**< deprecated functionality is used */ + RS_RET_DS_PROP_SEQ_ERR = -2308,/**< property sequence error deserializing object */ + RS_RET_TPL_INVLD_PROP = -2309,/**< property name error in template (unknown name) */ + RS_RET_NO_RULEBASE = -2310,/**< mmnormalize: rulebase can not be found or otherwise invalid */ + RS_RET_INVLD_MODE = -2311,/**< invalid mode specified in configuration */ + RS_RET_INVLD_ANON_BITS = -2312,/**< mmanon: invalid number of bits to anonymize specified */ + RS_RET_REPLCHAR_IGNORED = -2313,/**< mmanon: replacementChar parameter is ignored */ + RS_RET_SIGPROV_ERR = -2320,/**< error in signature provider */ + RS_RET_CRYPROV_ERR = -2321,/**< error in cryptography encryption provider */ + RS_RET_EI_OPN_ERR = -2322,/**< error opening an .encinfo file */ + RS_RET_EI_NO_EXISTS = -2323,/**< .encinfo file does not exist (status, not necessarily error!)*/ + RS_RET_EI_WR_ERR = -2324,/**< error writing an .encinfo file */ + RS_RET_EI_INVLD_FILE = -2325,/**< header indicates the file is no .encinfo file */ + RS_RET_CRY_INVLD_ALGO = -2326,/**< user specified invalid (unkonwn) crypto algorithm */ + RS_RET_CRY_INVLD_MODE = -2327,/**< user specified invalid (unkonwn) crypto mode */ + RS_RET_QUEUE_DISK_NO_FN = -2328,/**< disk queue configured, but filename not set */ /* RainerScript error messages (range 1000.. 1999) */ RS_RET_SYSVAR_NOT_FOUND = 1001, /**< system variable could not be found (maybe misspelled) */ + RS_RET_FIELD_NOT_FOUND = 1002, /**< field() function did not find requested field */ /* some generic error/status codes */ RS_RET_OK = 0, /**< operation successful */ @@ -417,7 +449,7 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth /** Object ID. These are for internal checking. Each * object is assigned a specific ID. This is contained in - * all Object structs (just like C++ RTTI). We can use + * all Object structs (just like C++ RTTI). We can use * this field to see if we have been passed a correct ID. * Other than that, there is currently no other use for * the object id. @@ -449,7 +481,7 @@ typedef enum rsObjectID rsObjID; #endif /** - * This macro should be used to free objects. + * This macro should be used to free objects. * It aids in interpreting dumps during debugging. */ #ifdef NDEBUG @@ -516,7 +548,7 @@ rsRetVal rsrtSetErrLogger(rsRetVal (*errLogger)(int, uchar*)); /* TODO: remove this -- this is only for transition of the config system */ extern rsconf_t *ourConf; /* defined by syslogd.c, a hack for functions that do not - yet receive a copy, so that we can incrementially + yet receive a copy, so that we can incrementially compile and change... -- rgerhars, 2011-04-19 */ #endif /* multi-include protection */ diff --git a/runtime/rule.c b/runtime/rule.c deleted file mode 100644 index fc1e740f..00000000 --- a/runtime/rule.c +++ /dev/null @@ -1,479 +0,0 @@ -/* rule.c - rsyslog's rule object - * - * See file comment in rule.c for the overall structure of rule processing. - * - * Module begun 2009-06-10 by Rainer Gerhards - * - * Copyright 2009-2012 Adiscon GmbH. - * - * This file is part of the rsyslog runtime library. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * -or- - * see COPYING.ASL20 in the source distribution - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "config.h" -#include <stdlib.h> -#include <string.h> -#include <assert.h> -#include <ctype.h> - -#include "rsyslog.h" -#include "obj.h" -#include "action.h" -#include "rule.h" -#include "errmsg.h" -#include "srUtils.h" -#include "batch.h" -#include "parserif.h" -#include "unicode-helper.h" - -/* static data */ -DEFobjStaticHelpers -DEFobjCurrIf(errmsg) - - -/* support for simple textual representation of FIOP names - * rgerhards, 2005-09-27 - */ -static char* -getFIOPName(unsigned iFIOP) -{ - char *pRet; - switch(iFIOP) { - case FIOP_CONTAINS: - pRet = "contains"; - break; - case FIOP_ISEQUAL: - pRet = "isequal"; - break; - case FIOP_STARTSWITH: - pRet = "startswith"; - break; - case FIOP_REGEX: - pRet = "regex"; - break; - case FIOP_EREREGEX: - pRet = "ereregex"; - break; - case FIOP_ISEMPTY: - pRet = "isempty"; - break; - default: - pRet = "NOP"; - break; - } - return pRet; -} - - -/* iterate over all actions, this is often needed, for example when HUP processing - * must be done or a shutdown is pending. - */ -static rsRetVal -iterateAllActions(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) -{ - return llExecFunc(&pThis->llActList, pFunc, pParam); -} - - -/* helper to processMsg(), used to call the configured actions. It is - * executed from within llExecFunc() of the action list. - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(processBatchDoActions) -{ - DEFiRet; - rsRetVal iRetMod; /* return value of module - we do not always pass that back */ - action_t *pAction = (action_t*) pData; - batch_t *pBatch = (batch_t*) pParam; - - DBGPRINTF("Processing next action\n"); - iRetMod = pAction->submitToActQ(pAction, pBatch); - - RETiRet; -} - - -/* This functions looks at the given message and checks if it matches the - * provided filter condition. - */ -static rsRetVal -shouldProcessThisMessage(rule_t *pRule, msg_t *pMsg, sbool *bProcessMsg) -{ - DEFiRet; - unsigned short pbMustBeFreed; - uchar *pszPropVal; - int bRet = 0; - size_t propLen; - - ISOBJ_TYPE_assert(pRule, rule); - assert(pMsg != NULL); - - /* we first have a look at the global, BSD-style block filters (for tag - * and host). Only if they match, we evaluate the actual filter. - * rgerhards, 2005-10-18 - */ - if(pRule->eHostnameCmpMode == HN_NO_COMP) { - /* EMPTY BY INTENSION - we check this value first, because - * it is the one most often used, so this saves us time! - */ - } else if(pRule->eHostnameCmpMode == HN_COMP_MATCH) { - if(rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, so we are already done... */ - DBGPRINTF("hostname filter '+%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } else { /* must be -hostname */ - if(!rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, SO WE ARe already done... */ - DBGPRINTF("hostname filter '-%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } - - if(pRule->pCSProgNameComp != NULL) { - int bInv = 0, bEqv = 0, offset = 0; - if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp)) == '-') { - if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp) + 1) == '-') - offset = 1; - else { - bInv = 1; - offset = 1; - } - } - if(!rsCStrOffsetSzStrCmp(pRule->pCSProgNameComp, offset, - (uchar*) getProgramName(pMsg, LOCK_MUTEX), getProgramNameLen(pMsg, LOCK_MUTEX))) - bEqv = 1; - - if((!bEqv && !bInv) || (bEqv && bInv)) { - /* not equal or inverted selection, so we are already done... */ - DBGPRINTF("programname filter '%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp), getProgramName(pMsg, LOCK_MUTEX)); - FINALIZE; - } - } - - /* done with the BSD-style block filters */ - - if(pRule->f_filter_type == FILTER_PRI) { - /* skip messages that are incorrect priority */ - if ( (pRule->f_filterData.f_pmask[pMsg->iFacility] == TABLE_NOPRI) || \ - ((pRule->f_filterData.f_pmask[pMsg->iFacility] & (1<<pMsg->iSeverity)) == 0) ) - bRet = 0; - else - bRet = 1; - DBGPRINTF("testing filter, f_pmask %d, result %d\n", pRule->f_filterData.f_pmask[pMsg->iFacility], bRet); - } else if(pRule->f_filter_type == FILTER_EXPR) { - bRet = cnfexprEvalBool(pRule->f_filterData.expr, pMsg); - DBGPRINTF("result of rainerscript filter evaluation: %d\n", bRet); - } else { - assert(pRule->f_filter_type == FILTER_PROP); /* assert() just in case... */ - if(pRule->f_filterData.prop.propID == PROP_INVALID) { - DBGPRINTF("invalid property ID, filter always returns 0\n"); - bRet = 0; - } else { - pszPropVal = MsgGetProp(pMsg, NULL, pRule->f_filterData.prop.propID, - pRule->f_filterData.prop.propName, &propLen, &pbMustBeFreed); - - /* Now do the compares (short list currently ;)) */ - switch(pRule->f_filterData.prop.operation ) { - case FIOP_CONTAINS: - if(rsCStrLocateInSzStr(pRule->f_filterData.prop.pCSCompValue, (uchar*) pszPropVal) != -1) - bRet = 1; - break; - case FIOP_ISEMPTY: - if(propLen == 0) - bRet = 1; /* process message! */ - break; - case FIOP_ISEQUAL: - if(rsCStrSzStrCmp(pRule->f_filterData.prop.pCSCompValue, - pszPropVal, ustrlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_STARTSWITH: - if(rsCStrSzStrStartsWithCStr(pRule->f_filterData.prop.pCSCompValue, - pszPropVal, ustrlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_REGEX: - if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 0, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - case FIOP_EREREGEX: - if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 1, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - default: - /* here, it handles NOP (for performance reasons) */ - assert(pRule->f_filterData.prop.operation == FIOP_NOP); - bRet = 1; /* as good as any other default ;) */ - break; - } - - /* now check if the value must be negated */ - if(pRule->f_filterData.prop.isNegated) - bRet = (bRet == 1) ? 0 : 1; - - if(Debug) { - char *cstr; - if(pRule->f_filterData.prop.propID == PROP_CEE) { - cstr = es_str2cstr(pRule->f_filterData.prop.propName, NULL); - dbgprintf("Filter: check for CEE property '%s' (value '%s') ", - cstr, pszPropVal); - free(cstr); - } else { - dbgprintf("Filter: check for property '%s' (value '%s') ", - propIDToName(pRule->f_filterData.prop.propID), pszPropVal); - } - if(pRule->f_filterData.prop.isNegated) - dbgprintf("NOT "); - if(pRule->f_filterData.prop.operation == FIOP_ISEMPTY) { - dbgprintf("%s : %s\n", - getFIOPName(pRule->f_filterData.prop.operation), - bRet ? "TRUE" : "FALSE"); - } else { - dbgprintf("%s '%s': %s\n", - getFIOPName(pRule->f_filterData.prop.operation), - rsCStrGetSzStrNoNULL(pRule->f_filterData.prop.pCSCompValue), - bRet ? "TRUE" : "FALSE"); - } - } - - /* cleanup */ - if(pbMustBeFreed) - free(pszPropVal); - } - } - -finalize_it: - *bProcessMsg = bRet; - RETiRet; -} - - - -/* Process (consume) a batch of messages. Calls the actions configured. - * rgerhards, 2005-10-13 - */ -static rsRetVal -processBatch(rule_t *pThis, batch_t *pBatch) -{ - int i; - rsRetVal localRet; - DEFiRet; - - ISOBJ_TYPE_assert(pThis, rule); - assert(pBatch != NULL); - - /* first check the filters and reset status variables */ - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - localRet = shouldProcessThisMessage(pThis, (msg_t*)(pBatch->pElem[i].pUsrp), - &(pBatch->pElem[i].bFilterOK)); - if(localRet != RS_RET_OK) { - DBGPRINTF("processBatch: iRet %d returned from shouldProcessThisMessage, " - "ignoring message\n", localRet); - pBatch->pElem[i].bFilterOK = 0; - } - if(pBatch->pElem[i].bFilterOK) { - /* re-init only when actually needed (cache write cost!) */ - pBatch->pElem[i].bPrevWasSuspended = 0; - } - } - CHKiRet(llExecFunc(&pThis->llActList, processBatchDoActions, pBatch)); - -finalize_it: - RETiRet; -} - - -/* Standard-Constructor - */ -BEGINobjConstruct(rule) /* be sure to specify the object type also in END macro! */ -ENDobjConstruct(rule) - - -/* ConstructionFinalizer - * rgerhards, 2008-01-09 - */ -static rsRetVal -ruleConstructFinalize(rule_t *pThis) -{ - DEFiRet; - ISOBJ_TYPE_assert(pThis, rule); - - /* note: actionDestruct is from action.c API! */ - CHKiRet(llInit(&pThis->llActList, actionDestruct, NULL, NULL)); - -finalize_it: - RETiRet; -} - - -/* destructor for the rule object */ -BEGINobjDestruct(rule) /* be sure to specify the object type also in END and CODESTART macros! */ -CODESTARTobjDestruct(rule) - if(pThis->pCSHostnameComp != NULL) - rsCStrDestruct(&pThis->pCSHostnameComp); - if(pThis->pCSProgNameComp != NULL) - rsCStrDestruct(&pThis->pCSProgNameComp); - - if(pThis->f_filter_type == FILTER_PROP) { - if(pThis->f_filterData.prop.pCSCompValue != NULL) - rsCStrDestruct(&pThis->f_filterData.prop.pCSCompValue); - if(pThis->f_filterData.prop.regex_cache != NULL) - rsCStrRegexDestruct(&pThis->f_filterData.prop.regex_cache); - if(pThis->f_filterData.prop.propName != NULL) - es_deleteStr(pThis->f_filterData.prop.propName); - } else if(pThis->f_filter_type == FILTER_EXPR) { - cnfexprDestruct(pThis->f_filterData.expr); - } - - llDestroy(&pThis->llActList); -ENDobjDestruct(rule) - - -/* set the associated ruleset */ -static rsRetVal -setAssRuleset(rule_t *pThis, ruleset_t *pRuleset) -{ - DEFiRet; - ISOBJ_TYPE_assert(pThis, rule); - ISOBJ_TYPE_assert(pRuleset, ruleset); - pThis->pRuleset = pRuleset; - RETiRet; -} - -/* get the associated ruleset (may be NULL if not set!) */ -static ruleset_t* -getAssRuleset(rule_t *pThis) -{ - ISOBJ_TYPE_assert(pThis, rule); - return pThis->pRuleset; -} - - -/* helper to DebugPrint, to print out all actions via - * the llExecFunc() facility. - */ -DEFFUNC_llExecFunc(dbgPrintInitInfoAction) -{ - DEFiRet; - iRet = actionDbgPrint((action_t*) pData); - dbgprintf("\n"); - RETiRet; -} - - -/* debugprint for the rule object */ -BEGINobjDebugPrint(rule) /* be sure to specify the object type also in END and CODESTART macros! */ - int i; - char *cstr; -CODESTARTobjDebugPrint(rule) - dbgoprint((obj_t*) pThis, "rsyslog rule:\n"); - if(pThis->pCSProgNameComp != NULL) - dbgprintf("tag: '%s'\n", rsCStrGetSzStrNoNULL(pThis->pCSProgNameComp)); - if(pThis->eHostnameCmpMode != HN_NO_COMP) - dbgprintf("hostname: %s '%s'\n", - pThis->eHostnameCmpMode == HN_COMP_MATCH ? - "only" : "allbut", - rsCStrGetSzStrNoNULL(pThis->pCSHostnameComp)); - if(pThis->f_filter_type == FILTER_PRI) { - for (i = 0; i <= LOG_NFACILITIES; i++) - if (pThis->f_filterData.f_pmask[i] == TABLE_NOPRI) - dbgprintf(" X "); - else - dbgprintf("%2X ", pThis->f_filterData.f_pmask[i]); - } else if(pThis->f_filter_type == FILTER_EXPR) { - dbgprintf("EXPRESSION-BASED Filter: can currently not be displayed"); - } else { - dbgprintf("PROPERTY-BASED Filter:\n"); - dbgprintf("\tProperty.: '%s'\n", propIDToName(pThis->f_filterData.prop.propID)); - if(pThis->f_filterData.prop.propID != PROP_INVALID) { - if(pThis->f_filterData.prop.propName != NULL) { - cstr = es_str2cstr(pThis->f_filterData.prop.propName, NULL); - dbgprintf("\tCEE-Prop.: '%s'\n", cstr); - free(cstr); - } - dbgprintf("\tOperation: "); - if(pThis->f_filterData.prop.isNegated) - dbgprintf("NOT "); - dbgprintf("'%s'\n", getFIOPName(pThis->f_filterData.prop.operation)); - dbgprintf("\tValue....: '%s'\n", - rsCStrGetSzStrNoNULL(pThis->f_filterData.prop.pCSCompValue)); - } - dbgprintf("\tAction...: "); - } - - dbgprintf("\nActions:\n"); - llExecFunc(&pThis->llActList, dbgPrintInitInfoAction, NULL); /* actions */ - - dbgprintf("\n"); -ENDobjDebugPrint(rule) - - -/* queryInterface function - * rgerhards, 2008-02-21 - */ -BEGINobjQueryInterface(rule) -CODESTARTobjQueryInterface(rule) - if(pIf->ifVersion != ruleCURR_IF_VERSION) { /* check for current version, increment on each change */ - ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); - } - - /* ok, we have the right interface, so let's fill it - * Please note that we may also do some backwards-compatibility - * work here (if we can support an older interface version - that, - * of course, also affects the "if" above). - */ - pIf->Construct = ruleConstruct; - pIf->ConstructFinalize = ruleConstructFinalize; - pIf->Destruct = ruleDestruct; - pIf->DebugPrint = ruleDebugPrint; - - pIf->IterateAllActions = iterateAllActions; - pIf->ProcessBatch = processBatch; - pIf->SetAssRuleset = setAssRuleset; - pIf->GetAssRuleset = getAssRuleset; -finalize_it: -ENDobjQueryInterface(rule) - - -/* Exit the rule class. - * rgerhards, 2009-04-06 - */ -BEGINObjClassExit(rule, OBJ_IS_CORE_MODULE) /* class, version */ - objRelease(errmsg, CORE_COMPONENT); -ENDObjClassExit(rule) - - -/* Initialize the rule class. Must be called as the very first method - * before anything else is called inside this class. - * rgerhards, 2008-02-19 - */ -BEGINObjClassInit(rule, 1, OBJ_IS_CORE_MODULE) /* class, version */ - /* request objects we use */ - CHKiRet(objUse(errmsg, CORE_COMPONENT)); - - /* set our own handlers */ - OBJSetMethodHandler(objMethod_DEBUGPRINT, ruleDebugPrint); - OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, ruleConstructFinalize); -ENDObjClassInit(rule) - -/* vi:set ai: - */ diff --git a/runtime/rule.h b/runtime/rule.h deleted file mode 100644 index 1b07279b..00000000 --- a/runtime/rule.h +++ /dev/null @@ -1,78 +0,0 @@ -/* The rule object. - * - * This implements rules within rsyslog. - * - * Copyright 2009-2012 Adiscon GmbH. - * - * This file is part of the rsyslog runtime library. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * -or- - * see COPYING.ASL20 in the source distribution - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -#ifndef INCLUDED_RULE_H -#define INCLUDED_RULE_H - -#include "libestr.h" -#include "linkedlist.h" -#include "regexp.h" -#include "rainerscript.h" - -/* the rule object */ -struct rule_s { - BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - /* filter properties */ - enum { - FILTER_PRI = 0, /* traditional PRI based filer */ - FILTER_PROP = 1, /* extended filter, property based */ - FILTER_EXPR = 2 /* extended filter, expression based */ - } f_filter_type; - EHostnameCmpMode eHostnameCmpMode; - cstr_t *pCSHostnameComp; /* hostname to check */ - cstr_t *pCSProgNameComp; /* tag to check or NULL, if not to be checked */ - union { - u_char f_pmask[LOG_NFACILITIES+1]; /* priority mask */ - struct { - fiop_t operation; - regex_t *regex_cache; /* cache for compiled REs, if such are used */ - cstr_t *pCSCompValue; /* value to "compare" against */ - sbool isNegated; - propid_t propID; /* ID of the requested property */ - es_str_t *propName; /* name of property for CEE-based filters */ - } prop; - struct cnfexpr *expr; /* expression object */ - } f_filterData; - - ruleset_t *pRuleset; /* associated ruleset */ - linkedList_t llActList; /* list of configured actions */ -}; - -/* interfaces */ -BEGINinterface(rule) /* name must also be changed in ENDinterface macro! */ - INTERFACEObjDebugPrint(rule); - rsRetVal (*Construct)(rule_t **ppThis); - rsRetVal (*ConstructFinalize)(rule_t __attribute__((unused)) *pThis); - rsRetVal (*Destruct)(rule_t **ppThis); - rsRetVal (*IterateAllActions)(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void *pParam); - rsRetVal (*ProcessBatch)(rule_t *pThis, batch_t *pBatch); - rsRetVal (*SetAssRuleset)(rule_t *pThis, ruleset_t*); - ruleset_t* (*GetAssRuleset)(rule_t *pThis); -ENDinterface(rule) -#define ruleCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ -/* change for v2: ProcessMsg replaced by ProcessBatch - 2010-06-10 */ - - -/* prototypes */ -PROTOTYPEObj(rule); - -#endif /* #ifndef INCLUDED_RULE_H */ diff --git a/runtime/ruleset.c b/runtime/ruleset.c index 5cb34148..5bf7ac03 100644 --- a/runtime/ruleset.c +++ b/runtime/ruleset.c @@ -11,27 +11,24 @@ * * Module begun 2009-06-10 by Rainer Gerhards * - * Copyright 2009-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * - * The rsyslog runtime library is free software: you can redistribute it and/or modify - * it under the terms of the GNU Lesser General Public License as published by - * the Free Software Foundation, either version 3 of the License, or - * (at your option) any later version. - * - * The rsyslog runtime library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. - * - * A copy of the GPL can be found in the file "COPYING" in this distribution. - * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. */ - #include "config.h" #include <stdlib.h> #include <assert.h> @@ -42,22 +39,36 @@ #include "cfsysline.h" #include "msg.h" #include "ruleset.h" -#include "rule.h" #include "errmsg.h" #include "parser.h" #include "batch.h" #include "unicode-helper.h" #include "rsconf.h" +#include "action.h" +#include "rainerscript.h" +#include "srUtils.h" +#include "modules.h" #include "dirty.h" /* for main ruleset queue creation */ /* static data */ DEFobjStaticHelpers DEFobjCurrIf(errmsg) -DEFobjCurrIf(rule) DEFobjCurrIf(parser) +/* tables for interfacing with the v6 config system (as far as we need to) */ +static struct cnfparamdescr rspdescr[] = { + { "name", eCmdHdlrString, CNFPARAM_REQUIRED }, + { "parser", eCmdHdlrArray, 0 } +}; +static struct cnfparamblk rspblk = + { CNFPARAMBLK_VERSION, + sizeof(rspdescr)/sizeof(struct cnfparamdescr), + rspdescr + }; + /* forward definitions */ static rsRetVal processBatch(batch_t *pBatch); +static rsRetVal scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active); /* ---------- linked-list key handling functions (ruleset) ---------- */ @@ -73,45 +84,61 @@ rulesetKeyDestruct(void __attribute__((unused)) *pData) /* ---------- END linked-list key handling functions (ruleset) ---------- */ +/* iterate over all actions in a script (stmt subtree) */ +static void +scriptIterateAllActions(struct cnfstmt *root, rsRetVal (*pFunc)(void*, void*), void* pParam) +{ + struct cnfstmt *stmt; + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + switch(stmt->nodetype) { + case S_NOP: + case S_STOP: + case S_CALL:/* call does not need to do anything - done in called ruleset! */ + break; + case S_ACT: + DBGPRINTF("iterateAllActions calling into action %p\n", stmt->d.act); + pFunc(stmt->d.act, pParam); + break; + case S_IF: + if(stmt->d.s_if.t_then != NULL) + scriptIterateAllActions(stmt->d.s_if.t_then, + pFunc, pParam); + if(stmt->d.s_if.t_else != NULL) + scriptIterateAllActions(stmt->d.s_if.t_else, + pFunc, pParam); + break; + case S_PRIFILT: + if(stmt->d.s_prifilt.t_then != NULL) + scriptIterateAllActions(stmt->d.s_prifilt.t_then, + pFunc, pParam); + if(stmt->d.s_prifilt.t_else != NULL) + scriptIterateAllActions(stmt->d.s_prifilt.t_else, + pFunc, pParam); + break; + case S_PROPFILT: + scriptIterateAllActions(stmt->d.s_propfilt.t_then, + pFunc, pParam); + break; + default: + dbgprintf("error: unknown stmt type %u during iterateAll\n", + (unsigned) stmt->nodetype); + break; + } + } +} /* driver to iterate over all of this ruleset actions */ typedef struct iterateAllActions_s { rsRetVal (*pFunc)(void*, void*); void *pParam; } iterateAllActions_t; -DEFFUNC_llExecFunc(doIterateRulesetActions) -{ - DEFiRet; - rule_t* pRule = (rule_t*) pData; - iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; - iRet = rule.IterateAllActions(pRule, pMyParam->pFunc, pMyParam->pParam); - RETiRet; -} -/* iterate over all actions of THIS rule set. - */ -static rsRetVal -iterateRulesetAllActions(ruleset_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) -{ - iterateAllActions_t params; - DEFiRet; - assert(pFunc != NULL); - - params.pFunc = pFunc; - params.pParam = pParam; - CHKiRet(llExecFunc(&(pThis->llRules), doIterateRulesetActions, ¶ms)); - -finalize_it: - RETiRet; -} - - /* driver to iterate over all actions */ DEFFUNC_llExecFunc(doIterateAllActions) { DEFiRet; ruleset_t* pThis = (ruleset_t*) pData; iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; - iRet = iterateRulesetAllActions(pThis, pMyParam->pFunc, pMyParam->pParam); + scriptIterateAllActions(pThis->root, pMyParam->pFunc, pMyParam->pParam); RETiRet; } /* iterate over ALL actions present in the WHOLE system. @@ -134,30 +161,10 @@ finalize_it: } - -/* helper to processBatch(), used to call the configured actions. It is - * executed from within llExecFunc() of the action list. - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(processBatchDoRules) -{ - rsRetVal iRet; - ISOBJ_TYPE_assert(pData, rule); - DBGPRINTF("Processing next rule\n"); - iRet = rule.ProcessBatch((rule_t*) pData, (batch_t*) pParam); - DBGPRINTF("ruleset: get iRet %d from rule.ProcessMsg()\n", iRet); - return iRet; -} - - - /* This function is similar to processBatch(), but works on a batch that * contains rules from multiple rulesets. In this case, we can not push * the whole batch through the ruleset. Instead, we examine it and * partition it into sub-rulesets which we then push through the system. - * Note that when we evaluate which message must be processed, we do NOT need - * to look at bFilterOK, because this value is only set in a later processing - * stage. Doing so caused a bug during development ;) * rgerhards, 2010-06-15 */ static inline rsRetVal @@ -174,7 +181,7 @@ processBatchMultiRuleset(batch_t *pBatch) do { bHaveUnprocessed = 0; /* search for first unprocessed element */ - for(iStart = 0 ; iStart < pBatch->nElem && pBatch->pElem[iStart].state == BATCH_STATE_DISC ; ++iStart) + for(iStart = 0 ; iStart < pBatch->nElem && pBatch->eltState[iStart] == BATCH_STATE_DISC ; ++iStart) /* just search, no action */; if(iStart == pBatch->nElem) break; /* everything processed */ @@ -187,11 +194,11 @@ processBatchMultiRuleset(batch_t *pBatch) for(i = iStart ; i < pBatch->nElem ; ++i) { if(batchElemGetRuleset(pBatch, i) == currRuleset) { /* for performance reasons, we copy only those members that we actually need */ - snglRuleBatch.pElem[iNew].pUsrp = pBatch->pElem[i].pUsrp; - snglRuleBatch.pElem[iNew].state = pBatch->pElem[i].state; + snglRuleBatch.pElem[iNew].pMsg = pBatch->pElem[i].pMsg; + snglRuleBatch.eltState[iNew] = pBatch->eltState[i]; ++iNew; /* We indicate the element also as done, so it will not be processed again */ - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } else { bHaveUnprocessed = 1; } @@ -207,6 +214,348 @@ finalize_it: RETiRet; } +/* return a new "active" structure for the batch. Free with freeActive(). */ +static inline sbool *newActive(batch_t *pBatch) +{ + return malloc(sizeof(sbool) * batchNumMsgs(pBatch)); + +} +static inline void freeActive(sbool *active) { free(active); } + + +/* for details, see scriptExec() header comment! */ +/* call action for all messages with filter on */ +static rsRetVal +execAct(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + DEFiRet; +dbgprintf("RRRR: execAct [%s]: batch of %d elements, active %p\n", modGetName(stmt->d.act->pMod), batchNumMsgs(pBatch), active); + pBatch->active = active; + stmt->d.act->submitToActQ(stmt->d.act, pBatch); + RETiRet; +} + +static rsRetVal +execSet(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + int i; + struct var result; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + cnfexprEval(stmt->d.s_set.expr, &result, pBatch->pElem[i].pMsg); + msgSetJSONFromVar(pBatch->pElem[i].pMsg, stmt->d.s_set.varname, + &result); + varDelete(&result); + } + } + RETiRet; +} + +static rsRetVal +execUnset(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + int i; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + msgUnsetJSON(pBatch->pElem[i].pMsg, stmt->d.s_unset.varname); + } + } + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +/* "stop" simply discards the filtered items - it's just a (hopefully more intuitive + * shortcut for users. + */ +static rsRetVal +execStop(batch_t *pBatch, sbool *active) +{ + int i; + DEFiRet; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { + if( pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) { + pBatch->eltState[i] = BATCH_STATE_DISC; + } + } + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +// save current filter, evaluate new one +// perform then (if any message) +// if ELSE given: +// set new filter, inverted +// perform else (if any messages) +static rsRetVal +execIf(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *newAct; + int i; + sbool bRet; + sbool allInactive = 1; + DEFiRet; + newAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + FINALIZE; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + if(active == NULL || active[i]) { + bRet = cnfexprEvalBool(stmt->d.s_if.expr, pBatch->pElem[i].pMsg); + allInactive = 0; + } else + bRet = 0; + newAct[i] = bRet; + DBGPRINTF("batch: item %d: expr eval: %d\n", i, bRet); + } + + if(allInactive) { + DBGPRINTF("execIf: all batch elements are inactive, holding execution\n"); + freeActive(newAct); + FINALIZE; + } + + if(stmt->d.s_if.t_then != NULL) { + scriptExec(stmt->d.s_if.t_then, pBatch, newAct); + } + if(stmt->d.s_if.t_else != NULL) { + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + FINALIZE; + if(pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) + newAct[i] = !newAct[i]; + } + scriptExec(stmt->d.s_if.t_else, pBatch, newAct); + } + freeActive(newAct); +finalize_it: + RETiRet; +} + +/* for details, see scriptExec() header comment! */ +static void +execPRIFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *newAct; + msg_t *pMsg; + int bRet; + int i; + newAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + pMsg = pBatch->pElem[i].pMsg; + if(active == NULL || active[i]) { + if( (stmt->d.s_prifilt.pmask[pMsg->iFacility] == TABLE_NOPRI) || + ((stmt->d.s_prifilt.pmask[pMsg->iFacility] + & (1<<pMsg->iSeverity)) == 0) ) + bRet = 0; + else + bRet = 1; + } else + bRet = 0; + newAct[i] = bRet; + DBGPRINTF("batch: item %d PRIFILT %d\n", i, newAct[i]); + } + + if(stmt->d.s_prifilt.t_then != NULL) { + scriptExec(stmt->d.s_prifilt.t_then, pBatch, newAct); + } + if(stmt->d.s_prifilt.t_else != NULL) { + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] != BATCH_STATE_DISC + && (active == NULL || active[i])) + newAct[i] = !newAct[i]; + } + scriptExec(stmt->d.s_prifilt.t_else, pBatch, newAct); + } + freeActive(newAct); +} + + +/* helper to execPROPFILT(), as the evaluation itself is quite lengthy */ +static int +evalPROPFILT(struct cnfstmt *stmt, msg_t *pMsg) +{ + unsigned short pbMustBeFreed; + uchar *pszPropVal; + int bRet = 0; + rs_size_t propLen; + + if(stmt->d.s_propfilt.propID == PROP_INVALID) + goto done; + + pszPropVal = MsgGetProp(pMsg, NULL, stmt->d.s_propfilt.propID, + stmt->d.s_propfilt.propName, &propLen, + &pbMustBeFreed, NULL); + + /* Now do the compares (short list currently ;)) */ + switch(stmt->d.s_propfilt.operation ) { + case FIOP_CONTAINS: + if(rsCStrLocateInSzStr(stmt->d.s_propfilt.pCSCompValue, (uchar*) pszPropVal) != -1) + bRet = 1; + break; + case FIOP_ISEMPTY: + if(propLen == 0) + bRet = 1; /* process message! */ + break; + case FIOP_ISEQUAL: + if(rsCStrSzStrCmp(stmt->d.s_propfilt.pCSCompValue, + pszPropVal, propLen) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_STARTSWITH: + if(rsCStrSzStrStartsWithCStr(stmt->d.s_propfilt.pCSCompValue, + pszPropVal, propLen) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_REGEX: + if(rsCStrSzStrMatchRegex(stmt->d.s_propfilt.pCSCompValue, + (unsigned char*) pszPropVal, 0, &stmt->d.s_propfilt.regex_cache) == RS_RET_OK) + bRet = 1; + break; + case FIOP_EREREGEX: + if(rsCStrSzStrMatchRegex(stmt->d.s_propfilt.pCSCompValue, + (unsigned char*) pszPropVal, 1, &stmt->d.s_propfilt.regex_cache) == RS_RET_OK) + bRet = 1; + break; + default: + /* here, it handles NOP (for performance reasons) */ + assert(stmt->d.s_propfilt.operation == FIOP_NOP); + bRet = 1; /* as good as any other default ;) */ + break; + } + + /* now check if the value must be negated */ + if(stmt->d.s_propfilt.isNegated) + bRet = (bRet == 1) ? 0 : 1; + + if(Debug) { + char *cstr; + if(stmt->d.s_propfilt.propID == PROP_CEE) { + cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + DBGPRINTF("Filter: check for CEE property '%s' (value '%s') ", + cstr, pszPropVal); + free(cstr); + } else { + DBGPRINTF("Filter: check for property '%s' (value '%s') ", + propIDToName(stmt->d.s_propfilt.propID), pszPropVal); + } + if(stmt->d.s_propfilt.isNegated) + DBGPRINTF("NOT "); + if(stmt->d.s_propfilt.operation == FIOP_ISEMPTY) { + DBGPRINTF("%s : %s\n", + getFIOPName(stmt->d.s_propfilt.operation), + bRet ? "TRUE" : "FALSE"); + } else { + DBGPRINTF("%s '%s': %s\n", + getFIOPName(stmt->d.s_propfilt.operation), + rsCStrGetSzStrNoNULL(stmt->d.s_propfilt.pCSCompValue), + bRet ? "TRUE" : "FALSE"); + } + } + + /* cleanup */ + if(pbMustBeFreed) + free(pszPropVal); +done: + return bRet; +} + +/* for details, see scriptExec() header comment! */ +static void +execPROPFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +{ + sbool *thenAct; + sbool bRet; + int i; + thenAct = newActive(pBatch); + for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { + if(*(pBatch->pbShutdownImmediate)) + return; + if(pBatch->eltState[i] == BATCH_STATE_DISC) + continue; /* will be ignored in any case */ + if(active == NULL || active[i]) { + bRet = evalPROPFILT(stmt, pBatch->pElem[i].pMsg); + } else + bRet = 0; + thenAct[i] = bRet; + DBGPRINTF("batch: item %d PROPFILT %d\n", i, thenAct[i]); + } + + scriptExec(stmt->d.s_propfilt.t_then, pBatch, thenAct); + freeActive(thenAct); +} + +/* The rainerscript execution engine. It is debatable if that would be better + * contained in grammer/rainerscript.c, HOWEVER, that file focusses primarily + * on the parsing and object creation part. So as an actual executor, it is + * better suited here. + * param active: if NULL, all messages are active (to be processed), if non-null + * this is an array of the same size as the batch. If 1, the message + * is to be processed, otherwise not. + * NOTE: this function must receive batches which contain a single ruleset ONLY! + * rgerhards, 2012-09-04 + */ +static rsRetVal +scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active) +{ + DEFiRet; + struct cnfstmt *stmt; + + for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + if(Debug) { + dbgprintf("scriptExec: batch of %d elements, active %p, active[0]:%d\n", + batchNumMsgs(pBatch), active, (active == NULL ? 1 : active[0])); + cnfstmtPrintOnly(stmt, 2, 0); + } + switch(stmt->nodetype) { + case S_NOP: + break; + case S_STOP: + execStop(pBatch, active); + break; + case S_ACT: + execAct(stmt, pBatch, active); + break; + case S_SET: + execSet(stmt, pBatch, active); + break; + case S_UNSET: + execUnset(stmt, pBatch, active); + break; + case S_CALL: + scriptExec(stmt->d.s_call.stmt, pBatch, active); + break; + case S_IF: + execIf(stmt, pBatch, active); + break; + case S_PRIFILT: + execPRIFILT(stmt, pBatch, active); + break; + case S_PROPFILT: + execPROPFILT(stmt, pBatch, active); + break; + default: + dbgprintf("error: unknown stmt type %u during exec\n", + (unsigned) stmt->nodetype); + break; + } + } + RETiRet; +} + + /* Process (consume) a batch of messages. Calls the actions configured. * If the whole batch uses a singel ruleset, we can process the batch as * a whole. Otherwise, we need to process it slower, on a message-by-message @@ -226,7 +575,7 @@ processBatch(batch_t *pBatch) if(pThis == NULL) pThis = ourConf->rulesets.pDflt; ISOBJ_TYPE_assert(pThis, ruleset); - CHKiRet(llExecFunc(&pThis->llRules, processBatchDoRules, pBatch)); + CHKiRet(scriptExec(pThis->root, pBatch, NULL)); } else { CHKiRet(processBatchMultiRuleset(pBatch)); } @@ -248,34 +597,21 @@ GetParserList(rsconf_t *conf, msg_t *pMsg) } -/* Add a new rule to the end of the current rule set. We do a number - * of checks and ignore the rule if it does not pass them. - */ -static rsRetVal -addRule(ruleset_t *pThis, rule_t **ppRule) +/* Add a script block to the current ruleset */ +static void +addScript(ruleset_t *pThis, struct cnfstmt *script) { - int iActionCnt; - DEFiRet; - - ISOBJ_TYPE_assert(pThis, ruleset); - ISOBJ_TYPE_assert(*ppRule, rule); - - CHKiRet(llGetNumElts(&(*ppRule)->llActList, &iActionCnt)); - if(iActionCnt == 0) { - errmsg.LogError(0, NO_ERRCODE, "warning: selector line without actions will be discarded"); - rule.Destruct(ppRule); - } else { - CHKiRet(llAppend(&pThis->llRules, NULL, *ppRule)); - DBGPRINTF("selector line successfully processed, %d actions\n", iActionCnt); + if(pThis->last == NULL) + pThis->root = pThis->last = script; + else { + pThis->last->next = script; + pThis->last = script; } - -finalize_it: - RETiRet; } /* set name for ruleset */ -static rsRetVal setName(ruleset_t *pThis, uchar *pszName) +static rsRetVal rulesetSetName(ruleset_t *pThis, uchar *pszName) { DEFiRet; free(pThis->pszName); @@ -344,8 +680,7 @@ finalize_it: } -/* Set a new current rule set. If the ruleset can not be found, no change happens. - */ +/* Set a new current rule set. If the ruleset can not be found, no change happens */ static rsRetVal SetCurrRuleset(rsconf_t *conf, uchar *pszName) { @@ -362,23 +697,11 @@ finalize_it: } -/* destructor we need to destruct rules inside our linked list contents. - */ -static rsRetVal -doRuleDestruct(void *pData) -{ - rule_t *pRule = (rule_t *) pData; - DEFiRet; - rule.Destruct(&pRule); - RETiRet; -} - - /* Standard-Constructor */ BEGINobjConstruct(ruleset) /* be sure to specify the object type also in END macro! */ - CHKiRet(llInit(&pThis->llRules, doRuleDestruct, NULL, NULL)); -finalize_it: + pThis->root = NULL; + pThis->last = NULL; ENDobjConstruct(ruleset) @@ -399,9 +722,6 @@ rulesetConstructFinalize(rsconf_t *conf, ruleset_t *pThis) CHKmalloc(keyName = ustrdup(pThis->pszName)); CHKiRet(llAppend(&(conf->rulesets.llRulesets), keyName, pThis)); - /* this now also is the new current ruleset */ - conf->rulesets.pCurr = pThis; - /* and also the default, if so far none has been set */ if(conf->rulesets.pDflt == NULL) conf->rulesets.pDflt = pThis; @@ -421,8 +741,8 @@ CODESTARTobjDestruct(ruleset) if(pThis->pParserLst != NULL) { parser.DestructParserList(&pThis->pParserLst); } - llDestroy(&pThis->llRules); free(pThis->pszName); + cnfstmtDestructLst(pThis->root); ENDobjDestruct(ruleset) @@ -456,16 +776,13 @@ rulesetDestructForLinkedList(void *pData) return rulesetDestruct(&pThis); } -/* helper for debugPrint(), initiates rule printing */ -DEFFUNC_llExecFunc(doDebugPrintRule) -{ - return rule.DebugPrint((rule_t*) pData); -} /* debugprint for the ruleset object */ BEGINobjDebugPrint(ruleset) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDebugPrint(ruleset) dbgoprint((obj_t*) pThis, "rsyslog ruleset %s:\n", pThis->pszName); - llExecFunc(&pThis->llRules, doDebugPrintRule, NULL); + cnfstmtPrint(pThis->root, 0); + dbgoprint((obj_t*) pThis, "ruleset %s assigned parser list:\n", pThis->pszName); + printParserList(pThis->pParserLst); ENDobjDebugPrint(ruleset) @@ -486,6 +803,40 @@ debugPrintAll(rsconf_t *conf) RETiRet; } +static inline void +rulesetOptimize(ruleset_t *pRuleset) +{ + if(Debug) { + dbgprintf("ruleset '%s' before optimization:\n", + pRuleset->pszName); + rulesetDebugPrint((ruleset_t*) pRuleset); + } + cnfstmtOptimize(pRuleset->root); + if(Debug) { + dbgprintf("ruleset '%s' after optimization:\n", + pRuleset->pszName); + rulesetDebugPrint((ruleset_t*) pRuleset); + } +} + +/* helper for rulsetOptimizeAll(), optimizes a single ruleset */ +DEFFUNC_llExecFunc(doRulesetOptimizeAll) +{ + rulesetOptimize((ruleset_t*) pData); + return RS_RET_OK; +} +/* optimize all rulesets + */ +rsRetVal +rulesetOptimizeAll(rsconf_t *conf) +{ + DEFiRet; + dbgprintf("begin ruleset optimization phase\n"); + llExecFunc(&(conf->rulesets.llRulesets), doRulesetOptimizeAll, NULL); + dbgprintf("ruleset optimization phase finished.\n"); + RETiRet; +} + /* Create a ruleset-specific "main" queue for this ruleset. If one is already * defined, an error message is emitted but nothing else is done. @@ -517,7 +868,7 @@ doRulesetCreateQueue(rsconf_t *conf, int *pNewVal) rsname = (conf->rulesets.pCurr->pszName == NULL) ? (uchar*) "[ruleset]" : conf->rulesets.pCurr->pszName; DBGPRINTF("adding a ruleset-specific \"main\" queue for ruleset '%s'\n", rsname); - CHKiRet(createMainQueue(&conf->rulesets.pCurr->pQueue, rsname)); + CHKiRet(createMainQueue(&conf->rulesets.pCurr->pQueue, rsname, NULL)); finalize_it: RETiRet; @@ -539,13 +890,11 @@ rulesetCreateQueue(void __attribute__((unused)) *pVal, int *pNewVal) * rgerhards, 2009-11-04 */ static rsRetVal -doRulesetAddParser(rsconf_t *conf, uchar *pName) +doRulesetAddParser(ruleset_t *pRuleset, uchar *pName) { parser_t *pParser; DEFiRet; - assert(conf->rulesets.pCurr != NULL); - CHKiRet(objUse(parser, CORE_COMPONENT)); iRet = parser.FindParser(&pParser, pName); if(iRet == RS_RET_PARSER_NOT_FOUND) { @@ -557,9 +906,9 @@ doRulesetAddParser(rsconf_t *conf, uchar *pName) FINALIZE; } - CHKiRet(parser.AddParserToList(&conf->rulesets.pCurr->pParserLst, pParser)); + CHKiRet(parser.AddParserToList(&pRuleset->pParserLst, pParser)); - DBGPRINTF("added parser '%s' to ruleset '%s'\n", pName, conf->rulesets.pCurr->pszName); + DBGPRINTF("added parser '%s' to ruleset '%s'\n", pName, pRuleset->pszName); finalize_it: d_free(pName); /* no longer needed */ @@ -570,7 +919,72 @@ finalize_it: static rsRetVal rulesetAddParser(void __attribute__((unused)) *pVal, uchar *pName) { - return doRulesetAddParser(ourConf, pName); + return doRulesetAddParser(ourConf->rulesets.pCurr, pName); +} + + +/* Process ruleset() objects */ +rsRetVal +rulesetProcessCnf(struct cnfobj *o) +{ + struct cnfparamvals *pvals; + struct cnfparamvals *queueParams; + rsRetVal localRet; + uchar *rsName = NULL; + uchar *parserName; + int nameIdx, parserIdx; + ruleset_t *pRuleset; + struct cnfarray *ar; + int i; + uchar *rsname; + DEFiRet; + + pvals = nvlstGetParams(o->nvlst, &rspblk, NULL); + if(pvals == NULL) { + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + DBGPRINTF("ruleset param blk after rulesetProcessCnf:\n"); + cnfparamsPrint(&rspblk, pvals); + nameIdx = cnfparamGetIdx(&rspblk, "name"); + rsName = (uchar*)es_str2cstr(pvals[nameIdx].val.d.estr, NULL); + localRet = rulesetGetRuleset(loadConf, &pRuleset, rsName); + if(localRet == RS_RET_OK) { + errmsg.LogError(0, RS_RET_RULESET_EXISTS, + "error: ruleset '%s' specified more than once", + rsName); + cnfstmtDestructLst(o->script); + ABORT_FINALIZE(RS_RET_RULESET_EXISTS); + } else if(localRet != RS_RET_NOT_FOUND) { + ABORT_FINALIZE(localRet); + } + CHKiRet(rulesetConstruct(&pRuleset)); + CHKiRet(rulesetSetName(pRuleset, rsName)); + CHKiRet(rulesetConstructFinalize(loadConf, pRuleset)); + addScript(pRuleset, o->script); + + /* we have only two params, so we do NOT do the usual param loop */ + parserIdx = cnfparamGetIdx(&rspblk, "parser"); + if(parserIdx != -1 && pvals[parserIdx].bUsed) { + ar = pvals[parserIdx].val.d.ar; + for(i = 0 ; i < ar->nmemb ; ++i) { + parserName = (uchar*)es_str2cstr(ar->arr[i], NULL); + doRulesetAddParser(pRuleset, parserName); + free(parserName); + } + } + + /* pick up ruleset queue parameters */ + qqueueDoCnfParams(o->nvlst, &queueParams); + if(queueCnfParamsSet(queueParams)) { + rsname = (pRuleset->pszName == NULL) ? (uchar*) "[ruleset]" : pRuleset->pszName; + DBGPRINTF("adding a ruleset-specific \"main\" queue for ruleset '%s'\n", rsname); + CHKiRet(createMainQueue(&pRuleset->pQueue, rsname, queueParams)); + } + +finalize_it: + free(rsName); + cnfparamvalsDestruct(pvals, &rspblk); + RETiRet; } @@ -595,9 +1009,9 @@ CODESTARTobjQueryInterface(ruleset) pIf->IterateAllActions = iterateAllActions; pIf->DestructAllActions = destructAllActions; - pIf->AddRule = addRule; + pIf->AddScript = addScript; pIf->ProcessBatch = processBatch; - pIf->SetName = setName; + pIf->SetName = rulesetSetName; pIf->DebugPrintAll = debugPrintAll; pIf->GetCurrent = GetCurrent; pIf->GetRuleset = rulesetGetRuleset; @@ -614,7 +1028,6 @@ ENDobjQueryInterface(ruleset) */ BEGINObjClassExit(ruleset, OBJ_IS_CORE_MODULE) /* class, version */ objRelease(errmsg, CORE_COMPONENT); - objRelease(rule, CORE_COMPONENT); objRelease(parser, CORE_COMPONENT); ENDObjClassExit(ruleset) @@ -626,7 +1039,6 @@ ENDObjClassExit(ruleset) BEGINObjClassInit(ruleset, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); - CHKiRet(objUse(rule, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, rulesetDebugPrint); diff --git a/runtime/ruleset.h b/runtime/ruleset.h index f4443e18..cbf8243b 100644 --- a/runtime/ruleset.h +++ b/runtime/ruleset.h @@ -25,13 +25,15 @@ #include "queue.h" #include "linkedlist.h" +#include "rsconf.h" /* the ruleset object */ struct ruleset_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - linkedList_t llRules; /* this is NOT a pointer - no typo here ;) */ uchar *pszName; /* name of our ruleset */ qqueue_t *pQueue; /* "main" message queue, if the ruleset has its own (else NULL) */ + struct cnfstmt *root; + struct cnfstmt *last; parserList_t *pParserLst;/* list of parsers to use for this ruleset */ }; @@ -42,9 +44,7 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Construct)(ruleset_t **ppThis); rsRetVal (*ConstructFinalize)(rsconf_t *conf, ruleset_t __attribute__((unused)) *pThis); rsRetVal (*Destruct)(ruleset_t **ppThis); - rsRetVal (*IterateAllActions)(rsconf_t *conf, rsRetVal (*pFunc)(void*, void*), void* pParam); rsRetVal (*DestructAllActions)(rsconf_t *conf); - rsRetVal (*AddRule)(ruleset_t *pThis, rule_t **ppRule); rsRetVal (*SetName)(ruleset_t *pThis, uchar *pszName); rsRetVal (*ProcessBatch)(batch_t*); rsRetVal (*GetRuleset)(rsconf_t *conf, ruleset_t **ppThis, uchar*); @@ -60,8 +60,12 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ * removed conf ptr from SetName, AddRule as the flex/bison based * system uses globals in any case. */ + /* v7, 2012-09-04 */ + /* AddRule() removed */ + /*TODO:REMOVE*/rsRetVal (*IterateAllActions)(rsconf_t *conf, rsRetVal (*pFunc)(void*, void*), void* pParam); + void (*AddScript)(ruleset_t *pThis, struct cnfstmt *script); ENDinterface(ruleset) -#define rulesetCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define rulesetCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ /* prototypes */ @@ -87,5 +91,16 @@ rulesetGetName(ruleset_t *pRuleset) } +/* we will most probably convert this module back to traditional C + * calling sequence, so here we go... + */ rsRetVal rulesetGetRuleset(rsconf_t *conf, ruleset_t **ppRuleset, uchar *pszName); +rsRetVal rulesetOptimizeAll(rsconf_t *conf); +rsRetVal rulesetProcessCnf(struct cnfobj *o); + +/* Set a current rule set to already-known pointer */ +static inline void +rulesetSetCurrRulesetPtr(ruleset_t *pRuleset) { + loadConf->rulesets.pCurr = pRuleset; +} #endif /* #ifndef INCLUDED_RULESET_H */ diff --git a/runtime/sigprov.h b/runtime/sigprov.h new file mode 100644 index 00000000..82587b7d --- /dev/null +++ b/runtime/sigprov.h @@ -0,0 +1,37 @@ +/* The interface definition for (file) signature providers. + * + * This is just an abstract driver interface, which needs to be + * implemented by concrete classes. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_SIGPROV_H +#define INCLUDED_SIGPROV_H + +/* interface */ +BEGINinterface(sigprov) /* name must also be changed in ENDinterface macro! */ + rsRetVal (*Construct)(void *ppThis); + rsRetVal (*SetCnfParam)(void *ppThis, struct nvlst *lst); + rsRetVal (*Destruct)(void *ppThis); + rsRetVal (*OnFileOpen)(void *pThis, uchar *fn, void *pFileInstData); + rsRetVal (*OnRecordWrite)(void *pFileInstData, uchar *rec, rs_size_t lenRec); + rsRetVal (*OnFileClose)(void *pFileInstData); +ENDinterface(sigprov) +#define sigprovCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#endif /* #ifndef INCLUDED_SIGPROV_H */ diff --git a/runtime/srUtils.h b/runtime/srUtils.h index 3169fd94..8626a4bb 100644 --- a/runtime/srUtils.h +++ b/runtime/srUtils.h @@ -91,6 +91,7 @@ char *rs_strerror_r(int errnum, char *buf, size_t buflen); int decodeSyslogName(uchar *name, syslogName_t *codetab); int getSubString(uchar **ppSrc, char *pDst, size_t DstSize, char cSep); rsRetVal getFileSize(uchar *pszName, off_t *pSize); +int containsGlobWildcard(char *str); /* mutex operations */ /* some useful constants */ diff --git a/runtime/srutils.c b/runtime/srutils.c index f420c0f7..6a509b4a 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -92,6 +92,9 @@ syslogName_t syslogFacNames[] = { #if defined(LOG_FTP) {"ftp", LOG_FTP}, #endif +#if defined(LOG_AUDIT) + {"audit", LOG_AUDIT}, +#endif {"local0", LOG_LOCAL0}, {"local1", LOG_LOCAL1}, {"local2", LOG_LOCAL2}, @@ -524,8 +527,7 @@ char *rs_strerror_r(int errnum, char *buf, size_t buflen) { } -/* Decode a symbolic name to a numeric value - */ +/* Decode a symbolic name to a numeric value */ int decodeSyslogName(uchar *name, syslogName_t *codetab) { register syslogName_t *c; @@ -535,22 +537,23 @@ int decodeSyslogName(uchar *name, syslogName_t *codetab) ASSERT(name != NULL); ASSERT(codetab != NULL); - dbgprintf("symbolic name: %s", name); - if (isdigit((int) *name)) - { - dbgprintf("\n"); + DBGPRINTF("symbolic name: %s", name); + if(isdigit((int) *name)) { + DBGPRINTF("\n"); return (atoi((char*) name)); } strncpy((char*) buf, (char*) name, 79); - for (p = buf; *p; p++) + for(p = buf; *p; p++) { if (isupper((int) *p)) *p = tolower((int) *p); - for (c = codetab; c->c_name; c++) - if (!strcmp((char*) buf, (char*) c->c_name)) - { - dbgprintf(" ==> %d\n", c->c_val); + } + for(c = codetab; c->c_name; c++) { + if(!strcmp((char*) buf, (char*) c->c_name)) { + DBGPRINTF(" ==> %d\n", c->c_val); return (c->c_val); } + } + DBGPRINTF("\n"); return (-1); } @@ -627,6 +630,28 @@ finalize_it: RETiRet; } +/* Returns 1 if the given string contains a non-escaped glob(3) + * wildcard character and 0 otherwise (or if the string is empty). + */ +int +containsGlobWildcard(char *str) +{ + char *p; + if(!str) { + return 0; + } + /* From Linux Programmer's Guide: + * "A string is a wildcard pattern if it contains one of the characters '?', '*' or '['" + * "One can remove the special meaning of '?', '*' and '[' by preceding them by a backslash" + */ + for(p = str; *p != '\0'; p++) { + if((*p == '?' || *p == '*' || *p == '[') && + (p == str || *(p-1) != '\\')) { + return 1; + } + } + return 0; +} /* vim:set ai: */ diff --git a/runtime/stream.c b/runtime/stream.c index 3eb7708d..94fc0ca7 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -16,7 +16,7 @@ * it turns out to be problematic. Then, we need to quasi-refcount the number of accesses * to the object. * - * Copyright 2008-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -45,6 +45,7 @@ #include <pthread.h> #include <fcntl.h> #include <unistd.h> +#include <sys/types.h> #include <sys/stat.h> /* required for HP UX */ #include <errno.h> #include <pthread.h> @@ -56,6 +57,7 @@ #include "stream.h" #include "unicode-helper.h" #include "module-template.h" +#include "cryprov.h" #if HAVE_SYS_PRCTL_H # include <sys/prctl.h> #endif @@ -65,7 +67,6 @@ # define O_LARGEFILE 0 #endif #ifndef HAVE_LSEEK64 - typedef off_t off64_t; # define lseek64(fd, offset, whence) lseek(fd, offset, whence) #endif @@ -74,12 +75,14 @@ DEFobjStaticHelpers DEFobjCurrIf(zlibw) /* forward definitions */ -static rsRetVal strmFlushInternal(strm_t *pThis); +static rsRetVal strmFlushInternal(strm_t *pThis, int bFlushZip); static rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); static rsRetVal strmCloseFile(strm_t *pThis); static void *asyncWriterThread(void *pPtr); -static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); +static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush); +static rsRetVal doZipFinish(strm_t *pThis); static rsRetVal strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); +static rsRetVal strmSeekCurrOffs(strm_t *pThis); /* methods */ @@ -196,6 +199,7 @@ static rsRetVal doPhysOpen(strm_t *pThis) { int iFlags = 0; + struct stat statOpen; DEFiRet; ISOBJ_TYPE_assert(pThis, strm); @@ -233,15 +237,75 @@ doPhysOpen(strm_t *pThis) ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); else ABORT_FINALIZE(RS_RET_IO_ERROR); + } + + if(pThis->tOperationsMode == STREAMMODE_READ) { + if(fstat(pThis->fd, &statOpen) == -1) { + DBGPRINTF("Error: cannot obtain inode# for file %s\n", pThis->pszCurrFName); + ABORT_FINALIZE(RS_RET_IO_ERROR); + } + pThis->inode = statOpen.st_ino; + } + + if(!ustrcmp(pThis->pszCurrFName, UCHAR_CONSTANT(_PATH_CONSOLE)) || isatty(pThis->fd)) { + DBGPRINTF("file %d is a tty-type file\n", pThis->fd); + pThis->bIsTTY = 1; + } else { + pThis->bIsTTY = 0; + } + + if(pThis->cryprov != NULL) { + CHKiRet(pThis->cryprov->OnFileOpen(pThis->cryprovData, + pThis->pszCurrFName, &pThis->cryprovFileData)); + } +finalize_it: + RETiRet; +} + + +static rsRetVal +strmSetCurrFName(strm_t *pThis) +{ + DEFiRet; + + if(pThis->sType == STREAMTYPE_FILE_CIRCULAR) { + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, pThis->iFileNumDigits)); } else { - if(!ustrcmp(pThis->pszCurrFName, UCHAR_CONSTANT(_PATH_CONSOLE)) || isatty(pThis->fd)) { - DBGPRINTF("file %d is a tty-type file\n", pThis->fd); - pThis->bIsTTY = 1; + if(pThis->pszDir == NULL) { + if((pThis->pszCurrFName = ustrdup(pThis->pszFName)) == NULL) + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } else { - pThis->bIsTTY = 0; + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, -1, 0)); } } +finalize_it: + RETiRet; +} + +/* This function checks if the actual file has changed and, if so, resets the + * offset. This is support for monitoring files. It should be called after + * deserializing the strm object and before doing any other operation on it + * (most importantly not an open or seek!). + */ +static rsRetVal +CheckFileChange(strm_t *pThis) +{ + struct stat statName; + DEFiRet; + CHKiRet(strmSetCurrFName(pThis)); + if(stat((char*) pThis->pszCurrFName, &statName) == -1) + ABORT_FINALIZE(RS_RET_IO_ERROR); + DBGPRINTF("stream/after deserialize checking for file change on '%s', " + "inode %u/%u, size/currOffs %llu/%llu\n", + pThis->pszCurrFName, (unsigned) pThis->inode, + (unsigned) statName.st_ino, statName.st_size, pThis->iCurrOffs); + if(pThis->inode != statName.st_ino || statName.st_size < pThis->iCurrOffs) { + DBGPRINTF("stream: file %s has changed\n", pThis->pszCurrFName); + pThis->iCurrOffs = 0; + } finalize_it: RETiRet; } @@ -264,19 +328,8 @@ static rsRetVal strmOpenFile(strm_t *pThis) if(pThis->pszFName == NULL) ABORT_FINALIZE(RS_RET_FILE_PREFIX_MISSING); - if(pThis->sType == STREAMTYPE_FILE_CIRCULAR) { - CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, - pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, pThis->iFileNumDigits)); - } else { - if(pThis->pszDir == NULL) { - if((pThis->pszCurrFName = ustrdup(pThis->pszFName)) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } else { - CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, - pThis->pszFName, pThis->lenFName, -1, 0)); - } - } - + CHKiRet(strmSetCurrFName(pThis)); + CHKiRet(doPhysOpen(pThis)); pThis->iCurrOffs = 0; @@ -334,6 +387,7 @@ strmWaitAsyncWriterDone(strm_t *pThis) */ static rsRetVal strmCloseFile(strm_t *pThis) { + off64_t currOffs; DEFiRet; ASSERT(pThis != NULL); @@ -341,7 +395,10 @@ static rsRetVal strmCloseFile(strm_t *pThis) (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName); if(pThis->tOperationsMode != STREAMMODE_READ) { - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); + if(pThis->iZipLevel) { + doZipFinish(pThis); + } if(pThis->bAsyncWrite) { strmWaitAsyncWriterDone(pThis); } @@ -351,8 +408,14 @@ static rsRetVal strmCloseFile(strm_t *pThis) * against this. -- rgerhards, 2010-03-19 */ if(pThis->fd != -1) { + currOffs = lseek64(pThis->fd, 0, SEEK_CUR); close(pThis->fd); pThis->fd = -1; + pThis->inode = 0; + if(pThis->cryprov != NULL) { + pThis->cryprov->OnFileClose(pThis->cryprovFileData, currOffs); + pThis->cryprovFileData = NULL; + } } if(pThis->fdDir != -1) { @@ -361,7 +424,13 @@ static rsRetVal strmCloseFile(strm_t *pThis) pThis->fdDir = -1; } - if(pThis->bDeleteOnClose && pThis->pszCurrFName != NULL) { + if(pThis->bDeleteOnClose) { + if(pThis->pszCurrFName == NULL) { + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, + pThis->iFileNumDigits)); + } + DBGPRINTF("strmCloseFile: deleting '%s'\n", pThis->pszCurrFName); if(unlink((char*) pThis->pszCurrFName) == -1) { char errStr[1024]; int err = errno; @@ -369,12 +438,13 @@ static rsRetVal strmCloseFile(strm_t *pThis) DBGPRINTF("error %d unlinking '%s' - ignored: %s\n", errno, pThis->pszCurrFName, errStr); } - free(pThis->pszCurrFName); /* no longer needed in any case (just for open) */ + free(pThis->pszCurrFName); pThis->pszCurrFName = NULL; } pThis->iCurrOffs = 0; /* we are back at begin of file */ +finalize_it: RETiRet; } @@ -421,18 +491,15 @@ static rsRetVal strmHandleEOFMonitor(strm_t *pThis) { DEFiRet; - struct stat statOpen; struct stat statName; ISOBJ_TYPE_assert(pThis, strm); - if(fstat(pThis->fd, &statOpen) == -1) - ABORT_FINALIZE(RS_RET_IO_ERROR); if(stat((char*) pThis->pszCurrFName, &statName) == -1) ABORT_FINALIZE(RS_RET_IO_ERROR); - DBGPRINTF("stream checking for file change on '%s', inode %u/%u", - pThis->pszCurrFName, (unsigned) statOpen.st_ino, + DBGPRINTF("stream checking for file change on '%s', inode %u/%u\n", + pThis->pszCurrFName, (unsigned) pThis->inode, (unsigned) statName.st_ino); - if(statOpen.st_ino == statName.st_ino) { + if(pThis->inode == statName.st_ino) { ABORT_FINALIZE(RS_RET_EOF); } else { /* we had a file change! */ @@ -585,25 +652,33 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) * mode = 2 LF <not whitespace> mode, a log line starts at the beginning of a line, but following lines that are indented are part of the same log entry * This modal interface is not nearly as flexible as being able to define a regex for when a new record starts, but it's also not nearly as hard (or as slow) to implement */ - DEFiRet; uchar c; uchar finished; + rsRetVal readCharRet; + DEFiRet; ASSERT(pThis != NULL); ASSERT(ppCStr != NULL); CHKiRet(cstrConstruct(ppCStr)); - - /* now read the line */ CHKiRet(strmReadChar(pThis, &c)); - if (mode == 0){ - while(c != '\n') { + + if(mode == 0) { + /* append previous message to current message if necessary */ + if(pThis->prevLineSegment != NULL) { + CHKiRet(cstrAppendCStr(*ppCStr, pThis->prevLineSegment)); + cstrDestruct(&pThis->prevLineSegment); + } + while(c != '\n') { CHKiRet(cstrAppendChar(*ppCStr, c)); - CHKiRet(strmReadChar(pThis, &c)); + readCharRet = strmReadChar(pThis, &c); + if(readCharRet == RS_RET_EOF) {/* end of file reached without \n? */ + CHKiRet(rsCStrConstructFromCStr(&pThis->prevLineSegment, *ppCStr)); + } + CHKiRet(readCharRet); } CHKiRet(cstrFinalize(*ppCStr)); - } - if (mode == 1){ + } else if(mode == 1) { finished=0; while(finished == 0){ if(c != '\n') { @@ -624,8 +699,7 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) } } CHKiRet(cstrFinalize(*ppCStr)); - } - if (mode == 2){ + } else if(mode == 2) { /* indented follow-up lines */ finished=0; while(finished == 0){ @@ -675,9 +749,11 @@ BEGINobjConstruct(strm) /* be sure to specify the object type also in END macro! pThis->fd = -1; pThis->fdDir = -1; pThis->iUngetC = -1; + pThis->bVeryReliableZip = 0; pThis->sType = STREAMTYPE_FILE_SINGLE; pThis->sIOBufSize = glblGetIOBufSize(); pThis->tOpenMode = 0600; + pThis->prevLineSegment = NULL; ENDobjConstruct(strm) @@ -777,6 +853,7 @@ stopWriter(strm_t *pThis) BEGINobjDestruct(strm) /* be sure to specify the object type also in END and CODESTART macros! */ int i; CODESTARTobjDestruct(strm) + /* we need to stop the ZIP writer */ if(pThis->bAsyncWrite) /* Note: mutex will be unlocked in stopWriter! */ d_pthread_mutex_lock(&pThis->mut); @@ -919,14 +996,14 @@ finalize_it: /* write memory buffer to a stream object. */ static inline rsRetVal -doWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf) +doWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush) { DEFiRet; ASSERT(pThis != NULL); if(pThis->iZipLevel) { - CHKiRet(doZipWrite(pThis, pBuf, lenBuf)); + CHKiRet(doZipWrite(pThis, pBuf, lenBuf, bFlush)); } else { /* write without zipping */ CHKiRet(strmPhysWrite(pThis, pBuf, lenBuf)); @@ -971,7 +1048,7 @@ doAsyncWriteInternal(strm_t *pThis, size_t lenBuf) * the background thread. -- rgerhards, 2009-07-07 */ static rsRetVal -strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlushZip) { DEFiRet; @@ -990,7 +1067,7 @@ strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) if(pThis->bAsyncWrite) { CHKiRet(doAsyncWriteInternal(pThis, lenBuf)); } else { - CHKiRet(doWriteInternal(pThis, pBuf, lenBuf)); + CHKiRet(doWriteInternal(pThis, pBuf, lenBuf, bFlushZip)); } @@ -1011,17 +1088,20 @@ asyncWriterThread(void *pPtr) sbool bTimedOut = 0; strm_t *pThis = (strm_t*) pPtr; int err; + uchar thrdName[256] = "rs:"; ISOBJ_TYPE_assert(pThis, strm); BEGINfunc + ustrncpy(thrdName+3, pThis->pszFName, sizeof(thrdName)-4); + dbgOutputTID((char*)thrdName); # if HAVE_PRCTL && defined PR_SET_NAME - if(prctl(PR_SET_NAME, "rs:asyn strmwr", 0, 0, 0) != 0) { + if(prctl(PR_SET_NAME, (char*)thrdName, 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", "stream writer"); } # endif + d_pthread_mutex_lock(&pThis->mut); while(1) { /* loop broken inside */ - d_pthread_mutex_lock(&pThis->mut); while(pThis->iCnt == 0) { if(pThis->bStopWriter) { pthread_cond_broadcast(&pThis->isEmpty); @@ -1030,18 +1110,17 @@ asyncWriterThread(void *pPtr) } if(bTimedOut && pThis->iBufPtr > 0) { /* if we timed out, we need to flush pending data */ - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); bTimedOut = 0; - continue; /* now we should have data */ + d_pthread_mutex_unlock(&pThis->mut); + continue; } bTimedOut = 0; timeoutComp(&t, pThis->iFlushInterval * 1000); /* *1000 millisconds */ if(pThis->bDoTimedWait) { if((err = pthread_cond_timedwait(&pThis->notEmpty, &pThis->mut, &t)) != 0) { - if(err == ETIMEDOUT) { - bTimedOut = 1; - } else { - bTimedOut = 1; + bTimedOut = 1; /* simulate in any case */ + if(err != ETIMEDOUT) { char errStr[1024]; rs_strerror_r(err, errStr, sizeof(errStr)); DBGPRINTF("stream async writer timeout with error (%d): %s - ignoring\n", @@ -1056,8 +1135,12 @@ asyncWriterThread(void *pPtr) bTimedOut = 0; /* we may have timed out, but there *is* work to do... */ iDeq = pThis->iDeq++ % STREAM_ASYNC_NUMBUFS; - doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf); + + /* now we can do the actual write in parallel */ + d_pthread_mutex_unlock(&pThis->mut); + doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf, 0); // TODO: flush state // TODO: error check????? 2009-07-06 + d_pthread_mutex_lock(&pThis->mut); --pThis->iCnt; if(pThis->iCnt < STREAM_ASYNC_NUMBUFS) { @@ -1065,8 +1148,8 @@ asyncWriterThread(void *pPtr) if(pThis->iCnt == 0) pthread_cond_broadcast(&pThis->isEmpty); } - d_pthread_mutex_unlock(&pThis->mut); } + d_pthread_mutex_unlock(&pThis->mut); finalize_it: ENDfunc @@ -1128,9 +1211,16 @@ strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) DEFiRet; ISOBJ_TYPE_assert(pThis, strm); + DBGPRINTF("strmPhysWrite, stream %p, len %u\n", pThis, (unsigned)lenBuf); if(pThis->fd == -1) CHKiRet(strmOpenFile(pThis)); + /* here we place our crypto interface */ + if(pThis->cryprov != NULL) { + pThis->cryprov->Encrypt(pThis->cryprovFileData, pBuf, &lenBuf); + } + /* end crypto */ + iWritten = lenBuf; CHKiRet(doWriteCall(pThis, pBuf, &iWritten)); @@ -1166,63 +1256,97 @@ finalize_it: * rgerhards, 2009-06-04 */ static rsRetVal -doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf, int bFlush) { - z_stream zstrm; int zRet; /* zlib return state */ - sbool bzInitDone = RSFALSE; DEFiRet; + unsigned outavail; assert(pThis != NULL); assert(pBuf != NULL); - /* allocate deflate state */ - zstrm.zalloc = Z_NULL; - zstrm.zfree = Z_NULL; - zstrm.opaque = Z_NULL; - zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ - /* see note in file header for the params we use with deflateInit2() */ - zRet = zlibw.DeflateInit2(&zstrm, pThis->iZipLevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); - if(zRet != Z_OK) { - DBGPRINTF("error %d returned from zlib/deflateInit2()\n", zRet); - ABORT_FINALIZE(RS_RET_ZLIB_ERR); + if(!pThis->bzInitDone) { + /* allocate deflate state */ + pThis->zstrm.zalloc = Z_NULL; + pThis->zstrm.zfree = Z_NULL; + pThis->zstrm.opaque = Z_NULL; + /* see note in file header for the params we use with deflateInit2() */ + zRet = zlibw.DeflateInit2(&pThis->zstrm, pThis->iZipLevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateInit2()\n", zRet); + ABORT_FINALIZE(RS_RET_ZLIB_ERR); + } + pThis->bzInitDone = RSTRUE; } - bzInitDone = RSTRUE; /* now doing the compression */ - zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ - zstrm.avail_in = lenBuf; + pThis->zstrm.next_in = (Bytef*) pBuf; + pThis->zstrm.avail_in = lenBuf; /* run deflate() on buffer until everything has been compressed */ do { - DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", zstrm.avail_in, zstrm.total_in); - zstrm.avail_out = pThis->sIOBufSize; - zstrm.next_out = pThis->pZipBuf; - zRet = zlibw.Deflate(&zstrm, Z_FINISH); /* no bad return value */ - DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, zstrm.avail_out); - assert(zRet != Z_STREAM_ERROR); /* state not clobbered */ - if(zstrm.avail_out == pThis->sIOBufSize) - break; /* this is valid, indicates end of compression --> see zlib howto */ - CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, pThis->sIOBufSize - zstrm.avail_out)); - } while (zstrm.avail_out == 0); - assert(zstrm.avail_in == 0); /* all input will be used */ + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", pThis->zstrm.avail_in, pThis->zstrm.total_in); + pThis->zstrm.avail_out = pThis->sIOBufSize; + pThis->zstrm.next_out = pThis->pZipBuf; + zRet = zlibw.Deflate(&pThis->zstrm, bFlush ? Z_SYNC_FLUSH : Z_NO_FLUSH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pThis->zstrm.avail_out); + outavail =pThis->sIOBufSize - pThis->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, outavail)); + } + } while (pThis->zstrm.avail_out == 0); finalize_it: - if(bzInitDone) { - zRet = zlibw.DeflateEnd(&zstrm); - if(zRet != Z_OK) { - DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); - } + if(pThis->bzInitDone && pThis->bVeryReliableZip) { + doZipFinish(pThis); } - RETiRet; } + +/* finish zlib buffer, to be called before closing the ZIP file (if + * running in stream mode). + */ +static rsRetVal +doZipFinish(strm_t *pThis) +{ + int zRet; /* zlib return state */ + DEFiRet; + unsigned outavail; + assert(pThis != NULL); + + if(!pThis->bzInitDone) + goto done; + + pThis->zstrm.avail_in = 0; + /* run deflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", pThis->zstrm.avail_in, pThis->zstrm.total_in); + pThis->zstrm.avail_out = pThis->sIOBufSize; + pThis->zstrm.next_out = pThis->pZipBuf; + zRet = zlibw.Deflate(&pThis->zstrm, Z_FINISH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pThis->zstrm.avail_out); + outavail = pThis->sIOBufSize - pThis->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, outavail)); + } + } while (pThis->zstrm.avail_out == 0); + +finalize_it: + zRet = zlibw.DeflateEnd(&pThis->zstrm); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); + } + + pThis->bzInitDone = 0; +done: RETiRet; +} + /* flush stream output buffer to persistent storage. This can be called at any time * and is automatically called when the output buffer is full. * rgerhards, 2008-01-10 */ static rsRetVal -strmFlushInternal(strm_t *pThis) +strmFlushInternal(strm_t *pThis, int bFlushZip) { DEFiRet; @@ -1232,7 +1356,7 @@ strmFlushInternal(strm_t *pThis) (long) pThis->iBufPtr, (pThis->iBufPtr == 0) ? " (no need to flush)" : ""); if(pThis->tOperationsMode != STREAMMODE_READ && pThis->iBufPtr > 0) { - iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr); + iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr, bFlushZip); } RETiRet; @@ -1254,7 +1378,7 @@ strmFlush(strm_t *pThis) if(pThis->bAsyncWrite) d_pthread_mutex_lock(&pThis->mut); - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 1)); finalize_it: if(pThis->bAsyncWrite) @@ -1277,11 +1401,15 @@ static rsRetVal strmSeek(strm_t *pThis, off64_t offs) if(pThis->fd == -1) { CHKiRet(strmOpenFile(pThis)); } else { - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 0)); } long long i; DBGOPRINT((obj_t*) pThis, "file %d seek, pos %llu\n", pThis->fd, (long long unsigned) offs); - i = lseek64(pThis->fd, offs, SEEK_SET); // TODO: check error! + i = lseek64(pThis->fd, offs, SEEK_SET); + if(i != offs) { + DBGPRINTF("strmSeek: error %lld seeking to offset %lld\n", i, offs); + ABORT_FINALIZE(RS_RET_IO_ERROR); + } pThis->iCurrOffs = offs; /* we are now at *this* offset */ pThis->iBufPtr = 0; /* buffer invalidated */ @@ -1289,6 +1417,56 @@ finalize_it: RETiRet; } +/* multi-file seek, seeks to file number & offset within file. This + * is a support function for the queue, in circular mode. DO NOT USE + * IT FOR OTHER NEEDS - it may not work as expected. It will + * seek to the new position and delete interim files, as it skips them. + * Note: this code can be removed when the queue gets a new disk store + * handler (if and when it does ;)). + * The output parameter bytesDel receives the number of bytes that have + * been deleted (if a file is deleted) or 0 if nothing was deleted. + * rgerhards, 2012-11-07 + */ +rsRetVal +strmMultiFileSeek(strm_t *pThis, int FNum, off64_t offs, off64_t *bytesDel) +{ + struct stat statBuf; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, strm); + + if(FNum == 0 && offs == 0) { /* happens during queue init */ + *bytesDel = 0; + FINALIZE; + } + + if(pThis->iCurrFNum != FNum) { + /* Note: we assume that no more than one file is skipped - an + * assumption that is being used also by the whole rest of the + * code and most notably the queue subsystem. + */ + CHKiRet(genFileName(&pThis->pszCurrFName, pThis->pszDir, pThis->lenDir, + pThis->pszFName, pThis->lenFName, pThis->iCurrFNum, + pThis->iFileNumDigits)); + stat((char*)pThis->pszCurrFName, &statBuf); + *bytesDel = statBuf.st_size; + DBGPRINTF("strmMultiFileSeek: detected new filenum, was %d, new %d, " + "deleting '%s' (%lld bytes)\n", pThis->iCurrFNum, FNum, + pThis->pszCurrFName, (long long) *bytesDel); + unlink((char*)pThis->pszCurrFName); + free(pThis->pszCurrFName); + pThis->pszCurrFName = NULL; + pThis->iCurrFNum = FNum; + } else { + *bytesDel = 0; + } + pThis->iCurrOffs = offs; + +finalize_it: + RETiRet; +} + + /* seek to current offset. This is primarily a helper to readjust the OS file * pointer after a strm object has been deserialized. @@ -1320,7 +1498,7 @@ static rsRetVal strmWriteChar(strm_t *pThis, uchar c) /* if the buffer is full, we need to flush before we can write */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); + CHKiRet(strmFlushInternal(pThis, 0)); } /* we now always have space for one character, so we simply copy it */ *(pThis->pIOBuf + pThis->iBufPtr) = c; @@ -1380,17 +1558,17 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) ASSERT(pThis != NULL); ASSERT(pBuf != NULL); -//DBGPRINTF("strmWrite(%p, '%65.65s', %ld);, disabled %d, sizelim %ld, size %lld\n", pThis, pBuf,lenBuf, pThis->bDisabled, pThis->iSizeLimit, pThis->iCurrOffs); - if(pThis->bAsyncWrite) - d_pthread_mutex_lock(&pThis->mut); - + /* DEV DEBUG ONLY DBGPRINTF("strmWrite(%p[%s], '%65.65s', %ld);, disabled %d, sizelim %ld, size %lld\n", pThis, pThis->pszCurrFName, pBuf,(long) lenBuf, pThis->bDisabled, (long) pThis->iSizeLimit, (long long) pThis->iCurrOffs); */ if(pThis->bDisabled) ABORT_FINALIZE(RS_RET_STREAM_DISABLED); + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + iOffset = 0; do { if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis, 0)); /* get a new buffer for rest of data */ } iWrite = pThis->sIOBufSize - pThis->iBufPtr; /* this fits in current buf */ if(iWrite > lenBuf) @@ -1405,7 +1583,7 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) * write it. This seems more natural than waiting (hours?) for the next message... */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis, 0)); /* get a new buffer for rest of data */ } finalize_it: @@ -1433,11 +1611,14 @@ DEFpropSetMeth(strm, tOperationsMode, int) DEFpropSetMeth(strm, tOpenMode, mode_t) DEFpropSetMeth(strm, sType, strmType_t) DEFpropSetMeth(strm, iZipLevel, int) +DEFpropSetMeth(strm, bVeryReliableZip, int) DEFpropSetMeth(strm, bSync, int) DEFpropSetMeth(strm, sIOBufSize, size_t) DEFpropSetMeth(strm, iSizeLimit, off_t) DEFpropSetMeth(strm, iFlushInterval, int) DEFpropSetMeth(strm, pszSizeLimitCmd, uchar*) +DEFpropSetMeth(strm, cryprov, cryprov_if_t*) +DEFpropSetMeth(strm, cryprovData, void*) static rsRetVal strmSetiMaxFiles(strm_t *pThis, int iNewVal) { @@ -1564,7 +1745,7 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) ISOBJ_TYPE_assert(pThis, strm); ISOBJ_TYPE_assert(pStrm, strm); - strmFlushInternal(pThis); + strmFlushInternal(pThis, 0); CHKiRet(obj.BeginSerialize(pStrm, (obj_t*) pThis)); objSerializeSCALAR(pStrm, iCurrFNum, INT); @@ -1584,6 +1765,11 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) l = pThis->iCurrOffs; objSerializeSCALAR_VAR(pStrm, iCurrOffs, INT64, l); + l = pThis->inode; + objSerializeSCALAR_VAR(pStrm, inode, INT64, l); + + objSerializePTR(pStrm, prevLineSegment, PSZ); + CHKiRet(obj.EndSerialize(pStrm)); finalize_it: @@ -1681,6 +1867,8 @@ static rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) CHKiRet(strmSettOpenMode(pThis, pProp->val.num)); } else if(isProp("iCurrOffs")) { pThis->iCurrOffs = pProp->val.num; + } else if(isProp("inode")) { + pThis->inode = (ino_t) pProp->val.num; } else if(isProp("iMaxFileSize")) { CHKiRet(strmSetiMaxFileSize(pThis, pProp->val.num)); } else if(isProp("iMaxFiles")) { @@ -1689,6 +1877,8 @@ static rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) CHKiRet(strmSetiFileNumDigits(pThis, pProp->val.num)); } else if(isProp("bDeleteOnClose")) { CHKiRet(strmSetbDeleteOnClose(pThis, pProp->val.num)); + } else if(isProp("prevLineSegment")) { + CHKiRet(rsCStrConstructFromCStr(&pThis->prevLineSegment, pProp->val.pStr)); } finalize_it: @@ -1748,6 +1938,7 @@ CODESTARTobjQueryInterface(strm) pIf->GetCurrOffset = strmGetCurrOffset; pIf->Dup = strmDup; pIf->SetWCntr = strmSetWCntr; + pIf->CheckFileChange = CheckFileChange; /* set methods */ pIf->SetbDeleteOnClose = strmSetbDeleteOnClose; pIf->SetiMaxFileSize = strmSetiMaxFileSize; @@ -1757,11 +1948,14 @@ CODESTARTobjQueryInterface(strm) pIf->SettOpenMode = strmSettOpenMode; pIf->SetsType = strmSetsType; pIf->SetiZipLevel = strmSetiZipLevel; + pIf->SetbVeryReliableZip = strmSetbVeryReliableZip; pIf->SetbSync = strmSetbSync; pIf->SetsIOBufSize = strmSetsIOBufSize; pIf->SetiSizeLimit = strmSetiSizeLimit; pIf->SetiFlushInterval = strmSetiFlushInterval; pIf->SetpszSizeLimitCmd = strmSetpszSizeLimitCmd; + pIf->Setcryprov = strmSetcryprov; + pIf->SetcryprovData = strmSetcryprovData; finalize_it: ENDobjQueryInterface(strm) diff --git a/runtime/stream.h b/runtime/stream.h index 0828f6fd..4f4a4301 100644 --- a/runtime/stream.h +++ b/runtime/stream.h @@ -41,7 +41,7 @@ * deflateInit2(zstrmptr, 6, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); * -------------------------------------------------------------------------- * - * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -70,6 +70,7 @@ #include "glbl.h" #include "stream.h" #include "zlibw.h" +#include "cryprov.h" /* stream types */ typedef enum { @@ -112,6 +113,7 @@ typedef struct strm_s { int lenDir; int fd; /* the file descriptor, -1 if closed */ int fdDir; /* the directory's descriptor, in case bSync is requested (-1 if closed) */ + ino_t inode; /* current inode for files being monitored (undefined else) */ uchar *pszCurrFName; /* name of current file (if open) */ uchar *pIOBuf; /* the iobuffer currently in use to gather data */ size_t iBufPtrMax; /* current max Ptr in Buffer (if partial read!) */ @@ -124,6 +126,8 @@ typedef struct strm_s { sbool bAsyncWrite; /* do asynchronous writes (always if a flush interval is given) */ sbool bStopWriter; /* shall writer thread terminate? */ sbool bDoTimedWait; /* instruct writer thread to do a times wait to support flush timeouts */ + sbool bzInitDone; /* did we do an init of zstrm already? */ + sbool bVeryReliableZip; /* shall we write interim headers to create a very reliable ZIP file? */ int iFlushInterval; /* flush in which interval - 0, no flushing */ pthread_mutex_t mut;/* mutex for flush in async mode */ pthread_cond_t notFull; @@ -131,7 +135,11 @@ typedef struct strm_s { pthread_cond_t isEmpty; unsigned short iEnq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ unsigned short iDeq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ + cryprov_if_t *cryprov; /* ptr to crypto provider; NULL = do not encrypt */ + void *cryprovData; /* opaque data ptr for provider use */ + void *cryprovFileData;/* opaque data ptr for file instance */ short iCnt; /* current nbr of elements in buffer */ + z_stream zstrm; /* zip stream to use */ struct { uchar *pBuf; size_t lenBuf; @@ -141,6 +149,7 @@ typedef struct strm_s { off_t iSizeLimit; /* file size limit, 0 = no limit */ uchar *pszSizeLimitCmd; /* command to carry out when size limit is reached */ sbool bIsTTY; /* is this a tty file? */ + cstr_t *prevLineSegment; /* for ReadLine, previous, unwritten part of file */ } strm_t; @@ -180,11 +189,23 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ INTERFACEpropSetMeth(strm, pszSizeLimitCmd, uchar*); /* v6 added */ rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, int mode); + /* v7 added 2012-09-14 */ + INTERFACEpropSetMeth(strm, bVeryReliableZip, int); + /* v8 added 2013-03-21 */ + rsRetVal (*CheckFileChange)(strm_t *pThis); + /* v9 added 2013-04-04 */ + INTERFACEpropSetMeth(strm, cryprov, cryprov_if_t*); + INTERFACEpropSetMeth(strm, cryprovData, void*); ENDinterface(strm) -#define strmCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define strmCURR_IF_VERSION 9 /* increment whenever you change the interface structure! */ +static inline int +strmGetCurrFileNum(strm_t *pStrm) { + return pStrm->iCurrFNum; +} /* prototypes */ PROTOTYPEObjClassInit(strm); +rsRetVal strmMultiFileSeek(strm_t *pThis, int fileNum, off64_t offs, off64_t *bytesDel); #endif /* #ifndef STREAM_H_INCLUDED */ diff --git a/runtime/stringbuf.c b/runtime/stringbuf.c index e7fd72c2..cb4f0457 100644 --- a/runtime/stringbuf.c +++ b/runtime/stringbuf.c @@ -32,6 +32,7 @@ #include <assert.h> #include <string.h> #include <ctype.h> +#include <stdarg.h> #include <sys/types.h> #include <libestr.h> #include "rsyslog.h" @@ -104,6 +105,55 @@ finalize_it: } +/* a helper function for rsCStr*Strf() + */ +static rsRetVal rsCStrConstructFromszStrv(cstr_t **ppThis, uchar *fmt, va_list ap) +{ + DEFiRet; + cstr_t *pThis; + va_list ap2; + int len; + + assert(ppThis != NULL); + + va_copy(ap2, ap); + len = vsnprintf(NULL, 0, (char*)fmt, ap2); + va_end(ap2); + + if(len < 0) + ABORT_FINALIZE(RS_RET_ERR); + + CHKiRet(rsCStrConstruct(&pThis)); + + pThis->iBufSize = pThis->iStrLen = len; + len++; /* account for the \0 written by vsnprintf */ + if((pThis->pBuf = (uchar*) MALLOC(sizeof(uchar) * len)) == NULL) { + RSFREEOBJ(pThis); + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } + + vsnprintf((char*)pThis->pBuf, len, (char*)fmt, ap); + *ppThis = pThis; +finalize_it: + RETiRet; +} + + +/* construct from a printf-style formated string + */ +rsRetVal rsCStrConstructFromszStrf(cstr_t **ppThis, char *fmt, ...) +{ + DEFiRet; + va_list ap; + + va_start(ap, fmt); + iRet = rsCStrConstructFromszStrv(ppThis, (uchar*)fmt, ap); + va_end(ap); + + RETiRet; +} + + /* construct from es_str_t string * rgerhards 2010-12-03 */ @@ -256,6 +306,27 @@ rsRetVal cstrAppendCStr(cstr_t *pThis, cstr_t *pstrAppend) } +/* append a printf-style formated string + */ +rsRetVal rsCStrAppendStrf(cstr_t *pThis, uchar *fmt, ...) +{ + DEFiRet; + va_list ap; + cstr_t *pStr = NULL; + + va_start(ap, fmt); + iRet = rsCStrConstructFromszStrv(&pStr, fmt, ap); + va_end(ap); + + CHKiRet(iRet); + + iRet = cstrAppendCStr(pThis, pStr); + rsCStrDestruct(&pStr); +finalize_it: + RETiRet; +} + + rsRetVal rsCStrAppendInt(cstr_t *pThis, long i) { DEFiRet; @@ -482,6 +553,8 @@ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) register uchar *pC; rsCHECKVALIDOBJECT(pThis, OIDrsCStr); + if(pThis->iStrLen == 0) + goto done; /* empty string -> nothing to trim ;) */ i = pThis->iStrLen; pC = pThis->pBuf + i - 1; while(i > 0 && isspace((int)*pC)) { @@ -492,7 +565,7 @@ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) pThis->iStrLen = i; pThis->pBuf[pThis->iStrLen] = '0'; /* we always have this space */ - return RS_RET_OK; +done: return RS_RET_OK; } /* compare two string objects - works like strcmp(), but operates @@ -868,13 +941,7 @@ int rsCStrSzStrCmp(cstr_t *pCS1, uchar *psz, size_t iLenSz) * length, so we need to actually check if they * are equal. */ - register size_t i; - for(i = 0 ; i < iLenSz ; ++i) { - if(pCS1->pBuf[i] != psz[i]) - return pCS1->pBuf[i] - psz[i]; - } - /* if we arrive here, the strings are equal */ - return 0; + return strncmp((char*)pCS1->pBuf, (char*)psz, iLenSz); } else return pCS1->iStrLen - iLenSz; diff --git a/runtime/stringbuf.h b/runtime/stringbuf.h index bba004a0..d0502a5b 100644 --- a/runtime/stringbuf.h +++ b/runtime/stringbuf.h @@ -58,6 +58,7 @@ rsRetVal cstrConstruct(cstr_t **ppThis); rsRetVal cstrConstructFromESStr(cstr_t **ppThis, es_str_t *str); rsRetVal rsCStrConstructFromszStr(cstr_t **ppThis, uchar *sz); rsRetVal rsCStrConstructFromCStr(cstr_t **ppThis, cstr_t *pFrom); +rsRetVal rsCStrConstructFromszStrf(cstr_t **ppThis, char *fmt, ...) __attribute__((format(printf,2, 3))); /** * Destruct the string buffer object. @@ -173,6 +174,12 @@ rsRetVal rsCStrAppendStr(cstr_t *pThis, uchar* psz); */ rsRetVal rsCStrAppendStrWithLen(cstr_t *pThis, uchar* psz, size_t iStrLen); +/** + * Append a printf-style formated string to the buffer. + * + * \param fmt pointer to the format string (see man 3 printf for details). Must not be NULL. + */ +rsRetVal rsCStrAppendStrf(cstr_t *pThis, uchar *fmt, ...); /** * Append an integer to the string. No special formatting is diff --git a/runtime/strms_sess.c b/runtime/strms_sess.c index d14f0b37..2537e8d8 100644 --- a/runtime/strms_sess.c +++ b/runtime/strms_sess.c @@ -38,12 +38,14 @@ #include "errmsg.h" #include "netstrm.h" #include "msg.h" +#include "prop.h" #include "datetime.h" /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) DEFobjCurrIf(errmsg) DEFobjCurrIf(netstrm) DEFobjCurrIf(datetime) @@ -86,7 +88,8 @@ CODESTARTobjDestruct(strms_sess) } /* now destruct our own properties */ free(pThis->fromHost); - free(pThis->fromHostIP); + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); ENDobjDestruct(strms_sess) @@ -111,17 +114,18 @@ SetHost(strms_sess_t *pThis, uchar *pszHost) RETiRet; } -/* set the remote host's IP. Note that the caller *hands over* the string. That is, +/* set the remote host's IP. Note that the caller *hands over* the property. That is, * the caller no longer controls it once SetHostIP() has received it. Most importantly, - * the caller must not free it. -- rgerhards, 2008-05-16 + * the caller must not destruct it. -- rgerhards, 2008-05-16 */ static rsRetVal -SetHostIP(strms_sess_t *pThis, uchar *pszHostIP) +SetHostIP(strms_sess_t *pThis, prop_t *ip) { DEFiRet; ISOBJ_TYPE_assert(pThis, strms_sess); - free(pThis->fromHostIP); - pThis->fromHostIP = pszHostIP; + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); + pThis->fromHostIP = ip; RETiRet; } @@ -188,8 +192,8 @@ Close(strms_sess_t *pThis) netstrm.Destruct(&pThis->pStrm); free(pThis->fromHost); pThis->fromHost = NULL; /* not really needed, but... */ - free(pThis->fromHostIP); - pThis->fromHostIP = NULL; /* not really needed, but... */ + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); RETiRet; } @@ -284,6 +288,7 @@ BEGINObjClassInit(strms_sess, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANGE CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(netstrm, LM_NETSTRMS_FILENAME)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); iMaxLine = glbl.GetMaxLine(); /* get maximum size we currently support */ diff --git a/runtime/strms_sess.h b/runtime/strms_sess.h index 5c0309f8..86f692a8 100644 --- a/runtime/strms_sess.h +++ b/runtime/strms_sess.h @@ -33,9 +33,8 @@ struct strms_sess_s { strmsrv_t *pSrv; /* pointer back to my server (e.g. for callbacks) */ strmLstnPortList_t *pLstnInfo; /* pointer back to listener info */ netstrm_t *pStrm; -// uchar *pMsg; /* message (fragment) received */ uchar *fromHost; - uchar *fromHostIP; + prop_t *fromHostIP; void *pUsr; /* a user-pointer */ }; @@ -54,15 +53,17 @@ BEGINinterface(strms_sess) /* name must also be changed in ENDinterface macro! * rsRetVal (*SetUsrP)(strms_sess_t*, void*); void* (*GetUsrP)(strms_sess_t*); rsRetVal (*SetHost)(strms_sess_t *pThis, uchar*); - rsRetVal (*SetHostIP)(strms_sess_t *pThis, uchar*); + rsRetVal (*SetHostIP)(strms_sess_t *pThis, prop_t*); rsRetVal (*SetStrm)(strms_sess_t *pThis, netstrm_t*); rsRetVal (*SetOnMsgReceive)(strms_sess_t *pThis, rsRetVal (*OnMsgReceive)(strms_sess_t*, uchar*, int)); ENDinterface(strms_sess) -#define strms_sessCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define strms_sessCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface changes * to version v2, rgerhards, 2009-05-22 * - Data structures changed * - SetLstnInfo entry point added + * version 3, rgerhads, 2013-01-21: + * - signature of SetHostIP() changed */ diff --git a/runtime/strmsrv.c b/runtime/strmsrv.c index 8310e832..e8b544b8 100644 --- a/runtime/strmsrv.c +++ b/runtime/strmsrv.c @@ -70,6 +70,7 @@ #include "netstrm.h" #include "nssel.h" #include "errmsg.h" +#include "prop.h" #include "unicode-helper.h" MODULE_TYPE_LIB @@ -89,6 +90,7 @@ DEFobjCurrIf(net) DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) DEFobjCurrIf(nssel) +DEFobjCurrIf(prop) /* forward definitions */ static rsRetVal create_strm_socket(strmsrv_t *pThis); @@ -418,7 +420,7 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes int iSess = -1; struct sockaddr_storage *addr; uchar *fromHostFQDN = NULL; - uchar *fromHostIP = NULL; + prop_t *ip = NULL; ISOBJ_TYPE_assert(pThis, strmsrv); assert(pLstnInfo != NULL); @@ -444,7 +446,7 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes /* get the host name */ CHKiRet(netstrm.GetRemoteHName(pNewStrm, &fromHostFQDN)); - CHKiRet(netstrm.GetRemoteIP(pNewStrm, &fromHostIP)); + CHKiRet(netstrm.GetRemoteIP(pNewStrm, &ip)); CHKiRet(netstrm.GetRemAddr(pNewStrm, &addr)); /* TODO: check if we need to strip the domain name here -- rgerhards, 2008-04-24 */ @@ -467,8 +469,8 @@ SessAccept(strmsrv_t *pThis, strmLstnPortList_t *pLstnInfo, strms_sess_t **ppSes */ CHKiRet(strms_sess.SetHost(pSess, fromHostFQDN)); fromHostFQDN = NULL; /* we handed this string over */ - CHKiRet(strms_sess.SetHostIP(pSess, fromHostIP)); - fromHostIP = NULL; /* we handed this string over */ + CHKiRet(strms_sess.SetHostIP(pSess, ip)); + ip = NULL; /* we handed this string over */ CHKiRet(strms_sess.SetStrm(pSess, pNewStrm)); pNewStrm = NULL; /* prevent it from being freed in error handler, now done in strms_sess! */ CHKiRet(strms_sess.ConstructFinalize(pSess)); @@ -489,7 +491,8 @@ finalize_it: if(pNewStrm != NULL) netstrm.Destruct(&pNewStrm); free(fromHostFQDN); - free(fromHostIP); + if(ip != NULL) + prop.Destruct(&ip); } RETiRet; @@ -908,6 +911,7 @@ CODESTARTObjClassExit(strmsrv) objRelease(strms_sess, DONT_LOAD_LIB); objRelease(conf, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(netstrms, DONT_LOAD_LIB); objRelease(nssel, DONT_LOAD_LIB); @@ -930,6 +934,7 @@ BEGINObjClassInit(strmsrv, 1, OBJ_IS_LOADABLE_MODULE) /* class, version - CHANGE CHKiRet(objUse(strms_sess, DONT_LOAD_LIB)); CHKiRet(objUse(conf, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, strmsrvDebugPrint); diff --git a/runtime/strmsrv.h b/runtime/strmsrv.h index 9ef28e47..f3d56d16 100644 --- a/runtime/strmsrv.h +++ b/runtime/strmsrv.h @@ -43,7 +43,7 @@ struct strmsrv_s { uchar *pszInputName; /**< value to be used as input name */ permittedPeers_t *pPermPeers;/**< driver's permitted peers */ int iLstnMax; /**< max nbr of listeners currently supported */ - netstrm_t **ppLstn; /**< our netstream listners */ + netstrm_t **ppLstn; /**< our netstream listeners */ strmLstnPortList_t **ppLstnPort; /**< pointer to relevant listen port description */ int iSessMax; /**< max number of sessions supported */ strmLstnPortList_t *pLstnPorts; /**< head pointer for listen ports */ diff --git a/runtime/typedefs.h b/runtime/typedefs.h index 4e7f1622..d3f68b4a 100644 --- a/runtime/typedefs.h +++ b/runtime/typedefs.h @@ -3,7 +3,7 @@ * * Begun 2010-11-25 RGerhards * - * Copyright (C) 2005-2008 by Rainer Gerhards and Adiscon GmbH + * Copyright (C) 2005-2013 by Rainer Gerhards and Adiscon GmbH * * This file is part of the rsyslog runtime library. * @@ -25,6 +25,13 @@ */ #ifndef INCLUDED_TYPEDEFS_H #define INCLUDED_TYPEDEFS_H +#if defined(__FreeBSD__) +#include <sys/types.h> +#endif + +#ifndef HAVE_LSEEK64 +#include <unistd.h> +#endif /* some universal fixed size integer defines ... */ typedef long long int64; @@ -92,6 +99,10 @@ typedef struct cfgmodules_etry_s cfgmodules_etry_t; typedef struct outchannels_s outchannels_t; typedef struct modConfData_s modConfData_t; typedef struct instanceConf_s instanceConf_t; +typedef struct ratelimit_s ratelimit_t; +typedef struct action_s action_t; +typedef int rs_size_t; /* we do never need more than 2Gig strings, signed permits to + * use -1 as a special flag. */ typedef rsRetVal (*prsf_t)(struct vmstk_s*, int); /* pointer to a RainerScript function */ typedef uint64 qDeqID; /* queue Dequeue order ID. 32 bits is considered dangerously few */ @@ -144,6 +155,10 @@ typedef enum { FIOP_ISEMPTY = 6 /* string empty <=> strlen(s) == 0 ?*/ } fiop_t; +#ifndef HAVE_LSEEK64 + typedef off_t off64_t; +#endif + /* types of configuration handlers */ typedef enum cslCmdHdlrType { @@ -162,6 +177,7 @@ typedef enum cslCmdHdlrType { eCmdHdlrSeverity, eCmdHdlrGetWord, eCmdHdlrString, + eCmdHdlrArray, eCmdHdlrQueueType, eCmdHdlrGoneAway /* statment existed, but is no longer supported */ } ecslCmdHdrlType; diff --git a/runtime/wtp.c b/runtime/wtp.c index a53a9888..19151e7c 100644 --- a/runtime/wtp.c +++ b/runtime/wtp.c @@ -381,6 +381,7 @@ wtpWorker(void *arg) /* the arg is actually a wti object, even though we are in if(prctl(PR_SET_NAME, thrdName, 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", wtpGetDbgHdr(pThis)); } + dbgOutputTID((char*)thrdName); # endif pthread_cleanup_push(wtpWrkrExecCancelCleanup, pWti); diff --git a/tcps_sess.c b/tcps_sess.c index e7149cb7..5821e443 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -47,6 +47,7 @@ #include "msg.h" #include "datetime.h" #include "prop.h" +#include "ratelimit.h" #include "debug.h" @@ -139,24 +140,20 @@ finalize_it: RETiRet; } -/* set the remote host's IP. Note that the caller *hands over* the string. That is, +/* set the remote host's IP. Note that the caller *hands over* the property. That is, * the caller no longer controls it once SetHostIP() has received it. Most importantly, - * the caller must not free it. -- rgerhards, 2008-05-16 + * the caller must not destruct it. -- rgerhards, 2008-05-16 */ static rsRetVal -SetHostIP(tcps_sess_t *pThis, uchar *pszHostIP) +SetHostIP(tcps_sess_t *pThis, prop_t *ip) { DEFiRet; - ISOBJ_TYPE_assert(pThis, tcps_sess); - if(pThis->fromHostIP == NULL) - CHKiRet(prop.Construct(&pThis->fromHostIP)); - - CHKiRet(prop.SetString(pThis->fromHostIP, pszHostIP, ustrlen(pszHostIP))); - -finalize_it: - free(pszHostIP); + if(pThis->fromHostIP != NULL) { + prop.Destruct(&pThis->fromHostIP); + } + pThis->fromHostIP = ip; RETiRet; } @@ -264,14 +261,7 @@ defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttG MsgSetRuleset(pMsg, pThis->pLstnInfo->pRuleset); STATSCOUNTER_INC(pThis->pLstnInfo->ctrSubmit, pThis->pLstnInfo->mutCtrSubmit); - if(pMultiSub == NULL) { - CHKiRet(submitMsg(pMsg)); - } else { - pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; - if(pMultiSub->nElem == pMultiSub->maxElem) - CHKiRet(multiSubmitMsg(pMultiSub)); - } - + ratelimitAddMsg(pThis->pLstnInfo->ratelimiter, pMultiSub, pMsg); finalize_it: /* reset status variables */ @@ -368,7 +358,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt ISOBJ_TYPE_assert(pThis, tcps_sess); if(pThis->inputState == eAtStrtFram) { - if(pThis->bSuppOctetFram && isdigit((int) c)) { + if(pThis->bSuppOctetFram && c >= '0' && c <= '9') { pThis->inputState = eInOctetCnt; pThis->iOctetsRemain = 0; pThis->eFraming = TCP_FRAMING_OCTET_COUNTING; @@ -379,7 +369,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt } if(pThis->inputState == eInOctetCnt) { - if(isdigit(c)) { + if(c >= '0' && c <= '9') { /* isdigit() the faster way */ pThis->iOctetsRemain = pThis->iOctetsRemain * 10 + c - '0'; } else { /* done with the octet count, so this must be the SP terminator */ DBGPRINTF("TCP Message with octet-counter, size %d.\n", pThis->iOctetsRemain); @@ -487,11 +477,7 @@ DataRcvd(tcps_sess_t *pThis, char *pData, size_t iLen) while(pData < pEnd) { CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); } - - if(multiSub.nElem > 0) { - /* submit anything that was not yet submitted */ - CHKiRet(multiSubmitMsg(&multiSub)); - } + iRet = multiSubmitFlush(&multiSub); finalize_it: RETiRet; diff --git a/tcps_sess.h b/tcps_sess.h index 054ce397..4506cf07 100644 --- a/tcps_sess.h +++ b/tcps_sess.h @@ -66,16 +66,18 @@ BEGINinterface(tcps_sess) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetLstnInfo)(tcps_sess_t *pThis, tcpLstnPortList_t *pLstnInfo); rsRetVal (*SetUsrP)(tcps_sess_t*, void*); rsRetVal (*SetHost)(tcps_sess_t *pThis, uchar*); - rsRetVal (*SetHostIP)(tcps_sess_t *pThis, uchar*); + rsRetVal (*SetHostIP)(tcps_sess_t *pThis, prop_t*); rsRetVal (*SetStrm)(tcps_sess_t *pThis, netstrm_t*); rsRetVal (*SetMsgIdx)(tcps_sess_t *pThis, int); rsRetVal (*SetOnMsgReceive)(tcps_sess_t *pThis, rsRetVal (*OnMsgReceive)(tcps_sess_t*, uchar*, int)); ENDinterface(tcps_sess) -#define tcps_sessCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define tcps_sessCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface changes * to version v2, rgerhards, 2009-05-22 * - Data structures changed * - SetLstnInfo entry point added + * version 3, rgerhards, 2013-01-21: + * - signature of SetHostIP() changed */ @@ -72,6 +72,7 @@ #include "nspoll.h" #include "errmsg.h" #include "ruleset.h" +#include "ratelimit.h" #include "unicode-helper.h" @@ -151,6 +152,9 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort, int bSuppOctetFram) snprintf((char*)statname, sizeof(statname), "%s(%s)", pThis->pszInputName, pszPort); statname[sizeof(statname)-1] = '\0'; /* just to be on the save side... */ CHKiRet(statsobj.SetName(pEntry->stats, statname)); + CHKiRet(ratelimitNew(&pEntry->ratelimiter, "tcperver", NULL)); + ratelimitSetLinuxLike(pEntry->ratelimiter, pThis->ratelimitInterval, pThis->ratelimitBurst); + ratelimitSetThreadSafe(pEntry->ratelimiter); STATSCOUNTER_INIT(pEntry->ctrSubmit, pEntry->mutCtrSubmit); CHKiRet(statsobj.AddCounter(pEntry->stats, UCHAR_CONSTANT("submitted"), ctrType_IntCtr, &(pEntry->ctrSubmit))); @@ -295,6 +299,7 @@ static void deinit_tcp_listener(tcpsrv_t *pThis) while(pEntry != NULL) { free(pEntry->pszPort); prop.Destruct(&pEntry->pInputName); + ratelimitDestruct(pEntry->ratelimiter); pDel = pEntry; pEntry = pEntry->pNext; free(pDel); @@ -419,7 +424,7 @@ SessAccept(tcpsrv_t *pThis, tcpLstnPortList_t *pLstnInfo, tcps_sess_t **ppSess, int iSess = -1; struct sockaddr_storage *addr; uchar *fromHostFQDN = NULL; - uchar *fromHostIP = NULL; + prop_t *fromHostIP; ISOBJ_TYPE_assert(pThis, tcpsrv); assert(pLstnInfo != NULL); @@ -470,7 +475,6 @@ SessAccept(tcpsrv_t *pThis, tcpLstnPortList_t *pLstnInfo, tcps_sess_t **ppSess, CHKiRet(tcps_sess.SetHost(pSess, fromHostFQDN)); fromHostFQDN = NULL; /* we handed this string over */ CHKiRet(tcps_sess.SetHostIP(pSess, fromHostIP)); - fromHostIP = NULL; /* we handed this string over */ CHKiRet(tcps_sess.SetStrm(pSess, pNewStrm)); pNewStrm = NULL; /* prevent it from being freed in error handler, now done in tcps_sess! */ CHKiRet(tcps_sess.SetMsgIdx(pSess, 0)); @@ -493,7 +497,6 @@ finalize_it: if(pNewStrm != NULL) netstrm.Destruct(&pNewStrm); free(fromHostFQDN); - free(fromHostIP); } RETiRet; @@ -913,6 +916,8 @@ BEGINobjConstruct(tcpsrv) /* be sure to specify the object type also in END macr pThis->addtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; pThis->bDisableLFDelim = 0; pThis->OnMsgReceive = NULL; + pThis->ratelimitInterval = 0; + pThis->ratelimitBurst = 10000; pThis->bUseFlowControl = 1; ENDobjConstruct(tcpsrv) @@ -1120,6 +1125,17 @@ finalize_it: } +/* Set the linux-like ratelimiter settings */ +static rsRetVal +SetLinuxLikeRatelimiters(tcpsrv_t *pThis, int ratelimitInterval, int ratelimitBurst) +{ + DEFiRet; + pThis->ratelimitInterval = ratelimitInterval; + pThis->ratelimitBurst = ratelimitBurst; + RETiRet; +} + + /* Set the ruleset (ptr) to use */ static rsRetVal SetRuleset(tcpsrv_t *pThis, ruleset_t *pRuleset) @@ -1270,6 +1286,7 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetCBOnErrClose = SetCBOnErrClose; pIf->SetOnMsgReceive = SetOnMsgReceive; pIf->SetRuleset = SetRuleset; + pIf->SetLinuxLikeRatelimiters = SetLinuxLikeRatelimiters; pIf->SetNotificationOnRemoteClose = SetNotificationOnRemoteClose; finalize_it: @@ -42,6 +42,7 @@ struct tcpLstnPortList_s { ruleset_t *pRuleset; /**< associated ruleset */ statsobj_t *stats; /**< associated stats object */ sbool bSuppOctetFram; /**< do we support octect-counted framing? (if no->legay only!)*/ + ratelimit_t *ratelimiter; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) tcpLstnPortList_t *pNext; /**< next port or NULL */ }; @@ -62,14 +63,16 @@ struct tcpsrv_s { sbool bUsingEPoll; /**< are we in epoll mode (means we do not need to keep track of sessions!) */ sbool bUseFlowControl; /**< use flow control (make light delayable) */ int iLstnCurr; /**< max nbr of listeners currently supported */ - netstrm_t **ppLstn; /**< our netstream listners */ + netstrm_t **ppLstn; /**< our netstream listeners */ tcpLstnPortList_t **ppLstnPort; /**< pointer to relevant listen port description */ - int iLstnMax; /**< max number of listners supported */ + int iLstnMax; /**< max number of listeners supported */ int iSessMax; /**< max number of sessions supported */ tcpLstnPortList_t *pLstnPorts; /**< head pointer for listen ports */ int addtlFrameDelim; /**< additional frame delimiter for plain TCP syslog framing (e.g. to handle NetScreen) */ int bDisableLFDelim; /**< if 1, standard LF frame delimiter is disabled (*very dangerous*) */ + int ratelimitInterval; + int ratelimitBurst; tcps_sess_t **pSessions;/**< array of all of our sessions */ void *pUsr; /**< a user-settable pointer (provides extensibility for "derived classes")*/ /* callbacks */ @@ -142,8 +145,10 @@ BEGINinterface(tcpsrv) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetUseFlowControl)(tcpsrv_t*, int); /* added v11 -- rgerhards, 2011-05-09 */ rsRetVal (*SetKeepAlive)(tcpsrv_t*, int); + /* added v13 -- rgerhards, 2012-10-15 */ + rsRetVal (*SetLinuxLikeRatelimiters)(tcpsrv_t *pThis, int interval, int burst); ENDinterface(tcpsrv) -#define tcpsrvCURR_IF_VERSION 12 /* increment whenever you change the interface structure! */ +#define tcpsrvCURR_IF_VERSION 13 /* increment whenever you change the interface structure! */ /* change for v4: * - SetAddtlFrameDelim() added -- rgerhards, 2008-12-10 * - SetInputName() added -- rgerhards, 2008-12-10 @@ -44,6 +44,7 @@ #include "errmsg.h" #include "strgen.h" #include "rsconf.h" +#include "msg.h" #include "unicode-helper.h" /* static data */ @@ -54,9 +55,10 @@ DEFobjCurrIf(strgen) /* tables for interfacing with the v6 config system */ static struct cnfparamdescr cnfparamdescr[] = { { "name", eCmdHdlrString, 1 }, - { "type", eCmdHdlrString, 0 }, + { "type", eCmdHdlrString, 1 }, { "string", eCmdHdlrString, 0 }, { "plugin", eCmdHdlrString, 0 }, + { "subtree", eCmdHdlrString, 0 }, { "option.stdsql", eCmdHdlrBinary, 0 }, { "option.sql", eCmdHdlrBinary, 0 }, { "option.json", eCmdHdlrBinary, 0 } @@ -77,6 +79,7 @@ static struct cnfparamdescr cnfparamdescrProperty[] = { { "format", eCmdHdlrString, 0 }, { "position.from", eCmdHdlrInt, 0 }, { "position.to", eCmdHdlrInt, 0 }, + { "position.relativetoend", eCmdHdlrBinary, 0 }, { "field.number", eCmdHdlrInt, 0 }, { "field.delimiter", eCmdHdlrInt, 0 }, { "regex.expression", eCmdHdlrString, 0 }, @@ -139,14 +142,16 @@ finalize_it: * offers big performance improvements. * rewritten 2009-06-19 rgerhards */ -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf) +rsRetVal +tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf, + struct syslogTime *ttNow) { DEFiRet; struct templateEntry *pTpe; size_t iBuf; unsigned short bMustBeFreed = 0; uchar *pVal; - size_t iLenVal = 0; + rs_size_t iLenVal = 0; assert(pTpl != NULL); assert(pMsg != NULL); @@ -158,6 +163,23 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t * FINALIZE; } + if(pTpl->subtree != NULL) { + /* only a single CEE subtree must be provided */ + /* note: we could optimize the code below, however, this is + * not worth the effort, as this passing mode is not expected + * in subtree mode and so most probably only used for debug & test. + */ + getCEEPropVal(pMsg, pTpl->subtree, &pVal, &iLenVal, &bMustBeFreed); + if(iLenVal >= (rs_size_t)*pLenBuf) /* we reserve one char for the final \0! */ + CHKiRet(ExtendBuf(ppBuf, pLenBuf, iLenVal + 1)); + memcpy(*ppBuf, pVal, iLenVal+1); + if(bMustBeFreed) + free(pVal); + FINALIZE; + } + + /* we have a "regular" template with template entries */ + /* loop through the template. We obtain one value * and copy it over to our dynamic string buffer. Then, we * free the obtained value (if requested). We continue this @@ -172,7 +194,8 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t * bMustBeFreed = 0; } else if(pTpe->eEntryType == FIELD) { pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &iLenVal, &bMustBeFreed); + pTpe->data.field.propName, &iLenVal, + &bMustBeFreed, ttNow); /* we now need to check if we should use SQL option. In this case, * we must go over the generated string and escape '\'' characters. * rgerhards, 2005-09-22: the option values below look somewhat misplaced, @@ -226,13 +249,14 @@ finalize_it: * is indicated by a NULL pointer. * rgerhards, 2009-04-03 */ -rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) +rsRetVal +tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime *ttNow) { DEFiRet; struct templateEntry *pTpe; uchar **pArr; int iArr; - size_t propLen; + rs_size_t propLen; unsigned short bMustBeFreed; uchar *pVal; @@ -240,11 +264,24 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) assert(pMsg != NULL); assert(ppArr != NULL); + if(pTpl->subtree) { + /* Note: this mode is untested, as there is no official plugin + * using array passing, so I simply could not test it. + */ + CHKmalloc(pArr = calloc(2, sizeof(uchar*))); + getCEEPropVal(pMsg, pTpl->subtree, &pVal, &propLen, &bMustBeFreed); + if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ + pArr[0] = pVal; /* ... so we can use it! */ + } else { + CHKmalloc(pArr[0] = (uchar*)strdup((char*) pVal)); + } + FINALIZE; + } + /* loop through the template. We obtain one value, create a * private copy (if necessary), add it to the string array * and then on to the next until we have processed everything. */ - CHKmalloc(pArr = calloc(pTpl->tpenElements + 1, sizeof(uchar*))); iArr = 0; @@ -254,7 +291,8 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) CHKmalloc(pArr[iArr] = (uchar*)strdup((char*) pTpe->data.constant.pConstant)); } else if(pTpe->eEntryType == FIELD) { pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, &bMustBeFreed); + pTpe->data.field.propName, &propLen, + &bMustBeFreed, ttNow); if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ pArr[iArr] = pVal; /* ... so we can use it! */ } else { @@ -277,15 +315,28 @@ finalize_it: * tpltoString(). * rgerhards, 2012-08-29 */ -rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson) +rsRetVal +tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson, struct syslogTime *ttNow) { struct templateEntry *pTpe; - size_t propLen; + rs_size_t propLen; unsigned short bMustBeFreed; uchar *pVal; struct json_object *json, *jsonf; + rsRetVal localRet; DEFiRet; + if(pTpl->subtree != NULL){ + localRet = jsonFind(pMsg, pTpl->subtree, pjson); + if(*pjson == NULL) { + /* we need to have a root object! */ + *pjson = json_object_new_object(); + } else { + json_object_get(*pjson); /* inc refcount */ + } + FINALIZE; + } + json = json_object_new_object(); for(pTpe = pTpl->pEntryRoot ; pTpe != NULL ; pTpe = pTpe->pNext) { if(pTpe->eEntryType == CONSTANT) { @@ -294,19 +345,34 @@ rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjso jsonf = json_object_new_string((char*) pTpe->data.constant.pConstant); json_object_object_add(json, (char*)pTpe->fieldName, jsonf); } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, &bMustBeFreed); - if(pTpe->data.field.options.bMandatory || propLen > 0) { - jsonf = json_object_new_string_len((char*)pVal, propLen); - json_object_object_add(json, (char*)pTpe->fieldName, jsonf); - } - if(bMustBeFreed) { /* json-c makes its own private copy! */ - free(pVal); + if(pTpe->data.field.propid == PROP_CEE) { + localRet = msgGetCEEPropJSON(pMsg, pTpe->data.field.propName, &jsonf); + if(localRet == RS_RET_OK) { + json_object_object_add(json, (char*)pTpe->fieldName, json_object_get(jsonf)); + } else { + DBGPRINTF("tplToJSON: error %d looking up property\n", + localRet); + if(pTpe->data.field.options.bMandatory) { + json_object_object_add(json, (char*)pTpe->fieldName, NULL); + } + } + } else { + pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, + pTpe->data.field.propName, &propLen, + &bMustBeFreed, ttNow); + if(pTpe->data.field.options.bMandatory || propLen > 0) { + jsonf = json_object_new_string_len((char*)pVal, propLen); + json_object_object_add(json, (char*)pTpe->fieldName, jsonf); + } + if(bMustBeFreed) { /* json-c makes its own private copy! */ + free(pVal); + } } } } - *pjson = (iRet == RS_RET_OK) ? json : NULL; + +finalize_it: RETiRet; } @@ -364,7 +430,7 @@ static void doEmergencyEscape(register uchar *p, int mode) * 2005-09-22 rgerhards */ rsRetVal -doEscape(uchar **pp, size_t *pLen, unsigned short *pbMustBeFreed, int mode) +doEscape(uchar **pp, rs_size_t *pLen, unsigned short *pbMustBeFreed, int mode) { DEFiRet; uchar *p = NULL; @@ -487,14 +553,20 @@ tplConstruct(rsconf_t *conf) /* helper to tplAddLine. Parses a constant and generates * the necessary structure. - * returns: 0 - ok, 1 - failure + * Paramter "bDoEscapes" is to support legacy vs. v6+ config system. In + * legacy, we must do escapes ourselves, whereas v6+ passes in already + * escaped strings (which we are NOT permitted to further escape, this would + * cause invalid result strings!). Note: if escapes are not permitted, + * quotes (") are just a regular character and do NOT terminate the constant! */ -static int do_Constant(unsigned char **pp, struct template *pTpl) +static rsRetVal +do_Constant(unsigned char **pp, struct template *pTpl, int bDoEscapes) { register unsigned char *p; cstr_t *pStrB; struct templateEntry *pTpe; int i; + DEFiRet; assert(pp != NULL); assert(*pp != NULL); @@ -502,13 +574,12 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) p = *pp; - if(cstrConstruct(&pStrB) != RS_RET_OK) - return 1; + CHKiRet(cstrConstruct(&pStrB)); /* process the message and expand escapes * (additional escapes can be added here if needed) */ - while(*p && *p != '%' && *p != '\"') { - if(*p == '\\') { + while(*p && *p != '%' && !(bDoEscapes && *p == '\"')) { + if(bDoEscapes && *p == '\\') { switch(*++p) { case '\0': /* the best we can do - it's invalid anyhow... */ @@ -557,7 +628,7 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) if((pTpe = tpeConstruct(pTpl)) == NULL) { rsCStrDestruct(&pStrB); - return 1; + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } pTpe->eEntryType = CONSTANT; cstrFinalize(pStrB); @@ -567,12 +638,12 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) * 2005-09-09 rgerhards */ pTpe->data.constant.iLenConstant = rsCStrLen(pStrB); - if(cstrConvSzStrAndDestruct(pStrB, &pTpe->data.constant.pConstant, 0) != RS_RET_OK) - return 1; + CHKiRet(cstrConvSzStrAndDestruct(pStrB, &pTpe->data.constant.pConstant, 0)); *pp = p; - return 0; +finalize_it: + RETiRet; } @@ -642,6 +713,8 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe) pTpe->data.field.options.bSecPathDrop = 1; } else if(!strcmp((char*)Buf, "secpath-replace")) { pTpe->data.field.options.bSecPathReplace = 1; + } else if(!strcmp((char*)Buf, "pos-end-relative")) { + pTpe->data.field.options.bFromPosEndRelative = 1; } else if(!strcmp((char*)Buf, "csv")) { if(pTpe->data.field.options.bJSON || pTpe->data.field.options.bJSONf) { errmsg.LogError(0, NO_ERRCODE, "error: can only specify " @@ -677,11 +750,11 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe) /* helper to tplAddLine. Parses a parameter and generates * the necessary structure. - * returns: 0 - ok, 1 - failure */ -static int do_Parameter(unsigned char **pp, struct template *pTpl) +static rsRetVal +do_Parameter(uchar **pp, struct template *pTpl) { - unsigned char *p; + uchar *p; cstr_t *pStrProp; cstr_t *pStrField = NULL; struct templateEntry *pTpe; @@ -693,21 +766,15 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) unsigned char *regex_char; unsigned char *regex_end; #endif + DEFiRet; assert(pp != NULL); assert(*pp != NULL); assert(pTpl != NULL); - p = (unsigned char*) *pp; - - if(cstrConstruct(&pStrProp) != RS_RET_OK) - return 1; - - if((pTpe = tpeConstruct(pTpl)) == NULL) { - /* TODO: add handler */ - dbgprintf("Could not allocate memory for template parameter!\n"); - return 1; - } + p = (uchar*) *pp; + CHKiRet(cstrConstruct(&pStrProp)); + CHKmalloc(pTpe = tpeConstruct(pTpl)); pTpe->eEntryType = FIELD; while(*p && *p != '%' && *p != ':') { @@ -719,19 +786,22 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) cstrFinalize(pStrProp); if(propNameToID(pStrProp, &pTpe->data.field.propid) != RS_RET_OK) { + errmsg.LogError(0, RS_RET_TPL_INVLD_PROP, "template '%s': invalid parameter '%s'", + pTpl->pszName, cstrGetSzStrNoNULL(pStrProp)); cstrDestruct(&pStrProp); - return 1; + ABORT_FINALIZE(RS_RET_TPL_INVLD_PROP); } if(pTpe->data.field.propid == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - if((pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(pStrProp)+2, cstrLen(pStrProp)-2)) == NULL) { + if((pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(pStrProp)+1, cstrLen(pStrProp)-1)) == NULL) { cstrDestruct(&pStrProp); - return 1; + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } } /* Check frompos, if it has an R, then topos should be a regex */ if(*p == ':') { + pTpe->bComplexProcessing = 1; ++p; /* eat ':' */ #ifdef FEATURE_REGEXP if(*p == 'R') { @@ -791,8 +861,8 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) * comma itself is already part of the next field. */ } else { - errmsg.LogError(0, NO_ERRCODE, "error: invalid regular expression type, rest of line %s", - (char*) p); + errmsg.LogError(0, NO_ERRCODE, "template %s error: invalid regular expression type, rest of line %s", + pTpl->pszName, (char*) p); } } @@ -898,9 +968,7 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) #ifdef FEATURE_REGEXP if (pTpe->data.field.has_regex) { - dbgprintf("debug: has regex \n"); - /* APR 2005-09 I need the string that represent the regex */ /* The regex end is: "--end" */ /* TODO : this is hardcoded and cant be escaped, please change */ @@ -916,16 +984,13 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) if(regex_char == NULL) { dbgprintf("Could not allocate memory for template parameter!\n"); pTpe->data.field.has_regex = 0; - return 1; - /* TODO: RGer: check if we can recover better... (probably not) */ + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } /* Get the regex string for compiling later */ memcpy(regex_char, p, longitud); regex_char[longitud] = '\0'; - dbgprintf("debug: regex detected: '%s'\n", regex_char); - /* Now i compile the regex */ /* Remember that the re is an attribute of the Template entry */ if((iRetLocal = objUse(regexp, LM_REGEXP_FILENAME)) == RS_RET_OK) { @@ -995,23 +1060,31 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) #endif /* #ifdef FEATURE_REGEXP */ } - if(pTpe->data.field.iToPos < pTpe->data.field.iFromPos) { - iNum = pTpe->data.field.iToPos; - pTpe->data.field.iToPos = pTpe->data.field.iFromPos; - pTpe->data.field.iFromPos = iNum; - } - /* check options */ if(*p == ':') { ++p; /* eat ':' */ doOptions(&p, pTpe); } + if(pTpe->data.field.options.bFromPosEndRelative) { + if(pTpe->data.field.iToPos > pTpe->data.field.iFromPos) { + iNum = pTpe->data.field.iToPos; + pTpe->data.field.iToPos = pTpe->data.field.iFromPos; + pTpe->data.field.iFromPos = iNum; + } + } else { + if(pTpe->data.field.iToPos < pTpe->data.field.iFromPos) { + iNum = pTpe->data.field.iToPos; + pTpe->data.field.iToPos = pTpe->data.field.iFromPos; + pTpe->data.field.iFromPos = iNum; + } + } + + /* check field name */ if(*p == ':') { ++p; /* eat ':' */ - if(cstrConstruct(&pStrField) != RS_RET_OK) - return 1; + CHKiRet(cstrConstruct(&pStrField)); while(*p != ':' && *p != '%' && *p != '\0') { cstrAppendChar(pStrField, *p); ++p; @@ -1031,18 +1104,18 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) } } else { pTpe->fieldName = ustrdup(cstrGetSzStrNoNULL(pStrField)); - pTpe->lenFieldName = cstrLen(pStrProp); + pTpe->lenFieldName = ustrlen(pTpe->fieldName); cstrDestruct(&pStrField); } - if(pTpe->fieldName == NULL) - return 1; - + if(pTpe->fieldName == NULL) { + DBGPRINTF("template/do_Parameter: fieldName is NULL!\n"); + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } cstrDestruct(&pStrProp); - if(*p) ++p; /* eat '%' */ - *pp = p; - return 0; +finalize_it: + RETiRet; } @@ -1108,10 +1181,10 @@ struct template *tplAddLine(rsconf_t *conf, char* pName, uchar** ppRestOfConfLin assert(pName != NULL); assert(ppRestOfConfLine != NULL); - if((pTpl = tplConstruct(conf)) == NULL) return NULL; + DBGPRINTF("tplAddLine processing template '%s'\n", pName); pTpl->iLenName = strlen(pName); pTpl->pszName = (char*) MALLOC(sizeof(char) * (pTpl->iLenName + 1)); if(pTpl->pszName == NULL) { @@ -1171,10 +1244,13 @@ struct template *tplAddLine(rsconf_t *conf, char* pName, uchar** ppRestOfConfLin break; case '%': /* parameter */ ++p; /* eat '%' */ - do_Parameter(&p, pTpl); + if(do_Parameter(&p, pTpl) != RS_RET_OK) { + dbgprintf("tplAddLine error: parameter invalid"); + return NULL; + }; break; default: /* constant */ - do_Constant(&p, pTpl); + do_Constant(&p, pTpl, 1); break; } if(*p == '"') {/* end of template string? */ @@ -1235,7 +1311,7 @@ static rsRetVal createConstantTpe(struct template *pTpl, struct cnfobj *o) { struct templateEntry *pTpe; - es_str_t *value; + es_str_t *value = NULL; /* init just to keep compiler happy - mandatory parameter */ int i; struct cnfparamvals *pvals = NULL; uchar *outname = NULL; @@ -1292,6 +1368,8 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) int fielddelim = 9; /* default is HT (USACSII 9) */ int re_matchToUse = 0; int re_submatchToUse = 0; + int bComplexProcessing = 0; + int bPosRelativeToEnd = 0; char *re_expr = NULL; struct cnfparamvals *pvals = NULL; enum {F_NONE, F_CSV, F_JSON, F_JSONF} formatType = F_NONE; @@ -1317,23 +1395,33 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) free(tmpstr); } else if(!strcmp(pblkProperty.descr[i].name, "droplastlf")) { droplastlf = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "mandatory")) { mandatory = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "spifno1stsp")) { spifno1stsp = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "outname")) { outname = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(pblkProperty.descr[i].name, "position.from")) { frompos = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "position.to")) { topos = pvals[i].val.d.n; + bComplexProcessing = 1; + } else if(!strcmp(pblkProperty.descr[i].name, "position.relativetoend")) { + bPosRelativeToEnd = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "field.number")) { fieldnum = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "field.delimiter")) { fielddelim = pvals[i].val.d.n; + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "regex.expression")) { re_expr = es_str2cstr(pvals[i].val.d.estr, NULL); + bComplexProcessing = 1; } else if(!strcmp(pblkProperty.descr[i].name, "regex.type")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"BRE", sizeof("BRE")-1)) { re_type = TPL_REGEX_BRE; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"ERE", sizeof("ERE")-1)) { @@ -1346,6 +1434,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "regex.nomatchmode")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"DFLT", sizeof("DFLT")-1)) { re_nomatchType = TPL_REGEX_NOMATCH_USE_DFLTSTR; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"BLANK", sizeof("BLANK")-1)) { @@ -1362,10 +1451,13 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "regex.match")) { + bComplexProcessing = 1; re_matchToUse = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "regex.submatch")) { + bComplexProcessing = 1; re_submatchToUse = pvals[i].val.d.n; } else if(!strcmp(pblkProperty.descr[i].name, "format")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"csv", sizeof("csv")-1)) { formatType = F_CSV; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"json", sizeof("json")-1)) { @@ -1380,6 +1472,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "controlcharacters")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"escape", sizeof("escape")-1)) { controlchr = CC_ESCAPE; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"space", sizeof("space")-1)) { @@ -1394,6 +1487,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "securepath")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"drop", sizeof("drop")-1)) { secpath = SP_DROP; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"replace", sizeof("replace")-1)) { @@ -1406,6 +1500,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) ABORT_FINALIZE(RS_RET_ERR); } } else if(!strcmp(pblkProperty.descr[i].name, "caseconversion")) { + bComplexProcessing = 1; if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"lower", sizeof("lower")-1)) { caseconv = tplCaseConvLower; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"upper", sizeof("upper")-1)) { @@ -1458,10 +1553,18 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) topos = 2000000000; /* large enough ;) */ if(frompos == -1 && topos != -1) frompos = 0; - if(topos < frompos) { - errmsg.LogError(0, RS_RET_ERR, "position.to=%d is lower than postion.from=%d\n", - topos, frompos); - ABORT_FINALIZE(RS_RET_ERR); + if(bPosRelativeToEnd) { + if(topos > frompos) { + errmsg.LogError(0, RS_RET_ERR, "position.to=%d is higher than postion.from=%d in 'relativeToEnd' mode\n", + topos, frompos); + ABORT_FINALIZE(RS_RET_ERR); + } + } else { + if(topos < frompos) { + errmsg.LogError(0, RS_RET_ERR, "position.to=%d is lower than postion.from=%d\n", + topos, frompos); + ABORT_FINALIZE(RS_RET_ERR); + } } if(fieldnum != -1 && re_expr != NULL) { errmsg.LogError(0, RS_RET_ERR, "both field extraction and regex extraction " @@ -1475,8 +1578,8 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) CHKiRet(propNameToID(name, &pTpe->data.field.propid)); if(pTpe->data.field.propid == PROP_CEE) { /* in CEE case, we need to preserve the actual property name */ - pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(name)+2, - cstrLen(name)-2); + pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(name)+1, + cstrLen(name)-1); } pTpe->data.field.options.bDropLastLF = droplastlf; pTpe->data.field.options.bSPIffNo1stSP = spifno1stsp; @@ -1524,6 +1627,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) pTpe->fieldName = outname; if(outname != NULL) pTpe->lenFieldName = ustrlen(outname); + pTpe->bComplexProcessing = bComplexProcessing; pTpe->data.field.eDateFormat = datefmt; if(fieldnum != -1) { pTpe->data.field.has_fields = 1; @@ -1533,6 +1637,7 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) if(frompos != -1) { pTpe->data.field.iFromPos = frompos; pTpe->data.field.iToPos = topos; + pTpe->data.field.options.bFromPosEndRelative = bPosRelativeToEnd; } if(re_expr != NULL) { rsRetVal iRetLocal; @@ -1604,12 +1709,14 @@ tplProcessCnf(struct cnfobj *o) { struct template *pTpl = NULL; struct cnfparamvals *pvals = NULL; - int lenName; + int lenName = 0; /* init just to keep compiler happy: mandatory parameter */ char *name = NULL; uchar *tplStr = NULL; uchar *plugin = NULL; + es_str_t *subtree = NULL; uchar *p; - enum { T_STRING, T_PLUGIN, T_LIST } tplType; + enum { T_STRING, T_PLUGIN, T_LIST, T_SUBTREE } + tplType = T_STRING; /* init just to keep compiler happy: mandatory parameter */ int i; int o_sql=0, o_stdsql=0, o_json=0; /* options */ int numopts; @@ -1632,6 +1739,8 @@ tplProcessCnf(struct cnfobj *o) tplType = T_PLUGIN; } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"list", sizeof("list")-1)) { tplType = T_LIST; + } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"subtree", sizeof("subtree")-1)) { + tplType = T_SUBTREE; } else { uchar *typeStr = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); errmsg.LogError(0, RS_RET_ERR, "invalid template type '%s'", @@ -1641,6 +1750,22 @@ tplProcessCnf(struct cnfobj *o) } } else if(!strcmp(pblk.descr[i].name, "string")) { tplStr = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(pblk.descr[i].name, "subtree")) { + uchar *st_str = es_getBufAddr(pvals[i].val.d.estr); + if(st_str[0] != '$' || st_str[1] != '!') { + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_ERR, "invalid subtree " + "parameter, variable must start with '$!' but " + "var name is '%s'", cstr); + free(cstr); + free(name); /* overall assigned */ + ABORT_FINALIZE(RS_RET_ERR); + } else { + /* TODO: unify strings! */ + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + subtree = es_newStrFromBuf(cstr+1, es_strlen(pvals[i].val.d.estr)-1); + free(cstr); + } } else if(!strcmp(pblk.descr[i].name, "plugin")) { plugin = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(pblk.descr[i].name, "option.stdsql")) { @@ -1682,6 +1807,19 @@ tplProcessCnf(struct cnfobj *o) } } + if(subtree == NULL) { + if(tplType == T_SUBTREE) { + errmsg.LogError(0, RS_RET_ERR, "template '%s' of type subtree needs " + "subtree parameter", name); + ABORT_FINALIZE(RS_RET_ERR); + } + } else { + if(tplType != T_SUBTREE) { + errmsg.LogError(0, RS_RET_ERR, "template '%s' is not a subtree " + "template but has a subtree specified - ignored", name); + } + } + if(o->subobjs == NULL) { if(tplType == T_LIST) { errmsg.LogError(0, RS_RET_ERR, "template '%s' of type list has " @@ -1719,10 +1857,10 @@ tplProcessCnf(struct cnfobj *o) switch(*p) { case '%': /* parameter */ ++p; /* eat '%' */ - do_Parameter(&p, pTpl); + CHKiRet(do_Parameter(&p, pTpl)); break; default: /* constant */ - do_Constant(&p, pTpl); + do_Constant(&p, pTpl, 0); break; } } @@ -1739,6 +1877,8 @@ tplProcessCnf(struct cnfobj *o) break; case T_LIST: createListTpl(pTpl, o); break; + case T_SUBTREE: pTpl->subtree = subtree; + break; } pTpl->optFormatEscape = NO_ESCAPE; @@ -1840,8 +1980,9 @@ void tplDeleteAll(rsconf_t *conf) } pTplDel = pTpl; pTpl = pTpl->pNext; - if(pTplDel->pszName != NULL) - free(pTplDel->pszName); + free(pTplDel->pszName); + if(pTplDel->subtree != NULL) + es_deleteStr(pTplDel->subtree); free(pTplDel); } ENDfunc @@ -1898,8 +2039,9 @@ void tplDeleteNew(rsconf_t *conf) } pTplDel = pTpl; pTpl = pTpl->pNext; - if(pTplDel->pszName != NULL) - free(pTplDel->pszName); + free(pTplDel->pszName); + if(pTplDel->subtree != NULL) + es_deleteStr(pTplDel->subtree); free(pTplDel); } ENDfunc @@ -1965,8 +2107,14 @@ void tplPrintList(rsconf_t *conf) case tplFmtUnixDate: dbgprintf("[Format as Unix timestamp] "); break; + case tplFmtSecFrac: + dbgprintf("[fractional seconds, only] "); + break; + case tplFmtRFC3164BuggyDate: + dbgprintf("[Format as buggy RFC3164-Date] "); + break; default: - dbgprintf("[INVALID eDateFormat %d] ", pTpe->data.field.eDateFormat); + dbgprintf("[UNKNOWN eDateFormat %d] ", pTpe->data.field.eDateFormat); } switch(pTpe->data.field.eCaseConv) { case tplCaseConvNo: @@ -2022,6 +2170,8 @@ void tplPrintList(rsconf_t *conf) } break; } + if(pTpe->bComplexProcessing) + dbgprintf("[COMPLEX]"); dbgprintf("\n"); pTpe = pTpe->pNext; } @@ -2035,8 +2185,6 @@ int tplGetEntryCount(struct template *pTpl) return(pTpl->tpenElements); } -/* our init function. TODO: remove once converted to a class - */ rsRetVal templateInit() { DEFiRet; @@ -39,7 +39,8 @@ struct template { struct template *pNext; char *pszName; int iLenName; - rsRetVal (*pStrgen)(msg_t*, uchar**, size_t *); /* name of strgen to use (bound if non-NULL!) */ + rsRetVal (*pStrgen)(msg_t*, uchar**, size_t *); + es_str_t *subtree; /* subtree name for subtree-type templates */ int tpenElements; /* number of elements in templateEntry list */ struct templateEntry *pEntryRoot; struct templateEntry *pEntryLast; @@ -71,6 +72,7 @@ struct templateEntry { enum EntryTypes eEntryType; uchar *fieldName; /**< field name to be used for structured output */ int lenFieldName; + sbool bComplexProcessing; /**< set if complex processing (options, etc) is required */ union { struct { uchar *pConstant; /* pointer to constant value */ @@ -117,6 +119,7 @@ struct templateEntry { unsigned bJSON: 1; /* format field JSON escaped */ unsigned bJSONf: 1; /* format field JSON *field* (n/v pair) */ unsigned bMandatory: 1; /* mandatory field - emit even if empty */ + unsigned bFromPosEndRelative: 1;/* is From/To-Pos relative to end of string? */ } options; /* options as bit fields */ } field; } data; @@ -141,15 +144,16 @@ void tplDeleteNew(rsconf_t *conf); void tplPrintList(rsconf_t *conf); void tplLastStaticInit(rsconf_t *conf, struct template *tpl); rsRetVal ExtendBuf(uchar **pBuf, size_t *pLenBuf, size_t iMinSize); +int tplRequiresDateCall(struct template *pTpl); /* note: if a compiler warning for undefined type tells you to look at this * code line below, the actual cause is that you currently MUST include template.h * BEFORE msg.h, even if your code file does not actually need it. * rgerhards, 2007-08-06 */ -rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr); -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz, size_t *); -rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **); -rsRetVal doEscape(uchar **pp, size_t *pLen, unsigned short *pbMustBeFreed, int escapeMode); +rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime *ttNow); +rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz, size_t *, struct syslogTime *ttNow); +rsRetVal tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **, struct syslogTime *ttNow); +rsRetVal doEscape(uchar **pp, rs_size_t *pLen, unsigned short *pbMustBeFreed, int escapeMode); rsRetVal templateInit(); rsRetVal tplProcessCnf(struct cnfobj *o); diff --git a/tests/Makefile.am b/tests/Makefile.am index e7122653..b339e797 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1,6 +1,6 @@ if ENABLE_TESTBENCH # TODO: reenable TESTRUNS = rt_init rscript -check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq msleep randomgen diagtalker uxsockrcvr syslog_caller syslog_inject inputfilegen +check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq msleep randomgen diagtalker uxsockrcvr syslog_caller syslog_inject inputfilegen minitcpsrv TESTS = $(TESTRUNS) #TESTS = $(TESTRUNS) cfg.sh @@ -65,9 +65,18 @@ TESTS += \ failover-no-basic.sh \ rcvr_fail_restore.sh \ rscript_contains.sh \ + rscript_field.sh \ + rscript_stop.sh \ + rscript_stop2.sh \ + rscript_prifilt.sh \ + rscript_optimizer1.sh \ + rscript_ruleset_call.sh \ + cee_simple.sh \ + cee_diskqueue.sh \ incltest.sh \ incltest_dir.sh \ incltest_dir_wildcard.sh \ + incltest_dir_empty_wildcard.sh \ linkedlistqueue.sh if HAVE_VALGRIND @@ -172,6 +181,10 @@ test_files = testbench.h runtime-dummy.c EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ validation-run.sh \ + tls-certs/ca-key.pem \ + tls-certs/ca.pem \ + tls-certs/cert.pem \ + tls-certs/key.pem \ testsuites/x.509/ca.pem \ testsuites/x.509/ca-key.pem \ testsuites/x.509/client-cert.pem \ @@ -270,10 +283,28 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/arrayqueue.conf \ rscript_contains.sh \ testsuites/rscript_contains.conf \ + rscript_field.sh \ + testsuites/rscript_field.conf \ + rscript_stop.sh \ + testsuites/rscript_stop.conf \ + rscript_stop2.sh \ + testsuites/rscript_stop2.conf \ + rscript_prifilt.sh \ + testsuites/rscript_prifilt.conf \ + rscript_optimizer1.sh \ + testsuites/rscript_optimizer1.conf \ + rscript_ruleset_call.sh \ + testsuites/rscript_ruleset_call.conf \ + cee_simple.sh \ + testsuites/cee_simple.conf \ + cee_diskqueue.sh \ + testsuites/cee_diskqueue.conf \ incltest.sh \ testsuites/incltest.conf \ incltest_dir.sh \ testsuites/incltest_dir.conf \ + incltest_dir_empty_wildcard.sh \ + testsuites/incltest_dir_empty_wildcard.conf \ incltest_dir_wildcard.sh \ testsuites/incltest_dir_wildcard.conf \ testsuites/incltest.d/include.conf \ @@ -505,6 +536,9 @@ if ENABLE_GNUTLS tcpflood_LDADD += -lgcrypt endif +minitcpsrv_SOURCES = minitcpsrvr.c +minitcpsrv_LDADD = $(SOL_LIBS) + syslog_caller_SOURCES = syslog_caller.c syslog_caller_LDADD = $(SOL_LIBS) diff --git a/tests/cee_diskqueue.sh b/tests/cee_diskqueue.sh new file mode 100755 index 00000000..4e19855b --- /dev/null +++ b/tests/cee_diskqueue.sh @@ -0,0 +1,14 @@ +# check if CEE properties are properly saved & restored to/from disk queue +# added 2012-09-19 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[cee_diskqueue.sh\]: CEE and diskqueue test +source $srcdir/diag.sh init +source $srcdir/diag.sh startup cee_diskqueue.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/cee_simple.sh b/tests/cee_simple.sh new file mode 100755 index 00000000..32f56393 --- /dev/null +++ b/tests/cee_simple.sh @@ -0,0 +1,13 @@ +# added 2012-09-19 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[cee_simple.sh\]: basic CEE property test +source $srcdir/diag.sh init +source $srcdir/diag.sh startup cee_simple.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/diag.sh b/tests/diag.sh index bd38b29d..7883d93b 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -16,7 +16,6 @@ case $1 in 'init') $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason cp $srcdir/testsuites/diag-common.conf diag-common.conf cp $srcdir/testsuites/diag-common2.conf diag-common2.conf - rm -f rsyslog.action.*.include rm -f rsyslogd.started work-*.conf rsyslog.random.data rm -f rsyslogd2.started work-*.conf rm -f work rsyslog.out.log rsyslog2.out.log rsyslog.out.log.save # common work files @@ -24,6 +23,10 @@ case $1 in rm -f rsyslog.out.*.log work-presort rsyslog.pipe rm -f rsyslog.input rsyslog.empty rm -f core.* vgcore.* + # Note: rsyslog.action.*.include must NOT be deleted, as it + # is used to setup some parameters BEFORE calling init. This + # happens in chained test scripts. Delete on exit is fine, + # though. mkdir test-spool ;; 'exit') rm -f rsyslogd.started work-*.conf diag-common.conf @@ -106,6 +109,7 @@ case $1 in echo Shutting down instance 2 fi $srcdir/diag.sh wait-queueempty $2 + ./msleep 100 # wait 100 milliseconds kill `cat rsyslog$2.pid` # note: we do not wait for the actual termination! ;; diff --git a/tests/imptcp_addtlframedelim.sh b/tests/imptcp_addtlframedelim.sh index b26fc85b..00276ab3 100755 --- a/tests/imptcp_addtlframedelim.sh +++ b/tests/imptcp_addtlframedelim.sh @@ -3,7 +3,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_addtlframedelim.sh\]: test imptcp additional frame delimiter -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imptcp_addtlframedelim.conf source $srcdir/diag.sh tcpflood -m20000 -F0 -P129 diff --git a/tests/imptcp_conndrop.sh b/tests/imptcp_conndrop.sh index 684de6b5..0cf0ba5e 100755 --- a/tests/imptcp_conndrop.sh +++ b/tests/imptcp_conndrop.sh @@ -4,9 +4,8 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_conndrop.sh\]: test imptcp with random connection drops -cat rsyslog.action.1.include source $srcdir/diag.sh init -source $srcdir/diag.sh startup imptcp_large.conf +source $srcdir/diag.sh startup imptcp_conndrop.conf # 100 byte messages to gain more practical data use source $srcdir/diag.sh tcpflood -c20 -m50000 -r -d100 -P129 -D sleep 4 # due to large messages, we need this time for the tcp receiver to settle... diff --git a/tests/imptcp_large.sh b/tests/imptcp_large.sh index b4d130bb..43027069 100755 --- a/tests/imptcp_large.sh +++ b/tests/imptcp_large.sh @@ -4,7 +4,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imptcp_large.sh\]: test imptcp with large-size messages -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imptcp_large.conf # send 4000 messages of 10.000bytes plus header max, randomized diff --git a/tests/imtcp_addtlframedelim.sh b/tests/imtcp_addtlframedelim.sh index 8de7ca58..4c1fd9cb 100755 --- a/tests/imtcp_addtlframedelim.sh +++ b/tests/imtcp_addtlframedelim.sh @@ -3,7 +3,6 @@ # This file is part of the rsyslog project, released under GPLv3 echo ==================================================================================== echo TEST: \[imtcp_addtlframedelim.sh\]: test imtcp additional frame delimiter -cat rsyslog.action.1.include source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp_addtlframedelim.conf source $srcdir/diag.sh tcpflood -m20000 -F0 -P129 diff --git a/tests/incltest_dir_empty_wildcard.sh b/tests/incltest_dir_empty_wildcard.sh new file mode 100755 index 00000000..6cdb3b21 --- /dev/null +++ b/tests/incltest_dir_empty_wildcard.sh @@ -0,0 +1,13 @@ +# This test checks if an empty includeConfig directory causes problems. It +# should not, as this is a valid situation that by default exists on many +# distros. +echo =============================================================================== +echo \[incltest_dir_empty_wildcard.sh\]: test $IncludeConfig for \"empty\" wildcard +source $srcdir/diag.sh init +source $srcdir/diag.sh startup incltest_dir_empty_wildcard.conf +# 100 messages are enough - the question is if the include is read ;) +source $srcdir/diag.sh injectmsg 0 100 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 99 +source $srcdir/diag.sh exit diff --git a/tests/minitcpsrvr.c b/tests/minitcpsrvr.c new file mode 100644 index 00000000..8ac59f25 --- /dev/null +++ b/tests/minitcpsrvr.c @@ -0,0 +1,62 @@ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/socket.h> +#include <fcntl.h> +#include <unistd.h> +#include <arpa/inet.h> + +static void +errout(char *reason) +{ + perror(reason); + exit(1); +} + +int +main(int argc, char *argv[]) +{ + int fds; + int fdc; + int fdf; + struct sockaddr_in srvAddr; + struct sockaddr_in cliAddr; + unsigned int srvAddrLen; + unsigned int cliAddrLen; + char wrkBuf[4096]; + ssize_t nRead; + + if(argc != 4) { + fprintf(stderr, "usage: minitcpsrvr ip-addr port outfile\n"); + exit(1); + } + + if(!strcmp(argv[3], "-")) { + fdf = 1; + } else { + fdf = open(argv[3], O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR|S_IWUSR); + if(fdf == -1) errout(argv[3]); + } + + fds = socket(AF_INET, SOCK_STREAM, 0); + srvAddr.sin_family = AF_INET; + srvAddr.sin_addr.s_addr = inet_addr(argv[1]); + srvAddr.sin_port = htons(atoi(argv[2])); + srvAddrLen = sizeof(srvAddr); + if(bind(fds, (struct sockaddr *)&srvAddr, srvAddrLen) != 0) + errout("bind"); + if(listen(fds, 20) != 0) errout("listen"); + cliAddrLen = sizeof(cliAddr); + + fdc = accept(fds, (struct sockaddr *)&cliAddr, &cliAddrLen); + while(1) { + nRead = read(fdc, wrkBuf, sizeof(wrkBuf)); + if(nRead == 0) break; + if(write(fdf, wrkBuf, nRead) != nRead) + errout("write"); + } + /* let the OS do the cleanup */ + return 0; +} diff --git a/tests/nettester.c b/tests/nettester.c index 9e68ebcc..4e4fe55a 100644 --- a/tests/nettester.c +++ b/tests/nettester.c @@ -243,7 +243,7 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) { int pipefd[2]; pid_t cpid; - char *newargv[] = {"../tools/rsyslogd", "dummy", "-c4", "-u2", "-n", "-irsyslog.pid", + char *newargv[] = {"../tools/rsyslogd", "dummy", "-u2", "-n", "-irsyslog.pid", "-M../runtime/.libs:../.libs", NULL, NULL}; char confFile[1024]; diff --git a/tests/queue-persist-drvr.sh b/tests/queue-persist-drvr.sh index 53fbcb8b..823fed6c 100755 --- a/tests/queue-persist-drvr.sh +++ b/tests/queue-persist-drvr.sh @@ -24,6 +24,12 @@ source $srcdir/diag.sh check-mainq-spool echo "#" > work-delay.conf source $srcdir/diag.sh startup queue-persist.conf source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +./msleep 500 $srcdir/diag.sh wait-shutdown -source $srcdir/diag.sh seq-check 0 4999 +# note: we need to permit duplicate messages, as due to the forced +# shutdown some messages may be flagged as "unprocessed" while they +# actually were processed. This is inline with rsyslog's philosophy +# to better duplicate than loose messages. Duplicate messages are +# permitted by the -d seq-check option. +source $srcdir/diag.sh seq-check 0 4999 -d source $srcdir/diag.sh exit diff --git a/tests/rscript_field.sh b/tests/rscript_field.sh new file mode 100755 index 00000000..e989e666 --- /dev/null +++ b/tests/rscript_field.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_field.sh\]: testing rainerscript field\(\) function +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_field.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_optimizer1.sh b/tests/rscript_optimizer1.sh new file mode 100755 index 00000000..1d2dcf87 --- /dev/null +++ b/tests/rscript_optimizer1.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_optimizer1.sh\]: testing rainerscript optimizer +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_optimizer1.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_prifilt.sh b/tests/rscript_prifilt.sh new file mode 100755 index 00000000..815492ab --- /dev/null +++ b/tests/rscript_prifilt.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_prifilt.sh\]: testing rainerscript prifield\(\) function +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_prifilt.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_ruleset_call.sh b/tests/rscript_ruleset_call.sh new file mode 100755 index 00000000..e29f21da --- /dev/null +++ b/tests/rscript_ruleset_call.sh @@ -0,0 +1,13 @@ +# added 2012-10-29 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_ruleset_call.sh\]: testing rainerscript ruleset\(\) and call statement +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_ruleset_call.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_stop.sh b/tests/rscript_stop.sh new file mode 100755 index 00000000..e532a522 --- /dev/null +++ b/tests/rscript_stop.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_stop.sh\]: testing rainerscript STOP statement +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_stop.conf +source $srcdir/diag.sh injectmsg 0 8000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/rscript_stop2.sh b/tests/rscript_stop2.sh new file mode 100755 index 00000000..eae36cce --- /dev/null +++ b/tests/rscript_stop2.sh @@ -0,0 +1,13 @@ +# added 2012-09-20 by rgerhards +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[rscript_stop2.sh\]: testing rainerscript STOP statement, alternate method +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rscript_stop2.conf +source $srcdir/diag.sh injectmsg 0 8000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/runtime-dummy.c b/tests/runtime-dummy.c index 5a9039bf..f6f2d07f 100644 --- a/tests/runtime-dummy.c +++ b/tests/runtime-dummy.c @@ -30,7 +30,6 @@ #include "rsyslog.h" int bReduceRepeatMsgs = 0; -int repeatinterval = 30; int bActExecWhenPrevSusp = 0; int iActExecOnceInterval = 1; int MarkInterval = 30; diff --git a/tests/tcp_forwarding_tpl.sh b/tests/tcp_forwarding_tpl.sh new file mode 100755 index 00000000..61114507 --- /dev/null +++ b/tests/tcp_forwarding_tpl.sh @@ -0,0 +1,30 @@ +# This test tests tcp forwarding with assigned template. To do so, a simple +# tcp listener service is started. +# added 2012-10-30 by Rgerhards. Released under GNU GPLv3+ +echo =============================================================================== +echo \[tcp_forwarding_tpl.sh\]: test for tcp forwarding with assigned template + +# create the pipe and start a background process that copies data from +# it to the "regular" work file +source $srcdir/diag.sh init +./minitcpsrvr 127.0.0.1 13514 rsyslog.out.log & +BGPROCESS=$! +echo background minitcpsrvr process id is $BGPROCESS + +# now do the usual run +source $srcdir/diag.sh startup tcp_forwarding_tpl.conf +# 10000 messages should be enough +source $srcdir/diag.sh injectmsg 0 10000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown + +# note: minitcpsrvr shuts down automatically if the connection is closed! +# (we still leave the code here in in case we need it later) +#echo shutting down minitcpsrv... +#kill $BGPROCESS +#wait $BGPROCESS +#echo background process has terminated, continue test... + +# and continue the usual checks +source $srcdir/diag.sh seq-check 0 9999 +source $srcdir/diag.sh exit diff --git a/tests/tcpflood.c b/tests/tcpflood.c index 4a3199c8..b3cef2e0 100644 --- a/tests/tcpflood.c +++ b/tests/tcpflood.c @@ -258,7 +258,7 @@ int openConnections(void) return setupUDP(); if(bShowProgress) - write(1, " open connections", sizeof(" open connections")-1); + if(write(1, " open connections", sizeof(" open connections")-1)){} # ifdef ENABLE_GNUTLS sessArray = calloc(numConnections, sizeof(gnutls_session_t)); # endif @@ -278,7 +278,7 @@ int openConnections(void) } if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d open connections\n", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)) {} } return 0; @@ -303,12 +303,12 @@ void closeConnections(void) return; if(bShowProgress) - write(1, " close connections", sizeof(" close connections")-1); + if(write(1, " close connections", sizeof(" close connections")-1)){} for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)){} } } if(sockArray[i] != -1) { @@ -325,7 +325,7 @@ void closeConnections(void) } if(bShowProgress) { lenMsg = sprintf(msgBuf, "\r%5.5d close connections\n", i); - write(1, msgBuf, lenMsg); + if(write(1, msgBuf, lenMsg)){} } } @@ -347,7 +347,7 @@ genMsg(char *buf, size_t maxBuf, int *pLenBuf, struct instdata *inst) /* get message from file */ do { done = 1; - *pLenBuf = fread(buf, 1, 1024, dataFP); + *pLenBuf = fread(buf, 1, MAX_EXTRADATA_LEN + 1024, dataFP); if(*pLenBuf == 0) { if(--numFileIterations > 0) { rewind(dataFP); diff --git a/tests/testsuites/cee_diskqueue.conf b/tests/testsuites/cee_diskqueue.conf new file mode 100644 index 00000000..a9b98e80 --- /dev/null +++ b/tests/testsuites/cee_diskqueue.conf @@ -0,0 +1,9 @@ +$IncludeConfig diag-common.conf + +global(workDirectory="/tmp") +template(name="outfmt" type="string" string="%$!usr!msg:F,58:2%\n") + +set $!usr!msg = $msg; +if $msg contains 'msgnum' then + action(type="omfile" file="./rsyslog.out.log" template="outfmt" + queue.type="disk" queue.filename="rsyslog-act1") diff --git a/tests/testsuites/cee_simple.conf b/tests/testsuites/cee_simple.conf new file mode 100644 index 00000000..1bcf83c1 --- /dev/null +++ b/tests/testsuites/cee_simple.conf @@ -0,0 +1,6 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="string" string="%$!usr!msg:F,58:2%\n") +set $!usr!msg = $msg; +if $msg contains 'msgnum' then + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/complex1.conf b/tests/testsuites/complex1.conf index 9e2441d4..9b6a9f35 100644 --- a/tests/testsuites/complex1.conf +++ b/tests/testsuites/complex1.conf @@ -3,6 +3,8 @@ $MaxMessageSize 10k $IncludeConfig diag-common.conf +$MainMsgQueueTimeoutEnqueue 5000 + $ModLoad ../plugins/imtcp/.libs/imtcp $MainMsgQueueTimeoutShutdown 10000 @@ -13,6 +15,7 @@ $template dynfile,"rsyslog.out.%inputname%.%msg:F,58:2%.log" $Ruleset R13514 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 @@ -36,6 +39,7 @@ $InputTCPServerRun 13514 $Ruleset R13515 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 @@ -60,6 +64,7 @@ $InputTCPServerRun 13515 $Ruleset R13516 # queue params: $ActionQueueTimeoutShutdown 60000 +$ActionQueueTimeoutEnqueue 5000 $ActionQueueSize 5000 $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 4900 diff --git a/tests/testsuites/imptcp_addtlframedelim.conf b/tests/testsuites/imptcp_addtlframedelim.conf index eb7ed0c4..bf302fb4 100644 --- a/tests/testsuites/imptcp_addtlframedelim.conf +++ b/tests/testsuites/imptcp_addtlframedelim.conf @@ -9,5 +9,4 @@ $template outfmt,"%msg:F,58:2%\n" $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/imptcp_conndrop.conf b/tests/testsuites/imptcp_conndrop.conf index 677e33f6..77a5d79a 100644 --- a/tests/testsuites/imptcp_conndrop.conf +++ b/tests/testsuites/imptcp_conndrop.conf @@ -12,5 +12,4 @@ $template dynfile,"rsyslog.out.log" # trick to use relative path names! $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ?dynfile;outfmt diff --git a/tests/testsuites/imptcp_large.conf b/tests/testsuites/imptcp_large.conf index 677e33f6..77a5d79a 100644 --- a/tests/testsuites/imptcp_large.conf +++ b/tests/testsuites/imptcp_large.conf @@ -12,5 +12,4 @@ $template dynfile,"rsyslog.out.log" # trick to use relative path names! $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ?dynfile;outfmt diff --git a/tests/testsuites/imtcp_addtlframedelim.conf b/tests/testsuites/imtcp_addtlframedelim.conf index 3b4759c5..6558c519 100644 --- a/tests/testsuites/imtcp_addtlframedelim.conf +++ b/tests/testsuites/imtcp_addtlframedelim.conf @@ -9,5 +9,4 @@ $template outfmt,"%msg:F,58:2%\n" $OMFileFlushOnTXEnd off $OMFileFlushInterval 2 $OMFileIOBufferSize 256k -$IncludeConfig rsyslog.action.1.include local0.* ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/incltest_dir_empty_wildcard.conf b/tests/testsuites/incltest_dir_empty_wildcard.conf new file mode 100644 index 00000000..5e750c5b --- /dev/null +++ b/tests/testsuites/incltest_dir_empty_wildcard.conf @@ -0,0 +1,11 @@ +# see .sh file for description +# rgerhards, 2009-11-30 +$IncludeConfig diag-common.conf + +# the following include leads to no files actually being included +# but MUST NOT abort rsyslog's startup sequence. No files matching +# the wildcard is valid (as long as the path exists)! +$IncludeConfig testsuites/incltest.d/*.conf-not-there + +$template outfmt,"%msg:F,58:2%\n" +:msg, contains, "msgnum:" ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/rscript_field.conf b/tests/testsuites/rscript_field.conf new file mode 100644 index 00000000..d7eb9066 --- /dev/null +++ b/tests/testsuites/rscript_field.conf @@ -0,0 +1,11 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if $msg contains 'msgnum' then { + set $!usr!msgnum = field($msg, 58, 2); + action(type="omfile" file="./rsyslog.out.log" template="outfmt") +} diff --git a/tests/testsuites/rscript_optimizer1.conf b/tests/testsuites/rscript_optimizer1.conf new file mode 100644 index 00000000..7720af7a --- /dev/null +++ b/tests/testsuites/rscript_optimizer1.conf @@ -0,0 +1,12 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + +/* tcpflood uses local4.=debug */ +if prifilt("syslog.*") then + stop # it actually doesn't matter what we do here +else + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/rscript_prifilt.conf b/tests/testsuites/rscript_prifilt.conf new file mode 100644 index 00000000..8cb13a0f --- /dev/null +++ b/tests/testsuites/rscript_prifilt.conf @@ -0,0 +1,10 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + +/* tcpflood uses local4.=debug, we use a bit more generic filter */ +if prifilt("local4.*") then + action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/rscript_ruleset_call.conf b/tests/testsuites/rscript_ruleset_call.conf new file mode 100644 index 00000000..96eab293 --- /dev/null +++ b/tests/testsuites/rscript_ruleset_call.conf @@ -0,0 +1,22 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="msg" field.delimiter="58" field.number="2") + constant(value="\n") +} + + +# we deliberately include continue/stop to make sure we have more than +# one statement. This catches grammar erorrs +ruleset(name="rs2") { + continue + action(type="omfile" file="./rsyslog.out.log" template="outfmt") + stop +} + +# this time we make sure a single statement is properly supported +ruleset(name="rs1") { + call rs2 +} + +if $msg contains 'msgnum' then call rs1 diff --git a/tests/testsuites/rscript_stop.conf b/tests/testsuites/rscript_stop.conf new file mode 100644 index 00000000..ab9569e5 --- /dev/null +++ b/tests/testsuites/rscript_stop.conf @@ -0,0 +1,13 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if $msg contains 'msgnum' then { + set $!usr!msgnum = field($msg, 58, 2); + if cnum($!usr!msgnum) >= 5000 then + stop + action(type="omfile" file="./rsyslog.out.log" template="outfmt") +} diff --git a/tests/testsuites/rscript_stop2.conf b/tests/testsuites/rscript_stop2.conf new file mode 100644 index 00000000..9ac9143e --- /dev/null +++ b/tests/testsuites/rscript_stop2.conf @@ -0,0 +1,18 @@ +$IncludeConfig diag-common.conf + +template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") +} + +if not ($msg contains 'msgnum') then + stop + +set $!usr!msgnum = field($msg, 58, 2); +if cnum($!usr!msgnum) >= 5000 then + stop +/* We could use yet another method, but we like to have the action statement + * without a filter in rsyslog.conf top level hierarchy - so this test, as + * a side-effect, also tests this ability. + */ +action(type="omfile" file="./rsyslog.out.log" template="outfmt") diff --git a/tests/testsuites/tcp_forwarding_tpl.conf b/tests/testsuites/tcp_forwarding_tpl.conf new file mode 100644 index 00000000..686c73ac --- /dev/null +++ b/tests/testsuites/tcp_forwarding_tpl.conf @@ -0,0 +1,7 @@ +$IncludeConfig diag-common.conf +$MainMsgQueueTimeoutShutdown 10000 +template(name="outfmt" type="string" string="%msg:F,58:2%\n") + +if $msg contains "msgnum:" then + action(type="omfwd" template="outfmt" + target="127.0.0.1" port="13514" protocol="tcp") diff --git a/tests/threadingmq.sh b/tests/threadingmq.sh index ea5d7837..98f195d4 100755 --- a/tests/threadingmq.sh +++ b/tests/threadingmq.sh @@ -11,6 +11,9 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup threadingmq.conf source $srcdir/diag.sh injectmsg 0 100000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +# we give an extra seconds for things to settle, especially +# important on slower test machines +./msleep 1000 source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 99999 source $srcdir/diag.sh exit diff --git a/tests/tls-certs/ca-key.pem b/tests/tls-certs/ca-key.pem new file mode 100644 index 00000000..181a8ad9 --- /dev/null +++ b/tests/tls-certs/ca-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXQIBAAKBgQDDaz5X5YIruPH0wukMPik7xIKqrpCcr8Gm28oz5h4GtX253eWr +piBuk2a/f/CKDjeuqmiWqTs90PFNb+Z1c+Yzvagqv80VzZwDI4RcrwlNaKrBz/9X +iowCcoV8s7GvV2vtZEPSThNzz4FYkxCMvbOYZeJIYQVhZggUcuadfhmDIwIDAQAB +AoGAIG5AUD2jmYDzD+UhiultVgtkifyNaEtsuQsZu/zbt85P2VQ0z4SINlbvrXvc +iJ9tEzzEPa3udHGj/MTDe3OAB4TK5tImX1pe2gw+zaOB/DaH5i4QhXeltU7epCHF +oUv9EVNzL8Bl00MFiWcLY0LisQVfHeW5rcN9U7EbvTlWbRkCQQDR2/Qn1ceavwDU +qYt2TbEicJVC8aQMYYyc6Xvi4mZaNa8gGCpWpurgQop0Ln0QE8vA0601UVs6N3tm +g8FJ8rXpAkEA7mKCtp2MXCbHMdkZCyQt6drUYCyU9N/HtmBEtFGpqt1PdMyUI07m +rlVFDwUH9JFmg18RP1X2ufj7+ZbJzaMtKwJBAJgbw1Z0P19Mfj+mPC2dlnyN+cIx +/2Px+Mdq/J6w1tsf+jVbDqUMC0ZNNKmNYJycnJzBUNRKicMin9DoQttkjrECQQCC +s/aRY+6adBSRi0QE7NBTwUzicm81mCDrKPtilsfdTDyNgMHUXiVy/oO/yXVkLfi0 +HQLa5CpEK3UUkw2Qt2BDAkA0XXvQzW0+tEHiktLNljIluhiyOAx2bBywY/9Qmn6C +hv4sOSCzTR39jNmuNZ0X6ZZvt4VsWTHhpche/ud1+3p6 +-----END RSA PRIVATE KEY----- diff --git a/tests/tls-certs/ca.pem b/tests/tls-certs/ca.pem new file mode 100644 index 00000000..6324c7d5 --- /dev/null +++ b/tests/tls-certs/ca.pem @@ -0,0 +1,15 @@ +-----BEGIN CERTIFICATE----- +MIICYjCCAc2gAwIBAgIBATALBgkqhkiG9w0BAQUwWDELMAkGA1UEBhMCREUxHTAb +BgNVBAoTFHJzeXNsb2cgdGVzdCByb290IENBMQswCQYDVQQLEwJDQTEdMBsGA1UE +AxMUcnN5c2xvZy10ZXN0LXJvb3QtY2EwHhcNMDgwNTIwMTI1ODEyWhcNMTgwNTE4 +MTI1ODI0WjBYMQswCQYDVQQGEwJERTEdMBsGA1UEChMUcnN5c2xvZyB0ZXN0IHJv +b3QgQ0ExCzAJBgNVBAsTAkNBMR0wGwYDVQQDExRyc3lzbG9nLXRlc3Qtcm9vdC1j +YTCBnDALBgkqhkiG9w0BAQEDgYwAMIGIAoGAw2s+V+WCK7jx9MLpDD4pO8SCqq6Q +nK/BptvKM+YeBrV9ud3lq6YgbpNmv3/wig43rqpolqk7PdDxTW/mdXPmM72oKr/N +Fc2cAyOEXK8JTWiqwc//V4qMAnKFfLOxr1dr7WRD0k4Tc8+BWJMQjL2zmGXiSGEF +YWYIFHLmnX4ZgyMCAwEAAaNDMEEwDwYDVR0TAQH/BAUwAwEB/zAPBgNVHQ8BAf8E +BQMDBwYAMB0GA1UdDgQWBBQzYQQgUm0YLNdarJnc2c1LxYVClDALBgkqhkiG9w0B +AQUDgYEAuGWtH7Jkpa0n/izqQ5ddDQP/LT6taivCwlpEYEU9aumpQPWWxtYywKaP +RfM1JTMLAiYd8MS7TJ8TYRvvR32Y02Y+OhXn11xERkWvBT2M9yzqX6hDfRueN7RT +fPWsfm/NBTVojzjaECcTFenZid7PC5JiFbcU6PSUMZ49/JPhxAo= +-----END CERTIFICATE----- diff --git a/tests/tls-certs/cert.pem b/tests/tls-certs/cert.pem new file mode 100644 index 00000000..6b5b13cd --- /dev/null +++ b/tests/tls-certs/cert.pem @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIIChjCCAfGgAwIBAgIBADALBgkqhkiG9w0BAQUwWDELMAkGA1UEBhMCREUxHTAb +BgNVBAoTFHJzeXNsb2cgdGVzdCByb290IENBMQswCQYDVQQLEwJDQTEdMBsGA1UE +AxMUcnN5c2xvZy10ZXN0LXJvb3QtY2EwHhcNMDgwNTIwMTMwNDE5WhcNMTgwNTE4 +MTMwNDI2WjA6MQswCQYDVQQGEwJERTEQMA4GA1UEChMHcnN5c2xvZzEZMBcGA1UE +CxMQdGVzdCBjZXJ0aWZpY2F0ZTCBnDALBgkqhkiG9w0BAQEDgYwAMIGIAoGAxmHe +fztJgaGxFYEceiUg0hdMlRVWBqoZelJ8BeXTDnXcu/5F2HtM+l+QDyDaGjKlx+NI +K4rkj7d6Wd3AKPgOYS0VSDZe3a1xf9rRYzOthWTv7tYi4/LTqPXqN5lKE71dgrB/ +/gOmvV/1YD776FIxVGCSAT0hHwkFC3slmpJSwD8CAwEAAaOBhDCBgTAMBgNVHRMB +Af8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHREECzAJ +ggdyc3lzbG9nMB0GA1UdDgQWBBQYu6eC9UALvC+5K5VOnFRi5OC98TAfBgNVHSME +GDAWgBQzYQQgUm0YLNdarJnc2c1LxYVClDALBgkqhkiG9w0BAQUDgYEAXaymqsG9 +PNBhhWIRFvXCDMaDM71vUtgSFoNUbxIV607ua2HQosPPM4EHIda6N6hdBK1bMQoG +yqBwhvw0JVaVaO70Kbs2m2Ypk3YcpJtRqyp8q8+2y/w1Mk1QazFZC29aYgX2iNVf +X4/x38YEL7Gu5vqPrTn++agnV4ZXECKuvLQ= +-----END CERTIFICATE----- diff --git a/tests/tls-certs/key.pem b/tests/tls-certs/key.pem new file mode 100644 index 00000000..3ff507f0 --- /dev/null +++ b/tests/tls-certs/key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICWwIBAAKBgQDGYd5/O0mBobEVgRx6JSDSF0yVFVYGqhl6UnwF5dMOddy7/kXY +e0z6X5APINoaMqXH40griuSPt3pZ3cAo+A5hLRVINl7drXF/2tFjM62FZO/u1iLj +8tOo9eo3mUoTvV2CsH/+A6a9X/VgPvvoUjFUYJIBPSEfCQULeyWaklLAPwIDAQAB +AoGARIwKqmHc+0rYenq7UUVE+vMMBjNyHyllVkvsCMmpzMRS+i5ZCf1I0vZ0O5X5 +ZrX7bH8PL+R1J2eZgjXKMR3NMZBuyKHewItD9t2rIC0eD/ITlwq3VybbaMsw666e +INxSmax+dS5CEcLevHHP3c+Q7S7QAFiWV43TdFUGXWJktIkCQQDPQ5WAZ+/Tvv0Q +vtRjXMeTVaw/bSuKNUeDzFkmGyePnFeCReNFtJLE9PFSQWcPuYcbZgU59JTfA5ac +Un+cHm31AkEA9Qek+q7PcJ+kON9E6SNodCZn6gLyHjnWrq4tf8pZO3NvoX2QiuD4 +rwF7KWjr6q1JzADpLtwXnuYEhyiLFjJA4wJAcElMCEnG2y+ASH8p7z7HfKGQdLg/ +O1wMB3JA5e0WLK5lllUogI4IaZ3N02NNY25+rLBDqpc/w+ZcxQnIypqNtQJATs9p +ofON5wSB1oUBbhckZo9fxuWxqEUkJsUA/2Q+9R843XE8h166vdc1HOmRT8bywHne +hmLl+gazmCFTMw1wzwJAHng+3zGUl4D8Ov3MPFD6hwYYK6/pEdtz/NUsCSazF7eK +XuuP+DXPHNhXOuF1A3tP74pfc/fC1uCUH2G5z3Fy0Q== +-----END RSA PRIVATE KEY----- @@ -184,8 +184,10 @@ static void* thrdStarter(void *arg) assert(pThis->pUsrThrdMain != NULL); # if HAVE_PRCTL && defined PR_SET_NAME - /* set thread name - we ignore if the call fails, has no harsh consequences... */ ustrncpy(thrdName+3, pThis->name, 20); + dbgOutputTID((char*)thrdName); + + /* set thread name - we ignore if the call fails, has no harsh consequences... */ if(prctl(PR_SET_NAME, thrdName, 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", pThis->name); } else { diff --git a/tools/Makefile.am b/tools/Makefile.am index e634076c..6832494e 100644 --- a/tools/Makefile.am +++ b/tools/Makefile.am @@ -1,5 +1,7 @@ sbin_PROGRAMS = bin_PROGRAMS = +CLEANFILES = +man1_MANS = man_MANS = rsyslogd.8 rsyslog.conf.5 sbin_PROGRAMS += rsyslogd @@ -36,13 +38,20 @@ rsyslogd_SOURCES = \ pidfile.h \ \ ../dirty.h -rsyslogd_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) $(CNF_LIBS) +rsyslogd_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) # note: it looks like librsyslog.la must be explicitely given on LDDADD, # otherwise dependencies are not properly calculated (resulting in a # potentially incomplete build, a problem we had several times...) -rsyslogd_LDADD = ../grammar/libgrammar.la ../runtime/librsyslog.la $(ZLIB_LIBS) $(PTHREADS_LIBS) $(RSRT_LIBS) $(SOL_LIBS) $(LIBEE_LIBS) $(LIBLOGNORM_LIBS) $(LIBUUID_LIBS) +rsyslogd_LDADD = ../grammar/libgrammar.la ../runtime/librsyslog.la $(ZLIB_LIBS) $(PTHREADS_LIBS) $(RSRT_LIBS) $(SOL_LIBS) $(LIBUUID_LIBS) rsyslogd_LDFLAGS = -export-dynamic +EXTRA_DIST = $(man_MANS) \ + rsgtutil.rst \ + rsgtutil.1 \ + rscryutil.rst \ + rscryutil.1 \ + recover_qi.pl + if ENABLE_DIAGTOOLS sbin_PROGRAMS += rsyslog_diag_hostname msggen zpipe rsyslog_diag_hostname_SOURCES = gethostn.c @@ -58,7 +67,26 @@ logctl_SOURCES = logctl.c logctl_CPPFLAGS = $(LIBMONGO_CLIENT_CFLAGS) logctl_LDADD = $(LIBMONGO_CLIENT_LIBS) endif +if ENABLE_GUARDTIME +bin_PROGRAMS += rsgtutil +rsgtutil = rsgtutil.c +rsgtutil_CPPFLAGS = $(RSRT_CFLAGS) $(GUARDTIME_CFLAGS) +rsgtutil_LDADD = ../runtime/librsgt.la $(GUARDTIME_LIBS) +rsgtutil.1: rsgtutil.rst + $(AM_V_GEN) $(RST2MAN) $< $@ +man1_MANS += rsgtutil.1 +CLEANFILES += rsgtutil.1 +EXTRA_DIST+= rsgtutil.1 +endif +if ENABLE_LIBGCRYPT +bin_PROGRAMS += rscryutil +rscryutil = rscryutil.c +rscryutil_CPPFLAGS = -I../runtime $(RSRT_CFLAGS) $(LIBGCRYPT_CFLAGS) +rscryutil_LDADD = ../runtime/libgcry.la $(LIBGCRYPT_LIBS) +rscryutil.1: rscryutil.rst + $(AM_V_GEN) $(RST2MAN) $< $@ +man1_MANS += rscryutil.1 +CLEANFILES += rscryutil.1 +EXTRA_DIST+= rscryutil.1 +endif endif - -EXTRA_DIST = $(man_MANS) \ - recover_qi.pl diff --git a/tools/logctl.c b/tools/logctl.c index df332bc2..1ab8ead0 100644 --- a/tools/logctl.c +++ b/tools/logctl.c @@ -143,7 +143,6 @@ struct ofields* get_data(struct results *res) struct ofields *fields; const char *msg; const char *prog; - const char *level; const char *syslog_tag; gint64 date_r; bson_cursor *c; @@ -263,7 +262,7 @@ struct select_doc* create_select() struct query_doc* create_query(struct queryopt *opt) { struct query_doc *qu_doc; - bson *query_what, *order_what, *order_how, *msg_what, *date_what; + bson *query_what, *order_what, *msg_what, *date_what; struct tm tm; time_t t; gint64 ts; @@ -417,7 +416,6 @@ int main (int argc, char *argv[]) struct queryopt opt; struct ofields *fields; - struct bson_doc *doc; struct select_doc *s_doc; struct query_doc *qu_doc; struct db_connect *db_conn; diff --git a/tools/logsigner.c b/tools/logsigner.c new file mode 100644 index 00000000..f6887696 --- /dev/null +++ b/tools/logsigner.c @@ -0,0 +1,159 @@ +/* This is a tool for offline signing logfiles via the guardtime API. + * + * NOTE: this currently is only a PoC and WiP! NOT suitable for + * production use! + * + * Current hardcoded timestamper (use this if you do not have an + * idea of which one else to use): + * http://stamper.guardtime.net/gt-signingservice + * Check the GuardTime website for the URLs of nearest public services. + * + * Copyright 2013 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either exprs or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <gt_base.h> +#include <gt_http.h> + +#include "librsgt.h" + + +#if 0 +void +outputhash(GTDataHash *hash) +{ + int i; + for(i = 0 ; i < hash->digest_length ; ++i) + printf("%2.2x", hash->digest[i]); + printf("\n"); +} + +void +timestampIt(GTDataHash *hash) +{ + int r = GT_OK; + GTTimestamp *timestamp = NULL; + unsigned char *der = NULL; + char *sigFile = "logsigner.TIMESTAMP"; + size_t der_len; + + /* Get the timestamp. */ + r = GTHTTP_createTimestampHash(hash, + "http://stamper.guardtime.net/gt-signingservice", ×tamp); + + if(r != GT_OK) { + fprintf(stderr, "GTHTTP_createTimestampHash() failed: %d (%s)\n", + r, GTHTTP_getErrorString(r)); + goto done; + } + + /* Encode timestamp. */ + r = GTTimestamp_getDEREncoded(timestamp, &der, &der_len); + if(r != GT_OK) { + fprintf(stderr, "GTTimestamp_getDEREncoded() failed: %d (%s)\n", + r, GT_getErrorString(r)); + goto done; + } + + /* Save DER-encoded timestamp to file. */ + r = GT_saveFile(sigFile, der, der_len); + if(r != GT_OK) { + fprintf(stderr, "Cannot save timestamp to file %s: %d (%s)\n", + sigFile, r, GT_getErrorString(r)); + if(r == GT_IO_ERROR) { + fprintf(stderr, "\t%d (%s)\n", errno, strerror(errno)); + } + goto done; + } + printf("Timestamping succeeded!\n"); +done: + GT_free(der); + GTTimestamp_free(timestamp); +} + + +void +sign(const char *buf, const size_t len) +{ + int r; + GTDataHash *hash = NULL; + + printf("hash for '%s' is ", buf); + r = GTDataHash_create(GT_HASHALG_SHA256, (const unsigned char*)buf, len, &hash); + if(r != GT_OK) { + fprintf(stderr, "GTTDataHash_create() failed: %d (%s)\n", + r, GT_getErrorString(r)); + goto done; + } + outputhash(hash); + timestampIt(hash); /* of course, this needs to be moved to once at end ;) */ +done: GTDataHash_free(hash); +} +#endif + +void +processFile(char *name) +{ + FILE *fp; + size_t len; + char line[64*1024+1]; + gtctx ctx = NULL; + + ctx = rsgtCtxNew((unsigned char*)"SIGFILE", GT_HASHALG_SHA256); + sigblkInit(ctx); + if(!strcmp(name, "-")) + fp = stdin; + else + fp = fopen(name, "r"); + + while(1) { + if(fgets(line, sizeof(line), fp) == NULL) { + if(!feof(fp)) + perror(name); + break; + } + len = strlen(line); + if(line[len-1] == '\n') { + --len; + line[len] = '\0'; + } + //sign(line, len); + sigblkAddRecord(ctx, (unsigned char*)line, len); + } + + if(fp != stdin) + fclose(fp); + sigblkFinish(ctx); + rsgtCtxDel(ctx); +} + + +int +main(int argc, char *argv[]) +{ + rsgtInit("rsyslog logsigner " VERSION); + processFile("-"); + rsgtExit(); + return 0; +} diff --git a/tools/omdiscard.c b/tools/omdiscard.c index 182c4b63..15c6ea82 100644 --- a/tools/omdiscard.c +++ b/tools/omdiscard.c @@ -35,6 +35,7 @@ #include "syslogd-types.h" #include "omdiscard.h" #include "module-template.h" +#include "errmsg.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -42,6 +43,7 @@ MODULE_TYPE_NOKEEP /* internal structures */ DEF_OMOD_STATIC_DATA +DEFobjCurrIf(errmsg); typedef struct _instanceData { EMPTY_STRUCT @@ -92,8 +94,10 @@ CODE_STD_STRING_REQUESTparseSelectorAct(0) p = *pp; if(*p == '~') { - /* TODO: check the rest of the selector line - error reporting */ dbgprintf("discard\n"); + errmsg.LogError(0, RS_RET_DEPRECATED, "warning: ~ action " + "is deprecated, consider using the 'stop' " + "statement instead"); } else { iRet = RS_RET_CONFLINE_UNPROCESSED; } @@ -103,6 +107,7 @@ ENDparseSelectorAct BEGINmodExit CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); ENDmodExit @@ -116,6 +121,7 @@ BEGINmodInit(Discard) CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); ENDmodInit /* * vi:set ai: diff --git a/tools/omfile.c b/tools/omfile.c index 715b218c..ba9f7f70 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -17,7 +17,7 @@ * pipes. These have been moved to ompipe, to reduced the entanglement * between the two different functionalities. -- rgerhards * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -68,6 +68,9 @@ #include "stream.h" #include "unicode-helper.h" #include "atomic.h" +#include "statsobj.h" +#include "sigprov.h" +#include "cryprov.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -81,6 +84,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(strm) +DEFobjCurrIf(statsobj) /* for our current LRU mechanism, we need a monotonically increasing counters. We use * it much like a "Lamport logical clock": we do not need the actual time, we just need @@ -116,6 +120,7 @@ getClockFileAccess(void) struct s_dynaFileCacheEntry { uchar *pName; /* name currently open, if dynamic name */ strm_t *pStrm; /* our output stream */ + void *sigprovFileData; /* opaque data ptr for provider use */ uint64 clkTickAccessed;/* for LRU - based on clockFileAccess */ }; typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; @@ -141,6 +146,18 @@ typedef struct _instanceData { gid_t fileGID; gid_t dirGID; int bFailOnChown; /* fail creation if chown fails? */ + uchar *sigprovName; /* signature provider */ + uchar *sigprovNameFull;/* full internal signature provider name */ + sigprov_if_t sigprov; /* ptr to signature provider interface */ + void *sigprovData; /* opaque data ptr for provider use */ + void *sigprovFileData;/* opaque data ptr for file instance */ + sbool useSigprov; /* quicker than checkig ptr (1 vs 8 bytes!) */ + uchar *cryprovName; /* crypto provider */ + uchar *cryprovNameFull;/* full internal crypto provider name */ + void *cryprovData; /* opaque data ptr for provider use */ + void *cryprovFileData;/* opaque data ptr for file instance */ + cryprov_if_t cryprov; /* ptr to crypto provider interface */ + sbool useCryprov; /* quicker than checkig ptr (1 vs 8 bytes!) */ int iCurrElt; /* currently active cache element (-1 = none) */ int iCurrCacheSize; /* currently cache size (1-based) */ int iDynaFileCacheSize; /* size of file handle cache */ @@ -156,6 +173,13 @@ typedef struct _instanceData { int iFlushInterval; /* how fast flush buffer on inactivity? */ sbool bFlushOnTXEnd; /* flush write buffers when transaction has ended? */ sbool bUseAsyncWriter; /* use async stream writer? */ + sbool bVeryRobustZip; + statsobj_t *stats; /* dynafile, primarily cache stats */ + STATSCOUNTER_DEF(ctrRequests, mutCtrRequests); + STATSCOUNTER_DEF(ctrLevel0, mutCtrLevel0); + STATSCOUNTER_DEF(ctrEvict, mutCtrEvict); + STATSCOUNTER_DEF(ctrMiss, mutCtrMiss); + STATSCOUNTER_DEF(ctrMax, mutCtrMax); } instanceData; @@ -205,6 +229,7 @@ static struct cnfparamdescr actpdescr[] = { { "ziplevel", eCmdHdlrInt, 0 }, /* legacy: omfileziplevel */ { "flushinterval", eCmdHdlrInt, 0 }, /* legacy: omfileflushinterval */ { "asyncwriting", eCmdHdlrBinary, 0 }, /* legacy: omfileasyncwriting */ + { "veryrobustzip", eCmdHdlrBinary, 0 }, { "flushontxend", eCmdHdlrBinary, 0 }, /* legacy: omfileflushontxend */ { "iobuffersize", eCmdHdlrSize, 0 }, /* legacy: omfileiobuffersize */ { "dirowner", eCmdHdlrUID, 0 }, /* legacy: dirowner */ @@ -218,7 +243,9 @@ static struct cnfparamdescr actpdescr[] = { { "sync", eCmdHdlrBinary, 0 }, /* legacy: actionfileenablesync */ { "file", eCmdHdlrString, 0 }, /* either "file" or ... */ { "dynafile", eCmdHdlrString, 0 }, /* "dynafile" MUST be present */ - { "template", eCmdHdlrGetWord, 0 }, + { "sig.provider", eCmdHdlrGetWord, 0 }, + { "cry.provider", eCmdHdlrGetWord, 0 }, + { "template", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -269,7 +296,8 @@ CODESTARTdbgPrintInstInfo dbgprintf("\tflush on TX end=%d\n", pData->bFlushOnTXEnd); dbgprintf("\tflush interval=%d\n", pData->iFlushInterval); dbgprintf("\tfile cache size=%d\n", pData->iDynaFileCacheSize); - dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "yes" : "no"); + dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "on" : "off"); + dbgprintf("\tvery robust zip: %s\n", pData->bCreateDirs ? "on" : "off"); dbgprintf("\tfile owner %d, group %d\n", (int) pData->fileUID, (int) pData->fileGID); dbgprintf("\tdirectory owner %d, group %d\n", (int) pData->dirUID, (int) pData->dirGID); dbgprintf("\tdir create mode 0%3.3o, file create mode 0%3.3o\n", @@ -292,7 +320,7 @@ setLegacyDfltTpl(void __attribute__((unused)) *pVal, uchar* newVal) if(loadModConf != NULL && loadModConf->tplName != NULL) { free(newVal); - errmsg.LogError(0, RS_RET_ERR, "omfile default template already set via module " + errmsg.LogError(0, RS_RET_ERR, "omfile: default template already set via module " "global parameter - can no longer be changed"); ABORT_FINALIZE(RS_RET_ERR); } @@ -405,15 +433,16 @@ finalize_it: * if the entry should be d_free()ed and 0 if not. */ static rsRetVal -dynaFileDelCacheEntry(dynaFileCacheEntry **pCache, int iEntry, int bFreeEntry) +dynaFileDelCacheEntry(instanceData *pData, int iEntry, int bFreeEntry) { + dynaFileCacheEntry **pCache = pData->dynCache; DEFiRet; ASSERT(pCache != NULL); if(pCache[iEntry] == NULL) FINALIZE; - DBGPRINTF("Removed entry %d for file '%s' from dynaCache.\n", iEntry, + DBGPRINTF("Removing entry %d for file '%s' from dynaCache.\n", iEntry, pCache[iEntry]->pName == NULL ? UCHAR_CONSTANT("[OPEN FAILED]") : pCache[iEntry]->pName); if(pCache[iEntry]->pName != NULL) { @@ -423,8 +452,10 @@ dynaFileDelCacheEntry(dynaFileCacheEntry **pCache, int iEntry, int bFreeEntry) if(pCache[iEntry]->pStrm != NULL) { strm.Destruct(&pCache[iEntry]->pStrm); - if(pCache[iEntry]->pStrm != NULL) /* safety check -- TODO: remove if no longer necessary */ - abort(); + if(pData->useSigprov) { + pData->sigprov.OnFileClose(pCache[iEntry]->sigprovFileData); + pCache[iEntry]->sigprovFileData = NULL; + } } if(bFreeEntry) { @@ -449,7 +480,7 @@ dynaFileFreeCacheEntries(instanceData *pData) BEGINfunc; for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { - dynaFileDelCacheEntry(pData->dynCache, i, 1); + dynaFileDelCacheEntry(pData, i, 1); } pData->iCurrElt = -1; /* invalidate current element */ ENDfunc; @@ -470,6 +501,29 @@ static void dynaFileFreeCache(instanceData *pData) } +/* close current file */ +static rsRetVal +closeFile(instanceData *pData) +{ + DEFiRet; + if(pData->useSigprov) { + pData->sigprov.OnFileClose(pData->sigprovFileData); + pData->sigprovFileData = NULL; + } + strm.Destruct(&pData->pStrm); + RETiRet; +} + + +/* This prepares the signature provider to process a file */ +static rsRetVal +sigprovPrepare(instanceData *pData, uchar *fn) +{ + DEFiRet; + pData->sigprov.OnFileOpen(pData->sigprovData, fn, &pData->sigprovFileData); + RETiRet; +} + /* This is now shared code for all types of files. It simply prepares * file access, which, among others, means the the file wil be opened * and any directories in between will be created (based on config, of @@ -536,12 +590,17 @@ prepareFile(instanceData *pData, uchar *newFileName) CHKiRet(strm.SetFName(pData->pStrm, szBaseName, ustrlen(szBaseName))); CHKiRet(strm.SetDir(pData->pStrm, szDirName, ustrlen(szDirName))); CHKiRet(strm.SetiZipLevel(pData->pStrm, pData->iZipLevel)); + CHKiRet(strm.SetbVeryReliableZip(pData->pStrm, pData->bVeryRobustZip)); CHKiRet(strm.SetsIOBufSize(pData->pStrm, (size_t) pData->iIOBufSize)); CHKiRet(strm.SettOperationsMode(pData->pStrm, STREAMMODE_WRITE_APPEND)); CHKiRet(strm.SettOpenMode(pData->pStrm, cs.fCreateMode)); CHKiRet(strm.SetbSync(pData->pStrm, pData->bSyncFile)); CHKiRet(strm.SetsType(pData->pStrm, STREAMTYPE_FILE_SINGLE)); CHKiRet(strm.SetiSizeLimit(pData->pStrm, pData->iSizeLimit)); + if(pData->useCryprov) { + CHKiRet(strm.Setcryprov(pData->pStrm, &pData->cryprov)); + CHKiRet(strm.SetcryprovData(pData->pStrm, pData->cryprovData)); + } /* set the flush interval only if we actually use it - otherwise it will activate * async processing, which is a real performance waste if we do not do buffered * writes! -- rgerhards, 2009-07-06 @@ -551,11 +610,14 @@ prepareFile(instanceData *pData, uchar *newFileName) if(pData->pszSizeLimitCmd != NULL) CHKiRet(strm.SetpszSizeLimitCmd(pData->pStrm, ustrdup(pData->pszSizeLimitCmd))); CHKiRet(strm.ConstructFinalize(pData->pStrm)); + + if(pData->useSigprov) + sigprovPrepare(pData, szNameBuf); finalize_it: if(iRet != RS_RET_OK) { if(pData->pStrm != NULL) { - strm.Destruct(&pData->pStrm); + closeFile(pData); } } RETiRet; @@ -586,14 +648,13 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) pCache = pData->dynCache; - /* first check, if we still have the current file - * I *hope* this will be a performance enhancement. - */ + /* first check, if we still have the current file */ if( (pData->iCurrElt != -1) && !ustrcmp(newFileName, pCache[pData->iCurrElt]->pName)) { /* great, we are all set */ pCache[pData->iCurrElt]->clkTickAccessed = getClockFileAccess(); - // LRU needs only a strictly monotonically increasing counter, so such a one could do + STATSCOUNTER_INC(pData->ctrLevel0, pData->mutCtrLevel0); + /* LRU needs only a strictly monotonically increasing counter, so such a one could do */ FINALIZE; } @@ -609,9 +670,11 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) if(iFirstFree == -1) iFirstFree = i; } else { /* got an element, let's see if it matches */ - if(!ustrcmp(newFileName, pCache[i]->pName)) { // RG: name == NULL? + if(!ustrcmp(newFileName, pCache[i]->pName)) { /* we found our element! */ pData->pStrm = pCache[i]->pStrm; + if(pData->useSigprov) + pData->sigprovFileData = pCache[i]->sigprovFileData; pData->iCurrElt = i; pCache[i]->clkTickAccessed = getClockFileAccess(); /* update "timestamp" for LRU */ FINALIZE; @@ -625,6 +688,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) } /* we have not found an entry */ + STATSCOUNTER_INC(pData->ctrMiss, pData->mutCtrMiss); /* invalidate iCurrElt as we may error-exit out of this function when the currrent * iCurrElt has been freed or otherwise become unusable. This is a precaution, and @@ -637,11 +701,12 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) * but it could be triggered in the common case of a failed open() system call. * rgerhards, 2010-03-22 */ - pData->pStrm = NULL; + pData->pStrm = NULL, pData->sigprovFileData = NULL; if(iFirstFree == -1 && (pData->iCurrCacheSize < pData->iDynaFileCacheSize)) { /* there is space left, so set it to that index */ iFirstFree = pData->iCurrCacheSize++; + STATSCOUNTER_SETMAX_NOMUT(pData->ctrMax, (unsigned) pData->iCurrCacheSize); } /* Note that the following code sequence does not work with the cache entry itself, @@ -649,13 +714,11 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) * The cache array is only updated after the open was successful. -- rgerhards, 2010-03-21 */ if(iFirstFree == -1) { - dynaFileDelCacheEntry(pCache, iOldest, 0); + dynaFileDelCacheEntry(pData, iOldest, 0); + STATSCOUNTER_INC(pData->ctrEvict, pData->mutCtrEvict); iFirstFree = iOldest; /* this one *is* now free ;) */ } else { /* we need to allocate memory for the cache structure */ - /* TODO: performance note: we could alloc all entries on startup, thus saving malloc - * overhead -- this may be something to consider in v5... - */ CHKmalloc(pCache[iFirstFree] = (dynaFileCacheEntry*) calloc(1, sizeof(dynaFileCacheEntry))); } @@ -678,10 +741,12 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) } if((pCache[iFirstFree]->pName = ustrdup(newFileName)) == NULL) { - strm.Destruct(&pData->pStrm); /* need to free failed entry! */ + closeFile(pData); /* need to free failed entry! */ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } pCache[iFirstFree]->pStrm = pData->pStrm; + if(pData->useSigprov) + pCache[iFirstFree]->sigprovFileData = pData->sigprovFileData; pCache[iFirstFree]->clkTickAccessed = getClockFileAccess(); pData->iCurrElt = iFirstFree; DBGPRINTF("Added new entry %d for file cache, file '%s'.\n", iFirstFree, newFileName); @@ -706,7 +771,9 @@ doWrite(instanceData *pData, uchar *pszBuf, int lenBuf) DBGPRINTF("write to stream, pData->pStrm %p, lenBuf %d\n", pData->pStrm, lenBuf); if(pData->pStrm != NULL){ CHKiRet(strm.Write(pData->pStrm, pszBuf, lenBuf)); - FINALIZE; + if(pData->useSigprov) { + CHKiRet(pData->sigprov.OnRecordWrite(pData->sigprovFileData, pszBuf, lenBuf)); + } } finalize_it: @@ -714,10 +781,7 @@ finalize_it: } -/* rgerhards 2004-11-11: write to a file output. This - * will be called for all outputs using file semantics, - * for example also for pipes. - */ +/* rgerhards 2004-11-11: write to a file output. */ static rsRetVal writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) { @@ -825,7 +889,21 @@ CODESTARTfreeInstance if(pData->bDynamicName) { dynaFileFreeCache(pData); } else if(pData->pStrm != NULL) - strm.Destruct(&pData->pStrm); + closeFile(pData); + if(pData->useSigprov) { + pData->sigprov.Destruct(&pData->sigprovData); + obj.ReleaseObj(__FILE__, pData->sigprovNameFull+2, pData->sigprovNameFull, + (void*) &pData->sigprov); + free(pData->sigprovName); + free(pData->sigprovNameFull); + } + if(pData->useCryprov) { + pData->cryprov.Destruct(&pData->cryprovData); + obj.ReleaseObj(__FILE__, pData->cryprovNameFull+2, pData->cryprovNameFull, + (void*) &pData->cryprov); + free(pData->cryprovName); + free(pData->cryprovNameFull); + } ENDfreeInstance @@ -843,7 +921,12 @@ BEGINendTransaction CODESTARTendTransaction /* Note: pStrm may be NULL if there was an error opening the stream */ if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { - CHKiRet(strm.Flush(pData->pStrm)); + /* if we have an async writer, it controls the flush via + * a timeout. However, without it, we actually need to flush, + * else incomplete records are written. + */ + if(!pData->bUseAsyncWriter) + CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: ENDendTransaction @@ -851,7 +934,10 @@ ENDendTransaction BEGINdoAction CODESTARTdoAction - DBGPRINTF("file to log to: %s\n", pData->f_fname); + DBGPRINTF("file to log to: %s\n", + (pData->bDynamicName) ? ppString[1] : pData->f_fname); + DBGPRINTF("omfile: start of data: '%.128s'\n", ppString[0]); + STATSCOUNTER_INC(pData->ctrRequests, pData->mutCtrRequests); CHKiRet(writeFile(ppString, iMsgOpts, pData)); if(!bCoreSupportsBatching && pData->bFlushOnTXEnd) { CHKiRet(strm.Flush(pData->pStrm)); @@ -878,14 +964,143 @@ setInstParamDefaults(instanceData *pData) pData->bCreateDirs = 1; pData->bSyncFile = 0; pData->iZipLevel = 0; + pData->bVeryRobustZip = 0; pData->bFlushOnTXEnd = FLUSHONTX_DFLT; pData->iIOBufSize = IOBUF_DFLT_SIZE; pData->iFlushInterval = FLUSH_INTRVL_DFLT; pData->bUseAsyncWriter = USE_ASYNCWRITER_DFLT; + pData->sigprovName = NULL; + pData->cryprovName = NULL; + pData->useSigprov = 0; + pData->useCryprov = 0; +} + + +static rsRetVal +setupInstStatsCtrs(instanceData *pData) +{ + uchar ctrName[512]; + DEFiRet; + + if(!pData->bDynamicName) { + FINALIZE; + } + + /* support statistics gathering */ + snprintf((char*)ctrName, sizeof(ctrName), "dynafile cache %s", pData->f_fname); + ctrName[sizeof(ctrName)-1] = '\0'; /* be on the save side */ + CHKiRet(statsobj.Construct(&(pData->stats))); + CHKiRet(statsobj.SetName(pData->stats, ctrName)); + STATSCOUNTER_INIT(pData->ctrRequests, pData->mutCtrRequests); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("requests"), + ctrType_IntCtr, &(pData->ctrRequests))); + STATSCOUNTER_INIT(pData->ctrLevel0, pData->mutCtrLevel0); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("level0"), + ctrType_IntCtr, &(pData->ctrLevel0))); + STATSCOUNTER_INIT(pData->ctrMiss, pData->mutCtrMiss); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("missed"), + ctrType_IntCtr, &(pData->ctrMiss))); + STATSCOUNTER_INIT(pData->ctrEvict, pData->mutCtrEvict); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("evicted"), + ctrType_IntCtr, &(pData->ctrEvict))); + STATSCOUNTER_INIT(pData->ctrMax, pData->mutCtrMax); + CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("maxused"), + ctrType_IntCtr, &(pData->ctrMax))); + CHKiRet(statsobj.ConstructFinalize(pData->stats)); + +finalize_it: + RETiRet; +} + +static inline void +initSigprov(instanceData *pData, struct nvlst *lst) +{ + uchar szDrvrName[1024]; + + if(snprintf((char*)szDrvrName, sizeof(szDrvrName), "lmsig_%s", pData->sigprovName) + == sizeof(szDrvrName)) { + errmsg.LogError(0, RS_RET_ERR, "omfile: signature provider " + "name is too long: '%s' - signatures disabled", + pData->sigprovName); + goto done; + } + pData->sigprovNameFull = ustrdup(szDrvrName); + + pData->sigprov.ifVersion = sigprovCURR_IF_VERSION; + /* The pDrvrName+2 below is a hack to obtain the object name. It + * safes us to have yet another variable with the name without "lm" in + * front of it. If we change the module load interface, we may re-think + * about this hack, but for the time being it is efficient and clean enough. + */ + if(obj.UseObj(__FILE__, szDrvrName, szDrvrName, (void*) &pData->sigprov) + != RS_RET_OK) { + errmsg.LogError(0, RS_RET_LOAD_ERROR, "omfile: could not load " + "signature provider '%s' - signatures disabled", + szDrvrName); + goto done; + } + + if(pData->sigprov.Construct(&pData->sigprovData) != RS_RET_OK) { + errmsg.LogError(0, RS_RET_SIGPROV_ERR, "omfile: error constructing " + "signature provider %s dataset - signatures disabled", + szDrvrName); + goto done; + } + pData->sigprov.SetCnfParam(pData->sigprovData, lst); + + dbgprintf("loaded signature provider %s, data instance at %p\n", + szDrvrName, pData->sigprovData); + pData->useSigprov = 1; +done: return; +} + +static inline rsRetVal +initCryprov(instanceData *pData, struct nvlst *lst) +{ + uchar szDrvrName[1024]; + DEFiRet; + + if(snprintf((char*)szDrvrName, sizeof(szDrvrName), "lmcry_%s", pData->cryprovName) + == sizeof(szDrvrName)) { + errmsg.LogError(0, RS_RET_ERR, "omfile: crypto provider " + "name is too long: '%s' - encryption disabled", + pData->cryprovName); + ABORT_FINALIZE(RS_RET_ERR); + } + pData->cryprovNameFull = ustrdup(szDrvrName); + + pData->cryprov.ifVersion = cryprovCURR_IF_VERSION; + /* The pDrvrName+2 below is a hack to obtain the object name. It + * safes us to have yet another variable with the name without "lm" in + * front of it. If we change the module load interface, we may re-think + * about this hack, but for the time being it is efficient and clean enough. + */ + if(obj.UseObj(__FILE__, szDrvrName, szDrvrName, (void*) &pData->cryprov) + != RS_RET_OK) { + errmsg.LogError(0, RS_RET_LOAD_ERROR, "omfile: could not load " + "crypto provider '%s' - encryption disabled", + szDrvrName); + ABORT_FINALIZE(RS_RET_CRYPROV_ERR); + } + + if(pData->cryprov.Construct(&pData->cryprovData) != RS_RET_OK) { + errmsg.LogError(0, RS_RET_CRYPROV_ERR, "omfile: error constructing " + "crypto provider %s dataset - encryption disabled", + szDrvrName); + ABORT_FINALIZE(RS_RET_CRYPROV_ERR); + } + CHKiRet(pData->cryprov.SetCnfParam(pData->cryprovData, lst)); + + dbgprintf("loaded crypto provider %s, data instance at %p\n", + szDrvrName, pData->cryprovData); + pData->useCryprov = 1; +finalize_it: + RETiRet; } BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; CODESTARTnewActInst DBGPRINTF("newActInst (omfile)\n"); @@ -914,6 +1129,8 @@ CODESTARTnewActInst pData->iZipLevel = (int) pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "flushinterval")) { pData->iFlushInterval = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "veryrobustzip")) { + pData->bVeryRobustZip = pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "asyncwriting")) { pData->bUseAsyncWriter = pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "flushontxend")) { @@ -948,6 +1165,10 @@ CODESTARTnewActInst pData->bDynamicName = 1; } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "sig.provider")) { + pData->sigprovName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "cry.provider")) { + pData->cryprovName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { dbgprintf("omfile: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); @@ -960,7 +1181,16 @@ CODESTARTnewActInst ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); } - CHKiRet(OMSRsetEntry(*ppOMSR, 0, ustrdup(getDfltTpl()), OMSR_NO_RQD_TPL_OPTS)); + if(pData->sigprovName != NULL) { + initSigprov(pData, lst); + } + + if(pData->cryprovName != NULL) { + CHKiRet(initCryprov(pData, lst)); + } + + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); if(pData->bDynamicName) { /* "filename" is actually a template name, we need this as string 1. So let's add it @@ -970,10 +1200,11 @@ CODESTARTnewActInst // TODO: create unified code for this (legacy+v6 system) /* we now allocate the cache table */ CHKmalloc(pData->dynCache = (dynaFileCacheEntry**) - calloc(cs.iDynaFileCacheSize, sizeof(dynaFileCacheEntry*))); + calloc(pData->iDynaFileCacheSize, sizeof(dynaFileCacheEntry*))); pData->iCurrElt = -1; /* no current element */ } // TODO: add pData->iSizeLimit = 0; /* default value, use outchannels to configure! */ + setupInstStatsCtrs(pData); CODE_STD_FINALIZERnewActInst cnfparamvalsDestruct(pvals, &actpblk); @@ -1062,6 +1293,8 @@ CODESTARTparseSelectorAct pData->iIOBufSize = (int) cs.iIOBufSize; pData->iFlushInterval = cs.iFlushInterval; pData->bUseAsyncWriter = cs.bUseAsyncWriter; + pData->bVeryRobustZip = 0; /* cannot be specified via legacy conf */ + setupInstStatsCtrs(pData); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -1088,7 +1321,6 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a cs.bUseAsyncWriter = USE_ASYNCWRITER_DFLT; free(pszFileDfltTplName); pszFileDfltTplName = NULL; - return RS_RET_OK; } @@ -1099,8 +1331,7 @@ CODESTARTdoHUP dynaFileFreeCacheEntries(pData); } else { if(pData->pStrm != NULL) { - strm.Destruct(&pData->pStrm); - pData->pStrm = NULL; + closeFile(pData); } } ENDdoHUP @@ -1110,6 +1341,7 @@ BEGINmodExit CODESTARTmodExit objRelease(errmsg, CORE_COMPONENT); objRelease(strm, CORE_COMPONENT); + objRelease(statsobj, CORE_COMPONENT); DESTROY_ATOMIC_HELPER_MUT(mutClock); ENDmodExit @@ -1132,6 +1364,7 @@ CODEmodInit_QueryRegCFSLineHdlr INITLegCnfVars CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(strm, CORE_COMPONENT)); + CHKiRet(objUse(statsobj, CORE_COMPONENT)); INIT_ATOMIC_HELPER_MUT(mutClock); diff --git a/tools/omfwd.c b/tools/omfwd.c index 2fd24bdf..129392d2 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -138,6 +138,7 @@ static struct cnfparamdescr actpdescr[] = { { "streamdriverauthmode", eCmdHdlrGetWord, 0 }, { "streamdriverpermittedpeers", eCmdHdlrGetWord, 0 }, { "resendlastmsgonreconnect", eCmdHdlrBinary, 0 }, + { "template", eCmdHdlrGetWord, 0 }, }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -760,6 +761,7 @@ setInstParamDefaults(instanceData *pData) BEGINnewActInst struct cnfparamvals *pvals; + uchar *tplToUse; int i; rsRetVal localRet; CODESTARTnewActInst @@ -881,7 +883,8 @@ CODESTARTnewActInst } CODE_STD_STRING_REQUESTnewActInst(1) - CHKiRet(OMSRsetEntry(*ppOMSR, 0, ustrdup(getDfltTpl()), OMSR_NO_RQD_TPL_OPTS)); + tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); + CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); CHKiRet(initTCP(pData)); CODE_STD_FINALIZERnewActInst diff --git a/tools/pidfile.c b/tools/pidfile.c index e9601232..8298b94e 100644 --- a/tools/pidfile.c +++ b/tools/pidfile.c @@ -55,7 +55,8 @@ int read_pid (char *pidfile) if (!(f=fopen(pidfile,"r"))) return 0; - fscanf(f,"%d", &pid); + if(fscanf(f,"%d", &pid) != 1) + pid = 0; fclose(f); return pid; } @@ -113,7 +114,8 @@ int write_pid (char *pidfile) #if HAVE_FLOCK if (flock(fd, LOCK_EX|LOCK_NB) == -1) { - fscanf(f, "%d", &pid); + if(fscanf(f, "%d", &pid) != 1) + pid = 0; fclose(f); printf("Can't lock, lock is held by pid %d.\n", pid); return 0; diff --git a/tools/pmrfc3164.c b/tools/pmrfc3164.c index bcded428..5dfa74f0 100644 --- a/tools/pmrfc3164.c +++ b/tools/pmrfc3164.c @@ -138,7 +138,7 @@ CODESTARTparse */ if(lenMsg > 0 && pMsg->msgFlags & PARSE_HOSTNAME) { i = 0; - while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' + while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '_' || p2parse[i] == '-') && i < (CONF_HOSTNAME_MAXSIZE - 1)) { bufParseHOSTNAME[i] = p2parse[i]; ++i; diff --git a/tools/recover_qi.pl b/tools/recover_qi.pl index 4e2cf9d5..4e2cf9d5 100644..100755 --- a/tools/recover_qi.pl +++ b/tools/recover_qi.pl diff --git a/tools/rscryutil.c b/tools/rscryutil.c new file mode 100644 index 00000000..2591b2cc --- /dev/null +++ b/tools/rscryutil.c @@ -0,0 +1,512 @@ +/* This is a tool for processing rsyslog encrypted log files. + * + * Copyright 2013 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either exprs or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <unistd.h> +#include <stdio.h> +#include <getopt.h> +#include <fcntl.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <gcrypt.h> + +#include "rsyslog.h" +#include "libgcry.h" + + +static enum { MD_DECRYPT, MD_WRITE_KEYFILE +} mode = MD_DECRYPT; +static int verbose = 0; +static gcry_cipher_hd_t gcry_chd; +static size_t blkLength; + +static char *keyfile = NULL; +static char *keyprog = NULL; +static int randomKeyLen = -1; +static char *cry_key = NULL; +static unsigned cry_keylen = 0; +static int cry_algo = GCRY_CIPHER_AES128; +static int cry_mode = GCRY_CIPHER_MODE_CBC; +static int optionForce = 0; + +/* rectype/value must be EIF_MAX_*_LEN+1 long! + * returns 0 on success or something else on error/EOF + */ +static int +eiGetRecord(FILE *eifp, char *rectype, char *value) +{ + int r; + unsigned short i, j; + char buf[EIF_MAX_RECTYPE_LEN+EIF_MAX_VALUE_LEN+128]; + /* large enough for any valid record */ + + if(fgets(buf, sizeof(buf), eifp) == NULL) { + r = 1; goto done; + } + + for(i = 0 ; i < EIF_MAX_RECTYPE_LEN && buf[i] != ':' ; ++i) + if(buf[i] == '\0') { + r = 2; goto done; + } else + rectype[i] = buf[i]; + rectype[i] = '\0'; + j = 0; + for(++i ; i < EIF_MAX_VALUE_LEN && buf[i] != '\n' ; ++i, ++j) + if(buf[i] == '\0') { + r = 3; goto done; + } else + value[j] = buf[i]; + value[j] = '\0'; + r = 0; +done: return r; +} + +static int +eiCheckFiletype(FILE *eifp) +{ + char rectype[EIF_MAX_RECTYPE_LEN+1]; + char value[EIF_MAX_VALUE_LEN+1]; + int r; + + if((r = eiGetRecord(eifp, rectype, value)) != 0) goto done; + if(strcmp(rectype, "FILETYPE") || strcmp(value, RSGCRY_FILETYPE_NAME)) { + fprintf(stderr, "invalid filetype \"cookie\" in encryption " + "info file\n"); + fprintf(stderr, "\trectype: '%s', value: '%s'\n", rectype, value); + r = 1; goto done; + } + r = 0; +done: return r; +} + +static int +eiGetIV(FILE *eifp, char *iv, size_t leniv) +{ + char rectype[EIF_MAX_RECTYPE_LEN+1]; + char value[EIF_MAX_VALUE_LEN+1]; + size_t valueLen; + unsigned short i, j; + int r; + unsigned char nibble; + + if((r = eiGetRecord(eifp, rectype, value)) != 0) goto done; + if(strcmp(rectype, "IV")) { + fprintf(stderr, "no IV record found when expected, record type " + "seen is '%s'\n", rectype); + r = 1; goto done; + } + valueLen = strlen(value); + if(valueLen/2 != leniv) { + fprintf(stderr, "length of IV is %d, expected %d\n", + valueLen/2, leniv); + r = 1; goto done; + } + + for(i = j = 0 ; i < valueLen ; ++i) { + if(value[i] >= '0' && value[i] <= '9') + nibble = value[i] - '0'; + else if(value[i] >= 'a' && value[i] <= 'f') + nibble = value[i] - 'a' + 10; + else { + fprintf(stderr, "invalid IV '%s'\n", value); + r = 1; goto done; + } + if(i % 2 == 0) + iv[j] = nibble << 4; + else + iv[j++] |= nibble; + } + r = 0; +done: return r; +} + +static int +eiGetEND(FILE *eifp, off64_t *offs) +{ + char rectype[EIF_MAX_RECTYPE_LEN+1]; + char value[EIF_MAX_VALUE_LEN+1]; + int r; + + if((r = eiGetRecord(eifp, rectype, value)) != 0) goto done; + if(strcmp(rectype, "END")) { + fprintf(stderr, "no END record found when expected, record type " + "seen is '%s'\n", rectype); + r = 1; goto done; + } + *offs = atoll(value); + r = 0; +done: return r; +} + +static int +initCrypt(FILE *eifp) +{ + int r = 0; + gcry_error_t gcryError; + char iv[4096]; + + blkLength = gcry_cipher_get_algo_blklen(cry_algo); + if(blkLength > sizeof(iv)) { + fprintf(stderr, "internal error[%s:%d]: block length %d too large for " + "iv buffer\n", __FILE__, __LINE__, blkLength); + r = 1; goto done; + } + if((r = eiGetIV(eifp, iv, blkLength)) != 0) goto done; + + size_t keyLength = gcry_cipher_get_algo_keylen(cry_algo); + if(strlen(cry_key) != keyLength) { + fprintf(stderr, "invalid key length; key is %u characters, but " + "exactly %u characters are required\n", cry_keylen, + keyLength); + r = 1; goto done; + } + + gcryError = gcry_cipher_open(&gcry_chd, cry_algo, cry_mode, 0); + if (gcryError) { + printf("gcry_cipher_open failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + r = 1; goto done; + } + + gcryError = gcry_cipher_setkey(gcry_chd, cry_key, keyLength); + if (gcryError) { + printf("gcry_cipher_setkey failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + r = 1; goto done; + } + + gcryError = gcry_cipher_setiv(gcry_chd, iv, blkLength); + if (gcryError) { + printf("gcry_cipher_setiv failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + r = 1; goto done; + } +done: return r; +} + +static inline void +removePadding(char *buf, size_t *plen) +{ + unsigned len = (unsigned) *plen; + unsigned iSrc, iDst; + char *frstNUL; + + frstNUL = memchr(buf, 0x00, *plen); + if(frstNUL == NULL) + goto done; + iDst = iSrc = frstNUL - buf; + + while(iSrc < len) { + if(buf[iSrc] != 0x00) + buf[iDst++] = buf[iSrc]; + ++iSrc; + } + + *plen = iDst; +done: return; +} + +static void +decryptBlock(FILE *fpin, FILE *fpout, off64_t blkEnd, off64_t *pCurrOffs) +{ + gcry_error_t gcryError; + size_t nRead, nWritten; + size_t toRead; + size_t leftTillBlkEnd; + char buf[64*1024]; + + leftTillBlkEnd = blkEnd - *pCurrOffs; + while(1) { + toRead = sizeof(buf) <= leftTillBlkEnd ? sizeof(buf) : leftTillBlkEnd; + toRead = toRead - toRead % blkLength; + nRead = fread(buf, 1, toRead, fpin); + if(nRead == 0) + break; + leftTillBlkEnd -= nRead, *pCurrOffs += nRead; + gcryError = gcry_cipher_decrypt( + gcry_chd, // gcry_cipher_hd_t + buf, // void * + nRead, // size_t + NULL, // const void * + 0); // size_t + if (gcryError) { + fprintf(stderr, "gcry_cipher_decrypt failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + return; + } + removePadding(buf, &nRead); + nWritten = fwrite(buf, 1, nRead, fpout); + if(nWritten != nRead) { + perror("fpout"); + return; + } + } +} + + +static int +doDecrypt(FILE *logfp, FILE *eifp, FILE *outfp) +{ + off64_t blkEnd; + off64_t currOffs = 0; + int r; + + while(1) { + /* process block */ + if(initCrypt(eifp) != 0) + goto done; + if((r = eiGetEND(eifp, &blkEnd)) != 0) goto done; + decryptBlock(logfp, outfp, blkEnd, &currOffs); + gcry_cipher_close(gcry_chd); + } + r = 0; +done: return r; +} + +static void +decrypt(char *name) +{ + FILE *logfp = NULL, *eifp = NULL; + int r = 0; + char eifname[4096]; + + if(!strcmp(name, "-")) { + fprintf(stderr, "decrypt mode cannot work on stdin\n"); + goto err; + } else { + if((logfp = fopen(name, "r")) == NULL) { + perror(name); + goto err; + } + snprintf(eifname, sizeof(eifname), "%s%s", name, ENCINFO_SUFFIX); + eifname[sizeof(eifname)-1] = '\0'; + if((eifp = fopen(eifname, "r")) == NULL) { + perror(eifname); + goto err; + } + if(eiCheckFiletype(eifp) != 0) + goto err; + } + + doDecrypt(logfp, eifp, stdout); + + fclose(logfp); logfp = NULL; + fclose(eifp); eifp = NULL; + return; + +err: + fprintf(stderr, "error %d processing file %s\n", r, name); + if(logfp != NULL) + fclose(logfp); +} + +static void +write_keyfile(char *fn) +{ + int fd; + int r; + mode_t fmode; + + fmode = O_WRONLY|O_CREAT; + if(!optionForce) + fmode |= O_EXCL; + if((fd = open(fn, fmode, S_IRUSR)) == -1) { + fprintf(stderr, "error opening keyfile "); + perror(fn); + exit(1); + } + if((r = write(fd, cry_key, cry_keylen)) != (ssize_t)cry_keylen) { + fprintf(stderr, "error writing keyfile (ret=%d) ", r); + perror(fn); + exit(1); + } + close(fd); +} + +static void +getKeyFromFile(char *fn) +{ + int r; + r = gcryGetKeyFromFile(fn, &cry_key, &cry_keylen); + if(r != 0) { + fprintf(stderr, "Error %d reading key from file '%s'\n", r, fn); + exit(1); + } +} + +static void +getRandomKey(void) +{ + int fd; + cry_keylen = randomKeyLen; + cry_key = malloc(randomKeyLen); /* do NOT zero-out! */ + /* if we cannot obtain data from /dev/urandom, we use whatever + * is present at the current memory location as random data. Of + * course, this is very weak and we should consider a different + * option, especially when not running under Linux (for Linux, + * unavailability of /dev/urandom is just a theoretic thing, it + * will always work...). -- TODO -- rgerhards, 2013-03-06 + */ + if((fd = open("/dev/urandom", O_RDONLY)) > 0) { + if(read(fd, cry_key, randomKeyLen)) {}; /* keep compiler happy */ + close(fd); + } +} + + +static void +setKey() +{ + if(randomKeyLen != -1) + getRandomKey(); + else if(keyfile != NULL) + getKeyFromFile(keyfile); + else if(keyprog != NULL) + gcryGetKeyFromProg(keyprog, &cry_key, &cry_keylen); + if(cry_key == NULL) { + fprintf(stderr, "ERROR: key must be set via some method\n"); + exit(1); + } +} + +static struct option long_options[] = +{ + {"verbose", no_argument, NULL, 'v'}, + {"version", no_argument, NULL, 'V'}, + {"decrypt", no_argument, NULL, 'd'}, + {"force", no_argument, NULL, 'f'}, + {"write-keyfile", required_argument, NULL, 'W'}, + {"key", required_argument, NULL, 'K'}, + {"generate-random-key", required_argument, NULL, 'r'}, + {"keyfile", required_argument, NULL, 'k'}, + {"key-program", required_argument, NULL, 'p'}, + {"algo", required_argument, NULL, 'a'}, + {"mode", required_argument, NULL, 'm'}, + {NULL, 0, NULL, 0} +}; + +int +main(int argc, char *argv[]) +{ + int i; + int opt; + int temp; + char *newKeyFile = NULL; + + while(1) { + opt = getopt_long(argc, argv, "a:dfk:K:m:p:r:vVW:", long_options, NULL); + if(opt == -1) + break; + switch(opt) { + case 'd': + mode = MD_DECRYPT; + break; + case 'W': + mode = MD_WRITE_KEYFILE; + newKeyFile = optarg; + break; + case 'k': + keyfile = optarg; + break; + case 'p': + keyprog = optarg; + break; + case 'f': + optionForce = 1; + break; + case 'r': + randomKeyLen = atoi(optarg); + if(randomKeyLen > 64*1024) { + fprintf(stderr, "ERROR: keys larger than 64KiB are " + "not supported\n"); + exit(1); + } + break; + case 'K': + fprintf(stderr, "WARNING: specifying the actual key " + "via the command line is highly insecure\n" + "Do NOT use this for PRODUCTION use.\n"); + cry_key = optarg; + cry_keylen = strlen(cry_key); + break; + case 'a': + temp = rsgcryAlgoname2Algo(optarg); + if(temp == GCRY_CIPHER_NONE) { + fprintf(stderr, "ERROR: algorithm \"%s\" is not " + "kown/supported\n", optarg); + exit(1); + } + cry_algo = temp; + break; + case 'm': + temp = rsgcryModename2Mode(optarg); + if(temp == GCRY_CIPHER_MODE_NONE) { + fprintf(stderr, "ERROR: cipher mode \"%s\" is not " + "kown/supported\n", optarg); + exit(1); + } + cry_mode = temp; + break; + case 'v': + verbose = 1; + break; + case 'V': + fprintf(stderr, "rsgtutil " VERSION "\n"); + exit(0); + break; + case '?': + break; + default:fprintf(stderr, "getopt_long() returns unknown value %d\n", opt); + return 1; + } + } + + setKey(); + + if(mode == MD_WRITE_KEYFILE) { + if(optind != argc) { + fprintf(stderr, "ERROR: no file parameters permitted in " + "--write-keyfile mode\n"); + exit(1); + } + write_keyfile(newKeyFile); + } else { + if(optind == argc) + decrypt("-"); + else { + for(i = optind ; i < argc ; ++i) + decrypt(argv[i]); + } + } + + memset(cry_key, 0, cry_keylen); /* zero-out key store */ + cry_keylen = 0; + return 0; +} diff --git a/tools/rscryutil.rst b/tools/rscryutil.rst new file mode 100644 index 00000000..dfd447d2 --- /dev/null +++ b/tools/rscryutil.rst @@ -0,0 +1,199 @@ +========= +rscryutil +========= + +-------------------------- +Manage Encrypted Log Files +-------------------------- + +:Author: Rainer Gerhards <rgerhards@adiscon.com> +:Date: 2013-04-15 +:Manual section: 1 + +SYNOPSIS +======== + +:: + + rscryutil [OPTIONS] [FILE] ... + + +DESCRIPTION +=========== + +This tool performs various operations on encrypted log files. +Most importantly, it provides the ability to decrypt them. + + +OPTIONS +======= + +-d, --decrypt + Select decryption mode. This is the default mode. + +-W, --write-keyfile <file> + Utility function to write a key to a keyfile. The key can be obtained + via any method. + +-v, --verbose + Select verbose mode. + +-f, --force + Forces operations that otherwise would fail. + +-k, --keyfile <file> + Reads the key from <file>. File _must_ contain the key, only, no headers + or other meta information. Keyfiles can be generated via the + *--write-keyfile* option. + +-p, --key-program <path-to-program> + In this mode, the key is provided by a so-called "key program". This program + is executed and must return the key to (as well as some meta information) + via stdout. The core idea of key programs is that using this interface the + user can implement as complex (and secure) method to obtain keys as + desired, all without the need to make modifications to rsyslog. + +-K, --key <KEY> + TESTING AID, NOT FOR PRODUCTION USE. This uses the KEY specified + on the command line. This is the actual key, and as such this mode + is highly insecure. However, it can be useful for intial testing + steps. This option may be removed in the future. + +-a, --algo <algo> + Sets the encryption algorightm (cipher) to be used. See below + for supported algorithms. The default is "AES128". + +-m, --mode <mode> + Sets the ciphermode to be used. See below for supported modes. + The default is "CBC". + +-r, --generate-random-key <bytes> + Generates a random key of length <bytes>. This option is + meant to be used together with *--write-keyfile* (and it is hard + to envision any other valid use for it). + +OPERATION MODES +=============== + +The operation mode specifies what exactly the tool does with the provided +files. The default operation mode is "dump", but this may change in the future. +Thus, it is recommended to always set the operations mode explicitely. If +multiple operations mode are set on the command line, results are +unpredictable. + +decrypt +------- + +The provided log files are decrypted. Note that the *.encinfo* side files +must exist and be accessible in order for decryption to to work. + +write-keyfile +------------- + +In this mode no log files are processed; thus it is an error to specify +any on the command line. The specified keyfile is written. The key itself +is obtained via the usual key commands. If *--keyfile* is used, that +file is effectively copied. + +For security reasons, existing key files are _not_ overwritten. To permit +this, specify the *--force* option. When doing so, keep in mind that lost +keys cannot be recovered and data encrypted with them may also be considered +lost. + +Keyfiles are always created with 0400 permission, that is read access for only +the user. An exception is when an existing file is overwritten via the +*--force* option, in which case the former permissions still apply. + +EXIT CODES +========== + +The command returns an exit code of 0 if everything went fine, and some +other code in case of failures. + + +SUPPORTED ALGORITHMS +==================== + +We basically support what libgcrypt supports. This is: + + 3DES + CAST5 + BLOWFISH + AES128 + AES192 + AES256 + TWOFISH + TWOFISH128 + ARCFOUR + DES + SERPENT128 + SERPENT192 + SERPENT256 + RFC2268_40 + SEED + CAMELLIA128 + CAMELLIA192 + CAMELLIA256 + + +SUPPORTED CIPHER MODES +====================== + +We basically support what libgcrypt supports. This is: + + ECB + CFB + CBC + STREAM + OFB + CTR + AESWRAP + +EXAMPLES +======== + +**rscryutil logfile** + +Decrypts "logfile" and sends data to stdout. + + +**rscryutil --generate-random-key 16 --keyfile /some/secured/path/keyfile** + +Generates random key and stores it in the specified keyfile. + +LOG SIGNATURES +============== + +Encrypted log files can be used together with signing. To verify such a file, +it must be decrypted first, and the verification tool **rsgtutil(1)** must be +run on the decrypted file. + +SECURITY CONSIDERATIONS +======================= + +Specifying keys directly on the command line (*--key* option) is very +insecure and should +not be done, except for testing purposes with test keys. Even then it is +recommended to use keyfiles, which are also easy to handle during testing. +Keep in mind that command history is usally be kept by bash and can also +easily be monitored. + +Local keyfiles are also a security risk. At a minimum, they should be +used with very restrictive file permissions. For this reason, +the *rscryutil* tool creates them with read permissions for the user, +only, no matter what umask is set to. + +When selecting cipher algorithms and modes, care needs to be taken. The +defaults should be reasonable safe to use, but this tends to change over +time. Keep up with the most current crypto recommendations. + + +SEE ALSO +======== +**rsgtutil(1)**, **rsyslogd(8)** + +COPYRIGHT +========= + +This page is part of the *rsyslog* project, and is available under +LGPLv2. diff --git a/tools/rsgtutil.c b/tools/rsgtutil.c new file mode 100644 index 00000000..095b8066 --- /dev/null +++ b/tools/rsgtutil.c @@ -0,0 +1,431 @@ +/* This is a tool for dumpoing the content of GuardTime TLV + * files in a (somewhat) human-readable manner. + * + * Copyright 2013 Adiscon GmbH + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either exprs or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <unistd.h> +#include <gt_base.h> +#include <gt_http.h> +#include <getopt.h> + +#include "librsgt.h" + +typedef unsigned char uchar; + +static enum { MD_DUMP, MD_DETECT_FILE_TYPE, MD_SHOW_SIGBLK_PARAMS, + MD_VERIFY, MD_EXTEND +} mode = MD_DUMP; +static int verbose = 0; + +static void +dumpFile(char *name) +{ + FILE *fp; + uchar hdr[9]; + void *obj; + tlvrecord_t rec; + int r = -1; + + if(!strcmp(name, "-")) + fp = stdin; + else { + printf("Processing file %s:\n", name); + if((fp = fopen(name, "r")) == NULL) { + perror(name); + goto err; + } + } + if((r = rsgt_tlvrdHeader(fp, hdr)) != 0) goto err; + printf("File Header: '%s'\n", hdr); + while(1) { /* we will err out on EOF */ + if((r = rsgt_tlvrd(fp, &rec, &obj)) != 0) { + if(feof(fp)) + break; + else + goto err; + } + rsgt_tlvprint(stdout, rec.tlvtype, obj, verbose); + rsgt_objfree(rec.tlvtype, obj); + } + + if(fp != stdin) + fclose(fp); + return; +err: fprintf(stderr, "error %d processing file %s\n", r, name); +} + +static void +showSigblkParams(char *name) +{ + FILE *fp; + block_sig_t *bs; + uint8_t bHasRecHashes, bHasIntermedHashes; + uint64_t blkCnt = 0; + int r = -1; + + if(!strcmp(name, "-")) + fp = stdin; + else { + if((fp = fopen(name, "r")) == NULL) { + perror(name); + goto err; + } + } + if((r = rsgt_chkFileHdr(fp, "LOGSIG10")) != 0) goto err; + + while(1) { /* we will err out on EOF */ + if((r = rsgt_getBlockParams(fp, 0, &bs, &bHasRecHashes, + &bHasIntermedHashes)) != 0) + goto err; + ++blkCnt; + rsgt_printBLOCK_SIG(stdout, bs, verbose); + printf("\t***META INFORMATION:\n"); + printf("\tBlock Nbr in File...: %llu\n", blkCnt); + printf("\tHas Record Hashes...: %d\n", bHasRecHashes); + printf("\tHas Tree Hashes.....: %d\n", bHasIntermedHashes); + } + + if(fp != stdin) + fclose(fp); + return; +err: + if(r != RSGTE_EOF) + fprintf(stderr, "error %d processing file %s\n", r, name); +} + +static void +detectFileType(char *name) +{ + FILE *fp; + char *typeName; + char hdr[9]; + int r = -1; + + if(!strcmp(name, "-")) + fp = stdin; + else { + if((fp = fopen(name, "r")) == NULL) { + perror(name); + goto err; + } + } + if((r = rsgt_tlvrdHeader(fp, (uchar*)hdr)) != 0) goto err; + if(!strcmp(hdr, "LOGSIG10")) + typeName = "Log Signature File, Version 10"; + else if(!strcmp(hdr, "GTSTAT10")) + typeName = "rsyslog GuardTime Signature State File, Version 10"; + else + typeName = "unknown"; + + printf("%s: %s [%s]\n", name, hdr, typeName); + + if(fp != stdin) + fclose(fp); + return; +err: fprintf(stderr, "error %d processing file %s\n", r, name); +} + +static inline int +doVerifyRec(FILE *logfp, FILE *sigfp, FILE *nsigfp, + block_sig_t *bs, gtfile gf, gterrctx_t *ectx, uint8_t bInBlock) +{ + int r; + size_t lenRec; + char line[128*1024]; + + if(fgets(line, sizeof(line), logfp) == NULL) { + if(feof(logfp)) { + r = RSGTE_EOF; + } else { + perror("log file input"); + r = RSGTE_IO; + } + goto done; + } + lenRec = strlen(line); + if(line[lenRec-1] == '\n') { + line[lenRec-1] = '\0'; + --lenRec; + rsgt_errctxSetErrRec(ectx, line); + } + + /* we need to preserve the first line (record) of each block for + * error-reporting purposes (bInBlock==0 meanst start of block) + */ + if(bInBlock == 0) + rsgt_errctxFrstRecInBlk(ectx, line); + + r = rsgt_vrfy_nextRec(bs, gf, sigfp, nsigfp, (unsigned char*)line, lenRec, ectx); +done: + return r; +} + +/* We handle both verify and extend with the same function as they + * are very similiar. + * + * note: here we need to have the LOG file name, not signature! + */ +static void +verify(char *name) +{ + FILE *logfp = NULL, *sigfp = NULL, *nsigfp = NULL; + block_sig_t *bs = NULL; + gtfile gf; + uint8_t bHasRecHashes, bHasIntermedHashes; + uint8_t bInBlock; + int r = 0; + char sigfname[4096]; + char oldsigfname[4096]; + char nsigfname[4096]; + gterrctx_t ectx; + + if(!strcmp(name, "-")) { + fprintf(stderr, "%s mode cannot work on stdin\n", + mode == MD_VERIFY ? "verify" : "extend"); + goto err; + } else { + snprintf(sigfname, sizeof(sigfname), "%s.gtsig", name); + sigfname[sizeof(sigfname)-1] = '\0'; + if((logfp = fopen(name, "r")) == NULL) { + perror(name); + goto err; + } + if((sigfp = fopen(sigfname, "r")) == NULL) { + perror(sigfname); + goto err; + } + if(mode == MD_EXTEND) { + snprintf(nsigfname, sizeof(nsigfname), "%s.gtsig.new", name); + nsigfname[sizeof(nsigfname)-1] = '\0'; + if((nsigfp = fopen(nsigfname, "w")) == NULL) { + perror(nsigfname); + goto err; + } + snprintf(oldsigfname, sizeof(oldsigfname), + "%s.gtsig.old", name); + oldsigfname[sizeof(oldsigfname)-1] = '\0'; + } + } + + rsgtInit("rsyslog rsgtutil " VERSION); + rsgt_errctxInit(&ectx); + ectx.verbose = verbose; + ectx.fp = stderr; + ectx.filename = strdup(sigfname); + + if((r = rsgt_chkFileHdr(sigfp, "LOGSIG10")) != 0) goto done; + if(mode == MD_EXTEND) { + if(fwrite("LOGSIG10", 8, 1, nsigfp) != 1) { + perror(nsigfname); + r = RSGTE_IO; + goto done; + } + } + gf = rsgt_vrfyConstruct_gf(); + if(gf == NULL) { + fprintf(stderr, "error initializing signature file structure\n"); + goto done; + } + + bInBlock = 0; + ectx.blkNum = 0; + ectx.recNumInFile = 0; + + while(!feof(logfp)) { + if(bInBlock == 0) { + if(bs != NULL) + rsgt_objfree(0x0902, bs); + if((r = rsgt_getBlockParams(sigfp, 1, &bs, &bHasRecHashes, + &bHasIntermedHashes)) != 0) + goto done; + rsgt_vrfyBlkInit(gf, bs, bHasRecHashes, bHasIntermedHashes); + ectx.recNum = 0; + ++ectx.blkNum; + } + ++ectx.recNum, ++ectx.recNumInFile; + if((r = doVerifyRec(logfp, sigfp, nsigfp, bs, gf, &ectx, bInBlock)) != 0) + goto done; + if(ectx.recNum == bs->recCount) { + if((r = verifyBLOCK_SIG(bs, gf, sigfp, nsigfp, + (mode == MD_EXTEND) ? 1 : 0, &ectx)) != 0) + goto done; + bInBlock = 0; + } else bInBlock = 1; + } + +done: + if(r != RSGTE_EOF) + goto err; + + fclose(logfp); logfp = NULL; + fclose(sigfp); sigfp = NULL; + if(nsigfp != NULL) { + fclose(nsigfp); nsigfp = NULL; + } + + /* everything went fine, so we rename files if we updated them */ + if(mode == MD_EXTEND) { + if(unlink(oldsigfname) != 0) { + if(errno != ENOENT) { + perror("unlink oldsig"); + r = RSGTE_IO; + goto err; + } + } + if(link(sigfname, oldsigfname) != 0) { + perror("link oldsig"); + r = RSGTE_IO; + goto err; + } + if(unlink(sigfname) != 0) { + perror("unlink cursig"); + r = RSGTE_IO; + goto err; + } + if(link(nsigfname, sigfname) != 0) { + perror("link newsig"); + fprintf(stderr, "WARNING: current sig file has been " + "renamed to %s - you need to manually recover " + "it.\n", oldsigfname); + r = RSGTE_IO; + goto err; + } + if(unlink(nsigfname) != 0) { + perror("unlink newsig"); + fprintf(stderr, "WARNING: current sig file has been " + "renamed to %s - you need to manually recover " + "it.\n", oldsigfname); + r = RSGTE_IO; + goto err; + } + } + rsgtExit(); + rsgt_errctxExit(&ectx); + return; + +err: + fprintf(stderr, "error %d processing file %s\n", r, name); + if(logfp != NULL) + fclose(logfp); + if(sigfp != NULL) + fclose(sigfp); + if(nsigfp != NULL) { + fclose(nsigfp); + unlink(nsigfname); + } + rsgtExit(); + rsgt_errctxExit(&ectx); +} + +static void +processFile(char *name) +{ + switch(mode) { + case MD_DETECT_FILE_TYPE: + detectFileType(name); + break; + case MD_DUMP: + dumpFile(name); + break; + case MD_SHOW_SIGBLK_PARAMS: + showSigblkParams(name); + break; + case MD_VERIFY: + case MD_EXTEND: + verify(name); + break; + } +} + + +static struct option long_options[] = +{ + {"dump", no_argument, NULL, 'D'}, + {"verbose", no_argument, NULL, 'v'}, + {"version", no_argument, NULL, 'V'}, + {"detect-file-type", no_argument, NULL, 'T'}, + {"show-sigblock-params", no_argument, NULL, 'B'}, + {"verify", no_argument, NULL, 't'}, /* 't' as in "test signatures" */ + {"extend", no_argument, NULL, 'e'}, + {"publications-server", optional_argument, NULL, 'P'}, + {"show-verified", no_argument, NULL, 's'}, + {NULL, 0, NULL, 0} +}; + +int +main(int argc, char *argv[]) +{ + int i; + int opt; + + while(1) { + opt = getopt_long(argc, argv, "DvVTBtPs", long_options, NULL); + if(opt == -1) + break; + switch(opt) { + case 'v': + verbose = 1; + break; + case 's': + rsgt_read_showVerified = 1; + break; + case 'V': + fprintf(stderr, "rsgtutil " VERSION "\n"); + exit(0); + case 'D': + mode = MD_DUMP; + break; + case 'B': + mode = MD_SHOW_SIGBLK_PARAMS; + break; + case 'P': + rsgt_read_puburl = optarg; + break; + case 'T': + mode = MD_DETECT_FILE_TYPE; + break; + case 't': + mode = MD_VERIFY; + break; + case 'e': + mode = MD_EXTEND; + break; + case '?': + break; + default:fprintf(stderr, "getopt_long() returns unknown value %d\n", opt); + return 1; + } + } + + if(optind == argc) + processFile("-"); + else { + for(i = optind ; i < argc ; ++i) + processFile(argv[i]); + } + + return 0; +} diff --git a/tools/rsgtutil.rst b/tools/rsgtutil.rst new file mode 100644 index 00000000..37958450 --- /dev/null +++ b/tools/rsgtutil.rst @@ -0,0 +1,177 @@ +======== +rsgtutil +======== + +----------------------------------- +Manage (GuardTime) Signed Log Files +----------------------------------- + +:Author: Rainer Gerhards <rgerhards@adiscon.com> +:Date: 2013-03-25 +:Manual section: 1 + +SYNOPSIS +======== + +:: + + rsgtutil [OPTIONS] [FILE] ... + + +DESCRIPTION +=========== + +This tool performs various maintenance operations on signed log files. +It specifically supports the GuardTime signature provider. + +The *rsgtutil* tool is the primary tool to verify log file signatures, +dump signature file contents and carry out other maintenance operations. +The tool offers different operation modes, which are selected via +command line options. + +The processing of multiple files is permitted. Depending on operation +mode, either the signature file or the base log file must be specified. +Within a single call, only a single operations mode is permitted. To +use different modes on different files, multiple calles, one for each +mode, must be made. + +If no file is specified on the command line, stdin is used instead. Note +that not all operation modes support stdin. + +OPTIONS +======= + +-D, --dump + Select "dump" operations mode. + +-t, --verify + Select "verify" operations mode. + +-T, --detect-file-type + Select "detect-file-type" operations mode. + +-B, --show-sigblock-params + Select "show-sigblock-params" operations mode. + +-s, --show-verified + Prints out information about correctly verified blocks (by default, only + errors are printed). + +-v, --verbose + Select verbose mode. Most importantly, hashes and signatures are printed + in full length (can be **very** lengthy) rather than the usual abbreviation. + +-e, --extend + Select extend mode. This extends the RFC3161 signatures. Note that this + mode also implies a full verification. If there are verify errors, extending + will also fail. + +-P <URL>, --publications-server <URL> + Sets the publications server. If not set but required by the operation a + default server is used. The default server is not necessarily optimal + in regard to performance and reliability. + + +OPERATION MODES +=============== + +The operation mode specifies what exactly the tool does with the provided +files. The default operation mode is "dump", but this may change in the future. +Thus, it is recommended to always set the operations mode explicitely. If +multiple operations mode are set on the command line, results are +unpredictable. + +dump +---- + +The provided *signature* files are dumped. For each top-level record, the*u +type code is printed as well as q short description. If there is additional +information available, it will be printed in tab-indented lines below the +main record dump. The actual *log* files need not to be present. + +verify +------ + +This mode does not work with stdin. On the command line, the *log* file names +are specified. The corresponding *signature* files (ending on ".gtsig") must also +be preset at the same location as the log file. In verify mode, both the log +and signature file is read and the validity of the log file checked. If verification +errors are detected these are printed and processing of the file aborted. By default, +each file is verified individually, without taking cross-file hash chains into +account (so the order of files on the command line does not matter). + +Note that the actual amount of what can be verified depends on the parameters with +which the signature file was written. If record and tree hashes are present, they +will be verified and thus fine-granular error reporting is possible. If they are +not present, only the block signature itself is verified. + +By default, only errors are printed. To also print successful verifications, use the +**--show-verified** option. + + +extend +------ +This extends the RFC3161 signatures. This includes a full verification +of the file. If there are verification errors, extending will also fail. +Note that a signature can only be extended when the required hash has been +published. Currently, these hashes are created at the 15th of each month at +0:00hrs UTC. It takes another few days to get them finally published. As such, +it can be assumed that extending is only possible after this happend (which +means it may take slightly above a month). + +To prevent data corruption, a copy of the signature file is created during +extension. So there must be enough disk space available for both files, +otherwise the operation will fail. If the log file is named logfile, the +signature file is logfile.gtsig and the temporary work file is named +logfile.gtsig.new. When extending finished successfully, the original +signature file (logfile.gtsig in our example) is renamed with the .old +postfix (logfile.gtsig.old) and the temporary file written under the +original name. The .old file can be deleted. It is just kept as a +precaution to prevent signature loss. Note that any already existing +.old or .new files are overwritten by these operations. + + +detect-file-type +---------------- +This mode is used to detect the type of some well-know files used inside the +signature system. The detection is based on the file header. This mode is +primarily a debug aid. + + +show-sigblock-params +-------------------- +This mode is used to print signature block parameters. It is similar to *dump* +mode, but will ignore everything except signature blocks. Also, some additional +meta information is printed. This mode is primarily a debug aid. + +EXIT CODES +========== + +The command returns an exit code of 0 if everything went fine, and some +other code in case of failures. + + +EXAMPLES +======== + +**rsgtutil --verify logfile** + +This verifies the file "logfile" via its associated signature file +"logfile.gtsig". If errors are detected, these are reported to stderr. +Otherwise, rsgtutil terminates without messages. + +**rsgtutil --dump logfile.gtsig** + +This dumps the content of the signature file "logfile.gtsig". The +actual log file is not being processed and does not even need to be +present. + +SEE ALSO +======== +**rsyslogd(8)** + +COPYRIGHT +========= + +This page is part of the *rsyslog* project, and is available under +LGPLv2. diff --git a/tools/rsyslog.conf.5 b/tools/rsyslog.conf.5 index dcc9b7c7..07da6ffd 100644 --- a/tools/rsyslog.conf.5 +++ b/tools/rsyslog.conf.5 @@ -17,7 +17,7 @@ .\" along with this program; if not, write to the Free Software .\" Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. .\" -.TH RSYSLOG.CONF 5 "11 July 2008" "Version 3.18.0" "Linux System Administration" +.TH RSYSLOG.CONF 5 "22 October 2012" "Version 7.2.0" "Linux System Administration" .SH NAME rsyslog.conf \- rsyslogd(8) configuration file .SH DESCRIPTION @@ -218,7 +218,7 @@ beginning with a slash ('/'). .B Example: .RS -*.* /var/log/traditionalfile.log;RSYSLOG_TraditionalFormat # log to a file in the traditional format +*.* /var/log/traditionalfile.log;RSYSLOG_TraditionalFileFormat # log to a file in the traditional format .RE Note: if you would like to use high-precision timestamps in your log files, @@ -335,13 +335,6 @@ Rsyslog offers three different types "filter conditions": * expression-based filters .RE -.SS Blocks -Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block of lines is separated from -the previous block by a program or hostname specification. A block will only log messages -corresponding to the most recent program and hostname specifications given. Thus, a block which -selects "ppp" as the program, directly followed by a block that selects messages from the -hostname "dialhost", then the second block will only log messages from the ppp program on dialhost. - .SS Selectors .B Selectors are the traditional way of filtering syslog messages. They have been kept in rsyslog with their original syntax, because it is well-known, highly @@ -422,7 +415,7 @@ To escape: .sp 0 \\ = \\\\ --> '\\' is used to escape (as in C) .sp 0 -$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n" +$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n" Properties can be accessed by the property replacer (see there for details). @@ -494,7 +487,7 @@ A template that resembles traditional syslogd file output: .RS $template TraditionalFormat,"%timegenerated% %HOSTNAME% .sp 0 -%syslogtag%%msg:::drop-last-lf%\n" +%syslogtag%%msg:::drop-last-lf%\\n" .RE A template that tells you a little more about the message: @@ -502,7 +495,7 @@ A template that tells you a little more about the message: .RS $template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%, .sp 0 -%syslogtag%,%msg%\n" +%syslogtag%,%msg%\\n" .RE A template for RFC 3164 format: @@ -514,7 +507,7 @@ $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" A template for the format traditionally used for user messages: .sp .RS -$template usermsg," XXXX%syslogtag%%msg%\n\r" +$template usermsg," XXXX%syslogtag%%msg%\\n\\r" .RE And a template with the traditional wall-message format: diff --git a/tools/rsyslogd.8 b/tools/rsyslogd.8 index 36f29769..620006f2 100644 --- a/tools/rsyslogd.8 +++ b/tools/rsyslogd.8 @@ -10,6 +10,7 @@ rsyslogd \- reliable and extended syslogd .RB [ " \-6 " ] .RB [ " \-A " ] .RB [ " \-d " ] +.RB [ " \-D " ] .RB [ " \-f " .I config file ] @@ -120,10 +121,15 @@ If neither -4 nor -6 is given, listens to all configured addresses of the system. .TP .BI "\-c " "version" -This option has been obsolted and has no function any longer. It is still +This option has been obsoleted and has no function any longer. It is still accepted in order not to break existing scripts. However, future versions may not support it. .TP +.B "\-D" +Runs the Bison config parser in debug mode. This may help when hard to find +syntax errors are reported. Please note that the output generated is deeply +technical and orignally targeted towards developers. +.TP .B "\-d" Turns on debug mode. Using this the daemon will not proceed a .BR fork (2) diff --git a/tools/syslogd.c b/tools/syslogd.c index a89c7e57..a8a733d6 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -44,7 +44,6 @@ #include "rsyslog.h" #define DEFUPRI (LOG_USER|LOG_NOTICE) -#define TIMERINTVL 30 /* interval for checking flush, mark */ #include <unistd.h> #include <stdlib.h> @@ -52,7 +51,6 @@ #include <stddef.h> #include <ctype.h> #include <limits.h> -#define GNU_SOURCE #include <string.h> #include <stdarg.h> #include <time.h> @@ -93,6 +91,8 @@ #include <zlib.h> #endif +extern int yydebug; /* interface to flex */ + #include <netdb.h> #include "pidfile.h" @@ -119,13 +119,13 @@ #include "batch.h" #include "unicode-helper.h" #include "ruleset.h" -#include "rule.h" #include "net.h" #include "prop.h" #include "rsconf.h" #include "dnscache.h" #include "sd-daemon.h" #include "rainerscript.h" +#include "ratelimit.h" /* definitions for objects we access */ DEFobjCurrIf(obj) @@ -134,7 +134,6 @@ DEFobjCurrIf(datetime) /* TODO: make go away! */ DEFobjCurrIf(conf) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) -DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) DEFobjCurrIf(prop) DEFobjCurrIf(parser) @@ -195,7 +194,6 @@ static prop_t *pInternalInputName = NULL; /* there is only one global inputName static uchar *ConfFile = (uchar*) _PATH_LOGCONF; /* read-only after startup */ static char *PidFile = _PATH_LOGPID; /* read-only after startup */ -static pid_t myPid; /* our pid for use in self-generated messages, e.g. on startup */ /* mypid is read-only after the initial fork() */ static int bHadHUP = 0; /* did we have a HUP? */ @@ -205,13 +203,6 @@ static int bFinished = 0; /* used by termination signal handler, read-only excep */ int iConfigVerify = 0; /* is this just a config verify run? */ -/* Intervals at which we flush out "message repeated" messages, - * in seconds after previous message is logged. After each flush, - * we move to the next interval until we reach the largest. - * TODO: this shall go into action object! -- rgerhards, 2008-01-29 - */ -int repeatinterval[2] = { 30, 60 }; /* # of secs before flush */ - #define LIST_DELIMITER ':' /* delimiter between two hosts */ static pid_t ppid; /* This is a quick and dirty hack used for spliting main/startup thread */ @@ -222,9 +213,11 @@ struct queuefilenames_s { } *queuefilenames = NULL; +static ratelimit_t *dflt_ratelimiter = NULL; /* ratelimiter for submits without explicit one */ +static ratelimit_t *internalMsg_ratelimiter = NULL; /* ratelimiter for rsyslog-own messages */ int MarkInterval = 20 * 60; /* interval between marks in seconds - read-only after startup */ int send_to_all = 0; /* send message to all IPv4/IPv6 addresses */ -static int NoFork = 0; /* don't fork - don't run in daemon mode - read-only after startup */ +static int doFork = 1; /* fork - run in daemon mode - read-only after startup */ int bHaveMainQueue = 0;/* set to 1 if the main queue - in queueing mode - is available * If the main queue is either not yet ready or not running in * queueing mode (mode DIRECT!), then this is set to 0. @@ -413,7 +406,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla CHKiRet(prop.Destruct(&pProp)); CHKiRet(MsgSetRcvFromIPStr(pMsg, hnameIP, ustrlen(hnameIP), &pProp)); CHKiRet(prop.Destruct(&pProp)); - CHKiRet(submitMsg(pMsg)); + CHKiRet(submitMsg2(pMsg)); finalize_it: RETiRet; @@ -433,6 +426,12 @@ submitErrMsg(int iErr, uchar *msg) } +static inline rsRetVal +submitMsgWithDfltRatelimiter(msg_t *pMsg) +{ + return ratelimitAddMsg(dflt_ratelimiter, NULL, pMsg); +} + /* rgerhards 2004-11-09: the following is a function that can be used * to log a message orginating from the syslogd itself. */ @@ -473,7 +472,7 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) * permits us to process unmodified config files which otherwise contain a * supressor statement. */ - if(((Debug == DEBUG_FULL || NoFork) && ourConf->globals.bErrMsgToStderr) || iConfigVerify) { + if(((Debug == DEBUG_FULL || !doFork) && ourConf->globals.bErrMsgToStderr) || iConfigVerify) { if(LOG_PRI(pri) == LOG_ERR) fprintf(stderr, "rsyslogd: %s\n", msg); } @@ -484,50 +483,13 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) /* we have the queue, so we can simply provide the * message to the queue engine. */ - submitMsg(pMsg); + ratelimitAddMsg(internalMsg_ratelimiter, NULL, pMsg); + //submitMsgWithDfltRatelimiter(pMsg); } finalize_it: RETiRet; } -/* check message against ACL set - * rgerhards, 2009-11-16 - */ -#if 0 -static inline rsRetVal -chkMsgAgainstACL() { - /* if we reach this point, we had a good receive and can process the packet received */ - /* check if we have a different sender than before, if so, we need to query some new values */ - if(net.CmpHost(&frominet, frominetPrev, socklen) != 0) { - CHKiRet(net.cvthname(&frominet, fromHost, fromHostFQDN, fromHostIP)); - memcpy(frominetPrev, &frominet, socklen); /* update cache indicator */ - /* Here we check if a host is permitted to send us - * syslog messages. If it isn't, we do not further - * process the message but log a warning (if we are - * configured to do this). - * rgerhards, 2005-09-26 - */ - *pbIsPermitted = net.isAllowedSender((uchar*)"UDP", - (struct sockaddr *)&frominet, (char*)fromHostFQDN); - - if(!*pbIsPermitted) { - DBGPRINTF("%s is not an allowed sender\n", (char*)fromHostFQDN); - if(glbl.GetOption_DisallowWarning) { - time_t tt; - - datetime.GetTime(&tt); - if(tt > ttLastDiscard + 60) { - ttLastDiscard = tt; - errmsg.LogError(0, NO_ERRCODE, - "UDP message from disallowed sender %s discarded", - (char*)fromHost); - } - } - } - } -} -#endif - /* preprocess a batch of messages, that is ready them for actual processing. This is done * as a first stage and totally in parallel to any other worker active in the system. So @@ -536,9 +498,9 @@ chkMsgAgainstACL() { */ static inline rsRetVal preprocessBatch(batch_t *pBatch) { - uchar fromHost[NI_MAXHOST]; - uchar fromHostIP[NI_MAXHOST]; - uchar fromHostFQDN[NI_MAXHOST]; + prop_t *ip; + prop_t *fqdn; + prop_t *localName; prop_t *propFromHost = NULL; prop_t *propFromHostIP = NULL; int bSingleRuleset; @@ -550,31 +512,31 @@ preprocessBatch(batch_t *pBatch) { DEFiRet; bSingleRuleset = 1; - batchRuleset = (pBatch->nElem > 0) ? ((msg_t*) pBatch->pElem[0].pUsrp)->pRuleset : NULL; + batchRuleset = (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; for(i = 0 ; i < pBatch->nElem && !*(pBatch->pbShutdownImmediate) ; i++) { - pMsg = (msg_t*) pBatch->pElem[i].pUsrp; + pMsg = pBatch->pElem[i].pMsg; if((pMsg->msgFlags & NEEDS_ACLCHK_U) != 0) { DBGPRINTF("msgConsumer: UDP ACL must be checked for message (hostname-based)\n"); - if(net.cvthname(pMsg->rcvFrom.pfrominet, fromHost, fromHostFQDN, fromHostIP) != RS_RET_OK) + if(net.cvthname(pMsg->rcvFrom.pfrominet, &localName, &fqdn, &ip) != RS_RET_OK) continue; bIsPermitted = net.isAllowedSender2((uchar*)"UDP", - (struct sockaddr *)pMsg->rcvFrom.pfrominet, (char*)fromHostFQDN, 1); + (struct sockaddr *)pMsg->rcvFrom.pfrominet, (char*)propGetSzStr(fqdn), 1); if(!bIsPermitted) { DBGPRINTF("Message from '%s' discarded, not a permitted sender host\n", - fromHostFQDN); - pBatch->pElem[i].state = BATCH_STATE_DISC; + propGetSzStr(fqdn)); + pBatch->eltState[i] = BATCH_STATE_DISC; } else { /* save some of the info we obtained */ - MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); - CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); + MsgSetRcvFrom(pMsg, localName); + CHKiRet(MsgSetRcvFromIP(pMsg, ip)); pMsg->msgFlags &= ~NEEDS_ACLCHK_U; } } if((pMsg->msgFlags & NEEDS_PARSING) != 0) { if((localRet = parser.ParseMsg(pMsg)) != RS_RET_OK) { DBGPRINTF("Message discarded, parsing error %d\n", localRet); - pBatch->pElem[i].state = BATCH_STATE_DISC; + pBatch->eltState[i] = BATCH_STATE_DISC; } } if(pMsg->pRuleset != batchRuleset) @@ -609,7 +571,7 @@ msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch, int *pbShu //do not have this yet and so we emulate -- 2010-06-10 int i; for(i = 0 ; i < pBatch->nElem && !*pbShutdownImmediate ; i++) { - pBatch->pElem[i].state = BATCH_STATE_COMM; + pBatch->eltState[i] = BATCH_STATE_COMM; } RETiRet; } @@ -620,7 +582,7 @@ int i; * rgerhards, 2008-02-13 */ rsRetVal -submitMsg(msg_t *pMsg) +submitMsg2(msg_t *pMsg) { qqueue_t *pQueue; ruleset_t *pRuleset; @@ -633,26 +595,31 @@ submitMsg(msg_t *pMsg) /* if a plugin logs a message during shutdown, the queue may no longer exist */ if(pQueue == NULL) { - DBGPRINTF("submitMsg() could not submit message - " + DBGPRINTF("submitMsg2() could not submit message - " "queue does (no longer?) exist - ignored\n"); FINALIZE; } - qqueueEnqObj(pQueue, pMsg->flowCtlType, (void*) pMsg); + qqueueEnqMsg(pQueue, pMsg->flowCtlType, pMsg); finalize_it: RETiRet; } +rsRetVal +submitMsg(msg_t *pMsg) +{ + return submitMsgWithDfltRatelimiter(pMsg); +} + /* submit multiple messages at once, very similar to submitMsg, just * for multi_submit_t. All messages need to go into the SAME queue! * rgerhards, 2009-06-16 */ rsRetVal -multiSubmitMsg(multi_submit_t *pMultiSub) +multiSubmitMsg2(multi_submit_t *pMultiSub) { - int i; qqueue_t *pQueue; ruleset_t *pRuleset; DEFiRet; @@ -677,8 +644,23 @@ multiSubmitMsg(multi_submit_t *pMultiSub) finalize_it: RETiRet; } +rsRetVal +multiSubmitMsg(multi_submit_t *pMultiSub) /* backward compat. level */ +{ + return multiSubmitMsg2(pMultiSub); +} +/* flush multiSubmit, e.g. at end of read records */ +rsRetVal +multiSubmitFlush(multi_submit_t *pMultiSub) +{ + DEFiRet; + if(pMultiSub->nElem > 0) { + iRet = multiSubmitMsg2(pMultiSub); + } + RETiRet; +} static void @@ -697,43 +679,6 @@ reapchild() } -/* helper to doFlushRptdMsgs() to flush the individual action links via llExecFunc - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(flushRptdMsgsActions) -{ - action_t *pAction = (action_t*) pData; - assert(pAction != NULL); - - BEGINfunc - d_pthread_mutex_lock(&pAction->mutAction); - /* TODO: time() performance: the call below could be moved to - * the beginn of the llExec(). This makes it slightly less correct, but - * in an acceptable way. -- rgerhards, 2008-09-16 - */ - if (pAction->f_prevcount && datetime.GetTime(NULL) >= REPEATTIME(pAction)) { - DBGPRINTF("flush %s: repeated %d times, %d sec.\n", - module.GetStateName(pAction->pMod), pAction->f_prevcount, - repeatinterval[pAction->f_repeatcount]); - actionWriteToAction(pAction); - BACKOFF(pAction); - } - d_pthread_mutex_unlock(&pAction->mutAction); - - ENDfunc - return RS_RET_OK; /* we ignore errors, we can not do anything either way */ -} - - -/* This method flushes repeat messages. - */ -static void -doFlushRptdMsgs(void) -{ - ruleset.IterateAllActions(runConf, flushRptdMsgsActions, NULL); -} - - static void debug_switch() { time_t tTime; @@ -774,8 +719,11 @@ static void debug_switch() * a minimal delay, but it is much cleaner than the approach of doing everything * inside the signal handler. * rgerhards, 2005-10-26 - * Note: we do not call DBGPRINTF() as this may cause us to block in case something - * with the threading is wrong. + * Note: + * - we do not call DBGPRINTF() as this may cause us to block in case something + * with the threading is wrong. + * - we do not really care about the return state of write(), but we need this + * strange check we do to silence compiler warnings (thanks, Ubuntu!) */ static void doDie(int sig) { @@ -783,11 +731,13 @@ static void doDie(int sig) # define MSG2 "DoDie called 5 times - unconditional exit\n" static int iRetries = 0; /* debug aid */ dbgprintf(MSG1); - if(Debug == DEBUG_FULL) - write(1, MSG1, sizeof(MSG1) - 1); + if(Debug == DEBUG_FULL) { + if(write(1, MSG1, sizeof(MSG1) - 1)) {} + } if(iRetries++ == 4) { - if(Debug == DEBUG_FULL) - write(1, MSG2, sizeof(MSG2) - 1); + if(Debug == DEBUG_FULL) { + if(write(1, MSG2, sizeof(MSG2) - 1)) {} + } abort(); } bFinished = sig; @@ -843,7 +793,7 @@ die(int sig) (void) snprintf(buf, sizeof(buf) / sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"]" " exiting on signal %d.", - (int) myPid, sig); + (int) glblGetOurPid(), sig); errno = 0; logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)buf, 0); } @@ -1106,7 +1056,7 @@ finalize_it: * the time being (remember that we want to restructure config processing at large!). * rgerhards, 2009-10-27 */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName) +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams) { struct queuefilenames_s *qfn; uchar *qfname = NULL; @@ -1122,60 +1072,65 @@ rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName) /* name our main queue object (it's not fatal if it fails...) */ obj.SetName((obj_t*) (*ppQueue), pszQueueName); - /* ... set some properties ... */ -# define setQPROP(func, directive, data) \ - CHKiRet_Hdlr(func(*ppQueue, data)) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } -# define setQPROPstr(func, directive, data) \ - CHKiRet_Hdlr(func(*ppQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } + if(queueParams == NULL) { /* use legacy parameters? */ + /* ... set some properties ... */ + # define setQPROP(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data)) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } + # define setQPROPstr(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } - if(ourConf->globals.mainQ.pszMainMsgQFName != NULL) { - /* check if the queue file name is unique, else emit an error */ - for(qfn = queuefilenames ; qfn != NULL ; qfn = qfn->next) { - dbgprintf("check queue file name '%s' vs '%s'\n", qfn->name, ourConf->globals.mainQ.pszMainMsgQFName ); - if(!ustrcmp(qfn->name, ourConf->globals.mainQ.pszMainMsgQFName)) { - snprintf((char*)qfrenamebuf, sizeof(qfrenamebuf), "%d-%s-%s", - ++qfn_renamenum, ourConf->globals.mainQ.pszMainMsgQFName, - (pszQueueName == NULL) ? "NONAME" : (char*)pszQueueName); - qfname = ustrdup(qfrenamebuf); - errmsg.LogError(0, NO_ERRCODE, "Error: queue file name '%s' already in use " - " - using '%s' instead", ourConf->globals.mainQ.pszMainMsgQFName, qfname); - break; + if(ourConf->globals.mainQ.pszMainMsgQFName != NULL) { + /* check if the queue file name is unique, else emit an error */ + for(qfn = queuefilenames ; qfn != NULL ; qfn = qfn->next) { + dbgprintf("check queue file name '%s' vs '%s'\n", qfn->name, ourConf->globals.mainQ.pszMainMsgQFName ); + if(!ustrcmp(qfn->name, ourConf->globals.mainQ.pszMainMsgQFName)) { + snprintf((char*)qfrenamebuf, sizeof(qfrenamebuf), "%d-%s-%s", + ++qfn_renamenum, ourConf->globals.mainQ.pszMainMsgQFName, + (pszQueueName == NULL) ? "NONAME" : (char*)pszQueueName); + qfname = ustrdup(qfrenamebuf); + errmsg.LogError(0, NO_ERRCODE, "Error: queue file name '%s' already in use " + " - using '%s' instead", ourConf->globals.mainQ.pszMainMsgQFName, qfname); + break; + } } + if(qfname == NULL) + qfname = ustrdup(ourConf->globals.mainQ.pszMainMsgQFName); + qfn = malloc(sizeof(struct queuefilenames_s)); + qfn->name = qfname; + qfn->next = queuefilenames; + queuefilenames = qfn; } - if(qfname == NULL) - qfname = ustrdup(ourConf->globals.mainQ.pszMainMsgQFName); - qfn = malloc(sizeof(struct queuefilenames_s)); - qfn->name = qfname; - qfn->next = queuefilenames; - queuefilenames = qfn; - } - setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", ourConf->globals.mainQ.iMainMsgQueMaxFileSize); - setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", ourConf->globals.mainQ.iMainMsgQueMaxDiskSpace); - setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", ourConf->globals.mainQ.iMainMsgQueDeqBatchSize); - setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", qfname); - setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", ourConf->globals.mainQ.iMainMsgQPersistUpdCnt); - setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", ourConf->globals.mainQ.bMainMsgQSyncQeueFiles); - setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", ourConf->globals.mainQ.iMainMsgQtoQShutdown ); - setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", ourConf->globals.mainQ.iMainMsgQtoActShutdown); - setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", ourConf->globals.mainQ.iMainMsgQtoWrkShutdown); - setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", ourConf->globals.mainQ.iMainMsgQtoEnq); - setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", ourConf->globals.mainQ.iMainMsgQHighWtrMark); - setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", ourConf->globals.mainQ.iMainMsgQLowWtrMark); - setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", ourConf->globals.mainQ.iMainMsgQDiscardMark); - setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", ourConf->globals.mainQ.iMainMsgQDiscardSeverity); - setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", ourConf->globals.mainQ.iMainMsgQWrkMinMsgs); - setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", ourConf->globals.mainQ.bMainMsgQSaveOnShutdown); - setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", ourConf->globals.mainQ.iMainMsgQDeqSlowdown); - setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", ourConf->globals.mainQ.iMainMsgQueueDeqtWinFromHr); - setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", ourConf->globals.mainQ.iMainMsgQueueDeqtWinToHr); - -# undef setQPROP -# undef setQPROPstr + setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", ourConf->globals.mainQ.iMainMsgQueMaxFileSize); + setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", ourConf->globals.mainQ.iMainMsgQueMaxDiskSpace); + setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", ourConf->globals.mainQ.iMainMsgQueDeqBatchSize); + setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", qfname); + setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", ourConf->globals.mainQ.iMainMsgQPersistUpdCnt); + setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", ourConf->globals.mainQ.bMainMsgQSyncQeueFiles); + setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", ourConf->globals.mainQ.iMainMsgQtoQShutdown ); + setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", ourConf->globals.mainQ.iMainMsgQtoActShutdown); + setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", ourConf->globals.mainQ.iMainMsgQtoWrkShutdown); + setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", ourConf->globals.mainQ.iMainMsgQtoEnq); + setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", ourConf->globals.mainQ.iMainMsgQHighWtrMark); + setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", ourConf->globals.mainQ.iMainMsgQLowWtrMark); + setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", ourConf->globals.mainQ.iMainMsgQDiscardMark); + setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", ourConf->globals.mainQ.iMainMsgQDiscardSeverity); + setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", ourConf->globals.mainQ.iMainMsgQWrkMinMsgs); + setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", ourConf->globals.mainQ.bMainMsgQSaveOnShutdown); + setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", ourConf->globals.mainQ.iMainMsgQDeqSlowdown); + setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", ourConf->globals.mainQ.iMainMsgQueueDeqtWinFromHr); + setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", ourConf->globals.mainQ.iMainMsgQueueDeqtWinToHr); + + # undef setQPROP + # undef setQPROPstr + } else { /* use new style config! */ + qqueueSetDefaultsRulesetQueue(*ppQueue); + qqueueApplyCnfParam(*ppQueue, queueParams); + } /* ... and finally start the queue! */ CHKiRet_Hdlr(qqueueStart(*ppQueue)) { @@ -1212,7 +1167,7 @@ init(void) snprintf(bufStartUpMsg, sizeof(bufStartUpMsg)/sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] start", - (int) myPid); + (int) glblGetOurPid()); logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)bufStartUpMsg, 0); } @@ -1254,7 +1209,7 @@ static inline void processImInternal(void) msg_t *pMsg; while(iminternalRemoveMsg(&pMsg) == RS_RET_OK) { - submitMsg(pMsg); + submitMsgWithDfltRatelimiter(pMsg); } } @@ -1291,7 +1246,7 @@ doHUP(void) snprintf(buf, sizeof(buf) / sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed", - (int) myPid); + (int) glblGetOurPid()); errno = 0; logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)buf, 0); } @@ -1321,49 +1276,22 @@ mainloop(void) while(!bFinished){ /* this is now just a wait - please note that we do use a near-"eternal" - * timeout of 1 day if we do not have repeated message reduction turned on - * (which it is not by default). This enables us to help safe the environment + * timeout of 1 day. This enables us to help safe the environment * by not unnecessarily awaking rsyslog on a regular tick (just think * powertop, for example). In that case, we primarily wait for a signal, * but a once-a-day wakeup should be quite acceptable. -- rgerhards, 2008-06-09 */ - tvSelectTimeout.tv_sec = (runConf->globals.bReduceRepeatMsgs == 1) ? TIMERINTVL : 86400 /*1 day*/; - //tvSelectTimeout.tv_sec = TIMERINTVL; /* TODO: change this back to the above code when we have a better solution for apc */ + tvSelectTimeout.tv_sec = 86400 /*1 day*/; tvSelectTimeout.tv_usec = 0; select(1, NULL, NULL, NULL, &tvSelectTimeout); if(bFinished) - break; /* exit as quickly as possible - see long comment below */ - - /* If we received a HUP signal, we call doFlushRptdMsgs() a bit early. This - * doesn't matter, because doFlushRptdMsgs() checks timestamps. What may happen, - * however, is that the too-early call may lead to a bit too-late output - * of "last message repeated n times" messages. But that is quite acceptable. - * rgerhards, 2007-12-21 - * ... and just to explain, we flush here because that is exactly what the mainloop - * shall do - provide a periodic interval in which not-yet-flushed messages will - * be flushed. Be careful, there is a potential race condition: doFlushRptdMsgs() - * needs to aquire a lock on the action objects. If, however, long-running consumers - * cause the main queue worker threads to lock them for a long time, we may receive - * a starvation condition, resulting in the mainloop being held on lock for an extended - * period of time. That, in turn, could lead to unresponsiveness to termination - * requests. It is especially important that the bFinished flag is checked before - * doFlushRptdMsgs() is called (I know because I ran into that situation). I am - * not yet sure if the remaining probability window of a termination-related - * problem is large enough to justify changing the code - I would consider it - * extremely unlikely that the problem ever occurs in practice. Fixing it would - * require not only a lot of effort but would cost considerable performance. So - * for the time being, I think the remaining risk can be accepted. - * rgerhards, 2008-01-10 - */ - if(runConf->globals.bReduceRepeatMsgs == 1) - doFlushRptdMsgs(); + break; /* exit as quickly as possible */ if(bHadHUP) { doHUP(); bHadHUP = 0; continue; } - // TODO: remove execScheduled(); /* handle Apc calls (if any) */ } ENDfunc } @@ -1409,6 +1337,11 @@ static void printVersion(void) #else printf("\tRuntime Instrumentation (slow code):\tNo\n"); #endif +#ifdef USE_LIBUUID + printf("\tuuid support:\t\t\t\tYes\n"); +#else + printf("\tuuid support:\t\t\t\tNo\n"); +#endif printf("\nSee http://www.rsyslog.com for more information.\n"); } @@ -1437,8 +1370,6 @@ InitGlobalClasses(void) CHKiRet(objUse(module, CORE_COMPONENT)); pErrObj = "datetime"; CHKiRet(objUse(datetime, CORE_COMPONENT)); - pErrObj = "rule"; - CHKiRet(objUse(rule, CORE_COMPONENT)); pErrObj = "ruleset"; CHKiRet(objUse(ruleset, CORE_COMPONENT)); pErrObj = "conf"; @@ -1461,6 +1392,7 @@ InitGlobalClasses(void) CHKiRet(objUse(net, LM_NET_FILENAME)); dnscacheInit(); initRainerscript(); + ratelimitModInit(); finalize_it: if(iRet != RS_RET_OK) { @@ -1492,7 +1424,6 @@ GlobalClassExit(void) objRelease(prop, CORE_COMPONENT); objRelease(conf, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); - objRelease(rule, CORE_COMPONENT); parserClassExit(); /* this is hack, currently core_modules do not get this automatically called */ rsconfClassExit(); /* this is hack, currently core_modules do not get this automatically called */ objRelease(datetime, CORE_COMPONENT); @@ -1500,6 +1431,7 @@ GlobalClassExit(void) /* TODO: implement the rest of the deinit */ /* dummy "classes */ strExit(); + ratelimitModExit(); #if 0 CHKiRet(objGetObjInterface(&obj)); /* this provides the root pointer for all other queries */ @@ -1595,6 +1527,13 @@ queryLocalHostname(void) */ glbl.SetLocalHostName(LocalHostName); glbl.SetLocalDomain(LocalDomain); + + if ( strlen((char*)LocalDomain) ) { + CHKmalloc(LocalFQDNName = (uchar*)malloc(strlen((char*)LocalDomain)+strlen((char*)LocalHostName)+2));/* one for dot, one for NUL! */ + if ( sprintf((char*)LocalFQDNName,"%s.%s",(char*)LocalHostName,(char*)LocalDomain) ) + glbl.SetLocalFQDNName(LocalFQDNName); + } + glbl.GenerateLocalHostNameProperty(); /* must be redone after conf processing, FQDN setting may have changed */ finalize_it: RETiRet; @@ -1681,8 +1620,7 @@ doGlblProcessInit(void) thrdInit(); - if( !(Debug == DEBUG_FULL || NoFork) ) - { + if(doFork) { DBGPRINTF("Checking pidfile '%s'.\n", PidFile); if (!check_pid(PidFile)) { @@ -1694,16 +1632,23 @@ doGlblProcessInit(void) /* stop writing debug messages to stdout (if debugging is on) */ stddbg = -1; + dbgprintf("ready for forking\n"); if (fork()) { /* Parent process */ - sleep(300); - /* Not reached unless something major went wrong. 5 - * minutes should be a fair amount of time to wait. - * Please note that this procedure is important since - * the father must not exit before syslogd isn't - * initialized or the klogd won't be able to flush its - * logs. -Joey + dbgprintf("parent process going to sleep for 60 secs\n"); + sleep(60); + /* Not reached unless something major went wrong. 1 + * minute should be a fair amount of time to wait. + * The parent should not exit before rsyslogd is + * properly initilized (at least almost) or the init + * system may get a wrong impression of our readyness. + * Note that we exit before being completely initialized, + * but at this point it is very, very unlikely that something + * bad can happen. We do this here, because otherwise we would + * need to have much more code to handle priv drop (which we + * don't consider worth for the init system, especially as it + * is going away on the majority of distros). */ exit(1); /* "good" exit - after forking, not diasabling anything */ } @@ -1712,6 +1657,7 @@ doGlblProcessInit(void) close(0); /* we keep stdout and stderr open in case we have to emit something */ i = 3; + dbgprintf("in child, finalizing initialization\n"); /* if (sd_booted()) */ { const char *e; @@ -1745,7 +1691,8 @@ doGlblProcessInit(void) i = SD_LISTEN_FDS_START + sd_fds; } for ( ; i < num_fds; i++) - (void) close(i); + if(i != dbgGetDbglogFd()) + close(i); untty(); } else { @@ -1770,7 +1717,7 @@ doGlblProcessInit(void) fputs("Pidfile (and pid) already exist.\n", stderr); exit(1); /* exit during startup - questionable */ } - myPid = getpid(); /* save our pid for further testing (also used for messages) */ + glblSetOurPid(getpid()); memset(&sigAct, 0, sizeof (sigAct)); sigemptyset(&sigAct.sa_mask); @@ -1825,7 +1772,7 @@ int realMain(int argc, char **argv) * of other options, we do this during the inital option processing. * rgerhards, 2008-04-04 */ - while((ch = getopt(argc, argv, "46a:Ac:def:g:hi:l:m:M:nN:op:qQr::s:t:T:u:vwx")) != EOF) { + while((ch = getopt(argc, argv, "46a:Ac:dDef:g:hi:l:m:M:nN:op:qQr::s:t:T:u:vwx")) != EOF) { switch((char)ch) { case '4': case '6': @@ -1853,11 +1800,15 @@ int realMain(int argc, char **argv) CHKiRet(bufOptAdd(ch, optarg)); break; case 'c': /* compatibility mode */ - fprintf(stderr, "rsyslogd: error: option -c is no longer supported - ignored"); + fprintf(stderr, "rsyslogd: error: option -c is no longer supported - ignored\n"); break; case 'd': /* debug - must be handled now, so that debug is active during init! */ debugging_on = 1; Debug = 1; + yydebug = 1; + break; + case 'D': /* BISON debug */ + yydebug = 1; break; case 'e': /* log every message (no repeat message supression) */ bEOptionWasGiven = 1; @@ -1952,7 +1903,7 @@ int realMain(int argc, char **argv) fprintf(stderr, "rsyslogd: error -m is no longer supported - use immark instead"); break; case 'n': /* don't fork */ - NoFork = 1; + doFork = 0; break; case 'N': /* enable config verify mode */ iConfigVerify = atoi(arg); @@ -2034,6 +1985,12 @@ int realMain(int argc, char **argv) } CHKiRet(localRet); + CHKiRet(ratelimitNew(&dflt_ratelimiter, "rsyslogd", "dflt")); + /* TODO: add linux-type limiting capability */ + CHKiRet(ratelimitNew(&internalMsg_ratelimiter, "rsyslogd", "internal_messages")); + ratelimitSetLinuxLike(internalMsg_ratelimiter, 5, 500); + /* TODO: make internalMsg ratelimit settings configurable */ + if(bChDirRoot) { if(chdir("/") != 0) fprintf(stderr, "Can not do 'cd /' - still trying to run\n"); @@ -2050,17 +2007,16 @@ int realMain(int argc, char **argv) if(!iConfigVerify) CHKiRet(doGlblProcessInit()); + /* Send a signal to the parent so it can terminate. */ + if(glblGetOurPid() != ppid) + kill(ppid, SIGTERM); + CHKiRet(init()); if(Debug && debugging_on) { dbgprintf("Debugging enabled, SIGUSR1 to turn off debugging.\n"); } - /* Send a signal to the parent so it can terminate. */ - if(myPid != ppid) - kill(ppid, SIGTERM); - - /* END OF INTIALIZATION */ DBGPRINTF("initialization completed, transitioning to regular run mode\n"); @@ -2071,12 +2027,14 @@ int realMain(int argc, char **argv) * is still in its infancy (and not really done), we currently accept this issue. * rgerhards, 2009-06-29 */ - if(!(Debug == DEBUG_FULL || NoFork)) { + if(doFork) { close(1); close(2); ourConf->globals.bErrMsgToStderr = 0; } + sd_notify(0, "READY=1"); + mainloop(); /* do any de-init's that need to be done AFTER this comment */ |