diff options
175 files changed, 19835 insertions, 3014 deletions
@@ -1,4 +1,48 @@ --------------------------------------------------------------------------- +Version 8.1.0 [devel] 2013-11-15 +- rewritten core engine for higher performance and new features + In detail: + * completely rewritten rule execution engine + * completely changed output module interface + * remodelled output module interface + * enabled important output modules to support full concurrent + operation + The core engine has been considerably changed and must be considered + experimental at this stage. Note that it does not yet include all + features planned for v8, but is close to this goal. In theory, the + engine should perform much better, especially on complex configurations + and busy servers. Most importantly, actions instances can now be called + concurrently from worker threads and many important output modules + support multiple concurrent action instances natively. +- module omruleset is no longer enabled by default. + Note that it has been deprecated in v7 and been replaced by the "call" + statement. Also, it can still be build without problems, the option must + just explicitely be given. +--------------------------------------------------------------------------- +Version 7.5.7 [devel] 2013-11-?? +- bugfix: ommysql lost configfile/section parameters after first close + This means that when a connection was broken, it was probably + re-instantiated with different parameters than configured. +- bugfix: regression in template processing with subtrees in templates + Thanks to Pavel Levshin for the fix +- bugfix: regular worker threads are not properly (re)started if DA + mode is active. + This occurs only under rare conditions, but definitely is a bug that + needed to be addressed. It probably is present since version 4. + Note that this patch has not been applied to v7.4-stable, as it + is very unlikely to happen and the fix itself has some regression + potential (the fix looks very solid, but it addresses a core component). + Thanks to Pavel Levshin for the fix +- worker thread pool handling has been improved + Among others, permits pool to actually shrink (was quite hard with + previous implementation. This will also improve performance and/or + lower system overhead on busy systems. + Thanks to Pavel Levshin for the enhancement. +- now emit warning message if om with msg passing mode uses action queue + These can modify the message, and this causes races. +- bugfix: $SystemLogUseSysTimeStamp/$SystemLogUsePIDFromSystem did not work + Thanks to Tomas Heinrich for the patch. +--------------------------------------------------------------------------- Version 7.4.7 [v7.4-stable] 2013-11-?? - bugfix: imuxsock: UseSysTimeStamp config parameter did not work correctly Thanks to Tomas Heinrich for alerting us and provinding a solution @@ -9,6 +53,202 @@ Version 7.4.7 [v7.4-stable] 2013-11-?? - bugfix: call to ruleset with async queue did not use the queue closes: http://bugzilla.adiscon.com/show_bug.cgi?id=443 --------------------------------------------------------------------------- +Version 7.5.6 [devel] 2013-10-29 +- impstats: add capability to bind to a ruleset +- improved performance of RainerScript variable access + by refactoring the whole body of variable handling code. This also + solves some of the anomalies experienced in some versions of rsyslog. + All variable types are now handled in unified code, including + access via templates. +- RainerScript: make use of 64 bit for numbers where available + Thanks to Pavel Levshin for enhancement. +- slight performance optimization if GCC is used + We give branch prediction hints for the frequent RETiRet macro which is + used for error handling. Some slight performance gain is to be expected + from that. +- removed global variable support + The original idea was not well thought out and global variables, as + implemented, worked far different from what anybody would expect. As + such, we consider the current approach as an experiment that did not + work out and opt to removing it, clearing the way for a better future + solution. Note: global vars were introduced in 7.5.3 on Sept, 11th 2013. +- new module mmsequence, primarily used for action load balancing + Thanks to Pavel Levshin for contributing this module. +- bugfix: unset statement always worked on message var, even if local + var was given +- imudp: support for binding to ruleset added +- bugfix: segfault if variable was assigned to non-container subtree + Thanks to Pavel Levshin for the fix +- bugfix: imuxsock did not suport addtl sockets if syssock was disabled + Thanks to Pavel Levshin for the fix +- bugfix: running imupd on multiple threads lead to segfault if recvmmsg + is available +- bugfix: imudp when using recvmmsg could report wrong sender IP +- bugfix: segfault if re_extract() function was used and no match found +- bugfix: omelasticsearch did not compile on platforms without atomic + instructions +- bugfix: potential misadressing on startup if property-filter was used + This could happen if the property name was longer than 127 chars, a case + that would not happen in practice. +- bugfix: invalid property filter was not properly disabled in ruleset + Note: the cosmetic memory leak introduced with that patch in 7.4.5 is + now also fixed. +- imported bugfixes from 7.4.6 stable release +--------------------------------------------------------------------------- +Version 7.5.5 [devel] 2013-10-16 +- imfile: permit to monitor an unlimited number of files +- imptcp: add "defaultTZ" input parameter +- imudp: support for multiple receiver threads added +- imudp: add "dfltTZ" input config parameter +- bugfix: memory leak in mmnormalize +- bugfix: mmutf8fix did not properly handle invalid UTF-8 at END of message + if the very last character sequence was too long, this was not detected + Thanks to Risto Vaarandi for reporting this problem. +- mmanon: removed the check for specific "terminator characters" after + last octet. As it turned out, this didn't work in practice as there + was an enormous set of potential terminator chars -- so removing + them was the best thing to do. Note that this may change behaviour of + existing installations. Yet, we still consider this an important + bugfix, that should be applied to the stable branch. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=477 + Thanks to Muri Cicanor for initiating the discussion +- now requires libestr 0.1.7 as early versions had a nasty bug in + string comparisons +- bugfix: mmanon did not detect all IP addresses in rewrite mode + The problem occured if two IPs were close to each other and the first one + was shrunk. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=485 + Thanks to micah-at-riseup.net for reporting this bug +- bugfix: mmanon sometimes used invalid replacement char in simple mode + depending on configuration sequence, the replacement character was set + to 's' instead of the correct value. Most importantly, it was set to + 's' if simple mode was selected and no replacement char set. + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=484 + Thanks to micah-at-riseup.net for reporting this bug +- bugfix: memory leak in mmnormalize +- bugfix: array-based ==/!= comparisions lead to invalid results + This was a regression introduced in 7.3.5 bei the PRI optimizer +--------------------------------------------------------------------------- +Version 7.5.4 [devel] 2013-10-07 +- mmpstrucdata: new module to parse RFC5424 structured data into json + message properties +- change main/ruleset queue defaults to be more enterprise-like + new defaults are queue.size 100,000 max workers 2, worker + activation after 40,000 msgs are queued, batch size 256. These settings + are much more useful for enterprises and will not hurt low-end systems + that much. This is part of our re-focus on enterprise needs. +- omfwd: new action parameter "maxErrorMessages" added +- omfile: new module parameters to set action defaults added + * dirCreateMode + * fileCreateMode +- mmutf8fix: new module to fix invalid UTF-8 sequences +- imuxsock: handle unlimited number of additional listen sockets +- doc: improve usability by linking to relevant web ressources + The idea is to enable users to quickly find additional information, + samples, HOWTOs and the like on the main site. + At the same time, (very) slightly remove memory footprint when + few listeners are monitored. +- bugfix: omfwd parameter streamdrivermmode was not properly handled + it was always overwritten by whatever value was set via the + legacy directive $ActionSendStreamDriverMode +- imtcp: add streamdriver.name module parameter + permits overriding the system default stream driver (gtls, ptcp) +- bugfix: build system: libgcrypt.h needed even if libgrcypt was disabled + Thanks to Jonny Törnbom for reporting this problem +- imported bugfixes from 7.4.4 +--------------------------------------------------------------------------- +Version 7.5.3 [devel] 2013-09-11 +- imfile: support for escaping LF characters added + embedded LF in syslog messages cause a lot of trouble. imfile now has + the capability to escape them to "#012" (just like the regular control + character escape option). This requires new-style input statements to be + used. If legacy configuration statements are used, LF escaping is always + turned off to preserve compatibility. + NOTE: if input() statements were already used, there is a CHANGE OF + BEHAVIOUR: starting with this version, escaping is enabled by + default. So if you do not want it, you need to add + escapeLF="off" + to the input statement. Given the trouble LFs cause and the fact + that the majority of installations still use legacy config, we + considered this behaviour change acceptable and useful. + see also: http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html +- add support for global and local variables +- bugfix: queue file size was not correctly processed + this could lead to using one queue file per message for sizes >2GiB + Thanks to Tomas Heinrich for the patch. +- add main_queue() configuration object to configure main message queue +- bugfix: stream compression in imptcp caused timestamp to be corrupted +- imudp: add ability to specify SO_RCVBUF size (rcvbufSize parameter) +- imudp: use inputname for statistics, if configured +- impstats: add process resource usage counters [via getrusage()] +- impstats: add paramter "resetCounters" to report delta values + possible for most, but not all, counters. See doc for details. +- librelp 1.2.0 is now required +- make use of new librelp generic error reporting facility + This leads to more error messages being passed to the user and + thus simplified troubleshooting. +- bugfix: very small memory leak in imrelp + more or less cosmetic, a single memory block was not freed, but this + only happens immediately before termination (when the OS automatically + frees all memory). Still an annoyance e.g. in valgrind. +- fix compile problem in debug build +- imported fixes from 7.4.4 +--------------------------------------------------------------------------- +Version 7.5.2 [devel] 2013-07-04 +- librelp 1.1.4 is now required + We use API extensions for better error reporting and higher performance. +- omrelp: use transactional mode to make imrelp emit bulk sends +- omrelp: add "windowSize" parameter to set custom RELP window size +- bugfix: double-free in omelasticsearch + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=461 + a security advisory for this bug is available at: + http://www.lsexperts.de/advisories/lse-2013-07-03.txt + CVE: CVE-2013-4758 + PLEASE NOTE: This issue only existed if omelasticsearch was used + in a non-default configuration, where the "errorfile" parameter + was specified. Without that parameter set, the bug could not + be triggered. + Thanks to Markus Vervier and Marius Ionescu for providing a detailled + bug report. Special thanks to Markus for coordinating his security + advisory with us. +- doc: fixed various typos + closes: http://bugzilla.adiscon.com/show_bug.cgi?id=391 + Thanks to Georgi Georgiev for the patch. +--------------------------------------------------------------------------- +Version 7.5.1 [devel] 2013-06-26 +- librelp 1.1.3 is required - older versions can lead to a segfault +- add mmfields, which among others supports easy parsing of CEF messages +- omrelp: + * new parameter "compression.prioritystring" to control encryption + parameters used by GnuTLS +- imrelp: + * new parameter "compression.dhbits" to control the number of + bits being used for Diffie-Hellman key generation + * new parameter "compression.prioritystring" to control encryption + parameters used by GnuTLS + * support for impstats added + * support for setting permitted peers (client authentication) added + * bugfix: potential segfault at startup on invalid config parameters +- imjournal: imported patches from 7.4.1 +- omprog: add support for command line parameters +- added experimental TCP stream compression (imptcp only, currently) +- added BSD-specific syslog facilities + * "console" + * "bsd_security" - this is called "security" under BSD, but that name + was unfortunately already taken by some standard facility. So I + did the (hopefully) second-best thing and renamed it a little. +- imported fixes from 7.4.2 (especially build problems on FreeBSD) +- bugfix: imptcp did not properly initialize compression status variable + could lead to segfault if stream:always compression mode was selected +--------------------------------------------------------------------------- +Version 7.5.0 [devel] 2013-06-11 +- imrelp: implement "ruleset" module parameter +- imrelp/omrelp: add TLS & compression (zip) support +- omrelp: add "rebindInterval" parameter +- add -S command line option to specify IP address to use for RELP client + connections + Thanks to Axel Rau for the patch. +--------------------------------------------------------------------------- Version 7.4.6 [v7.4-stable] 2013-10-31 - bugfix: potential abort during HUP This could happen when one of imklog, imzmq3, imkmsg, impstats, diff --git a/Makefile.am b/Makefile.am index 567b8769..6eb7b5b8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -241,6 +241,30 @@ if ENABLE_MMANON SUBDIRS += plugins/mmanon endif +if ENABLE_MMUTF8FIX +SUBDIRS += plugins/mmutf8fix +endif + +if ENABLE_MMCOUNT +SUBDIRS += plugins/mmcount +endif + +if ENABLE_MMSEQUENCE +SUBDIRS += plugins/mmsequence +endif + +if ENABLE_MMFIELDS +SUBDIRS += plugins/mmfields +endif + +if ENABLE_MMPSTRUCDATA +SUBDIRS += plugins/mmpstrucdata +endif + +if ENABLE_MMRFC5424ADDHMAC +SUBDIRS += plugins/mmrfc5424addhmac +endif + if ENABLE_ORACLE SUBDIRS += plugins/omoracle endif @@ -290,5 +314,5 @@ DISTCHECK_CONFIGURE_FLAGS= --enable-gssapi_krb5 \ --with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir) # temporarily disable these checks for make distcheck 2012-09-06 rgerhards # --enable-extended-tests \ -# --enable-pgsql \ +# --enable-pgsql ACLOCAL_AMFLAGS = -I m4 @@ -13,18 +13,15 @@ * The different modes (and calling sequence) are: * * if set iExecEveryNthOccur > 1 || iSecsExecOnceInterval - * - doSubmitToActionQComplexBatch - * - helperSubmitToActionQComplexBatch - * - doActionCallAction + * - doSubmitToActionQComplex * handles mark message reduction, but in essence calls * - actionWriteToAction * - qqueueEnqObj * (now queue engine processing) - * if(pThis->bWriteAllMarkMsgs == RSFALSE) - this is the DEFAULT - * - doSubmitToActionQNotAllMarkBatch - * - doSubmitToActionQBatch (and from here like in the else case below!) + * if(pThis->bWriteAllMarkMsgs == RSFALSE) + * - doSubmitToActionQNotAllMark + * - doSubmitToActionQ (and from here like in the else case below!) * else - * - doSubmitToActionQBatch * - doSubmitToActionQ * - qqueueEnqObj * (now queue engine processing) @@ -36,9 +33,6 @@ * * After dequeue, processing is as follows: * - processBatchMain - * - processAction - * - submitBatch - * - tryDoAction * - ... * * MORE ON PROCESSING, QUEUES and FILTERING @@ -69,7 +63,7 @@ * beast. * rgerhards, 2011-06-15 * - * Copyright 2007-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -114,18 +108,18 @@ #include "unicode-helper.h" #include "atomic.h" #include "ruleset.h" +#include "parserif.h" #include "statsobj.h" #define NO_TIME_PROVIDED 0 /* indicate we do not provide any cached time */ /* forward definitions */ -static rsRetVal processBatchMain(action_t *pAction, batch_t *pBatch, int*); -static rsRetVal doSubmitToActionQComplexBatch(action_t *pAction, batch_t *pBatch); -static rsRetVal doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch); -static rsRetVal doSubmitToActionQBatch(action_t *pAction, batch_t *pBatch); +static rsRetVal processBatchMain(void *pVoid, batch_t *pBatch, wti_t *pWti); +static rsRetVal doSubmitToActionQ(action_t *pAction, wti_t *pWti, msg_t*); +static rsRetVal doSubmitToActionQComplex(action_t *pAction, wti_t *pWti, msg_t*); +static rsRetVal doSubmitToActionQNotAllMark(action_t *pAction, wti_t *pWti, msg_t*); /* object static data (once for all instances) */ -/* TODO: make this an object! DEFobjStaticHelpers -- rgerhards, 2008-03-05 */ DEFobjCurrIf(obj) DEFobjCurrIf(datetime) DEFobjCurrIf(module) @@ -175,10 +169,9 @@ configSettings_t cs_save; /* our saved (scope!) config settings */ /* the counter below counts actions created. It is used to obtain unique IDs for the action. They * should not be relied on for any long-term activity (e.g. disk queue names!), but they are nice * to have during one instance of an rsyslogd run. For example, I use them to name actions when there - * is no better name available. Note that I do NOT recover previous numbers on HUP - we simply keep - * counting. -- rgerhards, 2008-01-29 + * is no better name available. */ -static int iActionNbr = 0; +int iActionNbr = 0; /* tables for interfacing with the v6 config system */ static struct cnfparamdescr cnfparamdescr[] = { @@ -308,7 +301,6 @@ rsRetVal actionDestruct(action_t *pThis) pThis->pMod->freeInstance(pThis->pModData); pthread_mutex_destroy(&pThis->mutAction); - pthread_mutex_destroy(&pThis->mutActExec); d_free(pThis->pszName); d_free(pThis->ppTpl); @@ -334,14 +326,15 @@ rsRetVal actionConstruct(action_t **ppThis) pThis->iResumeInterval = 30; pThis->iResumeRetryCount = 0; pThis->pszName = NULL; - pThis->bWriteAllMarkMsgs = RSFALSE; + pThis->bWriteAllMarkMsgs = 1; pThis->iExecEveryNthOccur = 0; pThis->iExecEveryNthOccurTO = 0; pThis->iSecsExecOnceInterval = 0; pThis->bExecWhenPrevSusp = 0; pThis->bRepMsgHasMsg = 0; + pThis->bDisabled = 0; pThis->tLastOccur = datetime.GetTime(NULL); /* done once per action on startup only */ - pthread_mutex_init(&pThis->mutActExec, NULL); + pThis->iActionNbr = iActionNbr; pthread_mutex_init(&pThis->mutAction, NULL); INIT_ATOMIC_HELPER_MUT(pThis->mutCAS); @@ -357,7 +350,7 @@ finalize_it: /* action construction finalizer */ rsRetVal -actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) +actionConstructFinalize(action_t *pThis, struct nvlst *lst) { DEFiRet; uchar pszAName[64]; /* friendly name of our action */ @@ -382,11 +375,11 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) STATSCOUNTER_INIT(pThis->ctrProcessed, pThis->mutCtrProcessed); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("processed"), - ctrType_IntCtr, &pThis->ctrProcessed)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrProcessed)); STATSCOUNTER_INIT(pThis->ctrFail, pThis->mutCtrFail); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("failed"), - ctrType_IntCtr, &pThis->ctrFail)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrFail)); CHKiRet(statsobj.ConstructFinalize(pThis->statsobj)); @@ -412,13 +405,13 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) DBGPRINTF("info: firehose mode disabled for action because " "iExecEveryNthOccur=%d, iSecsExecOnceInterval=%d\n", pThis->iExecEveryNthOccur, pThis->iSecsExecOnceInterval); - pThis->submitToActQ = doSubmitToActionQComplexBatch; - } else if(pThis->bWriteAllMarkMsgs == RSFALSE) { - /* nearly full-speed submission mode, default case */ - pThis->submitToActQ = doSubmitToActionQNotAllMarkBatch; + pThis->submitToActQ = doSubmitToActionQComplex; + } else if(pThis->bWriteAllMarkMsgs) { + /* full firehose submission mode, default case*/ + pThis->submitToActQ = doSubmitToActionQ; } else { - /* full firehose submission mode */ - pThis->submitToActQ = doSubmitToActionQBatch; + /* nearly full-speed submission mode */ + pThis->submitToActQ = doSubmitToActionQNotAllMark; } /* create queue */ @@ -428,11 +421,11 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) * spec. -- rgerhards, 2008-01-30 */ CHKiRet(qqueueConstruct(&pThis->pQueue, cs.ActionQueType, 1, cs.iActionQueueSize, - (rsRetVal (*)(void*, batch_t*, int*))processBatchMain)); + processBatchMain)); obj.SetName((obj_t*) pThis->pQueue, pszAName); qqueueSetpAction(pThis->pQueue, pThis); - if(queueParams == NULL) { /* use legacy params? */ + if(lst == NULL) { /* use legacy params? */ /* ... set some properties ... */ # define setQPROP(func, directive, data) \ CHKiRet_Hdlr(func(pThis->pQueue, data)) { \ @@ -466,7 +459,7 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) } else { /* we have v6-style config params */ qqueueSetDefaultsActionQueue(pThis->pQueue); - qqueueApplyCnfParam(pThis->pQueue, queueParams); + qqueueApplyCnfParam(pThis->pQueue, lst); } # undef setQPROP @@ -475,6 +468,12 @@ actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams) qqueueDbgPrint(pThis->pQueue); DBGPRINTF("Action %p: queue %p created\n", pThis, pThis->pQueue); + + if(pThis->eParamPassing == ACT_MSG_PASSING && pThis->pQueue->qType != QUEUETYPE_DIRECT) { + parser_warnmsg("module %s with message passing mode uses " + "non-direct queue. This most probably leads to undesired " + "results", (char*)modGetName(pThis->pMod)); + } /* and now reset the queue params (see comment in its function header!) */ actionResetQueueParams(); @@ -498,9 +497,9 @@ rsRetVal actionSetGlobalResumeInterval(int iNewVal) * returned string must not be modified. * rgerhards, 2009-05-07 */ -static uchar *getActStateName(action_t *pThis) +static uchar *getActStateName(action_t *pThis, wti_t *pWti) { - switch(pThis->eState) { + switch(getActionState(pWti, pThis)) { case ACT_STATE_RDY: return (uchar*) "rdy"; case ACT_STATE_ITX: @@ -509,8 +508,6 @@ static uchar *getActStateName(action_t *pThis) return (uchar*) "rtry"; case ACT_STATE_SUSP: return (uchar*) "susp"; - case ACT_STATE_DIED: - return (uchar*) "died"; case ACT_STATE_COMM: return (uchar*) "comm"; default: @@ -522,12 +519,12 @@ static uchar *getActStateName(action_t *pThis) /* returns a suitable return code based on action state * rgerhards, 2009-05-07 */ -static rsRetVal getReturnCode(action_t *pThis) +static rsRetVal getReturnCode(action_t *pThis, wti_t *pWti) { DEFiRet; ASSERT(pThis != NULL); - switch(pThis->eState) { + switch(getActionState(pWti, pThis)) { case ACT_STATE_RDY: iRet = RS_RET_OK; break; @@ -543,12 +540,11 @@ static rsRetVal getReturnCode(action_t *pThis) iRet = RS_RET_SUSPENDED; break; case ACT_STATE_SUSP: - case ACT_STATE_DIED: iRet = RS_RET_ACTION_FAILED; break; default: - DBGPRINTF("Invalid action engine state %d, program error\n", - (int) pThis->eState); + DBGPRINTF("Invalid action engine state %u, program error\n", + getActionState(pWti, pThis)); iRet = RS_RET_ERR; break; } @@ -560,29 +556,31 @@ static rsRetVal getReturnCode(action_t *pThis) /* set the action to a new state * rgerhards, 2007-08-02 */ -static inline void actionSetState(action_t *pThis, action_state_t newState) +static inline void +actionSetState(action_t *pThis, wti_t *pWti, uint8_t newState) { - pThis->eState = newState; - DBGPRINTF("Action %p transitioned to state: %s\n", pThis, getActStateName(pThis)); + setActionState(pWti, pThis, newState); + DBGPRINTF("Action %d transitioned to state: %s\n", + pThis->iActionNbr, getActStateName(pThis, pWti)); } /* Handles the transient commit state. So far, this is * mostly a dummy... * rgerhards, 2007-08-02 */ -static void actionCommitted(action_t *pThis) +static void actionCommitted(action_t *pThis, wti_t *pWti) { - actionSetState(pThis, ACT_STATE_RDY); + actionSetState(pThis, pWti, ACT_STATE_RDY); } /* set action to "rtry" state. * rgerhards, 2007-08-02 */ -static void actionRetry(action_t *pThis) +static void actionRetry(action_t *pThis, wti_t *pWti) { - actionSetState(pThis, ACT_STATE_RTRY); - pThis->iResumeOKinRow++; + actionSetState(pThis, pWti, ACT_STATE_RTRY); + incActionResumeInRow(pWti, pThis); } @@ -593,7 +591,7 @@ static void actionRetry(action_t *pThis) */ static void actionDisable(action_t *pThis) { - actionSetState(pThis, ACT_STATE_DIED); + pThis->bDisabled = 1; } @@ -605,7 +603,7 @@ static void actionDisable(action_t *pThis) * CPU time. TODO: maybe a config option for that? * rgerhards, 2007-08-02 */ -static inline void actionSuspend(action_t *pThis) +static inline void actionSuspend(action_t *pThis, wti_t *pWti) { time_t ttNow; @@ -613,8 +611,9 @@ static inline void actionSuspend(action_t *pThis) * since caching, and this would break logic (and it actually did so!) */ datetime.GetTime(&ttNow); - pThis->ttResumeRtry = ttNow + pThis->iResumeInterval * (pThis->iNbrResRtry / 10 + 1); - actionSetState(pThis, ACT_STATE_SUSP); + pThis->ttResumeRtry = ttNow + pThis->iResumeInterval * + (getActionNbrResRtry(pWti, pThis) / 10 + 1); + actionSetState(pThis, pWti, ACT_STATE_SUSP); DBGPRINTF("action suspended, earliest retry=%d\n", (int) pThis->ttResumeRtry); } @@ -627,15 +626,15 @@ static inline void actionSuspend(action_t *pThis) * entry point. This is invalid, but has harsh consequences: it will cause the rsyslog * engine to go into a tight loop. That obviously is not acceptable. As such, we track the * count of iterations that a tryResume returning RS_RET_OK is immediately followed by - * an unsuccessful call to doAction(). If that happens more than 1,000 times, we assume + * an unsuccessful call to doAction(). If that happens more than 10 times, we assume * the return acutally is a RS_RET_SUSPENDED. In order to go through the various - * resumption stages, we do this for every 1000 requests. This magic number 1000 may + * resumption stages, we do this for every 10 requests. This magic number 10 may * not be the most appropriate, but it should be thought of a "if nothing else helps" * kind of facility: in the first place, the module should return a proper indication * of its inability to recover. -- rgerhards, 2010-04-26. */ -static inline rsRetVal -actionDoRetry(action_t *pThis, int *pbShutdownImmediate) +static rsRetVal +actionDoRetry(action_t *pThis, wti_t *pWti) { int iRetries; int iSleepPeriod; @@ -645,31 +644,31 @@ actionDoRetry(action_t *pThis, int *pbShutdownImmediate) ASSERT(pThis != NULL); iRetries = 0; - while((*pbShutdownImmediate == 0) && pThis->eState == ACT_STATE_RTRY) { + while((*pWti->pbShutdownImmediate == 0) && getActionState(pWti, pThis) == ACT_STATE_RTRY) { DBGPRINTF("actionDoRetry: enter loop, iRetries=%d\n", iRetries); - iRet = pThis->pMod->tryResume(pThis->pModData); + iRet = pThis->pMod->tryResume(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData); DBGPRINTF("actionDoRetry: action->tryResume returned %d\n", iRet); - if((pThis->iResumeOKinRow > 9) && (pThis->iResumeOKinRow % 10 == 0)) { + if((getActionResumeInRow(pWti, pThis) > 9) && (getActionResumeInRow(pWti, pThis) % 10 == 0)) { bTreatOKasSusp = 1; - pThis->iResumeOKinRow = 0; + setActionResumeInRow(pWti, pThis, 0); } else { bTreatOKasSusp = 0; } if((iRet == RS_RET_OK) && (!bTreatOKasSusp)) { DBGPRINTF("actionDoRetry: had success RDY again (iRet=%d)\n", iRet); - actionSetState(pThis, ACT_STATE_RDY); + actionSetState(pThis, pWti, ACT_STATE_RDY); } else if(iRet == RS_RET_SUSPENDED || bTreatOKasSusp) { /* max retries reached? */ DBGPRINTF("actionDoRetry: check for max retries, iResumeRetryCount %d, iRetries %d\n", pThis->iResumeRetryCount, iRetries); if((pThis->iResumeRetryCount != -1 && iRetries >= pThis->iResumeRetryCount)) { - actionSuspend(pThis); + actionSuspend(pThis, pWti); } else { - ++pThis->iNbrResRtry; + incActionNbrResRtry(pWti, pThis); ++iRetries; iSleepPeriod = pThis->iResumeInterval; srSleep(iSleepPeriod, 0); - if(*pbShutdownImmediate) { + if(*pWti->pbShutdownImmediate) { ABORT_FINALIZE(RS_RET_FORCE_TERM); } } @@ -678,8 +677,8 @@ actionDoRetry(action_t *pThis, int *pbShutdownImmediate) } } - if(pThis->eState == ACT_STATE_RDY) { - pThis->iNbrResRtry = 0; + if(getActionState(pWti, pThis) == ACT_STATE_RDY) { + setActionNbrResRtry(pWti, pThis, 0); } finalize_it: @@ -687,17 +686,33 @@ finalize_it: } +static rsRetVal +actionCheckAndCreateWrkrInstance(action_t *pThis, wti_t *pWti) +{ + DEFiRet; + if(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData == NULL) { +dbgprintf("DDDD: wti %p create new worker instance for action %d\n", pWti, pThis->iActionNbr); + DBGPRINTF("we need to create a new action worker instance for " + "action %d\n", pThis->iActionNbr); + CHKiRet(pThis->pMod->mod.om.createWrkrInstance(&(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData), + pThis->pModData)); + pWti->actWrkrInfo[pThis->iActionNbr].pAction = pThis; + setActionState(pWti, pThis, ACT_STATE_RDY); /* action is enabled */ + } +finalize_it: + RETiRet; +} + /* try to resume an action -- rgerhards, 2007-08-02 * changed to new action state engine -- rgerhards, 2009-05-07 */ -static rsRetVal actionTryResume(action_t *pThis, int *pbShutdownImmediate) +static rsRetVal +actionTryResume(action_t *pThis, wti_t *pWti) { DEFiRet; time_t ttNow = NO_TIME_PROVIDED; - ASSERT(pThis != NULL); - - if(pThis->eState == ACT_STATE_SUSP) { + if(getActionState(pWti, pThis) == ACT_STATE_SUSP) { /* if we are suspended, we need to check if the timeout expired. * for this handling, we must always obtain a fresh timestamp. We used * to use the action timestamp, but in this case we will never reach a @@ -707,19 +722,19 @@ static rsRetVal actionTryResume(action_t *pThis, int *pbShutdownImmediate) */ datetime.GetTime(&ttNow); /* cache "now" */ if(ttNow >= pThis->ttResumeRtry) { - actionSetState(pThis, ACT_STATE_RTRY); /* back to retries */ + actionSetState(pThis, pWti, ACT_STATE_RTRY); /* back to retries */ } } - if(pThis->eState == ACT_STATE_RTRY) { + if(getActionState(pWti, pThis) == ACT_STATE_RTRY) { if(ttNow == NO_TIME_PROVIDED) /* use cached result if we have it */ datetime.GetTime(&ttNow); - CHKiRet(actionDoRetry(pThis, pbShutdownImmediate)); + CHKiRet(actionDoRetry(pThis, pWti)); } - if(Debug && (pThis->eState == ACT_STATE_RTRY ||pThis->eState == ACT_STATE_SUSP)) { + if(Debug && (getActionState(pWti, pThis) == ACT_STATE_RTRY ||getActionState(pWti, pThis) == ACT_STATE_SUSP)) { DBGPRINTF("actionTryResume: action %p state: %s, next retry (if applicable): %u [now %u]\n", - pThis, getActStateName(pThis), (unsigned) pThis->ttResumeRtry, (unsigned) ttNow); + pThis, getActStateName(pThis, pWti), (unsigned) pThis->ttResumeRtry, (unsigned) ttNow); } finalize_it: @@ -731,24 +746,27 @@ finalize_it: * depending on its current state. * rgerhards, 2009-05-07 */ -static inline rsRetVal actionPrepare(action_t *pThis, int *pbShutdownImmediate) +static inline rsRetVal +actionPrepare(action_t *pThis, wti_t *pWti) { DEFiRet; assert(pThis != NULL); - CHKiRet(actionTryResume(pThis, pbShutdownImmediate)); + CHKiRet(actionCheckAndCreateWrkrInstance(pThis, pWti)); + CHKiRet(actionTryResume(pThis, pWti)); /* if we are now ready, we initialize the transaction and advance * action state accordingly */ - if(pThis->eState == ACT_STATE_RDY) { - iRet = pThis->pMod->mod.om.beginTransaction(pThis->pModData); + if(getActionState(pWti, pThis) == ACT_STATE_RDY) { +dbgprintf("DDDDD: calling beginTransaction for action %d\n", pThis->iActionNbr); + iRet = pThis->pMod->mod.om.beginTransaction(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData); switch(iRet) { case RS_RET_OK: - actionSetState(pThis, ACT_STATE_ITX); + actionSetState(pThis, pWti, ACT_STATE_ITX); break; case RS_RET_SUSPENDED: - actionRetry(pThis); + actionRetry(pThis, pWti); break; case RS_RET_DISABLE_ACTION: actionDisable(pThis); @@ -762,10 +780,11 @@ finalize_it: } +#if 0 // TODO: remove? /* debug-print the contents of an action object * rgerhards, 2007-08-02 */ -rsRetVal actionDbgPrint(action_t *pThis) +static rsRetVal actionDbgPrint(action_t *pThis) { DEFiRet; char *sz; @@ -775,11 +794,12 @@ rsRetVal actionDbgPrint(action_t *pThis) dbgprintf("\n"); dbgprintf("\tInstance data: 0x%lx\n", (unsigned long) pThis->pModData); dbgprintf("\tResume Interval: %d\n", pThis->iResumeInterval); - if(pThis->eState == ACT_STATE_SUSP) { +#if 0 // do we need this ??? + if(getActionState(pWti, pThis) == ACT_STATE_SUSP) { dbgprintf("\tresume next retry: %u, number retries: %d", (unsigned) pThis->ttResumeRtry, pThis->iNbrResRtry); } - dbgprintf("\tState: %s\n", getActStateName(pThis)); +#endif dbgprintf("\tExec only when previous is suspended: %d\n", pThis->bExecWhenPrevSusp); if(pThis->submitToActQ == doSubmitToActionQComplexBatch) { sz = "slow, but feature-rich"; @@ -795,46 +815,52 @@ rsRetVal actionDbgPrint(action_t *pThis) RETiRet; } +#endif /* prepare the calling parameters for doAction() * rgerhards, 2009-05-07 */ static rsRetVal -prepareDoActionParams(action_t *pAction, batch_obj_t *pElem, struct syslogTime *ttNow) +prepareDoActionParams(action_t *pAction, wti_t *pWti, msg_t *pMsg, struct syslogTime *ttNow) { int i; - msg_t *pMsg; struct json_object *json; + actWrkrIParams_t *iparams; + actWrkrInfo_t *pWrkrInfo; DEFiRet; - ASSERT(pAction != NULL); - ASSERT(pElem != NULL); + pWrkrInfo = &(pWti->actWrkrInfo[pAction->iActionNbr]); + if(pAction->eParamPassing == ACT_STRING_PASSING) { + CHKiRet(wtiNewIParam(pWti, pAction, &iparams)); + } - pMsg = pElem->pMsg; /* here we must loop to process all requested strings */ for(i = 0 ; i < pAction->iNumTpls ; ++i) { +dbgprintf("DDDDD: generating template #%d\n", i); switch(pAction->eParamPassing) { case ACT_STRING_PASSING: - CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(pElem->staticActStrings[i]), - &pElem->staticLenStrings[i], ttNow)); - pElem->staticActParams[i] = pElem->staticActStrings[i]; + iparams->msgFlags = pMsg->msgFlags; + CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(iparams->staticActStrings[i]), + &iparams->staticLenStrings[i], ttNow)); + iparams->staticActParams[i] = iparams->staticActStrings[i]; break; case ACT_ARRAY_PASSING: - CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pElem->staticActParams[i]), ttNow)); + CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pWrkrInfo->staticActParams[i]), ttNow)); break; case ACT_MSG_PASSING: - pElem->staticActParams[i] = (void*) pMsg; + pWrkrInfo->staticActParams[i] = (void*) pMsg; break; case ACT_JSON_PASSING: CHKiRet(tplToJSON(pAction->ppTpl[i], pMsg, &json, ttNow)); - pElem->staticActParams[i] = (void*) json; + pWrkrInfo->staticActParams[i] = (void*) json; break; default:dbgprintf("software bug/error: unknown pAction->eParamPassing %d in prepareDoActionParams\n", (int) pAction->eParamPassing); assert(0); /* software bug if this happens! */ break; } +dbgprintf("DDDDD: template #%d is: '%s'\n", i, iparams->staticActStrings[i]); } finalize_it: @@ -842,96 +868,85 @@ finalize_it: } -/* free a batches ressources, but not string buffers (because they will - * most probably be reused). String buffers are only deleted upon final - * destruction of the batch. - * This function here must be called only when the batch is actually no - * longer used, also not for retrying actions or such. It invalidates - * buffers. - * rgerhards, 2010-12-17 - */ -static rsRetVal releaseBatch(action_t *pAction, batch_t *pBatch) +static void +releaseDoActionParams(action_t *pAction, wti_t *pWti) { int jArr; - int i, j; - batch_obj_t *pElem; + int j; + actWrkrInfo_t *pWrkrInfo; uchar ***ppMsgs; - DEFiRet; - - ASSERT(pAction != NULL); if(pAction->eParamPassing == ACT_STRING_PASSING || pAction->eParamPassing == ACT_MSG_PASSING) goto done; /* we need to do nothing with these types! */ - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - pElem = &(pBatch->pElem[i]); - if(batchIsValidElem(pBatch, i)) { - switch(pAction->eParamPassing) { - case ACT_ARRAY_PASSING: - ppMsgs = (uchar***) pElem->staticActParams; - for(j = 0 ; j < pAction->iNumTpls ; ++j) { - if(((uchar**)ppMsgs)[j] != NULL) { - jArr = 0; - while(ppMsgs[j][jArr] != NULL) { - d_free(ppMsgs[j][jArr]); - ppMsgs[j][jArr] = NULL; - ++jArr; - } - d_free(((uchar**)ppMsgs)[j]); - ((uchar**)ppMsgs)[j] = NULL; - } + pWrkrInfo = &(pWti->actWrkrInfo[pAction->iActionNbr]); + switch(pAction->eParamPassing) { + case ACT_ARRAY_PASSING: + ppMsgs = (uchar***) pWrkrInfo->staticActParams; + for(j = 0 ; j < pAction->iNumTpls ; ++j) { + if(((uchar**)ppMsgs)[j] != NULL) { + jArr = 0; + while(ppMsgs[j][jArr] != NULL) { + free(ppMsgs[j][jArr]); + ppMsgs[j][jArr] = NULL; + ++jArr; } - break; - case ACT_JSON_PASSING: - for(j = 0 ; j < pAction->iNumTpls ; ++j) { - json_object_put((struct json_object*) - pElem->staticActParams[j]); - pElem->staticActParams[j] = NULL; - } - break; - case ACT_STRING_PASSING: - case ACT_MSG_PASSING: - /* can never happen, just to keep compiler happy! */ - break; + free(((uchar**)ppMsgs)[j]); + ((uchar**)ppMsgs)[j] = NULL; } } + break; + case ACT_JSON_PASSING: + for(j = 0 ; j < pAction->iNumTpls ; ++j) { + json_object_put((struct json_object*) + pWrkrInfo->staticActParams[j]); + pWrkrInfo->staticActParams[j] = NULL; + } + break; + case ACT_STRING_PASSING: + case ACT_MSG_PASSING: + /* can never happen, just to keep compiler happy! */ + break; } -done: RETiRet; +done: return; } /* call the DoAction output plugin entry point * rgerhards, 2008-01-28 */ -rsRetVal -actionCallDoAction(action_t *pThis, msg_t *pMsg, void *actParams) +static rsRetVal +actionCallDoAction(action_t *pThis, int msgFlags, void *actParams, wti_t *pWti) { DEFiRet; ASSERT(pThis != NULL); - ISOBJ_TYPE_assert(pMsg, msg); - DBGPRINTF("entering actionCalldoAction(), state: %s\n", getActStateName(pThis)); + DBGPRINTF("entering actionCalldoAction(), state: %s, actionNbr %d\n", + getActStateName(pThis, pWti), pThis->iActionNbr); + + CHKiRet(actionCheckAndCreateWrkrInstance(pThis, pWti)); pThis->bHadAutoCommit = 0; - iRet = pThis->pMod->mod.om.doAction(actParams, pMsg->msgFlags, pThis->pModData); + iRet = pThis->pMod->mod.om.doAction(actParams, msgFlags, + pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData); switch(iRet) { case RS_RET_OK: - actionCommitted(pThis); - pThis->iResumeOKinRow = 0; /* we had a successful call! */ + actionCommitted(pThis, pWti); + setActionResumeInRow(pWti, pThis, 0); break; case RS_RET_DEFER_COMMIT: - pThis->iResumeOKinRow = 0; /* we had a successful call! */ + setActionResumeInRow(pWti, pThis, 0); /* we are done, action state remains the same */ break; case RS_RET_PREVIOUS_COMMITTED: /* action state remains the same, but we had a commit. */ pThis->bHadAutoCommit = 1; - pThis->iResumeOKinRow = 0; /* we had a successful call! */ + setActionResumeInRow(pWti, pThis, 0); break; case RS_RET_SUSPENDED: - actionRetry(pThis); + actionRetry(pThis, pWti); break; case RS_RET_DISABLE_ACTION: actionDisable(pThis); @@ -941,7 +956,7 @@ actionCallDoAction(action_t *pThis, msg_t *pMsg, void *actParams) */ FINALIZE; } - iRet = getReturnCode(pThis); + iRet = getReturnCode(pThis, pWti); finalize_it: RETiRet; @@ -952,57 +967,90 @@ finalize_it: * this readies the action and then calls doAction() * rgerhards, 2008-01-28 */ -static inline rsRetVal -actionProcessMessage(action_t *pThis, msg_t *pMsg, void *actParams, int *pbShutdownImmediate) +rsRetVal +actionProcessMessage(action_t *pThis, int msgFlags, void *actParams, wti_t *pWti) { DEFiRet; - ASSERT(pThis != NULL); - ISOBJ_TYPE_assert(pMsg, msg); - - CHKiRet(actionPrepare(pThis, pbShutdownImmediate)); + CHKiRet(actionPrepare(pThis, pWti)); if(pThis->pMod->mod.om.SetShutdownImmdtPtr != NULL) - pThis->pMod->mod.om.SetShutdownImmdtPtr(pThis->pModData, pbShutdownImmediate); - if(pThis->eState == ACT_STATE_ITX) - CHKiRet(actionCallDoAction(pThis, pMsg, actParams)); + pThis->pMod->mod.om.SetShutdownImmdtPtr(pThis->pModData, pWti->pbShutdownImmediate); + if(getActionState(pWti, pThis) == ACT_STATE_ITX) + CHKiRet(actionCallDoAction(pThis, msgFlags, actParams, pWti)); - iRet = getReturnCode(pThis); + iRet = getReturnCode(pThis, pWti); finalize_it: RETiRet; } -/* finish processing a batch. Most importantly, that means we commit if we - * need to do so. - * rgerhards, 2008-01-28 - */ +/* the following functions simulates a potential future new omo callback */ static rsRetVal -finishBatch(action_t *pThis, batch_t *pBatch) +doTransaction(action_t *pThis, wti_t *pWti) { + actWrkrInfo_t *wrkrInfo; + actWrkrIParams_t *iparamCurr; int i; DEFiRet; - ASSERT(pThis != NULL); + wrkrInfo = &(pWti->actWrkrInfo[pThis->iActionNbr]); + dbgprintf("DDDD: doTransaction: action %d, currIParams %d\n", pThis->iActionNbr, wrkrInfo->currIParam); + for(i = 0 ; i < wrkrInfo->currIParam ; ++i) { + iparamCurr = wrkrInfo->iparams + i; + iRet = actionProcessMessage(pThis, iparamCurr->msgFlags, + iparamCurr->staticActParams, + pWti); + dbgprintf("DDDD: doTransaction loop, iRet %d\n", iRet); + } + RETiRet; +} - if(pThis->eState == ACT_STATE_RDY) { - /* we just need to flag the batch as commited */ - FINALIZE; /* nothing to do */ + +static void +actionFreeParams(action_t *pThis, wti_t *pWti) +{ + actWrkrInfo_t *wrkrInfo; + actWrkrIParams_t *iparamCurr; + int i; + int j; + + wrkrInfo = &(pWti->actWrkrInfo[pThis->iActionNbr]); + dbgprintf("DDDD: actionFreeParams: action %d, currIParam %d\n", pThis->iActionNbr, wrkrInfo->currIParam); + for(i = 0 ; i < wrkrInfo->currIParam ; ++i) { + iparamCurr = wrkrInfo->iparams + i; + for(j = 0 ; j < CONF_OMOD_NUMSTRINGS_MAXSIZE ; ++j) { + /* TODO: we can save time by not freeing everything, + * but that's left for a later optimization. + */ + free(iparamCurr->staticActStrings[j]); + iparamCurr->staticActStrings[j] = NULL; + iparamCurr->staticLenStrings[j] = 0; + } } + wrkrInfo->currIParam = 0; /* reset to beginning */ + releaseDoActionParams(pThis, pWti); +} - CHKiRet(actionPrepare(pThis, pBatch->pbShutdownImmediate)); - if(pThis->eState == ACT_STATE_ITX) { - iRet = pThis->pMod->mod.om.endTransaction(pThis->pModData); + +/* Commit try committing (do not handle retry processing and such) */ +static rsRetVal +actionTryCommit(action_t *pThis, wti_t *pWti) +{ + //actWrkrInfo_t *wrkrInfo; + DEFiRet; + + doTransaction(pThis, pWti); + + CHKiRet(actionPrepare(pThis, pWti)); + if(getActionState(pWti, pThis) == ACT_STATE_ITX) { +dbgprintf("DDDDD: calling endTransaction for action %d\n", pThis->iActionNbr); + iRet = pThis->pMod->mod.om.endTransaction(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData); switch(iRet) { case RS_RET_OK: - actionCommitted(pThis); - /* flag messages as committed */ - for(i = 0 ; i < pBatch->nElem ; ++i) { - batchSetElemState(pBatch, i, BATCH_STATE_COMM); - pBatch->pElem[i].bPrevWasSuspended = 0; /* we had success! */ - } + actionCommitted(pThis, pWti); break; case RS_RET_SUSPENDED: - actionRetry(pThis); + actionRetry(pThis, pWti); break; case RS_RET_DISABLE_ACTION: actionDisable(pThis); @@ -1010,12 +1058,12 @@ finishBatch(action_t *pThis, batch_t *pBatch) case RS_RET_DEFER_COMMIT: DBGPRINTF("output plugin error: endTransaction() returns RS_RET_DEFER_COMMIT " "- ignored\n"); - actionCommitted(pThis); + actionCommitted(pThis, pWti); break; case RS_RET_PREVIOUS_COMMITTED: DBGPRINTF("output plugin error: endTransaction() returns RS_RET_PREVIOUS_COMMITTED " "- ignored\n"); - actionCommitted(pThis); + actionCommitted(pThis, pWti); break; default:/* permanent failure of this message - no sense in retrying. This is * not yet handled (but easy TODO) @@ -1023,304 +1071,138 @@ finishBatch(action_t *pThis, batch_t *pBatch) FINALIZE; } } - iRet = getReturnCode(pThis); + iRet = getReturnCode(pThis, pWti); finalize_it: + actionFreeParams(pThis, pWti); RETiRet; } - -/* try to submit a partial batch of elements. - * rgerhards, 2009-05-12 - */ -static inline rsRetVal -tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) -{ - int i; - int iElemProcessed; - int iCommittedUpTo; - msg_t *pMsg; - rsRetVal localRet; - DEFiRet; - - assert(pBatch != NULL); - assert(pnElem != NULL); - - i = pBatch->iDoneUpTo; /* all messages below that index are processed */ - iElemProcessed = 0; - iCommittedUpTo = i; - DBGPRINTF("tryDoAction %p, pnElem %d, nElem %d\n", pAction, *pnElem, pBatch->nElem); - while(iElemProcessed <= *pnElem && i < pBatch->nElem) { - if(*(pBatch->pbShutdownImmediate)) - ABORT_FINALIZE(RS_RET_FORCE_TERM); - /* NOTE: do NOT extend the filter below! Anything else must be done on the - * enq side of the queue (see file header comment)! -- rgerhards, 2011-06-15 - */ - if(batchIsValidElem(pBatch, i)) { - pMsg = pBatch->pElem[i].pMsg; - localRet = actionProcessMessage(pAction, pMsg, pBatch->pElem[i].staticActParams, - pBatch->pbShutdownImmediate); - DBGPRINTF("action %p call returned %d\n", pAction, localRet); - /* Note: we directly modify the batch object state, because we know that - * wo do not overwrite BATCH_STATE_DISC indicators! - */ - if(localRet == RS_RET_OK) { - /* mark messages as committed */ - while(iCommittedUpTo <= i) { - pBatch->pElem[iCommittedUpTo].bPrevWasSuspended = 0; /* we had success! */ - batchSetElemState(pBatch, iCommittedUpTo, BATCH_STATE_COMM); - ++iCommittedUpTo; - //pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; - } - } else if(localRet == RS_RET_PREVIOUS_COMMITTED) { - /* mark messages as committed */ - while(iCommittedUpTo < i) { - pBatch->pElem[iCommittedUpTo].bPrevWasSuspended = 0; /* we had success! */ - batchSetElemState(pBatch, iCommittedUpTo, BATCH_STATE_COMM); - ++iCommittedUpTo; - //pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; - } - pBatch->eltState[i] = BATCH_STATE_SUB; - } else if(localRet == RS_RET_DEFER_COMMIT) { - pBatch->eltState[i] = BATCH_STATE_SUB; - } else if(localRet == RS_RET_DISCARDMSG) { - pBatch->eltState[i] = BATCH_STATE_DISC; - } else { - dbgprintf("tryDoAction: unexpected error code %d[nElem %d, Commited UpTo %d], finalizing\n", - localRet, *pnElem, iCommittedUpTo); - iRet = localRet; - FINALIZE; - } - } - ++i; - ++iElemProcessed; - } - -finalize_it: - if(pBatch->iDoneUpTo != iCommittedUpTo) { - pBatch->iDoneUpTo = iCommittedUpTo; - } - RETiRet; -} - -/* submit a batch for actual action processing. - * The first nElem elements are processed. This function calls itself - * recursively if it needs to handle errors. - * Note: we don't need the number of the first message to be processed as a parameter, - * because this is kept track of inside the batch itself (iDoneUpTo). - * rgerhards, 2009-05-12 +/* Note: we currently need to return an iRet, as this is used in + * direct mode. TODO: However, it may be worth further investigating this, + * as it looks like there is no ultimate consumer of this code. + * rgerhards, 2013-11-06 */ static rsRetVal -submitBatch(action_t *pAction, batch_t *pBatch, int nElem) +actionCommit(action_t *pThis, wti_t *pWti) { - int i; - int bDone; - rsRetVal localRet; - int wasDoneTo; + sbool bDone; DEFiRet; - assert(pBatch != NULL); - - DBGPRINTF("submitBatch: enter, nElem %d\n", nElem); - wasDoneTo = pBatch->iDoneUpTo; + /* even more TODO: + This is the place where retry processing needs to go in. If the action + permanently fails, we should - as a new feature - add the capability to + write an error file. This is already done be omelasticsearch, and IMHO + pretty useful. + For the time being, I do NOT implement all of this (not even retry!) + as I want to get the rest of the engine to SISD (non-SIMD ;)) so that + I know any potential suprises and complications that arise out of this. + When this is done, I can come back here and complete this work. Obviously, + many features do not work in the mean time (but it is not planned to release + any of these partial implementations). + rgerhards, 2013-11-04 + */ bDone = 0; do { - localRet = tryDoAction(pAction, pBatch, &nElem); - if(localRet == RS_RET_FORCE_TERM) { + iRet = actionTryCommit(pThis, pWti); + DBGPRINTF("actionCommit, in retry loop, iRet %d\n", iRet); + if(iRet == RS_RET_FORCE_TERM) { ABORT_FINALIZE(RS_RET_FORCE_TERM); - } - if( localRet == RS_RET_OK - || localRet == RS_RET_PREVIOUS_COMMITTED - || localRet == RS_RET_DEFER_COMMIT) { - /* try commit transaction, once done, we can simply do so as if - * that return state was returned from tryDoAction(). - */ - localRet = finishBatch(pAction, pBatch); - } - - if( localRet == RS_RET_OK - || localRet == RS_RET_PREVIOUS_COMMITTED - || localRet == RS_RET_DEFER_COMMIT) { + } else if(iRet == RS_RET_OK || iRet == RS_RET_ACTION_FAILED) { bDone = 1; - } else if(localRet == RS_RET_SUSPENDED) { - DBGPRINTF("action ret RS_RET_SUSPENDED - retry full batch\n"); - /* do nothing, this will retry the full batch */ - } else if(localRet == RS_RET_ACTION_FAILED) { - /* in this case, everything not yet committed is BAD */ - for(i = pBatch->iDoneUpTo ; i < wasDoneTo + nElem ; ++i) { - if( pBatch->eltState[i] != BATCH_STATE_DISC - && pBatch->eltState[i] != BATCH_STATE_COMM ) { - pBatch->eltState[i] = BATCH_STATE_BAD; - pBatch->pElem[i].bPrevWasSuspended = 1; - STATSCOUNTER_INC(pAction->ctrFail, pAction->mutCtrFail); - } - } - bDone = 1; - } else { - if(nElem == 1) { - batchSetElemState(pBatch, pBatch->iDoneUpTo, BATCH_STATE_BAD); - bDone = 1; - } else { - /* retry with half as much. Depth is log_2 batchsize, so recursion is not too deep */ - DBGPRINTF("submitBatch recursing trying to find and exclude the culprit " - "for iRet %d\n", localRet); - submitBatch(pAction, pBatch, nElem / 2); - submitBatch(pAction, pBatch, nElem - (nElem / 2)); - bDone = 1; - } } - } while(!bDone && !*(pBatch->pbShutdownImmediate)); /* do .. while()! */ - - if(*(pBatch->pbShutdownImmediate)) - ABORT_FINALIZE(RS_RET_FORCE_TERM); - + } while(!bDone); finalize_it: RETiRet; } - -/* copy "active" array of batch, as we need to modify it. The caller - * must make sure the new array is freed and the orginal batch - * pointer is restored (thus the caller must save it). If active - * is currently NULL, this is properly handled. - * Note: the batches active pointer is modified, so it must be - * saved BEFORE calling this function! - * rgerhards, 2012-09-12 - */ -static rsRetVal -copyActive(batch_t *pBatch) -{ - sbool *active; - DEFiRet; - - CHKmalloc(active = malloc(batchNumMsgs(pBatch) * sizeof(sbool))); - if(pBatch->active == NULL) - memset(active, 1, batchNumMsgs(pBatch)); - else - memcpy(active, pBatch->active, batchNumMsgs(pBatch)); - pBatch->active = active; -finalize_it: - RETiRet; -} - -/* The following function prepares a batch for processing, that it is - * reinitializes batch states, generates strings and does everything else - * that needs to be done in order to make the batch ready for submission to - * the actual output module. Note that we look at the precomputed - * filter OK condition and process only those messages, that actually matched - * the filter. - * rgerhards, 2010-06-14 - */ -static inline rsRetVal -prepareBatch(action_t *pAction, batch_t *pBatch, sbool **activeSave, int *bMustRestoreActivePtr) +/* Commit all active transactions in *DIRECT mode* */ +void +actionCommitAllDirect(wti_t *pWti) { int i; - batch_obj_t *pElem; - struct syslogTime ttNow; - DEFiRet; - - /* indicate we have not yet read the date */ - ttNow.year = 0; + action_t *pAction; - pBatch->iDoneUpTo = 0; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - pElem = &(pBatch->pElem[i]); - if(batchIsValidElem(pBatch, i)) { - pBatch->eltState[i] = BATCH_STATE_RDY; - if(prepareDoActionParams(pAction, pElem, &ttNow) != RS_RET_OK) { - /* make sure we have our copy of "active" array */ - if(!*bMustRestoreActivePtr) { - *activeSave = pBatch->active; - copyActive(pBatch); - } - pBatch->active[i] = RSFALSE; - } - } + for(i = 0 ; i < iActionNbr ; ++i) { + dbgprintf("DDDD: actionCommitAll: action %d, state %u, nbr to commit %d\n", + i, getActionStateByNbr(pWti, i), pWti->actWrkrInfo->currIParam); + pAction = pWti->actWrkrInfo[i].pAction; + if(pAction != NULL && pAction->pQueue->qType == QUEUETYPE_DIRECT) + actionCommit(pAction, pWti); } - RETiRet; } - -/* receive a batch and process it. This includes retry handling. - * rgerhards, 2009-05-12 +/* process a single message. This is both called if we run from the + * cosumer side of an action queue as well as directly from the main + * queue thread if the action queue is set to "direct". */ -static inline rsRetVal -processAction(action_t *pAction, batch_t *pBatch) +static rsRetVal +processMsgMain(action_t *pAction, wti_t *pWti, msg_t *pMsg, struct syslogTime *ttNow) { DEFiRet; - assert(pBatch != NULL); - CHKiRet(submitBatch(pAction, pBatch, pBatch->nElem)); - iRet = finishBatch(pAction, pBatch); + if(pAction->bExecWhenPrevSusp && !pWti->execState.bPrevWasSuspended) { + DBGPRINTF("action %d: NOT executing, as previous action was " + "not suspended\n", pAction->iActionNbr); + FINALIZE; + } + +dbgprintf("DDDD: processMsgMain[act %d], %s\n", pAction->iActionNbr, pMsg->pszRawMsg); + iRet = prepareDoActionParams(pAction, pWti, pMsg, ttNow); + if(pAction->eParamPassing == ACT_STRING_PASSING) { + pWti->actWrkrInfo[pAction->iActionNbr].pAction = pAction; + dbgprintf("DDDD: action %d is string passing - executing in commit phase\n", pAction->iActionNbr); + iRet = getReturnCode(pAction, pWti); + FINALIZE; + } + iRet = actionProcessMessage(pAction, pMsg->msgFlags, + pWti->actWrkrInfo[pAction->iActionNbr].staticActParams, + pWti); + releaseDoActionParams(pAction, pWti); finalize_it: + if(pWti->execState.bDoAutoCommit) + iRet = actionCommit(pAction, pWti); + pWti->execState.bPrevWasSuspended = (iRet == RS_RET_SUSPENDED || iRet == RS_RET_ACTION_FAILED); +dbgprintf("DDDD: bPrevWasSuspended now %d, action state %d\n", (int)pWti->execState.bPrevWasSuspended, getActionState(pWti, pAction)); RETiRet; } - -#pragma GCC diagnostic ignored "-Wempty-body" -/* receive an array of to-process user pointers and submit them - * for processing. - * rgerhards, 2009-04-22 +/* This entry point is called by the ACTION queue (not main queue!) */ static rsRetVal -processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) +processBatchMain(void *pVoid, batch_t *pBatch, wti_t *pWti) { - int *pbShutdownImmdtSave; - sbool *activeSave; - int bMustRestoreActivePtr = 0; - rsRetVal localRet; + action_t *pAction = (action_t*) pVoid; + msg_t *pMsg; + int i; + struct syslogTime ttNow; DEFiRet; - assert(pBatch != NULL); - - if(pbShutdownImmediate != NULL) { - pbShutdownImmdtSave = pBatch->pbShutdownImmediate; - pBatch->pbShutdownImmediate = pbShutdownImmediate; - } - CHKiRet(prepareBatch(pAction, pBatch, &activeSave, &bMustRestoreActivePtr)); - - /* We now must guard the output module against execution by multiple threads. The - * plugin interface specifies that output modules must not be thread-safe (except - * if they notify us they are - functionality not yet implemented...). - * rgerhards, 2008-01-30 - */ - d_pthread_mutex_lock(&pAction->mutActExec); - pthread_cleanup_push(mutexCancelCleanup, &pAction->mutActExec); - - iRet = processAction(pAction, pBatch); - - pthread_cleanup_pop(1); /* unlock mutex */ - - /* even if processAction failed, we need to release the batch (else we - * have a memory leak). So we do this first, and then check if we need to - * return an error code. If so, the code from processAction has priority. - * rgerhards, 2010-12-17 - */ - localRet = releaseBatch(pAction, pBatch); + wtiResetExecState(pWti, pBatch); + /* indicate we have not yet read the date */ + ttNow.year = 0; - if(iRet == RS_RET_OK) - iRet = localRet; - - if(bMustRestoreActivePtr) { - free(pBatch->active); - pBatch->active = activeSave; + for(i = 0 ; i < batchNumMsgs(pBatch) && !*pWti->pbShutdownImmediate ; ++i) { + if(batchIsValidElem(pBatch, i)) { + pMsg = pBatch->pElem[i].pMsg; + iRet = processMsgMain(pAction, pWti, pMsg, &ttNow); + // TODO: we must refactor this! flag messages as committed + batchSetElemState(pBatch, i, BATCH_STATE_COMM); + } } -finalize_it: - if(pbShutdownImmediate != NULL) - pBatch->pbShutdownImmediate = pbShutdownImmdtSave; + if(!pWti->execState.bDoAutoCommit) + iRet = actionCommit(pAction, pWti); +dbgprintf("DDDD: processBatchMain - end\n"); RETiRet; } -#pragma GCC diagnostic warning "-Wempty-body" -/* call the HUP handler for a given action, if such a handler is defined. The - * action mutex is locked, because the HUP handler most probably needs to modify - * some internal state information. - * rgerhards, 2008-10-22 +/* call the HUP handler for a given action, if such a handler is defined. + * Note that the action must be able to service HUP requests concurrently + * to any current doAction() processing. */ -#pragma GCC diagnostic ignored "-Wempty-body" rsRetVal actionCallHUPHdlr(action_t *pAction) { @@ -1329,19 +1211,13 @@ actionCallHUPHdlr(action_t *pAction) ASSERT(pAction != NULL); DBGPRINTF("Action %p checks HUP hdlr: %p\n", pAction, pAction->pMod->doHUP); - if(pAction->pMod->doHUP == NULL) { - FINALIZE; /* no HUP handler, so we are done ;) */ + if(pAction->pMod->doHUP != NULL) { + CHKiRet(pAction->pMod->doHUP(pAction->pModData)); } - d_pthread_mutex_lock(&pAction->mutActExec); - pthread_cleanup_push(mutexCancelCleanup, &pAction->mutActExec); - CHKiRet(pAction->pMod->doHUP(pAction->pModData)); - pthread_cleanup_pop(1); /* unlock mutex */ - finalize_it: RETiRet; } -#pragma GCC diagnostic warning "-Wempty-body" /* set the action message queue mode @@ -1376,27 +1252,28 @@ static rsRetVal setActionQueType(void __attribute__((unused)) *pVal, uchar *pszT /* This submits the message to the action queue in case we do NOT need to handle repeat * message processing. That case permits us to gain lots of freedom during processing - * and thus speed. This is also utilized to submit messages in complex case once + * and thus speed. This is also utilized to submit messages in more complex cases once * the complex logic has been applied ;) * rgerhards, 2010-06-08 */ -static inline rsRetVal -doSubmitToActionQ(action_t *pAction, msg_t *pMsg) +static rsRetVal +doSubmitToActionQ(action_t *pAction, wti_t *pWti, msg_t *pMsg) { + struct syslogTime ttNow; // TODO: think if we can buffer this in pWti DEFiRet; - if(pAction->eState == ACT_STATE_DIED) { - DBGPRINTF("action %p died, do NOT execute\n", pAction); - FINALIZE; - } + DBGPRINTF("Called action, logging to %s\n", module.GetStateName(pAction->pMod)); STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); - if(pAction->pQueue->qType == QUEUETYPE_DIRECT) - iRet = qqueueEnqMsgDirect(pAction->pQueue, MsgAddRef(pMsg)); - else + if(pAction->pQueue->qType == QUEUETYPE_DIRECT) { + ttNow.year = 0; + iRet = processMsgMain(pAction, pWti, pMsg, &ttNow); + } else {/* in this case, we do single submits to the queue. + * TODO: optimize this, we may do at least a multi-submit! + */ iRet = qqueueEnqMsg(pAction->pQueue, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); + } -finalize_it: RETiRet; } @@ -1409,7 +1286,7 @@ finalize_it: * be filtered out before calling us (what is done currently!). */ rsRetVal -actionWriteToAction(action_t *pAction, msg_t *pMsg) +actionWriteToAction(action_t *pAction, msg_t *pMsg, wti_t *pWti) { DEFiRet; @@ -1464,44 +1341,46 @@ actionWriteToAction(action_t *pAction, msg_t *pMsg) /* When we reach this point, we have a valid, non-disabled action. * So let's enqueue our message for execution. -- rgerhards, 2007-07-24 */ - iRet = doSubmitToActionQ(pAction, pMsg); + iRet = doSubmitToActionQ(pAction, pWti, pMsg); finalize_it: RETiRet; } -/* helper to actonCallAction, mostly needed because of this damn - * pthread_cleanup_push() POSIX macro... +/* Call configured action, most complex case with all features supported (and thus slow). + * rgerhards, 2010-06-08 */ -static inline rsRetVal -doActionCallAction(action_t *pAction, batch_t *pBatch, int idxBtch) +#pragma GCC diagnostic ignored "-Wempty-body" +static rsRetVal +doSubmitToActionQComplex(action_t *pAction, wti_t *pWti, msg_t *pMsg) { - msg_t *pMsg; DEFiRet; - pMsg = pBatch->pElem[idxBtch].pMsg; + d_pthread_mutex_lock(&pAction->mutAction); + pthread_cleanup_push(mutexCancelCleanup, &pAction->mutAction); + DBGPRINTF("Called action %p (complex case), logging to %s\n", + pAction, module.GetStateName(pAction->pMod)); + pAction->tActNow = -1; /* we do not yet know our current time (clear prev. value) */ + // TODO: can we optimize the "now" handling again (was batch, I guess...)? /* don't output marks to recently written outputs */ - if(pAction->bWriteAllMarkMsgs == RSFALSE + if(pAction->bWriteAllMarkMsgs == 0 && (pMsg->msgFlags & MARK) && (getActNow(pAction) - pAction->f_time) < MarkInterval / 2) { ABORT_FINALIZE(RS_RET_OK); } /* call the output driver */ - iRet = actionWriteToAction(pAction, pMsg); + iRet = actionWriteToAction(pAction, pMsg, pWti); finalize_it: - /* we need to update the batch to handle failover processing correctly */ - if(iRet == RS_RET_OK) { - pBatch->pElem[idxBtch].bPrevWasSuspended = 0; - } else if(iRet == RS_RET_ACTION_FAILED) { - pBatch->pElem[idxBtch].bPrevWasSuspended = 1; - } + d_pthread_mutex_unlock(&pAction->mutAction); + pthread_cleanup_pop(0); /* remove mutex cleanup handler */ RETiRet; } +#pragma GCC diagnostic warning "-Wempty-body" /* helper to activateActions, it activates a specific action. @@ -1550,195 +1429,42 @@ activateActions(void) * rgerhards, 2010-06-08 */ static rsRetVal -doSubmitToActionQNotAllMarkBatch(action_t *pAction, batch_t *pBatch) +doSubmitToActionQNotAllMark(action_t *pAction, wti_t *pWti, msg_t *pMsg) { - time_t now = 0; + int doProcess = 1; time_t lastAct; - int i; - sbool *activeSave; DEFiRet; - activeSave = pBatch->active; - copyActive(pBatch); - - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if((pBatch->eltState[i] == BATCH_STATE_DISC) || !pBatch->active[i]) - continue; - if(now == 0) { - now = datetime.GetTime(NULL); /* good time call - the only one done */ - } - /* CAS loop, we write back a bit early, but that's OK... */ - /* we use reception time, not dequeue time - this is considered more appropriate and - * also faster ;) -- rgerhards, 2008-09-17 */ - do { - lastAct = pAction->f_time; - if(pBatch->pElem[i].pMsg->msgFlags & MARK) { - if((now - lastAct) < MarkInterval / 2) { - pBatch->active[i] = 0; - DBGPRINTF("batch item %d: action was recently called, ignoring " - "mark message\n", i); - break; /* do not update timestamp for non-written mark messages */ - } - } - } while(ATOMIC_CAS_time_t(&pAction->f_time, lastAct, - pBatch->pElem[i].pMsg->ttGenTime, &pAction->mutCAS) == 0); - if(pBatch->active[i]) { - DBGPRINTF("Called action(NotAllMark), processing batch[%d] via '%s'\n", - i, module.GetStateName(pAction->pMod)); - } - } - - iRet = doSubmitToActionQBatch(pAction, pBatch); - - free(pBatch->active); - pBatch->active = activeSave; - - RETiRet; -} - -static inline void -countStatsBatchEnq(action_t *pAction, batch_t *pBatch) -{ - int i; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( batchIsValidElem(pBatch, i)) { - STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); - } - } -} - - -/* enqueue a batch in direct mode. We have put this into its own function just to avoid - * cluttering the actual submit function. - * rgerhards, 2011-06-16 - */ -static inline rsRetVal -doQueueEnqObjDirectBatch(action_t *pAction, batch_t *pBatch) -{ - sbool bNeedSubmit; - sbool *activeSave; - int i; - DEFiRet; - - activeSave = pBatch->active; - copyActive(pBatch); - - /* note: for direct mode, we need to adjust the filter property. For non-direct - * this is not necessary, because in that case we enqueue only what actually needs - * to be processed. + /* TODO: think about the whole logic. If messages come in out of order, things + * tend to become a bit unreliable. On the other hand, this only happens if we have + * very high traffic, in which this use case here is not really affected (as the + * MarkInterval is pretty corase). */ - if(pAction->bExecWhenPrevSusp) { - bNeedSubmit = 0; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if(!pBatch->pElem[i].bPrevWasSuspended) { - DBGPRINTF("action enq stage: change active to 0 due to " - "failover case in elem %d\n", i); - pBatch->active[i] = 0; - } - if(batchIsValidElem(pBatch, i)) { - STATSCOUNTER_INC(pAction->ctrProcessed, pAction->mutCtrProcessed); - bNeedSubmit = 1; + /* CAS loop, we write back a bit early, but that's OK... */ + /* we use reception time, not dequeue time - this is considered more appropriate and + * also faster ;) -- rgerhards, 2008-09-17 */ + do { + lastAct = pAction->f_time; + if(pMsg->msgFlags & MARK) { + if((pMsg->ttGenTime - lastAct) < MarkInterval / 2) { + doProcess = 0; + DBGPRINTF("action was recently called, ignoring mark message\n"); + break; /* do not update timestamp for non-written mark messages */ } - DBGPRINTF("action %p[%d]: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, i, batchIsValidElem(pBatch, i), pBatch->eltState[i], - pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - } - if(bNeedSubmit) { - /* note: stats were already computed above */ - iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); - } else { - DBGPRINTF("no need to submit batch, all invalid\n"); } - } else { - if(GatherStats) - countStatsBatchEnq(pAction, pBatch); - iRet = qqueueEnqObjDirectBatch(pAction->pQueue, pBatch); - } + } while(ATOMIC_CAS_time_t(&pAction->f_time, lastAct, + pMsg->ttGenTime, &pAction->mutCAS) == 0); - free(pBatch->active); - pBatch->active = activeSave; - RETiRet; -} - -/* This submits the message to the action queue in case we do NOT need to handle repeat - * message processing. That case permits us to gain lots of freedom during processing - * and thus speed. - * rgerhards, 2010-06-08 - */ -static rsRetVal -doSubmitToActionQBatch(action_t *pAction, batch_t *pBatch) -{ - int i; - DEFiRet; - - DBGPRINTF("Called action(Batch), logging to %s\n", module.GetStateName(pAction->pMod)); - - if(pAction->pQueue->qType == QUEUETYPE_DIRECT) { - iRet = doQueueEnqObjDirectBatch(pAction, pBatch); - } else {/* in this case, we do single submits to the queue. - * TODO: optimize this, we may do at least a multi-submit! - */ - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], - pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( batchIsValidElem(pBatch, i) - && (pAction->bExecWhenPrevSusp == 0 || pBatch->pElem[i].bPrevWasSuspended == 1)) { - doSubmitToActionQ(pAction, pBatch->pElem[i].pMsg); - } - } + if(doProcess) { + DBGPRINTF("Called action(NotAllMark), processing via '%s'\n", + module.GetStateName(pAction->pMod)); + iRet = doSubmitToActionQ(pAction, pWti, pMsg); } RETiRet; } - -/* Helper to submit a batch of actions to the engine. Note that we have rather - * complicated processing here, so we need to do this one message after another. - * rgerhards, 2010-06-23 - */ -static inline rsRetVal -helperSubmitToActionQComplexBatch(action_t *pAction, batch_t *pBatch) -{ - int i; - DEFiRet; - - DBGPRINTF("Called action %p (complex case), logging to %s\n", - pAction, module.GetStateName(pAction->pMod)); - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - DBGPRINTF("action %p: valid:%d state:%d execWhenPrev:%d prevWasSusp:%d\n", - pAction, batchIsValidElem(pBatch, i), pBatch->eltState[i], - pAction->bExecWhenPrevSusp, pBatch->pElem[i].bPrevWasSuspended); - if( batchIsValidElem(pBatch, i) - && ((pAction->bExecWhenPrevSusp == 0) || pBatch->pElem[i].bPrevWasSuspended) ) { - doActionCallAction(pAction, pBatch, i); - } - } - - RETiRet; -} - -/* Call configured action, most complex case with all features supported (and thus slow). - * rgerhards, 2010-06-08 - */ -#pragma GCC diagnostic ignored "-Wempty-body" -static rsRetVal -doSubmitToActionQComplexBatch(action_t *pAction, batch_t *pBatch) -{ - DEFiRet; - - d_pthread_mutex_lock(&pAction->mutAction); - pthread_cleanup_push(mutexCancelCleanup, &pAction->mutAction); - iRet = helperSubmitToActionQComplexBatch(pAction, pBatch); - d_pthread_mutex_unlock(&pAction->mutAction); - pthread_cleanup_pop(0); /* remove mutex cleanup handler */ - - RETiRet; -} -#pragma GCC diagnostic warning "-Wempty-body" - - /* apply all params from param block to action. This supports the v6 config system. * Defaults must have been set appropriately during action construct! * rgerhards, 2011-08-01 @@ -1788,7 +1514,7 @@ actionApplyCnfParam(action_t *pAction, struct cnfparamvals *pvals) rsRetVal addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringRequest_t *pOMSR, struct cnfparamvals *actParams, - struct cnfparamvals *queueParams, int bSuspended) + struct nvlst *lst) { DEFiRet; int i; @@ -1817,7 +1543,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, pAction->bRepMsgHasMsg = cs.bActionRepMsgHasMsg; cs.iActExecEveryNthOccur = 0; /* auto-reset */ cs.iActExecEveryNthOccurTO = 0; /* auto-reset */ - cs.bActionWriteAllMarkMsgs = RSFALSE; /* auto-reset */ + cs.bActionWriteAllMarkMsgs = 1; /* auto-reset */ cs.pszActionName = NULL; /* free again! */ } else { actionApplyCnfParam(pAction, actParams); @@ -1875,15 +1601,10 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, pAction->pMod = pMod; pAction->pModData = pModData; - /* check if the module is compatible with select features (currently no such features exist) */ - pAction->eState = ACT_STATE_RDY; /* action is enabled */ - - if(bSuspended) - actionSuspend(pAction); - CHKiRet(actionConstructFinalize(pAction, queueParams)); + CHKiRet(actionConstructFinalize(pAction, lst)); - /* TODO: if we exit here, we have a memory leak... */ + /* TODO: if we exit here, we have a (quite acceptable...) memory leak */ *ppAction = pAction; /* finally store the action pointer */ @@ -1919,7 +1640,7 @@ resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unus static inline void initConfigVariables(void) { - cs.bActionWriteAllMarkMsgs = RSFALSE; + cs.bActionWriteAllMarkMsgs = 1; cs.glbliActionResumeRetryCount = 0; cs.bActExecWhenPrevSusp = 0; cs.iActExecOnceInterval = 0; @@ -1940,44 +1661,29 @@ rsRetVal actionNewInst(struct nvlst *lst, action_t **ppAction) { struct cnfparamvals *paramvals; - struct cnfparamvals *queueParams; modInfo_t *pMod; uchar *cnfModName = NULL; omodStringRequest_t *pOMSR; void *pModData; action_t *pAction; - int typeIdx; DEFiRet; paramvals = nvlstGetParams(lst, &pblk, NULL); if(paramvals == NULL) { - ABORT_FINALIZE(RS_RET_ERR); + ABORT_FINALIZE(RS_RET_PARAM_ERROR); } dbgprintf("action param blk after actionNewInst:\n"); cnfparamsPrint(&pblk, paramvals); - typeIdx = cnfparamGetIdx(&pblk, "type"); - if(paramvals[typeIdx].bUsed == 0) { - errmsg.LogError(0, RS_RET_CONF_RQRD_PARAM_MISSING, "action type missing"); - ABORT_FINALIZE(RS_RET_CONF_RQRD_PARAM_MISSING); // TODO: move this into rainerscript handlers - } cnfModName = (uchar*)es_str2cstr(paramvals[cnfparamGetIdx(&pblk, ("type"))].val.d.estr, NULL); if((pMod = module.FindWithCnfName(loadConf, cnfModName, eMOD_OUT)) == NULL) { errmsg.LogError(0, RS_RET_MOD_UNKNOWN, "module name '%s' is unknown", cnfModName); ABORT_FINALIZE(RS_RET_MOD_UNKNOWN); } - iRet = pMod->mod.om.newActInst(cnfModName, lst, &pModData, &pOMSR); - // TODO: check if RS_RET_SUSPENDED is still valid in v6! - if(iRet != RS_RET_OK && iRet != RS_RET_SUSPENDED) { - FINALIZE; /* iRet is already set to error state */ - } + CHKiRet(pMod->mod.om.newActInst(cnfModName, lst, &pModData, &pOMSR)); - qqueueDoCnfParams(lst, &queueParams); - - if((iRet = addAction(&pAction, pMod, pModData, pOMSR, paramvals, queueParams, - (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { + if((iRet = addAction(&pAction, pMod, pModData, pOMSR, paramvals, lst)) == RS_RET_OK) { /* check if the module is compatible with select features * (currently no such features exist) */ - pAction->eState = ACT_STATE_RDY; /* action is enabled */ loadConf->actions.nbrActions++; /* one more active action! */ } *ppAction = pAction; @@ -1988,8 +1694,6 @@ finalize_it: RETiRet; } -/* TODO: we are not yet a real object, the ClassInit here just looks like it is.. - */ rsRetVal actionClassInit(void) { DEFiRet; @@ -35,15 +35,6 @@ extern int glbliActionResumeRetryCount; -typedef enum { - ACT_STATE_DIED = 0, /* action permanently failed and now disabled - MUST BE ZERO! */ - ACT_STATE_RDY = 1, /* action ready, waiting for new transaction */ - ACT_STATE_ITX = 2, /* transaction active, waiting for new data or commit */ - ACT_STATE_COMM = 3, /* transaction finished (a transient state) */ - ACT_STATE_RTRY = 4, /* failure occured, trying to restablish ready state */ - ACT_STATE_SUSP = 5 /* suspended due to failure (return fail until timeout expired) */ -} action_state_t; - /* the following struct defines the action object data structure */ struct action_s { @@ -51,16 +42,15 @@ struct action_s { time_t tActNow; /* the current time for an action execution. Initially set to -1 and populated on an as-needed basis. This is a performance optimization. */ time_t tLastExec; /* time this action was last executed */ + int iActionNbr; /* this action's number (ID) */ sbool bExecWhenPrevSusp;/* execute only when previous action is suspended? */ sbool bWriteAllMarkMsgs;/* should all mark msgs be written (not matter how recent the action was executed)? */ - int iSecsExecOnceInterval; /* if non-zero, minimum seconds to wait until action is executed again */ - action_state_t eState; /* current state of action */ sbool bHadAutoCommit; /* did an auto-commit happen during doAction()? */ + sbool bDisabled; + int iSecsExecOnceInterval; /* if non-zero, minimum seconds to wait until action is executed again */ time_t ttResumeRtry; /* when is it time to retry the resume? */ - int iResumeOKinRow; /* number of times in a row that resume said OK with an immediate failure following */ int iResumeInterval;/* resume interval for this action */ int iResumeRetryCount;/* how often shall we retry a suspended action? (-1 --> eternal) */ - int iNbrResRtry; /* number of retries since last suspend */ int iNbrNoExec; /* number of matches that did not yet yield to an exec */ int iExecEveryNthOccur;/* execute this action only every n-th occurence (with n=0,1 -> always) */ int iExecEveryNthOccurTO;/* timeout for n-th occurence feature */ @@ -68,7 +58,7 @@ struct action_s { struct modInfo_s *pMod;/* pointer to output module handling this selector */ void *pModData; /* pointer to module data - content is module-specific */ sbool bRepMsgHasMsg; /* "message repeated..." has msg fragment in it (0-no, 1-yes) */ - rsRetVal (*submitToActQ)(action_t *, batch_t *);/* function submit message to action queue */ + rsRetVal (*submitToActQ)(action_t *, wti_t*, msg_t*);/* function submit message to action queue */ rsRetVal (*qConstruct)(struct queue_s *pThis); enum { ACT_STRING_PASSING = 0, ACT_ARRAY_PASSING = 1, ACT_MSG_PASSING = 2, ACT_JSON_PASSING = 3} @@ -78,8 +68,7 @@ struct action_s { * in this order. */ qqueue_t *pQueue; /* action queue */ pthread_mutex_t mutAction; /* primary action mutex */ - pthread_mutex_t mutActExec; /* mutex to guard actual execution of doAction for single-threaded modules */ - uchar *pszName; /* action name (for documentation) */ + uchar *pszName; /* action name */ DEF_ATOMIC_HELPER_MUT(mutCAS); /* for statistics subsystem */ statsobj_t *statsobj; @@ -91,17 +80,21 @@ struct action_s { /* function prototypes */ rsRetVal actionConstruct(action_t **ppThis); -rsRetVal actionConstructFinalize(action_t *pThis, struct cnfparamvals *queueParams); +rsRetVal actionConstructFinalize(action_t *pThis, struct nvlst *lst); rsRetVal actionDestruct(action_t *pThis); -rsRetVal actionDbgPrint(action_t *pThis); +//rsRetVal actionDbgPrint(action_t *pThis); rsRetVal actionSetGlobalResumeInterval(int iNewVal); rsRetVal actionDoAction(action_t *pAction); -rsRetVal actionWriteToAction(action_t *pAction, msg_t *pMsg); +rsRetVal actionWriteToAction(action_t *pAction, msg_t *pMsg, wti_t*); rsRetVal actionCallHUPHdlr(action_t *pAction); rsRetVal actionClassInit(void); -rsRetVal addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringRequest_t *pOMSR, struct cnfparamvals *actParams, struct cnfparamvals *queueParams, int bSuspended); +rsRetVal addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringRequest_t *pOMSR, struct cnfparamvals *actParams, struct nvlst *lst); rsRetVal activateActions(void); rsRetVal actionNewInst(struct nvlst *lst, action_t **ppAction); rsRetVal actionProcessCnf(struct cnfobj *o); +void actionCommitAllDirect(wti_t *pWti); + +/* external data */ +extern int iActionNbr; #endif /* #ifndef ACTION_H_INCLUDED */ diff --git a/build/rhel/rsyslog/rsyslog.spec b/build/rhel/rsyslog/rsyslog.spec index 4aa91df3..7bb367d4 100644 --- a/build/rhel/rsyslog/rsyslog.spec +++ b/build/rhel/rsyslog/rsyslog.spec @@ -47,6 +47,11 @@ Summary: ElasticSearch output module for rsyslog Group: System Environment/Daemons Requires: %name = %version-%release +%package mmcount +Summary: Message counting support for rsyslog +Group: System Environment/Daemons +Requires: %name = %version-%release + %package mmjsonparse Summary: JSON enhanced logging support Group: System Environment/Daemons @@ -154,6 +159,11 @@ relay chains. This module provides the capability for rsyslog to feed logs directly into Elasticsearch. +%description mmcount +This module provides the capability to count log messages by severity +or json property of given app-name. The count value is added into the +log message in json property named 'mmcount' + %description mmjsonparse This module provides the capability to recognize and parse JSON enhanced syslog messages. @@ -226,6 +236,7 @@ export PKG_CONFIG=/usr/bin/pkg-config %configure --disable-static \ --disable-testbench \ --enable-elasticsearch \ + --enable-mmcount \ --enable-mmjsonparse \ --enable-mmnormalize \ --enable-imzmq3 \ @@ -339,6 +350,10 @@ fi %defattr(-,root,root) %{_libdir}/rsyslog/omelasticsearch.so +%files mmcount +%defattr(-,root,root) +%{_libdir}/rsyslog/mmcount.so + %files mmjsonparse %defattr(-,root,root) %{_libdir}/rsyslog/mmjsonparse.so diff --git a/configure.ac b/configure.ac index ee3394b5..875a4ca0 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[7.4.6],[rsyslog@lists.adiscon.com]) +AC_INIT([rsyslog],[8.1.0],[rsyslog@lists.adiscon.com]) AM_INIT_AUTOMAKE m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) @@ -37,6 +37,10 @@ PKG_CHECK_MODULES([JSON_C], [json],, [ PKG_CHECK_MODULES([JSON_C], [json-c]) ]) +# if int64 is supported, use it +AC_CHECK_LIB(json-c, json_object_new_object,,) +AC_CHECK_FUNCS(json_object_new_int64,,) + case "${host}" in *-*-linux*) AC_DEFINE([OS_LINUX], [1], [Indicator for a Linux OS]) @@ -121,7 +125,7 @@ AC_TYPE_SIGNAL AC_FUNC_STAT AC_FUNC_STRERROR_R AC_FUNC_VPRINTF -AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setsid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64]) +AC_CHECK_FUNCS([flock recvmmsg basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setsid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64]) # getifaddrs is in libc (mostly) or in libsocket (eg Solaris 11) or not defined (eg Solaris 10) AC_SEARCH_LIBS([getifaddrs], [socket], [AC_DEFINE(HAVE_GETIFADDRS, [1], [set define])]) @@ -153,6 +157,14 @@ AC_TRY_COMPILE([ AC_MSG_RESULT(no; defined as 64) ) +# Check for __builtin_expect() +AC_MSG_CHECKING([for __builtin_expect()]) +AC_LINK_IFELSE([AC_LANG_PROGRAM(, return __builtin_expect(main != 0, 1))], + [AC_DEFINE(HAVE_BUILTIN_EXPECT, 1, + Define to 1 if compiler supports __builtin_expect) + AC_MSG_RESULT([yes])], + [AC_MSG_RESULT([no])]) + # check for availability of atomic operations RS_ATOMIC_OPERATIONS RS_ATOMIC_OPERATIONS_64BIT @@ -964,6 +976,89 @@ AC_ARG_ENABLE(mmanon, AM_CONDITIONAL(ENABLE_MMANON, test x$enable_mmanon = xyes) +# mmutf8fix +AC_ARG_ENABLE(mmutf8fix, + [AS_HELP_STRING([--enable-mmutf8fix],[Enable building mmutf8fix support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmutf8fix="yes" ;; + no) enable_mmutf8fix="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmutf8fix) ;; + esac], + [enable_mmutf8fix=no] +) +AM_CONDITIONAL(ENABLE_MMUTF8FIX, test x$enable_mmutf8fix = xyes) + + +# mmcount +AC_ARG_ENABLE(mmcount, + [AS_HELP_STRING([--enable-mmcount],[Enable message counting @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmcount="yes" ;; + no) enable_mmcount="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmcount) ;; + esac], + [enable_mmcount=no] +) +AM_CONDITIONAL(ENABLE_MMCOUNT, test x$enable_mmcount = xyes) + + +# mmsequence +AC_ARG_ENABLE(mmsequence, + [AS_HELP_STRING([--enable-mmsequence],[Enable sequence generator @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmsequence="yes" ;; + no) enable_mmsequence="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmsequence) ;; + esac], + [enable_mmsequence=no] +) +AM_CONDITIONAL(ENABLE_MMSEQUENCE, test x$enable_mmsequence = xyes) + + +# mmfields +AC_ARG_ENABLE(mmfields, + [AS_HELP_STRING([--enable-mmfields],[Enable building mmfields support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmfields="yes" ;; + no) enable_mmfields="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmfields) ;; + esac], + [enable_mmfields=no] +) +AM_CONDITIONAL(ENABLE_MMFIELDS, test x$enable_mmfields = xyes) + +# mmpstrucdata +AC_ARG_ENABLE(mmpstrucdata, + [AS_HELP_STRING([--enable-mmpstrucdata],[Enable building mmpstrucdata support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmpstrucdata="yes" ;; + no) enable_mmpstrucdata="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmpstrucdata) ;; + esac], + [enable_mmpstrucdata=no] +) +AM_CONDITIONAL(ENABLE_MMPSTRUCDATA, test x$enable_mmpstrucdata = xyes) + + +# mmrfc5424addhmac +AC_ARG_ENABLE(mmrfc5424addhmac, + [AS_HELP_STRING([--enable-mmrfc5424addhmac],[Enable building mmrfc5424addhmac support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_mmrfc5424addhmac="yes" ;; + no) enable_mmrfc5424addhmac="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmrfc5424addhmac) ;; + esac], + [enable_mmrfc5424addhmac=no] +) +if test "x$enable_mmrfc5424addhmac" = "xyes"; then + PKG_CHECK_MODULES(OPENSSL, openssl >= 0.9.7) +#AC_CHECK_LIB([crypto],[CRYPTO_new_ex_data], [], [AC_MSG_ERROR([OpenSSL libraries required])]) +#AC_CHECK_LIB([ssl],[SSL_library_init], [], [AC_MSG_ERROR([OpenSSL libraries required])]) +#AC_CHECK_HEADERS([openssl/crypto.h openssl/x509.h openssl/pem.h openssl/ssl.h openssl/err.h],[],[AC_MSG_ERROR([OpenSSL headers required])]) +fi +AM_CONDITIONAL(ENABLE_MMRFC5424ADDHMAC, test x$enable_mmrfc5424addhmac = xyes) + + # RELP support AC_ARG_ENABLE(relp, [AS_HELP_STRING([--enable-relp],[Enable RELP support @<:@default=no@:>@])], @@ -975,7 +1070,7 @@ AC_ARG_ENABLE(relp, [enable_relp=no] ) if test "x$enable_relp" = "xyes"; then - PKG_CHECK_MODULES(RELP, relp >= 1.0.3) + PKG_CHECK_MODULES(RELP, relp >= 1.2.0) fi AM_CONDITIONAL(ENABLE_RELP, test x$enable_relp = xyes) @@ -1258,7 +1353,7 @@ AC_ARG_ENABLE(omruleset, no) enable_omruleset="no" ;; *) AC_MSG_ERROR(bad value ${enableval} for --enable-omruleset) ;; esac], - [enable_omruleset=yes] + [enable_omruleset=no] ) AM_CONDITIONAL(ENABLE_OMRULESET, test x$enable_omruleset = xyes) @@ -1480,6 +1575,12 @@ AC_CONFIG_FILES([Makefile \ plugins/mmjsonparse/Makefile \ plugins/mmaudit/Makefile \ plugins/mmanon/Makefile \ + plugins/mmutf8fix/Makefile \ + plugins/mmcount/Makefile \ + plugins/mmsequence/Makefile \ + plugins/mmfields/Makefile \ + plugins/mmpstrucdata/Makefile \ + plugins/mmrfc5424addhmac/Makefile \ plugins/omelasticsearch/Makefile \ plugins/sm_cust_bindcdr/Makefile \ plugins/mmsnmptrapd/Makefile \ @@ -1503,6 +1604,8 @@ echo " uuid support enabled: $enable_uuid" echo " Log file signing support: $enable_guardtime" echo " Log file encryption support: $enable_libgcrypt" echo " anonymization support enabled: $enable_mmanon" +echo " message counting support enabled: $enable_mmcount" +echo " mmfields enabled: $enable_mmfields" echo echo "---{ input plugins }---" echo " Klog functionality enabled: $enable_klog ($os_type)" @@ -1541,6 +1644,10 @@ echo " mmnormalize module will be compiled: $enable_mmnormalize" echo " mmjsonparse module will be compiled: $enable_mmjsonparse" echo " mmjaduit module will be compiled: $enable_mmaudit" echo " mmsnmptrapd module will be compiled: $enable_mmsnmptrapd" +echo " mmutf8fix enabled: $enable_mmutf8fix" +echo " mmrfc5424addhmac enabled: $enable_mmrfc5424addhmac" +echo " mmpstrucdata enabled: $enable_mmpstrucdata" +echo " mmsequence enabled: $enable_mmsequence" echo echo "---{ strgen modules }---" echo " sm_cust_bindcdr module will be compiled: $enable_sm_cust_bindcdr" @@ -35,7 +35,8 @@ rsRetVal multiSubmitFlush(multi_submit_t *pMultiSub); rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags); rsRetVal __attribute__((deprecated)) parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime, ruleset_t *pRuleset); rsRetVal diagGetMainMsgQSize(int *piSize); /* for imdiag */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams); +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct nvlst *lst); +rsRetVal startMainQueue(qqueue_t *pQueue); extern int MarkInterval; extern qqueue_t *pMsgQueue; /* the main message queue */ diff --git a/doc/Makefile.am b/doc/Makefile.am index e1757644..f7a9efe0 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -37,6 +37,15 @@ html_files = \ omjournal.html \ imjournal.html \ mmanon.html \ + mmsequence.html \ + mmcount.html + mmfields.html \ + mmrfc5424addhmac.html \ + mmutf8fix.html \ + omelasticsearch.html \ + ommongodb.html \ + queue_parameters.html + mmpstrucdata.html \ omusrmsg.html \ omstdout.html \ omudpspoof.html \ @@ -112,6 +121,9 @@ html_files = \ v3compatibility.html \ v4compatibility.html \ v5compatibility.html \ + v6compatibility.html \ + v7compatibility.html \ + v8compatibility.html \ im3195.html \ netstream.html \ ns_gtls.html \ @@ -148,11 +160,10 @@ html_files = \ rsconf1_omfileforcechown.html \ rsyslog_queue_pointers.jpeg \ rsyslog_queue_pointers2.jpeg \ - v6compatibility.html \ - v7compatibility.html \ rsyslog_conf_basic_structure.html \ rsyslog_conf_sysklogd_compatibility.html \ imkmsg.html \ + rainerscript_call.html \ src/classes.dia grfx_files = \ diff --git a/doc/build_from_repo.html b/doc/build_from_repo.html index a06863e1..693704a8 100644 --- a/doc/build_from_repo.html +++ b/doc/build_from_repo.html @@ -8,7 +8,7 @@ distribution tarball to generate it. But there may be situations where it is des to build directly from the source repository. This is useful for people who would like to participate in development or who would like to use the latest, not-yet-released code. The later may especially be the case if you are asked to try out an experimental version. -<p>Building from the repsitory is not much different than building from the source +<p>Building from the repository is not much different than building from the source tarball, but some files are missing because they are output files and thus do not belong into the repository. <h2>Obtaining the Source</h2> diff --git a/doc/debug.html b/doc/debug.html index 229aeb08..9f05c687 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -27,7 +27,7 @@ be replaced by something else.</p> <p>There are two environment variables that set several debug settings: <ul> <li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") -writes (allmost) +writes (almost) all debug message to the specified log file in addition to stdout. Some system messages (e.g. segfault or abort message) are not written to the file as we can not capture them. @@ -133,7 +133,7 @@ turned on. threads and their calling stack by sending SIGUSR2. However, the usefulness of that information is very much depending on rsyslog compile-time settings, must importantly the --enable-rtinst configure flag. Note that activating this option causes additional overhead -and slows down rsyslgod considerable. So if you do that, you need to check if it is +and slows down rsyslogd considerable. So if you do that, you need to check if it is capable to handle the workload. Also, threading behavior is modified by the runtime instrumentation. <p>Sending SIGUSR2 writes new process state information to the log file each time @@ -143,13 +143,13 @@ some diagnostic information on the current processing state. In that case, turni on the mutex debugging options (see above) is probably useful. <h2>Interpreting the Logs</h2> <p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable -information to users. Just be warned that logs sometimes contains informaton the looks like +information to users. Just be warned that logs sometimes contains information the looks like an error, but actually is none. We put a lot of extra information into the logs, and there are some cases where it is OK for an error to happen, we just wanted to record it inside the log. The code handles many cases automatically. So, in short, the log may not make sense to you, but it (hopefully) makes sense to a developer. Note that we developers often need many lines of the log file, it is relatively rare that a problem can be diagnosed by -looking at just a couple of (hundered) log records. +looking at just a couple of (hundred) log records. <h2>Security Risks</h2> <p>The debug log will reveal potentially sensible information, including user accounts and passwords, to anyone able to read the log file. As such, it is recommended to properly @@ -159,8 +159,9 @@ attack or try to hide some information from the log file. As such, it is suggest enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled, SIGUSR1 and SIGUSR2 are completely ignored. <p>When running in any of the debug modes (including on demand mode), an interactive -instance of rsyslogd can be aborted by pressing ctl-c. -<p><b>See Also</b> +instance of rsyslogd can be aborted by pressing ctrl-c. +<p> +<h2>See Also</h2> <ul> <li><a href="http://www.rsyslog.com/how-to-use-debug-on-demand/">How to use debug on demand</a></li> </ul> diff --git a/doc/dev_oplugins.html b/doc/dev_oplugins.html index 4a9cd15d..bd2bfc3f 100644 --- a/doc/dev_oplugins.html +++ b/doc/dev_oplugins.html @@ -15,13 +15,15 @@ and pointers than to have nothing. <p>The best to get started with rsyslog plugin development is by looking at existing plugins. All that start with "om" are <b>o</b>utput <b>m</b>odules. That means they are primarily thought of being message sinks. In theory, however, output -plugins may aggergate other functionality, too. Nobody has taken this route so far +plugins may aggregate other functionality, too. Nobody has taken this route so far so if you would like to do that, it is highly suggested to post your plan on the rsyslog mailing list, first (so that we can offer advise). <p>The rsyslog distribution tarball contains the omstdout plugin which is extremely well targeted for getting started. Just note that this plugin itself is not meant for production use. But it is very simplistic and so a really good starting point to -grasp the core ideas. +grasp the core ideas. Also, it supports two different parameter-passing modes and +offers some light functionality. Note, however, that in order to use omstdout as is, you +need to run rsyslog interactively as otherwise stdout is redirected. <p>In any case, you should also read the comments in ./runtime/module-template.h. Output plugins are build based on a large set of code-generating macros. These macros handle most of the plumbing needed by the interface. As long as no @@ -44,7 +46,7 @@ that shares a single instanceData structure. <p>So as long as you do not mess around with global data, you do not need to think about multithreading (and can apply a purely sequential programming methodology). -<p>Please note that duringt the configuraton parsing stage of execution, access to +<p>Please note that during the configuration parsing stage of execution, access to global variables for the configuration system is safe. In that stage, the core will only call sequentially into the plugin. <h3>Getting Message Data</h3> @@ -69,7 +71,7 @@ get it into the core (so far, we could accept all such suggestions - no promise, request access to the template components. The typical use case seems to be databases, where you would like to access properties via specific fields. With that mode, you receive a char ** array, where each array element points to one field from the template (from -left to right). Fields start at arrray index 0 and a NULL pointer means you have +left to right). Fields start at array index 0 and a NULL pointer means you have reached the end of the array (the typical Unix "poor man's linked list in an array" design). Note, however, that each of the individual components is a string. It is not a date stamp, number or whatever, but a string. This is because rsyslog processes @@ -132,19 +134,19 @@ for example in MongoDB or ElasticSearch. a single-message interface was supported. <p>With the <b>single message</b> plugin interface, each message is passed via a separate call to the plugin. Most importantly, the rsyslog engine assumes that each call to the plugin is a complete transaction -and as such assumes that messages be properly commited after the plugin returns to the engine. +and as such assumes that messages be properly committed after the plugin returns to the engine. <p>With the <b>batching</b> interface, rsyslog employs something along the line of "transactions". Obviously, the rsyslog core can not make non-transactional outputs to be fully transactional. But what it can is support that the output tells the core which -messages have been commited by the output and which not yet. The core can than take care -of those uncommited messages when problems occur. For example, if a plugin has received -50 messages but not yet told the core that it commited them, and then returns an error state, the +messages have been committed by the output and which not yet. The core can than take care +of those uncommitted messages when problems occur. For example, if a plugin has received +50 messages but not yet told the core that it committed them, and then returns an error state, the core assumes that all these 50 messages were <b>not</b> written to the output. The core then -requeues all 50 messages and does the usual retry processing. Once the output plugin tells the +re-queues all 50 messages and does the usual retry processing. Once the output plugin tells the core that it is ready again to accept messages, the rsyslog core will provide it with these 50 -not yet commited messages again (actually, at this point, the rsyslog core no longer knows that -it is re-submiting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 -messages were commited (before it failed), then only 10 would have been requeued and resubmitted. +not yet committed messages again (actually, at this point, the rsyslog core no longer knows that +it is re-submitting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 +messages were committed (before it failed), then only 10 would have been re-queued and resubmitted. <p>In order to provide an efficient implementation, there are some (mild) constraints in that transactional model: first of all, rsyslog itself specifies the ultimate transaction boundaries. That is, it tells the plugin when a transaction begins and when it must finish. The plugin @@ -155,7 +157,7 @@ transaction support. Note that batch sizes are variable within the range of 1 to maximum limit. Most importantly, that means that plugins may receive batches of single messages, so they are required to commit each message individually. If the plugin tries to be "smarter" than the rsyslog engine and does not commit messages in those cases (for example), the plugin -puts message stream integrity at risk: once rsyslog has notified the plugin of transacton end, +puts message stream integrity at risk: once rsyslog has notified the plugin of transaction end, it discards all messages as it considers them committed and save. If now something goes wrong, the rsyslog core does not try to recover lost messages (and keep in mind that "goes wrong" includes such uncontrollable things like connection loss to a database server). So it is @@ -170,8 +172,8 @@ This is also under evaluation and, once decided, the core will offer an interfac to preserve message stream integrity for properly-crafted plugins). <p>The second restriction is that if a plugin makes commits in between (what is perfectly legal) those commits must be in-order. So if a commit is made for message ten out of 50, -this means that messages one to nine are also commited. It would be possible to remove -this restriction, but we have decided to deliberately introduce it to simpify things. +this means that messages one to nine are also committed. It would be possible to remove +this restriction, but we have decided to deliberately introduce it to simplify things. <h3>Output Plugin Transaction Interface</h3> <p>In order to keep compatible with existing output plugins (and because it introduces no complexity), the transactional plugin interface is build on the traditional @@ -231,7 +233,7 @@ But they convey additional information about the commit status as follows: <table border="0"> <tr> <td valign="top"><i>RS_RET_OK</i></td> -<td>The record and all previous inside the batch has been commited. +<td>The record and all previous inside the batch has been committed. <i>Note:</i> this definition is what makes integrating plugins without the transaction being/end calls so easy - this is the traditional "success" return state and if every call returns it, there is no need for actually calling @@ -239,7 +241,7 @@ state and if every call returns it, there is no need for actually calling </tr> <tr> <td valign="top"><i>RS_RET_DEFER_COMMIT</i></td> -<td>The record has been processed, but is not yet commited. This is the +<td>The record has been processed, but is not yet committed. This is the expected state for transactional-aware plugins.</td> </tr> <tr> @@ -248,7 +250,7 @@ expected state for transactional-aware plugins.</td> current one not yet. This state is introduced to support sources that fill up buffers and commit once a buffer is completely filled. That may occur halfway in the next record, so it may be important to be able to tell the -engine the everything up to the previouos record is commited</td> +engine the everything up to the previous record is committed</td> </tr> </table> <p>Note that the typical <b>calling cycle</b> is <code>beginTransaction()</code>, @@ -269,7 +271,7 @@ exists. So we introduce it with that release. What the means is if a rsyslog cor not provide this query interface, it is a core that was build before batching support was available. So the absence of a query interface indicates that the transactional interface is not available. One might now be tempted the think there is no need to do -the actual check, but is is recommended to ask the rsyslog engine explicitely if +the actual check, but is is recommended to ask the rsyslog engine explicitly if the transactional interface is present and will be honored. This enables us to create versions in the future which have, for whatever reason we do not yet know, no support for this interface. diff --git a/doc/dev_queue.html b/doc/dev_queue.html index bf2af7f0..6d5fe73f 100644 --- a/doc/dev_queue.html +++ b/doc/dev_queue.html @@ -80,7 +80,7 @@ and terminating while waiting on the primary queue to fill. In practice, this is highly unlikely (but only for the main message queue) because rsyslog issues a startup message. HOWEVER, we can not rely on that, it would introduce a race. If the primary rsyslog thread (the one that issues the message) is scheduled very -late and there is a low inactivty timeout for queue workers, the queue worker +late and there is a low inactivity timeout for queue workers, the queue worker may terminate before the startup message is issued. And if the on-disk queue holds only a few messages, it may become empty before the DA worker is re-initiated again. So it is possible that the DA run mode termination criteria @@ -105,7 +105,7 @@ clean shutdown of the DA queue).</p> <p>One might think that it would be more natural for the DA queue to detect being idle and shut down itself. However, there are some issues associated with that. Most importantly, all queue worker threads need to be shut down during -queue destruction. Only after that has happend, final destruction steps can +queue destruction. Only after that has happened, final destruction steps can happen (else we would have a myriad of races). However, it is the DA queues worker thread that detects it is empty (empty queue detection always happens at the consumer side and must so). That would lead to the DA queue worker thread to @@ -115,7 +115,7 @@ destructed). Obviously, this does not work out (and I didn't even mention the other issues - so let's forget about it). As such, the thread that enqueues messages must destruct the queue - and that is the primary queue's DA worker thread.</p> -<p>There are some subleties due to thread synchronization and the fact that the +<p>There are some subtleties due to thread synchronization and the fact that the DA consumer may not be running (in a <b>case-2 startup</b>). So it is not trivial to reliably change the queue back from DA run mode to regular run mode. The priority is a clean switch. We accept the fact that there may be situations @@ -132,7 +132,7 @@ most probably even lead to worse performance under regular conditions).</p> </ol> <p>Case 2 is unlikely, but may happen (see info above on a case 2 startup).</p> <p><b>The DA worker may also not wait at all,</b> because it is actively -executing and shuffeling messages between the queues. In that case, however, the +executing and shuffling messages between the queues. In that case, however, the program flow passes both of the two wait conditions but simply does not wait.</p> <p><b>Finally, the DA worker may be inactive </b>(again, with a case-2 startup). In that case no work(er) at all is executed. Most importantly, without the DA @@ -247,4 +247,4 @@ no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/droppriv.html b/doc/droppriv.html index 7293e872..75773e64 100644 --- a/doc/droppriv.html +++ b/doc/droppriv.html @@ -18,7 +18,7 @@ rsyslogd needs to start up as root. user. That is probably the safest way of operations. However, if a startup as root is required, you can use the $PrivDropToGroup and $PrivDropToUser config directives to specify a group and/or user that rsyslogd should drop to after initialization. -Once this happend, the daemon runs without high privileges (depending, of +Once this happens, the daemon runs without high privileges (depending, of course, on the permissions of the user account you specified). <p>There is some additional information available in the <a href="http://wiki.rsyslog.com/index.php/Security#Dropping_Privileges">rsyslog wiki</a>. diff --git a/doc/features.html b/doc/features.html index 626ff65d..ecf6a014 100644 --- a/doc/features.html +++ b/doc/features.html @@ -42,7 +42,7 @@ into syslog messages (one per line)</li> <li>ability to configure backup syslog/database servers - if the primary fails, control is switched to a prioritized list of backups</li> <li>support for receiving messages via reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php"> -RFC 3195</a> delivery (a bit clumpsy to build right now...)</li> +RFC 3195</a> delivery (a bit clumsy to build right now...)</li> <li>ability to generate file names and directories (log targets) dynamically, based on many different properties</li> <li>control of log output format, including ability to present @@ -95,7 +95,7 @@ via custom plugins</li> <li> an easy-to-write to plugin interface</li> <li> ability to send SNMP trap messages</li> <li> ability to filter out messages based on sequence of arrival</li> -<li>support for comma-seperated-values (CSV) output generation +<li>support for comma-separated-values (CSV) output generation (via the "csv" property replace option). The CSV format supported is that from RFC 4180.</li> <li>support for arbitrary complex boolean, string and diff --git a/doc/free_support.html b/doc/free_support.html index 182a82cd..a3a9a69a 100644 --- a/doc/free_support.html +++ b/doc/free_support.html @@ -18,7 +18,7 @@ reason is quite simple: If I do personal support, you gain some advantage withou contributing something back. Think about it: if you ask your question on the public forum or mailing list, other with the same problem can you and, most importantly, even years later find your post (and the answer) and get the problem solved. So by -solving your issue in public, you help create a great community ressource and also +solving your issue in public, you help create a great community resource and also help your fellow users finding solutions quicker. In the long term, this also contributes to improved code because the more questions users can find solutions to themselves, the fewer I need to look at. diff --git a/doc/history.html b/doc/history.html index 57b64004..a071461e 100644 --- a/doc/history.html +++ b/doc/history.html @@ -117,7 +117,7 @@ the need to have complex expression support, which was also the first use case. On February, 28th rsyslog 3.12.0 was released, the first version to contain expression support. This also meant that rsyslog from that date on supported all syslog-ng major features, but had a -number of major features exlusive to it. With 3.12.0, I consider +number of major features exclusive to it. With 3.12.0, I consider rsyslog fully superior to syslog-ng (except for platform support).</p> <p>Following the Fedora Developer's conference in Brno <b>2012</b>, rsyslog diff --git a/doc/im3195.html b/doc/im3195.html index aad9f3d1..317ab840 100644 --- a/doc/im3195.html +++ b/doc/im3195.html @@ -25,7 +25,7 @@ The port on which imklog listens for RFC 3195 messages. The default port is 601 to this input module, but we have NOT conducted any testing. Also, the module does not yet properly handle the recovery case. If someone intends to put this module into production, good testing should be -cunducted. It also is a good idea to notify the rsyslog project that you intend to use +conducted. It also is a good idea to notify the rsyslog project that you intend to use it in production. In this case, we'll probably give the module another cleanup. We don't do this now because so far it looks just like a big waste of time. diff --git a/doc/imfile.html b/doc/imfile.html index a69f62e9..8c9a425f 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -106,6 +106,15 @@ This mode should defined when having multiline messages. The value can range fro <br>0 (<strong>default</strong>) - line based (Each line is a new message) <br>1 - paragraph (There is a blank line between log messages) <br>2 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it) +<li><b>escapeLF</b> [<b>on</b>/off] (requires v7.5.3+)<br> +This is only meaningful if multi-line messages are to be processed. LF characters embedded into +syslog messages cause a lot of trouble, as most tools and even the legacy syslog TCP protocol +do not expect these. If set to "on", this option avoid this trouble by properly escaping +LF characters to the 4-byte sequence "#012". This is consistent with other rsyslog control character +escaping. By default, escaping is turned on. If you turn it off, make sure you test very carefully +with all associated tools. Please note that if you intend to use plain TCP syslog with embedded +LF characters, you need to enable octet-counted framing. For more details, see Rainer's blog posting on +<a href="http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html">imfile LF escaping</a>. <li><b>MaxLinesAtOnce</b> [number]</b> <br> This is useful if multiple files need to be monitored. If set to 0, each file @@ -160,6 +169,11 @@ input(type="imfile" File="/path/to/file2" <p><b>Legacy Configuration Directives</b>:</p> +<p>Note: in order to preserve compatibility with previous versions, the +LF escaping in multi-line messages is turned off for legacy-configured +file monitors (the "escapeLF" input parameter). This can cause serious problems. +So it is highly suggested that new deployments use the new input() statement +and keep LF escaping turned on. <ul> <li><strong>$InputFileName /path/to/file</strong><br> equivalent to: File </li> diff --git a/doc/imklog.html b/doc/imklog.html index 1f195b16..f2e36fd2 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -25,7 +25,7 @@ imklog generates some messages of itself (e.g. on problems, startup and shutdown) and these do not stem from the kernel. Historically, under Linux, these too have "kern" facility. Thus, on Linux platforms the default is "kern" while on others it is "syslogd". You usually do not -need to specify this configuratin directive - it is included primarily +need to specify this configuration directive - it is included primarily for few limited cases where it is needed for good reason. Bottom line: if you don't have a good idea why you should use this setting, do not touch it.</li> diff --git a/doc/imkmsg.html b/doc/imkmsg.html index 23b96147..ba73715d 100644 --- a/doc/imkmsg.html +++ b/doc/imkmsg.html @@ -16,7 +16,7 @@ Milan Bartos <p>Reads messages from the /dev/kmsg structured kernel log and submits them to the syslog engine.</p> <p> -The printk log buffer constains log records. These records are exported by /dev/kmsg +The printk log buffer contains log records. These records are exported by /dev/kmsg device as structured data in the following format:<br /> "level,sequnum,timestamp;<message text>\n"<br /> There could be continuation lines starting with space that contains key/value pairs.<br /> diff --git a/doc/impstats.html b/doc/impstats.html index 392fc431..3b206941 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -12,7 +12,7 @@ <p><b>Description</b>:</p> <p>This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So -availabilty and format of counters may change and is not yet stable (so be +availability and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version). <p>The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). @@ -20,41 +20,55 @@ Be sure that your configuration records the counter messages (default is syslog. Besides logging to the regular syslog stream, the module can also be configured to write statistics data into a (local) file. <p>Note that loading this module has impact on rsyslog performance. Depending on -settings, this impact may be noticable (for high-load environments). +settings, this impact may be noticeable (for high-load environments). <p>The rsyslog website has an updated overview of available <a href="http://rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a>. </p> <p><b>Module Confguration Parameters</b>:</p> <p>This module supports module parameters, only. <ul> - <li><strong>interval </strong>[seconds] (default 300 [5minutes])<br> + <li><b>interval </b>[seconds] (default 300 [5minutes])<br> Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the actual interval may be a bit longer. We do not try to be precise and so the interval is actually a sleep period which is entered after generating all messages. So the actual interval is what is configured here plus the actual time required to generate messages. In general, the difference should not really matter. <br></li> - <li><strong>facility </strong>[templateName]<br> + <li><b>facility </b>[facility number]<br> The numerical syslog facility code to be used for generated messages. Default is 5 (syslog). This is useful for filtering messages. <br></li> - <li><strong>severity </strong>[templateName]<br> + <li><b>severity </b>[severity number]<br> The numerical syslog severity code to be used for generated messages. Default is 6 (info).This is useful for filtering messages. <br></li> - <li><strong>format </strong>[json/cee/<b>legacy</b>](rsyslog v6.3.8+ only)<br> + <li><b>resetCounters </b>[<b>off</b>/on] - available since 7.5.3<br> + When set to "on", counters are automatically reset after they are emitted. In that + case, the contain only deltas to the last value emitted. When set + to "off", counters always accumulate their values. + Note that in auto-reset mode not all counters can be reset. Some counters (like queue size) + are directly obtained from internal object and cannot be modified. Also, auto-resetting + introduces some additional slight inaccuracies due to the multi-threaded nature of + rsyslog and the fact that for performance reasons it cannot serialize access + to counter variables.</br> + As an alternative to auto-reset mode, you can use rsyslog's statistics + manipulation scripts to create delta values from the regular statistic + logs. This is the suggested method if deltas are not necessarily needed in + real-time. + <br></li> + <li><b>format </b>[json/cee/<b>legacy</b>] - available since 6.3.8<br> Specifies the format of emitted stats messages. The default of "legacy" is compatible with pre v6-rsyslog. The other options provide support for structured formats (note the "cee" is actually "project lumberack" logging). <br></li> - <li><strong>log.syslog </strong>[<b>on</b>/off] - available since 7.3.6<br> + <li><b>log.syslog </b>[<b>on</b>/off] - available since 7.3.6<br> This is a boolean setting specifying if data should be sent to the usual syslog stream. This is useful if custom formatting or more elaborate processing is desired. However, output is placed under the same restrictions as regular syslog data, especially in regard to the queue position (stats data may sit for an extended period of time in queues if they are full).<br></li> - <li><strong>log.file </strong>[file name] - available since 7.3.6<br> + <li><b>log.file </b>[file name] - available since 7.3.6<br> If specified, statistics data is written the specified file. For robustness, this should be a local file. The file format cannot be customized, it consists of a date header, followed by a colon, @@ -66,6 +80,8 @@ settings, this impact may be noticable (for high-load environments). on file logging does NOT turn of syslog logging. If that is desired log.syslog="off" must be explicitely set. <br></li> + <li><b>Ruleset</b> [ruleset] - available since 7.5.6<br> + Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <p><b>Legacx Configuration Directives</b>:</p> diff --git a/doc/imptcp.html b/doc/imptcp.html index aece428d..e1e35239 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -77,7 +77,7 @@ The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. <li><b>KeepAlive.Interval</b> <number><br> -The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime. +The interval between subsequent keepalive probes, regardless of what the connection has exchanged in the meantime. The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. @@ -96,12 +96,28 @@ the message was received from. Binds specified ruleset to next server defined. <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> <li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. </li> <li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting burst in number of messages. Default is 10,000. +<li><b>compression.mode</b><i>mode</i><br> +<i>mode</i> is one of "none" or "stream:always". +It is the counterpart to the compression modes set in +<a href="omfile.html">omfile</a>. +Please see it's documentation for details. </li> </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imrelp.html b/doc/imrelp.html index f7fcc4b3..66e8b054 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -28,14 +28,84 @@ nits outlined above, is a much more reliable solution than plain tcp syslog and so it is highly suggested to use RELP instead of plain tcp. Clients send messages to the RELP server via omrelp.</p> -<p><b>Configuration Directives</b>:</p> +<p><b>Module Parameters</b>:</p> +<ul> + <li><b>Ruleset</b> <name></br> + Binds the specified ruleset to <b>all</b> RELP listeners. +</ul> +<p><b>Input Parameters</b>:</p> <ul> <li><b>Port</b> <port><br> Starts a RELP server on selected port</li> +<li><b>tls</b> (not mandatory, values "on","off", default "off")<br> +If set to "on", the RELP connection will be encrypted by TLS, +so that the data is protected against observers. Please note +that both the client and the server must have set TLS to +either "on" or "off". Other combinations lead to unpredictable +results. +</li> +<li><b>tls.compression</b> (not mandatory, values "on","off", default "off")<br> +The controls if the TLS stream should be compressed (zipped). While this +increases CPU use, the network bandwidth should be reduced. Note that +typical text-based log records usually compress rather well. +</li> +<li><b>tls.dhbits</b> (not mandatory, integer)<br> +This setting controls how many bits are used for Diffie-Hellman key +generation. If not set, the librelp default is used. For secrity +reasons, at least 1024 bits should be used. Please note that the number +of bits must be supported by GnuTLS. If an invalid number is given, rsyslog +will report an error when the listener is started. We do this to be transparent +to changes/upgrades in GnuTLS (to check at config processing time, we would need +to hardcode the supported bits and keep them in sync with GnuTLS - this is +even impossible when custom GnuTLS changes are made...). +</li> +<li><b>tls.permittedPeer</b> peer</br> +Places access restrictions on this listener. Only peers which +have been listed in this parameter may connect. The validation +bases on the certificate the remote peer presents.<br> +The <i>peer</i> parameter lists permitted certificate +fingerprints. Note that it is an array parameter, so either +a single or multiple fingerprints can be listed. When a +non-permitted peer connects, the refusal is logged together +with it's fingerprint. So if the administrator knows this was +a valid request, he can simple add the fingerprint by copy and +paste from the logfile to rsyslog.conf. +<br>To specify multiple fingerprints, just enclose them +in braces like this: +<br>tls.permittedPeer=["SHA1:...1", "SHA1:....2"] +<br>To specify just a single peer, you can either +specify the string directly or enclose it in braces. +</li> +<li><b>tls.authMode</b> mode</br> +Sets the mode used for mutual authentication. Supported values are +either "<i>fingerprint</i>" or "<i>name"</i>. +<br>Fingerprint mode basically is what SSH +does. It does not require a full PKI to be present, instead self-signed +certs can be used on all peers. Even if a CA certificate is given, the +validity of the peer cert is NOT verified against it. Only the +certificate fingerprint counts. +<br>In "name" mode, certificate validation happens. Here, the matching +is done against the certificate's subjectAltName and, as a fallback, +the subject common name. If the certificate contains multiple names, +a match on any one of these names is considered good and permits the +peer to talk to rsyslog. +<li><b>tls.prioritystring</b> (not mandatory, string)<br> +This parameter permits to specify the so-called "priority string" to +GnuTLS. This string gives complete control over all crypto parameters, +including compression setting. For this reason, when the prioritystring +is specified, the "tls.compression" parameter has no effect and is +ignored. +<br>Full information about how to construct a priority string can be +found in the GnuTLS manual. At the time of this writing, this +information was contained in +<a href="http://gnutls.org/manual/html_node/Priority-Strings.html">section 6.10 of the GnuTLS manual</a>. +<br><b>Note: this is an expert parameter.</b> Do not use if you do +not exactly know what you are doing. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>ruleset can only be bound via legacy configuration format</li> +<li>see description</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> @@ -45,20 +115,19 @@ not specific ones. This is due to a currently existing limitation in librelp. <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> </p> -<textarea rows="15" cols="60">module(load="imrelp") # needs to be done just once +<textarea rows="5" cols="60">module(load="imrelp") # needs to be done just once input(type="imrelp" port="20514") </textarea> <p><b>Legacy Configuration Directives</b>:</p> <ul> <li>InputRELPServerBindRuleset <name> (available in 6.3.6+)</br> -Binds the specified ruleset to all RELP listeners. +equivalent to: RuleSet <li>InputRELPServerRun <port><br> equivalent to: Port</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>see description</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> @@ -68,14 +137,14 @@ not specific ones. This is due to a currently existing limitation in librelp. <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> </p> -<textarea rows="15" cols="60">$ModLoad imrelp # needs to be done just once +<textarea rows="5" cols="60">$ModLoad imrelp # needs to be done just once $InputRELPServerRun 20514 </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imsolaris.html b/doc/imsolaris.html index ce0e7e84..cc9a745f 100644 --- a/doc/imsolaris.html +++ b/doc/imsolaris.html @@ -18,7 +18,7 @@ is no special kernel input device. Instead, both kernel messages as well as messages emitted via syslog() are received from a single source. <p>This module obeys the Solaris door() mechanism to detect a running syslogd instance. As such, only one can be active at one time. If it detects another -active intance at startup, the module disables itself, but rsyslog will +active instance at startup, the module disables itself, but rsyslog will continue to run. <p><b>Configuration Directives</b>:</p> <ul> diff --git a/doc/imtcp.html b/doc/imtcp.html index b9f0b056..1323252a 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -14,7 +14,7 @@ <p><b>Multi-Ruleset Support: </b>since 4.5.0 and 5.1.1 <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via TCP. -Encryption is natively provided by selecting the approprioate network stream driver and +Encryption is natively provided by selecting the appropriate network stream driver and can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> (an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> @@ -63,13 +63,22 @@ the sender is throttled a bit when the queue becomes near-full. This is done in to preserve some queue space for inputs that can not throttle (like UDP), but it may have some undesired effect in some configurations. Still, we consider this as a useful setting and thus it is the default. To turn the handling off, simply -configure that explicitely. +configure that explicitly. </li> <li><b>MaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> <li><b>MaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>StreamDriver.Name</b> <name><br> +Sets the driver name and overrides the system default. This enables e.g. to +define a system default of "gtls" (for TLS transmission) and override it to +"ptcp" (traditional unprotected plain tcp). Note, however, that this is a module +parameter. Currently, imtcp does not support mixed TLS/non-TLS listeners. If this +is desired, use imtcp for TLS, and imptcp for non-TLS. However, setting the +stream driver enables you to use e.g. plain tcp for the imtcp listeners while +setting the system default to TLS, which is then used by multiple forwarding (omfwd) +actions. <li><b>StreamDriver.Mode</b> <number><br> -Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> +Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specific.</li> <li><b>StreamDriver.AuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> <li><b>PermittedPeer</b> <id-string><br> @@ -80,7 +89,7 @@ AuthMode and <a href="netstream.html">network stream driver</a>. Permitted <br>Single peer: PermittedPeer="127.0.0.1" <br>Array of peers: PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."]</li> </ul> -<p><b>Action Directives</b>:</p> +<p><b>Input Parameters</b>:</p> <ul> <li><b>Port</b> <port><br> Starts a TCP server on selected port</li> @@ -96,6 +105,17 @@ activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> <li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. diff --git a/doc/imudp.html b/doc/imudp.html index a8dbca31..b57eb713 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -29,13 +29,42 @@ only activated if messages come in at a very fast rate, so doing less frequent time calls should usually be acceptable. The default value is two, because we have seen that even without optimization the kernel often returns twice the identical time. You can set this value as high as you like, but do so at your own risk. The higher -the value, the less precise the timestamp. +the value, the less precise the timestamp.<br> +<b>Note:</b> the timeRequery is done based on executed system calls (<b>not</b> +messages received). So when batch +sizes are used, multiple messages are received with one system call. All of these +messages always receive the same timestamp, as they are effectively received at the +same time. When there is very high traffic and successive system calls immediately +return the next batch of messages, the time requery logic kicks in, which means that +by default time is only queried for every second batch. Again, this should not cause +a too-much deviation as it requires messages to come in very rapidly. However, we advise +not to set the "timeRequery" parameter to a large value (larger than 10) if input +batches are used. <li><b>SchedulingPolicy</b> <rr/fifo/other><br> Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time processing under Linux (and thus reduce chance of packet loss). <li><b>SchedulingPriority</b> <number><br> Scheduling priority to use. +<li><b>batchSize</b> <number><br> +This parameter is only meaningful if the system support recvmmsg() (newer Linux +OSs do this). The parameter is silently ignored if the system does not support +it. If supported, it sets the maximum number of UDP messages that can be obtained +with a single OS call. For systems with high UDP traffic, a relatively high batch +size can reduce system overhead and improve performance. However, this parameter +should not be overdone. For each buffer, max message size bytes are statically +required. Also, a too-high number leads to reduced efficiency, as some structures +need to be completely initialized before the OS call is done. We would suggest to not +set it above a value of 128, except if experimental results show that this is useful. +<li><b>threads</b> <number> (default 1), available since 7.5.5<br> +Number of worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +threads (but not more than there are CPUs/Cores) can help improving +performance and avoiding message loss. Note that with too many threads, performance +can suffer. +There is a hard upper limit on the number of threads that can be defined. +Currently, this limit is set to 32. It may increase in the future when massive +multicore processors become available. </ul> <p><b>Input Parameters</b>:</p> <ul> @@ -75,7 +104,36 @@ are defined for a single input and each of the inputnames shall be unique. Note that there currently is no differentiation between IPv4/v6 listeners on the same port. </li> +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> +<li><b>rcvbufSize</b> [size] - (available since 7.5.3) +This request a socket receive buffer of specific size from the operating system. +It is an expert parameter, which should only be changed for a good reason. Note that +setting this parameter disables Linux auto-tuning, which usually works pretty well. +The default value is 0, which means "keep the OS buffer size unchanged". This is a size +value. So in addition to pure integer values, sizes like "256k", "1m" and the like can +be specified. Note that setting very large sizes may require root or other special +privileges. Also note that the OS may slightly adjust the value or shrink it to a +system-set max value if the user is not sufficiently privileged. Technically, this +parameter will result in a setsockopt() call with SO_RCVBUF (and SO_RCVBUFFORCE if it +is available). </ul> +<p><b>See Also</b> +<ul> +<li>Description of +<a href="http://www.rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a> +This also describes all imudp counters. +</ul> +<p> <b>Caveats/Known Bugs:</b> <ul> <li>Scheduling parameters are set <b>after</b> privileges have been dropped. @@ -91,6 +149,13 @@ user account. input(type="imudp" port="514") </textarea> +<p>The following sample is mostly equivalent to the first one, but request a +larger rcvuf size. Note that 1m most probably will not be honored by the OS +until the user is sufficiently privileged.</p> +<textarea rows="3" cols="60">module(load="imudp") # needs to be done just once +input(type="imudp" port="514" rcvbufSize="1m") +</textarea> + <p>In the next example, we set up three listeners at ports 10514, 10515 and 10516 and assign a listner name of "udp" to it, followed by the port number: </p> @@ -110,7 +175,6 @@ input(type="imudp" port=["10514","10515","10516"] inputname="" inputname.appendPort="on") </textarea> - <p><b>Legacy Configuration Directives</b>:</p> <p>Multiple receivers may be configured by specifying $UDPServerRun multiple times. @@ -129,7 +193,7 @@ equivalent to: SchedulingPolicy <li>$IMUDPSchedulingPriority <number> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPriority </ul> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>This sets up an UPD server on port 514:<br> </p> <textarea rows="3" cols="60">$ModLoad imudp # needs to be done just once diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 123771fa..02b2f187 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -42,7 +42,7 @@ severity and configure things accordingly. To turn off rate limiting, set the interval to zero. <p><b>Unix log sockets can be flow-controlled.</b> That is, if processing queues fill up, the unix socket reader is blocked for a short while. This may be useful to prevent overruning -the queues (which may cause exessive disk-io where it actually would not be needed). However, +the queues (which may cause excessive disk-io where it actually would not be needed). However, flow-controlling a log socket (and especially the system log socket) can lead to a very unresponsive system. As such, flow control is disabled by default. That means any log records are places as quickly as possible into the processing queues. If you would like to have @@ -125,7 +125,7 @@ module documentation for a more in-depth description. to the next socket.</li> <li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number -of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen +of seconds (5 recommended) to activate rate-limiting. The default of 0 has been chosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> diff --git a/doc/licensing.html b/doc/licensing.html index 93a50930..0a4ab1f4 100644 --- a/doc/licensing.html +++ b/doc/licensing.html @@ -22,7 +22,7 @@ real details, check source files and the files COPYING and COPYING.LESSER inside <p>Each of these components can be thought of as individual projects. In fact, some of the plugins have different main authors than the rest of the rsyslog package. All of these components are currently put together into a single "rsyslog" package (tarball) for -convinience: this makes it easier to distribute a consistent version where everything +convenience: this makes it easier to distribute a consistent version where everything is included (and in the right versions) to build a full system. Platform package maintainers in general take the overall package and split off the individual components, so that users can install only what they need. In source installations, this can be done via the diff --git a/doc/log_rotation_fix_size.html b/doc/log_rotation_fix_size.html index 51edf033..7cdecc6c 100644 --- a/doc/log_rotation_fix_size.html +++ b/doc/log_rotation_fix_size.html @@ -31,7 +31,7 @@ Channels to achieve this. Putting the following directive</p> <p><pre> # start log rotation via outchannel -# outchannel definiation +# outchannel definition $outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script # activate the channel and log everything to it *.* :omfile:$log_rotation @@ -49,13 +49,13 @@ mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1 <p>This moves the original log to a kind of backup log file. After the action was successfully performed rsyslog creates a new /var/log/log_rotation.log -file and fill it up with new logs. So the latest logs are always in log_roatation.log.</p> +file and fill it up with new logs. So the latest logs are always in log_rotation.log.</p> <h2>Conclusion</h2> <p>With this approach two files for logging are used, each with a maximum size of 50 MB. So we can say we have successfully configured a log rotation which satisfies our requirement. -We keep the logs at a fixed-size level of100 MB.</p> +We keep the logs at a fixed-size level of 100 MB.</p> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html index d72810f1..b7fcff04 100644 --- a/doc/lookup_tables.html +++ b/doc/lookup_tables.html @@ -104,7 +104,7 @@ lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") <h3>lookup() Function</h3> <p>This function is used to actually do the table lookup. Format: <pre> -lookup_table("name", indexvalue) +lookup("name", indexvalue) </pre> <h4>Parameters</h4> <ul> diff --git a/doc/manual.html b/doc/manual.html index a8477bcd..cce3863f 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ professional services</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 7.4.6 (v7.4-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 8.1.0 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -27,16 +27,18 @@ want to lend us a helping hand. </b>It doesn't require a lot of time - even a single mouse click helps. Learn <a href="how2help.html">how to help the rsyslog project</a>. Due to popular demand, there is now a <a href="rsyslog_ng_comparison.html">side-by-side comparison between rsyslog and syslog-ng</a>.</p> -<p>If you are upgrading from rsyslog v2 or stock sysklogd, -<a href="v3compatibility.html">be sure to read the rsyslog v3 compatibility notes</a>, -and if you are upgrading from v3, read the -<a href="v4compatibility.html">rsyslog v4 compatibility notes</a>, -if you upgrade from v4, read the -<a href="v5compatibility.html">rsyslog v5 compatibility notes</a>, and -if you upgrade from v5, read the -<a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. +If you upgrade from v7, read the +<a href="v8compatibility.html">rsyslog v8 compatibility notes</a>. if you upgrade from v6, read the <a href="v7compatibility.html">rsyslog v7 compatibility notes</a>. +if you upgrade from v5, read the +<a href="v6compatibility.html">rsyslog v6 compatibility notes</a>. +if you upgrade from v4, read the +<a href="v5compatibility.html">rsyslog v5 compatibility notes</a>. +if you upgrade from v3, read the +<a href="v4compatibility.html">rsyslog v4 compatibility notes</a>. +<p>If you are upgrading from rsyslog v2 or stock sysklogd, +<a href="v3compatibility.html">be sure to read the rsyslog v3 compatibility notes</a>, <p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> <p><b>Follow the links below for the</b></p> @@ -44,7 +46,7 @@ if you do not read the doc, but doing so will definitely improve your experience <li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> <li><a href="rsyslog_conf.html">configuration file format (rsyslog.conf)</a></li> <li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> -<li> <a href="property_replacer.html">property replacer, an important core component</a></li> +<li><a href="property_replacer.html">property replacer, an important core component</a></li> <li><a href="bugs.html">rsyslog bug list</a></li> <li><a href="messageparser.html">understanding rsyslog message parsers</a></li> <li><a href="generic_design.html">backgrounder on generic syslog application design</a></li> diff --git a/doc/messageparser.html b/doc/messageparser.html index 370db59f..d22021dd 100644 --- a/doc/messageparser.html +++ b/doc/messageparser.html @@ -23,7 +23,7 @@ the rsyslog code). like input and output modules). That means that new message parsers can be added without modifying the rsyslog core, even without contributing something back to the project. -<p>But that doesn't answer what a message parser really is. What does ist mean to "parse a +<p>But that doesn't answer what a message parser really is. What does it mean to "parse a message" and, maybe more importantly, what is a message? To answer these questions correctly, we need to dig down into the relevant standards. <a href="http://tools.ietf.org/html/rfc5424">RFC5424</a> specifies a layered architecture @@ -49,7 +49,7 @@ reason) a single message into two and encapsulates these into two frames, there a message parser could undo that. <p>A typical example may be a multi-line message: let's assume some originator has generated a message for the format "A\nB" (where \n means LF). If that message is being transmitted -via plain tcp syslog, the frame delimiter is LF. So the sender will delimite the frame with +via plain tcp syslog, the frame delimiter is LF. So the sender will delimit the frame with LF, but otherwise send the message unmodified onto the wire (because that is how things are -unfortunately- done in plain tcp syslog...). So wire will see "A\nB\n". When this arrives at the receiver, the transport layer will undo the framing. When it sees the LF @@ -58,7 +58,7 @@ the receive will extract one complete message A and one complete message B, not that they once were both part of a large multi-line message. These two messages are then passed to the upper layers, where the message parsers receive them and extract information. However, the message parsers never know (or even have a chance to see) that A and B -belonged together. Even further, in rsyslog there is no guarnatee that A will be parsed +belonged together. Even further, in rsyslog there is no guarantee that A will be parsed before B - concurrent operations may cause the reverse order (and do so very validly). <p>The important lesson is: <b>message parsers can not be used to fix a broken framing</b>. You need a full protocol implementation to do that, what is the domain of input and @@ -73,10 +73,10 @@ the real-world evil that you can usually see. So I won't repeat that here. But i real problem is not the framing, but how to make malformed messages well-looking. <p><b>This is what message parsers permit you to do: take a (well-known) malformed message, parse it according to its semantics and generate perfectly valid internal message representations -from it.</b> So as long as messages are consistenly in the same wrong format (and they usually +from it.</b> So as long as messages are consistently in the same wrong format (and they usually are!), a message parser can look at that format, parse it, and make the message processable just -like it were wellformed in the first place. Plus, one can abuse the interface to do some other -"intersting" tricks, but that would take us to far. +like it were well formed in the first place. Plus, one can abuse the interface to do some other +"interesting" tricks, but that would take us to far. <p>While this functionality may not sound exciting, it actually solves a very big issue (that you only really understand if you have managed a system with various different syslog sources). Note that we were often able to process malformed messages in the past with the help of the @@ -113,15 +113,15 @@ interface probably extended, to support generic filter modules. These would need to the root of the parser chain. As mentioned, the current system already supports this. <p>The position inside the parser chain can be thought of as a priority: parser sitting earlier in the chain take precedence over those sitting later in it. So more specific -parser should go ealier in the chain. A good example of how this works is the default parser +parser should go earlier in the chain. A good example of how this works is the default parser set provided by rsyslog: rsyslog.rfc5424 and rsyslog.rfc3164, each one parses according to the rfc that has named it. RFC5424 was designed to be distinguishable from RFC3164 message by the sequence "1 " immediately after the so-called PRI-part (don't worry about these words, it is -sufficient if you understand there is a well-defined sequence used to indentify RFC5424 +sufficient if you understand there is a well-defined sequence used to identify RFC5424 messages). In contrary, RFC3164 actually permits everything as a valid message. Thus the RFC3164 parser will always parse a message, sometimes with quite unexpected outcome (there is a lot of guesswork involved in that parser, which unfortunately is unavoidable due to -existing techology limits). So the default parser chain is to try the RFC5424 parser first +existing technology limits). So the default parser chain is to try the RFC5424 parser first and after it the RFC3164 parser. If we have a 5424-formatted message, that parser will identify and parse it and the rsyslog engine will stop processing. But if we receive a legacy syslog message, the RFC5424 will detect that it can not parse it, return this status @@ -139,16 +139,16 @@ case, rsyslog has no other choice than to discard the message. If it does so, it a warning message, but only in the first 1,000 incidents. This limit is a safety measure against message-loops, which otherwise could quickly result from a parser chain misconfiguration. <b>If you do not tolerate loss of unparsable messages, you must ensure -that each message can be parsed.</b> You can easily achive this by always using the +that each message can be parsed.</b> You can easily achieve this by always using the "rsyslog-rfc3164" parser as the <i>last</i> parser inside parser chains. That may result in invalid parsing, but you will have a chance to see the invalid message (in debug mode, a warning message will be written to the debug log each time a message is dropped due to inability to parse it). <h3>Where are parser chains used?</h3> <p>We now know what parser chains are and how they operate. The question is now how many -parser chains can be active and how it is decicded which parser chain is used on which message. +parser chains can be active and how it is decided which parser chain is used on which message. This is controlled via <a href="multi_ruleset.html">rsyslog's rulesets</a>. In short, multiple -rulesets can be defined and there always exist at least one ruleset (for specifcs, follow +rulesets can be defined and there always exist at least one ruleset (for specifics, follow the <a href="multi_ruleset.html">link</a>). A parser chain is bound to a specific ruleset. This is done by virtue of defining parsers via the <a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive (for specifics, @@ -161,22 +161,22 @@ is added to the end of the (initially empty) ruleset's parser chain. <p>The correct answer is: generally yes, but it depends. First of all, remember that input modules (and specific listeners) may be bound to specific rulesets. As parser chains "reside" in rulesets, binding to a ruleset also binds to the parser chain that is bound to that ruleset. -As a number one prequisite, the input module must support binding to different rulesets. Not +As a number one prerequisite, the input module must support binding to different rulesets. Not all do, but their number is growing. For example, the important <a href="imudp.html">imudp</a> and <a href="imtcp.html">imtcp</a> input modules support that functionality. Those that do not (for example <a href="im3195">im3195</a>) can only utilize the default ruleset and thus the parser chain defined in that ruleset. <p>If you do not know if the input module in question supports ruleset binding, check -its documentation page. Those that support it have the requiered directives. +its documentation page. Those that support it have the required directives. <p>Note that it is currently under evaluation if rsyslog will support binding parser chains to specific inputs directly, without depending on the ruleset. There are some concerns that this may not be necessary but adds considerable complexity to the configuration. So this may or may not be possible in the future. In any case, if we decide to add it, input modules need to support it, so this functionality would require some time to implement. -<p>The coockbook recipe for using different parsers for different devices is given +<p>The cookbook recipe for using different parsers for different devices is given as an actual in-depth example in the <a href="rscon1_rulesetsparser.html">$RulesetParser</a> -configuration directive doc page. In short, it is acomplished by defining specific rulesets -for the required parser chains, definining different listener ports for each of the devices +configuration directive doc page. In short, it is accomplished by defining specific rulesets +for the required parser chains, defining different listener ports for each of the devices with different format and binding these listeners to the correct ruleset (and thus parser chains). Using that approach, a variety of different message formats can be supported via a single rsyslog instance. @@ -185,19 +185,19 @@ via a single rsyslog instance. <p>As of this writing, there exist only two message parsers, one for RFC5424 format and one for legacy syslog (loosely described in <a href="http://tools.ietf.org/html/rfc3164">RFC3164</a>). These parsers are built-in and -must not be explicitely loaded. However, message parsers can be added with relative ease +must not be explicitly loaded. However, message parsers can be added with relative ease by anyone knowing to code in C. Then, they can be loaded via $ModLoad just like any other loadable module. It is expected that the rsyslog project will be contributed additional message parsers over time, so that at some point there hopefully is a rich choice of them (I intend to add a browsable repository as soon as new parsers pop up). <h3>How to write a message parser?</h3> -<p>As a prequisite, you need to know the exact format that the device is sending. Then, you need +<p>As a prerequisite, you need to know the exact format that the device is sending. Then, you need moderate C coding skills, and a little bit of rsyslog internals. I guess the rsyslog specific part should not be that hard, as almost all information can be gained from the existing parsers. They are rather simple in structure and can be found under the "./tools" directory. They are named pmrfc3164.c and pmrfc5424.c. You need to follow the usual loadable module guidelines. It is my expectation that writing a parser should typically not take longer than a single -day, with maybe a day more to get aquainted with rsyslog. Of course, I am not sure if the number +day, with maybe a day more to get acquainted with rsyslog. Of course, I am not sure if the number is actually right. <p>If you can not program or have no time to do it, Adiscon can also write a message parser for you as @@ -209,7 +209,7 @@ provide a fast and efficient solution for this problem. Different parsers can be different devices, and they all convert message information into rsyslog's well-defined internal format. Message parsers were first introduced in rsyslog 5.3.4 and also offer some interesting ideas that may be explored in the future - up to full message normalization -capabilities. It is strongly recommended that anyone with a heterogenous environment take +capabilities. It is strongly recommended that anyone with a heterogeneous environment take a look at message parser capabilities. <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual diff --git a/doc/mmanon.html b/doc/mmanon.html index e14d75cf..984872bf 100644 --- a/doc/mmanon.html +++ b/doc/mmanon.html @@ -60,6 +60,12 @@ part is always zero-filled and replacementChar is of no use. If it is specified, an error message is emitted and the parameter ignored. </ul> +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/">Howto anonymize messages that go to specific files</a> +</ul> + + <p><b>Caveats/Known Bugs:</b> <ul> <li><b>only IPv4</b> is supported diff --git a/doc/mmcount.html b/doc/mmcount.html new file mode 100644 index 00000000..1d06340d --- /dev/null +++ b/doc/mmcount.html @@ -0,0 +1,58 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmcount</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>mmcount</h1> +<p><b>Module Name: mmcount</b></p> +<p><b>Author: </b>Bala.FA <barumuga@redhat.com></p> +<p><b>Status: </b>Non project-supported module - contact author +or rsyslog mailing list for questions +<p><b>Available since</b>: 7.5.0</p> +<p><b>Description</b>:</p> +<p> +<pre> + mmcount: message modification plugin which counts messages + + This module provides the capability to count log messages by severity + or json property of given app-name. The count value is added into the + log message as json property named 'mmcount' + + Example usage of the module in the configuration file + + module(load="mmcount") + + # count each severity of appname gluster + action(type="mmcount" appname="gluster") + + # count each value of gf_code of appname gluster + action(type="mmcount" appname="glusterd" key="!gf_code") + + # count value 9999 of gf_code of appname gluster + action(type="mmcount" appname="glusterfsd" key="!gf_code" value="9999") + + # send email for every 50th mmcount + if $app-name == 'glusterfsd' and $!mmcount <> 0 and $!mmcount % 50 == 0 then { + $ActionMailSMTPServer smtp.example.com + $ActionMailFrom rsyslog@example.com + $ActionMailTo glusteradmin@example.com + $template mailSubject,"50th message of gf_code=9999 on %hostname%" + $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" + $ActionMailSubject mailSubject + $ActionExecOnlyOnceEveryInterval 30 + :ommail:;RSYSLOG_SyslogProtocol23Format + } +</pre> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmfields.html b/doc/mmfields.html new file mode 100644 index 00000000..885d6bca --- /dev/null +++ b/doc/mmfields.html @@ -0,0 +1,91 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Field Extraction Module (mmfields)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Fields Extraction Module (mmfields)</h1> +<p><b>Module Name: mmfields</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.1</p> +<p><b>Description</b>:</p> +<p>The mmfield module permits to extract fields. It is an alternate to using +the property replacer field extraction capabilities. In contrast to the +property replacer, all fields are extracted as once and stored inside the +structured data part (more precisely: they become Lumberjack [JSON] properties). +<p>Using this module is of special advantage if a field-based log format +is to be processed, like for example CEF <b>and</b> and either a large +number of fields is needed or a specific field is used multiple times +inside filters. In these scenarios, mmfields potentially offers better +performance than the property replacer of the RainerScript field extraction +method. The reason is that mmfields extracts all fields as one big sweep, +whereas the other methods extract fields individually, which requires +multiple passes through the same data. On the other hand, adding field +content to the rsyslog property dictionary also has some overhead, +so for high-performance use cases it is suggested to do some performance +testing before finally deciding which method to use. This is most important +if only a smaller subset of the fields is actually needed. +<p>In any case, mmfields provides a very handy and easy to use way to +parse structured data into a it's individual data items. Again, a primiary +use case was support for CEF (Common Event Format), which is made +extremely easy to do with this module. +<p>This module is implemented via the action interface. Thus it +can be conditionally used depending on some prequisites. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>separator</b> - separatorChar (default ',')<br> +This is the character used to separate fields. Currently, only a single +character is permitted, while the RainerScript method permits to +specify multi-character separator strings. For CEF, this is not required. +If there is actual need to support multi-character separator strings, +support can relatively easy be added. It is suggested to request it on the +rsyslog mailing list, together with the use case - we intend to add +functionality only if there is a real use case behind the request +(in the past we too-often implemented things that actually never got used). +<br>The fields are named f<i>nbr</i>, where <i>nbr</i> is the field number +starting with one and being incremented for each field. +<li><b>jsonRoot</b> - path (default "!")<br> +This parameters specifies into which json path the extracted fields shall +be written. The default is to use the json root object itself. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>Currently none. +</ul> + +<p><b>Samples:</b></p> +<p>This is a very simple use case where each message is +parsed. The default separator character of comma is being used. +<p><textarea rows="5" cols="60">module(load="mmfields") +template(name="ftpl" type=string string="%$!%\n") +action(type="omfields") +action(type="omfile" file="/path/to/logfile" template="ftpl") +</textarea> + +<p>The following sample is similar to the previous one, but +this time the colon is used as separator and data is written +into the "$!mmfields" json path. +<p><textarea rows="5" cols="60">module(load="mmfields") +template(name="ftpl" type=string string="%$!%\n") +action(type="omfields" separator=":" jsonRoot="!mmfields") +action(type="omfile" file="/path/to/logfile" template="ftpl") +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmjsonparse.html b/doc/mmjsonparse.html index c2c862d7..20a44f7a 100644 --- a/doc/mmjsonparse.html +++ b/doc/mmjsonparse.html @@ -15,6 +15,16 @@ that follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked and, if present, the JSON-encoded structured message content is parsed. The properties are than available as original message properties. </p> +<p>The "CEE cookie" is the character squence "@cee:" which must prepend the +actual JSON. Note that the JSON must be valid and MUST NOT be followed by +any non-JSON message. If either of these conditions is not true, mmjsonparse +will <b>not</b> parse the associated JSON. This is based on the cookie +definition used in CEE/project lumberjack and is meant to aid against +an errornous detection of a message as being CEE where it is not. +<p>This also means that mmjsonparse currently is NOT a generic JSON +parser that picks up JSON from whereever it may occur in the message. This +is intentional, but future versions may support config parameters to +relax the format requirements. <p><b>Action specific Configuration Directives</b>:</p> <p>currently none <ul> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html index 81100235..fc6ec6d2 100644 --- a/doc/mmnormalize.html +++ b/doc/mmnormalize.html @@ -17,7 +17,7 @@ a normal form. This is done so quickly, that it should be possible to normalize events in realtime. <p>This module is implemented via the output module interface. This means that mmnormalize should be called just like an action. After it has been called, -the normalized message properties are avaialable and can be accessed. These properties +the normalized message properties are available and can be accessed. These properties are called the "CEE/lumberjack" properties, because liblognorm creates a format that is inspired by the CEE/lumberjack approach. <p><b>Please note:</b> CEE/lumberjack properties are different from regular properties. diff --git a/doc/mmpstrucdata.html b/doc/mmpstrucdata.html new file mode 100644 index 00000000..b4003062 --- /dev/null +++ b/doc/mmpstrucdata.html @@ -0,0 +1,59 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>RFC5424 structured data parsing module (mmpstrucdata)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>RFC5424 structured data parsing module (mmpstrucdata)</h1> +<p><b>Module Name: mmpstrucdata</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.4</p> +<p><b>Description</b>:</p> +<p>The mmpstrucdata parses RFC5424 structured data into the message +json variable tree. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>jsonRoot</b> - default "!"<br> +Specifies into which json container the data shall be parsed to. +</ul> + +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/">Howto anonymize messages that go to specific files</a> +</ul> + + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>this module is currently experimental; feedback is appreciated +<li>structured data with duplicate SD-IDs and SD-PARAMS is not + properly processed +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we parse the message and emit all json variable to a file +with the message anonymized. Note that once mmpstrucdata has run, access to the +original message is no longer possible (execept if stored in user +variables before anonymization). +<p><textarea rows="5" cols="60">module(load="mmpstrucdata") +action(type="mmpstrucdata") +template(name="jsondump" type="string" string="%msg%: %$!%\n") +action(type="omfile" file="/path/to/log" template="jsondump") +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmrfc5424addhmac.html b/doc/mmrfc5424addhmac.html new file mode 100644 index 00000000..a54908c4 --- /dev/null +++ b/doc/mmrfc5424addhmac.html @@ -0,0 +1,88 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmrfc5424addhmac</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>mmrfc5424addhmac</h1> +<p><b>Module Name: mmrfc5424addhmac</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.6</p> +<p><b>Description</b>:</p> +<p>This module adds a hmac to RFC5424 structured data if not already present. +This is a custom module and uses openssl as requested by the sponsor. +This works exclusively for RFC5424 formatted messages; all others are ignored. +<p>If both <a href="mmpstrucdata.html">mmpstrucdata</a> +and mmrfc5424addhmac are to be used, the recommended calling sequence is +<ol> +<li>mmrfc5424addhmac +<li>mmpstrucdata +</ol> +with that sequence, the generated hash will become available for mmpstrucdata. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>key</b><br> +The "key" (string) to be used to generate the hmac. +<li><b>hashfunction</b><br> +An openssl hash function name for the function to be used. This is passed +on to openssl, so see the openssl list of supported function names. +<li><b>sd_id</b><br> +The RFC5424 structured data ID to be used by this module. This is the +SD-ID that will be added. Note that nothing is added if this SD-ID +is already present. +</ul> + +<p><b>Verification method</b> +<p>rsyslog does not contain any tools to verify a log file (this was not +part of the custom project). So you need to write your own verifier. +<p>When writing the verifier, keep in mind that the log file contains messages +with the hash SD-ID included. For obvious reasons, this SD-ID was not present when +the hash was created. So before the actual verification is done, this SD-ID must be +removed, and the remaining (original) message be verified. Also, it is important to +note that the output template must write the exact same message format that was +received. Otherwise, a verification failure will obviously occur - and must +so, because the message content actually was altered. +<p>So in a more formal description, verification of a message m can be done as follows: +<ol> +<li>let m' be m with the configured SD-ID removed (everything between []). Otherwise, +m' must be an exact duplicate of m. +<li>call openssl's HMAC function as follows:<br> +<code>HMAC(hashfunction, key, len(key), m', len(m'), hash, &hashlen);</code></br> +Where hashfunction and key are the configured values and hash is an output +buffer for the hash. +<li>let h be the extracted hash value obtained from m within the relevant SD-ID. Be sure to convert the hex string back to the actual byte values. +<li>now compare hash and h under consideration of the sizes. If these values match +the verification succeeds, otherwise the message was modified. +</ol> +<p>If you neeed help implementing a verifier function or want to sponsor development +of a verification tool, please simply email +<a href="sales@adiscon.com">sales@adiscon.com</a> for a quote. + +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/how-to-add-a-hmac-to-rfc5424-structured-data-messages/">How to +add a HMAC to RFC5424 messages</a> +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>none +</ul> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmsequence.html b/doc/mmsequence.html new file mode 100644 index 00000000..75ac57b4 --- /dev/null +++ b/doc/mmsequence.html @@ -0,0 +1,148 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmsequence</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Number generator and counter module (mmsequence)</h1> +<p><b>Module Name: mmsequence</b></p> +<p><b>Author: </b>Pavel Levshin <pavel@levshin.spb.ru></p> +<p><b>Status: </b>Non project-supported module - contact author +or rsyslog mailing list for questions</p> +<p><b>Available since</b>: 7.5.6</p> +<p><b>Description</b>:</p> +<p>This module generates numeric sequences of different kinds. It can be used +to count messages up to a limit and to number them. It can generate random +numbers in a given range.</p> + +<p>This module is implemented via the output module interface, so it is +called just as an action. The number generated is stored in a variable.</p> +<p> </p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>mode</b> "random" or "instance" or "key" + <p>Specifies mode of the action. In "random" mode, the module + generates uniformly distributed integer numbers in a range defined + by "from" and "to".</p> + + <p>In "instance" mode, which is default, the action produces a counter + in range [from, to). This counter is specific to this action instance.</p> + + <p>In "key" mode, the counter can be shared between multiple instances. + This counter is identified by a name, which is defined with "key" + parameter.</p> +</li> +<li><b>from</b> [non-negative integer], default "0" + <p>Starting value for counters and lower margin for random generator.</p> +</li> +<li><b>to</b> [positive integer], default "INT_MAX" + <p>Upper margin for all sequences. Note that this margin is not + inclusive. When next value for a counter is equal or greater than + this parameter, the counter resets to the starting value.</p> +</li> +<li><b>step</b> [non-negative integer], default "1" + <p>Increment for counters. If step is "0", it can be used to fetch + current value without modification. The latter not applies to + "random" mode. This is useful in "key" mode or to get constant + values in "instance" mode.</p> +</li> +<li><b>key</b> [word], default "" + <p>Name of the global counter which is used in this action.</p> +</li> +<li><b>var</b> [word], default "$!mmsequence" + <p>Name of the variable where the number will be stored. + Should start with "$".</p> +</li> +</ul> + + +<p><b>Sample</b>:</p> +<pre> +# load balance +Ruleset( + name="logd" + queue.workerthreads="5" + ){ + + Action( + type="mmsequence" + mode="instance" + from="0" + to="2" + var="$.seq" + ) + + if $.seq == "0" then { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } else { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } + + # output logic here +} + # generate random numbers + action( + type="mmsequence" + mode="random" + to="100" + var="$!rndz" + ) + # count from 0 to 99 + action( + type="mmsequence" + mode="instance" + to="100" + var="$!cnt1" + ) + # the same as before but the counter is global + action( + type="mmsequence" + mode="key" + key="key1" + to="100" + var="$!cnt2" + ) + # count specific messages but place the counter in every message + if $msg contains "txt" then + action( + type="mmsequence" + mode="key" + to="100" + var="$!cnt3" + ) + else + action( + type="mmsequence" + mode="key" + to="100" + step="0" + var="$!cnt3" + key="" + ) +</pre> + + +<p><b>Legacy Configuration Directives</b>:</p> + + <p>Not supported.</p> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmsnmptrapd.html b/doc/mmsnmptrapd.html index 699049d3..fb50f6c6 100644 --- a/doc/mmsnmptrapd.html +++ b/doc/mmsnmptrapd.html @@ -34,8 +34,8 @@ The following logic is applied to all message being processed: snmptrapd/severity/hostname. A configurable mapping table will be used to drive a new severity value from that severity string. If no mapping has been defined, the original severity is not changed. -<li>It replaces the "FromHost" value with the derived value from step2 -<li>It replaces the "Severity" value with the derived value from step 3 +<li>It replaces the "FromHost" value with the derived value from step 2 +<li>It replaces the "Severity" value with the derived value from step 3 </ol> <p>Note that the placement of this module inside the configuration is important. All actions before this modules is called will work on the unmodified message. All messages after it's call @@ -55,11 +55,11 @@ tells the module which start string inside the tag to look for. The default is matching incoming messages. It MUST not be given, except if two slashes are required for whatever reasons (so "tag/" results in a check for "tag//" at the start of the tag field). -<li><b>$mmsnmptrapdSeverityMapping</b> [severtiymap]<br> +<li><b>$mmsnmptrapdSeverityMapping</b> [severitymap]<br> This specifies the severity mapping table. It needs to be specified as a list. Note that due to the current config system <b>no whitespace</b> is supported inside the list, so be sure not to use any whitespace inside it.<br> -The list is constructed of Severtiy-Name/Severity-Value pairs, delimited by comma. +The list is constructed of Severity-Name/Severity-Value pairs, delimited by comma. Severity-Name is a case-sensitive string, e.g. "warning" and an associated numerical value (e.g. 4). Possible values are in the rage 0..7 and are defined in RFC5424, table 2. The diff --git a/doc/mmutf8fix.html b/doc/mmutf8fix.html new file mode 100644 index 00000000..6275c17e --- /dev/null +++ b/doc/mmutf8fix.html @@ -0,0 +1,105 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Fix invalid UTF-8 Sequences (mmutf8fix)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Fix invalid UTF-8 Sequences (mmutf8fix)</h1> +<p><b>Module Name: mmutf8fix</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.4</p> +<p><b>Description</b>:</p> +<p>The mmutf8fix module permits to fix invalid UTF-8 sequences. +Most often, such invalid sequences result from syslog sources sending +in non-UTF character sets, e.g. ISO 8859. As syslog does not have a way +to convey the character set information, these sequences are not properly +handled. While they are typically uncritical with plain text files, they can +cause big headache with database sources as well as systems like ElasticSearch. +<p>The module supports different "fixing" modes and fixes. The current +implementation will always replace invalid bytes with a single US ASCII +character. Additional replacement modes will probably be added in the future, +depending on user demand. In the longer term +it could also be evolved into an any-charset-to-UTF8 converter. But +first let's see if it really gets into widespread enough use. + +<p><b>Proper Usage</b>:</p> +<p>Some notes are due for proper use of this module. This is a message modification +module utilizing the action interface, which means you call it like an action. +This gives great flexibility on the question on when and how to call this module. +Note that once it has been called, it actually modifies the message. The original +messsage is then no longer available. However, this does <b>not</b> change any +properties set, used or extracted before the modification is done. +<p>One potential use case is to normalize all messages. This is done by simply calling +mmutf8fix right in front of all other actions. +<p>If only a specific source (or set of sources) is known to cause problems, +mmutf8fix can be conditionally called only on messages from them. This also offers +performance benefits. If such multiple sources exists, it probably is a good idea +to define different listeners for their incoming traffic, bind them to specific +<a href="multi_ruleset.html">ruleset</a> and call mmutf8fix as first action in this +ruleset. + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>mode</b> - <b>utf-8</b>/controlcharacters<br> +This sets the basic detection mode. +<br>In <b>utf-8</b> mode (the default), proper +UTF-8 encoding is checked and bytes which are not proper UTF-8 sequences +are acted on. If a proper multi-byte start sequence byte is detected but +any of the following bytes is invalid, the whole sequence is replaced by +the replacement method. This mode is most useful with non-US-ASCII character +sets, which validly includes multibyte sequences. Note that in this mode +control characters are NOT being replaced, because they are valid UTF-8. +<br>In <b>controlcharacters</b> mode, all bytes which do not represent a +printable US-ASCII character (codes 32 to 126) are replaced. Note that this +also mangles valid UTF-8 multi-byte sequences, as these are (deliberately) outside +of that character range. This mode is most useful if it is known that no +characters outside of the US-ASCII alphabet need to be processed. +<li><b>replacementChar</b> - default " " (space), a single character<br> +This is the character that invalid sequences are replaced by. Currently, it +MUST be a <b>printable</b> US-ASCII character. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>overlong UTF-8 encodings are currently not detected in utf-8 mode. +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we write one file without fixing UTF-8 and another one +with the message fixed. Note that once mmutf8fix has run, access to the +original message is no longer possible. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +action(type="omfile" file="/path/to/non-fixed.log") +action(type="mmutf8fix") +action(type="omfile" file="/path/to/fixed.log") +</textarea> + +<p>In this sample, we fix only message originating from host 10.0.0.1. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix") +# all other actions here... +</textarea> + +<p>This is mostly the same as the previous sample, but uses "controlcharacters" +processing mode. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix" mode="controlcharacters") +# all other actions here... +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/modules.html b/doc/modules.html index 4eae6db3..0ed7d4fe 100644 --- a/doc/modules.html +++ b/doc/modules.html @@ -70,7 +70,7 @@ other internal structures). Besides security, this also greatly simplifies the job of the output module developer.</p> <h2>Action Selectors</h2> <p>Modules (and rsyslog) need to know when they are called. For this, there must -a an action identification in selector lines. There are two syntaxes: the +be an action identification in selector lines. There are two syntaxes: the single-character syntax, where a single characters identifies a module (e.g. "*" for a wall message) and the modules designator syntax, where the module name is given between colons (e.g. ":ommysql:"). The single character syntax is diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index 83c495ca..1be2c81e 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -5,7 +5,7 @@ <h1>Multiple Rulesets in rsyslog</h1> <p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports multiple rulesets within a single configuration. -This is especially useful for routing the recpetion of remote messages to a set of specific rules. +This is especially useful for routing the reception of remote messages to a set of specific rules. Note that the input module must support binding to non-standard rulesets, so the functionality may not be available with all inputs. <p>In this document, I am using <a href="imtcp.html">imtcp</a>, an input module @@ -39,7 +39,7 @@ is the name space reserved for rsyslog use). If it finds this directive, it begi rule set (if the name was not yet know) or switches to an already-existing one (if the name was known). All rules defined between this $RuleSet directive and the next one are appended to the named ruleset. Note that the reserved name "RSYSLOG_DefaultRuleset" is used to -specify rsyslogd's default ruleset. You can use that name whereever you can use a ruleset name, +specify rsyslogd's default ruleset. You can use that name wherever you can use a ruleset name, including when binding an input to it. <p>Inside a ruleset, messages are processed as described above: they start with the first rule @@ -48,7 +48,7 @@ there are no more rules or the discard action is executed. Note that with multip no longer <b>all</b> rsyslog.conf rules are executed but <b>only</b> those that are contained within the specific ruleset. -<p>Inputs must explicitely bind to rulesets. If they don't do, the default ruleset is bound. +<p>Inputs must explicitly bind to rulesets. If they don't do, the default ruleset is bound. <p>This brings up the next question: @@ -58,10 +58,10 @@ it means that a specific input, or part of an input (like a tcp listener) will u ruleset to "pass its messages to". So when a new message arrives, it will be processed via the bound ruleset. Rule from all other rulesets are irrelevant and will never be processed. <p>This makes multiple rulesets very handy to process local and remote message via -seperate means: bind the respective receivers to different rule sets, and you do not need -to seperate the messages by any other method. +separate means: bind the respective receivers to different rule sets, and you do not need +to separate the messages by any other method. -<p>Binding to rulesets is input-specifc. For imtcp, this is done via the +<p>Binding to rulesets is input-specific. For imtcp, this is done via the <pre>input(type="imptcp" port="514" ruleset="rulesetname"); </pre> @@ -72,7 +72,7 @@ I personally think that it is best to define all rule sets at the top of rsyslog define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but seems to be a really useful and straightforward way of doing things. <h2>Why are rulesets important for different parser configurations?</h2> -<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid) +<p>Custom message parsers, used to handle different (and potentially otherwise-invalid) message formats, can be bound to rulesets. So multiple rulesets can be a very useful way to handle devices sending messages in different malformed formats in a consistent way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation @@ -89,6 +89,15 @@ modify the ruleset to which the next input is bound but rather provides a system default rule set for those inputs that did not explicitly bind to one. As such, the directive can not be used as a work-around to bind inputs to non-default rulesets that do not support ruleset binding. + +<h2>Rulesets and Main Queues</h2> +By default, rulesets do not have their own queue. It must be activated via +the $RulesetCreateMainQueue directive, or if using rainerscript format, by +specifying queue parameters on the ruleset directive, e.g. +ruleset(name="whatever" queue.type="fixedArray" queue. ...) +See <a href="http://www.rsyslog.com/doc/queue_parameters.html">http://www.rsyslog.com/doc/queue_parameters.html</a> +for more details. + <h2>Examples</h2> <h3>Split local and remote logging</h3> <p>Let's say you have a pretty standard system that logs its local messages to the usual @@ -222,7 +231,7 @@ needs to insert messages into the main message queue. So if each of them wants t submit a newly arrived message into the queue at the same time, only one can do so while the others need to wait. With multiple rulesets, its own queue can be created for each ruleset. If now each listener is bound to its own ruleset, concurrent message submission is -possible. On a machine with a sufficiently large number of corse, this can result in +possible. On a machine with a sufficiently large number of cores, this can result in dramatic performance improvement. <p>It is highly advised that high-performance systems define a dedicated ruleset, with a dedicated queue for each of the inputs. diff --git a/doc/multi_ruleset_legacy_format.html b/doc/multi_ruleset_legacy_format.html index 273a4a09..03586ca7 100644 --- a/doc/multi_ruleset_legacy_format.html +++ b/doc/multi_ruleset_legacy_format.html @@ -5,18 +5,18 @@ <h1>Multiple Rulesets in rsyslog</h1> <p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports multiple rulesets within a single configuration. -This is especially useful for routing the recpetion of remote messages to a set of specific rules. +This is especially useful for routing the reception of remote messages to a set of specific rules. Note that the input module must support binding to non-standard rulesets, so the functionality -may not be available with all inputs.<p> +may not be available with all inputs.</p> <b>Attention: this guide is shortened and only contains the samples in legacy format.</b> -Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html<a> +Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html</a> <h2>Examples</h2> <h3>Split local and remote logging</h3> <p>Let's say you have a pretty standard system that logs its local messages to the usual bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf -might look like this: +might look like this:</p> <pre> # ... module loading ... @@ -34,7 +34,7 @@ cron.* /var/log/cron <p>Now, you want to add receive messages from a remote system and log these to a special file, but you do not want to have these messages written to the files specified above. The traditional approach is to add a rule in front of all others that -filters on the message, processes it and then discards it: +filters on the message, processes it and then discards it:</p> <pre> # ... module loading ... @@ -55,7 +55,7 @@ cron.* /var/log/cron </pre> <p>Note the tilde character, which is the discard action!. Also note that we assume that -192.0.2.1 is the sole remote sender (to keep it simple). +192.0.2.1 is the sole remote sender (to keep it simple).</p> <p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception case and bind it to the receiver. This may be written as follows: @@ -88,7 +88,7 @@ cron.* /var/log/cron <p>Here, we need to switch back to the default ruleset after we have defined our custom one. This is why I recommend a different ordering, which I find more intuitive. The sample -below has it, and it leads to the same results: +below has it, and it leads to the same results:</p> <pre> # ... module loading ... @@ -116,27 +116,27 @@ $InputTCPServerRun 10514 </pre> <p>Here, we do not switch back to the default ruleset, because this is not needed as it is -completely defined when we begin the "remote" ruleset. +completely defined when we begin the "remote" ruleset.</p> <p>Now look at the examples and compare them to the single-ruleset solution. You will notice that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use "*.*" as all messages now means all messages that are being processed by this rule set and all of them come in via the TCP receiver! This is what makes using multiple -rulesets so much easier. +rulesets so much easier.</p> <h3>Split local and remote logging for three different ports</h3> <p>This example is almost like the first one, but it extends it a little bit. While it is very similar, I hope it is different enough to provide a useful example why you may want -to have more than two rulesets. +to have more than two rulesets.</p> <p>Again, we would like to use the "regular" log files for local logging, only. But this time we set up three syslog/tcp listeners, each one listening to a different port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into different files. Also, logs received from 10516 (and only from that port!) with "mail.*" priority, shall be written into a specif file and <b>not</b> be -written to 10516's general log file. +written to 10516's general log file.</p> -<p>This is the config: +<p>This is the config:</p> <pre> # ... module loading ... @@ -180,12 +180,12 @@ $InputTCPServerRun 10516 </pre> <p>Note that the "mail.*" rule inside the "remote10516" ruleset does -not affect processing inside any other rule set, including the default rule set. +not affect processing inside any other rule set, including the default rule set.</p> <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> +project.<br/> Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/ns_gtls.html b/doc/ns_gtls.html index 0d02ad02..21a7f19c 100644 --- a/doc/ns_gtls.html +++ b/doc/ns_gtls.html @@ -10,7 +10,7 @@ library</a>.</p> <p><b>Available since:</b> 3.19.0 (suggested minimum 3.19.8 and above)</p> <p style="font-weight: bold;">Supported Driver Modes</p> <ul> -<li>0 - unencrypted trasmission (just like <a href="ns_ptcp.html">ptcp</a> driver)</li> +<li>0 - unencrypted transmission (just like <a href="ns_ptcp.html">ptcp</a> driver)</li> <li>1 - TLS-protected operation</li> </ul> Note: mode 0 does not provide any benefit over the ptcp driver. This @@ -38,7 +38,7 @@ unauthorized access. It is recommended NOT to use this mode.</p> <p>x509/certvalid is a nonstandard mode. It validates the remote peers certificate, but does not check the subject name. This is weak authentication that may be useful in scenarios where multiple -devices are deployed and it is sufficient proof of authenticy when +devices are deployed and it is sufficient proof of authenticity when their certificates are signed by the CA the server trusts. This is better than anon authentication, but still not recommended. <b>Known Problems</b><br> diff --git a/doc/omfile.html b/doc/omfile.html index 0f64f26f..2ad95d05 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -11,19 +11,37 @@ <p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> <p><b>Description</b>:</p> <p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> -<p> </p> <p><b>Module Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li> + Set the default template to be used if an action is not + configured to use a specific template.<br></li> + + <li><strong>DirCreateMode </strong>[default 0700]<br> + Sets the default DirCreateMode to be used for an action + if no explicit one is specified.</br> + <li><strong>FileCreateMode </strong>[default 0644]<br> + Sets the default DirCreateMode to be used for an action + if no explicit one is specified.</br> </ul> <p> </p> <p><b>Action Parameters</b>:</p> <ul> - <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + <li><strong>Template </strong>[templateName]<br> + Sets the template to be used for this action. If not specified, the + default template is applied.<br></li><br> + + <li><strong>DynaFileCacheSize </strong>(not mandatory, default 10)<br> + Applies only if dynamic filenames are used.<br> + Specifies the number of DynaFiles that will be kept open. The default is + 10. Note that this is a per-action value, so if you have multiple + dynafile actions, each of them have their individual caches (which means + the numbers sum up). Ideally, the cache size exactly matches the + need. You can use <a href="impstats.html">impstats</a> to tune this + value. Note that a too-low cache size can be a very considerable + performance bottleneck.<br></li><br> <li><strong>ZipLevel </strong>0..9 [default 0]<br> if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> @@ -39,7 +57,8 @@ </li><br> <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + Defines, in seconds, the interval after which unwritten data is + flushed.<br></li><br> <li><strong>ASyncWriting </strong>on/off [default off]<br> if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable FlushInterval, AsyncWriting must be set to "on". Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is "on" but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. <br></li><br> @@ -62,12 +81,12 @@ <li><strong>FileGroup </strong><br> Set the group for files newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> - <li><strong>DirCreateMode </strong>[defaul 0700]<br> - This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> - - <li><strong>FileCreateMode </strong>[default 0644]<br> + <li><strong>FileCreateMode </strong>[default equelly-named module parameter]<br> The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> + <li><strong>DirCreateMode </strong>[default equelly-named module parameter]<br> + This is the same as FileCreateMode, but for directories automatically generated.<br></li><br> + <li><strong>FailOnCHOwnFailure </strong>on/off [default on]<br> This option modifies behaviour of file creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> @@ -75,7 +94,8 @@ create directories on an as-needed basis<br></li><br> <li><strong>Sync </strong>on/off [default off]<br> - enables file syncing capability of omfile.<br></li><br> + enables file syncing capability of omfile. Note that this causes + an enormous performance hit if enabled.<br></li><br> <li><strong>File </strong><br> If the file already exists, new data is appended to it. Existing data is not truncated. If the file does not already exist, it is created. Files are kept open as long as rsyslogd is active. This conflicts with external log file rotation. In order to close a file after rotation, send rsyslogd a HUP signal after the file has been rotated away. <br></li><br> @@ -93,9 +113,6 @@ there only is one provider called "<a href="cryprov_gcry.html">gcry</a>".<br></li><br> - <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li><br> - </ul> <p><b>See Also</b> <ul> @@ -124,11 +141,10 @@ unusable. </ul> <p><b>Sample:</b></p> <p>The following command writes all syslog messages into a file.</p> -<textarea rows="5" cols="60">Module (load="builtin:omfile") -*.* action(type="omfile" - DirCreateMode="0700" - FileCreateMode="0644" - File="/var/log/messages") +<textarea rows="5" cols="60">action(type="omfile" + DirCreateMode="0700" + FileCreateMode="0644" + File="/var/log/messages") </textarea> <br><br> diff --git a/doc/omfwd.html b/doc/omfwd.html index a541dd27..d535b29b 100644 --- a/doc/omfwd.html +++ b/doc/omfwd.html @@ -13,14 +13,21 @@ <p>The omfwd plug-in provides the core functionality of traditional message forwarding via UDP and plain TCP. It is a built-in module that does not need to be loaded. </p> <p> </p> -<p><b>Global Configuration Directives</b>:</p> +<p><b><i>Note: this documentation describes features present in v7+ of +rsyslog. If you use an older version, scroll down to "legacy parameters".</i></b> +If you prefer, you can also +<a href="http://www.rsyslog.com/how-to-obtain-a-specific-doc-version/">obtain +a specific version of the rsyslog documentation</a>. +<p> </p> + +<p><b>Module Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> sets a non-standard default template for this module.<br></li> </ul> <p> </p> -<p><b>Action specific Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> <li><strong>Target </strong>string<br> Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> @@ -35,7 +42,67 @@ Framing-Mode to be for forwarding. This affects only TCP-based protocols. It is ignored for UDP. In protocol engineering, ``framing'' means how multiple messages over the same connection are separated. Usually, this is transparent to users. Unfortunately, the early syslog protocol evolved, and so there are cases where users need to specify the framing. The traditional framing is nontransparent. With it, messages are end when a LF (aka ``line break'', ``return'') is encountered, and the next message starts immediately after the LF. If multi-line messages are received, these are essentially broken up into multiple message, usually with all but the first message segment being incorrectly formatted. The octet-counting framing solves this issue. With it, each message is prefixed with the actual message length, so that a receivers knows exactly where the message ends. Multi-line messages cause no problem here. This mode is very close to the method described in RFC5425 for TLS-enabled syslog. Unfortunately, only few syslogd implementations support octet-counted framing. As such, the traditional framing is set as default, even though it has defects. If it is known that the receiver supports octet-counted framing, it is suggested to use that framing mode. <br></li><br> <li><strong>ZipLevel </strong>0..9 [default 0]<br> - Compression level for messages. Rsyslog implements a proprietary capability to zip transmitted messages. Note that compression happens on a message-per-message basis. As such, there is a performance gain only for larger messages. Before compressing a message, rsyslog checks if there is some gain by compression. If so, the message is sent compressed. If not, it is sent uncompressed. As such, it is totally valid that compressed and uncompressed messages are intermixed within a conversation. <br>The compression level is specified via the usual factor of 0 to 9, with 9 being the strongest compression (taking up most processing time) and 0 being no compression at all (taking up no extra processing time). <br></li><br> + Compression level for messages. + <br>Up until rsyslog 7.5.1, this was the only compression setting that + rsyslog understood. Starting with 7.5.1, we have different compression + modes. All of them are affected by the ziplevel. If, however, no mode + is explicitely set, setting ziplevel also turns on "single" + compression mode, so pre 7.5.1 configuration will continue to work + as expected. + <br>The compression level is specified via the usual factor of 0 to 9, with 9 being the strongest compression (taking up most processing time) and 0 being no compression at all (taking up no extra processing time). <br></li><br> + + <li><b>maxErrorMessages </b>integer [default 5], available since 7.5.4<br> + This sets the maximum number of error messages that omfwd + emits during regular operations. The reason for such an upper + limit is that error messages are conveyed back to rsyslog's + input message stream. So if there would be no limit, an endless + loop could be initiated if the failing action would need to + process its own error messages and the emit a new one. This is + also the reason why the default is very conservatively low. + Note that version prior to 7.5.4 did not report any error + messages for the same reason. Also note that with the initial + implementation only errors during UDP forwarding are logged.<br></li><br> + + <li><b>compression.mode</b> <i>mode</i><br> + <i>mode</i> is one of "none", "single", or "stream:always". The + default is "none", in which no compression happens at all. + <br>In "single" compression mode, Rsyslog implements a proprietary + capability to zip transmitted messages. That compression happens + on a message-per-message basis. As such, there is a performance gain + only for larger messages. Before compressing a message, rsyslog checks + if there is some gain by compression. If so, the message is sent + compressed. If not, it is sent uncompressed. As such, it is totally + valid that compressed and uncompressed messages are intermixed + within a conversation. + <br>In "stream:always" compression mode the full stream is being + compressed. This also uses non-standard protocol and is compatible + only with receives that have the same abilities. This mode offers + potentially very high compression ratios. With typical syslog + messages, it can be as high as 95+% compression (so only one twentieth + of data is actually transmitted!). Note that this mode introduces + extra latency, as data is only sent when the compressor emits new + compressed data. For typical syslog messages, this can mean that + some hundered messages may be held in local buffers before they are + actually sent. This mode has been introduced in 7.5.1. + <br><b>Note: currently only imptcp supports receiving stream-compressed + data.</b> + <br></li><br> + + <li><b>compression.stream.flushOnTXEnd</b> <i>[<b>on</b>/off</i>] (requires 7.5.3+)<br> + This setting affects stream compression mode, only. If enabled (the default), the compression + buffer will by emptied at the end of a rsyslog batch. If set to "off", + end of batch will not affect compression at all.<br> + While setting it to "off" can potentially greatly improve compression + ratio, it will also introduce severe delay between when a message is being processed + by rsyslog and actually sent out to the network. We have seen cases where for + several thousand message not a single byte was sent. This is good in the sense that + it can happen only if we have a great compression ratio. This is most probably + a very good mode for busy machines which will process several thousand messages + per second and te resulting short delay will not pose any problems. However, + the default is more conservative, while it works more "naturally" with even low + message traffic. Even in flush mode, notable compression should be achivable + (but we do not yet have practice reports on actual compression ratios). + <br></li><br> <li><strong>RebindInterval </strong>integer<br> Permits to specify an interval at which the current connection is broken and re-established. This setting is primarily an aid to load balancers. After the configured number of messages has been transmitted, the current connection is terminated and a new one started. Note that this setting applies to both TCP and UDP traffic. For UDP, the new ``connection'' uses a different source port (ports are cycled and not reused too frequently). This usually is perceived as a ``new connection'' by load balancers, which in turn forward messages to another physical target system. <br></li><br> @@ -64,12 +131,11 @@ <p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> <p><b>Sample:</b></p> <p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> -<textarea rows="5" cols="60">Module (load="builtin:omfwd") -*.* action(type="omfwd" -Target="192.168.2.11" -Port="10514" -Protocol="tcp" -) +<textarea rows="5" cols="60">action(type="omfwd" + Target="192.168.2.11" + Port="10514" + Protocol="tcp" + ) </textarea> <br><br> @@ -116,7 +182,7 @@ Protocol="tcp" index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omprog.html b/doc/omprog.html index 471ab224..4f369735 100644 --- a/doc/omprog.html +++ b/doc/omprog.html @@ -25,7 +25,34 @@ con re-using existing binaries. For the time being, it simply is not done. In th we may add an option for such pooling, provided that some demand for that is voiced. You can also mimic the same effect by defining multiple rulesets and including them (at the price of some slight performance loss). -<p><b>Configuration Directives</b>:</p> + +<p> </p> + +<p><b>Module Parameters</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action Parameters</b>:</p> +<ul> + <li><strong>binary </strong><br> + Mostly equivalent to the "binary" action parameter, but must contain the binary name + only. In legacy config, it is <b>not possible</b> to specify command line parameters. +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command writes all syslog messages into a file.</p> +<textarea rows="5" cols="85">Module (load="omprog") +*.* action(type="omprog" + binary="/pathto/omprog.py --parm1=\"value 1\" --parm2=value2" + template="RSYSLOG_TraditionalFileFormat") +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> <ul> <li><b>$ActionOMProgBinary</b> <binary><br> The binary program to be executed. @@ -36,7 +63,7 @@ The binary program to be executed. [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omrelp.html b/doc/omrelp.html index 8858f884..8049ebaf 100644 --- a/doc/omrelp.html +++ b/doc/omrelp.html @@ -24,16 +24,89 @@ implementation).</p> rsyslog 7.3.10. For older versions, legacy configuration directives must be used. <ul> - <li><b>target </b>(mandatory)<br> + <li><b>target</b> (mandatory)<br> The target server to connect to. </li> - <li><b>template </b>(not mandatory, default "RSYSLOG_ForwardFormat")<br> + <li><b>template</b> (not mandatory, default "RSYSLOG_ForwardFormat")<br> Defines the template to be used for the output. </li> - <li><b>timeout </b>(not mandatory, default 90)<br> + <li><b>timeout</b> (not mandatory, default 90)<br> Timeout for relp sessions. If set too low, valid sessions may be considered dead and tried to recover. </li> + <li><b>windowSize</b> (not mandatory, default 0)<br> + This is an <b>expert parameter</b>. It permits to override the + RELP window size being used by the client. Changing the window + size has both an effect on performance as well as potential + message duplication in failure case. A larger window size means + more performance, but also potentially more duplicated + messages - and vice versa. The default 0 means that librelp's + default window size is being used, which is considered a + compromise between goals reached. For your information: + at the time of this writing, the librelp default window size + is 128 messages, but this may change at any time. + <br>Note that there is no equivalent server parameter, as the + client proposes and manages the window size in RELP protocol. + <li><b>tls</b> (not mandatory, values "on","off", default "off")<br> + If set to "on", the RELP connection will be encrypted by TLS, so that the data is protected against observers. Please note that both the client and the server must have set TLS to either "on" or "off". Other combinations lead to unpredictable results. + </li> + <li><b>tls.compression</b> (not mandatory, values "on","off", default "off")<br> + The controls if the TLS stream should be compressed (zipped). While this + increases CPU use, the network bandwidth should be reduced. Note that + typical text-based log records usually compress rather well. + </li> + <li><b>tls.permittedPeer</b> peer</br> + Places access restrictions on this forwarder. Only peers which + have been listed in this parameter may be connected to. + This guards against rouge servers and man-in-the-middle + attacks. The validation + bases on the certficate the remote peer presents.<br> + The <i>peer</i> parameter lists permitted certificate + fingerprints. Note that it is an array parameter, so either + a single or multiple fingerprints can be listed. When a + non-permitted peer is connected to, the refusal is logged together + with it's fingerprint. So if the administrator knows this was + a valid request, he can simple add the fingerprint by copy and + paste from the logfile to rsyslog.conf. It must be noted, though, + that this situation should usually not happen after initial + client setup and administrators should be alert in this case. + <br>Note that usually a single remote peer should be all that + is ever needed. Support for multiple peers is primarily included + in support of load balancing scenarios. If the connection + goes to a specific server, only one specific certificate is ever + expected (just like when connecting to a specific ssh server). + <br>To specify multiple fingerprints, just enclose them + in braces like this: + <br>tls.permittedPeer=["SHA1:...1", "SHA1:....2"] + <br>To specify just a single peer, you can either + specify the string directly or enclose it in braces. + </li> + <li><b>tls.authMode</b> mode</br> + Sets the mode used for mutual authentication. Supported values are + either "<i>fingerprint</i>" or "<i>name"</i>. + <br>Fingerprint mode basically is what SSH + does. It does not require a full PKI to be present, instead self-signed + certs can be used on all peers. Even if a CA certificate is given, the + validity of the peer cert is NOT verified against it. Only the + certificate fingerprint counts. + <br>In "name" mode, certificate validation happens. Here, the matching + is done against the certificate's subjectAltName and, as a fallback, + the subject common name. If the certificate contains multiple names, + a match on any one of these names is considered good and permits the + peer to talk to rsyslog. + <li><b>tls.prioritystring</b> (not mandatory, string)<br> + This parameter permits to specify the so-called "priority string" to + GnuTLS. This string gives complete control over all crypto parameters, + including compression setting. For this reason, when the prioritystring + is specified, the "tls.compression" parameter has no effect and is + ignored. + <br>Full information about how to construct a priority string can be + found in the GnuTLS manual. At the time of this writing, this + information was contained in + <a href="http://gnutls.org/manual/html_node/Priority-Strings.html">section 6.10 of the GnuTLS manual</a>. + <br><b>Note: this is an expert parameter.</b> Do not use if you do + not exactly know what you are doing. + </li> </ul> <p><b>Sample:</b></p> <p>The following sample sends all messages to the central server diff --git a/doc/omruleset.html b/doc/omruleset.html index f0d5f7bd..3911929d 100644 --- a/doc/omruleset.html +++ b/doc/omruleset.html @@ -10,6 +10,20 @@ <p><b>Module Name: omruleset</b></p> <p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> <p><b>Available Since</b>: 5.3.4</p> +<p><b><font color="red">Deprecated in</font></b>: 7.2.0+</p> +<p><b>Deprecation note</b></p> +<p><font color="red">This module exists only for backwards-compatibility +reasons.</font> +<b>Do no longer use it in new configurations.</b> It has been +replaced by the much more efficient +<a href="rainerscript_call.html">"call" RainerScript statement</a>. The +"call" statement supports everything omruleset does, but in an easier +to use way. +<br> +<br> +<br> +<br> +<br> <p><b>Description</b>:</p> <p>This is a very special "output" module. It permits to pass a message object to another rule set. While this is a very simple action, it enables very diff --git a/doc/queue_parameters.html b/doc/queue_parameters.html new file mode 100644 index 00000000..0237a475 --- /dev/null +++ b/doc/queue_parameters.html @@ -0,0 +1,84 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>rsyslog queues parameters</title></head> +<body> +<a href="rsyslog_conf_global.html">back</a> +<p><h2>General Queue Parameters</h2></p> +<p>Queue parameters can be used together with the following statements: +<ul> +<li><a href="rsyslog_conf_actions.html">action()</a> +<li>ruleset()</a> +<li>main_queue()</a> +</ul> +<p> +Queues need to be configured in the action or ruleset it should affect. If nothing is configured, +default values will be used. Thus, the default ruleset has only the default main queue. Specific Action +queues are not set up by default.</p> +<ul> + <li><strong>queue.filename</strong> name</li> + <li><strong>queue.size</strong> number <br> + This is the maximum size of the queue in number of messages. + Note that setting the queue size to very small values (roughly + below 100 messages) is not supported and can lead to + unpredictable results.<br> + For more information on the current status of this restriction + see the <a href="http://www.rsyslog.com/lower-bound-for-queue-sizes/">rsyslog + FAQ: "lower bound for queue sizes"</a>.</li> + <li><strong>queue.dequeuebatchsize</strong> number + <br>default 16</li> + <li><strong>queue.maxdiskspace</strong> number</li> + <li><strong>queue.highwatermark</strong> number + <br>default 8000</li> + <li><strong>queue.lowwatermark</strong> number + <br>default 2000</li> + <li><strong>queue.fulldelaymark</strong> number</li> + <li><strong>queue.lightdelaymark</strong> number</li> + <li><strong>queue.discardmark</strong> number + <br>default 9750]</li> + <li><strong>queue.discardseverity</strong> number + <br>*numerical* severity! default 8 (nothing discarded)</li> + <li><strong>queue.checkpointinterval</strong> number</li> + <li><strong>queue.syncqueuefiles</strong> on/off</li> + <li><strong>queue.type</strong> [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> + <li><strong>queue.workerthreads</strong> number + <br>number of worker threads, default 1, recommended 1</li> + <li><strong>queue.timeoutshutdown</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 0 (indefinite)</li> + <li><strong>queue.timeoutactioncompletion</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 1000, 0 means immediate!</li> + <li><strong>queue.timeoutenqueue</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite</li> + <li><strong>queue.timeoutworkerthreadshutdown</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 60000 (1 minute)</li> + <li><strong>queue.workerthreadminimummessages</strong> number + <br>default 100</li> + <li><strong>queue.maxfilesize</strong> size_nbr + <br> default 1m</li> + <li><strong>queue.saveonshutdown</strong> on/<b>off</b></li> + <li><strong>queue.dequeueslowdown</strong> number + <br>number is timeout in microseconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!</li> + <li><strong>queue.dequeuetimebegin</strong> number</li> + <li><strong>queue.dequeuetimeend</strong> number</li> +</ul> +<p><b>Sample:</b></p> +<p>The following is a sample of a TCP forwarding action with its own queue.</p> +<textarea rows="7" cols="60">action(type="omfwd" + target="192.168.2.11" + port="10514" + protocol="tcp" + queue.filename="forwarding" + queue.size="1000000" + queue.type="LinkedList" +) +</textarea> +<br><br> +[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index 7cbbfa9f..0a780ac4 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -34,6 +34,39 @@ return a valid result, as you can't really add two letters (to concatenate them, use the concatenation operator &). However, all type conversions are automatically done by the script interpreter when there is need to do so.<br> +<h2>Variable (Property) types</h2> +<p>All rsyslog properties (see the <a href="property_replacer.html">property +replacer</a> page for a list) can be used in RainerScript. In addition, it also +supports local variables. Local variables are local to the current message, but are +NOT message properties (e.g. the "$!" all JSON property does not contain +them). +<p>Only message json (CEE/Lumberjack) properties can be modified by +the "set" and "unset" statements, not any other message property. Obviously, +local variables are also modifieable. +<p>Message JSON property names start with "$!" where the bang character +represents the root. +<p>Local variables names start with "$.", where the dot denotes the root. +<p>Both JSON properties as well as local variables may contain an arbitrary +deep path before the final element. The bang character is always used as path +separator, no matter if it is a message property or a local variable. For example +"$!path1!path2!varname" is a three-level deep message property where as +the very similar looking "$.path1!path2!varname" specifies a three-level +deep local variable. +The bang or dot character immediately following the +dollar sign is used by rsyslog to separate the different types. +<h2>configuration objects</h2> +<h3>main_queue()</h3> +<p><i>This object is available since 7.5.3.</i> +This permits to specify parameters for the main message queue. Note that +only <a href="queue_parameters.html">queue-parameters</a> are permitted for this +config object. This permits to set the same options like in ruleset and action +queues. A special statement is needed for the main queue, because it is a +different object and cannot be configured via any other object. +<p>Note that when the main_queue() object is configured, the legacy +$MainMsgQ... statements are ignored. +<p>Example:</p> +<textarea rows="2" cols="60">main_queue(queue.size="100000" queue.type="LinkedList") +</textarea> <h2>Expressions</h2> The language supports arbitrary complex expressions. All usual operators are supported. The precedence of operations is as follows diff --git a/doc/rainerscript_call.html b/doc/rainerscript_call.html new file mode 100644 index 00000000..faab5c24 --- /dev/null +++ b/doc/rainerscript_call.html @@ -0,0 +1,51 @@ +<html> +<head> +<title>rsyslog "call" statement</title> +</head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h1>The rsyslog "call" statement</h1> +<p>The rsyslog "call" statement is used to tie rulesets together. +It is modelled after the usual programming langauge "call" statement. Think +of a ruleset as a subroutine (what it really is!) and you get the picture. + +<p>The "call" statement can be used to call into any type of rulesets. +If a rule set has a queue assigned, the message will be posted to that queue +and processed asynchronously. Otherwise, the ruleset will be executed +synchronously and control returns to right after the call when the rule set +has finished execution. + +<p>Note that there is an important difference between asynchronous and +synchronous execution in regard to the "stop" statement. It will not affect +processing of the original message when run asynchronously. + +<p>The "call" statement replaces the deprecated omruleset module. It offers +all capabilities omruleset has, but works in a much more efficient way. +Note that omruleset was a hack that made calling rulesets possible +within the constraints of the pre-v7 engine. "call" is the clean solution for +the new engine. Especially for rulesets without associated queues (synchronous +operation), it has zero overhead (really!). omruleset always needs to +duplicate messages, which usually means at least ~250 bytes of memory writes, +some allocs and frees - and even more performance-intense operations. + +<h2>syntax</h2> +<p><code>call rulesetname</code></p> +<p>Where "rulesetname" is the name of a ruleset that is defined elsewhere +inside the configration. If the call is synchronous or asynchronous depends +on the ruleset parameters. This cannot be overriden by the "call" statement. + +<h2>related links</h2> +<ul> +<li><a href="http://blog.gerhards.net/2012/10/how-to-use-rsyslogs-ruleset-and-call.html">Blog posting announcing "call" statement (with sample)</a> +</ul> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 50b13a07..260d1f2b 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -3,9 +3,14 @@ <body> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> +<p><b><i>Note: this documentation describes features present in v7+ of +rsyslog. If you use an older version, scroll down to "legacy parameters".</i></b> +If you prefer, you can also +<a href="http://www.rsyslog.com/how-to-obtain-a-specific-doc-version/">obtain +a specific version of the rsyslog documentation</a>. <h2>Actions</h2> Action object describe what is to be done with a message. They are -implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. +implemented via <a href="rsyslog_conf_modules.html#om">output modules</a>. <p>The action object has different parameters: <ul> <li>those that apply to all actions and are action specific. These @@ -13,8 +18,11 @@ implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. <li>parameters for the action queue. While they also apply to all parameters, they are queue-specific, not action-specific (they are the same that are used in rulesets, for example). + The are documented separately under + <a href="queue_parameters.html">queue parameters</a>. <li>action-specific parameters. These are specific to a certain - type of actions. They are documented by the output module + type of actions. They are documented by the + <a href="rsyslog_conf_modules.html#om">output module</a> in question. </ul> <h3>General Action Parameters</h3> @@ -23,8 +31,14 @@ implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. <br>used for statistics gathering and documentation <li><b>type</b> string <br>Mandatory parameter for every action. The name of the module that should be used. </li> - <li><b>action.writeAllMarkMessages</b> on/off - <br>Normally, mark messages are written to actions only if the action was not recently executed (by default, recently means within the past 20 minutes). If this setting is switched to "on", mark messages are always sent to actions, no matter how recently they have been executed. In this mode, mark messages can be used as a kind of heartbeat.</li> + <li><b>action.writeAllMarkMessages</b> <i>on</i>/off + <br>This setting tells if mark messages are always written ("on", the default) or only + if the action was not recently executed ("off"). By default, recently means within the + past 20 minutes. If this setting is "on", mark messages are always sent to actions, no + matter how recently they have been executed. In this mode, mark messages can be used as + a kind of heartbeat. This mode also enables faster processing inside the rule engine. So + it should be set to "off" only when there is a good reason to do so. + </li> <li><b>action.execOnlyEveryNthTime</b> integer <br>If configured, the next action will only be executed every n-th time. For example, if configured to 3, the first two messages that go into the action will be dropped, the 3rd will actually cause the action to execute, the 4th and 5th will be dropped, the 6th executed under the action, ... and so on.</li> <li><b>action.execOnlyEveryNthTimeout</b> integer @@ -56,7 +70,19 @@ When an action is suspended (e.g. destination can not be connected), the action </ul> -<h2>Legacy Format</h2> +<h2>Useful Links</h2> +<ul> +<li>Rainer's blog posting on the performance of + <a href="http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html">main + and action queue worker threads</a> +</ul> + +<br> +<br> +<br> + + +<h1>Legacy Format</h1> <p><b>Be warned that legacy action format is hard to get right. It is recommended to use RainerScript-Style action format whenever possible!</b> A key problem with legacy format is that a single action is defined via diff --git a/doc/rsyslog_conf_basic_structure.html b/doc/rsyslog_conf_basic_structure.html index f5d4891a..54fb721c 100644 --- a/doc/rsyslog_conf_basic_structure.html +++ b/doc/rsyslog_conf_basic_structure.html @@ -43,7 +43,7 @@ after the stop statement are never evaluated. <ul> <li><b>if expr then ... else ...</b> - conditional execution <li><b>stop</b> - stops processing the current message -<li><b>call</b> - calls a ruleset (just like a subroutine call) +<li><b><a href="rainerscript_call.html">call</a></b> - calls a ruleset (just like a subroutine call) <li><b>continue</b> - a NOP, useful e.g. inside the then part of an if </ul> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 18d6b8a1..99557215 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -53,28 +53,28 @@ to message generators. and messages be transmitted to various different targets. <ul> <li><a href="omfile.html">omfile</a> - file output module</li> -<li><a href="omfwd.html">omfwd</a> - syslog forwarding output module</li> +<li><a href="omfwd.html">omfwd</a> (does NOT yet work in v8) - syslog forwarding output module</li> <li><a href="omjournal.html">omjournal</a> - Linux journal output module</li> <li><a href="ompipe.html">ompipe</a> - named pipe output module</li> <li><a href="omusrmsg.html">omusrmsg</a> - user message output module</li> -<li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> +<li><a href="omsnmp.html">omsnmp</a> (does NOT yet work in v8) - SNMP trap output module</li> <li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li> -<li><a href="omrelp.html">omrelp</a> - RELP output module</li> +<li><a href="omrelp.html">omrelp</a> (does NOT yet work in v8) - RELP output module</li> <li><a href="omruleset.html">omruleset</a> - forward message to another ruleset</li> -<li>omgssapi - output module for GSS-enabled syslog</li> +<li>omgssapi (does NOT yet work in v8) - output module for GSS-enabled syslog</li> <li><a href="ommysql.html">ommysql</a> - output module for MySQL</li> -<li>ompgsql - output module for PostgreSQL</li> -<li><a href="omlibdbi.html">omlibdbi</a> - +<li>ompgsql (does NOT yet work in v8) - output module for PostgreSQL</li> +<li><a href="omlibdbi.html">omlibdbi</a> (does NOT yet work in v8) - generic database output module (Firebird/Interbase, MS SQL, Sybase, SQLLite, Ingres, Oracle, mSQL)</li> -<li><a href="ommail.html">ommail</a> - +<li><a href="ommail.html">ommail</a> (does NOT yet work in v8) - permits rsyslog to alert folks by mail if something important happens</li> -<li><a href="omprog.html">omprog</a> - permits sending messages to a program for custom processing</li> -<li><a href="omoracle.html">omoracle</a> - output module for Oracle (native OCI interface)</li> -<li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> -<li><a href="omuxsock.html">omuxsock</a> - output module Unix domain sockets</li> -<li><a href="omhdfs.html">omhdfs</a> - output module for Hadoop's HDFS file system</li> -<li><a href="ommongodb.html">ommongodb</a> - output module for MongoDB</li> +<li><a href="omprog.html">omprog</a> (does NOT yet work in v8) - permits sending messages to a program for custom processing</li> +<li><a href="omoracle.html">omoracle</a> (orphaned) - output module for Oracle (native OCI interface)</li> +<li><a href="omudpspoof.html">omudpspoof</a> (does NOT yet work in v8) - output module sending UDP syslog messages with a spoofed address</li> +<li><a href="omuxsock.html">omuxsock</a> (does NOT yet work in v8) - output module Unix domain sockets</li> +<li><a href="omhdfs.html">omhdfs</a> (does NOT yet work in v8) - output module for Hadoop's HDFS file system</li> +<li><a href="ommongodb.html">ommongodb</a> (does NOT yet work in v8) - output module for MongoDB</li> <li><a href="omelasticsearch.html">omelasticsearch</a> - output module for ElasticSearch</li> </ul> @@ -111,13 +111,22 @@ probably an excellent starting base for writing a new module. Currently, the fol modules exist inside the source tree: <ul> <li><a href="mmanon.html">mmanon</a> - used to anonymize log messages. +<li><a href="mmcount.html">mmcount</a> - message modification plugin which counts messages +<li><a href="mmfields.html">mmfields</a> - used to extract fields from +specially formatted messages (e.g. CEF) <li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. Note that this actually is a <b>generic</b> module. <li><a href="mmjsonparse.html">mmjsonparse</a> - used to interpret CEE/lumberjack enabled structured log messages. +<li><a href="mmpstrucdata.html">mmpstrucdata</a> - used to parse RFC5424 structured data +into json message properties <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. +<li><a href="mmutf8fix.html">mmutf8fix</a> - used to fix invalid UTF-8 character sequences +<li><a href="mmrfc5424addhmac.html">mmrfc5424addhmac</a> - custom module for adding HMACs to +rfc5424-formatted messages if not already present +<li><a href="mmsequence.html">mmsequence</a> - sequence generator and counter plugin </ul> <a name="lm"></a><h2>String Generator Modules</h2> diff --git a/doc/v8compatibility.html b/doc/v8compatibility.html new file mode 100644 index 00000000..6dda2c45 --- /dev/null +++ b/doc/v8compatibility.html @@ -0,0 +1,120 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v8</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v8</h1> +This document describes things to keep in mind when moving from v7 to v8. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used v7 and want to use v8 without hassle. +<p>Version 8 offers a completely rewritten core rsyslog engine. This resulted in +a number of changes that are visible to users and (plugin) developers. +Most importantly, pre-v8 plugins <b>do not longer work</b> and need to +be updated to support the new calling interfaces. If you developed a plugin, +be sure to review the developer section below. +</p> +<h2>Mark Messages</h2> +<p>In previous versions, mark messages were by default only processed if an +action was not executed for some time. The default has now changed, and mark +messages are now always processed. Note that this enables faster processing +inside rsyslog. To change to previous behaviour, you need to add +action.writeAllMarkMessages="off" to the actions in question. + +<h2>Modules WITHOUT v8 Support</h2> +<p>The following modules will not properly build under rsyslog v8 and +need to be updated: +<ul> +<li>plugins/omgssapi +<li>plugins/omhdfs +<li>plugins/omhiredis - not project supported +<li>plugins/omrabbitmq - not project supported +<li>plugins/omzmq3 - not project supported +<li>plugins/mmrfc5424addhmac - was a custom project, requires sponsoring + for conversion +<li>plugins/omoracle - orphaned since a while -- use omlibdbi instead +<li>plugins/mmsnmptrapd - waiting for demand +<li>plugins/mmaudit - scheduled to be updated soon +<li>plugins/mmcount - scheduled to be updated soon +</ul> + +<h2>Untested Modules</h2> +<p>The following modules have been updated and successfully +build, but no "real" test were conducted. Users of these modules +should use extra care. +<ul> +<li>mmsequence +<li>omprog +<li>omsnmp +<li>mmfields +<li>mmpstrucdata +<li>omlibdbi - will be tested soon +<li>ommongodb - larger changes still outstanding +<li>ompgsql - larger chages still outstanding +<li>omuxsock +</ul> +<p>In addition to bug reports, success reports are also appreciated for +these modules (this may save us testing). + +<h2>What Developers need to Know</h2> +<h3>output plugin interface</h3> +<p>To support the new core engine, the output interface has been considerably +changed. It is suggested to review some of the project-provided plugins for +full details. In this doc, we describe the most important changes from a high +level perspective. +<p><b>NOTE: the v8 output module interface is not yet stable.</b> It is highly +likely that additional changes will be made within the next weeks. So if you +convert a module now, be prepared for additional mandatory changes. + +<p><b>Multi-thread awareness required</b></p> +<p>The new engine activates one <b>worker</b>instance of output actions on +each worker thread. This means an action has now three types of data: +<ul> +<li>global +<li>action-instance - previously known pData, one for each action inside the config +<li>worker-action-instance - one for each worker thread (called pWrkrData), note +that this is specific to exactly one pData +</ul> +The plugin <b>must</b> now by multi-threading aware. It may be called by multiple +threads concurrently, but it is guaranteed that each call is for a unique +pWrkrData structure. This still permits to write plugins easily, but enables the +engine to work with much higher performance. Note that plugin developers +should assume it is the norm that multiple concurrent worker action instances +are active a the some time. + +<p><b>New required entry points</b></p> +<p>In order to support the new threading model, new entry points are required. +Most importantly, only the plugin knows which data must be present in pData and +pWrkrData, so it must created and destroy these data structures on request of +the engine. Note that pWrkrData may be destroyed at any time and new ones +re-created later. Depending on workload structure and configuration, this can happen +frequently. +<p>New entry points are: +<ul> +<li>createWrkrInstance +<li>freeWrkrInstance +</ul> + +The calling interface for these entry points has changed. Basically, +they now receive a pWrkrData object instead pData. It is assumed that +createWrkrInstance populates pWrkrData->pData appropriately. +<ul> +<li>beginTransaction +<li>doAction +<li>endTransaction +</ul> + +<p><b>RS_RET_SUSPENDED is no longer supported when creating an action instance</b> +<p>This means a plugin must not try to establish any connections or the like +before any of its processing entry points (like beginTransaction or doAction) +is called. This was generally also the case von v7, but was not enforced in +all cases. In v8, creating action creation fails if anything but RS_RET_OK +is returned. + +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body></html> diff --git a/grammar/.gitignore b/grammar/.gitignore new file mode 100644 index 00000000..8bd546bd --- /dev/null +++ b/grammar/.gitignore @@ -0,0 +1 @@ +lexer.c diff --git a/grammar/lexer.l b/grammar/lexer.l index ed5d8a80..7fdb68af 100644 --- a/grammar/lexer.l +++ b/grammar/lexer.l @@ -9,7 +9,7 @@ * cases. So while we hope that cfsysline support can be dropped some time in * the future, we will probably keep these useful constructs. * - * Copyright 2011-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2011-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -129,7 +129,7 @@ int fileno(FILE *stream); <EXPR>0[0-7]+ | /* octal number */ <EXPR>0x[0-7a-f] | /* hex number, following rule is dec; strtoll handles all! */ <EXPR>([1-9][0-9]*|0) { yylval.n = strtoll(yytext, NULL, 0); return NUMBER; } -<EXPR>\$[$!]{0,1}[a-z][!a-z0-9\-_\.]* { yylval.s = strdup(yytext); return VAR; } +<EXPR>\$[$!./]{0,1}[a-z][!a-z0-9\-_\.]* { yylval.s = strdup(yytext+1); return VAR; } <EXPR>\'([^'\\]|\\['"\\$bntr]|\\x[0-9a-f][0-9a-f]|\\[0-7][0-7][0-7])*\' { yytext[yyleng-1] = '\0'; unescapeStr((uchar*)yytext+1, yyleng-2); @@ -176,6 +176,8 @@ int fileno(FILE *stream); <INCL>[^ \t\n]+ { if(cnfDoInclude(yytext) != 0) yyterminate(); BEGIN INITIAL; } +"main_queue"[ \n\t]*"(" { yylval.objType = CNFOBJ_MAINQ; + BEGIN INOBJ; return BEGINOBJ; } "global"[ \n\t]*"(" { yylval.objType = CNFOBJ_GLOBAL; BEGIN INOBJ; return BEGINOBJ; } "template"[ \n\t]*"(" { yylval.objType = CNFOBJ_TPL; @@ -190,6 +192,8 @@ int fileno(FILE *stream); BEGIN INOBJ; return BEGINOBJ; } "module"[ \n\t]*"(" { yylval.objType = CNFOBJ_MODULE; BEGIN INOBJ; return BEGINOBJ; } +"lookup_table"[ \n\t]*"(" { yylval.objType = CNFOBJ_LOOKUP_TABLE; + BEGIN INOBJ; return BEGINOBJ; } "action"[ \n\t]*"(" { BEGIN INOBJ; return BEGIN_ACTION; } ^[ \t]*:\$?[a-z\-]+[ ]*,[ ]*!?[a-z]+[ ]*,[ ]*\"(\\\"|[^\"])*\" { yylval.s = strdup(rmLeadingSpace(yytext)); diff --git a/grammar/parserif.h b/grammar/parserif.h index aa271ec4..21da0950 100644 --- a/grammar/parserif.h +++ b/grammar/parserif.h @@ -6,6 +6,7 @@ int yyparse(); char *cnfcurrfn; void dbgprintf(char *fmt, ...) __attribute__((format(printf, 1, 2))); void parser_errmsg(char *fmt, ...) __attribute__((format(printf, 1, 2))); +void parser_warnmsg(char *fmt, ...) __attribute__((format(printf, 1, 2))); void tellLexEndParsing(void); extern int yydebug; extern int yylineno; @@ -19,5 +20,4 @@ void cnfDoScript(struct cnfstmt *script); void cnfDoCfsysline(char *ln); void cnfDoBSDTag(char *ln); void cnfDoBSDHost(char *ln); -es_str_t *cnfGetVar(char *name, void *usrptr); #endif diff --git a/grammar/rainerscript.c b/grammar/rainerscript.c index a1de6442..0c695934 100644 --- a/grammar/rainerscript.c +++ b/grammar/rainerscript.c @@ -38,6 +38,7 @@ #include "rainerscript.h" #include "conf.h" #include "parserif.h" +#include "parse.h" #include "rsconf.h" #include "grammar.h" #include "queue.h" @@ -46,6 +47,8 @@ #include "obj.h" #include "modules.h" #include "ruleset.h" +#include "msg.h" +#include "unicode-helper.h" DEFobjCurrIf(obj) DEFobjCurrIf(regexp) @@ -144,6 +147,104 @@ getFIOPName(unsigned iFIOP) return pRet; } + +/* This function takes the filter part of a property + * based filter and decodes it. It processes the line up to the beginning + * of the action part. + */ +static rsRetVal +DecodePropFilter(uchar *pline, struct cnfstmt *stmt) +{ + rsParsObj *pPars = NULL; + cstr_t *pCSCompOp = NULL; + cstr_t *pCSPropName = NULL; + int iOffset; /* for compare operations */ + DEFiRet; + + ASSERT(pline != NULL); + + DBGPRINTF("Decoding property-based filter '%s'\n", pline); + + /* create parser object starting with line string without leading colon */ + if((iRet = rsParsConstructFromSz(&pPars, pline+1)) != RS_RET_OK) { + parser_errmsg("error %d constructing parser object", iRet); + ABORT_FINALIZE(iRet); + } + + /* read property */ + iRet = parsDelimCStr(pPars, &pCSPropName, ',', 1, 1, 1); + if(iRet != RS_RET_OK) { + parser_errmsg("error %d parsing filter property", iRet); + rsParsDestruct(pPars); + ABORT_FINALIZE(iRet); + } + CHKiRet(msgPropDescrFill(&stmt->d.s_propfilt.prop, cstrGetSzStrNoNULL(pCSPropName), + cstrLen(pCSPropName))); + + /* read operation */ + iRet = parsDelimCStr(pPars, &pCSCompOp, ',', 1, 1, 1); + if(iRet != RS_RET_OK) { + parser_errmsg("error %d compare operation property - ignoring selector", iRet); + rsParsDestruct(pPars); + ABORT_FINALIZE(iRet); + } + + /* we now first check if the condition is to be negated. To do so, we first + * must make sure we have at least one char in the param and then check the + * first one. + * rgerhards, 2005-09-26 + */ + if(rsCStrLen(pCSCompOp) > 0) { + if(*rsCStrGetBufBeg(pCSCompOp) == '!') { + stmt->d.s_propfilt.isNegated = 1; + iOffset = 1; /* ignore '!' */ + } else { + stmt->d.s_propfilt.isNegated = 0; + iOffset = 0; + } + } else { + stmt->d.s_propfilt.isNegated = 0; + iOffset = 0; + } + + if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "contains", 8)) { + stmt->d.s_propfilt.operation = FIOP_CONTAINS; + } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isequal", 7)) { + stmt->d.s_propfilt.operation = FIOP_ISEQUAL; + } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isempty", 7)) { + stmt->d.s_propfilt.operation = FIOP_ISEMPTY; + } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "startswith", 10)) { + stmt->d.s_propfilt.operation = FIOP_STARTSWITH; + } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "regex", 5)) { + stmt->d.s_propfilt.operation = FIOP_REGEX; + } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "ereregex", 8)) { + stmt->d.s_propfilt.operation = FIOP_EREREGEX; + } else { + parser_errmsg("error: invalid compare operation '%s'", + (char*) rsCStrGetSzStrNoNULL(pCSCompOp)); + ABORT_FINALIZE(RS_RET_ERR); + } + + if(stmt->d.s_propfilt.operation != FIOP_ISEMPTY) { + /* read compare value */ + iRet = parsQuotedCStr(pPars, &stmt->d.s_propfilt.pCSCompValue); + if(iRet != RS_RET_OK) { + parser_errmsg("error %d compare value property", iRet); + rsParsDestruct(pPars); + ABORT_FINALIZE(iRet); + } + } + +finalize_it: + if(pPars != NULL) + rsParsDestruct(pPars); + if(pCSCompOp != NULL) + rsCStrDestruct(&pCSCompOp); + if(pCSPropName != NULL) + cstrDestruct(&pCSPropName); + RETiRet; +} + static void prifiltInvert(struct funcData_prifilt *prifilt) { @@ -1091,7 +1192,11 @@ var2Number(struct var *r, int *bSuccess) n = es_str2num(r->d.estr, bSuccess); } else { if(r->datatype == 'J') { +#ifdef HAVE_JSON_OBJECT_NEW_INT64 + n = (r->d.json == NULL) ? 0 : json_object_get_int64(r->d.json); +#else /* HAVE_JSON_OBJECT_NEW_INT64 */ n = (r->d.json == NULL) ? 0 : json_object_get_int(r->d.json); +#endif /* HAVE_JSON_OBJECT_NEW_INT64 */ } else { n = r->d.n; } @@ -1142,6 +1247,23 @@ var2CString(struct var *r, int *bMustFree) return cstr; } +/* frees struct var members, but not the struct itself. This is because + * it usually is allocated on the stack. Callers why dynamically allocate + * struct var need to free the struct themselfes! + */ +static void +varFreeMembers(struct var *r) +{ + /* Note: we do NOT need to free JSON objects, as we use + * json_object_object_get() to obtain the values, which does not + * increment the reference count. So json_object_put() [free] is + * neither required nor permitted (would free the original object!). + * So for the time being the string data type is the only one that + * we currently need to free. + */ + if(r->datatype == 'S') es_deleteStr(r->d.estr); +} + static rsRetVal doExtractFieldByChar(uchar *str, uchar delim, int matchnbr, uchar **resstr) { @@ -1309,9 +1431,9 @@ doFunc_re_extract(struct cnffunc *func, struct var *ret, void* usrptr) finalize_it: if(bMustFree) free(str); - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); - if(r[2].datatype == 'S') es_deleteStr(r[2].d.estr); - if(r[3].datatype == 'S') es_deleteStr(r[3].d.estr); + varFreeMembers(&r[0]); + varFreeMembers(&r[2]); + varFreeMembers(&r[3]); if(bHadNoMatch) { cnfexprEval(func->expr[4], &r[4], usrptr); @@ -1380,7 +1502,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } ret->datatype = 'S'; if(bMustFree) es_deleteStr(estr); - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + varFreeMembers(&r[0]); free(str); break; case CNFFUNC_TOLOWER: @@ -1391,7 +1513,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) es_tolower(estr); ret->datatype = 'S'; ret->d.estr = estr; - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + varFreeMembers(&r[0]); break; case CNFFUNC_CSTR: cnfexprEval(func->expr[0], &r[0], usrptr); @@ -1400,7 +1522,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) estr = es_strdup(estr); ret->datatype = 'S'; ret->d.estr = estr; - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + varFreeMembers(&r[0]); break; case CNFFUNC_CNUM: if(func->expr[0]->nodetype == 'N') { @@ -1411,7 +1533,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } else { cnfexprEval(func->expr[0], &r[0], usrptr); ret->d.n = var2Number(&r[0], NULL); - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + varFreeMembers(&r[0]); } ret->datatype = 'N'; break; @@ -1429,7 +1551,7 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } ret->datatype = 'N'; if(bMustFree) free(str); - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); + varFreeMembers(&r[0]); break; case CNFFUNC_RE_EXTRACT: doFunc_re_extract(func, ret, usrptr); @@ -1462,9 +1584,9 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) } ret->datatype = 'S'; if(bMustFree) free(str); - if(r[0].datatype == 'S') es_deleteStr(r[0].d.estr); - if(r[1].datatype == 'S') es_deleteStr(r[1].d.estr); - if(r[2].datatype == 'S') es_deleteStr(r[2].d.estr); + varFreeMembers(&r[0]); + varFreeMembers(&r[1]); + varFreeMembers(&r[2]); break; case CNFFUNC_PRIFILT: pPrifilt = (struct funcData_prifilt*) func->funcdata; @@ -1476,6 +1598,19 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) ret->d.n = 1; ret->datatype = 'N'; break; + case CNFFUNC_LOOKUP: +dbgprintf("DDDD: executing lookup\n"); + ret->datatype = 'S'; + if(func->funcdata == NULL) { + ret->d.estr = es_newStrFromCStr("TABLE-NOT-FOUND", sizeof("TABLE-NOT-FOUND")-1); + break; + } + cnfexprEval(func->expr[1], &r[1], usrptr); + str = (char*) var2CString(&r[1], &bMustFree); + ret->d.estr = lookupKey_estr(func->funcdata, (uchar*)str); + if(bMustFree) free(str); + if(r[1].datatype == 'S') es_deleteStr(r[1].d.estr); + break; default: if(Debug) { fname = es_str2cstr(func->fname, NULL); @@ -1491,20 +1626,28 @@ doFuncCall(struct cnffunc *func, struct var *ret, void* usrptr) static inline void evalVar(struct cnfvar *var, void *usrptr, struct var *ret) { + rs_size_t propLen; + uchar *pszProp = NULL; + unsigned short bMustBeFreed = 0; rsRetVal localRet; - es_str_t *estr; struct json_object *json; - if(var->name[0] == '$' && var->name[1] == '!') { - /* TODO: unify string libs */ - estr = es_newStrFromBuf(var->name+1, strlen(var->name)-1); - localRet = msgGetCEEPropJSON((msg_t*)usrptr, estr, &json); - es_deleteStr(estr); + if(var->prop.id == PROP_CEE || + var->prop.id == PROP_LOCAL_VAR || + var->prop.id == PROP_GLOBAL_VAR ) { + localRet = msgGetJSONPropJSON((msg_t*)usrptr, &var->prop, &json); ret->datatype = 'J'; ret->d.json = (localRet == RS_RET_OK) ? json : NULL; + + DBGPRINTF("rainerscript: var %d:%s: '%s'\n", var->prop.id, var->prop.name, + (ret->d.json == NULL) ? "" : json_object_get_string(ret->d.json)); } else { ret->datatype = 'S'; - ret->d.estr = cnfGetVar(var->name, usrptr); + pszProp = (uchar*) MsgGetProp((msg_t*)usrptr, NULL, &var->prop, &propLen, &bMustBeFreed, NULL); + ret->d.estr = es_newStrFromCStr((char*)pszProp, propLen); + DBGPRINTF("rainerscript: var %d: '%s'\n", var->prop.id, pszProp); + if(bMustBeFreed) + free(pszProp); } } @@ -1550,8 +1693,8 @@ evalStrArrayCmp(es_str_t *estr_l, struct cnfarray* ar, int cmpop) } #define FREE_BOTH_RET \ - if(r.datatype == 'S') es_deleteStr(r.d.estr); \ - if(l.datatype == 'S') es_deleteStr(l.d.estr) + varFreeMembers(&r); \ + varFreeMembers(&l) #define COMP_NUM_BINOP(x) \ cnfexprEval(expr->l, &l, usrptr); \ @@ -1578,9 +1721,9 @@ evalStrArrayCmp(es_str_t *estr_l, struct cnfarray* ar, int cmpop) #define FREE_TWO_STRINGS \ if(bMustFree) es_deleteStr(estr_r); \ - if(expr->r->nodetype != 'S' && expr->r->nodetype != 'A' && r.datatype == 'S') es_deleteStr(r.d.estr); \ + if(expr->r->nodetype != 'S' && expr->r->nodetype != 'A') varFreeMembers(&r); \ if(bMustFree2) es_deleteStr(estr_l); \ - if(l.datatype == 'S') es_deleteStr(l.d.estr) + varFreeMembers(&l) /* evaluate an expression. * Note that we try to avoid malloc whenever possible (because of @@ -1631,7 +1774,7 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) if(bMustFree) es_deleteStr(estr_r); } } - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); } } else if(l.datatype == 'J') { estr_l = var2String(&l, &bMustFree); @@ -1653,7 +1796,7 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) if(bMustFree) es_deleteStr(estr_r); } } - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); } if(bMustFree) es_deleteStr(estr_l); } else { @@ -1670,9 +1813,9 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) } else { ret->d.n = (l.d.n == r.d.n); /*CMP*/ } - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); } - if(l.datatype == 'S') es_deleteStr(l.d.estr); + varFreeMembers(&l); break; case CMP_NE: cnfexprEval(expr->l, &l, usrptr); @@ -1900,9 +2043,9 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) ret->d.n = 1ll; else ret->d.n = 0ll; - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); } - if(l.datatype == 'S') es_deleteStr(l.d.estr); + varFreeMembers(&l); break; case AND: cnfexprEval(expr->l, &l, usrptr); @@ -1913,17 +2056,17 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) ret->d.n = 1ll; else ret->d.n = 0ll; - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); } else { ret->d.n = 0ll; } - if(l.datatype == 'S') es_deleteStr(l.d.estr); + varFreeMembers(&l); break; case NOT: cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; ret->d.n = !var2Number(&r, &convok_r); - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); break; case 'N': ret->datatype = 'N'; @@ -1974,7 +2117,7 @@ cnfexprEval(struct cnfexpr *expr, struct var *ret, void* usrptr) cnfexprEval(expr->r, &r, usrptr); ret->datatype = 'N'; ret->d.n = -var2Number(&r, &convok_r); - if(r.datatype == 'S') es_deleteStr(r.d.estr); + varFreeMembers(&r); break; case 'F': doFuncCall((struct cnffunc*) expr, ret, usrptr); @@ -2067,6 +2210,7 @@ cnfexprDestruct(struct cnfexpr *expr) break; case 'V': free(((struct cnfvar*)expr)->name); + msgPropDescrDestruct(&(((struct cnfvar*)expr)->prop)); break; case 'F': cnffuncDestruct((struct cnffunc*)expr); @@ -2284,7 +2428,7 @@ cnfstmtPrintOnly(struct cnfstmt *stmt, int indent, sbool subtree) free(cstr); break; case S_ACT: - doIndent(indent); dbgprintf("ACTION %p [%s:%s]\n", stmt->d.act, + doIndent(indent); dbgprintf("ACTION %d [%s:%s]\n", stmt->d.act->iActionNbr, modGetName(stmt->d.act->pMod), stmt->printable); break; case S_IF: @@ -2325,12 +2469,12 @@ cnfstmtPrintOnly(struct cnfstmt *stmt, int indent, sbool subtree) case S_PROPFILT: doIndent(indent); dbgprintf("PROPFILT\n"); doIndent(indent); dbgprintf("\tProperty.: '%s'\n", - propIDToName(stmt->d.s_propfilt.propID)); - if(stmt->d.s_propfilt.propName != NULL) { - cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + propIDToName(stmt->d.s_propfilt.prop.id)); + if(stmt->d.s_propfilt.prop.id == PROP_CEE || + stmt->d.s_propfilt.prop.id == PROP_LOCAL_VAR || + stmt->d.s_propfilt.prop.id == PROP_GLOBAL_VAR) { doIndent(indent); - dbgprintf("\tCEE-Prop.: '%s'\n", cstr); - free(cstr); + dbgprintf("\tCEE-Prop.: '%s'\n", stmt->d.s_propfilt.prop.name); } doIndent(indent); dbgprintf("\tOperation: "); if(stmt->d.s_propfilt.isNegated) @@ -2437,6 +2581,7 @@ cnfvarNew(char *name) if((var = malloc(sizeof(struct cnfvar))) != NULL) { var->nodetype = 'V'; var->name = name; + msgPropDescrFill(&var->prop, (uchar*)var->name, strlen(var->name)); } return var; } @@ -2490,8 +2635,7 @@ cnfstmtDestruct(struct cnfstmt *stmt) cnfstmtDestructLst(stmt->d.s_prifilt.t_else); break; case S_PROPFILT: - if(stmt->d.s_propfilt.propName != NULL) - es_deleteStr(stmt->d.s_propfilt.propName); + msgPropDescrDestruct(&stmt->d.s_propfilt.prop); if(stmt->d.s_propfilt.regex_cache != NULL) rsCStrRegexDestruct(&stmt->d.s_propfilt.regex_cache); if(stmt->d.s_propfilt.pCSCompValue != NULL) @@ -2576,11 +2720,11 @@ cnfstmtNewPROPFILT(char *propfilt, struct cnfstmt *t_then) if((cnfstmt = cnfstmtNew(S_PROPFILT)) != NULL) { cnfstmt->printable = (uchar*)propfilt; cnfstmt->d.s_propfilt.t_then = t_then; - cnfstmt->d.s_propfilt.propName = NULL; cnfstmt->d.s_propfilt.regex_cache = NULL; cnfstmt->d.s_propfilt.pCSCompValue = NULL; if(DecodePropFilter((uchar*)propfilt, cnfstmt) != RS_RET_OK) { cnfstmt->nodetype = S_NOP; /* disable action! */ + cnfstmtDestructLst(t_then); /* we do no longer need this */ } } return cnfstmt; @@ -3250,6 +3394,13 @@ funcName2ID(es_str_t *fname, unsigned short nParams) return CNFFUNC_INVALID; } return CNFFUNC_PRIFILT; + } else if(!es_strbufcmp(fname, (unsigned char*)"lookup", sizeof("lookup") - 1)) { + if(nParams != 2) { + parser_errmsg("number of parameters for lookup() must be two " + "but is %d.", nParams); + return CNFFUNC_INVALID; + } + return CNFFUNC_LOOKUP; } else { return CNFFUNC_INVALID; } @@ -3314,6 +3465,30 @@ finalize_it: } +static inline rsRetVal +initFunc_lookup(struct cnffunc *func) +{ + uchar *cstr = NULL; + DEFiRet; + + func->funcdata = NULL; + if(func->expr[0]->nodetype != 'S') { + parser_errmsg("table name (param 1) of lookup() must be a constant string"); + FINALIZE; + } + + cstr = (uchar*)es_str2cstr(((struct cnfstringval*) func->expr[0])->estr, NULL); + if((func->funcdata = lookupFindTable(cstr)) == NULL) { + parser_errmsg("lookup table '%s' not found", cstr); + FINALIZE; + } + +finalize_it: + free(cstr); + RETiRet; +} + + struct cnffunc * cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) { @@ -3351,6 +3526,9 @@ cnffuncNew(es_str_t *fname, struct cnffparamlst* paramlst) case CNFFUNC_PRIFILT: initFunc_prifilt(func); break; + case CNFFUNC_LOOKUP: + initFunc_lookup(func); + break; default:break; } } diff --git a/grammar/rainerscript.h b/grammar/rainerscript.h index 06573307..377dde90 100644 --- a/grammar/rainerscript.h +++ b/grammar/rainerscript.h @@ -7,7 +7,6 @@ #include <regex.h> #include "typedefs.h" - #define LOG_NFACILITIES 24 /* current number of syslog facilities */ #define CNFFUNC_MAX_ARGS 32 /**< maximum number of arguments that any function can have (among @@ -25,6 +24,8 @@ enum cnfobjType { CNFOBJ_TPL, CNFOBJ_PROPERTY, CNFOBJ_CONSTANT, + CNFOBJ_MAINQ, + CNFOBJ_LOOKUP_TABLE, CNFOBJ_INVALID = 0 }; @@ -56,6 +57,11 @@ cnfobjType2str(enum cnfobjType ot) case CNFOBJ_CONSTANT: return "constant"; break; + case CNFOBJ_MAINQ: + return "main_queue"; + case CNFOBJ_LOOKUP_TABLE: + return "lookup_table"; + break; default:return "error: invalid cnfobjType"; } } @@ -177,8 +183,7 @@ struct cnfstmt { regex_t *regex_cache;/* cache for compiled REs, if used */ struct cstr_s *pCSCompValue;/* value to "compare" against */ sbool isNegated; - uintTiny propID;/* ID of the requested property */ - es_str_t *propName;/* name of property for CEE-based filters */ + msgPropDescr_t prop; /* requested property */ struct cnfstmt *t_then; struct cnfstmt *t_else; } s_propfilt; @@ -205,6 +210,7 @@ struct cnfstringval { struct cnfvar { unsigned nodetype; char *name; + msgPropDescr_t prop; }; struct cnfarray { @@ -230,7 +236,8 @@ enum cnffuncid { CNFFUNC_RE_MATCH, CNFFUNC_RE_EXTRACT, CNFFUNC_FIELD, - CNFFUNC_PRIFILT + CNFFUNC_PRIFILT, + CNFFUNC_LOOKUP }; struct cnffunc { diff --git a/grammar/testdriver.c b/grammar/testdriver.c index b29626d4..58d204a3 100644 --- a/grammar/testdriver.c +++ b/grammar/testdriver.c @@ -87,15 +87,6 @@ void cnfDoBSDHost(char *ln) dbgprintf("global:BSD host: %s\n", ln); } -es_str_t* -cnfGetVar(char __attribute__((unused)) *name, - void __attribute__((unused)) *usrptr) -{ - es_str_t *estr; - estr = es_newStrFromCStr("", 1); - return estr; -} - int main(int argc, char *argv[]) { diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 9c824c18..9fde0580 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -5,7 +5,7 @@ * * Work originally begun on 2008-02-01 by Rainer Gerhards * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -31,6 +31,8 @@ #include <errno.h> #include <fcntl.h> #include <pthread.h> /* do NOT remove: will soon be done by the module generation macros */ +#include <sys/types.h> +#include <unistd.h> #ifdef HAVE_SYS_STAT_H # include <sys/stat.h> #endif @@ -70,8 +72,14 @@ static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config para #define NUM_MULTISUB 1024 /* default max number of submits */ #define DFLT_PollInterval 10 +#define INIT_FILE_TAB_SIZE 4 /* default file table size - is extended as needed, use 2^x value */ + +/* this structure is used in pure polling mode as well one of the support + * structures for inotify. + */ typedef struct fileInfo_s { uchar *pszFileName; + uchar *pszBasename; /* our files basename part */ uchar *pszTag; size_t lenTag; uchar *pszStateFile; /* file in which state between runs is to be stored */ @@ -81,7 +89,8 @@ typedef struct fileInfo_s { int nRecords; /**< How many records did we process before persisting the stream? */ int iPersistStateInterval; /**< how often should state be persisted? (0=on close only) */ strm_t *pStrm; /* its stream (NULL if not assigned) */ - int readMode; /* which mode to use in ReadMulteLine call? */ + uint8_t readMode; /* which mode to use in ReadMulteLine call? */ + sbool escapeLF; /* escape LF inside the MSG content? */ ruleset_t *pRuleset; /* ruleset to bind listener to (use system default if unspecified) */ ratelimit_t *ratelimiter; multi_submit_t multiSub; @@ -103,6 +112,8 @@ static struct configSettings_s { struct instanceConf_s { uchar *pszFileName; + uchar *pszDirName; + uchar *pszFileBaseName; uchar *pszTag; uchar *pszStateFile; uchar *pszBindRuleset; @@ -110,7 +121,8 @@ struct instanceConf_s { int iPersistStateInterval; int iFacility; int iSeverity; - int readMode; + uint8_t readMode; + sbool escapeLF; int maxLinesAtOnce; ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ struct instanceConf_s *next; @@ -132,8 +144,8 @@ static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current l static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ static int iFilPtr = 0; /* number of files to be monitored; pointer to next free spot during config */ -#define MAX_INPUT_FILES 100 -static fileInfo_t files[MAX_INPUT_FILES]; +static fileInfo_t *files = NULL; +static int currMaxFiles; static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this input */ @@ -156,6 +168,7 @@ static struct cnfparamdescr inppdescr[] = { { "facility", eCmdHdlrFacility, 0 }, { "ruleset", eCmdHdlrString, 0 }, { "readmode", eCmdHdlrInt, 0 }, + { "escapelf", eCmdHdlrBinary, 0 }, { "maxlinesatonce", eCmdHdlrInt, 0 }, { "maxsubmitatonce", eCmdHdlrInt, 0 }, { "persiststateinterval", eCmdHdlrInt, 0 } @@ -168,6 +181,7 @@ static struct cnfparamblk inppblk = #include "im-helper.h" /* must be included AFTER the type definitions! */ + /* enqueue the read file line as a message. The provided string is * not freed - thuis must be done by the caller. */ @@ -295,7 +309,7 @@ static rsRetVal pollFile(fileInfo_t *pThis, int *pbHadFileData) while(glbl.GetGlobalInputTermState() == 0) { if(pThis->maxLinesAtOnce != 0 && nProcessed >= pThis->maxLinesAtOnce) break; - CHKiRet(strm.ReadLine(pThis->pStrm, &pCStr, pThis->readMode)); + CHKiRet(strm.ReadLine(pThis->pStrm, &pCStr, pThis->readMode, pThis->escapeLF)); ++nProcessed; *pbHadFileData = 1; /* this is just a flag, so set it and forget it */ CHKiRet(enqLine(pThis, pCStr)); /* process line */ @@ -341,6 +355,7 @@ createInstance(instanceConf_t **pinst) inst->maxLinesAtOnce = 10240; inst->iPersistStateInterval = 0; inst->readMode = 0; + inst->escapeLF = 1; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -356,6 +371,65 @@ finalize_it: } +/* this function checks instance parameters and does some required pre-processing + * (e.g. split filename in path and actual name) + * Note: we do NOT use dirname()/basename() as they have portability problems. + */ +static rsRetVal +checkInstance(instanceConf_t *inst) +{ + char dirn[MAXFNAME]; + char basen[MAXFNAME]; + int i; + int lenName; + struct stat sb; + int r; + int eno; + char errStr[512]; + DEFiRet; + + lenName = ustrlen(inst->pszFileName); + for(i = lenName ; i >= 0 ; --i) { + if(inst->pszFileName[i] == '/') { + /* found basename component */ + if(i == lenName) + basen[0] = '\0'; + else { + memcpy(basen, inst->pszFileName+i+1, lenName-i); + /* Note \0 is copied above! */ + //basen[(lenName-i+1)+1] = '\0'; + } + break; + } + } + memcpy(dirn, inst->pszFileName, i); /* do not copy slash */ + dirn[i] = '\0'; + CHKmalloc(inst->pszFileBaseName = ustrdup(basen)); + CHKmalloc(inst->pszDirName = ustrdup(dirn)); + + if(dirn[0] == '\0') { + dirn[0] = '/'; + dirn[1] = '\0'; + } + r = stat(dirn, &sb); + if(r != 0) { + eno = errno; + rs_strerror_r(eno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_CONFIG_ERROR, "imfile warning: directory '%s': %s", + dirn, errStr); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + if(!S_ISDIR(sb.st_mode)) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, "imfile warning: configured directory " + "'%s' is NOT a directory", dirn); + ABORT_FINALIZE(RS_RET_CONFIG_ERROR); + } + +finalize_it: + RETiRet; +} + + /* add a new monitor */ static rsRetVal addInstance(void __attribute__((unused)) *pVal, uchar *pNewVal) { @@ -389,6 +463,9 @@ static rsRetVal addInstance(void __attribute__((unused)) *pVal, uchar *pNewVal) inst->maxLinesAtOnce = cs.maxLinesAtOnce; inst->iPersistStateInterval = cs.iPersistStateInterval; inst->readMode = cs.readMode; + inst->escapeLF = 0; + + CHKiRet(checkInstance(inst)); /* reset legacy system */ cs.iPersistStateInterval = 0; @@ -405,33 +482,44 @@ static inline rsRetVal addListner(instanceConf_t *inst) { DEFiRet; + int newMax; + fileInfo_t *newFileTab; fileInfo_t *pThis; - if(iFilPtr < MAX_INPUT_FILES) { - pThis = &files[iFilPtr]; - //TODO: optimize, save strdup? - pThis->pszFileName = (uchar*) strdup((char*) inst->pszFileName); - pThis->pszTag = (uchar*) strdup((char*) inst->pszTag); - pThis->lenTag = ustrlen(pThis->pszTag); - pThis->pszStateFile = (uchar*) strdup((char*) inst->pszStateFile); - - CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); - CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(msg_t*))); - pThis->multiSub.maxElem = inst->nMultiSub; - pThis->multiSub.nElem = 0; - pThis->iSeverity = inst->iSeverity; - pThis->iFacility = inst->iFacility; - pThis->maxLinesAtOnce = inst->maxLinesAtOnce; - pThis->iPersistStateInterval = inst->iPersistStateInterval; - pThis->readMode = inst->readMode; - pThis->pRuleset = inst->pBindRuleset; - pThis->nRecords = 0; - } else { - errmsg.LogError(0, RS_RET_OUT_OF_DESRIPTORS, - "Too many file monitors configured - ignoring %s", - inst->pszFileName); - ABORT_FINALIZE(RS_RET_OUT_OF_DESRIPTORS); + if(iFilPtr == currMaxFiles) { + newMax = 2 * currMaxFiles; + newFileTab = realloc(files, newMax * sizeof(fileInfo_t)); + if(newFileTab == NULL) { + errmsg.LogError(0, RS_RET_OUT_OF_MEMORY, + "cannot alloc memory to monitor file '%s' - ignoring", + inst->pszFileName); + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } + files = newFileTab; + currMaxFiles = newMax; + DBGPRINTF("imfile: increased file table to %d entries\n", currMaxFiles); } + + /* if we reach this point, there is space in the file table for the new entry */ + pThis = &files[iFilPtr]; + pThis->pszFileName = (uchar*) strdup((char*) inst->pszFileName); + pThis->pszTag = (uchar*) strdup((char*) inst->pszTag); + pThis->lenTag = ustrlen(pThis->pszTag); + pThis->pszStateFile = (uchar*) strdup((char*) inst->pszStateFile); + + CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); + CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(msg_t*))); + pThis->multiSub.maxElem = inst->nMultiSub; + pThis->multiSub.nElem = 0; + pThis->iSeverity = inst->iSeverity; + pThis->iFacility = inst->iFacility; + pThis->maxLinesAtOnce = inst->maxLinesAtOnce; + pThis->iPersistStateInterval = inst->iPersistStateInterval; + pThis->readMode = inst->readMode; + pThis->escapeLF = inst->escapeLF; + pThis->pRuleset = inst->pBindRuleset; + pThis->nRecords = 0; + pThis->pStrm = NULL; ++iFilPtr; /* we got a new file to monitor */ resetConfigVariables(NULL, NULL); /* values are both dummies */ @@ -475,7 +563,9 @@ CODESTARTnewInpInst } else if(!strcmp(inppblk.descr[i].name, "facility")) { inst->iFacility = pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "readmode")) { - inst->readMode = pvals[i].val.d.n; + inst->readMode = (uint8_t) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "escapelf")) { + inst->escapeLF = (sbool) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "maxlinesatonce")) { inst->maxLinesAtOnce = pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "persiststateinterval")) { @@ -487,6 +577,7 @@ CODESTARTnewInpInst "param '%s'\n", inppblk.descr[i].name); } } + CHKiRet(checkInstance(inst)); finalize_it: CODE_STD_FINALIZERnewInpInst cnfparamvalsDestruct(pvals, &inppblk); @@ -591,9 +682,15 @@ BEGINactivateCnf instanceConf_t *inst; CODESTARTactivateCnf runModConf = pModConf; + free(files); /* clear any previous instance */ + CHKmalloc(files = (fileInfo_t*) malloc(sizeof(fileInfo_t) * INIT_FILE_TAB_SIZE)); + currMaxFiles = INIT_FILE_TAB_SIZE; + iFilPtr = 0; + for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { addListner(inst); } + /* if we could not set up any listeners, there is no point in running... */ if(iFilPtr == 0) { errmsg.LogError(0, NO_ERRCODE, "imfile: no file monitors could be started, " @@ -610,35 +707,18 @@ CODESTARTfreeCnf for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindRuleset); free(inst->pszFileName); + free(inst->pszDirName); + free(inst->pszFileBaseName); free(inst->pszTag); free(inst->pszStateFile); del = inst; inst = inst->next; free(del); } + free(files); ENDfreeCnf - -/* This function is the cancel cleanup handler. It is called when rsyslog decides the - * module must be stopped, what most probably happens during shutdown of rsyslogd. When - * this function is called, the runInput() function (below) is already terminated - somewhere - * in the middle of what it was doing. The cancel cleanup handler below should take - * care of any locked mutexes and such, things that really need to be cleaned up - * before processing continues. In general, many plugins do not need to provide - * any code at all here. - * - * IMPORTANT: the calling interface of this function can NOT be modified. It actually is - * called by pthreads. The provided argument is currently not being used. - */ -static void -inputModuleCleanup(void __attribute__((unused)) *arg) -{ - BEGINfunc - ENDfunc -} - - /* This function is called by the framework to gather the input. The module stays * most of its lifetime inside this function. It MUST NEVER exit this function. Doing * so would end module processing and rsyslog would NOT reschedule the module. If @@ -659,12 +739,10 @@ inputModuleCleanup(void __attribute__((unused)) *arg) * On spamming the main queue: keep in mind that it will automatically rate-limit * ourselfes if we begin to overrun it. So we really do not need to care here. */ -#pragma GCC diagnostic ignored "-Wempty-body" BEGINrunInput int i; int bHadFileData; /* were there at least one file with data during this run? */ CODESTARTrunInput - pthread_cleanup_push(inputModuleCleanup, NULL); while(glbl.GetGlobalInputTermState() == 0) { do { bHadFileData = 0; @@ -684,13 +762,8 @@ CODESTARTrunInput } DBGPRINTF("imfile: terminating upon request of rsyslog core\n"); - pthread_cleanup_pop(0); /* just for completeness, but never called... */ RETiRet; /* use it to make sure the housekeeping is done! */ ENDrunInput -#pragma GCC diagnostic warning "-Wempty-body" - /* END no-touch zone * - * ------------------------------------------------------------------------------------------ */ - /* The function is called by rsyslog before runInput() is called. It is a last chance @@ -860,8 +933,7 @@ std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, insta * complexity of processing is depending on the actual module. However, only * thing absolutely necessary should be done here. Actual app-level processing * is to be performed in runInput(). A good sample of what to do here may be to - * set some variable defaults. The most important thing probably is registration - * of config command handlers. + * set some variable defaults. */ BEGINmodInit() CODESTARTmodInit diff --git a/plugins/impstats/impstats.c b/plugins/impstats/impstats.c index 47378445..a883ef1b 100644 --- a/plugins/impstats/impstats.c +++ b/plugins/impstats/impstats.c @@ -1,7 +1,7 @@ /* impstats.c * A module to periodically output statistics gathered by rsyslog. * - * Copyright 2010-2012 Adiscon GmbH. + * Copyright 2010-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -32,6 +32,9 @@ #if defined(__FreeBSD__) #include <sys/stat.h> #endif +#include <errno.h> +#include <sys/time.h> +#include <sys/resource.h> #include "dirty.h" #include "cfsysline.h" @@ -43,6 +46,7 @@ #include "glbl.h" #include "statsobj.h" #include "prop.h" +#include "ruleset.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -59,6 +63,7 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(prop) DEFobjCurrIf(statsobj) DEFobjCurrIf(errmsg) +DEFobjCurrIf(ruleset) typedef struct configSettings_s { int iStatsInterval; @@ -74,10 +79,13 @@ struct modConfData_s { int iFacility; int iSeverity; int logfd; /* fd if logging to file, or -1 if closed */ + ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ statsFmtType_t statsFmt; sbool bLogToSyslog; + sbool bResetCtrs; char *logfile; sbool configSetViaV2Method; + uchar *pszBindRuleset; /* name of ruleset to bind to */ }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ @@ -92,8 +100,10 @@ static struct cnfparamdescr modpdescr[] = { { "facility", eCmdHdlrInt, 0 }, { "severity", eCmdHdlrInt, 0 }, { "log.syslog", eCmdHdlrBinary, 0 }, + { "resetcounters", eCmdHdlrBinary, 0 }, { "log.file", eCmdHdlrGetWord, 0 }, - { "format", eCmdHdlrGetWord, 0 } + { "format", eCmdHdlrGetWord, 0 }, + { "ruleset", eCmdHdlrString, 0 } }; static struct cnfparamblk modpblk = { CNFPARAMBLK_VERSION, @@ -101,6 +111,19 @@ static struct cnfparamblk modpblk = modpdescr }; + +/* resource use stats counters */ +static intctr_t st_ru_utime; +static intctr_t st_ru_stime; +static int st_ru_maxrss; +static int st_ru_minflt; +static int st_ru_majflt; +static int st_ru_inblock; +static int st_ru_oublock; +static int st_ru_nvcsw; +static int st_ru_nivcsw; +static statsobj_t *statsobj_resources; + BEGINmodExit CODESTARTmodExit prop.Destruct(&pInputName); @@ -109,6 +132,7 @@ CODESTARTmodExit objRelease(prop, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(statsobj, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); ENDmodExit @@ -144,6 +168,7 @@ doSubmitMsg(uchar *line) MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); MsgSetRcvFromIP(pMsg, glbl.GetLocalHostIP()); MsgSetMSGoffs(pMsg, 0); + MsgSetRuleset(pMsg, runModConf->pBindRuleset); MsgSetTAG(pMsg, UCHAR_CONSTANT("rsyslogd-pstats:"), sizeof("rsyslogd-pstats:") - 1); pMsg->iFacility = runModConf->iFacility; pMsg->iSeverity = runModConf->iSeverity; @@ -222,7 +247,23 @@ doStatsLine(void __attribute__((unused)) *usrptr, cstr_t *cstr) static inline void generateStatsMsgs(void) { - statsobj.GetAllStatsLines(doStatsLine, NULL, runModConf->statsFmt); + struct rusage ru; + int r; + r = getrusage(RUSAGE_SELF, &ru); + if(r != 0) { + dbgprintf("impstats: getrusage() failed with error %d, zeroing out\n", errno); + memset(&ru, 0, sizeof(ru)); + } + st_ru_utime = ru.ru_utime.tv_sec * 1000000 + ru.ru_utime.tv_usec; + st_ru_stime = ru.ru_stime.tv_sec * 1000000 + ru.ru_stime.tv_usec; + st_ru_maxrss = ru.ru_maxrss; + st_ru_minflt = ru.ru_minflt; + st_ru_majflt = ru.ru_majflt; + st_ru_inblock = ru.ru_inblock; + st_ru_oublock = ru.ru_oublock; + st_ru_nvcsw = ru.ru_nvcsw; + st_ru_nivcsw = ru.ru_nivcsw; + statsobj.GetAllStatsLines(doStatsLine, NULL, runModConf->statsFmt, runModConf->bResetCtrs); } @@ -238,7 +279,9 @@ CODESTARTbeginCnfLoad loadModConf->statsFmt = statsFmt_Legacy; loadModConf->logfd = -1; loadModConf->logfile = NULL; + loadModConf->pszBindRuleset = NULL; loadModConf->bLogToSyslog = 1; + loadModConf->bResetCtrs = 0; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config vars */ initConfigSettings(); @@ -273,6 +316,8 @@ CODESTARTsetModCnf loadModConf->iSeverity = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "log.syslog")) { loadModConf->bLogToSyslog = (sbool) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "resetcounters")) { + loadModConf->bResetCtrs = (sbool) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "log.file")) { loadModConf->logfile = es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "format")) { @@ -288,6 +333,8 @@ CODESTARTsetModCnf mode); } free(mode); + } else if(!strcmp(modpblk.descr[i].name, "ruleset")) { + loadModConf->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { dbgprintf("impstats: program error, non-handled " "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); @@ -321,6 +368,30 @@ CODESTARTendCnfLoad ENDendCnfLoad +/* we need our special version of checkRuleset(), as we do not have any instances */ +static inline rsRetVal +checkRuleset(modConfData_t *modConf) +{ + ruleset_t *pRuleset; + rsRetVal localRet; + DEFiRet; + + modConf->pBindRuleset = NULL; /* assume default ruleset */ + + if(modConf->pszBindRuleset == NULL) + FINALIZE; + + localRet = ruleset.GetRuleset(modConf->pConf, &pRuleset, modConf->pszBindRuleset); + if(localRet == RS_RET_NOT_FOUND) { + errmsg.LogError(0, NO_ERRCODE, "impstats: ruleset '%s' not found - " + "using default ruleset instead", modConf->pszBindRuleset); + } + CHKiRet(localRet); + modConf->pBindRuleset = pRuleset; +finalize_it: + RETiRet; +} + BEGINcheckCnf CODESTARTcheckCnf if(pModConf->iStatsInterval == 0) { @@ -328,6 +399,7 @@ CODESTARTcheckCnf "default of %d seconds", DEFAULT_STATS_PERIOD); pModConf->iStatsInterval = DEFAULT_STATS_PERIOD; } + iRet = checkRuleset(pModConf); ENDcheckCnf @@ -335,15 +407,41 @@ BEGINactivateCnf rsRetVal localRet; CODESTARTactivateCnf runModConf = pModConf; - DBGPRINTF("impstats: stats interval %d seconds, logToSyslog %d, logFile %s\n", - runModConf->iStatsInterval, runModConf->bLogToSyslog, + DBGPRINTF("impstats: stats interval %d seconds, reset %d, logToSyslog %d, logFile %s\n", + runModConf->iStatsInterval, runModConf->bResetCtrs, runModConf->bLogToSyslog, runModConf->logfile == NULL ? "deactivated" : (char*)runModConf->logfile); localRet = statsobj.EnableStats(); if(localRet != RS_RET_OK) { errmsg.LogError(0, localRet, "impstats: error enabling statistics gathering"); ABORT_FINALIZE(RS_RET_NO_RUN); } + /* initialize our own counters */ + CHKiRet(statsobj.Construct(&statsobj_resources)); + CHKiRet(statsobj.SetName(statsobj_resources, (uchar*)"resource-usage")); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("utime"), + ctrType_IntCtr, CTR_FLAG_NONE, &st_ru_utime)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("stime"), + ctrType_IntCtr, CTR_FLAG_NONE, &st_ru_stime)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("maxrss"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_maxrss)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("minflt"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_minflt)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("majflt"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_majflt)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("inblock"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_inblock)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("oublock"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_oublock)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("nvcsw"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_nvcsw)); + CHKiRet(statsobj.AddCounter(statsobj_resources, UCHAR_CONSTANT("nivcsw"), + ctrType_Int, CTR_FLAG_NONE, &st_ru_nivcsw)); + CHKiRet(statsobj.ConstructFinalize(statsobj_resources)); finalize_it: + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "impstats: error activating module"); + iRet = RS_RET_NO_RUN; + } ENDactivateCnf @@ -408,6 +506,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(statsobj, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); /* the pstatsinverval is an alias to support a previous screwed-up syntax... */ CHKiRet(regCfSysLineHdlr2((uchar *)"pstatsinterval", 0, eCmdHdlrInt, NULL, &cs.iStatsInterval, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); CHKiRet(regCfSysLineHdlr2((uchar *)"pstatinterval", 0, eCmdHdlrInt, NULL, &cs.iStatsInterval, STD_LOADABLE_MODULE_ID, &bLegacyCnfModGlobalsPermitted)); diff --git a/plugins/impstats/pygal-tooltips.js b/plugins/impstats/pygal-tooltips.js new file mode 100644 index 00000000..7b183de1 --- /dev/null +++ b/plugins/impstats/pygal-tooltips.js @@ -0,0 +1,121 @@ +<html>
+<head><title>301 Moved Permanently</title></head>
+<body bgcolor="white">
+<center><h1>301 Moved Permanently</h1></center>
+<hr><center>nginx</center>
+</body>
+</html>
+// Generated by CoffeeScript 1.6.3 +(function() { + var get_translation, init, padding, r_translation, tooltip_timeout; + + padding = 5; + + tooltip_timeout = 0; + + r_translation = /translate\((\d+)[ ,]+(\d+)\)/; + + get_translation = function($elt) { + return (r_translation.exec($elt.attr('transform')) || []).slice(1); + }; + + init = function(ctx) { + var tooltip, untooltip; + $vg('.text-overlay .series', ctx).hide(); + $vg('.reactive', ctx).hover((function() { + return $vg(this).addClass('active'); + }), (function() { + return $vg(this).removeClass('active'); + })); + $vg('.activate-serie', ctx).hover((function() { + var num; + num = this.id.replace('activate-serie-', ''); + $vg('.text-overlay .serie-' + num, ctx).show(); + return $vg('.serie-' + num + ' .reactive', ctx).addClass('active'); + }), function() { + var num; + num = this.id.replace('activate-serie-', ''); + $vg('.text-overlay .serie-' + num, ctx).hide(); + return $vg('.serie-' + num + ' .reactive', ctx).removeClass('active'); + }); + $vg('.tooltip-trigger', ctx).hover((function() { + return tooltip($vg(this)); + }), (function() { + return untooltip(); + })); + tooltip = function($elt) { + var $label, $rect, $text, $tooltip, $value, current_x, current_y, h, target, w, x, x_elt, xlink, y, y_elt, _ref; + clearTimeout(tooltip_timeout); + $tooltip = $vg('#tooltip', ctx).css({ + opacity: 1 + }); + $text = $tooltip.find('text'); + $label = $tooltip.find('tspan.label'); + $value = $tooltip.find('tspan.value'); + $rect = $tooltip.find('rect'); + if ($elt.siblings('.tooltip').size()) { + $label.text($elt.siblings('.tooltip').text()); + $value.text(''); + } else { + $label.text($elt.siblings('.label').text()); + $value.text($elt.siblings('.value').text()); + } + xlink = $elt.siblings('.xlink').text() || null; + target = $elt.parent().attr('target'); + if (xlink) { + $tooltip.find('a').attr('href', xlink).attr('target', target); + } + $text.attr('x', padding); + $text.attr('y', padding + this.config.tooltip_font_size); + $value.attr('x', padding); + $value.attr('dy', $label.text() ? this.config.tooltip_font_size + padding : 0); + w = $text.width() + 2 * padding; + h = $text.height() + 2 * padding; + $rect.attr('width', w); + $rect.attr('height', h); + x_elt = $elt.siblings('.x'); + y_elt = $elt.siblings('.y'); + x = parseInt(x_elt.text()); + if (x_elt.hasClass('centered')) { + x -= w / 2; + } else if (x_elt.hasClass('left')) { + x -= w; + } + y = parseInt(y_elt.text()); + if (y_elt.hasClass('centered')) { + y -= h / 2; + } else if (y_elt.hasClass('top')) { + y -= h; + } + _ref = get_translation($tooltip), current_x = _ref[0], current_y = _ref[1]; + if (current_x === x && current_y === y) { + return; + } + return $tooltip.attr('transform', "translate(" + x + " " + y + ")"); + }; + return untooltip = function() { + return tooltip_timeout = setTimeout((function() { + return $vg('#tooltip', ctx).css({ + opacity: 0 + }); + }), 1000); + }; + }; + + this.init_svg = function(ctx) { + return init($vg(ctx)); + }; + + $vg(function() { + var $charts; + $charts = $vg('.pygal-chart'); + if ($charts.size()) { + return $charts.each(function() { + return init_svg(this); + }); + } else { + return init(); + } + }); + +}).call(this); diff --git a/plugins/impstats/statslog-analyzer.py b/plugins/impstats/statslog-analyzer.py new file mode 100644 index 00000000..51f74a9a --- /dev/null +++ b/plugins/impstats/statslog-analyzer.py @@ -0,0 +1,267 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- + +# * Copyright (C) 2013 Adiscon GmbH. +# * This file is part of RSyslog +# * +# * This script processes impstats logfiles and searches for abnormalities +# * + +import sys +import datetime +import time +import os + +# Include regex definitions +import statslog_regex +from statslog_regex import * + +# Set default variables +szInput = "rsyslog-stats.log" +bHelpOutput = False +nEvictedAlarm = 5 # In Percent, when this specific amount of requests raises the evicted counter, a problem is reported. +nFailedAlarm = 1 # Number of fails to occur until a problem is reported. +nDiscardedAlarm = 1 # Number of discards to occur until a problem is reported. + +# Helper variables +nLogLineNum = 0 +nLogFileCount = 0 +outputData = {} + +# ArrayID's in Logdata +LN_DATE = 0 +LN_HOST = 1 +LN_LINENUM = 2 +LN_DATA = 3 + +# Open Errorlog on init +errorlog = open("statslog-analyzer.corrupted.log", 'w') + +# Process Arguments +for arg in sys.argv: # [-4:]: + if arg.find("--input=") != -1: + szInput = arg[8:] + elif arg.find("--evictedalarm=") != -1: + nEvictedAlarm = int(arg[15:]) + if nEvictedAlarm < 0 or nEvictedAlarm > 100: + nEvictedAlarm = 5 # Resett to default if value is invalid + elif arg.find("--failedalarm=") != -1: + nFailedAlarm = int(arg[14:]) + elif arg.find("--discardedalarm=") != -1: + nDiscardedAlarm = int(arg[17:]) + elif arg.find("--h") != -1 or arg.find("-h") != -1 or arg.find("--help") != -1: + bHelpOutput = True + +if bHelpOutput: + print "\n\nStatslog-analyzer command line options:" + print "=======================================" + print " --input=<filename> Contains the path and filename of your impstats logfile. " + print " Default is 'rsyslog-stats.log' \n" + print " --evictedalarm=<number> Default is 5 which is in percent." + print " So the value can be between 0 and 100.\n" + print " --failedalarm=<number> Number of fails that have to occur before raising an alarm.\n" + print " --discardedalarm=<number> Number of discards that have to occur before raising an alarm.\n" + print " --h / -h / --help Displays this help message. \n" + print "\n Sampleline: ./statslog-analyzer.py --input=rsyslog-stats.log --evictedalarm=5 --failedalarm=1 --discardedalarm=1" +else: + print " Start sorting impstats file '" + szInput+ "'" + + # Open inout file + inputfile = open(szInput, 'r') + for line in inputfile.readlines(): + if line.find("rsyslogd-pstats") != -1: + # Init variables + aData = {} + iLogRegExIndex = 0 + bLogProcessed = False + + # Loop through Regex parsers + for loglineregex in loglineregexes: + # Parse IMPStats Line! + result = loglineregex.split(line) + # Found valid logline, save into file! + if len(result) >= loglineindexes[iLogRegExIndex]["LN_LOGDATA"] and result[ loglineindexes[iLogRegExIndex]["LN_SYSLOGTAG"] ] == "rsyslogd-pstats": + # Convert Datetime! + try: + iMonth = int( result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] ) + filedate = datetime.datetime.strptime(result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] + " " + str(datetime.datetime.now().year) + " " + result[ loglineindexes[iLogRegExIndex]["LN_DAY"] ] + " " + result[ loglineindexes[iLogRegExIndex]["LN_TIME"] ] ,"%m %Y %d %H:%M:%S") + except ValueError: + filedate = datetime.datetime.strptime(result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] + " " + str(datetime.datetime.now().year) + " " + result[ loglineindexes[iLogRegExIndex]["LN_DAY"] ] + " " + result[ loglineindexes[iLogRegExIndex]["LN_TIME"] ] ,"%b %Y %d %H:%M:%S") + + # Split logdata into Array + aProperties = result[ loglineindexes[iLogRegExIndex]["LN_LOGDATA"] ].split(" ") + for szProperty in aProperties: + aProperty = szProperty.split("=") + if len(aProperty) > 1: + # aData.append(aProperty[1]) # Append FieldData + aData[aProperty[0]] = aProperty[1] + else: + errorlog.write("Corrupted Logline at line " + str(nLogLineNum) + " failed to parse: " + line) + break + + # Remove invalid characters from StatsID! +# szStatsID = re.sub("[^a-zA-Z0-9]", "_", result[ loglineindexes[iLogRegExIndex]["LN_LOGOBJECT"] ]) + szStatsID = result[ loglineindexes[iLogRegExIndex]["LN_LOGOBJECT"] ] + + # Open Statsdata per ID + if szStatsID not in outputData: + outputData[szStatsID] = [] + + # Add data into array + outputData[szStatsID].append( [ filedate.strftime("%Y/%b/%d %H:%M:%S"), + result[ loglineindexes[iLogRegExIndex]["LN_HOST"] ], + nLogLineNum, + aData ] ) + + # Set Log as processed, abort Loop! + bLogProcessed = True + + # Increment Logreged Counter + iLogRegExIndex += 1 + + # Abort Loop if processed + if bLogProcessed: + break + + # Debug output if format not detected + if bLogProcessed == False: + print "Fail parsing logline: " + print result + + # Increment helper counter + nLogLineNum += 1 + + # Close input file + inputfile.close() + + # Data sorted, now analyze data! + print " Sorting finished with '" + str(nLogLineNum) + "' loglines processed." + print " Start analyzing logdata now" + + #print outputData + #sys.exit(0) + + # Init variables + aPossibleProblems = {} + + # Loop through data + for szSingleStatsID in outputData: + # Init variables + aProblemFound = {} + bEvictedMsgs = False + bFailedMsgs = False + aPossibleProblems[szSingleStatsID] = [] + iLineNum = 0 + szLineDate = "" + szHost = "" + + # Loop through loglines + for singleStatLine in outputData[szSingleStatsID]: + iLineNum = singleStatLine[LN_LINENUM] + szLineDate = singleStatLine[LN_DATE] + szHost = singleStatLine[LN_HOST] + + # Init helper values + iLogEvictedData = 0 + iLogRequestsData = 0 + + # Loop through data + for logDataID in singleStatLine[LN_DATA]: + if logDataID.find("discarded") != -1 or logDataID.find("failed") != -1: + iLogData = int(singleStatLine[LN_DATA][logDataID]) + if iLogData > 0: + if logDataID not in aProblemFound: + # Init Value + aProblemFound[logDataID] = {} + aProblemFound[logDataID]['type'] = logDataID + aProblemFound[logDataID]['value'] = iLogData + aProblemFound[logDataID]['startline'] = iLineNum + aProblemFound[logDataID]['startdate'] = szLineDate + elif aProblemFound[logDataID]['value'] > iLogData: + # Logdata was resetted + aProblemFound[logDataID]['endline'] = iLineNum + aProblemFound[logDataID]['enddate'] = szLineDate + + # Add to possible Problems Array + if logDataID.find("discarded") != -1: + if aProblemFound[logDataID]['value'] >= nDiscardedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound[logDataID] ) + elif logDataID.find("failed") != -1: + if aProblemFound[logDataID]['value'] >= nFailedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound[logDataID] ) + + # Reinit + del aProblemFound[logDataID] + else: + aProblemFound[logDataID]['value'] = int(singleStatLine[LN_DATA][logDataID]) + aProblemFound[logDataID]['endline'] = iLineNum + aProblemFound[logDataID]['enddate'] = szLineDate + elif logDataID.find("evicted") != -1 or logDataID.find("requests") != -1: + iLogData = int(singleStatLine[LN_DATA][logDataID]) + if "omfile" not in aProblemFound: + # Init Value + aProblemFound['omfile'] = {} + aProblemFound['omfile']['type'] = "omfile" + aProblemFound['omfile']['evicted'] = 0 # Init value + aProblemFound['omfile']['requests'] = 0 # Init value + aProblemFound['omfile'][logDataID] = iLogData # Set value + aProblemFound['omfile']['startline'] = iLineNum + aProblemFound['omfile']['startdate'] = szLineDate + elif aProblemFound['omfile'][logDataID] > iLogData: + # Logdata was resetted + aProblemFound['omfile']['endline'] = iLineNum + aProblemFound['omfile']['enddate'] = szLineDate + + # Add to possible Problems Array if exceeds alarm level + if (aProblemFound['omfile']['evicted'] / (aProblemFound['omfile']['requests']/100)) >= nEvictedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound['omfile'] ) + + # Reinit + del aProblemFound['omfile'] + else: + aProblemFound['omfile'][logDataID] = int(singleStatLine[LN_DATA][logDataID]) + aProblemFound['omfile']['endline'] = iLineNum + aProblemFound['omfile']['enddate'] = szLineDate + + #print logDataID + "=" + singleStatLine[LN_DATA][logDataID] + + # Finish possible problems array + if len(aProblemFound) > 0: + for szProblemID in aProblemFound: + if aProblemFound[szProblemID]['type'].find('omfile') != -1: + # Only add if exceeds percentage + if (aProblemFound[szProblemID]['evicted'] / (aProblemFound[szProblemID]['requests']/100)) >= nEvictedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound[szProblemID] ) + elif aProblemFound[szProblemID]['type'].find('failed') != -1: + # Only add if exceeds Alarm Level + if aProblemFound[szProblemID]['value'] >= nFailedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound[szProblemID] ) + elif aProblemFound[szProblemID]['type'].find('discarded') != -1: + # Only add if exceeds Alarm Level + if aProblemFound[szProblemID]['value'] >= nDiscardedAlarm: + aPossibleProblems[szSingleStatsID].append( aProblemFound[szProblemID] ) + else: + # Add to problem array! + aPossibleProblems[szSingleStatsID].append( aProblemFound[szProblemID] ) + + # Debug break +# sys.exit(0) + + + print " End analyzing logdata, showing results now:" + + # Output all found problems! + for szProblemID in aPossibleProblems: + if len(aPossibleProblems[szProblemID]) > 0: + print "\n Problems found for Counter '" + szProblemID + "': " + for aProblem in aPossibleProblems[szProblemID]: + if aProblem['type'].find("discarded") != -1: + print " - " + str(aProblem['value']) + " Messages were discarded (" + aProblem['type'] + ") between line " + str(aProblem['startline']) + " and " + str(aProblem['endline']) + " (Startdate: '" + str(aProblem['startdate']) + "' Enddate: '" + str(aProblem['enddate']) + "') " + elif aProblem['type'].find("failed") != -1: + print " - " + str(aProblem['value']) + " Messages failed (" + aProblem['type'] + ") between line " + str(aProblem['startline']) + " and " + str(aProblem['endline']) + " (Startdate: '" + str(aProblem['startdate']) + "' Enddate: '" + str(aProblem['enddate']) + "') " + elif aProblem['type'].find("omfile") != -1: + nEvictedInPercent = aProblem['evicted'] / (aProblem['requests']/100) + print " - " + str(nEvictedInPercent) + "% File Requests were evicted ('" + str(aProblem['requests']) + "' requests, '" + str(aProblem['evicted']) + "' evicted) between line " + str(aProblem['startline']) + " and " + str(aProblem['endline']) + " (Startdate: '" + str(aProblem['startdate']) + "' Enddate: '" + str(aProblem['enddate']) + "') " + + # Finished + print "\n\n" diff --git a/plugins/impstats/statslog-graph.py b/plugins/impstats/statslog-graph.py new file mode 100755 index 00000000..a154d4b9 --- /dev/null +++ b/plugins/impstats/statslog-graph.py @@ -0,0 +1,245 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- + +# * Copyright (C) 2013 Adiscon GmbH. +# * This file is part of RSyslog +# * +# * This script processes csv stats logfiles created by statslog-splitter.py and creates graphs +# * Dependecies: - python pip -> Needed to install python packages +# * - python cairosvg -> Needed for PNG converting support! +# * - Install python packages using this command: +# * pip install CairoSVG tinycss cssselect pygal +# * + +import sys +import datetime +import time +import os +import pygal + +# Set default variables +szInput = "" +szOutputFile = "" +bHelpOutput = False +nMaxDataCount = 25 +bUseDateTime = True +bLineChart = True +bBarChart = False +bConvertPng = False +bLogarithmicChart = False +bFilledLineChart = False +bChartCalcDelta = False + +# Init variables +aFields = [] +aData = {} +aMajorXData = [] + +# Helper variables +nDataRecordCound = 0 +nLineCount = 0 +iStartSeconds = 0 + + +# Process Arguments +for arg in sys.argv: # [-4:]: + if arg.find("--input=") != -1: + szInput = arg[8:] + elif arg.find("--outputfile=") != -1: + szOutputFile = arg[13:] + elif arg.find("--maxdataxlabel=") != -1: + nMaxDataCount = int(arg[16:]) + elif arg.find("--xlabeldatetime") != -1: + bUseDateTime = True + elif arg.find("--xlabelseconds") != -1: + bUseDateTime = False + elif arg.find("--convertpng") != -1: + bConvertPng = True + elif arg.find("--linechart") != -1: + bLineChart = True + bBarChart = False + elif arg.find("--barchart") != -1: + bLineChart = False + bBarChart = True + elif arg.find("--logarithmic") != -1: + bLogarithmicChart = True + elif arg.find("--filledlinechart") != -1: + bFilledLineChart = True + elif arg.find("--chartscalcdelta") != -1: + bChartCalcDelta = True + elif arg.find("--h") != -1 or arg.find("-h") != -1 or arg.find("--help") != -1: + bHelpOutput = True + +if bHelpOutput == True: + print "\n\nStatslog-graph command line options:" + print "=======================================" + print " --input=<filename> Contains the path and filename of your impstats logfile. " + print " Default is 'rsyslog-stats.log' \n" + print " --outputfile=<dir> Output directory and file to be used. " + print " Default is '" + szOutputFile + "'. " + print " --maxdataxlabel=<num> Max Number of data shown on the x-label." + print " Default is 25 label entries." + print " --xlabeldatetime Use Readable Datetime for x label data. (Cannot be used with --xlabelseconds)" + print " Default is enabled." + print " --xlabelseconds Use seconds instead of datetime, starting at 0. (Cannot be used with --xlabeldatetime)" + print " Default is disabled." + print " --linechart Generates a Linechart (Default chart mode) (Cannot be used with --barchart)" + print " --barchart Generates a Barchart (Cannot be used with --linechart)" + print " --logarithmic Uses Logarithmic to scale the Y Axis, maybe useful in some cases. Default is OFF" + print " --filledlinechart Use filled lines on Linechart, maybe useful in some cases. Default is OFF" + print " --chartscalcdelta If set, charts will use calculated delta values instead of cumulative values." + print " --convertpng Generate PNG Output rather than SVG. " + print " Default is SVG output." + print " --h / -h / --help Displays this help message. \n" + print "\n Sampleline: ./statslog-graph.py --input=imuxsock.csv --outputfile=/home/user/csvgraphs/imuxsock.svg" +else: + # Generate output filename + if len(szInput) > 0: + # Only set output filename if not specified + if len(szOutputFile) == 0: + if szInput.rfind(".") == -1: + szOutputFile += szInput + ".svg" + else: + szOutputFile += szInput[:-4] + ".svg" + else: + print "Error, no input file specified!" + sys.exit(0) + + # Process inputfile + inputfile = open(szInput, 'r') + aLineDataPrev = [] # Helper variable for previous line! + for line in inputfile.readlines(): + if nLineCount == 0: + aFields = line.strip().split(";") + # remove last item if empty + if len(aFields[len(aFields)-1]) == 0: + aFields.pop() + #print aFields + #sys.exit(0) + + #Init data arrays + for field in aFields: + aData[field] = [] + else: + aLineData = line.strip().split(";") + # remove last item if empty + if len(aLineData[len(aLineData)-1]) == 0: + aLineData.pop() + + # Loop Through line data + iFieldNum = 0 + for field in aFields: + if iFieldNum == 0: + if bUseDateTime: + aData[field].append( datetime.datetime.strptime(aLineData[iFieldNum],"%Y/%b/%d %H:%M:%S") ) + else: + # Convert Time String into UNIX Timestamp + myDateTime = datetime.datetime.strptime(aLineData[iFieldNum],"%Y/%b/%d %H:%M:%S") + iTimeStamp = int(time.mktime(myDateTime.timetuple())) + + # Init Start Seconds + if iStartSeconds == 0: + iStartSeconds = iTimeStamp + + # Set data field + aData[field].append( iTimeStamp - iStartSeconds ) + elif iFieldNum > 2: + # Check if we need to calculate Deltas! + if bChartCalcDelta and len(aLineDataPrev) > 0: + iPreviousVal = int(aLineDataPrev[iFieldNum]) + iCurrentVal = int(aLineData[iFieldNum]) + if iCurrentVal != 0: # Calc DELTA + aData[field].append(iCurrentVal - iPreviousVal) + else: # Don't Calc delta value! + aData[field].append( iCurrentVal ) + else: + aData[field].append( int(aLineData[iFieldNum]) ) + else: + aData[field].append( aLineData[iFieldNum] ) + + iFieldNum += 1 + # print aData[field[nLineCount]] + + # Increment counter + nDataRecordCound += 1 + + # in case deltas need to be calculated, Store current line into previous line + if bChartCalcDelta: + aLineDataPrev = aLineData + + # Increment counter + nLineCount += 1 + +# if nLineCount > 25: +# break + +# if nMaxDataCount > 0: +# # Check if we need to reduce the data amount +# nTotalDataCount = len( aData[aFields[0]] ) +# nDataStepCount = nTotalDataCount / (nMaxDataCount) +# if nTotalDataCount > nMaxDataCount: +# for iDataNum in reversed(range(0, nTotalDataCount)): +# # Remove all entries who +# if iDataNum % nDataStepCount == 0: +# aMajorXData.append( aData[aFields[0]][iDataNum] ) + + # Import Style +# from pygal.style import LightSolarizedStyle + + # Create Config object + from pygal import Config + chartCfg = Config() + chartCfg.show_legend = True + chartCfg.human_readable = True + chartCfg.pretty_print=True + if bFilledLineChart: + chartCfg.fill = True + else: + chartCfg.fill = False + chartCfg.x_scale = 1 + chartCfg.y_scale = 1 + chartCfg.x_label_rotation = 45 + chartCfg.include_x_axis = True + chartCfg.show_dots=False + if nMaxDataCount > 0: + chartCfg.show_minor_x_labels=False + chartCfg.x_labels_major_count=nMaxDataCount + chartCfg.js = [ 'svg.jquery.js','pygal-tooltips.js' ] # Use script from local +# chartCfg.style = LightSolarizedStyle + chartCfg.print_values = False + chartCfg.print_zeroes = True + chartCfg.no_data_text = "All values are 0" + if bLogarithmicChart: + chartCfg.logarithmic=True # Makes chart Y-Axis data more readable + + # Create Linechart + if bLineChart: + myChart = pygal.Line(chartCfg) + myChart.title = 'Line Chart of "' + szInput + '"' + myChart.x_title = "Time elasped in seconds" + myChart.x_labels = map(str, aData[aFields[0]] ) +# myChart.x_labels_major = map(str, aMajorXData ) + for iChartNum in range(3, len(aFields) ): + myChart.add(aFields[iChartNum], aData[ aFields[iChartNum] ]) # Add some values + # Create BarChart + elif bBarChart: + myChart = pygal.Bar(chartCfg) + myChart.title = 'Bar Chart of "' + szInput + '"' + myChart.x_title = "Time elasped in seconds" + myChart.x_labels = map(str, aData[aFields[0]] ) +# myChart.x_labels_major = map(str, aMajorXData ) + for iChartNum in range(3, len(aFields) ): + myChart.add(aFields[iChartNum], aData[ aFields[iChartNum] ]) # Add some values + + # Render Chart now and output to file! + myChart.render_to_file(szOutputFile) + + # Convert to PNG and remove SVG + if bConvertPng: + szPngFileName = szOutputFile[:-4] + ".png" + iReturn = os.system("cairosvg " + szOutputFile + " -f png -o " + szPngFileName) + print "File SVG converted to PNG: '" + szPngFileName + "', return value of cairosvg: " + str(iReturn) + os.remove(szOutputFile) + + # Finished + sys.exit(0) diff --git a/plugins/impstats/statslog-splitter.py b/plugins/impstats/statslog-splitter.py new file mode 100755 index 00000000..715b17bf --- /dev/null +++ b/plugins/impstats/statslog-splitter.py @@ -0,0 +1,232 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- + +# * Copyright (C) 2013 Adiscon GmbH. +# * This file is part of RSyslog +# * +# * This script processes impstats logfiles and splits them into csv files +# * + +import sys +import datetime +import time +import os + +# Include regex definitions +import statslog_regex +from statslog_regex import * + +# Set default variables +szInput = "rsyslog-stats.log" +szOutputDir = "./" +bSingleObjectOutput = True +bHelpOutput = False +bEnableCharts = False +bLogarithmicChart = False +bLineChart = True +bBarChart = False +bFilledLineChart = False +bChartCalcDelta = False +szChartsFormat = "svg" + +# Helper variables +nLogLineNum = 0 +nLogFileCount = 0 +szChartAddArgs = "" + +# Init result with file handles +outputFiles = {} + +# Open Errorlog on init +errorlog = open("statslog-splitter.corrupted.log", 'w') + +# Process Arguments +for arg in sys.argv: # [-4:]: + if arg.find("--input=") != -1: + szInput = arg[8:] + elif arg.find("--outputdir=") != -1: + szOutputDir = arg[12:] + elif arg.find("--singlefile") != -1: + bSingleObjectOutput = True + elif arg.find("--enablecharts") != -1: + bEnableCharts = True + elif arg.find("--chartsformat=") != -1: + szChartsFormat = arg[15:] + elif arg.find("--logarithmic") != -1: + bLogarithmicChart = True + elif arg.find("--linechart") != -1: + bLineChart = True + bBarChart = False + elif arg.find("--barchart") != -1: + bLineChart = False + bBarChart = True + elif arg.find("--filledlinechart") != -1: + bFilledLineChart = True + elif arg.find("--chartscalcdelta") != -1: + bChartCalcDelta = True + elif arg.find("--h") != -1 or arg.find("-h") != -1 or arg.find("--help") != -1: + bHelpOutput = True + +if bHelpOutput: + print "\n\nStatslog-splitter command line options:" + print "=======================================" + print " --input=<filename> Contains the path and filename of your impstats logfile. " + print " Default is 'rsyslog-stats.log' \n" + print " --outputdir=<dir> Output directory to be used. " + print " Default is current directory. " + print " --h / -h / --help Displays this help message. \n" + print " --singlefile Splits the stats logfile into single CSV Files" + print " Default is enabled." + print " --enablecharts Generate Charts for each exported CSV File." + print " Default is disabled." + print " --chartsformat=<svg|png> Format which should be used for Charts." + print " Default is svg format" + print " --logarithmic Uses Logarithmic to scale the Y Axis, maybe useful in some cases. Default is OFF." + print " --linechart If set, line charts will be generated (Default)." + print " --barchart If set, bar charts will be generated." + print " --filledlinechart Use filled lines on Linechart, maybe useful in some cases. Default is OFF." + print " --chartscalcdelta If set, charts will use calculated delta values instead of cumulative values." + print "\n Sampleline: ./statslog-splitter.py singlefile --input=rsyslog-stats.log --outputdir=/home/user/csvlogs/ --enablecharts --chartsformat=png" +elif bSingleObjectOutput: + inputfile = open(szInput, 'r') + for line in inputfile.readlines(): + if line.find("rsyslogd-pstats") != -1: + # Init variables + aFields = [] + aData = [] + iLogRegExIndex = 0 + bLogProcessed = False + + # Loop through Regex parsers + for loglineregex in loglineregexes: + # Parse IMPStats Line! + result = loglineregex.split(line) + # Found valid logline, save into file! + if len(result) >= loglineindexes[iLogRegExIndex]["LN_LOGDATA"] and result[ loglineindexes[iLogRegExIndex]["LN_SYSLOGTAG"] ] == "rsyslogd-pstats": + # Convert Datetime! + try: + iMonth = int( result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] ) + filedate = datetime.datetime.strptime(result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] + " " + str(datetime.datetime.now().year) + " " + result[ loglineindexes[iLogRegExIndex]["LN_DAY"] ] + " " + result[ loglineindexes[iLogRegExIndex]["LN_TIME"] ] ,"%m %Y %d %H:%M:%S") + except ValueError: + filedate = datetime.datetime.strptime(result[ loglineindexes[iLogRegExIndex]["LN_MONTH"] ] + " " + str(datetime.datetime.now().year) + " " + result[ loglineindexes[iLogRegExIndex]["LN_DAY"] ] + " " + result[ loglineindexes[iLogRegExIndex]["LN_TIME"] ] ,"%b %Y %d %H:%M:%S") + + # Split logdata into Array + aProperties = result[ loglineindexes[iLogRegExIndex]["LN_LOGDATA"] ].split(" ") + for szProperty in aProperties: + aProperty = szProperty.split("=") + aFields.append(aProperty[0]) # Append FieldID + if len(aProperty) > 1: + aData.append(aProperty[1]) # Append FieldData + else: + errorlog.write("Corrupted Logline at line " + str(nLogLineNum) + " failed to parse: " + line) + break + + # Remove invalid characters for filename! + szFileName = re.sub("[^a-zA-Z0-9]", "_", result[ loglineindexes[iLogRegExIndex]["LN_LOGOBJECT"] ]) + ".csv" + + # Open file for writing! + if szFileName not in outputFiles: + print "Creating file : " + szOutputDir + "/" + szFileName + outputFiles[szFileName] = open(szOutputDir + "/" + szFileName, 'w') + nLogFileCount += 1 + + # Output CSV Header + outputFiles[szFileName].write("Date;") + outputFiles[szFileName].write("Host;") + outputFiles[szFileName].write("Object;") + for szField in aFields: + outputFiles[szFileName].write(szField + ";") + outputFiles[szFileName].write("\n") + #else: + # print "already open: " + szFileName + + # Output CSV Data + outputFiles[szFileName].write(filedate.strftime("%Y/%b/%d %H:%M:%S") + ";") + outputFiles[szFileName].write(result[ loglineindexes[iLogRegExIndex]["LN_HOST"] ] + ";") + outputFiles[szFileName].write(result[ loglineindexes[iLogRegExIndex]["LN_LOGOBJECT"] ] + ";") + for szData in aData: + outputFiles[szFileName].write(szData + ";") + outputFiles[szFileName].write("\n") + + # Set Log as processed, abort Loop! + bLogProcessed = True + + #print result[LN_LOGOBJECT] + #print result[LN_LOGDATA] + + # Increment Logreged Counter + iLogRegExIndex += 1 + + # Abort Loop if processed + if bLogProcessed: + break + + # Debug output if format not detected + if bLogProcessed == False: + print "Fail parsing logline: " + print result + + # Increment helper counter + nLogLineNum += 1 + + #print result + #break + + #print "IMPStats Line found: " + line + + # Close outfiles + for outFileName in outputFiles: + outputFiles[outFileName].close() + + # Close input file + inputfile.close() + + print "\n File " + szInput + " has been processed" + print " " + str(nLogFileCount) + " Logfiles have been exported to " + szOutputDir + + if bEnableCharts: + # Open HTML Code + szHtmlCode = "<!DOCTYPE html><html><head></head><body><center>" + + # Add mandetory args + if bLineChart: + szChartAddArgs += " --linechart" + elif bBarChart: + szChartAddArgs += " --barchart" + + # Add optional args + if bLogarithmicChart: + szChartAddArgs += " --logarithmic" + if bFilledLineChart: + szChartAddArgs += " --filledlinechart" + if bChartCalcDelta: + szChartAddArgs += " --chartscalcdelta" + + # Default SVG Format! + if szChartsFormat.find("svg") != -1: + for outFileName in outputFiles: + iReturn = os.system("./statslog-graph.py " + szChartAddArgs + " --input=" + szOutputDir + "/" + outFileName + "") + print "Chart SVG generated for '" + outFileName + "': " + str(iReturn) + szHtmlCode += "<figure><embed type=\"image/svg+xml\" src=\"" + outFileName[:-4] + ".svg" + "\" />" + "</figure><br/><br/>" + # Otherwise PNG Output! + else: + for outFileName in outputFiles: + iReturn = os.system("./statslog-graph.py " + szChartAddArgs + " --input=" + szOutputDir + "/" + outFileName + " --convertpng") + print "Chart PNG generated for '" + outFileName + "': " + str(iReturn) + szHtmlCode += "<img src=\"" + outFileName[:-4] + ".png" + "\" width=\"800\" height=\"600\"/>" + "<br/><br/>" + + print " " + str(nLogFileCount) + " Charts have been written " + szOutputDir + + # Close HTML Code + szHtmlCode += "</center></body></html>" + + # Write HTML Index site! + outHtmlFile = open(szOutputDir + "/index.html", 'w') + outHtmlFile.write(szHtmlCode) + outHtmlFile.close() + print " HTML Index with all charts has been written to " + szOutputDir + "/index.html" + + print "\n\n" + +# Close Error log on exit +errorlog.close() diff --git a/plugins/impstats/statslog_regex.py b/plugins/impstats/statslog_regex.py new file mode 100644 index 00000000..b929291e --- /dev/null +++ b/plugins/impstats/statslog_regex.py @@ -0,0 +1,26 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- + +# * Copyright (C) 2013 Adiscon GmbH. +# * This file is part of RSyslog +# * +# * Helper script which includes REGEX definitions +# * + +import sys +import re + +# Create regex for loglines +loglineregexes = [] +loglineindexes = [] + +# Traditional Format +# Sample Line: Jun 26 14:21:44 nhpljt084 rsyslogd-pstats: main Q[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 +loglineregexes.append( re.compile(r"(...)(?:.|..)([0-9]{1,2}) ([0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}) ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32}): (.*?): (.*?) \n") ) +loglineindexes.append( {"LN_YEAR":-1, "LN_MONTH":1, "LN_DAY":2, "LN_TIME":3, "LN_HOST":4, "LN_SYSLOGTAG":5, "LN_LOGOBJECT":6, "LN_LOGDATA":7} ) + +# Newer Format +# Sample format: 2013-07-03T17:22:55.680078+02:00 devdebian6 rsyslogd-pstats: main Q: size=358 enqueued=358 full=0 discarded.full=0 discarded.nf=0 maxqsize=358 +loglineregexes.append( re.compile(r"([0-9]{4,4})-([0-9]{1,2})-([0-9]{1,2})T([0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2})\.[0-9]{1,6}.[0-9]{1,2}:[0-9]{1,2} ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32}): (.*?): (.*?) \n") ) +loglineindexes.append( {"LN_YEAR":1, "LN_MONTH":2, "LN_DAY":3, "LN_TIME":4, "LN_HOST":5, "LN_SYSLOGTAG":6, "LN_LOGOBJECT":7, "LN_LOGDATA":8} ) + diff --git a/plugins/impstats/svg.jquery.js b/plugins/impstats/svg.jquery.js new file mode 100644 index 00000000..46cd42ac --- /dev/null +++ b/plugins/impstats/svg.jquery.js @@ -0,0 +1,9176 @@ +<html>
+<head><title>301 Moved Permanently</title></head>
+<body bgcolor="white">
+<center><h1>301 Moved Permanently</h1></center>
+<hr><center>nginx</center>
+</body>
+</html>
+/*! + * jQuery JavaScript Library v1.7.2 modified by Florian Mounier to work with SVG + * http://jquery.com/ + * + * Copyright 2011, John Resig + * Dual licensed under the MIT or GPL Version 2 licenses. + * http://jquery.org/license + * + * Includes Sizzle.js + * http://sizzlejs.com/ + * Copyright 2011, The Dojo Foundation + * Released under the MIT, BSD, and GPL Licenses. + * + * Date: Wed Mar 21 12:46:34 2012 -0700 + */ +(function( window, undefined ) { + +// Use the correct document accordingly with window argument (sandbox) +var document = window.document, + navigator = window.navigator, + location = window.location; +var jQuery = (function() { + +// Define a local copy of jQuery +var jQuery = function( selector, context ) { + // The jQuery object is actually just the init constructor 'enhanced' + return new jQuery.fn.init( selector, context, rootjQuery ); + }, + + // Map over jQuery in case of overwrite + _jQuery = window.jQuery, + + // Map over the $ in case of overwrite + _$ = window.$, + + // A central reference to the root jQuery(document) + rootjQuery, + + // A simple way to check for HTML strings or ID strings + // Prioritize #id over <tag> to avoid XSS via location.hash (#9521) + quickExpr = /^(?:[^#<]*(<[\w\W]+>)[^>]*$|#([\w\-]*)$)/, + + // Check if a string has a non-whitespace character in it + rnotwhite = /\S/, + + // Used for trimming whitespace + trimLeft = /^\s+/, + trimRight = /\s+$/, + + // Match a standalone tag + rsingleTag = /^<(\w+)\s*\/?>(?:<\/\1>)?$/, + + // JSON RegExp + rvalidchars = /^[\],:{}\s]*$/, + rvalidescape = /\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})/g, + rvalidtokens = /"[^"\\\n\r]*"|true|false|null|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g, + rvalidbraces = /(?:^|:|,)(?:\s*\[)+/g, + + // Useragent RegExp + rwebkit = /(webkit)[ \/]([\w.]+)/, + ropera = /(opera)(?:.*version)?[ \/]([\w.]+)/, + rmsie = /(msie) ([\w.]+)/, + rmozilla = /(mozilla)(?:.*? rv:([\w.]+))?/, + + // Matches dashed string for camelizing + rdashAlpha = /-([a-z]|[0-9])/ig, + rmsPrefix = /^-ms-/, + + // Used by jQuery.camelCase as callback to replace() + fcamelCase = function( all, letter ) { + return ( letter + "" ).toUpperCase(); + }, + + // Keep a UserAgent string for use with jQuery.browser + userAgent = navigator.userAgent, + + // For matching the engine and version of the browser + browserMatch, + + // The deferred used on DOM ready + readyList, + + // The ready event handler + DOMContentLoaded, + + // Save a reference to some core methods + toString = Object.prototype.toString, + hasOwn = Object.prototype.hasOwnProperty, + push = Array.prototype.push, + slice = Array.prototype.slice, + trim = String.prototype.trim, + indexOf = Array.prototype.indexOf, + + // [[Class]] -> type pairs + class2type = {}; + +jQuery.fn = jQuery.prototype = { + constructor: jQuery, + init: function( selector, context, rootjQuery ) { + var match, elem, ret, doc; + + // Handle $(""), $(null), or $(undefined) + if ( !selector ) { + return this; + } + + // Handle $(DOMElement) + if ( selector.nodeType ) { + this.context = this[0] = selector; + this.length = 1; + return this; + } + + // The body element only exists once, optimize finding it + if ( selector === "body" && !context && document.body ) { + this.context = document; + this[0] = document.body; + this.selector = selector; + this.length = 1; + return this; + } + + // Handle HTML strings + if ( typeof selector === "string" ) { + // Are we dealing with HTML string or an ID? + if ( selector.charAt(0) === "<" && selector.charAt( selector.length - 1 ) === ">" && selector.length >= 3 ) { + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = quickExpr.exec( selector ); + } + + // Verify a match, and that no context was specified for #id + if ( match && (match[1] || !context) ) { + + // HANDLE: $(html) -> $(array) + if ( match[1] ) { + context = context instanceof jQuery ? context[0] : context; + doc = ( context ? context.ownerDocument || context : document ); + + // If a single string is passed in and it's a single tag + // just do a createElement and skip the rest + ret = rsingleTag.exec( selector ); + + if ( ret ) { + if ( jQuery.isPlainObject( context ) ) { + selector = [ document.createElementNS( 'http://www.w3.org/2000/svg', ret[1] ) ]; + jQuery.fn.attr.call( selector, context, true ); + + } else { + selector = [ doc.createElementNS( 'http://www.w3.org/2000/svg', ret[1] ) ]; + } + + } else { + ret = jQuery.buildFragment( [ match[1] ], [ doc ] ); + selector = ( ret.cacheable ? jQuery.clone(ret.fragment) : ret.fragment ).childNodes; + } + + return jQuery.merge( this, selector ); + + // HANDLE: $("#id") + } else { + elem = document.getElementById( match[2] ); + + // Check parentNode to catch when Blackberry 4.6 returns + // nodes that are no longer in the document #6963 + if ( elem && elem.parentNode ) { + // Handle the case where IE and Opera return items + // by name instead of ID + if ( elem.id !== match[2] ) { + return rootjQuery.find( selector ); + } + + // Otherwise, we inject the element directly into the jQuery object + this.length = 1; + this[0] = elem; + } + + this.context = document; + this.selector = selector; + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || rootjQuery ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( jQuery.isFunction( selector ) ) { + return rootjQuery.ready( selector ); + } + + if ( selector.selector !== undefined ) { + this.selector = selector.selector; + this.context = selector.context; + } + + return jQuery.makeArray( selector, this ); + }, + + // Start with an empty selector + selector: "", + + // The current version of jQuery being used + jquery: "1.7.2", + + // The default length of a jQuery object is 0 + length: 0, + + // The number of elements contained in the matched element set + size: function() { + return this.length; + }, + + toArray: function() { + return slice.call( this, 0 ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + return num == null ? + + // Return a 'clean' array + this.toArray() : + + // Return just the object + ( num < 0 ? this[ this.length + num ] : this[ num ] ); + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems, name, selector ) { + // Build a new jQuery matched element set + var ret = this.constructor(); + + if ( jQuery.isArray( elems ) ) { + push.apply( ret, elems ); + + } else { + jQuery.merge( ret, elems ); + } + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + ret.context = this.context; + + if ( name === "find" ) { + ret.selector = this.selector + ( this.selector ? " " : "" ) + selector; + } else if ( name ) { + ret.selector = this.selector + "." + name + "(" + selector + ")"; + } + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + // (You can seed the arguments with an array of args, but this is + // only used internally.) + each: function( callback, args ) { + return jQuery.each( this, callback, args ); + }, + + ready: function( fn ) { + // Attach the listeners + jQuery.bindReady(); + + // Add the callback + readyList.add( fn ); + + return this; + }, + + eq: function( i ) { + i = +i; + return i === -1 ? + this.slice( i ) : + this.slice( i, i + 1 ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ), + "slice", slice.call(arguments).join(",") ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map(this, function( elem, i ) { + return callback.call( elem, i, elem ); + })); + }, + + end: function() { + return this.prevObject || this.constructor(null); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: [].sort, + splice: [].splice +}; + +// Give the init function the jQuery prototype for later instantiation +jQuery.fn.init.prototype = jQuery.fn; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[0] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + target = arguments[1] || {}; + // skip the boolean and the target + i = 2; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !jQuery.isFunction(target) ) { + target = {}; + } + + // extend jQuery itself if only one argument is passed + if ( length === i ) { + target = this; + --i; + } + + for ( ; i < length; i++ ) { + // Only deal with non-null/undefined values + if ( (options = arguments[ i ]) != null ) { + // Extend the base object + for ( name in options ) { + src = target[ name ]; + copy = options[ name ]; + + // Prevent never-ending loop + if ( target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject(copy) || (copyIsArray = jQuery.isArray(copy)) ) ) { + if ( copyIsArray ) { + copyIsArray = false; + clone = src && jQuery.isArray(src) ? src : []; + + } else { + clone = src && jQuery.isPlainObject(src) ? src : {}; + } + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend({ + noConflict: function( deep ) { + if ( window.$ === jQuery ) { + window.$ = _$; + } + + if ( deep && window.jQuery === jQuery ) { + window.jQuery = _jQuery; + } + + return jQuery; + }, + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Hold (or release) the ready event + holdReady: function( hold ) { + if ( hold ) { + jQuery.readyWait++; + } else { + jQuery.ready( true ); + } + }, + + // Handle when the DOM is ready + ready: function( wait ) { + // Either a released hold or an DOMready/load event and not yet ready + if ( (wait === true && !--jQuery.readyWait) || (wait !== true && !jQuery.isReady) ) { + // Make sure body exists, at least, in case IE gets a little overzealous (ticket #5443). + // if ( !document.body ) { + // return setTimeout( jQuery.ready, 1 ); + // } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.fireWith( document, [ jQuery ] ); + + // Trigger any bound ready events + if ( jQuery.fn.trigger ) { + jQuery( document ).trigger( "ready" ).off( "ready" ); + } + } + }, + + bindReady: function() { + if ( readyList ) { + return; + } + + readyList = jQuery.Callbacks( "once memory" ); + + // Catch cases where $(document).ready() is called after the + // browser event has already occurred. + if ( document.readyState === "complete" ) { + // Handle it asynchronously to allow scripts the opportunity to delay ready + return setTimeout( jQuery.ready, 1 ); + } + + // Mozilla, Opera and webkit nightlies currently support this event + if ( document.addEventListener ) { + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", DOMContentLoaded, false ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", jQuery.ready, false ); + + // If IE event model is used + } else if ( document.attachEvent ) { + // ensure firing before onload, + // maybe late but safe also for iframes + document.attachEvent( "onreadystatechange", DOMContentLoaded ); + + // A fallback to window.onload, that will always work + window.attachEvent( "onload", jQuery.ready ); + + // If IE and not a frame + // continually check to see if the document is ready + var toplevel = false; + + try { + toplevel = window.frameElement == null; + } catch(e) {} + + if ( document.documentElement.doScroll && toplevel ) { + doScrollCheck(); + } + } + }, + + // See test/unit/core.js for details concerning isFunction. + // Since version 1.3, DOM methods and functions like alert + // aren't supported. They return false on IE (#2968). + isFunction: function( obj ) { + return jQuery.type(obj) === "function"; + }, + + isArray: Array.isArray || function( obj ) { + return jQuery.type(obj) === "array"; + }, + + isWindow: function( obj ) { + return obj != null && obj == obj.window; + }, + + isNumeric: function( obj ) { + return !isNaN( parseFloat(obj) ) && isFinite( obj ); + }, + + type: function( obj ) { + return obj == null ? + String( obj ) : + class2type[ toString.call(obj) ] || "object"; + }, + + isPlainObject: function( obj ) { + // Must be an Object. + // Because of IE, we also have to check the presence of the constructor property. + // Make sure that DOM nodes and window objects don't pass through, as well + if ( !obj || jQuery.type(obj) !== "object" || obj.nodeType || jQuery.isWindow( obj ) ) { + return false; + } + + try { + // Not own constructor property must be Object + if ( obj.constructor && + !hasOwn.call(obj, "constructor") && + !hasOwn.call(obj.constructor.prototype, "isPrototypeOf") ) { + return false; + } + } catch ( e ) { + // IE8,9 Will throw exceptions on certain host objects #9897 + return false; + } + + // Own properties are enumerated firstly, so to speed up, + // if last one is own, then all properties are own. + + var key; + for ( key in obj ) {} + + return key === undefined || hasOwn.call( obj, key ); + }, + + isEmptyObject: function( obj ) { + for ( var name in obj ) { + return false; + } + return true; + }, + + error: function( msg ) { + throw new Error( msg ); + }, + + parseJSON: function( data ) { + if ( typeof data !== "string" || !data ) { + return null; + } + + // Make sure leading/trailing whitespace is removed (IE can't handle it) + data = jQuery.trim( data ); + + // Attempt to parse using the native JSON parser first + if ( window.JSON && window.JSON.parse ) { + return window.JSON.parse( data ); + } + + // Make sure the incoming data is actual JSON + // Logic borrowed from http://json.org/json2.js + if ( rvalidchars.test( data.replace( rvalidescape, "@" ) + .replace( rvalidtokens, "]" ) + .replace( rvalidbraces, "")) ) { + + return ( new Function( "return " + data ) )(); + + } + jQuery.error( "Invalid JSON: " + data ); + }, + + // Cross-browser xml parsing + parseXML: function( data ) { + if ( typeof data !== "string" || !data ) { + return null; + } + var xml, tmp; + try { + if ( window.DOMParser ) { // Standard + tmp = new DOMParser(); + xml = tmp.parseFromString( data , "text/xml" ); + } else { // IE + xml = new ActiveXObject( "Microsoft.XMLDOM" ); + xml.async = "false"; + xml.loadXML( data ); + } + } catch( e ) { + xml = undefined; + } + if ( !xml || !xml.documentElement || xml.getElementsByTagName( "parsererror" ).length ) { + jQuery.error( "Invalid XML: " + data ); + } + return xml; + }, + + noop: function() {}, + + // Evaluates a script in a global context + // Workarounds based on findings by Jim Driscoll + // http://weblogs.java.net/blog/driscoll/archive/2009/09/08/eval-javascript-global-context + globalEval: function( data ) { + if ( data && rnotwhite.test( data ) ) { + // We use execScript on Internet Explorer + // We use an anonymous function so that context is window + // rather than jQuery in Firefox + ( window.execScript || function( data ) { + window[ "eval" ].call( window, data ); + } )( data ); + } + }, + + // Convert dashed to camelCase; used by the css and data modules + // Microsoft forgot to hump their vendor prefix (#9572) + camelCase: function( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); + }, + + nodeName: function( elem, name ) { + return elem.nodeName && elem.nodeName.toUpperCase() === name.toUpperCase(); + }, + + // args is for internal usage only + each: function( object, callback, args ) { + var name, i = 0, + length = object.length, + isObj = length === undefined || jQuery.isFunction( object ); + + if ( args ) { + if ( isObj ) { + for ( name in object ) { + if ( callback.apply( object[ name ], args ) === false ) { + break; + } + } + } else { + for ( ; i < length; ) { + if ( callback.apply( object[ i++ ], args ) === false ) { + break; + } + } + } + + // A special, fast, case for the most common use of each + } else { + if ( isObj ) { + for ( name in object ) { + if ( callback.call( object[ name ], name, object[ name ] ) === false ) { + break; + } + } + } else { + for ( ; i < length; ) { + if ( callback.call( object[ i ], i, object[ i++ ] ) === false ) { + break; + } + } + } + } + + return object; + }, + + // Use native String.trim function wherever possible + trim: trim ? + function( text ) { + return text == null ? + "" : + trim.call( text ); + } : + + // Otherwise use our own trimming functionality + function( text ) { + return text == null ? + "" : + text.toString().replace( trimLeft, "" ).replace( trimRight, "" ); + }, + + // results is for internal usage only + makeArray: function( array, results ) { + var ret = results || []; + + if ( array != null ) { + // The window, strings (and functions) also have 'length' + // Tweaked logic slightly to handle Blackberry 4.7 RegExp issues #6930 + var type = jQuery.type( array ); + + if ( array.length == null || type === "string" || type === "function" || type === "regexp" || jQuery.isWindow( array ) ) { + push.call( ret, array ); + } else { + jQuery.merge( ret, array ); + } + } + + return ret; + }, + + inArray: function( elem, array, i ) { + var len; + + if ( array ) { + if ( indexOf ) { + return indexOf.call( array, elem, i ); + } + + len = array.length; + i = i ? i < 0 ? Math.max( 0, len + i ) : i : 0; + + for ( ; i < len; i++ ) { + // Skip accessing in sparse arrays + if ( i in array && array[ i ] === elem ) { + return i; + } + } + } + + return -1; + }, + + merge: function( first, second ) { + var i = first.length, + j = 0; + + if ( typeof second.length === "number" ) { + for ( var l = second.length; j < l; j++ ) { + first[ i++ ] = second[ j ]; + } + + } else { + while ( second[j] !== undefined ) { + first[ i++ ] = second[ j++ ]; + } + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, inv ) { + var ret = [], retVal; + inv = !!inv; + + // Go through the array, only saving the items + // that pass the validator function + for ( var i = 0, length = elems.length; i < length; i++ ) { + retVal = !!callback( elems[ i ], i ); + if ( inv !== retVal ) { + ret.push( elems[ i ] ); + } + } + + return ret; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var value, key, ret = [], + i = 0, + length = elems.length, + // jquery objects are treated as arrays + isArray = elems instanceof jQuery || length !== undefined && typeof length === "number" && ( ( length > 0 && elems[ 0 ] && elems[ length -1 ] ) || length === 0 || jQuery.isArray( elems ) ) ; + + // Go through the array, translating each of the items to their + if ( isArray ) { + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret[ ret.length ] = value; + } + } + + // Go through every key on the object, + } else { + for ( key in elems ) { + value = callback( elems[ key ], key, arg ); + + if ( value != null ) { + ret[ ret.length ] = value; + } + } + } + + // Flatten any nested arrays + return ret.concat.apply( [], ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // Bind a function to a context, optionally partially applying any + // arguments. + proxy: function( fn, context ) { + if ( typeof context === "string" ) { + var tmp = fn[ context ]; + context = fn; + fn = tmp; + } + + // Quick check to determine if target is callable, in the spec + // this throws a TypeError, but we will just return undefined. + if ( !jQuery.isFunction( fn ) ) { + return undefined; + } + + // Simulated bind + var args = slice.call( arguments, 2 ), + proxy = function() { + return fn.apply( context, args.concat( slice.call( arguments ) ) ); + }; + + // Set the guid of unique handler to the same of original handler, so it can be removed + proxy.guid = fn.guid = fn.guid || proxy.guid || jQuery.guid++; + + return proxy; + }, + + // Mutifunctional method to get and set values to a collection + // The value/s can optionally be executed if it's a function + access: function( elems, fn, key, value, chainable, emptyGet, pass ) { + var exec, + bulk = key == null, + i = 0, + length = elems.length; + + // Sets many values + if ( key && typeof key === "object" ) { + for ( i in key ) { + jQuery.access( elems, fn, i, key[i], 1, emptyGet, value ); + } + chainable = 1; + + // Sets one value + } else if ( value !== undefined ) { + // Optionally, function values get executed if exec is true + exec = pass === undefined && jQuery.isFunction( value ); + + if ( bulk ) { + // Bulk operations only iterate when executing function values + if ( exec ) { + exec = fn; + fn = function( elem, key, value ) { + return exec.call( jQuery( elem ), value ); + }; + + // Otherwise they run against the entire set + } else { + fn.call( elems, value ); + fn = null; + } + } + + if ( fn ) { + for (; i < length; i++ ) { + fn( elems[i], key, exec ? value.call( elems[i], i, fn( elems[i], key ) ) : value, pass ); + } + } + + chainable = 1; + } + + return chainable ? + elems : + + // Gets + bulk ? + fn.call( elems ) : + length ? fn( elems[0], key ) : emptyGet; + }, + + now: function() { + return ( new Date() ).getTime(); + }, + + // Use of jQuery.browser is frowned upon. + // More details: http://docs.jquery.com/Utilities/jQuery.browser + uaMatch: function( ua ) { + ua = ua.toLowerCase(); + + var match = rwebkit.exec( ua ) || + ropera.exec( ua ) || + rmsie.exec( ua ) || + ua.indexOf("compatible") < 0 && rmozilla.exec( ua ) || + []; + + return { browser: match[1] || "", version: match[2] || "0" }; + }, + + sub: function() { + function jQuerySub( selector, context ) { + return new jQuerySub.fn.init( selector, context ); + } + jQuery.extend( true, jQuerySub, this ); + jQuerySub.superclass = this; + jQuerySub.fn = jQuerySub.prototype = this(); + jQuerySub.fn.constructor = jQuerySub; + jQuerySub.sub = this.sub; + jQuerySub.fn.init = function init( selector, context ) { + if ( context && context instanceof jQuery && !(context instanceof jQuerySub) ) { + context = jQuerySub( context ); + } + + return jQuery.fn.init.call( this, selector, context, rootjQuerySub ); + }; + jQuerySub.fn.init.prototype = jQuerySub.fn; + var rootjQuerySub = jQuerySub(document); + return jQuerySub; + }, + + browser: {} +}); + +// Populate the class2type map +jQuery.each("Boolean Number String Function Array Date RegExp Object".split(" "), function(i, name) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); +}); + +browserMatch = jQuery.uaMatch( userAgent ); +if ( browserMatch.browser ) { + jQuery.browser[ browserMatch.browser ] = true; + jQuery.browser.version = browserMatch.version; +} + +// Deprecated, use jQuery.browser.webkit instead +if ( jQuery.browser.webkit ) { + jQuery.browser.safari = true; +} + +// IE doesn't match non-breaking spaces with \s +if ( rnotwhite.test( "\xA0" ) ) { + trimLeft = /^[\s\xA0]+/; + trimRight = /[\s\xA0]+$/; +} + +// All jQuery objects should point back to these +rootjQuery = jQuery(document); + +// Cleanup functions for the document ready method +if ( document.addEventListener ) { + DOMContentLoaded = function() { + document.removeEventListener( "DOMContentLoaded", DOMContentLoaded, false ); + jQuery.ready(); + }; + +} else if ( document.attachEvent ) { + DOMContentLoaded = function() { + // Make sure body exists, at least, in case IE gets a little overzealous (ticket #5443). + if ( document.readyState === "complete" ) { + document.detachEvent( "onreadystatechange", DOMContentLoaded ); + jQuery.ready(); + } + }; +} + +// The DOM ready check for Internet Explorer +function doScrollCheck() { + if ( jQuery.isReady ) { + return; + } + + try { + // If IE is used, use the trick by Diego Perini + // http://javascript.nwbox.com/IEContentLoaded/ + document.documentElement.doScroll("left"); + } catch(e) { + setTimeout( doScrollCheck, 1 ); + return; + } + + // and execute any waiting functions + jQuery.ready(); +} + +return jQuery; + +})(); + + +// String to Object flags format cache +var flagsCache = {}; + +// Convert String-formatted flags into Object-formatted ones and store in cache +function createFlags( flags ) { + var object = flagsCache[ flags ] = {}, + i, length; + flags = flags.split( /\s+/ ); + for ( i = 0, length = flags.length; i < length; i++ ) { + object[ flags[i] ] = true; + } + return object; +} + +/* + * Create a callback list using the following parameters: + * + * flags: an optional list of space-separated flags that will change how + * the callback list behaves + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible flags: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( flags ) { + + // Convert flags from String-formatted to Object-formatted + // (we check in cache first) + flags = flags ? ( flagsCache[ flags ] || createFlags( flags ) ) : {}; + + var // Actual callback list + list = [], + // Stack of fire calls for repeatable lists + stack = [], + // Last fire value (for non-forgettable lists) + memory, + // Flag to know if list was already fired + fired, + // Flag to know if list is currently firing + firing, + // First callback to fire (used internally by add and fireWith) + firingStart, + // End of the loop when firing + firingLength, + // Index of currently firing callback (modified by remove if needed) + firingIndex, + // Add one or several callbacks to the list + add = function( args ) { + var i, + length, + elem, + type, + actual; + for ( i = 0, length = args.length; i < length; i++ ) { + elem = args[ i ]; + type = jQuery.type( elem ); + if ( type === "array" ) { + // Inspect recursively + add( elem ); + } else if ( type === "function" ) { + // Add if not in unique mode and callback is not in + if ( !flags.unique || !self.has( elem ) ) { + list.push( elem ); + } + } + } + }, + // Fire callbacks + fire = function( context, args ) { + args = args || []; + memory = !flags.memory || [ context, args ]; + fired = true; + firing = true; + firingIndex = firingStart || 0; + firingStart = 0; + firingLength = list.length; + for ( ; list && firingIndex < firingLength; firingIndex++ ) { + if ( list[ firingIndex ].apply( context, args ) === false && flags.stopOnFalse ) { + memory = true; // Mark as halted + break; + } + } + firing = false; + if ( list ) { + if ( !flags.once ) { + if ( stack && stack.length ) { + memory = stack.shift(); + self.fireWith( memory[ 0 ], memory[ 1 ] ); + } + } else if ( memory === true ) { + self.disable(); + } else { + list = []; + } + } + }, + // Actual Callbacks object + self = { + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + var length = list.length; + add( arguments ); + // Do we need to add the callbacks to the + // current firing batch? + if ( firing ) { + firingLength = list.length; + // With memory, if we're not firing then + // we should call right away, unless previous + // firing was halted (stopOnFalse) + } else if ( memory && memory !== true ) { + firingStart = length; + fire( memory[ 0 ], memory[ 1 ] ); + } + } + return this; + }, + // Remove a callback from the list + remove: function() { + if ( list ) { + var args = arguments, + argIndex = 0, + argLength = args.length; + for ( ; argIndex < argLength ; argIndex++ ) { + for ( var i = 0; i < list.length; i++ ) { + if ( args[ argIndex ] === list[ i ] ) { + // Handle firingIndex and firingLength + if ( firing ) { + if ( i <= firingLength ) { + firingLength--; + if ( i <= firingIndex ) { + firingIndex--; + } + } + } + // Remove the element + list.splice( i--, 1 ); + // If we have some unicity property then + // we only need to do this once + if ( flags.unique ) { + break; + } + } + } + } + } + return this; + }, + // Control if a given callback is in the list + has: function( fn ) { + if ( list ) { + var i = 0, + length = list.length; + for ( ; i < length; i++ ) { + if ( fn === list[ i ] ) { + return true; + } + } + } + return false; + }, + // Remove all callbacks from the list + empty: function() { + list = []; + return this; + }, + // Have the list do nothing anymore + disable: function() { + list = stack = memory = undefined; + return this; + }, + // Is it disabled? + disabled: function() { + return !list; + }, + // Lock the list in its current state + lock: function() { + stack = undefined; + if ( !memory || memory === true ) { + self.disable(); + } + return this; + }, + // Is it locked? + locked: function() { + return !stack; + }, + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( stack ) { + if ( firing ) { + if ( !flags.once ) { + stack.push( [ context, args ] ); + } + } else if ( !( flags.once && memory ) ) { + fire( context, args ); + } + } + return this; + }, + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + + + +var // Static reference to slice + sliceDeferred = [].slice; + +jQuery.extend({ + + Deferred: function( func ) { + var doneList = jQuery.Callbacks( "once memory" ), + failList = jQuery.Callbacks( "once memory" ), + progressList = jQuery.Callbacks( "memory" ), + state = "pending", + lists = { + resolve: doneList, + reject: failList, + notify: progressList + }, + promise = { + done: doneList.add, + fail: failList.add, + progress: progressList.add, + + state: function() { + return state; + }, + + // Deprecated + isResolved: doneList.fired, + isRejected: failList.fired, + + then: function( doneCallbacks, failCallbacks, progressCallbacks ) { + deferred.done( doneCallbacks ).fail( failCallbacks ).progress( progressCallbacks ); + return this; + }, + always: function() { + deferred.done.apply( deferred, arguments ).fail.apply( deferred, arguments ); + return this; + }, + pipe: function( fnDone, fnFail, fnProgress ) { + return jQuery.Deferred(function( newDefer ) { + jQuery.each( { + done: [ fnDone, "resolve" ], + fail: [ fnFail, "reject" ], + progress: [ fnProgress, "notify" ] + }, function( handler, data ) { + var fn = data[ 0 ], + action = data[ 1 ], + returned; + if ( jQuery.isFunction( fn ) ) { + deferred[ handler ](function() { + returned = fn.apply( this, arguments ); + if ( returned && jQuery.isFunction( returned.promise ) ) { + returned.promise().then( newDefer.resolve, newDefer.reject, newDefer.notify ); + } else { + newDefer[ action + "With" ]( this === deferred ? newDefer : this, [ returned ] ); + } + }); + } else { + deferred[ handler ]( newDefer[ action ] ); + } + }); + }).promise(); + }, + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + if ( obj == null ) { + obj = promise; + } else { + for ( var key in promise ) { + obj[ key ] = promise[ key ]; + } + } + return obj; + } + }, + deferred = promise.promise({}), + key; + + for ( key in lists ) { + deferred[ key ] = lists[ key ].fire; + deferred[ key + "With" ] = lists[ key ].fireWith; + } + + // Handle state + deferred.done( function() { + state = "resolved"; + }, failList.disable, progressList.lock ).fail( function() { + state = "rejected"; + }, doneList.disable, progressList.lock ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( firstParam ) { + var args = sliceDeferred.call( arguments, 0 ), + i = 0, + length = args.length, + pValues = new Array( length ), + count = length, + pCount = length, + deferred = length <= 1 && firstParam && jQuery.isFunction( firstParam.promise ) ? + firstParam : + jQuery.Deferred(), + promise = deferred.promise(); + function resolveFunc( i ) { + return function( value ) { + args[ i ] = arguments.length > 1 ? sliceDeferred.call( arguments, 0 ) : value; + if ( !( --count ) ) { + deferred.resolveWith( deferred, args ); + } + }; + } + function progressFunc( i ) { + return function( value ) { + pValues[ i ] = arguments.length > 1 ? sliceDeferred.call( arguments, 0 ) : value; + deferred.notifyWith( promise, pValues ); + }; + } + if ( length > 1 ) { + for ( ; i < length; i++ ) { + if ( args[ i ] && args[ i ].promise && jQuery.isFunction( args[ i ].promise ) ) { + args[ i ].promise().then( resolveFunc(i), deferred.reject, progressFunc(i) ); + } else { + --count; + } + } + if ( !count ) { + deferred.resolveWith( deferred, args ); + } + } else if ( deferred !== firstParam ) { + deferred.resolveWith( deferred, length ? [ firstParam ] : [] ); + } + return promise; + } +}); + + + + +jQuery.support = { + // IE strips leading whitespace when .innerHTML is used + leadingWhitespace: true, + + // Make sure that tbody elements aren't automatically inserted + // IE will insert them into empty tables + tbody: false, + + // Make sure that link elements get serialized correctly by innerHTML + // This requires a wrapper element in IE + htmlSerialize: true, + + // Get the style information from getAttribute + // (IE uses .cssText instead) + style: true, + + // Make sure that URLs aren't manipulated + // (IE normalizes it by default) + hrefNormalized: true, + + // Make sure that element opacity exists + // (IE uses filter instead) + // Use a regex to work around a WebKit issue. See #5145 + opacity: true, + + // Verify style float existence + // (IE uses styleFloat instead of cssFloat) + cssFloat: true, + + // Make sure that if no value is specified for a checkbox + // that it defaults to "on". + // (WebKit defaults to "" instead) + checkOn: true, + + // Make sure that a selected-by-default option has a working selected property. + // (WebKit defaults to false instead of true, IE too, if it's in an optgroup) + optSelected: true, + + // Test setAttribute on camelCase class. If it works, we need attrFixes when doing get/setAttribute (ie6/7) + getSetAttribute: true, + + // Tests for enctype support on a form(#6743) + enctype: true, + + // Makes sure cloning an html5 element does not cause problems + // Where outerHTML is undefined, this still works + html5Clone: true, + + // Will be defined later + submitBubbles: true, + changeBubbles: true, + focusinBubbles: false, + deleteExpando: true, + noCloneEvent: true, + inlineBlockNeedsLayout: false, + shrinkWrapBlocks: false, + reliableMarginRight: true, + pixelMargin: true + }; + + + +var rbrace = /^(?:\{.*\}|\[.*\])$/, + rmultiDash = /([A-Z])/g; + +jQuery.extend({ + cache: {}, + + // Please use with caution + uuid: 0, + + // Unique for each copy of jQuery on the page + // Non-digits removed to match rinlinejQuery + expando: "jQuery" + ( jQuery.fn.jquery + Math.random() ).replace( /\D/g, "" ), + + // The following elements throw uncatchable exceptions if you + // attempt to add expando properties to them. + noData: { + "embed": true, + // Ban all objects except for Flash (which handle expandos) + "object": "clsid:D27CDB6E-AE6D-11cf-96B8-444553540000", + "applet": true + }, + + hasData: function( elem ) { + elem = elem.nodeType ? jQuery.cache[ elem[jQuery.expando] ] : elem[ jQuery.expando ]; + return !!elem && !isEmptyDataObject( elem ); + }, + + data: function( elem, name, data, pvt /* Internal Use Only */ ) { + if ( !jQuery.acceptData( elem ) ) { + return; + } + + var privateCache, thisCache, ret, + internalKey = jQuery.expando, + getByName = typeof name === "string", + + // We have to handle DOM nodes and JS objects differently because IE6-7 + // can't GC object references properly across the DOM-JS boundary + isNode = elem.nodeType, + + // Only DOM nodes need the global jQuery cache; JS object data is + // attached directly to the object so GC can occur automatically + cache = isNode ? jQuery.cache : elem, + + // Only defining an ID for JS objects if its cache already exists allows + // the code to shortcut on the same path as a DOM node with no cache + id = isNode ? elem[ internalKey ] : elem[ internalKey ] && internalKey, + isEvents = name === "events"; + + // Avoid doing any more work than we need to when trying to get data on an + // object that has no data at all + if ( (!id || !cache[id] || (!isEvents && !pvt && !cache[id].data)) && getByName && data === undefined ) { + return; + } + + if ( !id ) { + // Only DOM nodes need a new unique ID for each element since their data + // ends up in the global cache + if ( isNode ) { + elem[ internalKey ] = id = ++jQuery.uuid; + } else { + id = internalKey; + } + } + + if ( !cache[ id ] ) { + cache[ id ] = {}; + + // Avoids exposing jQuery metadata on plain JS objects when the object + // is serialized using JSON.stringify + if ( !isNode ) { + cache[ id ].toJSON = jQuery.noop; + } + } + + // An object can be passed to jQuery.data instead of a key/value pair; this gets + // shallow copied over onto the existing cache + if ( typeof name === "object" || typeof name === "function" ) { + if ( pvt ) { + cache[ id ] = jQuery.extend( cache[ id ], name ); + } else { + cache[ id ].data = jQuery.extend( cache[ id ].data, name ); + } + } + + privateCache = thisCache = cache[ id ]; + + // jQuery data() is stored in a separate object inside the object's internal data + // cache in order to avoid key collisions between internal data and user-defined + // data. + if ( !pvt ) { + if ( !thisCache.data ) { + thisCache.data = {}; + } + + thisCache = thisCache.data; + } + + if ( data !== undefined ) { + thisCache[ jQuery.camelCase( name ) ] = data; + } + + // Users should not attempt to inspect the internal events object using jQuery.data, + // it is undocumented and subject to change. But does anyone listen? No. + if ( isEvents && !thisCache[ name ] ) { + return privateCache.events; + } + + // Check for both converted-to-camel and non-converted data property names + // If a data property was specified + if ( getByName ) { + + // First Try to find as-is property data + ret = thisCache[ name ]; + + // Test for null|undefined property data + if ( ret == null ) { + + // Try to find the camelCased property + ret = thisCache[ jQuery.camelCase( name ) ]; + } + } else { + ret = thisCache; + } + + return ret; + }, + + removeData: function( elem, name, pvt /* Internal Use Only */ ) { + if ( !jQuery.acceptData( elem ) ) { + return; + } + + var thisCache, i, l, + + // Reference to internal data cache key + internalKey = jQuery.expando, + + isNode = elem.nodeType, + + // See jQuery.data for more information + cache = isNode ? jQuery.cache : elem, + + // See jQuery.data for more information + id = isNode ? elem[ internalKey ] : internalKey; + + // If there is already no cache entry for this object, there is no + // purpose in continuing + if ( !cache[ id ] ) { + return; + } + + if ( name ) { + + thisCache = pvt ? cache[ id ] : cache[ id ].data; + + if ( thisCache ) { + + // Support array or space separated string names for data keys + if ( !jQuery.isArray( name ) ) { + + // try the string as a key before any manipulation + if ( name in thisCache ) { + name = [ name ]; + } else { + + // split the camel cased version by spaces unless a key with the spaces exists + name = jQuery.camelCase( name ); + if ( name in thisCache ) { + name = [ name ]; + } else { + name = name.split( " " ); + } + } + } + + for ( i = 0, l = name.length; i < l; i++ ) { + delete thisCache[ name[i] ]; + } + + // If there is no data left in the cache, we want to continue + // and let the cache object itself get destroyed + if ( !( pvt ? isEmptyDataObject : jQuery.isEmptyObject )( thisCache ) ) { + return; + } + } + } + + // See jQuery.data for more information + if ( !pvt ) { + delete cache[ id ].data; + + // Don't destroy the parent cache unless the internal data object + // had been the only thing left in it + if ( !isEmptyDataObject(cache[ id ]) ) { + return; + } + } + + // Browsers that fail expando deletion also refuse to delete expandos on + // the window, but it will allow it on all other JS objects; other browsers + // don't care + // Ensure that `cache` is not a window object #10080 + if ( jQuery.support.deleteExpando || !cache.setInterval ) { + delete cache[ id ]; + } else { + cache[ id ] = null; + } + + // We destroyed the cache and need to eliminate the expando on the node to avoid + // false lookups in the cache for entries that no longer exist + if ( isNode ) { + // IE does not allow us to delete expando properties from nodes, + // nor does it have a removeAttribute function on Document nodes; + // we must handle all of these cases + if ( jQuery.support.deleteExpando ) { + delete elem[ internalKey ]; + } else if ( elem.removeAttribute ) { + elem.removeAttribute( internalKey ); + } else { + elem[ internalKey ] = null; + } + } + }, + + // For internal use only. + _data: function( elem, name, data ) { + return jQuery.data( elem, name, data, true ); + }, + + // A method for determining if a DOM node can handle the data expando + acceptData: function( elem ) { + if ( elem.nodeName ) { + var match = jQuery.noData[ elem.nodeName.toLowerCase() ]; + + if ( match ) { + return !(match === true || elem.getAttribute("classid") !== match); + } + } + + return true; + } +}); + +jQuery.fn.extend({ + data: function( key, value ) { + var parts, part, attr, name, l, + elem = this[0], + i = 0, + data = null; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = jQuery.data( elem ); + + if ( elem.nodeType === 1 && !jQuery._data( elem, "parsedAttrs" ) ) { + attr = elem.attributes; + for ( l = attr.length; i < l; i++ ) { + name = attr[i].name; + + if ( name.indexOf( "data-" ) === 0 ) { + name = jQuery.camelCase( name.substring(5) ); + + dataAttr( elem, name, data[ name ] ); + } + } + jQuery._data( elem, "parsedAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each(function() { + jQuery.data( this, key ); + }); + } + + parts = key.split( ".", 2 ); + parts[1] = parts[1] ? "." + parts[1] : ""; + part = parts[1] + "!"; + + return jQuery.access( this, function( value ) { + + if ( value === undefined ) { + data = this.triggerHandler( "getData" + part, [ parts[0] ] ); + + // Try to fetch any internally stored data first + if ( data === undefined && elem ) { + data = jQuery.data( elem, key ); + data = dataAttr( elem, key, data ); + } + + return data === undefined && parts[1] ? + this.data( parts[0] ) : + data; + } + + parts[1] = value; + this.each(function() { + var self = jQuery( this ); + + self.triggerHandler( "setData" + part, parts ); + jQuery.data( this, key, value ); + self.triggerHandler( "changeData" + part, parts ); + }); + }, null, value, arguments.length > 1, null, false ); + }, + + removeData: function( key ) { + return this.each(function() { + jQuery.removeData( this, key ); + }); + } +}); + +function dataAttr( elem, key, data ) { + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + + var name = "data-" + key.replace( rmultiDash, "-$1" ).toLowerCase(); + + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = data === "true" ? true : + data === "false" ? false : + data === "null" ? null : + jQuery.isNumeric( data ) ? +data : + rbrace.test( data ) ? jQuery.parseJSON( data ) : + data; + } catch( e ) {} + + // Make sure we set the data so it isn't changed later + jQuery.data( elem, key, data ); + + } else { + data = undefined; + } + } + + return data; +} + +// checks a cache object for emptiness +function isEmptyDataObject( obj ) { + for ( var name in obj ) { + + // if the public data object is empty, the private is still empty + if ( name === "data" && jQuery.isEmptyObject( obj[name] ) ) { + continue; + } + if ( name !== "toJSON" ) { + return false; + } + } + + return true; +} + + + + +function handleQueueMarkDefer( elem, type, src ) { + var deferDataKey = type + "defer", + queueDataKey = type + "queue", + markDataKey = type + "mark", + defer = jQuery._data( elem, deferDataKey ); + if ( defer && + ( src === "queue" || !jQuery._data(elem, queueDataKey) ) && + ( src === "mark" || !jQuery._data(elem, markDataKey) ) ) { + // Give room for hard-coded callbacks to fire first + // and eventually mark/queue something else on the element + setTimeout( function() { + if ( !jQuery._data( elem, queueDataKey ) && + !jQuery._data( elem, markDataKey ) ) { + jQuery.removeData( elem, deferDataKey, true ); + defer.fire(); + } + }, 0 ); + } +} + +jQuery.extend({ + + _mark: function( elem, type ) { + if ( elem ) { + type = ( type || "fx" ) + "mark"; + jQuery._data( elem, type, (jQuery._data( elem, type ) || 0) + 1 ); + } + }, + + _unmark: function( force, elem, type ) { + if ( force !== true ) { + type = elem; + elem = force; + force = false; + } + if ( elem ) { + type = type || "fx"; + var key = type + "mark", + count = force ? 0 : ( (jQuery._data( elem, key ) || 1) - 1 ); + if ( count ) { + jQuery._data( elem, key, count ); + } else { + jQuery.removeData( elem, key, true ); + handleQueueMarkDefer( elem, type, "mark" ); + } + } + }, + + queue: function( elem, type, data ) { + var q; + if ( elem ) { + type = ( type || "fx" ) + "queue"; + q = jQuery._data( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !q || jQuery.isArray(data) ) { + q = jQuery._data( elem, type, jQuery.makeArray(data) ); + } else { + q.push( data ); + } + } + return q || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + fn = queue.shift(), + hooks = {}; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + } + + if ( fn ) { + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + jQuery._data( elem, type + ".run", hooks ); + fn.call( elem, function() { + jQuery.dequeue( elem, type ); + }, hooks ); + } + + if ( !queue.length ) { + jQuery.removeData( elem, type + "queue " + type + ".run", true ); + handleQueueMarkDefer( elem, type, "queue" ); + } + } +}); + +jQuery.fn.extend({ + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[0], type ); + } + + return data === undefined ? + this : + this.each(function() { + var queue = jQuery.queue( this, type, data ); + + if ( type === "fx" && queue[0] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + }); + }, + dequeue: function( type ) { + return this.each(function() { + jQuery.dequeue( this, type ); + }); + }, + // Based off of the plugin by Clint Helfers, with permission. + // http://blindsignals.com/index.php/2009/07/jquery-delay/ + delay: function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = setTimeout( next, time ); + hooks.stop = function() { + clearTimeout( timeout ); + }; + }); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, object ) { + if ( typeof type !== "string" ) { + object = type; + type = undefined; + } + type = type || "fx"; + var defer = jQuery.Deferred(), + elements = this, + i = elements.length, + count = 1, + deferDataKey = type + "defer", + queueDataKey = type + "queue", + markDataKey = type + "mark", + tmp; + function resolve() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + } + while( i-- ) { + if (( tmp = jQuery.data( elements[ i ], deferDataKey, undefined, true ) || + ( jQuery.data( elements[ i ], queueDataKey, undefined, true ) || + jQuery.data( elements[ i ], markDataKey, undefined, true ) ) && + jQuery.data( elements[ i ], deferDataKey, jQuery.Callbacks( "once memory" ), true ) )) { + count++; + tmp.add( resolve ); + } + } + resolve(); + return defer.promise( object ); + } +}); + + + + +var rclass = /[\n\t\r]/g, + rspace = /\s+/, + rreturn = /\r/g, + rtype = /^(?:button|input)$/i, + rfocusable = /^(?:button|input|object|select|textarea)$/i, + rclickable = /^a(?:rea)?$/i, + rboolean = /^(?:autofocus|autoplay|async|checked|controls|defer|disabled|hidden|loop|multiple|open|readonly|required|scoped|selected)$/i, + getSetAttribute = jQuery.support.getSetAttribute, + nodeHook, boolHook, fixSpecified; + +jQuery.fn.extend({ + attr: function( name, value ) { + return jQuery.access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each(function() { + jQuery.removeAttr( this, name ); + }); + }, + + prop: function( name, value ) { + return jQuery.access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + name = jQuery.propFix[ name ] || name; + return this.each(function() { + // try/catch handles cases where IE balks (such as removing a property on window) + try { + this[ name ] = undefined; + delete this[ name ]; + } catch( e ) {} + }); + }, + + addClass: function( value ) { + var classNames, i, l, elem, + setClass, c, cl; + + if ( jQuery.isFunction( value ) ) { + return this.each(function( j ) { + jQuery( this ).addClass( value.call(this, j, this.className.baseVal) ); + }); + } + + if ( value && typeof value === "string" ) { + classNames = value.split( rspace ); + + for ( i = 0, l = this.length; i < l; i++ ) { + elem = this[ i ]; + + if ( elem.nodeType === 1 ) { + if ( !elem.className.baseVal && classNames.length === 1 ) { + elem.className.baseVal = value; + + } else { + setClass = " " + elem.className.baseVal + " "; + + for ( c = 0, cl = classNames.length; c < cl; c++ ) { + if ( !~setClass.indexOf( " " + classNames[ c ] + " " ) ) { + setClass += classNames[ c ] + " "; + } + } + elem.className.baseVal = jQuery.trim( setClass ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classNames, i, l, elem, className, c, cl; + + if ( jQuery.isFunction( value ) ) { + return this.each(function( j ) { + jQuery( this ).removeClass( value.call(this, j, this.className.baseVal) ); + }); + } + + if ( (value && typeof value === "string") || value === undefined ) { + classNames = ( value || "" ).split( rspace ); + + for ( i = 0, l = this.length; i < l; i++ ) { + elem = this[ i ]; + + if ( elem.nodeType === 1 && elem.className.baseVal ) { + if ( value ) { + className = (" " + elem.className.baseVal + " ").replace( rclass, " " ); + for ( c = 0, cl = classNames.length; c < cl; c++ ) { + className = className.replace(" " + classNames[ c ] + " ", " "); + } + elem.className.baseVal = jQuery.trim( className ); + + } else { + elem.className.baseVal = ""; + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isBool = typeof stateVal === "boolean"; + + if ( jQuery.isFunction( value ) ) { + return this.each(function( i ) { + jQuery( this ).toggleClass( value.call(this, i, this.className.baseVal, stateVal), stateVal ); + }); + } + + return this.each(function() { + if ( type === "string" ) { + // toggle individual class names + var className, + i = 0, + self = jQuery( this ), + state = stateVal, + classNames = value.split( rspace ); + + while ( (className = classNames[ i++ ]) ) { + // check each className given, space seperated list + state = isBool ? state : !self.hasClass( className ); + self[ state ? "addClass" : "removeClass" ]( className ); + } + + } else if ( type === "undefined" || type === "boolean" ) { + if ( this.className.baseVal ) { + // store className if set + jQuery._data( this, "__className__", this.className.baseVal ); + } + + // toggle whole className + this.className.baseVal = this.className.baseVal || value === false ? "" : jQuery._data( this, "__className__" ) || ""; + } + }); + }, + + hasClass: function( selector ) { + var className = " " + selector + " ", + i = 0, + l = this.length; + for ( ; i < l; i++ ) { + if ( this[i].nodeType === 1 && (" " + this[i].className.baseVal + " ").replace(rclass, " ").indexOf( className ) > -1 ) { + return true; + } + } + + return false; + }, + + val: function( value ) { + var hooks, ret, isFunction, + elem = this[0]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && "get" in hooks && (ret = hooks.get( elem, "value" )) !== undefined ) { + return ret; + } + + ret = elem.value; + + return typeof ret === "string" ? + // handle most common string cases + ret.replace(rreturn, "") : + // handle cases where value is null/undef or number + ret == null ? "" : ret; + } + + return; + } + + isFunction = jQuery.isFunction( value ); + + return this.each(function( i ) { + var self = jQuery(this), val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( isFunction ) { + val = value.call( this, i, self.val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + } else if ( typeof val === "number" ) { + val += ""; + } else if ( jQuery.isArray( val ) ) { + val = jQuery.map(val, function ( value ) { + return value == null ? "" : value + ""; + }); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !("set" in hooks) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + }); + } +}); + +jQuery.extend({ + valHooks: { + option: { + get: function( elem ) { + // attributes.value is undefined in Blackberry 4.7 but + // uses .value. See #6932 + var val = elem.attributes.value; + return !val || val.specified ? elem.value : elem.text; + } + }, + select: { + get: function( elem ) { + var value, i, max, option, + index = elem.selectedIndex, + values = [], + options = elem.options, + one = elem.type === "select-one"; + + // Nothing was selected + if ( index < 0 ) { + return null; + } + + // Loop through all the selected options + i = one ? index : 0; + max = one ? index + 1 : options.length; + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Don't return options that are disabled or in a disabled optgroup + if ( option.selected && (jQuery.support.optDisabled ? !option.disabled : option.getAttribute("disabled") === null) && + (!option.parentNode.disabled || !jQuery.nodeName( option.parentNode, "optgroup" )) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + // Fixes Bug #2551 -- select.val() broken in IE after form.reset() + if ( one && !values.length && options.length ) { + return jQuery( options[ index ] ).val(); + } + + return values; + }, + + set: function( elem, value ) { + var values = jQuery.makeArray( value ); + + jQuery(elem).find("option").each(function() { + this.selected = jQuery.inArray( jQuery(this).val(), values ) >= 0; + }); + + if ( !values.length ) { + elem.selectedIndex = -1; + } + return values; + } + } + }, + + attrFn: { + val: true, + css: true, + html: true, + text: true, + data: true, + width: true, + height: true, + offset: true + }, + + attr: function( elem, name, value, pass ) { + var ret, hooks, notxml, + nType = elem.nodeType; + + // don't get/set attributes on text, comment and attribute nodes + if ( !elem || nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( pass && name in jQuery.attrFn ) { + return jQuery( elem )[ name ]( value ); + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + notxml = nType !== 1 || !jQuery.isXMLDoc( elem ); + + // All attributes are lowercase + // Grab necessary hook if one is defined + if ( notxml ) { + name = name.toLowerCase(); + hooks = jQuery.attrHooks[ name ] || ( rboolean.test( name ) ? boolHook : nodeHook ); + } + + if ( value !== undefined ) { + + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + + } else if ( hooks && "set" in hooks && notxml && (ret = hooks.set( elem, value, name )) !== undefined ) { + return ret; + + } else { + if (name == 'href') { + elem.setAttributeNS( 'http://www.w3.org/1999/xlink', name, "" + value ); + } else { + elem.setAttribute( name, "" + value ); + } + return value; + } + + } else if ( hooks && "get" in hooks && notxml && (ret = hooks.get( elem, name )) !== null ) { + return ret; + + } else { + + ret = elem.getAttribute( name ); + + // Non-existent attributes return null, we normalize to undefined + return ret === null ? + undefined : + ret; + } + }, + + removeAttr: function( elem, value ) { + var propName, attrNames, name, l, isBool, + i = 0; + + if ( value && elem.nodeType === 1 ) { + attrNames = value.toLowerCase().split( rspace ); + l = attrNames.length; + + for ( ; i < l; i++ ) { + name = attrNames[ i ]; + + if ( name ) { + propName = jQuery.propFix[ name ] || name; + isBool = rboolean.test( name ); + + // See #9699 for explanation of this approach (setting first, then removal) + // Do not do this for boolean attributes (see #10870) + if ( !isBool ) { + jQuery.attr( elem, name, "" ); + } + elem.removeAttribute( getSetAttribute ? name : propName ); + + // Set corresponding property to false for boolean attributes + if ( isBool && propName in elem ) { + elem[ propName ] = false; + } + } + } + } + }, + + attrHooks: { + type: { + set: function( elem, value ) { + // We can't allow the type property to be changed (since it causes problems in IE) + if ( rtype.test( elem.nodeName ) && elem.parentNode ) { + jQuery.error( "type property can't be changed" ); + } else if ( !jQuery.support.radioValue && value === "radio" && jQuery.nodeName(elem, "input") ) { + // Setting the type on a radio button after the value resets the value in IE6-9 + // Reset value to it's default in case type is set after value + // This is for element creation + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + }, + // Use the value property for back compat + // Use the nodeHook for button elements in IE6/7 (#1954) + value: { + get: function( elem, name ) { + if ( nodeHook && jQuery.nodeName( elem, "button" ) ) { + return nodeHook.get( elem, name ); + } + return name in elem ? + elem.value : + null; + }, + set: function( elem, value, name ) { + if ( nodeHook && jQuery.nodeName( elem, "button" ) ) { + return nodeHook.set( elem, value, name ); + } + // Does not return so that setAttribute is also used + elem.value = value; + } + } + }, + + propFix: { + tabindex: "tabIndex", + readonly: "readOnly", + "for": "htmlFor", + "class": "className", + maxlength: "maxLength", + cellspacing: "cellSpacing", + cellpadding: "cellPadding", + rowspan: "rowSpan", + colspan: "colSpan", + usemap: "useMap", + frameborder: "frameBorder", + contenteditable: "contentEditable" + }, + + prop: function( elem, name, value ) { + var ret, hooks, notxml, + nType = elem.nodeType; + + // don't get/set properties on text, comment and attribute nodes + if ( !elem || nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + notxml = nType !== 1 || !jQuery.isXMLDoc( elem ); + + if ( notxml ) { + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && (ret = hooks.set( elem, value, name )) !== undefined ) { + return ret; + + } else { + return ( elem[ name ] = value ); + } + + } else { + if ( hooks && "get" in hooks && (ret = hooks.get( elem, name )) !== null ) { + return ret; + + } else { + return elem[ name ]; + } + } + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + // elem.tabIndex doesn't always return the correct value when it hasn't been explicitly set + // http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + var attributeNode = elem.getAttributeNode("tabindex"); + + return attributeNode && attributeNode.specified ? + parseInt( attributeNode.value, 10 ) : + rfocusable.test( elem.nodeName ) || rclickable.test( elem.nodeName ) && elem.href ? + 0 : + undefined; + } + } + } +}); + +// Add the tabIndex propHook to attrHooks for back-compat (different case is intentional) +jQuery.attrHooks.tabindex = jQuery.propHooks.tabIndex; + +// Hook for boolean attributes +boolHook = { + get: function( elem, name ) { + // Align boolean attributes with corresponding properties + // Fall back to attribute presence where some booleans are not supported + var attrNode, + property = jQuery.prop( elem, name ); + return property === true || typeof property !== "boolean" && ( attrNode = elem.getAttributeNode(name) ) && attrNode.nodeValue !== false ? + name.toLowerCase() : + undefined; + }, + set: function( elem, value, name ) { + var propName; + if ( value === false ) { + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + // value is true since we know at this point it's type boolean and not false + // Set boolean attributes to the same name and set the DOM property + propName = jQuery.propFix[ name ] || name; + if ( propName in elem ) { + // Only set the IDL specifically if it already exists on the element + elem[ propName ] = true; + } + + elem.setAttribute( name, name.toLowerCase() ); + } + return name; + } +}; + +// IE6/7 do not support getting/setting some attributes with get/setAttribute +if ( !getSetAttribute ) { + + fixSpecified = { + name: true, + id: true, + coords: true + }; + + // Use this for any attribute in IE6/7 + // This fixes almost every IE6/7 issue + nodeHook = jQuery.valHooks.button = { + get: function( elem, name ) { + var ret; + ret = elem.getAttributeNode( name ); + return ret && ( fixSpecified[ name ] ? ret.nodeValue !== "" : ret.specified ) ? + ret.nodeValue : + undefined; + }, + set: function( elem, value, name ) { + // Set the existing or create a new attribute node + var ret = elem.getAttributeNode( name ); + if ( !ret ) { + ret = document.createAttribute( name ); + elem.setAttributeNode( ret ); + } + return ( ret.nodeValue = value + "" ); + } + }; + + // Apply the nodeHook to tabindex + jQuery.attrHooks.tabindex.set = nodeHook.set; + + // Set width and height to auto instead of 0 on empty string( Bug #8150 ) + // This is for removals + jQuery.each([ "width", "height" ], function( i, name ) { + jQuery.attrHooks[ name ] = jQuery.extend( jQuery.attrHooks[ name ], { + set: function( elem, value ) { + if ( value === "" ) { + elem.setAttribute( name, "auto" ); + return value; + } + } + }); + }); + + // Set contenteditable to false on removals(#10429) + // Setting to empty string throws an error as an invalid value + jQuery.attrHooks.contenteditable = { + get: nodeHook.get, + set: function( elem, value, name ) { + if ( value === "" ) { + value = "false"; + } + nodeHook.set( elem, value, name ); + } + }; +} + + +// Some attributes require a special call on IE +if ( !jQuery.support.hrefNormalized ) { + jQuery.each([ "href", "src", "width", "height" ], function( i, name ) { + jQuery.attrHooks[ name ] = jQuery.extend( jQuery.attrHooks[ name ], { + get: function( elem ) { + var ret = elem.getAttribute( name, 2 ); + return ret === null ? undefined : ret; + } + }); + }); +} + +if ( !jQuery.support.style ) { + jQuery.attrHooks.style = { + get: function( elem ) { + // Return undefined in the case of empty string + // Normalize to lowercase since IE uppercases css property names + return elem.style.cssText.toLowerCase() || undefined; + }, + set: function( elem, value ) { + return ( elem.style.cssText = "" + value ); + } + }; +} + +// Safari mis-reports the default selected property of an option +// Accessing the parent's selectedIndex property fixes it +if ( !jQuery.support.optSelected ) { + jQuery.propHooks.selected = jQuery.extend( jQuery.propHooks.selected, { + get: function( elem ) { + var parent = elem.parentNode; + + if ( parent ) { + parent.selectedIndex; + + // Make sure that it also works with optgroups, see #5701 + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + return null; + } + }); +} + +// IE6/7 call enctype encoding +if ( !jQuery.support.enctype ) { + jQuery.propFix.enctype = "encoding"; +} + +// Radios and checkboxes getter/setter +if ( !jQuery.support.checkOn ) { + jQuery.each([ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + get: function( elem ) { + // Handle the case where in Webkit "" is returned instead of "on" if a value isn't specified + return elem.getAttribute("value") === null ? "on" : elem.value; + } + }; + }); +} +jQuery.each([ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = jQuery.extend( jQuery.valHooks[ this ], { + set: function( elem, value ) { + if ( jQuery.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery(elem).val(), value ) >= 0 ); + } + } + }); +}); + + + + +var rformElems = /^(?:textarea|input|select)$/i, + rtypenamespace = /^([^\.]*)?(?:\.(.+))?$/, + rhoverHack = /(?:^|\s)hover(\.\S+)?\b/, + rkeyEvent = /^key/, + rmouseEvent = /^(?:mouse|contextmenu)|click/, + rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + rquickIs = /^(\w*)(?:#([\w\-]+))?(?:\.([\w\-]+))?$/, + quickParse = function( selector ) { + var quick = rquickIs.exec( selector ); + if ( quick ) { + // 0 1 2 3 + // [ _, tag, id, class ] + quick[1] = ( quick[1] || "" ).toLowerCase(); + quick[3] = quick[3] && new RegExp( "(?:^|\\s)" + quick[3] + "(?:\\s|$)" ); + } + return quick; + }, + quickIs = function( elem, m ) { + var attrs = elem.attributes || {}; + return ( + (!m[1] || elem.nodeName.toLowerCase() === m[1]) && + (!m[2] || (attrs.id || {}).value === m[2]) && + (!m[3] || m[3].test( (attrs[ "class" ] || {}).value )) + ); + }, + hoverHack = function( events ) { + return jQuery.event.special.hover ? events : events.replace( rhoverHack, "mouseenter$1 mouseleave$1" ); + }; + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + add: function( elem, types, handler, data, selector ) { + + var elemData, eventHandle, events, + t, tns, type, namespaces, handleObj, + handleObjIn, quick, handlers, special; + + // Don't attach events to noData or text/comment nodes (allow plain objects tho) + if ( elem.nodeType === 3 || elem.nodeType === 8 || !types || !handler || !(elemData = jQuery._data( elem )) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + events = elemData.events; + if ( !events ) { + elemData.events = events = {}; + } + eventHandle = elemData.handle; + if ( !eventHandle ) { + elemData.handle = eventHandle = function( e ) { + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && (!e || jQuery.event.triggered !== e.type) ? + jQuery.event.dispatch.apply( eventHandle.elem, arguments ) : + undefined; + }; + // Add elem as a property of the handle fn to prevent a memory leak with IE non-native events + eventHandle.elem = elem; + } + + // Handle multiple events separated by a space + // jQuery(...).bind("mouseover mouseout", fn); + types = jQuery.trim( hoverHack(types) ).split( " " ); + for ( t = 0; t < types.length; t++ ) { + + tns = rtypenamespace.exec( types[t] ) || []; + type = tns[1]; + namespaces = ( tns[2] || "" ).split( "." ).sort(); + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend({ + type: type, + origType: tns[1], + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + quick: selector && quickParse( selector ), + namespace: namespaces.join(".") + }, handleObjIn ); + + // Init the event handler queue if we're the first + handlers = events[ type ]; + if ( !handlers ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener/attachEvent if the special events handler returns false + if ( !special.setup || special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + // Bind the global event handler to the element + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle, false ); + + } else if ( elem.attachEvent ) { + elem.attachEvent( "on" + type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + // Nullify elem to prevent memory leaks in IE + elem = null; + }, + + global: {}, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var elemData = jQuery.hasData( elem ) && jQuery._data( elem ), + t, tns, type, origType, namespaces, origCount, + j, events, special, handle, eventType, handleObj; + + if ( !elemData || !(events = elemData.events) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = jQuery.trim( hoverHack( types || "" ) ).split(" "); + for ( t = 0; t < types.length; t++ ) { + tns = rtypenamespace.exec( types[t] ) || []; + type = origType = tns[1]; + namespaces = tns[2]; + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector? special.delegateType : special.bindType ) || type; + eventType = events[ type ] || []; + origCount = eventType.length; + namespaces = namespaces ? new RegExp("(^|\\.)" + namespaces.split(".").sort().join("\\.(?:.*\\.)?") + "(\\.|$)") : null; + + // Remove matching events + for ( j = 0; j < eventType.length; j++ ) { + handleObj = eventType[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !namespaces || namespaces.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || selector === "**" && handleObj.selector ) ) { + eventType.splice( j--, 1 ); + + if ( handleObj.selector ) { + eventType.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( eventType.length === 0 && origCount !== eventType.length ) { + if ( !special.teardown || special.teardown.call( elem, namespaces ) === false ) { + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + handle = elemData.handle; + if ( handle ) { + handle.elem = null; + } + + // removeData also checks for emptiness and clears the expando if empty + // so use it instead of delete + jQuery.removeData( elem, [ "events", "handle" ], true ); + } + }, + + // Events that are safe to short-circuit if no handlers are attached. + // Native DOM events should not be added, they may have inline handlers. + customEvent: { + "getData": true, + "setData": true, + "changeData": true + }, + + trigger: function( event, data, elem, onlyHandlers ) { + // Don't do events on text and comment nodes + if ( elem && (elem.nodeType === 3 || elem.nodeType === 8) ) { + return; + } + + // Event object or event type + var type = event.type || event, + namespaces = [], + cache, exclusive, i, cur, old, ontype, special, handle, eventPath, bubbleType; + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "!" ) >= 0 ) { + // Exclusive events trigger only for the exact event (no namespaces) + type = type.slice(0, -1); + exclusive = true; + } + + if ( type.indexOf( "." ) >= 0 ) { + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split("."); + type = namespaces.shift(); + namespaces.sort(); + } + + if ( (!elem || jQuery.event.customEvent[ type ]) && !jQuery.event.global[ type ] ) { + // No jQuery handlers for this event type, and it can't have inline handlers + return; + } + + // Caller can pass in an Event, Object, or just an event type string + event = typeof event === "object" ? + // jQuery.Event object + event[ jQuery.expando ] ? event : + // Object literal + new jQuery.Event( type, event ) : + // Just the event type (string) + new jQuery.Event( type ); + + event.type = type; + event.isTrigger = true; + event.exclusive = exclusive; + event.namespace = namespaces.join( "." ); + event.namespace_re = event.namespace? new RegExp("(^|\\.)" + namespaces.join("\\.(?:.*\\.)?") + "(\\.|$)") : null; + ontype = type.indexOf( ":" ) < 0 ? "on" + type : ""; + + // Handle a global trigger + if ( !elem ) { + + // TODO: Stop taunting the data cache; remove global events and always attach to document + cache = jQuery.cache; + for ( i in cache ) { + if ( cache[ i ].events && cache[ i ].events[ type ] ) { + jQuery.event.trigger( event, data, cache[ i ].handle.elem, true ); + } + } + return; + } + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data != null ? jQuery.makeArray( data ) : []; + data.unshift( event ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + eventPath = [[ elem, special.bindType || type ]]; + if ( !onlyHandlers && !special.noBubble && !jQuery.isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + cur = rfocusMorph.test( bubbleType + type ) ? elem : elem.parentNode; + old = null; + for ( ; cur; cur = cur.parentNode ) { + eventPath.push([ cur, bubbleType ]); + old = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( old && old === elem.ownerDocument ) { + eventPath.push([ old.defaultView || old.parentWindow || window, bubbleType ]); + } + } + + // Fire handlers on the event path + for ( i = 0; i < eventPath.length && !event.isPropagationStopped(); i++ ) { + + cur = eventPath[i][0]; + event.type = eventPath[i][1]; + + handle = ( jQuery._data( cur, "events" ) || {} )[ event.type ] && jQuery._data( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + // Note that this is a bare JS function and not a jQuery handler + handle = ontype && cur[ ontype ]; + if ( handle && jQuery.acceptData( cur ) && handle.apply( cur, data ) === false ) { + event.preventDefault(); + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( (!special._default || special._default.apply( elem.ownerDocument, data ) === false) && + !(type === "click" && jQuery.nodeName( elem, "a" )) && jQuery.acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name name as the event. + // Can't use an .isFunction() check here because IE6/7 fails that test. + // Don't do default actions on window, that's where global variables be (#6170) + // IE<9 dies on focus/blur to hidden element (#1486) + if ( ontype && elem[ type ] && ((type !== "focus" && type !== "blur") || event.target.offsetWidth !== 0) && !jQuery.isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + old = elem[ ontype ]; + + if ( old ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + elem[ type ](); + jQuery.event.triggered = undefined; + + if ( old ) { + elem[ ontype ] = old; + } + } + } + } + + return event.result; + }, + + dispatch: function( event ) { + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( event || window.event ); + + var handlers = ( (jQuery._data( this, "events" ) || {} )[ event.type ] || []), + delegateCount = handlers.delegateCount, + args = [].slice.call( arguments, 0 ), + run_all = !event.exclusive && !event.namespace, + special = jQuery.event.special[ event.type ] || {}, + handlerQueue = [], + i, j, cur, jqcur, ret, selMatch, matched, matches, handleObj, sel, related; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[0] = event; + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers that should run if there are delegated events + // Avoid non-left-click bubbling in Firefox (#3861) + if ( delegateCount && !(event.button && event.type === "click") ) { + + // Pregenerate a single jQuery object for reuse with .is() + jqcur = jQuery(this); + jqcur.context = this.ownerDocument || this; + + for ( cur = event.target; cur != this; cur = cur.parentNode || this ) { + + // Don't process events on disabled elements (#6911, #8165) + if ( cur.disabled !== true ) { + selMatch = {}; + matches = []; + jqcur[0] = cur; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + sel = handleObj.selector; + + if ( selMatch[ sel ] === undefined ) { + selMatch[ sel ] = ( + handleObj.quick ? quickIs( cur, handleObj.quick ) : jqcur.is( sel ) + ); + } + if ( selMatch[ sel ] ) { + matches.push( handleObj ); + } + } + if ( matches.length ) { + handlerQueue.push({ elem: cur, matches: matches }); + } + } + } + } + + // Add the remaining (directly-bound) handlers + if ( handlers.length > delegateCount ) { + handlerQueue.push({ elem: this, matches: handlers.slice( delegateCount ) }); + } + + // Run delegates first; they may want to stop propagation beneath us + for ( i = 0; i < handlerQueue.length && !event.isPropagationStopped(); i++ ) { + matched = handlerQueue[ i ]; + event.currentTarget = matched.elem; + + for ( j = 0; j < matched.matches.length && !event.isImmediatePropagationStopped(); j++ ) { + handleObj = matched.matches[ j ]; + + // Triggered event must either 1) be non-exclusive and have no namespace, or + // 2) have namespace(s) a subset or equal to those in the bound event (both can have no namespace). + if ( run_all || (!event.namespace && !handleObj.namespace) || event.namespace_re && event.namespace_re.test( handleObj.namespace ) ) { + + event.data = handleObj.data; + event.handleObj = handleObj; + + ret = ( (jQuery.event.special[ handleObj.origType ] || {}).handle || handleObj.handler ) + .apply( matched.elem, args ); + + if ( ret !== undefined ) { + event.result = ret; + if ( ret === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + // Includes some event props shared by KeyEvent and MouseEvent + // *** attrChange attrName relatedNode srcElement are not normalized, non-W3C, deprecated, will be removed in 1.8 *** + props: "attrChange attrName relatedNode srcElement altKey bubbles cancelable ctrlKey currentTarget eventPhase metaKey relatedTarget shiftKey target timeStamp view which".split(" "), + + fixHooks: {}, + + keyHooks: { + props: "char charCode key keyCode".split(" "), + filter: function( event, original ) { + + // Add which for key events + if ( event.which == null ) { + event.which = original.charCode != null ? original.charCode : original.keyCode; + } + + return event; + } + }, + + mouseHooks: { + props: "button buttons clientX clientY fromElement offsetX offsetY pageX pageY screenX screenY toElement".split(" "), + filter: function( event, original ) { + var eventDoc, doc, body, + button = original.button, + fromElement = original.fromElement; + + // Calculate pageX/Y if missing and clientX/Y available + if ( event.pageX == null && original.clientX != null ) { + eventDoc = event.target.ownerDocument || document; + doc = eventDoc.documentElement; + body = eventDoc.body; + + event.pageX = original.clientX + ( doc && doc.scrollLeft || body && body.scrollLeft || 0 ) - ( doc && doc.clientLeft || body && body.clientLeft || 0 ); + event.pageY = original.clientY + ( doc && doc.scrollTop || body && body.scrollTop || 0 ) - ( doc && doc.clientTop || body && body.clientTop || 0 ); + } + + // Add relatedTarget, if necessary + if ( !event.relatedTarget && fromElement ) { + event.relatedTarget = fromElement === event.target ? original.toElement : fromElement; + } + + // Add which for click: 1 === left; 2 === middle; 3 === right + // Note: button is not normalized, so don't use it + if ( !event.which && button !== undefined ) { + event.which = ( button & 1 ? 1 : ( button & 2 ? 3 : ( button & 4 ? 2 : 0 ) ) ); + } + + return event; + } + }, + + fix: function( event ) { + if ( event[ jQuery.expando ] ) { + return event; + } + + // Create a writable copy of the event object and normalize some properties + var i, prop, + originalEvent = event, + fixHook = jQuery.event.fixHooks[ event.type ] || {}, + copy = fixHook.props ? this.props.concat( fixHook.props ) : this.props; + + event = jQuery.Event( originalEvent ); + + for ( i = copy.length; i; ) { + prop = copy[ --i ]; + event[ prop ] = originalEvent[ prop ]; + } + + // Fix target property, if necessary (#1925, IE 6/7/8 & Safari2) + if ( !event.target ) { + event.target = originalEvent.srcElement || document; + } + + // Target should not be a text node (#504, Safari) + if ( event.target.nodeType === 3 ) { + event.target = event.target.parentNode; + } + + // For mouse/key events; add metaKey if it's not there (#3368, IE6/7/8) + if ( event.metaKey === undefined ) { + event.metaKey = event.ctrlKey; + } + + return fixHook.filter? fixHook.filter( event, originalEvent ) : event; + }, + + special: { + ready: { + // Make sure the ready event is setup + setup: jQuery.bindReady + }, + + load: { + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + + focus: { + delegateType: "focusin" + }, + blur: { + delegateType: "focusout" + }, + + beforeunload: { + setup: function( data, namespaces, eventHandle ) { + // We only want to do this special case on windows + if ( jQuery.isWindow( this ) ) { + this.onbeforeunload = eventHandle; + } + }, + + teardown: function( namespaces, eventHandle ) { + if ( this.onbeforeunload === eventHandle ) { + this.onbeforeunload = null; + } + } + } + }, + + simulate: function( type, elem, event, bubble ) { + // Piggyback on a donor event to simulate a different one. + // Fake originalEvent to avoid donor's stopPropagation, but if the + // simulated event prevents default then we do the same on the donor. + var e = jQuery.extend( + new jQuery.Event(), + event, + { type: type, + isSimulated: true, + originalEvent: {} + } + ); + if ( bubble ) { + jQuery.event.trigger( e, null, elem ); + } else { + jQuery.event.dispatch.call( elem, e ); + } + if ( e.isDefaultPrevented() ) { + event.preventDefault(); + } + } +}; + +// Some plugins are using, but it's undocumented/deprecated and will be removed. +// The 1.7 special event interface should provide all the hooks needed now. +jQuery.event.handle = jQuery.event.dispatch; + +jQuery.removeEvent = document.removeEventListener ? + function( elem, type, handle ) { + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle, false ); + } + } : + function( elem, type, handle ) { + if ( elem.detachEvent ) { + elem.detachEvent( "on" + type, handle ); + } + }; + +jQuery.Event = function( src, props ) { + // Allow instantiation without the 'new' keyword + if ( !(this instanceof jQuery.Event) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = ( src.defaultPrevented || src.returnValue === false || + src.getPreventDefault && src.getPreventDefault() ) ? returnTrue : returnFalse; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || jQuery.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +function returnFalse() { + return false; +} +function returnTrue() { + return true; +} + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// http://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + preventDefault: function() { + this.isDefaultPrevented = returnTrue; + + var e = this.originalEvent; + if ( !e ) { + return; + } + + // if preventDefault exists run it on the original event + if ( e.preventDefault ) { + e.preventDefault(); + + // otherwise set the returnValue property of the original event to false (IE) + } else { + e.returnValue = false; + } + }, + stopPropagation: function() { + this.isPropagationStopped = returnTrue; + + var e = this.originalEvent; + if ( !e ) { + return; + } + // if stopPropagation exists run it on the original event + if ( e.stopPropagation ) { + e.stopPropagation(); + } + // otherwise set the cancelBubble property of the original event to true (IE) + e.cancelBubble = true; + }, + stopImmediatePropagation: function() { + this.isImmediatePropagationStopped = returnTrue; + this.stopPropagation(); + }, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse +}; + +// Create mouseenter/leave events using mouseover/out and event-time checks +jQuery.each({ + mouseenter: "mouseover", + mouseleave: "mouseout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var target = this, + related = event.relatedTarget, + handleObj = event.handleObj, + selector = handleObj.selector, + ret; + + // For mousenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || (related !== target && !jQuery.contains( target, related )) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +}); + +// IE submit delegation +if ( !jQuery.support.submitBubbles ) { + + jQuery.event.special.submit = { + setup: function() { + // Only need this for delegated form submit events + if ( jQuery.nodeName( this, "form" ) ) { + return false; + } + + // Lazy-add a submit handler when a descendant form may potentially be submitted + jQuery.event.add( this, "click._submit keypress._submit", function( e ) { + // Node name check avoids a VML-related crash in IE (#9807) + var elem = e.target, + form = jQuery.nodeName( elem, "input" ) || jQuery.nodeName( elem, "button" ) ? elem.form : undefined; + if ( form && !form._submit_attached ) { + jQuery.event.add( form, "submit._submit", function( event ) { + event._submit_bubble = true; + }); + form._submit_attached = true; + } + }); + // return undefined since we don't need an event listener + }, + + postDispatch: function( event ) { + // If form was submitted by the user, bubble the event up the tree + if ( event._submit_bubble ) { + delete event._submit_bubble; + if ( this.parentNode && !event.isTrigger ) { + jQuery.event.simulate( "submit", this.parentNode, event, true ); + } + } + }, + + teardown: function() { + // Only need this for delegated form submit events + if ( jQuery.nodeName( this, "form" ) ) { + return false; + } + + // Remove delegated handlers; cleanData eventually reaps submit handlers attached above + jQuery.event.remove( this, "._submit" ); + } + }; +} + +// IE change delegation and checkbox/radio fix +if ( !jQuery.support.changeBubbles ) { + + jQuery.event.special.change = { + + setup: function() { + + if ( rformElems.test( this.nodeName ) ) { + // IE doesn't fire change on a check/radio until blur; trigger it on click + // after a propertychange. Eat the blur-change in special.change.handle. + // This still fires onchange a second time for check/radio after blur. + if ( this.type === "checkbox" || this.type === "radio" ) { + jQuery.event.add( this, "propertychange._change", function( event ) { + if ( event.originalEvent.propertyName === "checked" ) { + this._just_changed = true; + } + }); + jQuery.event.add( this, "click._change", function( event ) { + if ( this._just_changed && !event.isTrigger ) { + this._just_changed = false; + jQuery.event.simulate( "change", this, event, true ); + } + }); + } + return false; + } + // Delegated event; lazy-add a change handler on descendant inputs + jQuery.event.add( this, "beforeactivate._change", function( e ) { + var elem = e.target; + + if ( rformElems.test( elem.nodeName ) && !elem._change_attached ) { + jQuery.event.add( elem, "change._change", function( event ) { + if ( this.parentNode && !event.isSimulated && !event.isTrigger ) { + jQuery.event.simulate( "change", this.parentNode, event, true ); + } + }); + elem._change_attached = true; + } + }); + }, + + handle: function( event ) { + var elem = event.target; + + // Swallow native change events from checkbox/radio, we already triggered them above + if ( this !== elem || event.isSimulated || event.isTrigger || (elem.type !== "radio" && elem.type !== "checkbox") ) { + return event.handleObj.handler.apply( this, arguments ); + } + }, + + teardown: function() { + jQuery.event.remove( this, "._change" ); + + return rformElems.test( this.nodeName ); + } + }; +} + +// Create "bubbling" focus and blur events +if ( !jQuery.support.focusinBubbles ) { + jQuery.each({ focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler while someone wants focusin/focusout + var attaches = 0, + handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ), true ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + if ( attaches++ === 0 ) { + document.addEventListener( orig, handler, true ); + } + }, + teardown: function() { + if ( --attaches === 0 ) { + document.removeEventListener( orig, handler, true ); + } + } + }; + }); +} + +jQuery.fn.extend({ + + on: function( types, selector, data, fn, /*INTERNAL*/ one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { // && selector != null + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + this.on( type, selector, data, types[ type ], one ); + } + return this; + } + + if ( data == null && fn == null ) { + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return this; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return this.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + }); + }, + one: function( types, selector, data, fn ) { + return this.on( types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + if ( types && types.preventDefault && types.handleObj ) { + // ( event ) dispatched jQuery.Event + var handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? handleObj.origType + "." + handleObj.namespace : handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + // ( types-object [, selector] ) + for ( var type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each(function() { + jQuery.event.remove( this, types, fn, selector ); + }); + }, + + bind: function( types, data, fn ) { + return this.on( types, null, data, fn ); + }, + unbind: function( types, fn ) { + return this.off( types, null, fn ); + }, + + live: function( types, data, fn ) { + jQuery( this.context ).on( types, this.selector, data, fn ); + return this; + }, + die: function( types, fn ) { + jQuery( this.context ).off( types, this.selector || "**", fn ); + return this; + }, + + delegate: function( selector, types, data, fn ) { + return this.on( types, selector, data, fn ); + }, + undelegate: function( selector, types, fn ) { + // ( namespace ) or ( selector, types [, fn] ) + return arguments.length == 1? this.off( selector, "**" ) : this.off( types, selector, fn ); + }, + + trigger: function( type, data ) { + return this.each(function() { + jQuery.event.trigger( type, data, this ); + }); + }, + triggerHandler: function( type, data ) { + if ( this[0] ) { + return jQuery.event.trigger( type, data, this[0], true ); + } + }, + + toggle: function( fn ) { + // Save reference to arguments for access in closure + var args = arguments, + guid = fn.guid || jQuery.guid++, + i = 0, + toggler = function( event ) { + // Figure out which function to execute + var lastToggle = ( jQuery._data( this, "lastToggle" + fn.guid ) || 0 ) % i; + jQuery._data( this, "lastToggle" + fn.guid, lastToggle + 1 ); + + // Make sure that clicks stop + event.preventDefault(); + + // and execute the function + return args[ lastToggle ].apply( this, arguments ) || false; + }; + + // link all the functions, so any of them can unbind this click handler + toggler.guid = guid; + while ( i < args.length ) { + args[ i++ ].guid = guid; + } + + return this.click( toggler ); + }, + + hover: function( fnOver, fnOut ) { + return this.mouseenter( fnOver ).mouseleave( fnOut || fnOver ); + } +}); + +jQuery.each( ("blur focus focusin focusout load resize scroll unload click dblclick " + + "mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave " + + "change select submit keydown keypress keyup error contextmenu").split(" "), function( i, name ) { + + // Handle event binding + jQuery.fn[ name ] = function( data, fn ) { + if ( fn == null ) { + fn = data; + data = null; + } + + return arguments.length > 0 ? + this.on( name, null, data, fn ) : + this.trigger( name ); + }; + + if ( jQuery.attrFn ) { + jQuery.attrFn[ name ] = true; + } + + if ( rkeyEvent.test( name ) ) { + jQuery.event.fixHooks[ name ] = jQuery.event.keyHooks; + } + + if ( rmouseEvent.test( name ) ) { + jQuery.event.fixHooks[ name ] = jQuery.event.mouseHooks; + } +}); + + + +/*! + * Sizzle CSS Selector Engine + * Copyright 2011, The Dojo Foundation + * Released under the MIT, BSD, and GPL Licenses. + * More information: http://sizzlejs.com/ + */ +(function(){ + +var chunker = /((?:\((?:\([^()]+\)|[^()]+)+\)|\[(?:\[[^\[\]]*\]|['"][^'"]*['"]|[^\[\]'"]+)+\]|\\.|[^ >+~,(\[\\]+)+|[>+~])(\s*,\s*)?((?:.|\r|\n)*)/g, + expando = "sizcache" + (Math.random() + '').replace('.', ''), + done = 0, + toString = Object.prototype.toString, + hasDuplicate = false, + baseHasDuplicate = true, + rBackslash = /\\/g, + rReturn = /\r\n/g, + rNonWord = /\W/; + +// Here we check if the JavaScript engine is using some sort of +// optimization where it does not always call our comparision +// function. If that is the case, discard the hasDuplicate value. +// Thus far that includes Google Chrome. +[0, 0].sort(function() { + baseHasDuplicate = false; + return 0; +}); + +var Sizzle = function( selector, context, results, seed ) { + results = results || []; + context = context || document; + + var origContext = context; + + if ( context.nodeType !== 1 && context.nodeType !== 9 ) { + return []; + } + + if ( !selector || typeof selector !== "string" ) { + return results; + } + + var m, set, checkSet, extra, ret, cur, pop, i, + prune = true, + contextXML = Sizzle.isXML( context ), + parts = [], + soFar = selector; + + // Reset the position of the chunker regexp (start from head) + do { + chunker.exec( "" ); + m = chunker.exec( soFar ); + + if ( m ) { + soFar = m[3]; + + parts.push( m[1] ); + + if ( m[2] ) { + extra = m[3]; + break; + } + } + } while ( m ); + + if ( parts.length > 1 && origPOS.exec( selector ) ) { + + if ( parts.length === 2 && Expr.relative[ parts[0] ] ) { + set = posProcess( parts[0] + parts[1], context, seed ); + + } else { + set = Expr.relative[ parts[0] ] ? + [ context ] : + Sizzle( parts.shift(), context ); + + while ( parts.length ) { + selector = parts.shift(); + + if ( Expr.relative[ selector ] ) { + selector += parts.shift(); + } + + set = posProcess( selector, set, seed ); + } + } + + } else { + // Take a shortcut and set the context if the root selector is an ID + // (but not if it'll be faster if the inner selector is an ID) + if ( !seed && parts.length > 1 && context.nodeType === 9 && !contextXML && + Expr.match.ID.test(parts[0]) && !Expr.match.ID.test(parts[parts.length - 1]) ) { + + ret = Sizzle.find( parts.shift(), context, contextXML ); + context = ret.expr ? + Sizzle.filter( ret.expr, ret.set )[0] : + ret.set[0]; + } + + if ( context ) { + ret = seed ? + { expr: parts.pop(), set: makeArray(seed) } : + Sizzle.find( parts.pop(), parts.length === 1 && (parts[0] === "~" || parts[0] === "+") && context.parentNode ? context.parentNode : context, contextXML ); + + set = ret.expr ? + Sizzle.filter( ret.expr, ret.set ) : + ret.set; + + if ( parts.length > 0 ) { + checkSet = makeArray( set ); + + } else { + prune = false; + } + + while ( parts.length ) { + cur = parts.pop(); + pop = cur; + + if ( !Expr.relative[ cur ] ) { + cur = ""; + } else { + pop = parts.pop(); + } + + if ( pop == null ) { + pop = context; + } + + Expr.relative[ cur ]( checkSet, pop, contextXML ); + } + + } else { + checkSet = parts = []; + } + } + + if ( !checkSet ) { + checkSet = set; + } + + if ( !checkSet ) { + Sizzle.error( cur || selector ); + } + + if ( toString.call(checkSet) === "[object Array]" ) { + if ( !prune ) { + results.push.apply( results, checkSet ); + + } else if ( context && context.nodeType === 1 ) { + for ( i = 0; checkSet[i] != null; i++ ) { + if ( checkSet[i] && (checkSet[i] === true || checkSet[i].nodeType === 1 && Sizzle.contains(context, checkSet[i])) ) { + results.push( set[i] ); + } + } + + } else { + for ( i = 0; checkSet[i] != null; i++ ) { + if ( checkSet[i] && checkSet[i].nodeType === 1 ) { + results.push( set[i] ); + } + } + } + + } else { + makeArray( checkSet, results ); + } + + if ( extra ) { + Sizzle( extra, origContext, results, seed ); + Sizzle.uniqueSort( results ); + } + + return results; +}; + +Sizzle.uniqueSort = function( results ) { + if ( sortOrder ) { + hasDuplicate = baseHasDuplicate; + results.sort( sortOrder ); + + if ( hasDuplicate ) { + for ( var i = 1; i < results.length; i++ ) { + if ( results[i] === results[ i - 1 ] ) { + results.splice( i--, 1 ); + } + } + } + } + + return results; +}; + +Sizzle.matches = function( expr, set ) { + return Sizzle( expr, null, null, set ); +}; + +Sizzle.matchesSelector = function( node, expr ) { + return Sizzle( expr, null, null, [node] ).length > 0; +}; + +Sizzle.find = function( expr, context, isXML ) { + var set, i, len, match, type, left; + + if ( !expr ) { + return []; + } + + for ( i = 0, len = Expr.order.length; i < len; i++ ) { + type = Expr.order[i]; + + if ( (match = Expr.leftMatch[ type ].exec( expr )) ) { + left = match[1]; + match.splice( 1, 1 ); + + if ( left.substr( left.length - 1 ) !== "\\" ) { + match[1] = (match[1] || "").replace( rBackslash, "" ); + set = Expr.find[ type ]( match, context, isXML ); + + if ( set != null ) { + expr = expr.replace( Expr.match[ type ], "" ); + break; + } + } + } + } + + if ( !set ) { + set = typeof context.getElementsByTagName !== "undefined" ? + context.getElementsByTagName( "*" ) : + []; + } + + return { set: set, expr: expr }; +}; + +Sizzle.filter = function( expr, set, inplace, not ) { + var match, anyFound, + type, found, item, filter, left, + i, pass, + old = expr, + result = [], + curLoop = set, + isXMLFilter = set && set[0] && Sizzle.isXML( set[0] ); + + while ( expr && set.length ) { + for ( type in Expr.filter ) { + if ( (match = Expr.leftMatch[ type ].exec( expr )) != null && match[2] ) { + filter = Expr.filter[ type ]; + left = match[1]; + + anyFound = false; + + match.splice(1,1); + + if ( left.substr( left.length - 1 ) === "\\" ) { + continue; + } + + if ( curLoop === result ) { + result = []; + } + + if ( Expr.preFilter[ type ] ) { + match = Expr.preFilter[ type ]( match, curLoop, inplace, result, not, isXMLFilter ); + + if ( !match ) { + anyFound = found = true; + + } else if ( match === true ) { + continue; + } + } + + if ( match ) { + for ( i = 0; (item = curLoop[i]) != null; i++ ) { + if ( item ) { + found = filter( item, match, i, curLoop ); + pass = not ^ found; + + if ( inplace && found != null ) { + if ( pass ) { + anyFound = true; + + } else { + curLoop[i] = false; + } + + } else if ( pass ) { + result.push( item ); + anyFound = true; + } + } + } + } + + if ( found !== undefined ) { + if ( !inplace ) { + curLoop = result; + } + + expr = expr.replace( Expr.match[ type ], "" ); + + if ( !anyFound ) { + return []; + } + + break; + } + } + } + + // Improper expression + if ( expr === old ) { + if ( anyFound == null ) { + Sizzle.error( expr ); + + } else { + break; + } + } + + old = expr; + } + + return curLoop; +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Utility function for retreiving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +var getText = Sizzle.getText = function( elem ) { + var i, node, + nodeType = elem.nodeType, + ret = ""; + + if ( nodeType ) { + if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + // Use textContent || innerText for elements + if ( typeof elem.textContent === 'string' ) { + return elem.textContent; + } else if ( typeof elem.innerText === 'string' ) { + // Replace IE's carriage returns + return elem.innerText.replace( rReturn, '' ); + } else { + // Traverse it's children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + } else { + + // If no nodeType, this is expected to be an array + for ( i = 0; (node = elem[i]); i++ ) { + // Do not traverse comment nodes + if ( node.nodeType !== 8 ) { + ret += getText( node ); + } + } + } + return ret; +}; + +var Expr = Sizzle.selectors = { + order: [ "ID", "NAME", "TAG" ], + + match: { + ID: /#((?:[\w\u00c0-\uFFFF\-]|\\.)+)/, + CLASS: /\.((?:[\w\u00c0-\uFFFF\-]|\\.)+)/, + NAME: /\[name=['"]*((?:[\w\u00c0-\uFFFF\-]|\\.)+)['"]*\]/, + ATTR: /\[\s*((?:[\w\u00c0-\uFFFF\-]|\\.)+)\s*(?:(\S?=)\s*(?:(['"])(.*?)\3|(#?(?:[\w\u00c0-\uFFFF\-]|\\.)*)|)|)\s*\]/, + TAG: /^((?:[\w\u00c0-\uFFFF\*\-]|\\.)+)/, + CHILD: /:(only|nth|last|first)-child(?:\(\s*(even|odd|(?:[+\-]?\d+|(?:[+\-]?\d*)?n\s*(?:[+\-]\s*\d+)?))\s*\))?/, + POS: /:(nth|eq|gt|lt|first|last|even|odd)(?:\((\d*)\))?(?=[^\-]|$)/, + PSEUDO: /:((?:[\w\u00c0-\uFFFF\-]|\\.)+)(?:\((['"]?)((?:\([^\)]+\)|[^\(\)]*)+)\2\))?/ + }, + + leftMatch: {}, + + attrMap: { + "class": "className", + "for": "htmlFor" + }, + + attrHandle: { + href: function( elem ) { + return elem.getAttribute( "href" ); + }, + type: function( elem ) { + return elem.getAttribute( "type" ); + } + }, + + relative: { + "+": function(checkSet, part){ + var isPartStr = typeof part === "string", + isTag = isPartStr && !rNonWord.test( part ), + isPartStrNotTag = isPartStr && !isTag; + + if ( isTag ) { + part = part.toLowerCase(); + } + + for ( var i = 0, l = checkSet.length, elem; i < l; i++ ) { + if ( (elem = checkSet[i]) ) { + while ( (elem = elem.previousSibling) && elem.nodeType !== 1 ) {} + + checkSet[i] = isPartStrNotTag || elem && elem.nodeName.toLowerCase() === part ? + elem || false : + elem === part; + } + } + + if ( isPartStrNotTag ) { + Sizzle.filter( part, checkSet, true ); + } + }, + + ">": function( checkSet, part ) { + var elem, + isPartStr = typeof part === "string", + i = 0, + l = checkSet.length; + + if ( isPartStr && !rNonWord.test( part ) ) { + part = part.toLowerCase(); + + for ( ; i < l; i++ ) { + elem = checkSet[i]; + + if ( elem ) { + var parent = elem.parentNode; + checkSet[i] = parent.nodeName.toLowerCase() === part ? parent : false; + } + } + + } else { + for ( ; i < l; i++ ) { + elem = checkSet[i]; + + if ( elem ) { + checkSet[i] = isPartStr ? + elem.parentNode : + elem.parentNode === part; + } + } + + if ( isPartStr ) { + Sizzle.filter( part, checkSet, true ); + } + } + }, + + "": function(checkSet, part, isXML){ + var nodeCheck, + doneName = done++, + checkFn = dirCheck; + + if ( typeof part === "string" && !rNonWord.test( part ) ) { + part = part.toLowerCase(); + nodeCheck = part; + checkFn = dirNodeCheck; + } + + checkFn( "parentNode", part, doneName, checkSet, nodeCheck, isXML ); + }, + + "~": function( checkSet, part, isXML ) { + var nodeCheck, + doneName = done++, + checkFn = dirCheck; + + if ( typeof part === "string" && !rNonWord.test( part ) ) { + part = part.toLowerCase(); + nodeCheck = part; + checkFn = dirNodeCheck; + } + + checkFn( "previousSibling", part, doneName, checkSet, nodeCheck, isXML ); + } + }, + + find: { + ID: function( match, context, isXML ) { + if ( typeof context.getElementById !== "undefined" && !isXML ) { + var m = context.getElementById(match[1]); + // Check parentNode to catch when Blackberry 4.6 returns + // nodes that are no longer in the document #6963 + return m && m.parentNode ? [m] : []; + } + }, + + NAME: function( match, context ) { + if ( typeof context.getElementsByName !== "undefined" ) { + var ret = [], + results = context.getElementsByName( match[1] ); + + for ( var i = 0, l = results.length; i < l; i++ ) { + if ( results[i].getAttribute("name") === match[1] ) { + ret.push( results[i] ); + } + } + + return ret.length === 0 ? null : ret; + } + }, + + TAG: function( match, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( match[1] ); + } + } + }, + preFilter: { + CLASS: function( match, curLoop, inplace, result, not, isXML ) { + match = " " + match[1].replace( rBackslash, "" ) + " "; + + if ( isXML ) { + return match; + } + + for ( var i = 0, elem; (elem = curLoop[i]) != null; i++ ) { + if ( elem ) { + if ( not ^ (elem.className && elem.className.baseVal && (" " + elem.className.baseVal + " ").replace(/[\t\n\r]/g, " ").indexOf(match) >= 0) ) { + if ( !inplace ) { + result.push( elem ); + } + + } else if ( inplace ) { + curLoop[i] = false; + } + } + } + + return false; + }, + + ID: function( match ) { + return match[1].replace( rBackslash, "" ); + }, + + TAG: function( match, curLoop ) { + return match[1].replace( rBackslash, "" ).toLowerCase(); + }, + + CHILD: function( match ) { + if ( match[1] === "nth" ) { + if ( !match[2] ) { + Sizzle.error( match[0] ); + } + + match[2] = match[2].replace(/^\+|\s*/g, ''); + + // parse equations like 'even', 'odd', '5', '2n', '3n+2', '4n-1', '-n+6' + var test = /(-?)(\d*)(?:n([+\-]?\d*))?/.exec( + match[2] === "even" && "2n" || match[2] === "odd" && "2n+1" || + !/\D/.test( match[2] ) && "0n+" + match[2] || match[2]); + + // calculate the numbers (first)n+(last) including if they are negative + match[2] = (test[1] + (test[2] || 1)) - 0; + match[3] = test[3] - 0; + } + else if ( match[2] ) { + Sizzle.error( match[0] ); + } + + // TODO: Move to normal caching system + match[0] = done++; + + return match; + }, + + ATTR: function( match, curLoop, inplace, result, not, isXML ) { + var name = match[1] = match[1].replace( rBackslash, "" ); + + if ( !isXML && Expr.attrMap[name] ) { + match[1] = Expr.attrMap[name]; + } + + // Handle if an un-quoted value was used + match[4] = ( match[4] || match[5] || "" ).replace( rBackslash, "" ); + + if ( match[2] === "~=" ) { + match[4] = " " + match[4] + " "; + } + + return match; + }, + + PSEUDO: function( match, curLoop, inplace, result, not ) { + if ( match[1] === "not" ) { + // If we're dealing with a complex expression, or a simple one + if ( ( chunker.exec(match[3]) || "" ).length > 1 || /^\w/.test(match[3]) ) { + match[3] = Sizzle(match[3], null, null, curLoop); + + } else { + var ret = Sizzle.filter(match[3], curLoop, inplace, true ^ not); + + if ( !inplace ) { + result.push.apply( result, ret ); + } + + return false; + } + + } else if ( Expr.match.POS.test( match[0] ) || Expr.match.CHILD.test( match[0] ) ) { + return true; + } + + return match; + }, + + POS: function( match ) { + match.unshift( true ); + + return match; + } + }, + + filters: { + enabled: function( elem ) { + return elem.disabled === false && elem.type !== "hidden"; + }, + + disabled: function( elem ) { + return elem.disabled === true; + }, + + checked: function( elem ) { + return elem.checked === true; + }, + + selected: function( elem ) { + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + parent: function( elem ) { + return !!elem.firstChild; + }, + + empty: function( elem ) { + return !elem.firstChild; + }, + + has: function( elem, i, match ) { + return !!Sizzle( match[3], elem ).length; + }, + + header: function( elem ) { + return (/h\d/i).test( elem.nodeName ); + }, + + text: function( elem ) { + var attr = elem.getAttribute( "type" ), type = elem.type; + // IE6 and 7 will map elem.type to 'text' for new HTML5 types (search, etc) + // use getAttribute instead to test this case + return elem.nodeName.toLowerCase() === "input" && "text" === type && ( attr === type || attr === null ); + }, + + radio: function( elem ) { + return elem.nodeName.toLowerCase() === "input" && "radio" === elem.type; + }, + + checkbox: function( elem ) { + return elem.nodeName.toLowerCase() === "input" && "checkbox" === elem.type; + }, + + file: function( elem ) { + return elem.nodeName.toLowerCase() === "input" && "file" === elem.type; + }, + + password: function( elem ) { + return elem.nodeName.toLowerCase() === "input" && "password" === elem.type; + }, + + submit: function( elem ) { + var name = elem.nodeName.toLowerCase(); + return (name === "input" || name === "button") && "submit" === elem.type; + }, + + image: function( elem ) { + return elem.nodeName.toLowerCase() === "input" && "image" === elem.type; + }, + + reset: function( elem ) { + var name = elem.nodeName.toLowerCase(); + return (name === "input" || name === "button") && "reset" === elem.type; + }, + + button: function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && "button" === elem.type || name === "button"; + }, + + input: function( elem ) { + return (/input|select|textarea|button/i).test( elem.nodeName ); + }, + + focus: function( elem ) { + return elem === elem.ownerDocument.activeElement; + } + }, + setFilters: { + first: function( elem, i ) { + return i === 0; + }, + + last: function( elem, i, match, array ) { + return i === array.length - 1; + }, + + even: function( elem, i ) { + return i % 2 === 0; + }, + + odd: function( elem, i ) { + return i % 2 === 1; + }, + + lt: function( elem, i, match ) { + return i < match[3] - 0; + }, + + gt: function( elem, i, match ) { + return i > match[3] - 0; + }, + + nth: function( elem, i, match ) { + return match[3] - 0 === i; + }, + + eq: function( elem, i, match ) { + return match[3] - 0 === i; + } + }, + filter: { + PSEUDO: function( elem, match, i, array ) { + var name = match[1], + filter = Expr.filters[ name ]; + + if ( filter ) { + return filter( elem, i, match, array ); + + } else if ( name === "contains" ) { + return (elem.textContent || elem.innerText || getText([ elem ]) || "").indexOf(match[3]) >= 0; + + } else if ( name === "not" ) { + var not = match[3]; + + for ( var j = 0, l = not.length; j < l; j++ ) { + if ( not[j] === elem ) { + return false; + } + } + + return true; + + } else { + Sizzle.error( name ); + } + }, + + CHILD: function( elem, match ) { + var first, last, + doneName, parent, cache, + count, diff, + type = match[1], + node = elem; + + switch ( type ) { + case "only": + case "first": + while ( (node = node.previousSibling) ) { + if ( node.nodeType === 1 ) { + return false; + } + } + + if ( type === "first" ) { + return true; + } + + node = elem; + + /* falls through */ + case "last": + while ( (node = node.nextSibling) ) { + if ( node.nodeType === 1 ) { + return false; + } + } + + return true; + + case "nth": + first = match[2]; + last = match[3]; + + if ( first === 1 && last === 0 ) { + return true; + } + + doneName = match[0]; + parent = elem.parentNode; + + if ( parent && (parent[ expando ] !== doneName || !elem.nodeIndex) ) { + count = 0; + + for ( node = parent.firstChild; node; node = node.nextSibling ) { + if ( node.nodeType === 1 ) { + node.nodeIndex = ++count; + } + } + + parent[ expando ] = doneName; + } + + diff = elem.nodeIndex - last; + + if ( first === 0 ) { + return diff === 0; + + } else { + return ( diff % first === 0 && diff / first >= 0 ); + } + } + }, + + ID: function( elem, match ) { + return elem.nodeType === 1 && elem.getAttribute("id") === match; + }, + + TAG: function( elem, match ) { + return (match === "*" && elem.nodeType === 1) || !!elem.nodeName && elem.nodeName.toLowerCase() === match; + }, + + CLASS: function( elem, match ) { + return (" " + ((elem.className && elem.className.baseVal) || elem.getAttribute("class")) + " ") + .indexOf( match ) > -1; + }, + + ATTR: function( elem, match ) { + var name = match[1], + result = Sizzle.attr ? + Sizzle.attr( elem, name ) : + Expr.attrHandle[ name ] ? + Expr.attrHandle[ name ]( elem ) : + elem[ name ] != null ? + elem[ name ] : + elem.getAttribute( name ), + value = result + "", + type = match[2], + check = match[4]; + + return result == null ? + type === "!=" : + !type && Sizzle.attr ? + result != null : + type === "=" ? + value === check : + type === "*=" ? + value.indexOf(check) >= 0 : + type === "~=" ? + (" " + value + " ").indexOf(check) >= 0 : + !check ? + value && result !== false : + type === "!=" ? + value !== check : + type === "^=" ? + value.indexOf(check) === 0 : + type === "$=" ? + value.substr(value.length - check.length) === check : + type === "|=" ? + value === check || value.substr(0, check.length + 1) === check + "-" : + false; + }, + + POS: function( elem, match, i, array ) { + var name = match[2], + filter = Expr.setFilters[ name ]; + + if ( filter ) { + return filter( elem, i, match, array ); + } + } + } +}; + +var origPOS = Expr.match.POS, + fescape = function(all, num){ + return "\\" + (num - 0 + 1); + }; + +for ( var type in Expr.match ) { + Expr.match[ type ] = new RegExp( Expr.match[ type ].source + (/(?![^\[]*\])(?![^\(]*\))/.source) ); + Expr.leftMatch[ type ] = new RegExp( /(^(?:.|\r|\n)*?)/.source + Expr.match[ type ].source.replace(/\\(\d+)/g, fescape) ); +} +// Expose origPOS +// "global" as in regardless of relation to brackets/parens +Expr.match.globalPOS = origPOS; + +var makeArray = function( array, results ) { + array = Array.prototype.slice.call( array, 0 ); + + if ( results ) { + results.push.apply( results, array ); + return results; + } + + return array; +}; + +// Perform a simple check to determine if the browser is capable of +// converting a NodeList to an array using builtin methods. +// Also verifies that the returned array holds DOM nodes +// (which is not the case in the Blackberry browser) +try { + Array.prototype.slice.call( document.documentElement.childNodes, 0 )[0].nodeType; + +// Provide a fallback method if it does not work +} catch( e ) { + makeArray = function( array, results ) { + var i = 0, + ret = results || []; + + if ( toString.call(array) === "[object Array]" ) { + Array.prototype.push.apply( ret, array ); + + } else { + if ( typeof array.length === "number" ) { + for ( var l = array.length; i < l; i++ ) { + ret.push( array[i] ); + } + + } else { + for ( ; array[i]; i++ ) { + ret.push( array[i] ); + } + } + } + + return ret; + }; +} + +var sortOrder, siblingCheck; + +if ( document.documentElement.compareDocumentPosition ) { + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + if ( !a.compareDocumentPosition || !b.compareDocumentPosition ) { + return a.compareDocumentPosition ? -1 : 1; + } + + return a.compareDocumentPosition(b) & 4 ? -1 : 1; + }; + +} else { + sortOrder = function( a, b ) { + // The nodes are identical, we can exit early + if ( a === b ) { + hasDuplicate = true; + return 0; + + // Fallback to using sourceIndex (in IE) if it's available on both nodes + } else if ( a.sourceIndex && b.sourceIndex ) { + return a.sourceIndex - b.sourceIndex; + } + + var al, bl, + ap = [], + bp = [], + aup = a.parentNode, + bup = b.parentNode, + cur = aup; + + // If the nodes are siblings (or identical) we can do a quick check + if ( aup === bup ) { + return siblingCheck( a, b ); + + // If no parents were found then the nodes are disconnected + } else if ( !aup ) { + return -1; + + } else if ( !bup ) { + return 1; + } + + // Otherwise they're somewhere else in the tree so we need + // to build up a full list of the parentNodes for comparison + while ( cur ) { + ap.unshift( cur ); + cur = cur.parentNode; + } + + cur = bup; + + while ( cur ) { + bp.unshift( cur ); + cur = cur.parentNode; + } + + al = ap.length; + bl = bp.length; + + // Start walking down the tree looking for a discrepancy + for ( var i = 0; i < al && i < bl; i++ ) { + if ( ap[i] !== bp[i] ) { + return siblingCheck( ap[i], bp[i] ); + } + } + + // We ended someplace up the tree so do a sibling check + return i === al ? + siblingCheck( a, bp[i], -1 ) : + siblingCheck( ap[i], b, 1 ); + }; + + siblingCheck = function( a, b, ret ) { + if ( a === b ) { + return ret; + } + + var cur = a.nextSibling; + + while ( cur ) { + if ( cur === b ) { + return -1; + } + + cur = cur.nextSibling; + } + + return 1; + }; +} + +// Check to see if the browser returns elements by name when +// querying by getElementById (and provide a workaround) +(function(){ + // We're going to inject a fake input element with a specified name + var form = document.createElement("div"), + id = "script" + (new Date()).getTime(), + root = document.documentElement; + + form.innerHTML = "<a name='" + id + "'/>"; + + // Inject it into the root element, check its status, and remove it quickly + root.insertBefore( form, root.firstChild ); + + // The workaround has to do additional checks after a getElementById + // Which slows things down for other browsers (hence the branching) + if ( document.getElementById( id ) ) { + Expr.find.ID = function( match, context, isXML ) { + if ( typeof context.getElementById !== "undefined" && !isXML ) { + var m = context.getElementById(match[1]); + + return m ? + m.id === match[1] || typeof m.getAttributeNode !== "undefined" && m.getAttributeNode("id").nodeValue === match[1] ? + [m] : + undefined : + []; + } + }; + + Expr.filter.ID = function( elem, match ) { + var node = typeof elem.getAttributeNode !== "undefined" && elem.getAttributeNode("id"); + + return elem.nodeType === 1 && node && node.nodeValue === match; + }; + } + + root.removeChild( form ); + + // release memory in IE + root = form = null; +})(); + +(function(){ + // Check to see if the browser returns only elements + // when doing getElementsByTagName("*") + + // Create a fake element + var div = document.createElement("div"); + div.appendChild( document.createComment("") ); + + // Make sure no comments are found + if ( div.getElementsByTagName("*").length > 0 ) { + Expr.find.TAG = function( match, context ) { + var results = context.getElementsByTagName( match[1] ); + + // Filter out possible comments + if ( match[1] === "*" ) { + var tmp = []; + + for ( var i = 0; results[i]; i++ ) { + if ( results[i].nodeType === 1 ) { + tmp.push( results[i] ); + } + } + + results = tmp; + } + + return results; + }; + } + + // Check to see if an attribute returns normalized href attributes + div.innerHTML = "<a href='#'></a>"; + + if ( div.firstChild && typeof div.firstChild.getAttribute !== "undefined" && + div.firstChild.getAttribute("href") !== "#" ) { + + Expr.attrHandle.href = function( elem ) { + return elem.getAttribute( "href", 2 ); + }; + } + + // release memory in IE + div = null; +})(); + +if ( document.querySelectorAll ) { + (function(){ + var oldSizzle = Sizzle, + div = document.createElement("div"), + id = "__sizzle__"; + + div.innerHTML = "<p class='TEST'></p>"; + + // Safari can't handle uppercase or unicode characters when + // in quirks mode. + if ( div.querySelectorAll && div.querySelectorAll(".TEST").length === 0 ) { + return; + } + + Sizzle = function( query, context, extra, seed ) { + context = context || document; + + // Only use querySelectorAll on non-XML documents + // (ID selectors don't work in non-HTML documents) + if ( !seed && !Sizzle.isXML(context) ) { + // See if we find a selector to speed up + var match = /^(\w+$)|^\.([\w\-]+$)|^#([\w\-]+$)/.exec( query ); + + if ( match && (context.nodeType === 1 || context.nodeType === 9) ) { + // Speed-up: Sizzle("TAG") + if ( match[1] ) { + return makeArray( context.getElementsByTagName( query ), extra ); + + // Speed-up: Sizzle(".CLASS") + } else if ( match[2] && Expr.find.CLASS && context.getElementsByClassName ) { + return makeArray( context.getElementsByClassName( match[2] ), extra ); + } + } + + if ( context.nodeType === 9 ) { + // Speed-up: Sizzle("body") + // The body element only exists once, optimize finding it + if ( query === "body" && context.body ) { + return makeArray( [ context.body ], extra ); + + // Speed-up: Sizzle("#ID") + } else if ( match && match[3] ) { + var elem = context.getElementById( match[3] ); + + // Check parentNode to catch when Blackberry 4.6 returns + // nodes that are no longer in the document #6963 + if ( elem && elem.parentNode ) { + // Handle the case where IE and Opera return items + // by name instead of ID + if ( elem.id === match[3] ) { + return makeArray( [ elem ], extra ); + } + + } else { + return makeArray( [], extra ); + } + } + + try { + return makeArray( context.querySelectorAll(query), extra ); + } catch(qsaError) {} + + // qSA works strangely on Element-rooted queries + // We can work around this by specifying an extra ID on the root + // and working up from there (Thanks to Andrew Dupont for the technique) + // IE 8 doesn't work on object elements + } else if ( context.nodeType === 1 && context.nodeName.toLowerCase() !== "object" ) { + var oldContext = context, + old = context.getAttribute( "id" ), + nid = old || id, + hasParent = context.parentNode, + relativeHierarchySelector = /^\s*[+~]/.test( query ); + + if ( !old ) { + context.setAttribute( "id", nid ); + } else { + nid = nid.replace( /'/g, "\\$&" ); + } + if ( relativeHierarchySelector && hasParent ) { + context = context.parentNode; + } + + try { + if ( !relativeHierarchySelector || hasParent ) { + return makeArray( context.querySelectorAll( "[id='" + nid + "'] " + query ), extra ); + } + + } catch(pseudoError) { + } finally { + if ( !old ) { + oldContext.removeAttribute( "id" ); + } + } + } + } + + return oldSizzle(query, context, extra, seed); + }; + + for ( var prop in oldSizzle ) { + Sizzle[ prop ] = oldSizzle[ prop ]; + } + + // release memory in IE + div = null; + })(); +} + +(function(){ + var html = document.documentElement, + matches = html.matchesSelector || html.mozMatchesSelector || html.webkitMatchesSelector || html.msMatchesSelector; + + if ( matches ) { + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9 fails this) + var disconnectedMatch = !matches.call( document.createElement( "div" ), "div" ), + pseudoWorks = false; + + try { + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( document.documentElement, "[test!='']:sizzle" ); + + } catch( pseudoError ) { + pseudoWorks = true; + } + + Sizzle.matchesSelector = function( node, expr ) { + // Make sure that attribute selectors are quoted + expr = expr.replace(/\=\s*([^'"\]]*)\s*\]/g, "='$1']"); + + if ( !Sizzle.isXML( node ) ) { + try { + if ( pseudoWorks || !Expr.match.PSEUDO.test( expr ) && !/!=/.test( expr ) ) { + var ret = matches.call( node, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || !disconnectedMatch || + // As well, disconnected nodes are said to be in a document + // fragment in IE 9, so check for that + node.document && node.document.nodeType !== 11 ) { + return ret; + } + } + } catch(e) {} + } + + return Sizzle(expr, null, null, [node]).length > 0; + }; + } +})(); + +(function(){ + var div = document.createElement("div"); + + div.innerHTML = "<div class='test e'></div><div class='test'></div>"; + + // Opera can't find a second classname (in 9.6) + // Also, make sure that getElementsByClassName actually exists + if ( !div.getElementsByClassName || div.getElementsByClassName("e").length === 0 ) { + return; + } + + // Safari caches class attributes, doesn't catch changes (in 3.2) + div.lastChild.className.baseVal = "e"; + + if ( div.getElementsByClassName("e").length === 1 ) { + return; + } + + Expr.order.splice(1, 0, "CLASS"); + Expr.find.CLASS = function( match, context, isXML ) { + if ( typeof context.getElementsByClassName !== "undefined" && !isXML ) { + return context.getElementsByClassName(match[1]); + } + }; + + // release memory in IE + div = null; +})(); + +function dirNodeCheck( dir, cur, doneName, checkSet, nodeCheck, isXML ) { + for ( var i = 0, l = checkSet.length; i < l; i++ ) { + var elem = checkSet[i]; + + if ( elem ) { + var match = false; + + elem = elem[dir]; + + while ( elem ) { + if ( elem[ expando ] === doneName ) { + match = checkSet[elem.sizset]; + break; + } + + if ( elem.nodeType === 1 && !isXML ){ + elem[ expando ] = doneName; + elem.sizset = i; + } + + if ( elem.nodeName.toLowerCase() === cur ) { + match = elem; + break; + } + + elem = elem[dir]; + } + + checkSet[i] = match; + } + } +} + +function dirCheck( dir, cur, doneName, checkSet, nodeCheck, isXML ) { + for ( var i = 0, l = checkSet.length; i < l; i++ ) { + var elem = checkSet[i]; + + if ( elem ) { + var match = false; + + elem = elem[dir]; + + while ( elem ) { + if ( elem[ expando ] === doneName ) { + match = checkSet[elem.sizset]; + break; + } + + if ( elem.nodeType === 1 ) { + if ( !isXML ) { + elem[ expando ] = doneName; + elem.sizset = i; + } + + if ( typeof cur !== "string" ) { + if ( elem === cur ) { + match = true; + break; + } + + } else if ( Sizzle.filter( cur, [elem] ).length > 0 ) { + match = elem; + break; + } + } + + elem = elem[dir]; + } + + checkSet[i] = match; + } + } +} + +if ( document.documentElement.contains ) { + Sizzle.contains = function( a, b ) { + return a !== b && (a.contains ? a.contains(b) : true); + }; + +} else if ( document.documentElement.compareDocumentPosition ) { + Sizzle.contains = function( a, b ) { + return !!(a.compareDocumentPosition(b) & 16); + }; + +} else { + Sizzle.contains = function() { + return false; + }; +} + +Sizzle.isXML = function( elem ) { + // documentElement is verified for cases where it doesn't yet exist + // (such as loading iframes in IE - #4833) + var documentElement = (elem ? elem.ownerDocument || elem : 0).documentElement; + + return documentElement ? documentElement.nodeName !== "HTML" : false; +}; + +var posProcess = function( selector, context, seed ) { + var match, + tmpSet = [], + later = "", + root = context.nodeType ? [context] : context; + + // Position selectors must be done after the filter + // And so must :not(positional) so we move all PSEUDOs to the end + while ( (match = Expr.match.PSEUDO.exec( selector )) ) { + later += match[0]; + selector = selector.replace( Expr.match.PSEUDO, "" ); + } + + selector = Expr.relative[selector] ? selector + "*" : selector; + + for ( var i = 0, l = root.length; i < l; i++ ) { + Sizzle( selector, root[i], tmpSet, seed ); + } + + return Sizzle.filter( later, tmpSet ); +}; + +// EXPOSE +// Override sizzle attribute retrieval +Sizzle.attr = jQuery.attr; +Sizzle.selectors.attrMap = {}; +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; +jQuery.expr[":"] = jQuery.expr.filters; +jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; + + +})(); + + +var runtil = /Until$/, + rparentsprev = /^(?:parents|prevUntil|prevAll)/, + // Note: This RegExp should be improved, or likely pulled from Sizzle + rmultiselector = /,/, + isSimple = /^.[^:#\[\.,]*$/, + slice = Array.prototype.slice, + POS = jQuery.expr.match.globalPOS, + // methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend({ + find: function( selector ) { + var self = this, + i, l; + + if ( typeof selector !== "string" ) { + return jQuery( selector ).filter(function() { + for ( i = 0, l = self.length; i < l; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + }); + } + + var ret = this.pushStack( "", "find", selector ), + length, n, r; + + for ( i = 0, l = this.length; i < l; i++ ) { + length = ret.length; + jQuery.find( selector, this[i], ret ); + + if ( i > 0 ) { + // Make sure that the results are unique + for ( n = length; n < ret.length; n++ ) { + for ( r = 0; r < length; r++ ) { + if ( ret[r] === ret[n] ) { + ret.splice(n--, 1); + break; + } + } + } + } + } + + return ret; + }, + + has: function( target ) { + var targets = jQuery( target ); + return this.filter(function() { + for ( var i = 0, l = targets.length; i < l; i++ ) { + if ( jQuery.contains( this, targets[i] ) ) { + return true; + } + } + }); + }, + + not: function( selector ) { + return this.pushStack( winnow(this, selector, false), "not", selector); + }, + + filter: function( selector ) { + return this.pushStack( winnow(this, selector, true), "filter", selector ); + }, + + is: function( selector ) { + return !!selector && ( + typeof selector === "string" ? + // If this is a positional selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + POS.test( selector ) ? + jQuery( selector, this.context ).index( this[0] ) >= 0 : + jQuery.filter( selector, this ).length > 0 : + this.filter( selector ).length > 0 ); + }, + + closest: function( selectors, context ) { + var ret = [], i, l, cur = this[0]; + + // Array (deprecated as of jQuery 1.7) + if ( jQuery.isArray( selectors ) ) { + var level = 1; + + while ( cur && cur.ownerDocument && cur !== context ) { + for ( i = 0; i < selectors.length; i++ ) { + + if ( jQuery( cur ).is( selectors[ i ] ) ) { + ret.push({ selector: selectors[ i ], elem: cur, level: level }); + } + } + + cur = cur.parentNode; + level++; + } + + return ret; + } + + // String + var pos = POS.test( selectors ) || typeof selectors !== "string" ? + jQuery( selectors, context || this.context ) : + 0; + + for ( i = 0, l = this.length; i < l; i++ ) { + cur = this[i]; + + while ( cur ) { + if ( pos ? pos.index(cur) > -1 : jQuery.find.matchesSelector(cur, selectors) ) { + ret.push( cur ); + break; + + } else { + cur = cur.parentNode; + if ( !cur || !cur.ownerDocument || cur === context || cur.nodeType === 11 ) { + break; + } + } + } + } + + ret = ret.length > 1 ? jQuery.unique( ret ) : ret; + + return this.pushStack( ret, "closest", selectors ); + }, + + // Determine the position of an element within + // the matched set of elements + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[0] && this[0].parentNode ) ? this.prevAll().length : -1; + } + + // index in selector + if ( typeof elem === "string" ) { + return jQuery.inArray( this[0], jQuery( elem ) ); + } + + // Locate the position of the desired element + return jQuery.inArray( + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[0] : elem, this ); + }, + + add: function( selector, context ) { + var set = typeof selector === "string" ? + jQuery( selector, context ) : + jQuery.makeArray( selector && selector.nodeType ? [ selector ] : selector ), + all = jQuery.merge( this.get(), set ); + + return this.pushStack( isDisconnected( set[0] ) || isDisconnected( all[0] ) ? + all : + jQuery.unique( all ) ); + }, + + andSelf: function() { + return this.add( this.prevObject ); + } +}); + +// A painfully simple check to see if an element is disconnected +// from a document (should be improved, where feasible). +function isDisconnected( node ) { + return !node || !node.parentNode || node.parentNode.nodeType === 11; +} + +jQuery.each({ + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return jQuery.dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, i, until ) { + return jQuery.dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return jQuery.nth( elem, 2, "nextSibling" ); + }, + prev: function( elem ) { + return jQuery.nth( elem, 2, "previousSibling" ); + }, + nextAll: function( elem ) { + return jQuery.dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return jQuery.dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, i, until ) { + return jQuery.dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, i, until ) { + return jQuery.dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return jQuery.sibling( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return jQuery.sibling( elem.firstChild ); + }, + contents: function( elem ) { + return jQuery.nodeName( elem, "iframe" ) ? + elem.contentDocument || elem.contentWindow.document : + jQuery.makeArray( elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var ret = jQuery.map( this, fn, until ); + + if ( !runtil.test( name ) ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + ret = jQuery.filter( selector, ret ); + } + + ret = this.length > 1 && !guaranteedUnique[ name ] ? jQuery.unique( ret ) : ret; + + if ( (this.length > 1 || rmultiselector.test( selector )) && rparentsprev.test( name ) ) { + ret = ret.reverse(); + } + + return this.pushStack( ret, name, slice.call( arguments ).join(",") ); + }; +}); + +jQuery.extend({ + filter: function( expr, elems, not ) { + if ( not ) { + expr = ":not(" + expr + ")"; + } + + return elems.length === 1 ? + jQuery.find.matchesSelector(elems[0], expr) ? [ elems[0] ] : [] : + jQuery.find.matches(expr, elems); + }, + + dir: function( elem, dir, until ) { + var matched = [], + cur = elem[ dir ]; + + while ( cur && cur.nodeType !== 9 && (until === undefined || cur.nodeType !== 1 || !jQuery( cur ).is( until )) ) { + if ( cur.nodeType === 1 ) { + matched.push( cur ); + } + cur = cur[dir]; + } + return matched; + }, + + nth: function( cur, result, dir, elem ) { + result = result || 1; + var num = 0; + + for ( ; cur; cur = cur[dir] ) { + if ( cur.nodeType === 1 && ++num === result ) { + break; + } + } + + return cur; + }, + + sibling: function( n, elem ) { + var r = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + r.push( n ); + } + } + + return r; + } +}); + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, keep ) { + + // Can't pass null or undefined to indexOf in Firefox 4 + // Set to 0 to skip string check + qualifier = qualifier || 0; + + if ( jQuery.isFunction( qualifier ) ) { + return jQuery.grep(elements, function( elem, i ) { + var retVal = !!qualifier.call( elem, i, elem ); + return retVal === keep; + }); + + } else if ( qualifier.nodeType ) { + return jQuery.grep(elements, function( elem, i ) { + return ( elem === qualifier ) === keep; + }); + + } else if ( typeof qualifier === "string" ) { + var filtered = jQuery.grep(elements, function( elem ) { + return elem.nodeType === 1; + }); + + if ( isSimple.test( qualifier ) ) { + return jQuery.filter(qualifier, filtered, !keep); + } else { + qualifier = jQuery.filter( qualifier, filtered ); + } + } + + return jQuery.grep(elements, function( elem, i ) { + return ( jQuery.inArray( elem, qualifier ) >= 0 ) === keep; + }); +} + + + + +function createSafeFragment( document ) { + var list = nodeNames.split( "|" ), + safeFrag = document.createDocumentFragment(); + + if ( safeFrag.createElement ) { + while ( list.length ) { + safeFrag.createElement( + list.pop() + ); + } + } + return safeFrag; +} + +var nodeNames = "abbr|article|aside|audio|bdi|canvas|data|datalist|details|figcaption|figure|footer|" + + "header|hgroup|mark|meter|nav|output|progress|section|summary|time|video", + rinlinejQuery = / jQuery\d+="(?:\d+|null)"/g, + rleadingWhitespace = /^\s+/, + rxhtmlTag = /<(?!area|br|col|embed|hr|img|input|link|meta|param)(([\w:]+)[^>]*)\/>/ig, + rtagName = /<([\w:]+)/, + rtbody = /<tbody/i, + rhtml = /<|&#?\w+;/, + rnoInnerhtml = /<(?:script|style)/i, + rnocache = /<(?:script|object|embed|option|style)/i, + rnoshimcache = new RegExp("<(?:" + nodeNames + ")[\\s/>]", "i"), + // checked="checked" or checked + rchecked = /checked\s*(?:[^=]|=\s*.checked.)/i, + rscriptType = /\/(java|ecma)script/i, + rcleanScript = /^\s*<!(?:\[CDATA\[|\-\-)/, + wrapMap = { + option: [ 1, "<select multiple='multiple'>", "</select>" ], + legend: [ 1, "<fieldset>", "</fieldset>" ], + thead: [ 1, "<table>", "</table>" ], + tr: [ 2, "<table><tbody>", "</tbody></table>" ], + td: [ 3, "<table><tbody><tr>", "</tr></tbody></table>" ], + col: [ 2, "<table><tbody></tbody><colgroup>", "</colgroup></table>" ], + area: [ 1, "<map>", "</map>" ], + _default: [ 0, "", "" ] + }, + safeFragment = createSafeFragment( document ); + +wrapMap.optgroup = wrapMap.option; +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// IE can't serialize <link> and <script> tags normally +if ( !jQuery.support.htmlSerialize ) { + wrapMap._default = [ 1, "div<div>", "</div>" ]; +} + +jQuery.fn.extend({ + text: function( value ) { + return jQuery.access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().append( ( this[0] && this[0].ownerDocument || document ).createTextNode( value ) ); + }, null, value, arguments.length ); + }, + + wrapAll: function( html ) { + if ( jQuery.isFunction( html ) ) { + return this.each(function(i) { + jQuery(this).wrapAll( html.call(this, i) ); + }); + } + + if ( this[0] ) { + // The elements to wrap the target around + var wrap = jQuery( html, this[0].ownerDocument ).eq(0).clone(true); + + if ( this[0].parentNode ) { + wrap.insertBefore( this[0] ); + } + + wrap.map(function() { + var elem = this; + + while ( elem.firstChild && elem.firstChild.nodeType === 1 ) { + elem = elem.firstChild; + } + + return elem; + }).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( jQuery.isFunction( html ) ) { + return this.each(function(i) { + jQuery(this).wrapInner( html.call(this, i) ); + }); + } + + return this.each(function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + }); + }, + + wrap: function( html ) { + var isFunction = jQuery.isFunction( html ); + + return this.each(function(i) { + jQuery( this ).wrapAll( isFunction ? html.call(this, i) : html ); + }); + }, + + unwrap: function() { + return this.parent().each(function() { + if ( !jQuery.nodeName( this, "body" ) ) { + jQuery( this ).replaceWith( this.childNodes ); + } + }).end(); + }, + + append: function() { + return this.domManip(arguments, true, function( elem ) { + if ( this.nodeType === 1 ) { + this.appendChild( elem ); + } + }); + }, + + prepend: function() { + return this.domManip(arguments, true, function( elem ) { + if ( this.nodeType === 1 ) { + this.insertBefore( elem, this.firstChild ); + } + }); + }, + + before: function() { + if ( this[0] && this[0].parentNode ) { + return this.domManip(arguments, false, function( elem ) { + this.parentNode.insertBefore( elem, this ); + }); + } else if ( arguments.length ) { + var set = jQuery.clean( arguments ); + set.push.apply( set, this.toArray() ); + return this.pushStack( set, "before", arguments ); + } + }, + + after: function() { + if ( this[0] && this[0].parentNode ) { + return this.domManip(arguments, false, function( elem ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + }); + } else if ( arguments.length ) { + var set = this.pushStack( this, "after", arguments ); + set.push.apply( set, jQuery.clean(arguments) ); + return set; + } + }, + + // keepData is for internal use only--do not document + remove: function( selector, keepData ) { + for ( var i = 0, elem; (elem = this[i]) != null; i++ ) { + if ( !selector || jQuery.filter( selector, [ elem ] ).length ) { + if ( !keepData && elem.nodeType === 1 ) { + jQuery.cleanData( elem.getElementsByTagName("*") ); + jQuery.cleanData( [ elem ] ); + } + + if ( elem.parentNode ) { + elem.parentNode.removeChild( elem ); + } + } + } + + return this; + }, + + empty: function() { + for ( var i = 0, elem; (elem = this[i]) != null; i++ ) { + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( elem.getElementsByTagName("*") ); + } + + // Remove any remaining nodes + while ( elem.firstChild ) { + elem.removeChild( elem.firstChild ); + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function () { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + }); + }, + + html: function( value ) { + return jQuery.access( this, function( value ) { + var elem = this[0] || {}, + i = 0, + l = this.length; + + if ( value === undefined ) { + return elem.nodeType === 1 ? + elem.innerHTML.replace( rinlinejQuery, "" ) : + null; + } + + + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + ( jQuery.support.leadingWhitespace || !rleadingWhitespace.test( value ) ) && + !wrapMap[ ( rtagName.exec( value ) || ["", ""] )[1].toLowerCase() ] ) { + + value = value.replace( rxhtmlTag, "<$1></$2>" ); + + try { + for (; i < l; i++ ) { + // Remove element nodes and prevent memory leaks + elem = this[i] || {}; + if ( elem.nodeType === 1 ) { + jQuery.cleanData( elem.getElementsByTagName( "*" ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch(e) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function( value ) { + if ( this[0] && this[0].parentNode ) { + // Make sure that the elements are removed from the DOM before they are inserted + // this can help fix replacing a parent with child elements + if ( jQuery.isFunction( value ) ) { + return this.each(function(i) { + var self = jQuery(this), old = self.html(); + self.replaceWith( value.call( this, i, old ) ); + }); + } + + if ( typeof value !== "string" ) { + value = jQuery( value ).detach(); + } + + return this.each(function() { + var next = this.nextSibling, + parent = this.parentNode; + + jQuery( this ).remove(); + + if ( next ) { + jQuery(next).before( value ); + } else { + jQuery(parent).append( value ); + } + }); + } else { + return this.length ? + this.pushStack( jQuery(jQuery.isFunction(value) ? value() : value), "replaceWith", value ) : + this; + } + }, + + detach: function( selector ) { + return this.remove( selector, true ); + }, + + domManip: function( args, table, callback ) { + var results, first, fragment, parent, + value = args[0], + scripts = []; + + // We can't cloneNode fragments that contain checked, in WebKit + if ( !jQuery.support.checkClone && arguments.length === 3 && typeof value === "string" && rchecked.test( value ) ) { + return this.each(function() { + jQuery(this).domManip( args, table, callback, true ); + }); + } + + if ( jQuery.isFunction(value) ) { + return this.each(function(i) { + var self = jQuery(this); + args[0] = value.call(this, i, table ? self.html() : undefined); + self.domManip( args, table, callback ); + }); + } + + if ( this[0] ) { + parent = value && value.parentNode; + + // If we're in a fragment, just use that instead of building a new one + if ( jQuery.support.parentNode && parent && parent.nodeType === 11 && parent.childNodes.length === this.length ) { + results = { fragment: parent }; + + } else { + results = jQuery.buildFragment( args, this, scripts ); + } + + fragment = results.fragment; + + if ( fragment.childNodes.length === 1 ) { + first = fragment = fragment.firstChild; + } else { + first = fragment.firstChild; + } + + if ( first ) { + table = table && jQuery.nodeName( first, "tr" ); + + for ( var i = 0, l = this.length, lastIndex = l - 1; i < l; i++ ) { + callback.call( + table ? + root(this[i], first) : + this[i], + // Make sure that we do not leak memory by inadvertently discarding + // the original fragment (which might have attached data) instead of + // using it; in addition, use the original fragment object for the last + // item instead of first because it can end up being emptied incorrectly + // in certain situations (Bug #8070). + // Fragments from the fragment cache must always be cloned and never used + // in place. + results.cacheable || ( l > 1 && i < lastIndex ) ? + jQuery.clone( fragment, true, true ) : + fragment + ); + } + } + + if ( scripts.length ) { + jQuery.each( scripts, function( i, elem ) { + if ( elem.src ) { + jQuery.ajax({ + type: "GET", + global: false, + url: elem.src, + async: false, + dataType: "script" + }); + } else { + jQuery.globalEval( ( elem.text || elem.textContent || elem.innerHTML || "" ).replace( rcleanScript, "/*$0*/" ) ); + } + + if ( elem.parentNode ) { + elem.parentNode.removeChild( elem ); + } + }); + } + } + + return this; + } +}); + +function root( elem, cur ) { + return jQuery.nodeName(elem, "table") ? + (elem.getElementsByTagName("tbody")[0] || + elem.appendChild(elem.ownerDocument.createElement("tbody"))) : + elem; +} + +function cloneCopyEvent( src, dest ) { + + if ( dest.nodeType !== 1 || !jQuery.hasData( src ) ) { + return; + } + + var type, i, l, + oldData = jQuery._data( src ), + curData = jQuery._data( dest, oldData ), + events = oldData.events; + + if ( events ) { + delete curData.handle; + curData.events = {}; + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + + // make the cloned public data object a copy from the original + if ( curData.data ) { + curData.data = jQuery.extend( {}, curData.data ); + } +} + +function cloneFixAttributes( src, dest ) { + var nodeName; + + // We do not need to do anything for non-Elements + if ( dest.nodeType !== 1 ) { + return; + } + + // clearAttributes removes the attributes, which we don't want, + // but also removes the attachEvent events, which we *do* want + if ( dest.clearAttributes ) { + dest.clearAttributes(); + } + + // mergeAttributes, in contrast, only merges back on the + // original attributes, not the events + if ( dest.mergeAttributes ) { + dest.mergeAttributes( src ); + } + + nodeName = dest.nodeName.toLowerCase(); + + // IE6-8 fail to clone children inside object elements that use + // the proprietary classid attribute value (rather than the type + // attribute) to identify the type of content to display + if ( nodeName === "object" ) { + dest.outerHTML = src.outerHTML; + + } else if ( nodeName === "input" && (src.type === "checkbox" || src.type === "radio") ) { + // IE6-8 fails to persist the checked state of a cloned checkbox + // or radio button. Worse, IE6-7 fail to give the cloned element + // a checked appearance if the defaultChecked value isn't also set + if ( src.checked ) { + dest.defaultChecked = dest.checked = src.checked; + } + + // IE6-7 get confused and end up setting the value of a cloned + // checkbox/radio button to an empty string instead of "on" + if ( dest.value !== src.value ) { + dest.value = src.value; + } + + // IE6-8 fails to return the selected option to the default selected + // state when cloning options + } else if ( nodeName === "option" ) { + dest.selected = src.defaultSelected; + + // IE6-8 fails to set the defaultValue to the correct value when + // cloning other types of input fields + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + + // IE blanks contents when cloning scripts + } else if ( nodeName === "script" && dest.text !== src.text ) { + dest.text = src.text; + } + + // Event data gets referenced instead of copied if the expando + // gets copied too + dest.removeAttribute( jQuery.expando ); + + // Clear flags for bubbling special change/submit events, they must + // be reattached when the newly cloned events are first activated + dest.removeAttribute( "_submit_attached" ); + dest.removeAttribute( "_change_attached" ); +} + +jQuery.buildFragment = function( args, nodes, scripts ) { + var fragment, cacheable, cacheresults, doc, + first = args[ 0 ]; + + // nodes may contain either an explicit document object, + // a jQuery collection or context object. + // If nodes[0] contains a valid object to assign to doc + if ( nodes && nodes[0] ) { + doc = nodes[0].ownerDocument || nodes[0]; + } + + // Ensure that an attr object doesn't incorrectly stand in as a document object + // Chrome and Firefox seem to allow this to occur and will throw exception + // Fixes #8950 + if ( !doc.createDocumentFragment ) { + doc = document; + } + + // Only cache "small" (1/2 KB) HTML strings that are associated with the main document + // Cloning options loses the selected state, so don't cache them + // IE 6 doesn't like it when you put <object> or <embed> elements in a fragment + // Also, WebKit does not clone 'checked' attributes on cloneNode, so don't cache + // Lastly, IE6,7,8 will not correctly reuse cached fragments that were created from unknown elems #10501 + if ( args.length === 1 && typeof first === "string" && first.length < 512 && doc === document && + first.charAt(0) === "<" && !rnocache.test( first ) && + (jQuery.support.checkClone || !rchecked.test( first )) && + (jQuery.support.html5Clone || !rnoshimcache.test( first )) ) { + + cacheable = true; + + cacheresults = jQuery.fragments[ first ]; + if ( cacheresults && cacheresults !== 1 ) { + fragment = cacheresults; + } + } + + if ( !fragment ) { + fragment = doc.createDocumentFragment(); + jQuery.clean( args, doc, fragment, scripts ); + } + + if ( cacheable ) { + jQuery.fragments[ first ] = cacheresults ? fragment : 1; + } + + return { fragment: fragment, cacheable: cacheable }; +}; + +jQuery.fragments = {}; + +jQuery.each({ + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var ret = [], + insert = jQuery( selector ), + parent = this.length === 1 && this[0].parentNode; + + if ( parent && parent.nodeType === 11 && parent.childNodes.length === 1 && insert.length === 1 ) { + insert[ original ]( this[0] ); + return this; + + } else { + for ( var i = 0, l = insert.length; i < l; i++ ) { + var elems = ( i > 0 ? this.clone(true) : this ).get(); + jQuery( insert[i] )[ original ]( elems ); + ret = ret.concat( elems ); + } + + return this.pushStack( ret, name, insert.selector ); + } + }; +}); + +function getAll( elem ) { + if ( typeof elem.getElementsByTagName !== "undefined" ) { + return elem.getElementsByTagName( "*" ); + + } else if ( typeof elem.querySelectorAll !== "undefined" ) { + return elem.querySelectorAll( "*" ); + + } else { + return []; + } +} + +// Used in clean, fixes the defaultChecked property +function fixDefaultChecked( elem ) { + if ( elem.type === "checkbox" || elem.type === "radio" ) { + elem.defaultChecked = elem.checked; + } +} +// Finds all inputs and passes them to fixDefaultChecked +function findInputs( elem ) { + var nodeName = ( elem.nodeName || "" ).toLowerCase(); + if ( nodeName === "input" ) { + fixDefaultChecked( elem ); + // Skip scripts, get other children + } else if ( nodeName !== "script" && typeof elem.getElementsByTagName !== "undefined" ) { + jQuery.grep( elem.getElementsByTagName("input"), fixDefaultChecked ); + } +} + +// Derived From: http://www.iecss.com/shimprove/javascript/shimprove.1-0-1.js +function shimCloneNode( elem ) { + var div = document.createElement( "div" ); + safeFragment.appendChild( div ); + + div.innerHTML = elem.outerHTML; + return div.firstChild; +} + +jQuery.extend({ + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var srcElements, + destElements, + i, + // IE<=8 does not properly clone detached, unknown element nodes + clone = jQuery.support.html5Clone || jQuery.isXMLDoc(elem) || !rnoshimcache.test( "<" + elem.nodeName + ">" ) ? + elem.cloneNode( true ) : + shimCloneNode( elem ); + + if ( (!jQuery.support.noCloneEvent || !jQuery.support.noCloneChecked) && + (elem.nodeType === 1 || elem.nodeType === 11) && !jQuery.isXMLDoc(elem) ) { + // IE copies events bound via attachEvent when using cloneNode. + // Calling detachEvent on the clone will also remove the events + // from the original. In order to get around this, we use some + // proprietary methods to clear the events. Thanks to MooTools + // guys for this hotness. + + cloneFixAttributes( elem, clone ); + + // Using Sizzle here is crazy slow, so we use getElementsByTagName instead + srcElements = getAll( elem ); + destElements = getAll( clone ); + + // Weird iteration because IE will replace the length property + // with an element if you are cloning the body and one of the + // elements on the page has a name or id of "length" + for ( i = 0; srcElements[i]; ++i ) { + // Ensure that the destination node is not null; Fixes #9587 + if ( destElements[i] ) { + cloneFixAttributes( srcElements[i], destElements[i] ); + } + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + cloneCopyEvent( elem, clone ); + + if ( deepDataAndEvents ) { + srcElements = getAll( elem ); + destElements = getAll( clone ); + + for ( i = 0; srcElements[i]; ++i ) { + cloneCopyEvent( srcElements[i], destElements[i] ); + } + } + } + + srcElements = destElements = null; + + // Return the cloned set + return clone; + }, + + clean: function( elems, context, fragment, scripts ) { + var checkScriptType, script, j, + ret = []; + + context = context || document; + + // !context.createElement fails in IE with an error but returns typeof 'object' + if ( typeof context.createElement === "undefined" ) { + context = context.ownerDocument || context[0] && context[0].ownerDocument || document; + } + + for ( var i = 0, elem; (elem = elems[i]) != null; i++ ) { + if ( typeof elem === "number" ) { + elem += ""; + } + + if ( !elem ) { + continue; + } + + // Convert html string into DOM nodes + if ( typeof elem === "string" ) { + if ( !rhtml.test( elem ) ) { + elem = context.createTextNode( elem ); + } else { + // Fix "XHTML"-style tags in all browsers + elem = elem.replace(rxhtmlTag, "<$1></$2>"); + + // Trim whitespace, otherwise indexOf won't work as expected + var tag = ( rtagName.exec( elem ) || ["", ""] )[1].toLowerCase(), + wrap = wrapMap[ tag ] || wrapMap._default, + depth = wrap[0], + div = context.createElement("div"), + safeChildNodes = safeFragment.childNodes, + remove; + + // Append wrapper element to unknown element safe doc fragment + if ( context === document ) { + // Use the fragment we've already created for this document + safeFragment.appendChild( div ); + } else { + // Use a fragment created with the owner document + createSafeFragment( context ).appendChild( div ); + } + + // Go to html and back, then peel off extra wrappers + div.innerHTML = wrap[1] + elem + wrap[2]; + + // Move to the right depth + while ( depth-- ) { + div = div.lastChild; + } + + // Remove IE's autoinserted <tbody> from table fragments + if ( !jQuery.support.tbody ) { + + // String was a <table>, *may* have spurious <tbody> + var hasBody = rtbody.test(elem), + tbody = tag === "table" && !hasBody ? + div.firstChild && div.firstChild.childNodes : + + // String was a bare <thead> or <tfoot> + wrap[1] === "<table>" && !hasBody ? + div.childNodes : + []; + + for ( j = tbody.length - 1; j >= 0 ; --j ) { + if ( jQuery.nodeName( tbody[ j ], "tbody" ) && !tbody[ j ].childNodes.length ) { + tbody[ j ].parentNode.removeChild( tbody[ j ] ); + } + } + } + + // IE completely kills leading whitespace when innerHTML is used + if ( !jQuery.support.leadingWhitespace && rleadingWhitespace.test( elem ) ) { + div.insertBefore( context.createTextNode( rleadingWhitespace.exec(elem)[0] ), div.firstChild ); + } + + elem = div.childNodes; + + // Clear elements from DocumentFragment (safeFragment or otherwise) + // to avoid hoarding elements. Fixes #11356 + if ( div ) { + div.parentNode.removeChild( div ); + + // Guard against -1 index exceptions in FF3.6 + if ( safeChildNodes.length > 0 ) { + remove = safeChildNodes[ safeChildNodes.length - 1 ]; + + if ( remove && remove.parentNode ) { + remove.parentNode.removeChild( remove ); + } + } + } + } + } + + // Resets defaultChecked for any radios and checkboxes + // about to be appended to the DOM in IE 6/7 (#8060) + var len; + if ( !jQuery.support.appendChecked ) { + if ( elem[0] && typeof (len = elem.length) === "number" ) { + for ( j = 0; j < len; j++ ) { + findInputs( elem[j] ); + } + } else { + findInputs( elem ); + } + } + + if ( elem.nodeType ) { + ret.push( elem ); + } else { + ret = jQuery.merge( ret, elem ); + } + } + + if ( fragment ) { + checkScriptType = function( elem ) { + return !elem.type || rscriptType.test( elem.type ); + }; + for ( i = 0; ret[i]; i++ ) { + script = ret[i]; + if ( scripts && jQuery.nodeName( script, "script" ) && (!script.type || rscriptType.test( script.type )) ) { + scripts.push( script.parentNode ? script.parentNode.removeChild( script ) : script ); + + } else { + if ( script.nodeType === 1 ) { + var jsTags = jQuery.grep( script.getElementsByTagName( "script" ), checkScriptType ); + + ret.splice.apply( ret, [i + 1, 0].concat( jsTags ) ); + } + fragment.appendChild( script ); + } + } + } + + return ret; + }, + + cleanData: function( elems ) { + var data, id, + cache = jQuery.cache, + special = jQuery.event.special, + deleteExpando = jQuery.support.deleteExpando; + + for ( var i = 0, elem; (elem = elems[i]) != null; i++ ) { + if ( elem.nodeName && jQuery.noData[elem.nodeName.toLowerCase()] ) { + continue; + } + + id = elem[ jQuery.expando ]; + + if ( id ) { + data = cache[ id ]; + + if ( data && data.events ) { + for ( var type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + + // Null the DOM reference to avoid IE6/7/8 leak (#7054) + if ( data.handle ) { + data.handle.elem = null; + } + } + + if ( deleteExpando ) { + delete elem[ jQuery.expando ]; + + } else if ( elem.removeAttribute ) { + elem.removeAttribute( jQuery.expando ); + } + + delete cache[ id ]; + } + } + } +}); + + + + +var ralpha = /alpha\([^)]*\)/i, + ropacity = /opacity=([^)]*)/, + // fixed for IE9, see #8346 + rupper = /([A-Z]|^ms)/g, + rnum = /^[\-+]?(?:\d*\.)?\d+$/i, + rnumnonpx = /^-?(?:\d*\.)?\d+(?!px)[^\d\s]+$/i, + rrelNum = /^([\-+])=([\-+.\de]+)/, + rmargin = /^margin/, + + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + + // order is important! + cssExpand = [ "Top", "Right", "Bottom", "Left" ], + + curCSS, + + getComputedStyle, + currentStyle; + +jQuery.fn.css = function( name, value ) { + return jQuery.access( this, function( elem, name, value ) { + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); +}; + +jQuery.extend({ + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + + } else { + return elem.style.opacity; + } + } + } + }, + + // Exclude the following css properties to add px + cssNumber: { + "fillOpacity": true, + "fontWeight": true, + "lineHeight": true, + "opacity": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: { + // normalize float css property + "float": jQuery.support.cssFloat ? "cssFloat" : "styleFloat" + }, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, origName = jQuery.camelCase( name ), + style = elem.style, hooks = jQuery.cssHooks[ origName ]; + + name = jQuery.cssProps[ origName ] || origName; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // convert relative number strings (+= or -=) to relative numbers. #7345 + if ( type === "string" && (ret = rrelNum.exec( value )) ) { + value = ( +( ret[1] + 1) * +ret[2] ) + parseFloat( jQuery.css( elem, name ) ); + // Fixes bug #9237 + type = "number"; + } + + // Make sure that NaN and null values aren't set. See: #7116 + if ( value == null || type === "number" && isNaN( value ) ) { + return; + } + + // If a number was passed in, add 'px' to the (except for certain CSS properties) + if ( type === "number" && !jQuery.cssNumber[ origName ] ) { + value += "px"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !("set" in hooks) || (value = hooks.set( elem, value )) !== undefined ) { + // Wrapped to prevent IE from throwing errors when 'invalid' values are provided + // Fixes bug #5509 + try { + style[ name ] = value; + } catch(e) {} + } + + } else { + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && (ret = hooks.get( elem, false, extra )) !== undefined ) { + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra ) { + var ret, hooks; + + // Make sure that we're working with the right name + name = jQuery.camelCase( name ); + hooks = jQuery.cssHooks[ name ]; + name = jQuery.cssProps[ name ] || name; + + // cssFloat needs a special treatment + if ( name === "cssFloat" ) { + name = "float"; + } + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks && (ret = hooks.get( elem, true, extra )) !== undefined ) { + return ret; + + // Otherwise, if a way to get the computed value exists, use that + } else if ( curCSS ) { + return curCSS( elem, name ); + } + }, + + // A method for quickly swapping in/out CSS properties to get correct calculations + swap: function( elem, options, callback ) { + var old = {}, + ret, name; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; + } +}); + +// DEPRECATED in 1.3, Use jQuery.css() instead +jQuery.curCSS = jQuery.css; + +if ( document.defaultView && document.defaultView.getComputedStyle ) { + getComputedStyle = function( elem, name ) { + var ret, defaultView, computedStyle, width, + style = elem.style; + + name = name.replace( rupper, "-$1" ).toLowerCase(); + + if ( (defaultView = elem.ownerDocument.defaultView) && + (computedStyle = defaultView.getComputedStyle( elem, null )) ) { + + ret = computedStyle.getPropertyValue( name ); + if ( ret === "" && !jQuery.contains( elem.ownerDocument.documentElement, elem ) ) { + ret = jQuery.style( elem, name ); + } + } + + // A tribute to the "awesome hack by Dean Edwards" + // WebKit uses "computed value (percentage if specified)" instead of "used value" for margins + // which is against the CSSOM draft spec: http://dev.w3.org/csswg/cssom/#resolved-values + if ( !jQuery.support.pixelMargin && computedStyle && rmargin.test( name ) && rnumnonpx.test( ret ) ) { + width = style.width; + style.width = ret; + ret = computedStyle.width; + style.width = width; + } + + return ret; + }; +} + +if ( document.documentElement.currentStyle ) { + currentStyle = function( elem, name ) { + var left, rsLeft, uncomputed, + ret = elem.currentStyle && elem.currentStyle[ name ], + style = elem.style; + + // Avoid setting ret to empty string here + // so we don't default to auto + if ( ret == null && style && (uncomputed = style[ name ]) ) { + ret = uncomputed; + } + + // From the awesome hack by Dean Edwards + // http://erik.eae.net/archives/2007/07/27/18.54.15/#comment-102291 + + // If we're not dealing with a regular pixel number + // but a number that has a weird ending, we need to convert it to pixels + if ( rnumnonpx.test( ret ) ) { + + // Remember the original values + left = style.left; + rsLeft = elem.runtimeStyle && elem.runtimeStyle.left; + + // Put in the new values to get a computed value out + if ( rsLeft ) { + elem.runtimeStyle.left = elem.currentStyle.left; + } + style.left = name === "fontSize" ? "1em" : ret; + ret = style.pixelLeft + "px"; + + // Revert the changed values + style.left = left; + if ( rsLeft ) { + elem.runtimeStyle.left = rsLeft; + } + } + + return ret === "" ? "auto" : ret; + }; +} + +curCSS = getComputedStyle || currentStyle; + +function getWidthOrHeight( elem, name, extra ) { + + // Start with offset property + var val = name === "width" ? elem.offsetWidth : elem.offsetHeight, + i = name === "width" ? 1 : 0, + len = 4; + if (val === undefined) { + val = name === "width" ? elem.getBBox().width : elem.getBBox().height; + } + if ( val > 0 ) { + if ( extra !== "border" ) { + for ( ; i < len; i += 2 ) { + if ( !extra ) { + val -= parseFloat( jQuery.css( elem, "padding" + cssExpand[ i ] ) ) || 0; + } + if ( extra === "margin" ) { + val += parseFloat( jQuery.css( elem, extra + cssExpand[ i ] ) ) || 0; + } else { + val -= parseFloat( jQuery.css( elem, "border" + cssExpand[ i ] + "Width" ) ) || 0; + } + } + } + + return val + "px"; + } + + // Fall back to computed then uncomputed css if necessary + val = curCSS( elem, name ); + if ( val < 0 || val == null ) { + val = elem.style[ name ]; + } + + // Computed unit is not pixels. Stop here and return. + if ( rnumnonpx.test(val) ) { + return val; + } + + // Normalize "", auto, and prepare for extra + val = parseFloat( val ) || 0; + + // Add padding, border, margin + if ( extra ) { + for ( ; i < len; i += 2 ) { + val += parseFloat( jQuery.css( elem, "padding" + cssExpand[ i ] ) ) || 0; + if ( extra !== "padding" ) { + val += parseFloat( jQuery.css( elem, "border" + cssExpand[ i ] + "Width" ) ) || 0; + } + if ( extra === "margin" ) { + val += parseFloat( jQuery.css( elem, extra + cssExpand[ i ]) ) || 0; + } + } + } + + return val + "px"; +} + +jQuery.each([ "height", "width" ], function( i, name ) { + jQuery.cssHooks[ name ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + if ( elem.offsetWidth !== 0 ) { + return getWidthOrHeight( elem, name, extra ); + } else { + return jQuery.swap( elem, cssShow, function() { + return getWidthOrHeight( elem, name, extra ); + }); + } + } + }, + + set: function( elem, value ) { + return rnum.test( value ) ? + value + "px" : + value; + } + }; +}); + +if ( !jQuery.support.opacity ) { + jQuery.cssHooks.opacity = { + get: function( elem, computed ) { + // IE uses filters for opacity + return ropacity.test( (computed && elem.currentStyle ? elem.currentStyle.filter : elem.style.filter) || "" ) ? + ( parseFloat( RegExp.$1 ) / 100 ) + "" : + computed ? "1" : ""; + }, + + set: function( elem, value ) { + var style = elem.style, + currentStyle = elem.currentStyle, + opacity = jQuery.isNumeric( value ) ? "alpha(opacity=" + value * 100 + ")" : "", + filter = currentStyle && currentStyle.filter || style.filter || ""; + + // IE has trouble with opacity if it does not have layout + // Force it by setting the zoom level + // style.zoom = 1; + + // if setting opacity to 1, and no other filters exist - attempt to remove filter attribute #6652 + // if ( value >= 1 && jQuery.trim( filter.replace( ralpha, "" ) ) === "" ) { + + // // Setting style.filter to null, "" & " " still leave "filter:" in the cssText + // // if "filter:" is present at all, clearType is disabled, we want to avoid this + // // style.removeAttribute is IE Only, but so apparently is this code path... + // style.removeAttribute( "filter" ); + + // // if there there is no filter style applied in a css rule, we are done + // if ( currentStyle && !currentStyle.filter ) { + // return; + // } + // } + + // otherwise, set new filter values + style.filter = ralpha.test( filter ) ? + filter.replace( ralpha, opacity ) : + filter + " " + opacity; + } + }; +} + +jQuery(function() { + // This hook cannot be added until DOM ready because the support test + // for it is not run until after DOM ready + if ( !jQuery.support.reliableMarginRight ) { + jQuery.cssHooks.marginRight = { + get: function( elem, computed ) { + // WebKit Bug 13343 - getComputedStyle returns wrong value for margin-right + // Work around by temporarily setting element display to inline-block + return jQuery.swap( elem, { "display": "inline-block" }, function() { + if ( computed ) { + return curCSS( elem, "margin-right" ); + } else { + return elem.style.marginRight; + } + }); + } + }; + } +}); + +if ( jQuery.expr && jQuery.expr.filters ) { + jQuery.expr.filters.hidden = function( elem ) { + var width = elem.offsetWidth, + height = elem.offsetHeight; + + return ( width === 0 && height === 0 ) || (!jQuery.support.reliableHiddenOffsets && ((elem.style && elem.style.display) || jQuery.css( elem, "display" )) === "none"); + }; + + jQuery.expr.filters.visible = function( elem ) { + return !jQuery.expr.filters.hidden( elem ); + }; +} + +// These hooks are used by animate to expand properties +jQuery.each({ + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i, + + // assumes a single number if not a string + parts = typeof value === "string" ? value.split(" ") : [ value ], + expanded = {}; + + for ( i = 0; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; +}); + + + + +var r20 = /%20/g, + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rhash = /#.*$/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)\r?$/mg, // IE leaves an \r character at EOL + rinput = /^(?:color|date|datetime|datetime-local|email|hidden|month|number|password|range|search|tel|text|time|url|week)$/i, + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app\-storage|.+\-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + rquery = /\?/, + rscript = /<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, + rselectTextarea = /^(?:select|textarea)/i, + rspacesAjax = /\s+/, + rts = /([?&])_=[^&]*/, + rurl = /^([\w\+\.\-]+:)(?:\/\/([^\/?#:]*)(?::(\d+))?)?/, + + // Keep a copy of the old load method + _load = jQuery.fn.load, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Document location + ajaxLocation, + + // Document location segments + ajaxLocParts, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = ["*/"] + ["*"]; + +// #8138, IE may throw an exception when accessing +// a field from window.location if document.domain has been set +try { + ajaxLocation = location.href; +} catch( e ) { + // Use the href attribute of an A element + // since IE will modify it given document.location + ajaxLocation = document.createElement( "a" ); + ajaxLocation.href = ""; + ajaxLocation = ajaxLocation.href; +} + +// Segment location into parts +ajaxLocParts = rurl.exec( ajaxLocation.toLowerCase() ) || []; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + if ( jQuery.isFunction( func ) ) { + var dataTypes = dataTypeExpression.toLowerCase().split( rspacesAjax ), + i = 0, + length = dataTypes.length, + dataType, + list, + placeBefore; + + // For each dataType in the dataTypeExpression + for ( ; i < length; i++ ) { + dataType = dataTypes[ i ]; + // We control if we're asked to add before + // any existing element + placeBefore = /^\+/.test( dataType ); + if ( placeBefore ) { + dataType = dataType.substr( 1 ) || "*"; + } + list = structure[ dataType ] = structure[ dataType ] || []; + // then we add to the structure accordingly + list[ placeBefore ? "unshift" : "push" ]( func ); + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR, + dataType /* internal */, inspected /* internal */ ) { + + dataType = dataType || options.dataTypes[ 0 ]; + inspected = inspected || {}; + + inspected[ dataType ] = true; + + var list = structure[ dataType ], + i = 0, + length = list ? list.length : 0, + executeOnly = ( structure === prefilters ), + selection; + + for ( ; i < length && ( executeOnly || !selection ); i++ ) { + selection = list[ i ]( options, originalOptions, jqXHR ); + // If we got redirected to another dataType + // we try there if executing only and not done already + if ( typeof selection === "string" ) { + if ( !executeOnly || inspected[ selection ] ) { + selection = undefined; + } else { + options.dataTypes.unshift( selection ); + selection = inspectPrefiltersOrTransports( + structure, options, originalOptions, jqXHR, selection, inspected ); + } + } + } + // If we're only executing or nothing was selected + // we try the catchall dataType if not done already + if ( ( executeOnly || !selection ) && !inspected[ "*" ] ) { + selection = inspectPrefiltersOrTransports( + structure, options, originalOptions, jqXHR, "*", inspected ); + } + // unnecessary when only executing (prefilters) + // but it'll be ignored by the caller in that case + return selection; +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } +} + +jQuery.fn.extend({ + load: function( url, params, callback ) { + if ( typeof url !== "string" && _load ) { + return _load.apply( this, arguments ); + + // Don't do a request if no elements are being requested + } else if ( !this.length ) { + return this; + } + + var off = url.indexOf( " " ); + if ( off >= 0 ) { + var selector = url.slice( off, url.length ); + url = url.slice( 0, off ); + } + + // Default to a GET request + var type = "GET"; + + // If the second parameter was provided + if ( params ) { + // If it's a function + if ( jQuery.isFunction( params ) ) { + // We assume that it's the callback + callback = params; + params = undefined; + + // Otherwise, build a param string + } else if ( typeof params === "object" ) { + params = jQuery.param( params, jQuery.ajaxSettings.traditional ); + type = "POST"; + } + } + + var self = this; + + // Request the remote document + jQuery.ajax({ + url: url, + type: type, + dataType: "html", + data: params, + // Complete callback (responseText is used internally) + complete: function( jqXHR, status, responseText ) { + // Store the response as specified by the jqXHR object + responseText = jqXHR.responseText; + // If successful, inject the HTML into all the matched elements + if ( jqXHR.isResolved() ) { + // #4825: Get the actual response in case + // a dataFilter is present in ajaxSettings + jqXHR.done(function( r ) { + responseText = r; + }); + // See if a selector was specified + self.html( selector ? + // Create a dummy div to hold the results + jQuery("<div>") + // inject the contents of the document in, removing the scripts + // to avoid any 'Permission Denied' errors in IE + .append(responseText.replace(rscript, "")) + + // Locate the specified elements + .find(selector) : + + // If not, just inject the full result + responseText ); + } + + if ( callback ) { + self.each( callback, [ responseText, status, jqXHR ] ); + } + } + }); + + return this; + }, + + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + + serializeArray: function() { + return this.map(function(){ + return this.elements ? jQuery.makeArray( this.elements ) : this; + }) + .filter(function(){ + return this.name && !this.disabled && + ( this.checked || rselectTextarea.test( this.nodeName ) || + rinput.test( this.type ) ); + }) + .map(function( i, elem ){ + var val = jQuery( this ).val(); + + return val == null ? + null : + jQuery.isArray( val ) ? + jQuery.map( val, function( val, i ){ + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + }) : + { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + }).get(); + } +}); + +// Attach a bunch of functions for handling common AJAX events +jQuery.each( "ajaxStart ajaxStop ajaxComplete ajaxError ajaxSuccess ajaxSend".split( " " ), function( i, o ){ + jQuery.fn[ o ] = function( f ){ + return this.on( o, f ); + }; +}); + +jQuery.each( [ "get", "post" ], function( i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + // shift arguments if data argument was omitted + if ( jQuery.isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + return jQuery.ajax({ + type: method, + url: url, + data: data, + success: callback, + dataType: type + }); + }; +}); + +jQuery.extend({ + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + if ( settings ) { + // Building a settings object + ajaxExtend( target, jQuery.ajaxSettings ); + } else { + // Extending ajaxSettings + settings = target; + target = jQuery.ajaxSettings; + } + ajaxExtend( target, settings ); + return target; + }, + + ajaxSettings: { + url: ajaxLocation, + isLocal: rlocalProtocol.test( ajaxLocParts[ 1 ] ), + global: true, + type: "GET", + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + processData: true, + async: true, + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + traditional: false, + headers: {}, + */ + + accepts: { + xml: "application/xml, text/xml", + html: "text/html", + text: "text/plain", + json: "application/json, text/javascript", + "*": allTypes + }, + + contents: { + xml: /xml/, + html: /html/, + json: /json/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText" + }, + + // List of data converters + // 1) key format is "source_type destination_type" (a single space in-between) + // 2) the catchall symbol "*" can be used for source_type + converters: { + + // Convert anything to text + "* text": window.String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": jQuery.parseJSON, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + context: true, + url: true + } + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + // Callbacks context + callbackContext = s.context || s, + // Context for global events + // It's the callbackContext if one was provided in the options + // and if it's a DOM node or a jQuery collection + globalEventContext = callbackContext !== s && + ( callbackContext.nodeType || callbackContext instanceof jQuery ) ? + jQuery( callbackContext ) : jQuery.event, + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + // Status-dependent callbacks + statusCode = s.statusCode || {}, + // ifModified key + ifModifiedKey, + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + // Response headers + responseHeadersString, + responseHeaders, + // transport + transport, + // timeout handle + timeoutTimer, + // Cross-domain detection vars + parts, + // The jqXHR state + state = 0, + // To know if global events are to be dispatched + fireGlobals, + // Loop variable + i, + // Fake xhr + jqXHR = { + + readyState: 0, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( !state ) { + var lname = name.toLowerCase(); + name = requestHeadersNames[ lname ] = requestHeadersNames[ lname ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Raw string + getAllResponseHeaders: function() { + return state === 2 ? responseHeadersString : null; + }, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( state === 2 ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[1].toLowerCase() ] = match[ 2 ]; + } + } + match = responseHeaders[ key.toLowerCase() ]; + } + return match === undefined ? null : match; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( !state ) { + s.mimeType = type; + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + statusText = statusText || "abort"; + if ( transport ) { + transport.abort( statusText ); + } + done( 0, statusText ); + return this; + } + }; + + // Callback for when everything is done + // It is defined here because jslint complains if it is declared + // at the end of the function (which would be more logical and readable) + function done( status, nativeStatusText, responses, headers ) { + + // Called once + if ( state === 2 ) { + return; + } + + // State is "done" now + state = 2; + + // Clear timeout if it exists + if ( timeoutTimer ) { + clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + var isSuccess, + success, + error, + statusText = nativeStatusText, + response = responses ? ajaxHandleResponses( s, jqXHR, responses ) : undefined, + lastModified, + etag; + + // If successful, handle type chaining + if ( status >= 200 && status < 300 || status === 304 ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + + if ( ( lastModified = jqXHR.getResponseHeader( "Last-Modified" ) ) ) { + jQuery.lastModified[ ifModifiedKey ] = lastModified; + } + if ( ( etag = jqXHR.getResponseHeader( "Etag" ) ) ) { + jQuery.etag[ ifModifiedKey ] = etag; + } + } + + // If not modified + if ( status === 304 ) { + + statusText = "notmodified"; + isSuccess = true; + + // If we have data + } else { + + try { + success = ajaxConvert( s, response ); + statusText = "success"; + isSuccess = true; + } catch(e) { + // We have a parsererror + statusText = "parsererror"; + error = e; + } + } + } else { + // We extract error from statusText + // then normalize statusText and status for non-aborts + error = statusText; + if ( !statusText || status ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = "" + ( nativeStatusText || statusText ); + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( "ajax" + ( isSuccess ? "Success" : "Error" ), + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + // Attach deferreds + deferred.promise( jqXHR ); + jqXHR.success = jqXHR.done; + jqXHR.error = jqXHR.fail; + jqXHR.complete = completeDeferred.add; + + // Status-dependent callbacks + jqXHR.statusCode = function( map ) { + if ( map ) { + var tmp; + if ( state < 2 ) { + for ( tmp in map ) { + statusCode[ tmp ] = [ statusCode[tmp], map[tmp] ]; + } + } else { + tmp = map[ jqXHR.status ]; + jqXHR.then( tmp, tmp ); + } + } + return this; + }; + + // Remove hash character (#7531: and string promotion) + // Add protocol if not provided (#5866: IE7 issue with protocol-less urls) + // We also use the url parameter if available + s.url = ( ( url || s.url ) + "" ).replace( rhash, "" ).replace( rprotocol, ajaxLocParts[ 1 ] + "//" ); + + // Extract dataTypes list + s.dataTypes = jQuery.trim( s.dataType || "*" ).toLowerCase().split( rspacesAjax ); + + // Determine if a cross-domain request is in order + if ( s.crossDomain == null ) { + parts = rurl.exec( s.url.toLowerCase() ); + s.crossDomain = !!( parts && + ( parts[ 1 ] != ajaxLocParts[ 1 ] || parts[ 2 ] != ajaxLocParts[ 2 ] || + ( parts[ 3 ] || ( parts[ 1 ] === "http:" ? 80 : 443 ) ) != + ( ajaxLocParts[ 3 ] || ( ajaxLocParts[ 1 ] === "http:" ? 80 : 443 ) ) ) + ); + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( state === 2 ) { + return false; + } + + // We can fire global events as of now if asked to + fireGlobals = s.global; + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // If data is available, append data to url + if ( s.data ) { + s.url += ( rquery.test( s.url ) ? "&" : "?" ) + s.data; + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Get ifModifiedKey before adding the anti-cache parameter + ifModifiedKey = s.url; + + // Add anti-cache in url if needed + if ( s.cache === false ) { + + var ts = jQuery.now(), + // try replacing _= if it is there + ret = s.url.replace( rts, "$1_=" + ts ); + + // if nothing was replaced, add timestamp to the end + s.url = ret + ( ( ret === s.url ) ? ( rquery.test( s.url ) ? "&" : "?" ) + "_=" + ts : "" ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + ifModifiedKey = ifModifiedKey || s.url; + if ( jQuery.lastModified[ ifModifiedKey ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ ifModifiedKey ] ); + } + if ( jQuery.etag[ ifModifiedKey ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ ifModifiedKey ] ); + } + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[0] ] ? + s.accepts[ s.dataTypes[0] ] + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || state === 2 ) ) { + // Abort if not done already + jqXHR.abort(); + return false; + + } + + // Install callbacks on deferreds + for ( i in { success: 1, error: 1, complete: 1 } ) { + jqXHR[ i ]( s[ i ] ); + } + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = setTimeout( function(){ + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + state = 1; + transport.send( requestHeaders, done ); + } catch (e) { + // Propagate exception as error if not done + if ( state < 2 ) { + done( -1, e ); + // Simply rethrow otherwise + } else { + throw e; + } + } + } + + return jqXHR; + }, + + // Serialize an array of form elements or a set of + // key/values into a query string + param: function( a, traditional ) { + var s = [], + add = function( key, value ) { + // If value is a function, invoke it and return its value + value = jQuery.isFunction( value ) ? value() : value; + s[ s.length ] = encodeURIComponent( key ) + "=" + encodeURIComponent( value ); + }; + + // Set traditional to true for jQuery <= 1.3.2 behavior. + if ( traditional === undefined ) { + traditional = jQuery.ajaxSettings.traditional; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( jQuery.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + }); + + } else { + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( var prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ).replace( r20, "+" ); + } +}); + +function buildParams( prefix, obj, traditional, add ) { + if ( jQuery.isArray( obj ) ) { + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + // If array item is non-scalar (array or object), encode its + // numeric index to resolve deserialization ambiguity issues. + // Note that rack (as of 1.0.0) can't currently deserialize + // nested arrays properly, and attempting to do so may cause + // a server error. Possible fixes are to modify rack's + // deserialization algorithm or to provide an option or flag + // to force array serialization to be shallow. + buildParams( prefix + "[" + ( typeof v === "object" ? i : "" ) + "]", v, traditional, add ); + } + }); + + } else if ( !traditional && jQuery.type( obj ) === "object" ) { + // Serialize object item. + for ( var name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + // Serialize scalar item. + add( prefix, obj ); + } +} + +// This is still on the jQuery object... for now +// Want to move this to jQuery.ajax some day +jQuery.extend({ + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {} + +}); + +/* Handles responses to an ajax request: + * - sets all responseXXX fields accordingly + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var contents = s.contents, + dataTypes = s.dataTypes, + responseFields = s.responseFields, + ct, + type, + finalDataType, + firstDataType; + + // Fill responseXXX fields + for ( type in responseFields ) { + if ( type in responses ) { + jqXHR[ responseFields[type] ] = responses[ type ]; + } + } + + // Remove auto dataType and get content-type in the process + while( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "content-type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[0] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +// Chain conversions given the request and the original response +function ajaxConvert( s, response ) { + + // Apply the dataFilter if provided + if ( s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + var dataTypes = s.dataTypes, + converters = {}, + i, + key, + length = dataTypes.length, + tmp, + // Current and previous dataTypes + current = dataTypes[ 0 ], + prev, + // Conversion expression + conversion, + // Conversion function + conv, + // Conversion functions (transitive conversion) + conv1, + conv2; + + // For each dataType in the chain + for ( i = 1; i < length; i++ ) { + + // Create converters map + // with lowercased keys + if ( i === 1 ) { + for ( key in s.converters ) { + if ( typeof key === "string" ) { + converters[ key.toLowerCase() ] = s.converters[ key ]; + } + } + } + + // Get the dataTypes + prev = current; + current = dataTypes[ i ]; + + // If current is auto dataType, update it to prev + if ( current === "*" ) { + current = prev; + // If no auto and dataTypes are actually different + } else if ( prev !== "*" && prev !== current ) { + + // Get the converter + conversion = prev + " " + current; + conv = converters[ conversion ] || converters[ "* " + current ]; + + // If there is no direct converter, search transitively + if ( !conv ) { + conv2 = undefined; + for ( conv1 in converters ) { + tmp = conv1.split( " " ); + if ( tmp[ 0 ] === prev || tmp[ 0 ] === "*" ) { + conv2 = converters[ tmp[1] + " " + current ]; + if ( conv2 ) { + conv1 = converters[ conv1 ]; + if ( conv1 === true ) { + conv = conv2; + } else if ( conv2 === true ) { + conv = conv1; + } + break; + } + } + } + } + // If we found no converter, dispatch an error + if ( !( conv || conv2 ) ) { + jQuery.error( "No conversion from " + conversion.replace(" "," to ") ); + } + // If found converter is not an equivalence + if ( conv !== true ) { + // Convert with 1 or 2 converters accordingly + response = conv ? conv( response ) : conv2( conv1(response) ); + } + } + } + return response; +} + + + + +var jsc = jQuery.now(), + jsre = /(\=)\?(&|$)|\?\?/i; + +// Default jsonp settings +jQuery.ajaxSetup({ + jsonp: "callback", + jsonpCallback: function() { + return jQuery.expando + "_" + ( jsc++ ); + } +}); + +// Detect, normalize options and install callbacks for jsonp requests +jQuery.ajaxPrefilter( "json jsonp", function( s, originalSettings, jqXHR ) { + + var inspectData = ( typeof s.data === "string" ) && /^application\/x\-www\-form\-urlencoded/.test( s.contentType ); + + if ( s.dataTypes[ 0 ] === "jsonp" || + s.jsonp !== false && ( jsre.test( s.url ) || + inspectData && jsre.test( s.data ) ) ) { + + var responseContainer, + jsonpCallback = s.jsonpCallback = + jQuery.isFunction( s.jsonpCallback ) ? s.jsonpCallback() : s.jsonpCallback, + previous = window[ jsonpCallback ], + url = s.url, + data = s.data, + replace = "$1" + jsonpCallback + "$2"; + + if ( s.jsonp !== false ) { + url = url.replace( jsre, replace ); + if ( s.url === url ) { + if ( inspectData ) { + data = data.replace( jsre, replace ); + } + if ( s.data === data ) { + // Add callback manually + url += (/\?/.test( url ) ? "&" : "?") + s.jsonp + "=" + jsonpCallback; + } + } + } + + s.url = url; + s.data = data; + + // Install callback + window[ jsonpCallback ] = function( response ) { + responseContainer = [ response ]; + }; + + // Clean-up function + jqXHR.always(function() { + // Set callback back to previous value + window[ jsonpCallback ] = previous; + // Call if it was a function and we have a response + if ( responseContainer && jQuery.isFunction( previous ) ) { + window[ jsonpCallback ]( responseContainer[ 0 ] ); + } + }); + + // Use data converter to retrieve json after script execution + s.converters["script json"] = function() { + if ( !responseContainer ) { + jQuery.error( jsonpCallback + " was not called" ); + } + return responseContainer[ 0 ]; + }; + + // force json dataType + s.dataTypes[ 0 ] = "json"; + + // Delegate to script + return "script"; + } +}); + + + + +// Install script dataType +jQuery.ajaxSetup({ + accepts: { + script: "text/javascript, application/javascript, application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /javascript|ecmascript/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +}); + +// Handle cache's special case and global +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + s.global = false; + } +}); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function(s) { + + // This transport only deals with cross domain requests + if ( s.crossDomain ) { + + var script, + head = document.head || document.getElementsByTagName( "head" )[0] || document.documentElement; + + return { + + send: function( _, callback ) { + + script = document.createElement( "script" ); + + script.async = "async"; + + if ( s.scriptCharset ) { + script.charset = s.scriptCharset; + } + + script.src = s.url; + + // Attach handlers for all browsers + script.onload = script.onreadystatechange = function( _, isAbort ) { + + if ( isAbort || !script.readyState || /loaded|complete/.test( script.readyState ) ) { + + // Handle memory leak in IE + script.onload = script.onreadystatechange = null; + + // Remove the script + if ( head && script.parentNode ) { + head.removeChild( script ); + } + + // Dereference the script + script = undefined; + + // Callback if not abort + if ( !isAbort ) { + callback( 200, "success" ); + } + } + }; + // Use insertBefore instead of appendChild to circumvent an IE6 bug. + // This arises when a base node is used (#2709 and #4378). + head.insertBefore( script, head.firstChild ); + }, + + abort: function() { + if ( script ) { + script.onload( 0, 1 ); + } + } + }; + } +}); + + + + +var // #5280: Internet Explorer will keep connections alive if we don't abort on unload + xhrOnUnloadAbort = window.ActiveXObject ? function() { + // Abort all pending requests + for ( var key in xhrCallbacks ) { + xhrCallbacks[ key ]( 0, 1 ); + } + } : false, + xhrId = 0, + xhrCallbacks; + +// Functions to create xhrs +function createStandardXHR() { + try { + return new window.XMLHttpRequest(); + } catch( e ) {} +} + +function createActiveXHR() { + try { + return new window.ActiveXObject( "Microsoft.XMLHTTP" ); + } catch( e ) {} +} + +// Create the request object +// (This is still attached to ajaxSettings for backward compatibility) +jQuery.ajaxSettings.xhr = window.ActiveXObject ? + /* Microsoft failed to properly + * implement the XMLHttpRequest in IE7 (can't request local files), + * so we use the ActiveXObject when it is available + * Additionally XMLHttpRequest can be disabled in IE7/IE8 so + * we need a fallback. + */ + function() { + return !this.isLocal && createStandardXHR() || createActiveXHR(); + } : + // For all other browsers, use the standard XMLHttpRequest object + createStandardXHR; + +// Determine support properties +(function( xhr ) { + jQuery.extend( jQuery.support, { + ajax: !!xhr, + cors: !!xhr && ( "withCredentials" in xhr ) + }); +})( jQuery.ajaxSettings.xhr() ); + +// Create transport if the browser can provide an xhr +if ( jQuery.support.ajax ) { + + jQuery.ajaxTransport(function( s ) { + // Cross domain only allowed if supported through XMLHttpRequest + if ( !s.crossDomain || jQuery.support.cors ) { + + var callback; + + return { + send: function( headers, complete ) { + + // Get a new xhr + var xhr = s.xhr(), + handle, + i; + + // Open the socket + // Passing null username, generates a login popup on Opera (#2865) + if ( s.username ) { + xhr.open( s.type, s.url, s.async, s.username, s.password ); + } else { + xhr.open( s.type, s.url, s.async ); + } + + // Apply custom fields if provided + if ( s.xhrFields ) { + for ( i in s.xhrFields ) { + xhr[ i ] = s.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( s.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( s.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !s.crossDomain && !headers["X-Requested-With"] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Need an extra try/catch for cross domain requests in Firefox 3 + try { + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + } catch( _ ) {} + + // Do send the request + // This may raise an exception which is actually + // handled in jQuery.ajax (so no try/catch here) + xhr.send( ( s.hasContent && s.data ) || null ); + + // Listener + callback = function( _, isAbort ) { + + var status, + statusText, + responseHeaders, + responses, + xml; + + // Firefox throws exceptions when accessing properties + // of an xhr when a network error occured + // http://helpful.knobs-dials.com/index.php/Component_returned_failure_code:_0x80040111_(NS_ERROR_NOT_AVAILABLE) + try { + + // Was never called and is aborted or complete + if ( callback && ( isAbort || xhr.readyState === 4 ) ) { + + // Only called once + callback = undefined; + + // Do not keep as active anymore + if ( handle ) { + xhr.onreadystatechange = jQuery.noop; + if ( xhrOnUnloadAbort ) { + delete xhrCallbacks[ handle ]; + } + } + + // If it's an abort + if ( isAbort ) { + // Abort it manually if needed + if ( xhr.readyState !== 4 ) { + xhr.abort(); + } + } else { + status = xhr.status; + responseHeaders = xhr.getAllResponseHeaders(); + responses = {}; + xml = xhr.responseXML; + + // Construct response list + if ( xml && xml.documentElement /* #4958 */ ) { + responses.xml = xml; + } + + // When requesting binary data, IE6-9 will throw an exception + // on any attempt to access responseText (#11426) + try { + responses.text = xhr.responseText; + } catch( _ ) { + } + + // Firefox throws an exception when accessing + // statusText for faulty cross-domain requests + try { + statusText = xhr.statusText; + } catch( e ) { + // We normalize with Webkit giving an empty statusText + statusText = ""; + } + + // Filter status for non standard behaviors + + // If the request is local and we have data: assume a success + // (success with no data won't get notified, that's the best we + // can do given current implementations) + if ( !status && s.isLocal && !s.crossDomain ) { + status = responses.text ? 200 : 404; + // IE - #1450: sometimes returns 1223 when it should be 204 + } else if ( status === 1223 ) { + status = 204; + } + } + } + } catch( firefoxAccessException ) { + if ( !isAbort ) { + complete( -1, firefoxAccessException ); + } + } + + // Call complete if needed + if ( responses ) { + complete( status, statusText, responses, responseHeaders ); + } + }; + + // if we're in sync mode or it's in cache + // and has been retrieved directly (IE6 & IE7) + // we need to manually fire the callback + if ( !s.async || xhr.readyState === 4 ) { + callback(); + } else { + handle = ++xhrId; + if ( xhrOnUnloadAbort ) { + // Create the active xhrs callbacks list if needed + // and attach the unload handler + if ( !xhrCallbacks ) { + xhrCallbacks = {}; + jQuery( window ).unload( xhrOnUnloadAbort ); + } + // Add to list of active xhrs callbacks + xhrCallbacks[ handle ] = callback; + } + xhr.onreadystatechange = callback; + } + }, + + abort: function() { + if ( callback ) { + callback(0,1); + } + } + }; + } + }); +} + + + + +var elemdisplay = {}, + iframe, iframeDoc, + rfxtypes = /^(?:toggle|show|hide)$/, + rfxnum = /^([+\-]=)?([\d+.\-]+)([a-z%]*)$/i, + timerId, + fxAttrs = [ + // height animations + [ "height", "marginTop", "marginBottom", "paddingTop", "paddingBottom" ], + // width animations + [ "width", "marginLeft", "marginRight", "paddingLeft", "paddingRight" ], + // opacity animations + [ "opacity" ] + ], + fxNow; + +jQuery.fn.extend({ + show: function( speed, easing, callback ) { + var elem, display; + + if ( speed || speed === 0 ) { + return this.animate( genFx("show", 3), speed, easing, callback ); + + } else { + for ( var i = 0, j = this.length; i < j; i++ ) { + elem = this[ i ]; + + if ( elem.style ) { + display = elem.style.display; + + // Reset the inline display of this element to learn if it is + // being hidden by cascaded rules or not + if ( !jQuery._data(elem, "olddisplay") && display === "none" ) { + display = elem.style.display = ""; + } + + // Set elements which have been overridden with display: none + // in a stylesheet to whatever the default browser style is + // for such an element + if ( (display === "" && jQuery.css(elem, "display") === "none") || + !jQuery.contains( elem.ownerDocument.documentElement, elem ) ) { + jQuery._data( elem, "olddisplay", defaultDisplay(elem.nodeName) ); + } + } + } + + // Set the display of most of the elements in a second loop + // to avoid the constant reflow + for ( i = 0; i < j; i++ ) { + elem = this[ i ]; + + if ( elem.style ) { + display = elem.style.display; + + if ( display === "" || display === "none" ) { + elem.style.display = jQuery._data( elem, "olddisplay" ) || ""; + } + } + } + + return this; + } + }, + + hide: function( speed, easing, callback ) { + if ( speed || speed === 0 ) { + return this.animate( genFx("hide", 3), speed, easing, callback); + + } else { + var elem, display, + i = 0, + j = this.length; + + for ( ; i < j; i++ ) { + elem = this[i]; + if ( elem.style ) { + display = jQuery.css( elem, "display" ); + + if ( display !== "none" && !jQuery._data( elem, "olddisplay" ) ) { + jQuery._data( elem, "olddisplay", display ); + } + } + } + + // Set the display of the elements in a second loop + // to avoid the constant reflow + for ( i = 0; i < j; i++ ) { + if ( this[i].style ) { + this[i].style.display = "none"; + } + } + + return this; + } + }, + + // Save the old toggle function + _toggle: jQuery.fn.toggle, + + toggle: function( fn, fn2, callback ) { + var bool = typeof fn === "boolean"; + + if ( jQuery.isFunction(fn) && jQuery.isFunction(fn2) ) { + this._toggle.apply( this, arguments ); + + } else if ( fn == null || bool ) { + this.each(function() { + var state = bool ? fn : jQuery(this).is(":hidden"); + jQuery(this)[ state ? "show" : "hide" ](); + }); + + } else { + this.animate(genFx("toggle", 3), fn, fn2, callback); + } + + return this; + }, + + fadeTo: function( speed, to, easing, callback ) { + return this.filter(":hidden").css("opacity", 0).show().end() + .animate({opacity: to}, speed, easing, callback); + }, + + animate: function( prop, speed, easing, callback ) { + var optall = jQuery.speed( speed, easing, callback ); + + if ( jQuery.isEmptyObject( prop ) ) { + return this.each( optall.complete, [ false ] ); + } + + // Do not change referenced properties as per-property easing will be lost + prop = jQuery.extend( {}, prop ); + + function doAnimation() { + // XXX 'this' does not always have a nodeName when running the + // test suite + + if ( optall.queue === false ) { + jQuery._mark( this ); + } + + var opt = jQuery.extend( {}, optall ), + isElement = this.nodeType === 1, + hidden = isElement && jQuery(this).is(":hidden"), + name, val, p, e, hooks, replace, + parts, start, end, unit, + method; + + // will store per property easing and be used to determine when an animation is complete + opt.animatedProperties = {}; + + // first pass over propertys to expand / normalize + for ( p in prop ) { + name = jQuery.camelCase( p ); + if ( p !== name ) { + prop[ name ] = prop[ p ]; + delete prop[ p ]; + } + + if ( ( hooks = jQuery.cssHooks[ name ] ) && "expand" in hooks ) { + replace = hooks.expand( prop[ name ] ); + delete prop[ name ]; + + // not quite $.extend, this wont overwrite keys already present. + // also - reusing 'p' from above because we have the correct "name" + for ( p in replace ) { + if ( ! ( p in prop ) ) { + prop[ p ] = replace[ p ]; + } + } + } + } + + for ( name in prop ) { + val = prop[ name ]; + // easing resolution: per property > opt.specialEasing > opt.easing > 'swing' (default) + if ( jQuery.isArray( val ) ) { + opt.animatedProperties[ name ] = val[ 1 ]; + val = prop[ name ] = val[ 0 ]; + } else { + opt.animatedProperties[ name ] = opt.specialEasing && opt.specialEasing[ name ] || opt.easing || 'swing'; + } + + if ( val === "hide" && hidden || val === "show" && !hidden ) { + return opt.complete.call( this ); + } + + if ( isElement && ( name === "height" || name === "width" ) ) { + // Make sure that nothing sneaks out + // Record all 3 overflow attributes because IE does not + // change the overflow attribute when overflowX and + // overflowY are set to the same value + opt.overflow = [ this.style.overflow, this.style.overflowX, this.style.overflowY ]; + + // Set display property to inline-block for height/width + // animations on inline elements that are having width/height animated + if ( jQuery.css( this, "display" ) === "inline" && + jQuery.css( this, "float" ) === "none" ) { + + // inline-level elements accept inline-block; + // block-level elements need to be inline with layout + if ( !jQuery.support.inlineBlockNeedsLayout || defaultDisplay( this.nodeName ) === "inline" ) { + this.style.display = "inline-block"; + + } else { + this.style.zoom = 1; + } + } + } + } + + if ( opt.overflow != null ) { + this.style.overflow = "hidden"; + } + + for ( p in prop ) { + e = new jQuery.fx( this, opt, p ); + val = prop[ p ]; + + if ( rfxtypes.test( val ) ) { + + // Tracks whether to show or hide based on private + // data attached to the element + method = jQuery._data( this, "toggle" + p ) || ( val === "toggle" ? hidden ? "show" : "hide" : 0 ); + if ( method ) { + jQuery._data( this, "toggle" + p, method === "show" ? "hide" : "show" ); + e[ method ](); + } else { + e[ val ](); + } + + } else { + parts = rfxnum.exec( val ); + start = e.cur(); + + if ( parts ) { + end = parseFloat( parts[2] ); + unit = parts[3] || ( jQuery.cssNumber[ p ] ? "" : "px" ); + + // We need to compute starting value + if ( unit !== "px" ) { + jQuery.style( this, p, (end || 1) + unit); + start = ( (end || 1) / e.cur() ) * start; + jQuery.style( this, p, start + unit); + } + + // If a +=/-= token was provided, we're doing a relative animation + if ( parts[1] ) { + end = ( (parts[ 1 ] === "-=" ? -1 : 1) * end ) + start; + } + + e.custom( start, end, unit ); + + } else { + e.custom( start, val, "" ); + } + } + } + + // For JS strict compliance + return true; + } + + return optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + + stop: function( type, clearQueue, gotoEnd ) { + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue && type !== false ) { + this.queue( type || "fx", [] ); + } + + return this.each(function() { + var index, + hadTimers = false, + timers = jQuery.timers, + data = jQuery._data( this ); + + // clear marker counters if we know they won't be + if ( !gotoEnd ) { + jQuery._unmark( true, this ); + } + + function stopQueue( elem, data, index ) { + var hooks = data[ index ]; + jQuery.removeData( elem, index, true ); + hooks.stop( gotoEnd ); + } + + if ( type == null ) { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && index.indexOf(".run") === index.length - 4 ) { + stopQueue( this, data, index ); + } + } + } else if ( data[ index = type + ".run" ] && data[ index ].stop ){ + stopQueue( this, data, index ); + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && (type == null || timers[ index ].queue === type) ) { + if ( gotoEnd ) { + + // force the next step to be the last + timers[ index ]( true ); + } else { + timers[ index ].saveState(); + } + hadTimers = true; + timers.splice( index, 1 ); + } + } + + // start the next in the queue if the last step wasn't forced + // timers currently will call their complete callbacks, which will dequeue + // but only if they were gotoEnd + if ( !( gotoEnd && hadTimers ) ) { + jQuery.dequeue( this, type ); + } + }); + } + +}); + +// Animations created synchronously will run synchronously +function createFxNow() { + setTimeout( clearFxNow, 0 ); + return ( fxNow = jQuery.now() ); +} + +function clearFxNow() { + fxNow = undefined; +} + +// Generate parameters to create a standard animation +function genFx( type, num ) { + var obj = {}; + + jQuery.each( fxAttrs.concat.apply([], fxAttrs.slice( 0, num )), function() { + obj[ this ] = type; + }); + + return obj; +} + +// Generate shortcuts for custom animations +jQuery.each({ + slideDown: genFx( "show", 1 ), + slideUp: genFx( "hide", 1 ), + slideToggle: genFx( "toggle", 1 ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +}); + +jQuery.extend({ + speed: function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + jQuery.isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !jQuery.isFunction( easing ) && easing + }; + + opt.duration = jQuery.fx.off ? 0 : typeof opt.duration === "number" ? opt.duration : + opt.duration in jQuery.fx.speeds ? jQuery.fx.speeds[ opt.duration ] : jQuery.fx.speeds._default; + + // normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function( noUnmark ) { + if ( jQuery.isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } else if ( noUnmark !== false ) { + jQuery._unmark( this ); + } + }; + + return opt; + }, + + easing: { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return ( -Math.cos( p*Math.PI ) / 2 ) + 0.5; + } + }, + + timers: [], + + fx: function( elem, options, prop ) { + this.options = options; + this.elem = elem; + this.prop = prop; + + options.orig = options.orig || {}; + } + +}); + +jQuery.fx.prototype = { + // Simple function for setting a style value + update: function() { + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + ( jQuery.fx.step[ this.prop ] || jQuery.fx.step._default )( this ); + }, + + // Get the current size + cur: function() { + if ( this.elem[ this.prop ] != null && (!this.elem.style || this.elem.style[ this.prop ] == null) ) { + if (this.elem[ this.prop ].baseVal) { + return this.elem[ this.prop ].baseVal.value; + } + return this.elem[ this.prop ]; + } + + var parsed, + r = jQuery.css( this.elem, this.prop ); + // Empty strings, null, undefined and "auto" are converted to 0, + // complex values such as "rotate(1rad)" are returned as is, + // simple values such as "10px" are parsed to Float. + return isNaN( parsed = parseFloat( r ) ) ? !r || r === "auto" ? 0 : r : parsed; + }, + + // Start an animation from one number to another + custom: function( from, to, unit ) { + var self = this, + fx = jQuery.fx; + + this.startTime = fxNow || createFxNow(); + this.end = to; + this.now = this.start = from; + this.pos = this.state = 0; + this.unit = unit || this.unit || ( jQuery.cssNumber[ this.prop ] ? "" : "px" ); + + function t( gotoEnd ) { + return self.step( gotoEnd ); + } + + t.queue = this.options.queue; + t.elem = this.elem; + t.saveState = function() { + if ( jQuery._data( self.elem, "fxshow" + self.prop ) === undefined ) { + if ( self.options.hide ) { + jQuery._data( self.elem, "fxshow" + self.prop, self.start ); + } else if ( self.options.show ) { + jQuery._data( self.elem, "fxshow" + self.prop, self.end ); + } + } + }; + + if ( t() && jQuery.timers.push(t) && !timerId ) { + timerId = setInterval( fx.tick, fx.interval ); + } + }, + + // Simple 'show' function + show: function() { + var dataShow = jQuery._data( this.elem, "fxshow" + this.prop ); + + // Remember where we started, so that we can go back to it later + this.options.orig[ this.prop ] = dataShow || jQuery.style( this.elem, this.prop ); + this.options.show = true; + + // Begin the animation + // Make sure that we start at a small width/height to avoid any flash of content + if ( dataShow !== undefined ) { + // This show is picking up where a previous hide or show left off + this.custom( this.cur(), dataShow ); + } else { + this.custom( this.prop === "width" || this.prop === "height" ? 1 : 0, this.cur() ); + } + + // Start by showing the element + jQuery( this.elem ).show(); + }, + + // Simple 'hide' function + hide: function() { + // Remember where we started, so that we can go back to it later + this.options.orig[ this.prop ] = jQuery._data( this.elem, "fxshow" + this.prop ) || jQuery.style( this.elem, this.prop ); + this.options.hide = true; + + // Begin the animation + this.custom( this.cur(), 0 ); + }, + + // Each step of an animation + step: function( gotoEnd ) { + var p, n, complete, + t = fxNow || createFxNow(), + done = true, + elem = this.elem, + options = this.options; + + if ( gotoEnd || t >= options.duration + this.startTime ) { + this.now = this.end; + this.pos = this.state = 1; + this.update(); + + options.animatedProperties[ this.prop ] = true; + + for ( p in options.animatedProperties ) { + if ( options.animatedProperties[ p ] !== true ) { + done = false; + } + } + + if ( done ) { + // Reset the overflow + if ( options.overflow != null && !jQuery.support.shrinkWrapBlocks ) { + + jQuery.each( [ "", "X", "Y" ], function( index, value ) { + elem.style[ "overflow" + value ] = options.overflow[ index ]; + }); + } + + // Hide the element if the "hide" operation was done + if ( options.hide ) { + jQuery( elem ).hide(); + } + + // Reset the properties, if the item has been hidden or shown + if ( options.hide || options.show ) { + for ( p in options.animatedProperties ) { + jQuery.style( elem, p, options.orig[ p ] ); + jQuery.removeData( elem, "fxshow" + p, true ); + // Toggle data is no longer needed + jQuery.removeData( elem, "toggle" + p, true ); + } + } + + // Execute the complete function + // in the event that the complete function throws an exception + // we must ensure it won't be called twice. #5684 + + complete = options.complete; + if ( complete ) { + + options.complete = false; + complete.call( elem ); + } + } + + return false; + + } else { + // classical easing cannot be used with an Infinity duration + if ( options.duration == Infinity ) { + this.now = t; + } else { + n = t - this.startTime; + this.state = n / options.duration; + + // Perform the easing function, defaults to swing + this.pos = jQuery.easing[ options.animatedProperties[this.prop] ]( this.state, n, 0, 1, options.duration ); + this.now = this.start + ( (this.end - this.start) * this.pos ); + } + // Perform the next step of the animation + this.update(); + } + + return true; + } +}; + +jQuery.extend( jQuery.fx, { + tick: function() { + var timer, + timers = jQuery.timers, + i = 0; + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + // Checks the timer has not already been removed + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + }, + + interval: 13, + + stop: function() { + clearInterval( timerId ); + timerId = null; + }, + + speeds: { + slow: 600, + fast: 200, + // Default speed + _default: 400 + }, + + step: { + opacity: function( fx ) { + jQuery.style( fx.elem, "opacity", fx.now ); + }, + + _default: function( fx ) { + if ( fx.elem.style && fx.elem.style[ fx.prop ] != null ) { + fx.elem.style[ fx.prop ] = fx.now + fx.unit; + } else { + fx.elem[ fx.prop ] = fx.now; + } + } + } +}); + +// Ensure props that can't be negative don't go there on undershoot easing +jQuery.each( fxAttrs.concat.apply( [], fxAttrs ), function( i, prop ) { + // exclude marginTop, marginLeft, marginBottom and marginRight from this list + if ( prop.indexOf( "margin" ) ) { + jQuery.fx.step[ prop ] = function( fx ) { + jQuery.style( fx.elem, prop, Math.max(0, fx.now) + fx.unit ); + }; + } +}); + +if ( jQuery.expr && jQuery.expr.filters ) { + jQuery.expr.filters.animated = function( elem ) { + return jQuery.grep(jQuery.timers, function( fn ) { + return elem === fn.elem; + }).length; + }; +} + +// Try to restore the default display value of an element +function defaultDisplay( nodeName ) { + + if ( !elemdisplay[ nodeName ] ) { + + var body = document.body, + elem = jQuery( "<" + nodeName + ">" ).appendTo( body ), + display = elem.css( "display" ); + elem.remove(); + + // If the simple way fails, + // get element's real default display by attaching it to a temp iframe + if ( display === "none" || display === "" ) { + // No iframe to use yet, so create it + if ( !iframe ) { + iframe = document.createElement( "iframe" ); + iframe.frameBorder = iframe.width = iframe.height = 0; + } + + body.appendChild( iframe ); + + // Create a cacheable copy of the iframe document on first call. + // IE and Opera will allow us to reuse the iframeDoc without re-writing the fake HTML + // document to it; WebKit & Firefox won't allow reusing the iframe document. + if ( !iframeDoc || !iframe.createElement ) { + iframeDoc = ( iframe.contentWindow || iframe.contentDocument ).document; + iframeDoc.write( ( jQuery.support.boxModel ? "<!doctype html>" : "" ) + "<html><body>" ); + iframeDoc.close(); + } + + elem = iframeDoc.createElement( nodeName ); + + iframeDoc.body.appendChild( elem ); + + display = jQuery.css( elem, "display" ); + body.removeChild( iframe ); + } + + // Store the correct default display + elemdisplay[ nodeName ] = display; + } + + return elemdisplay[ nodeName ]; +} + + + + +var getOffset, + rtable = /^t(?:able|d|h)$/i, + rroot = /^(?:body|html)$/i; + +if ( "getBoundingClientRect" in document.documentElement ) { + getOffset = function( elem, doc, docElem, box ) { + try { + box = elem.getBoundingClientRect(); + } catch(e) {} + + // Make sure we're not dealing with a disconnected DOM node + if ( !box || !jQuery.contains( docElem, elem ) ) { + return box ? { top: box.top, left: box.left } : { top: 0, left: 0 }; + } + + var body = doc.body, + win = getWindow( doc ), + clientTop = docElem.clientTop || body.clientTop || 0, + clientLeft = docElem.clientLeft || body.clientLeft || 0, + scrollTop = win.pageYOffset || jQuery.support.boxModel && docElem.scrollTop || body.scrollTop, + scrollLeft = win.pageXOffset || jQuery.support.boxModel && docElem.scrollLeft || body.scrollLeft, + top = box.top + scrollTop - clientTop, + left = box.left + scrollLeft - clientLeft; + + return { top: top, left: left }; + }; + +} else { + getOffset = function( elem, doc, docElem ) { + var computedStyle, + offsetParent = elem.offsetParent, + prevOffsetParent = elem, + body = doc.body, + defaultView = doc.defaultView, + prevComputedStyle = defaultView ? defaultView.getComputedStyle( elem, null ) : elem.currentStyle, + top = elem.offsetTop, + left = elem.offsetLeft; + + while ( (elem = elem.parentNode) && elem !== body && elem !== docElem ) { + if ( jQuery.support.fixedPosition && prevComputedStyle.position === "fixed" ) { + break; + } + + computedStyle = defaultView ? defaultView.getComputedStyle(elem, null) : elem.currentStyle; + top -= elem.scrollTop; + left -= elem.scrollLeft; + + if ( elem === offsetParent ) { + top += elem.offsetTop; + left += elem.offsetLeft; + + if ( jQuery.support.doesNotAddBorder && !(jQuery.support.doesAddBorderForTableAndCells && rtable.test(elem.nodeName)) ) { + top += parseFloat( computedStyle.borderTopWidth ) || 0; + left += parseFloat( computedStyle.borderLeftWidth ) || 0; + } + + prevOffsetParent = offsetParent; + offsetParent = elem.offsetParent; + } + + if ( jQuery.support.subtractsBorderForOverflowNotVisible && computedStyle.overflow !== "visible" ) { + top += parseFloat( computedStyle.borderTopWidth ) || 0; + left += parseFloat( computedStyle.borderLeftWidth ) || 0; + } + + prevComputedStyle = computedStyle; + } + + if ( prevComputedStyle.position === "relative" || prevComputedStyle.position === "static" ) { + top += body.offsetTop; + left += body.offsetLeft; + } + + if ( jQuery.support.fixedPosition && prevComputedStyle.position === "fixed" ) { + top += Math.max( docElem.scrollTop, body.scrollTop ); + left += Math.max( docElem.scrollLeft, body.scrollLeft ); + } + + return { top: top, left: left }; + }; +} + +jQuery.fn.offset = function( options ) { + if ( arguments.length ) { + return options === undefined ? + this : + this.each(function( i ) { + jQuery.offset.setOffset( this, options, i ); + }); + } + + var elem = this[0], + doc = elem && elem.ownerDocument; + + if ( !doc ) { + return null; + } + + if ( elem === doc.body ) { + return jQuery.offset.bodyOffset( elem ); + } + + return getOffset( elem, doc, doc.documentElement ); +}; + +jQuery.offset = { + + bodyOffset: function( body ) { + var top = body.offsetTop, + left = body.offsetLeft; + + if ( jQuery.support.doesNotIncludeMarginInBodyOffset ) { + top += parseFloat( jQuery.css(body, "marginTop") ) || 0; + left += parseFloat( jQuery.css(body, "marginLeft") ) || 0; + } + + return { top: top, left: left }; + }, + + setOffset: function( elem, options, i ) { + var position = jQuery.css( elem, "position" ); + + // set position first, in-case top/left are set even on static elem + if ( position === "static" ) { + elem.style.position = "relative"; + } + + var curElem = jQuery( elem ), + curOffset = curElem.offset(), + curCSSTop = jQuery.css( elem, "top" ), + curCSSLeft = jQuery.css( elem, "left" ), + calculatePosition = ( position === "absolute" || position === "fixed" ) && jQuery.inArray("auto", [curCSSTop, curCSSLeft]) > -1, + props = {}, curPosition = {}, curTop, curLeft; + + // need to be able to calculate position if either top or left is auto and position is either absolute or fixed + if ( calculatePosition ) { + curPosition = curElem.position(); + curTop = curPosition.top; + curLeft = curPosition.left; + } else { + curTop = parseFloat( curCSSTop ) || 0; + curLeft = parseFloat( curCSSLeft ) || 0; + } + + if ( jQuery.isFunction( options ) ) { + options = options.call( elem, i, curOffset ); + } + + if ( options.top != null ) { + props.top = ( options.top - curOffset.top ) + curTop; + } + if ( options.left != null ) { + props.left = ( options.left - curOffset.left ) + curLeft; + } + + if ( "using" in options ) { + options.using.call( elem, props ); + } else { + curElem.css( props ); + } + } +}; + + +jQuery.fn.extend({ + + position: function() { + if ( !this[0] ) { + return null; + } + + var elem = this[0], + + // Get *real* offsetParent + offsetParent = this.offsetParent(), + + // Get correct offsets + offset = this.offset(), + parentOffset = rroot.test(offsetParent[0].nodeName) ? { top: 0, left: 0 } : offsetParent.offset(); + + // Subtract element margins + // note: when an element has margin: auto the offsetLeft and marginLeft + // are the same in Safari causing offset.left to incorrectly be 0 + offset.top -= parseFloat( jQuery.css(elem, "marginTop") ) || 0; + offset.left -= parseFloat( jQuery.css(elem, "marginLeft") ) || 0; + + // Add offsetParent borders + parentOffset.top += parseFloat( jQuery.css(offsetParent[0], "borderTopWidth") ) || 0; + parentOffset.left += parseFloat( jQuery.css(offsetParent[0], "borderLeftWidth") ) || 0; + + // Subtract the two offsets + return { + top: offset.top - parentOffset.top, + left: offset.left - parentOffset.left + }; + }, + + offsetParent: function() { + return this.map(function() { + var offsetParent = this.offsetParent || document.body; + while ( offsetParent && (!rroot.test(offsetParent.nodeName) && jQuery.css(offsetParent, "position") === "static") ) { + offsetParent = offsetParent.offsetParent; + } + return offsetParent; + }); + } +}); + + +// Create scrollLeft and scrollTop methods +jQuery.each( {scrollLeft: "pageXOffset", scrollTop: "pageYOffset"}, function( method, prop ) { + var top = /Y/.test( prop ); + + jQuery.fn[ method ] = function( val ) { + return jQuery.access( this, function( elem, method, val ) { + var win = getWindow( elem ); + + if ( val === undefined ) { + return win ? (prop in win) ? win[ prop ] : + jQuery.support.boxModel && win.document.documentElement[ method ] || + win.document.body[ method ] : + elem[ method ]; + } + + if ( win ) { + win.scrollTo( + !top ? val : jQuery( win ).scrollLeft(), + top ? val : jQuery( win ).scrollTop() + ); + + } else { + elem[ method ] = val; + } + }, method, val, arguments.length, null ); + }; +}); + +function getWindow( elem ) { + return jQuery.isWindow( elem ) ? + elem : + elem.nodeType === 9 ? + elem.defaultView || elem.parentWindow : + false; +} + + + + +// Create width, height, innerHeight, innerWidth, outerHeight and outerWidth methods +jQuery.each( { Height: "height", Width: "width" }, function( name, type ) { + var clientProp = "client" + name, + scrollProp = "scroll" + name, + offsetProp = "offset" + name; + + // innerHeight and innerWidth + jQuery.fn[ "inner" + name ] = function() { + var elem = this[0]; + return elem ? + elem.style ? + parseFloat( jQuery.css( elem, type, "padding" ) ) : + this[ type ]() : + null; + }; + + // outerHeight and outerWidth + jQuery.fn[ "outer" + name ] = function( margin ) { + var elem = this[0]; + return elem ? + elem.style ? + parseFloat( jQuery.css( elem, type, margin ? "margin" : "border" ) ) : + this[ type ]() : + null; + }; + + jQuery.fn[ type ] = function( value ) { + return jQuery.access( this, function( elem, type, value ) { + var doc, docElemProp, orig, ret; + + if ( jQuery.isWindow( elem ) ) { + // 3rd condition allows Nokia support, as it supports the docElem prop but not CSS1Compat + doc = elem.document; + docElemProp = doc.documentElement[ clientProp ]; + return jQuery.support.boxModel && docElemProp || + doc.body && doc.body[ clientProp ] || docElemProp; + } + + // Get document width or height + if ( elem.nodeType === 9 ) { + // Either scroll[Width/Height] or offset[Width/Height], whichever is greater + doc = elem.documentElement; + + // when a window > document, IE6 reports a offset[Width/Height] > client[Width/Height] + // so we can't use max, as it'll choose the incorrect offset[Width/Height] + // instead we use the correct client[Width/Height] + // support:IE6 + if ( doc[ clientProp ] >= doc[ scrollProp ] ) { + return doc[ clientProp ]; + } + + return Math.max( + elem.body[ scrollProp ], doc[ scrollProp ], + elem.body[ offsetProp ], doc[ offsetProp ] + ); + } + + // Get width or height on the element + if ( value === undefined ) { + orig = jQuery.css( elem, type ); + ret = parseFloat( orig ); + return jQuery.isNumeric( ret ) ? ret : orig; + } + + // Set the width or height on the element + jQuery( elem ).css( type, value ); + }, type, value, arguments.length, null ); + }; +}); + + + + +// Expose jQuery to the global object +window.jQuerySvg = window.$vg = jQuery; + +// Expose jQuery as an AMD module, but only for AMD loaders that +// understand the issues with loading multiple versions of jQuery +// in a page that all might call define(). The loader will indicate +// they have special allowances for multiple jQuery versions by +// specifying define.amd.jQuery = true. Register as a named module, +// since jQuery can be concatenated with other files that may use define, +// but not use a proper concatenation script that understands anonymous +// AMD modules. A named AMD is safest and most robust way to register. +// Lowercase jquery is used because AMD module names are derived from +// file names, and jQuery is normally delivered in a lowercase file name. +// Do this after creating the global so that if an AMD module wants to call +// noConflict to hide this version of jQuery, it will work. +if ( typeof define === "function" && define.amd && define.amd.jQuery ) { + define( "jquery", [], function () { return jQuery; } ); +} + + + +})( window ); diff --git a/plugins/imptcp/imptcp.c b/plugins/imptcp/imptcp.c index 906521dd..55e6f947 100644 --- a/plugins/imptcp/imptcp.c +++ b/plugins/imptcp/imptcp.c @@ -10,7 +10,7 @@ * * File begun on 2010-08-10 by RGerhards * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -50,6 +50,8 @@ #include <sys/socket.h> #include <sys/epoll.h> #include <netinet/tcp.h> +#include <stdint.h> +#include <zlib.h> #if HAVE_FCNTL_H #include <fcntl.h> #endif @@ -93,6 +95,11 @@ static void * wrkr(void *myself); #define DFLT_wrkrMax 2 +#define COMPRESS_NEVER 0 +#define COMPRESS_SINGLE_MSG 1 /* old, single-message compression */ +/* all other settings are for stream-compression */ +#define COMPRESS_STREAM_ALWAYS 2 + /* config settings */ typedef struct configSettings_s { int bKeepAlive; /* support keep-alive packets */ @@ -117,11 +124,13 @@ struct instanceConf_s { int bEmitMsgOnClose; int bSuppOctetFram; /* support octet-counted framing? */ int iAddtlFrameDelim; + uint8_t compressionMode; uchar *pszBindPort; /* port to bind to */ uchar *pszBindAddr; /* IP to bind socket to */ uchar *pszBindRuleset; /* name of ruleset to bind to */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + uchar *dfltTZ; int ratelimitInterval; int ratelimitBurst; struct instanceConf_s *next; @@ -154,8 +163,10 @@ static struct cnfparamdescr inppdescr[] = { { "address", eCmdHdlrString, 0 }, { "name", eCmdHdlrString, 0 }, { "ruleset", eCmdHdlrString, 0 }, + { "defaulttz", eCmdHdlrString, 0 }, { "supportoctetcountedframing", eCmdHdlrBinary, 0 }, { "notifyonconnectionclose", eCmdHdlrBinary, 0 }, + { "compression.mode", eCmdHdlrGetWord, 0 }, { "keepalive", eCmdHdlrBinary, 0 }, { "keepalive.probes", eCmdHdlrInt, 0 }, { "keepalive.time", eCmdHdlrInt, 0 }, @@ -191,7 +202,9 @@ struct ptcpsrv_s { int iKeepAliveIntvl; int iKeepAliveProbes; int iKeepAliveTime; + uint8_t compressionMode; uchar *pszInputName; + uchar *dfltTZ; prop_t *pInputName; /* InputName in (fast to process) property format */ ruleset_t *pRuleset; ptcplstn_t *pLstn; /* root of our listeners */ @@ -207,11 +220,13 @@ struct ptcpsrv_s { * includes support for doubly-linked list. */ struct ptcpsess_s { -// ptcpsrv_t *pSrv; /* our server TODO: check remove! */ ptcplstn_t *pLstn; /* our listener */ ptcpsess_t *prev, *next; int sock; epolld_t *epd; + sbool bzInitDone; /* did we do an init of zstrm already? */ + z_stream zstrm; /* zip stream to use for tcp compression */ + uint8_t compressionMode; //--- from tcps_sess.h int iMsg; /* index of next char to store in msg */ int bAtStrtOfFram; /* are we at the very beginning of a new frame? */ @@ -239,6 +254,8 @@ struct ptcplstn_s { sbool bSuppOctetFram; epolld_t *epd; statsobj_t *stats; /* listener stats */ + intctr_t rcvdBytes; + intctr_t rcvdDecompressed; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) }; @@ -680,6 +697,8 @@ doSubmitMsg(ptcpsess_t *pThis, struct syslogTime *stTime, time_t ttGenTime, mult MsgSetRawMsg(pMsg, (char*)pThis->pMsg, pThis->iMsg); MsgSetInputName(pMsg, pSrv->pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); + if(pSrv->dfltTZ != NULL) + MsgSetDfltTZ(pMsg, (char*) pSrv->dfltTZ); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; MsgSetRcvFrom(pMsg, pThis->peerName); CHKiRet(MsgSetRcvFromIP(pMsg, pThis->peerIP)); @@ -806,19 +825,18 @@ processDataRcvd(ptcpsess_t *pThis, char c, struct syslogTime *stTime, time_t ttG * EXTRACT from tcps_sess.c */ static rsRetVal -DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) +DataRcvdUncompressed(ptcpsess_t *pThis, char *pData, size_t iLen, struct syslogTime *stTime, time_t ttGenTime) { multi_submit_t multiSub; msg_t *pMsgs[CONF_NUM_MULTISUB]; - struct syslogTime stTime; - time_t ttGenTime; char *pEnd; DEFiRet; assert(pData != NULL); assert(iLen > 0); - datetime.getCurrTime(&stTime, &ttGenTime); + if(ttGenTime == 0) + datetime.getCurrTime(stTime, &ttGenTime); multiSub.ppMsgs = pMsgs; multiSub.maxElem = CONF_NUM_MULTISUB; multiSub.nElem = 0; @@ -827,7 +845,7 @@ DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) pEnd = pData + iLen; /* this is one off, which is intensional */ while(pData < pEnd) { - CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); + CHKiRet(processDataRcvd(pThis, *pData++, stTime, ttGenTime, &multiSub)); } iRet = multiSubmitFlush(&multiSub); @@ -836,6 +854,71 @@ finalize_it: RETiRet; } +static rsRetVal +DataRcvdCompressed(ptcpsess_t *pThis, char *buf, size_t len) +{ + struct syslogTime stTime; + time_t ttGenTime; + int zRet; /* zlib return state */ + unsigned outavail; + uchar zipBuf[64*1024]; // TODO: alloc on heap, and much larger (512KiB? batch size!) + DEFiRet; + // TODO: can we do stats counters? Even if they are not 100% correct under all cases, + // by simply updating the input and output sizes? + uint64_t outtotal; + + datetime.getCurrTime(&stTime, &ttGenTime); + outtotal = 0; + + if(!pThis->bzInitDone) { + /* allocate deflate state */ + pThis->zstrm.zalloc = Z_NULL; + pThis->zstrm.zfree = Z_NULL; + pThis->zstrm.opaque = Z_NULL; + zRet = inflateInit(&pThis->zstrm); + if(zRet != Z_OK) { + DBGPRINTF("imptcp: error %d returned from zlib/inflateInit()\n", zRet); + ABORT_FINALIZE(RS_RET_ZLIB_ERR); + } + pThis->bzInitDone = RSTRUE; + } + + pThis->zstrm.next_in = (Bytef*) buf; + pThis->zstrm.avail_in = len; + /* run inflate() on buffer until everything has been uncompressed */ + do { + DBGPRINTF("imptcp: in inflate() loop, avail_in %d, total_in %ld\n", pThis->zstrm.avail_in, pThis->zstrm.total_in); + pThis->zstrm.avail_out = sizeof(zipBuf); + pThis->zstrm.next_out = zipBuf; + zRet = inflate(&pThis->zstrm, Z_SYNC_FLUSH); /* no bad return value */ + //zRet = inflate(&pThis->zstrm, Z_NO_FLUSH); /* no bad return value */ + DBGPRINTF("after inflate, ret %d, avail_out %d\n", zRet, pThis->zstrm.avail_out); + outavail = sizeof(zipBuf) - pThis->zstrm.avail_out; + if(outavail != 0) { + outtotal += outavail; + pThis->pLstn->rcvdDecompressed += outavail; + CHKiRet(DataRcvdUncompressed(pThis, (char*)zipBuf, outavail, &stTime, ttGenTime)); + } + } while (pThis->zstrm.avail_out == 0); + + dbgprintf("end of DataRcvCompress, sizes: in %lld, out %llu\n", (long long) len, outtotal); +finalize_it: + RETiRet; +} + +static rsRetVal +DataRcvd(ptcpsess_t *pThis, char *pData, size_t iLen) +{ + struct syslogTime stTime; + DEFiRet; + pThis->pLstn->rcvdBytes += iLen; + if(pThis->compressionMode >= COMPRESS_STREAM_ALWAYS) + iRet = DataRcvdCompressed(pThis, pData, iLen); + else + iRet = DataRcvdUncompressed(pThis, pData, iLen, &stTime, 0); + RETiRet; +} + /****************************************** --END-- TCP SUPPORT FUNCTIONS ***********************************/ @@ -935,7 +1018,15 @@ addLstn(ptcpsrv_t *pSrv, int sock, int isIPv6) CHKiRet(statsobj.SetName(pLstn->stats, statname)); STATSCOUNTER_INIT(pLstn->ctrSubmit, pLstn->mutCtrSubmit); CHKiRet(statsobj.AddCounter(pLstn->stats, UCHAR_CONSTANT("submitted"), - ctrType_IntCtr, &(pLstn->ctrSubmit))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pLstn->ctrSubmit))); + /* the following counters are not protected by mutexes; we accept + * that they may not be 100% correct */ + pLstn->rcvdBytes = 0, + pLstn->rcvdDecompressed = 0; + CHKiRet(statsobj.AddCounter(pLstn->stats, UCHAR_CONSTANT("bytes.received"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pLstn->rcvdBytes))); + CHKiRet(statsobj.AddCounter(pLstn->stats, UCHAR_CONSTANT("bytes.decompressed"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pLstn->rcvdDecompressed))); CHKiRet(statsobj.ConstructFinalize(pLstn->stats)); /* add to start of server's listener list */ @@ -968,9 +1059,11 @@ addSess(ptcplstn_t *pLstn, int sock, prop_t *peerName, prop_t *peerIP) pSess->bSuppOctetFram = pLstn->bSuppOctetFram; pSess->inputState = eAtStrtFram; pSess->iMsg = 0; + pSess->bzInitDone = 0; pSess->bAtStrtOfFram = 1; pSess->peerName = peerName; pSess->peerIP = peerIP; + pSess->compressionMode = pLstn->pSrv->compressionMode; /* add to start of server's listener list */ pSess->prev = NULL; @@ -988,6 +1081,45 @@ finalize_it: } +/* finish zlib buffer, to be called before closing the session. + */ +static rsRetVal +doZipFinish(ptcpsess_t *pSess) +{ + int zRet; /* zlib return state */ + DEFiRet; + unsigned outavail; + struct syslogTime stTime; + uchar zipBuf[32*1024]; // TODO: use "global" one from pSess + + if(!pSess->bzInitDone) + goto done; + + pSess->zstrm.avail_in = 0; + /* run inflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("doZipFinish: in inflate() loop, avail_in %d, total_in %ld\n", pSess->zstrm.avail_in, pSess->zstrm.total_in); + pSess->zstrm.avail_out = sizeof(zipBuf); + pSess->zstrm.next_out = zipBuf; + zRet = inflate(&pSess->zstrm, Z_FINISH); /* no bad return value */ + DBGPRINTF("after inflate, ret %d, avail_out %d\n", zRet, pSess->zstrm.avail_out); + outavail = sizeof(zipBuf) - pSess->zstrm.avail_out; + if(outavail != 0) { + pSess->pLstn->rcvdDecompressed += outavail; + CHKiRet(DataRcvdUncompressed(pSess, (char*)zipBuf, outavail, &stTime, 0)); // TODO: query time! + } + } while (pSess->zstrm.avail_out == 0); + +finalize_it: + zRet = inflateEnd(&pSess->zstrm); + if(zRet != Z_OK) { + DBGPRINTF("imptcp: error %d returned from zlib/inflateEnd()\n", zRet); + } + + pSess->bzInitDone = 0; +done: RETiRet; +} + /* close/remove a session * NOTE: we must first remove the fd from the epoll set and then close it -- else we * get an error "bad file descriptor" from epoll. @@ -998,6 +1130,9 @@ closeSess(ptcpsess_t *pSess) int sock; DEFiRet; + if(pSess->compressionMode >= COMPRESS_STREAM_ALWAYS) + doZipFinish(pSess); + sock = pSess->sock; CHKiRet(removeEPollSock(sock, pSess->epd)); close(sock); @@ -1044,10 +1179,12 @@ createInstance(instanceConf_t **pinst) inst->iKeepAliveProbes = 0; inst->iKeepAliveTime = 0; inst->bEmitMsgOnClose = 0; + inst->dfltTZ = NULL; inst->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; inst->pBindRuleset = NULL; inst->ratelimitBurst = 10000; /* arbitrary high limit */ inst->ratelimitInterval = 0; /* off */ + inst->compressionMode = COMPRESS_SINGLE_MSG; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -1127,6 +1264,8 @@ addListner(modConfData_t __attribute__((unused)) *modConf, instanceConf_t *inst) pSrv->iKeepAliveProbes = inst->iKeepAliveProbes; pSrv->iKeepAliveTime = inst->iKeepAliveTime; pSrv->bEmitMsgOnClose = inst->bEmitMsgOnClose; + pSrv->compressionMode = inst->compressionMode; + pSrv->dfltTZ = inst->dfltTZ; CHKiRet(ratelimitNew(&pSrv->ratelimiter, "imtcp", (char*)inst->pszBindPort)); ratelimitSetLinuxLike(pSrv->ratelimiter, inst->ratelimitInterval, inst->ratelimitBurst); ratelimitSetThreadSafe(pSrv->ratelimiter); @@ -1269,6 +1408,10 @@ sessActivity(ptcpsess_t *pSess) { int lenRcv; int lenBuf; + uchar *peerName; + int lenPeer; + int remsock = 0; /* init just to keep compiler happy... :-( */ + sbool bEmitOnClose = 0; char rcvBuf[128*1024]; DEFiRet; @@ -1285,13 +1428,15 @@ sessActivity(ptcpsess_t *pSess) } else if (lenRcv == 0) { /* session was closed, do clean-up */ if(pSess->pLstn->pSrv->bEmitMsgOnClose) { - uchar *peerName; - int lenPeer; - prop.GetString(pSess->peerName, &peerName, &lenPeer); - errmsg.LogError(0, RS_RET_PEER_CLOSED_CONN, "imptcp session %d closed by remote peer %s.\n", - pSess->sock, peerName); + prop.GetString(pSess->peerName, &peerName, &lenPeer), + remsock = pSess->sock; + bEmitOnClose = 1; + } + CHKiRet(closeSess(pSess)); /* close may emit more messages in strmzip mode! */ + if(bEmitOnClose) { + errmsg.LogError(0, RS_RET_PEER_CLOSED_CONN, "imptcp session %d closed by " + "remote peer %s.\n", remsock, peerName); } - CHKiRet(closeSess(pSess)); break; } else { if(errno == EAGAIN || errno == EWOULDBLOCK) @@ -1415,6 +1560,7 @@ wrkr(void *myself) BEGINnewInpInst struct cnfparamvals *pvals; instanceConf_t *inst; + char *cstr; int i; CODESTARTnewInpInst DBGPRINTF("newInpInst (imptcp)\n"); @@ -1443,6 +1589,19 @@ CODESTARTnewInpInst inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "supportoctetcountedframing")) { inst->bSuppOctetFram = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "compression.mode")) { + cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + if(!strcasecmp(cstr, "stream:always")) { + inst->compressionMode = COMPRESS_STREAM_ALWAYS; + } else if(!strcasecmp(cstr, "none")) { + inst->compressionMode = COMPRESS_NEVER; + } else { + errmsg.LogError(0, RS_RET_PARAM_ERROR, "omfwd: invalid value for 'compression.mode' " + "parameter (given is '%s')", cstr); + free(cstr); + ABORT_FINALIZE(RS_RET_PARAM_ERROR); + } + free(cstr); } else if(!strcmp(inppblk.descr[i].name, "keepalive")) { inst->bKeepAlive = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "keepalive.probes")) { @@ -1455,6 +1614,8 @@ CODESTARTnewInpInst inst->iAddtlFrameDelim = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "notifyonconnectionclose")) { inst->bEmitMsgOnClose = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "defaulttz")) { + inst->dfltTZ = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ratelimit.burst")) { inst->ratelimitBurst = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { @@ -1609,6 +1770,7 @@ CODESTARTfreeCnf free(inst->pszBindAddr); free(inst->pszBindRuleset); free(inst->pszInputName); + free(inst->dfltTZ); del = inst; inst = inst->next; free(del); @@ -1658,7 +1820,9 @@ shutdownSrv(ptcpsrv_t *pSrv) /* now unlink listner */ lstnDel = pLstn; pLstn = pLstn->next; - DBGPRINTF("imptcp shutdown listen socket %d\n", lstnDel->sock); + DBGPRINTF("imptcp shutdown listen socket %d (rcvd %lld bytes, " + "decompressed %lld)\n", lstnDel->sock, lstnDel->rcvdBytes, + lstnDel->rcvdDecompressed); free(lstnDel->epd); free(lstnDel); } diff --git a/plugins/imrelp/imrelp.c b/plugins/imrelp/imrelp.c index 5e0ae552..599d7559 100644 --- a/plugins/imrelp/imrelp.c +++ b/plugins/imrelp/imrelp.c @@ -4,7 +4,7 @@ * * File begun on 2008-03-13 by RGerhards * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -47,6 +47,7 @@ #include "prop.h" #include "ruleset.h" #include "glbl.h" +#include "statsobj.h" MODULE_TYPE_INPUT MODULE_TYPE_NOKEEP @@ -59,6 +60,7 @@ DEFobjCurrIf(prop) DEFobjCurrIf(errmsg) DEFobjCurrIf(ruleset) DEFobjCurrIf(glbl) +DEFobjCurrIf(statsobj) /* forward definitions */ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal); @@ -74,7 +76,31 @@ static struct configSettings_s { struct instanceConf_s { uchar *pszBindPort; /* port to bind to */ + sbool bEnableTLS; + sbool bEnableTLSZip; + int dhBits; + uchar *pristring; /* GnuTLS priority string (NULL if not to be provided) */ + uchar *authmode; /* TLS auth mode */ + uchar *caCertFile; + uchar *myCertFile; + uchar *myPrivKeyFile; + struct { + int nmemb; + uchar **name; + } permittedPeers; + struct instanceConf_s *next; + /* with librelp, this module does not have any own specific session + * or listener active data item. As a "work-around", we keep some + * data items inside the configuration object. To keep things + * decently clean, we put them all into their dedicated struct. So + * it is easy to judge what is actual configuration and what is + * dynamic runtime data. -- rgerhards, 2013-06-18 + */ + struct { + statsobj_t *stats; /* listener stats */ + STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) + } data; }; @@ -88,9 +114,28 @@ struct modConfData_s { static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ +/* module-global parameters */ +static struct cnfparamdescr modpdescr[] = { + { "ruleset", eCmdHdlrGetWord, 0 }, +}; +static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr + }; + /* input instance parameters */ static struct cnfparamdescr inppdescr[] = { - { "port", eCmdHdlrString, CNFPARAM_REQUIRED } + { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, + { "tls", eCmdHdlrBinary, 0 }, + { "tls.permittedpeer", eCmdHdlrArray, 0 }, + { "tls.authmode", eCmdHdlrString, 0 }, + { "tls.dhbits", eCmdHdlrInt, 0 }, + { "tls.prioritystring", eCmdHdlrString, 0 }, + { "tls.cacert", eCmdHdlrString, 0 }, + { "tls.mycert", eCmdHdlrString, 0 }, + { "tls.myprivkey", eCmdHdlrString, 0 }, + { "tls.compression", eCmdHdlrBinary, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -102,6 +147,30 @@ static struct cnfparamblk inppblk = /* ------------------------------ callbacks ------------------------------ */ +static void +onErr(void *pUsr, char *objinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) +{ + instanceConf_t *inst = (instanceConf_t*) pUsr; + errmsg.LogError(0, RS_RET_RELP_AUTH_FAIL, "imrelp[%s]: error '%s', object " + " '%s' - input may not work as intended", + inst->pszBindPort, errmesg, objinfo); +} + +static void +onGenericErr(char *objinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) +{ + errmsg.LogError(0, RS_RET_RELP_ERR, "imrelp: librelp error '%s', object " + " '%s' - input may not work as intended", errmesg, objinfo); +} + +static void +onAuthErr(void *pUsr, char *authinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) +{ + instanceConf_t *inst = (instanceConf_t*) pUsr; + errmsg.LogError(0, RS_RET_RELP_AUTH_FAIL, "imrelp[%s]: authentication error '%s', peer " + "is '%s'", inst->pszBindPort, errmesg, authinfo); +} + /* callback for receiving syslog messages. This function is invoked from the * RELP engine when a syslog message arrived. It must return a relpRetVal, * with anything else but RELP_RET_OK terminating the relp session. Please note @@ -113,10 +182,11 @@ static struct cnfparamblk inppblk = * we will only see the hostname (twice). -- rgerhards, 2009-10-14 */ static relpRetVal -onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *msg, size_t lenMsg) +onSyslogRcv(void *pUsr, uchar *pHostname, uchar *pIP, uchar *msg, size_t lenMsg) { prop_t *pProp = NULL; msg_t *pMsg; + instanceConf_t *inst = (instanceConf_t*) pUsr; DEFiRet; CHKiRet(msgConstruct(&pMsg)); @@ -134,6 +204,7 @@ onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *msg, size_t lenMsg) CHKiRet(MsgSetRcvFromIPStr(pMsg, pIP, ustrlen(pIP), &pProp)); CHKiRet(prop.Destruct(&pProp)); CHKiRet(submitMsg2(pMsg)); + STATSCOUNTER_INC(inst->data.ctrSubmit, inst->data.mutCtrSubmit); finalize_it: @@ -155,6 +226,15 @@ createInstance(instanceConf_t **pinst) inst->next = NULL; inst->pszBindPort = NULL; + inst->bEnableTLS = 0; + inst->bEnableTLSZip = 0; + inst->dhBits = 0; + inst->pristring = NULL; + inst->authmode = NULL; + inst->permittedPeers.nmemb = 0; + inst->caCertFile = NULL; + inst->myCertFile = NULL; + inst->myPrivKeyFile = NULL; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -179,7 +259,7 @@ std_checkRuleset_genErrMsg(modConfData_t *modConf, __attribute__((unused)) insta } -/* This function is called when a new listener instace shall be added to +/* This function is called when a new listener instance shall be added to * the current config object via the legacy config system. It just shuffles * all parameters to the listener in-memory instance. * rgerhards, 2011-05-04 @@ -204,19 +284,63 @@ finalize_it: static rsRetVal addListner(modConfData_t __attribute__((unused)) *modConf, instanceConf_t *inst) { + relpSrv_t *pSrv; + uchar statname[64]; + int i; DEFiRet; if(pRelpEngine == NULL) { CHKiRet(relpEngineConstruct(&pRelpEngine)); CHKiRet(relpEngineSetDbgprint(pRelpEngine, dbgprintf)); CHKiRet(relpEngineSetFamily(pRelpEngine, glbl.GetDefPFFamily())); CHKiRet(relpEngineSetEnableCmd(pRelpEngine, (uchar*) "syslog", eRelpCmdState_Required)); - CHKiRet(relpEngineSetSyslogRcv(pRelpEngine, onSyslogRcv)); + CHKiRet(relpEngineSetSyslogRcv2(pRelpEngine, onSyslogRcv)); + CHKiRet(relpEngineSetOnErr(pRelpEngine, onErr)); + CHKiRet(relpEngineSetOnGenericErr(pRelpEngine, onGenericErr)); + CHKiRet(relpEngineSetOnAuthErr(pRelpEngine, onAuthErr)); if (!glbl.GetDisableDNS()) { CHKiRet(relpEngineSetDnsLookupMode(pRelpEngine, 1)); } } - CHKiRet(relpEngineAddListner(pRelpEngine, inst->pszBindPort)); + CHKiRet(relpEngineListnerConstruct(pRelpEngine, &pSrv)); + CHKiRet(relpSrvSetLstnPort(pSrv, inst->pszBindPort)); + /* support statistics gathering */ + CHKiRet(statsobj.Construct(&(inst->data.stats))); + snprintf((char*)statname, sizeof(statname), "imrelp(%s)", + inst->pszBindPort); + statname[sizeof(statname)-1] = '\0'; /* just to be on the save side... */ + CHKiRet(statsobj.SetName(inst->data.stats, statname)); + STATSCOUNTER_INIT(inst->data.ctrSubmit, inst->data.mutCtrSubmit); + CHKiRet(statsobj.AddCounter(inst->data.stats, UCHAR_CONSTANT("submitted"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(inst->data.ctrSubmit))); + CHKiRet(statsobj.ConstructFinalize(inst->data.stats)); + /* end stats counters */ + relpSrvSetUsrPtr(pSrv, inst); + if(inst->bEnableTLS) { + relpSrvEnableTLS(pSrv); + if(inst->bEnableTLSZip) { + relpSrvEnableTLSZip(pSrv); + } + if(inst->dhBits) { + relpSrvSetDHBits(pSrv, inst->dhBits); + } + relpSrvSetGnuTLSPriString(pSrv, (char*)inst->pristring); + if(relpSrvSetAuthMode(pSrv, (char*)inst->authmode) != RELP_RET_OK) { + errmsg.LogError(0, RS_RET_RELP_ERR, + "imrelp: invalid auth mode '%s'\n", inst->authmode); + ABORT_FINALIZE(RS_RET_RELP_ERR); + } + if(relpSrvSetCACert(pSrv, (char*) inst->caCertFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpSrvSetOwnCert(pSrv, (char*) inst->myCertFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpSrvSetPrivKey(pSrv, (char*) inst->myPrivKeyFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + for(i = 0 ; i < inst->permittedPeers.nmemb ; ++i) { + relpSrvAddPermittedPeer(pSrv, (char*)inst->permittedPeers.name[i]); + } + } + CHKiRet(relpEngineListnerConstructFinalize(pRelpEngine, pSrv)); finalize_it: RETiRet; @@ -226,7 +350,7 @@ finalize_it: BEGINnewInpInst struct cnfparamvals *pvals; instanceConf_t *inst; - int i; + int i,j; CODESTARTnewInpInst DBGPRINTF("newInpInst (imrelp)\n"); @@ -249,6 +373,29 @@ CODESTARTnewInpInst continue; if(!strcmp(inppblk.descr[i].name, "port")) { inst->pszBindPort = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls")) { + inst->bEnableTLS = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "tls.dhbits")) { + inst->dhBits = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "tls.prioritystring")) { + inst->pristring = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls.authmode")) { + inst->authmode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls.compression")) { + inst->bEnableTLSZip = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "tls.cacert")) { + inst->caCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls.mycert")) { + inst->myCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls.myprivkey")) { + inst->myPrivKeyFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "tls.permittedpeer")) { + inst->permittedPeers.nmemb = pvals[i].val.d.ar->nmemb; + CHKmalloc(inst->permittedPeers.name = + malloc(sizeof(uchar*) * inst->permittedPeers.nmemb)); + for(j = 0 ; j < pvals[i].val.d.ar->nmemb ; ++j) { + inst->permittedPeers.name[j] = (uchar*)es_str2cstr(pvals[i].val.d.ar->arr[j], NULL); + } } else { dbgprintf("imrelp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); @@ -264,19 +411,58 @@ BEGINbeginCnfLoad CODESTARTbeginCnfLoad loadModConf = pModConf; pModConf->pConf = pConf; + pModConf->pszBindRuleset = NULL; + pModConf->pBindRuleset = NULL; /* init legacy config variables */ cs.pszBindRuleset = NULL; ENDbeginCnfLoad +BEGINsetModCnf + struct cnfparamvals *pvals = NULL; + int i; +CODESTARTsetModCnf + pvals = nvlstGetParams(lst, &modpblk, NULL); + if(pvals == NULL) { + errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "error processing module " + "config parameters [module(...)]"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(Debug) { + dbgprintf("module (global) param blk for imrelp:\n"); + cnfparamsPrint(&modpblk, pvals); + } + + for(i = 0 ; i < modpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(modpblk.descr[i].name, "ruleset")) { + loadModConf->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("imrelp: program error, non-handled " + "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); + } + } +finalize_it: + if(pvals != NULL) + cnfparamvalsDestruct(pvals, &modpblk); +ENDsetModCnf + BEGINendCnfLoad CODESTARTendCnfLoad - if((cs.pszBindRuleset == NULL) || (cs.pszBindRuleset[0] == '\0')) { - loadModConf->pszBindRuleset = NULL; + if(loadModConf->pszBindRuleset == NULL) { + if((cs.pszBindRuleset == NULL) || (cs.pszBindRuleset[0] == '\0')) { + loadModConf->pszBindRuleset = NULL; + } else { + CHKmalloc(loadModConf->pszBindRuleset = ustrdup(cs.pszBindRuleset)); + } } else { - CHKmalloc(loadModConf->pszBindRuleset = ustrdup(cs.pszBindRuleset)); + if((cs.pszBindRuleset != NULL) && (cs.pszBindRuleset[0] != '\0')) { + errmsg.LogError(0, RS_RET_DUP_PARAM, "imrelp: warning: ruleset " + "set via legacy directive ignored"); + } } - loadModConf->pBindRuleset = NULL; finalize_it: free(cs.pszBindRuleset); loadModConf = NULL; /* done loading */ @@ -293,6 +479,7 @@ CODESTARTcheckCnf if(pModConf->pszBindRuleset == NULL) { pModConf->pBindRuleset = NULL; } else { + DBGPRINTF("imrelp: using ruleset '%s'\n", pModConf->pszBindRuleset); localRet = ruleset.GetRuleset(pModConf->pConf, &pRuleset, pModConf->pszBindRuleset); if(localRet == RS_RET_NOT_FOUND) { std_checkRuleset_genErrMsg(pModConf, NULL); @@ -323,13 +510,21 @@ ENDactivateCnf BEGINfreeCnf instanceConf_t *inst, *del; + int i; CODESTARTfreeCnf for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindPort); + free(inst->pristring); + free(inst->authmode); + statsobj.Destruct(&(inst->data.stats)); + for(i = 0 ; i < inst->permittedPeers.nmemb ; ++i) { + free(inst->permittedPeers.name[i]); + } del = inst; inst = inst->next; free(del); } + free(pModConf->pszBindRuleset); ENDfreeCnf /* This is used to terminate the plugin. Note that the signal handler blocks @@ -390,6 +585,7 @@ CODESTARTmodExit prop.Destruct(&pInputName); /* release objects we used */ + objRelease(statsobj, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); objRelease(prop, CORE_COMPONENT); @@ -420,6 +616,7 @@ CODEqueryEtryPt_STD_IMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES ENDqueryEtryPt @@ -435,6 +632,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); CHKiRet(objUse(ruleset, CORE_COMPONENT)); + CHKiRet(objUse(statsobj, CORE_COMPONENT)); /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputrelpserverbindruleset", 0, eCmdHdlrGetWord, diff --git a/plugins/imtcp/imtcp.c b/plugins/imtcp/imtcp.c index e10a8ba3..51697574 100644 --- a/plugins/imtcp/imtcp.c +++ b/plugins/imtcp/imtcp.c @@ -4,7 +4,7 @@ * File begun on 2007-12-21 by RGerhards (extracted from syslogd.c, * which at the time of the rsyslog fork was BSD-licensed) * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -105,6 +105,7 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ uchar *pszInputName; /* value for inputname property, NULL is OK and handled by core engine */ + uchar *dfltTZ; int ratelimitInterval; int ratelimitBurst; int bSuppOctetFram; @@ -124,6 +125,7 @@ struct modConfData_s { sbool bUseFlowControl; /* use flow control, what means indicate ourselfs a "light delayable" */ sbool bKeepAlive; sbool bEmitMsgOnClose; /* emit an informational message on close by remote peer */ + uchar *pszStrmDrvrName; /* stream driver to use */ uchar *pszStrmDrvrAuthMode; /* authentication mode to use */ struct cnfarray *permittedPeers; sbool configSetViaV2Method; @@ -144,6 +146,7 @@ static struct cnfparamdescr modpdescr[] = { { "maxlisteners", eCmdHdlrPositiveInt, 0 }, { "streamdriver.mode", eCmdHdlrPositiveInt, 0 }, { "streamdriver.authmode", eCmdHdlrString, 0 }, + { "streamdriver.name", eCmdHdlrString, 0 }, { "permittedpeer", eCmdHdlrArray, 0 }, { "keepalive", eCmdHdlrBinary, 0 } }; @@ -157,6 +160,7 @@ static struct cnfparamblk modpblk = static struct cnfparamdescr inppdescr[] = { { "port", eCmdHdlrString, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ { "name", eCmdHdlrString, 0 }, + { "defaulttz", eCmdHdlrString, 0 }, { "ruleset", eCmdHdlrString, 0 }, { "supportOctetCountedFraming", eCmdHdlrBinary, 0 }, { "ratelimit.interval", eCmdHdlrInt, 0 }, @@ -255,6 +259,7 @@ createInstance(instanceConf_t **pinst) inst->next = NULL; inst->pszBindRuleset = NULL; inst->pszInputName = NULL; + inst->dfltTZ = NULL; inst->bSuppOctetFram = 1; inst->ratelimitInterval = 0; inst->ratelimitBurst = 10000; @@ -328,6 +333,9 @@ addListner(modConfData_t *modConf, instanceConf_t *inst) CHKiRet(tcpsrv.SetbDisableLFDelim(pOurTcpsrv, modConf->bDisableLFDelim)); CHKiRet(tcpsrv.SetNotificationOnRemoteClose(pOurTcpsrv, modConf->bEmitMsgOnClose)); /* now set optional params, but only if they were actually configured */ + if(modConf->pszStrmDrvrName != NULL) { + CHKiRet(tcpsrv.SetDrvrName(pOurTcpsrv, modConf->pszStrmDrvrName)); + } if(modConf->pszStrmDrvrAuthMode != NULL) { CHKiRet(tcpsrv.SetDrvrAuthMode(pOurTcpsrv, modConf->pszStrmDrvrAuthMode)); } @@ -341,6 +349,7 @@ addListner(modConfData_t *modConf, instanceConf_t *inst) CHKiRet(tcpsrv.SetRuleset(pOurTcpsrv, inst->pBindRuleset)); CHKiRet(tcpsrv.SetInputName(pOurTcpsrv, inst->pszInputName == NULL ? UCHAR_CONSTANT("imtcp") : inst->pszInputName)); + CHKiRet(tcpsrv.SetDfltTZ(pOurTcpsrv, (inst->dfltTZ == NULL) ? (uchar*)"" : inst->dfltTZ)); CHKiRet(tcpsrv.SetLinuxLikeRatelimiters(pOurTcpsrv, inst->ratelimitInterval, inst->ratelimitBurst)); tcpsrv.configureTCPListen(pOurTcpsrv, inst->pszBindPort, inst->bSuppOctetFram); @@ -380,6 +389,8 @@ CODESTARTnewInpInst inst->pszBindPort = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "name")) { inst->pszInputName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(inppblk.descr[i].name, "defaulttz")) { + inst->dfltTZ = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { inst->pszBindRuleset = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "supportOctetCountedFraming")) { @@ -413,6 +424,7 @@ CODESTARTbeginCnfLoad loadModConf->bEmitMsgOnClose = 0; loadModConf->iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; loadModConf->bDisableLFDelim = 0; + loadModConf->pszStrmDrvrName = NULL; loadModConf->pszStrmDrvrAuthMode = NULL; loadModConf->permittedPeers = NULL; loadModConf->configSetViaV2Method = 0; @@ -463,6 +475,8 @@ CODESTARTsetModCnf loadModConf->iStrmDrvrMode = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "streamdriver.authmode")) { loadModConf->pszStrmDrvrAuthMode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(modpblk.descr[i].name, "streamdriver.name")) { + loadModConf->pszStrmDrvrName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(modpblk.descr[i].name, "permittedpeer")) { loadModConf->permittedPeers = cnfarrayDup(pvals[i].val.d.ar); } else { @@ -563,6 +577,7 @@ ENDactivateCnf BEGINfreeCnf instanceConf_t *inst, *del; CODESTARTfreeCnf + free(pModConf->pszStrmDrvrName); free(pModConf->pszStrmDrvrAuthMode); if(pModConf->permittedPeers != NULL) { cnfarrayContentDestruct(pModConf->permittedPeers); @@ -571,6 +586,7 @@ CODESTARTfreeCnf for(inst = pModConf->root ; inst != NULL ; ) { free(inst->pszBindPort); free(inst->pszInputName); + free(inst->dfltTZ); del = inst; inst = inst->next; free(del); diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index 312645bd..f13bd148 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -31,6 +31,8 @@ #include <errno.h> #include <unistd.h> #include <netdb.h> +#include <sys/socket.h> +#include <pthread.h> #if HAVE_SYS_EPOLL_H # include <sys/epoll.h> #endif @@ -59,6 +61,7 @@ MODULE_TYPE_NOKEEP MODULE_CNFNAME("imudp") /* defines */ +#define MAX_WRKR_THREADS 32 /* Module static data */ DEF_IMOD_STATIC_DATA @@ -78,9 +81,11 @@ static struct lstn_s { prop_t *pInputName; statsobj_t *stats; /* listener stats */ ratelimit_t *ratelimiter; + uchar *dfltTZ; STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) } *lcnfRoot = NULL, *lcnfLast = NULL; + static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ static int bDoACLCheck; /* are ACL checks neeed? Cached once immediately before listener startup */ static int iMaxLine; /* maximum UDP message size supported */ @@ -88,11 +93,7 @@ static time_t ttLastDiscard = 0; /* timestamp when a message from a non-permitte * This shall prevent remote DoS when the "discard on disallowed sender" * message is configured to be logged on occurance of such a case. */ -static uchar *pRcvBuf = NULL; /* receive buffer (for a single packet). We use a global and alloc - * it so that we can check available memory in willRun() and request - * termination if we can not get it. -- rgerhards, 2007-12-27 - */ - +#define BATCH_SIZE_DFLT 32 /* do not overdo, has heavy toll on memory, especially with large msgs */ #define TIME_REQUERY_DFLT 2 #define SCHED_PRIO_UNSET -12345678 /* a value that indicates that the scheduling priority has not been set */ /* config vars for legacy config system */ @@ -110,12 +111,33 @@ struct instanceConf_s { uchar *pszBindRuleset; /* name of ruleset to bind to */ uchar *inputname; ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ + uchar *dfltTZ; int ratelimitInterval; int ratelimitBurst; + int rcvbuf; /* 0 means: do not set, keep OS default */ struct instanceConf_s *next; sbool bAppendPortToInpname; }; +/* The following structure controls the worker threads. Global data is + * needed for their access. + */ +static struct wrkrInfo_s { + pthread_t tid; /* the worker's thread ID */ + int id; + thrdInfo_t *pThrd; + statsobj_t *stats; /* worker thread stats */ + STATSCOUNTER_DEF(ctrCall_recvmmsg, mutCtrCall_recvmmsg) + STATSCOUNTER_DEF(ctrCall_recvmsg, mutCtrCall_recvmsg) + STATSCOUNTER_DEF(ctrMsgsRcvd, mutCtrMsgsRcvd) + uchar *pRcvBuf; /* receive buffer (for a single packet) */ +# ifdef HAVE_RECVMMSG + struct sockaddr_storage *frominet; + struct mmsghdr *recvmsg_mmh; + struct iovec *recvmsg_iov; +# endif +} wrkrInfo[MAX_WRKR_THREADS]; + struct modConfData_s { rsconf_t *pConf; /* our overall config object */ instanceConf_t *root, *tail; @@ -123,6 +145,8 @@ struct modConfData_s { int iSchedPolicy; /* scheduling policy as SCHED_xxx */ int iSchedPrio; /* scheduling priority */ int iTimeRequery; /* how often is time to be queried inside tight recv loop? 0=always */ + int batchSize; /* max nbr of input batch --> also recvmmsg() max count */ + int8_t wrkrMax; /* max nbr of worker threads */ sbool configSetViaV2Method; }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ @@ -132,6 +156,8 @@ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current lo static struct cnfparamdescr modpdescr[] = { { "schedulingpolicy", eCmdHdlrGetWord, 0 }, { "schedulingpriority", eCmdHdlrInt, 0 }, + { "batchsize", eCmdHdlrInt, 0 }, + { "threads", eCmdHdlrPositiveInt, 0 }, { "timerequery", eCmdHdlrInt, 0 } }; static struct cnfparamblk modpblk = @@ -143,12 +169,14 @@ static struct cnfparamblk modpblk = /* input instance parameters */ static struct cnfparamdescr inppdescr[] = { { "port", eCmdHdlrArray, CNFPARAM_REQUIRED }, /* legacy: InputTCPServerRun */ + { "defaulttz", eCmdHdlrString, 0 }, { "inputname", eCmdHdlrGetWord, 0 }, { "inputname.appendport", eCmdHdlrBinary, 0 }, { "address", eCmdHdlrString, 0 }, - { "ruleset", eCmdHdlrString, 0 }, { "ratelimit.interval", eCmdHdlrInt, 0 }, - { "ratelimit.burst", eCmdHdlrInt, 0 } + { "ratelimit.burst", eCmdHdlrInt, 0 }, + { "rcvbufsize", eCmdHdlrSize, 0 }, + { "ruleset", eCmdHdlrString, 0 } }; static struct cnfparamblk inppblk = { CNFPARAMBLK_VERSION, @@ -177,6 +205,8 @@ createInstance(instanceConf_t **pinst) inst->bAppendPortToInpname = 0; inst->ratelimitBurst = 10000; /* arbitrary high limit */ inst->ratelimitInterval = 0; /* off */ + inst->rcvbuf = 0; + inst->dfltTZ = NULL; /* node created, let's add to config */ if(loadModConf->tail == NULL) { @@ -252,7 +282,7 @@ addListner(instanceConf_t *inst) DBGPRINTF("Trying to open syslog UDP ports at %s:%s.\n", bindName, inst->pszBindPort); - newSocks = net.create_udp_socket(bindAddr, port, 1); + newSocks = net.create_udp_socket(bindAddr, port, 1, inst->rcvbuf); if(newSocks != NULL) { /* we now need to add the new sockets to the existing set */ /* ready to copy */ @@ -261,14 +291,15 @@ addListner(instanceConf_t *inst) newlcnfinfo->next = NULL; newlcnfinfo->sock = newSocks[iSrc]; newlcnfinfo->pRuleset = inst->pBindRuleset; - snprintf((char*)dispname, sizeof(dispname), "imudp(%s:%s)", bindName, port); - dispname[sizeof(dispname)-1] = '\0'; /* just to be on the save side... */ - CHKiRet(ratelimitNew(&newlcnfinfo->ratelimiter, (char*)dispname, NULL)); + newlcnfinfo->dfltTZ = inst->dfltTZ; if(inst->inputname == NULL) { inputname = (uchar*)"imudp"; } else { inputname = inst->inputname; } + snprintf((char*)dispname, sizeof(dispname), "%s(%s:%s)", inputname, bindName, port); + dispname[sizeof(dispname)-1] = '\0'; /* just to be on the save side... */ + CHKiRet(ratelimitNew(&newlcnfinfo->ratelimiter, (char*)dispname, NULL)); if(inst->bAppendPortToInpname) { snprintf((char*)inpnameBuf, sizeof(inpnameBuf), "%s%s", inputname, port); @@ -286,7 +317,7 @@ addListner(instanceConf_t *inst) CHKiRet(statsobj.SetName(newlcnfinfo->stats, dispname)); STATSCOUNTER_INIT(newlcnfinfo->ctrSubmit, newlcnfinfo->mutCtrSubmit); CHKiRet(statsobj.AddCounter(newlcnfinfo->stats, UCHAR_CONSTANT("submitted"), - ctrType_IntCtr, &(newlcnfinfo->ctrSubmit))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(newlcnfinfo->ctrSubmit))); CHKiRet(statsobj.ConstructFinalize(newlcnfinfo->stats)); /* link to list. Order must be preserved to take care for * conflicting matches. @@ -318,6 +349,155 @@ std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, insta } +/* This function processes received data. It provides unified handling + * in cases where recvmmsg() is available and not. + */ +static inline rsRetVal +processPacket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, + uchar *rcvBuf, ssize_t lenRcvBuf, struct syslogTime *stTime, time_t ttGenTime, + struct sockaddr_storage *frominet, socklen_t socklen, multi_submit_t *multiSub) +{ + DEFiRet; + msg_t *pMsg; + + assert(pThrd != NULL); + + if(lenRcvBuf == 0) + FINALIZE; /* this looks a bit strange, but practice shows it happens... */ + + /* if we reach this point, we had a good receive and can process the packet received */ + /* check if we have a different sender than before, if so, we need to query some new values */ + if(bDoACLCheck) { + socklen = sizeof(struct sockaddr_storage); + if(net.CmpHost(frominet, frominetPrev, socklen) != 0) { + memcpy(frominetPrev, frominet, socklen); /* update cache indicator */ + /* Here we check if a host is permitted to send us syslog messages. If it isn't, + * we do not further process the message but log a warning (if we are + * configured to do this). However, if the check would require name resolution, + * it is postponed to the main queue. See also my blog post at + * http://blog.gerhards.net/2009/11/acls-imudp-and-accepting-messages.html + * rgerhards, 2009-11-16 + */ + *pbIsPermitted = net.isAllowedSender2((uchar*)"UDP", + (struct sockaddr *)frominet, "", 0); + + if(*pbIsPermitted == 0) { + DBGPRINTF("msg is not from an allowed sender\n"); + if(glbl.GetOption_DisallowWarning) { + time_t tt; + datetime.GetTime(&tt); + if(tt > ttLastDiscard + 60) { + ttLastDiscard = tt; + errmsg.LogError(0, NO_ERRCODE, + "UDP message from disallowed sender discarded"); + } + } + } + } + } else { + *pbIsPermitted = 1; /* no check -> everything permitted */ + } + + DBGPRINTF("recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, rcvBuf); + + if(*pbIsPermitted != 0) { + /* we now create our own message object and submit it to the queue */ + CHKiRet(msgConstructWithTime(&pMsg, stTime, ttGenTime)); + MsgSetRawMsg(pMsg, (char*)rcvBuf, lenRcvBuf); + MsgSetInputName(pMsg, lstn->pInputName); + MsgSetRuleset(pMsg, lstn->pRuleset); + MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); + if(lstn->dfltTZ != NULL) + MsgSetDfltTZ(pMsg, (char*) lstn->dfltTZ); + pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME | NEEDS_DNSRESOL; + if(*pbIsPermitted == 2) + pMsg->msgFlags |= NEEDS_ACLCHK_U; /* request ACL check after resolution */ + CHKiRet(msgSetFromSockinfo(pMsg, frominet)); + CHKiRet(ratelimitAddMsg(lstn->ratelimiter, multiSub, pMsg)); + STATSCOUNTER_INC(lstn->ctrSubmit, lstn->mutCtrSubmit); + } + +finalize_it: + RETiRet; +} + + + + +/* The following "two" functions are helpers to runInput. Actually, it is + * just one function. Depending on whether or not we have recvmmsg(), + * an appropriate version is compiled (as such we need to maintain both!). + */ +#ifdef HAVE_RECVMMSG +static inline rsRetVal +processSocket(struct wrkrInfo_s *pWrkr, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted) +{ + DEFiRet; + int iNbrTimeUsed; + time_t ttGenTime; + struct syslogTime stTime; + char errStr[1024]; + msg_t *pMsgs[CONF_NUM_MULTISUB]; + multi_submit_t multiSub; + int nelem; + int i; + + multiSub.ppMsgs = pMsgs; + multiSub.maxElem = CONF_NUM_MULTISUB; + multiSub.nElem = 0; + iNbrTimeUsed = 0; + while(1) { /* loop is terminated if we have a "bad" receive, done below in the body */ + if(pWrkr->pThrd->bShallStop == RSTRUE) + ABORT_FINALIZE(RS_RET_FORCE_TERM); + memset(pWrkr->recvmsg_iov, 0, runModConf->batchSize * sizeof(struct iovec)); + memset(pWrkr->recvmsg_mmh, 0, runModConf->batchSize * sizeof(struct mmsghdr)); + for(i = 0 ; i < runModConf->batchSize ; ++i) { + pWrkr->recvmsg_iov[i].iov_base = pWrkr->pRcvBuf+(i*(iMaxLine+1)); + pWrkr->recvmsg_iov[i].iov_len = iMaxLine; + pWrkr->recvmsg_mmh[i].msg_hdr.msg_namelen = sizeof(struct sockaddr_storage); + pWrkr->recvmsg_mmh[i].msg_hdr.msg_name = &(pWrkr->frominet[i]); + pWrkr->recvmsg_mmh[i].msg_hdr.msg_iov = &(pWrkr->recvmsg_iov[i]); + pWrkr->recvmsg_mmh[i].msg_hdr.msg_iovlen = 1; + } + nelem = recvmmsg(lstn->sock, pWrkr->recvmsg_mmh, runModConf->batchSize, 0, NULL); + STATSCOUNTER_INC(pWrkr->ctrCall_recvmmsg, pWrkr->mutCtrCall_recvmmsg); + DBGPRINTF("imudp: recvmmsg returned %d\n", nelem); + if(nelem < 0 && errno == ENOSYS) { + /* be careful: some versions of valgrind do not support recvmmsg()! */ + DBGPRINTF("imudp: error ENOSYS on call to recvmmsg() - fall back to recvmsg\n"); + nelem = recvmsg(lstn->sock, &(pWrkr->recvmsg_mmh[0].msg_hdr), 0); + STATSCOUNTER_INC(pWrkr->ctrCall_recvmsg, pWrkr->mutCtrCall_recvmsg); + if(nelem >= 0) { + pWrkr->recvmsg_mmh[0].msg_len = nelem; + nelem = 1; + } + } + if(nelem < 0) { + if(errno != EINTR && errno != EAGAIN) { + rs_strerror_r(errno, errStr, sizeof(errStr)); + DBGPRINTF("INET socket error: %d = %s.\n", errno, errStr); + errmsg.LogError(errno, NO_ERRCODE, "imudp: error receiving on socket: %s", errStr); + } + ABORT_FINALIZE(RS_RET_ERR); // this most often is NOT an error, state is not checked by caller! + } + + if((runModConf->iTimeRequery == 0) || (iNbrTimeUsed++ % runModConf->iTimeRequery) == 0) { + datetime.getCurrTime(&stTime, &ttGenTime); + } + + pWrkr->ctrMsgsRcvd += nelem; + for(i = 0 ; i < nelem ; ++i) { + processPacket(pWrkr->pThrd, lstn, frominetPrev, pbIsPermitted, pWrkr->recvmsg_mmh[i].msg_hdr.msg_iov->iov_base, + pWrkr->recvmsg_mmh[i].msg_len, &stTime, ttGenTime, &(pWrkr->frominet[i]), + pWrkr->recvmsg_mmh[i].msg_hdr.msg_namelen, &multiSub); + } + } + +finalize_it: + multiSubmitFlush(&multiSub); + RETiRet; +} +#else /* we do not have recvmmsg() */ /* This function is a helper to runInput. I have extracted it * from the main loop just so that we do not have that large amount of code * in a single place. This function takes a socket and pulls messages from @@ -333,108 +513,61 @@ std_checkRuleset_genErrMsg(__attribute__((unused)) modConfData_t *modConf, insta * on scheduling order. -- rgerhards, 2008-10-02 */ static inline rsRetVal -processSocket(thrdInfo_t *pThrd, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted) +processSocket(struct wrkrInfo_s *pWrkr, struct lstn_s *lstn, struct sockaddr_storage *frominetPrev, int *pbIsPermitted) { int iNbrTimeUsed; time_t ttGenTime; struct syslogTime stTime; - socklen_t socklen; ssize_t lenRcvBuf; struct sockaddr_storage frominet; - msg_t *pMsg; - prop_t *propFromHost = NULL; - prop_t *propFromHostIP = NULL; multi_submit_t multiSub; msg_t *pMsgs[CONF_NUM_MULTISUB]; char errStr[1024]; + struct msghdr mh; + struct iovec iov[1]; DEFiRet; - assert(pThrd != NULL); multiSub.ppMsgs = pMsgs; multiSub.maxElem = CONF_NUM_MULTISUB; multiSub.nElem = 0; iNbrTimeUsed = 0; while(1) { /* loop is terminated if we have a bad receive, done below in the body */ - if(pThrd->bShallStop == RSTRUE) + if(pWrkr->pThrd->bShallStop == RSTRUE) ABORT_FINALIZE(RS_RET_FORCE_TERM); - socklen = sizeof(struct sockaddr_storage); - lenRcvBuf = recvfrom(lstn->sock, (char*) pRcvBuf, iMaxLine, 0, (struct sockaddr *)&frominet, &socklen); + memset(iov, 0, sizeof(iov)); + iov[0].iov_base = pWrkr->pRcvBuf; + iov[0].iov_len = iMaxLine; + memset(&mh, 0, sizeof(mh)); + mh.msg_name = &frominet; + mh.msg_namelen = sizeof(struct sockaddr_storage); + mh.msg_iov = iov; + mh.msg_iovlen = 1; + lenRcvBuf = recvmsg(lstn->sock, &mh, 0); + STATSCOUNTER_INC(pWrkr->ctrCall_recvmsg, pWrkr->mutCtrCall_recvmsg); if(lenRcvBuf < 0) { if(errno != EINTR && errno != EAGAIN) { rs_strerror_r(errno, errStr, sizeof(errStr)); DBGPRINTF("INET socket error: %d = %s.\n", errno, errStr); - errmsg.LogError(errno, NO_ERRCODE, "recvfrom inet"); + errmsg.LogError(errno, NO_ERRCODE, "imudp: error receiving on socket: %s", errStr); } ABORT_FINALIZE(RS_RET_ERR); // this most often is NOT an error, state is not checked by caller! } - if(lenRcvBuf == 0) - continue; /* this looks a bit strange, but practice shows it happens... */ - - /* if we reach this point, we had a good receive and can process the packet received */ - /* check if we have a different sender than before, if so, we need to query some new values */ - if(bDoACLCheck) { - if(net.CmpHost(&frominet, frominetPrev, socklen) != 0) { - memcpy(frominetPrev, &frominet, socklen); /* update cache indicator */ - /* Here we check if a host is permitted to send us syslog messages. If it isn't, - * we do not further process the message but log a warning (if we are - * configured to do this). However, if the check would require name resolution, - * it is postponed to the main queue. See also my blog post at - * http://blog.gerhards.net/2009/11/acls-imudp-and-accepting-messages.html - * rgerhards, 2009-11-16 - */ - *pbIsPermitted = net.isAllowedSender2((uchar*)"UDP", - (struct sockaddr *)&frominet, "", 0); - - if(*pbIsPermitted == 0) { - DBGPRINTF("msg is not from an allowed sender\n"); - if(glbl.GetOption_DisallowWarning) { - time_t tt; - datetime.GetTime(&tt); - if(tt > ttLastDiscard + 60) { - ttLastDiscard = tt; - errmsg.LogError(0, NO_ERRCODE, - "UDP message from disallowed sender discarded"); - } - } - } - } - } else { - *pbIsPermitted = 1; /* no check -> everything permitted */ + ++pWrkr->ctrMsgsRcvd; + if((runModConf->iTimeRequery == 0) || (iNbrTimeUsed++ % runModConf->iTimeRequery) == 0) { + datetime.getCurrTime(&stTime, &ttGenTime); } - DBGPRINTF("imudp:recv(%d,%d),acl:%d,msg:%s\n", lstn->sock, (int) lenRcvBuf, *pbIsPermitted, pRcvBuf); - - if(*pbIsPermitted != 0) { - if((runModConf->iTimeRequery == 0) || (iNbrTimeUsed++ % runModConf->iTimeRequery) == 0) { - datetime.getCurrTime(&stTime, &ttGenTime); - } - /* we now create our own message object and submit it to the queue */ - CHKiRet(msgConstructWithTime(&pMsg, &stTime, ttGenTime)); - MsgSetRawMsg(pMsg, (char*)pRcvBuf, lenRcvBuf); - MsgSetInputName(pMsg, lstn->pInputName); - MsgSetRuleset(pMsg, lstn->pRuleset); - MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); - pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME | NEEDS_DNSRESOL; - if(*pbIsPermitted == 2) - pMsg->msgFlags |= NEEDS_ACLCHK_U; /* request ACL check after resolution */ - CHKiRet(msgSetFromSockinfo(pMsg, &frominet)); - CHKiRet(ratelimitAddMsg(lstn->ratelimiter, &multiSub, pMsg)); - STATSCOUNTER_INC(lstn->ctrSubmit, lstn->mutCtrSubmit); - } + CHKiRet(processPacket(pWrkr->pThrd, lstn, frominetPrev, pbIsPermitted, pWrkr->pRcvBuf, lenRcvBuf, &stTime, + ttGenTime, &frominet, mh.msg_namelen, &multiSub)); } finalize_it: multiSubmitFlush(&multiSub); - - if(propFromHost != NULL) - prop.Destruct(&propFromHost); - if(propFromHostIP != NULL) - prop.Destruct(&propFromHostIP); - RETiRet; } +#endif /* #ifdef HAVE_RECVMMSG */ /* check configured scheduling priority. @@ -565,7 +698,7 @@ finalize_it: */ #if defined(HAVE_EPOLL_CREATE1) || defined(HAVE_EPOLL_CREATE) #define NUM_EPOLL_EVENTS 10 -rsRetVal rcvMainLoop(thrdInfo_t *pThrd) +rsRetVal rcvMainLoop(struct wrkrInfo_s *pWrkr) { DEFiRet; int nfds; @@ -628,11 +761,11 @@ rsRetVal rcvMainLoop(thrdInfo_t *pThrd) nfds = epoll_wait(efd, currEvt, NUM_EPOLL_EVENTS, -1); DBGPRINTF("imudp: epoll_wait() returned with %d fds\n", nfds); - if(pThrd->bShallStop == RSTRUE) + if(pWrkr->pThrd->bShallStop == RSTRUE) break; /* terminate input! */ for(i = 0 ; i < nfds ; ++i) { - processSocket(pThrd, currEvt[i].data.ptr, &frominetPrev, &bIsPermitted); + processSocket(pWrkr, currEvt[i].data.ptr, &frominetPrev, &bIsPermitted); } } @@ -644,7 +777,7 @@ finalize_it: } #else /* #if HAVE_EPOLL_CREATE1 */ /* this is the code for the select() interface */ -rsRetVal rcvMainLoop(thrdInfo_t *pThrd) +rsRetVal rcvMainLoop(thrdInfo_t *pWrkr) { DEFiRet; int maxfds; @@ -691,7 +824,7 @@ rsRetVal rcvMainLoop(thrdInfo_t *pThrd) for(lstn = lcnfRoot ; nfds && lstn != NULL ; lstn = lstn->next) { if(FD_ISSET(lstn->sock, &readfds)) { - processSocket(pThrd, lstn, &frominetPrev, &bIsPermitted); + processSocket(pWrkr, lstn, &frominetPrev, &bIsPermitted); --nfds; /* indicate we have processed one descriptor */ } } @@ -721,6 +854,8 @@ createListner(es_str_t *port, struct cnfparamvals *pvals) inst->inputname = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "inputname.appendport")) { inst->bAppendPortToInpname = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "defaulttz")) { + inst->dfltTZ = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "address")) { inst->pszBindAddr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { @@ -729,6 +864,8 @@ createListner(es_str_t *port, struct cnfparamvals *pvals) inst->ratelimitBurst = (int) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "ratelimit.interval")) { inst->ratelimitInterval = (int) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "rcvbufsize")) { + inst->rcvbuf = (int) pvals[i].val.d.n; } else { dbgprintf("imudp: program error, non-handled " "param '%s'\n", inppblk.descr[i].name); @@ -772,6 +909,8 @@ CODESTARTbeginCnfLoad pModConf->pConf = pConf; /* init our settings */ loadModConf->configSetViaV2Method = 0; + loadModConf->wrkrMax = 1; /* conservative, but least msg reordering */ + loadModConf->batchSize = BATCH_SIZE_DFLT; loadModConf->iTimeRequery = TIME_REQUERY_DFLT; loadModConf->iSchedPrio = SCHED_PRIO_UNSET; loadModConf->pszSchedPolicy = NULL; @@ -788,6 +927,7 @@ ENDbeginCnfLoad BEGINsetModCnf struct cnfparamvals *pvals = NULL; int i; + int wrkrMax; CODESTARTsetModCnf pvals = nvlstGetParams(lst, &modpblk, NULL); if(pvals == NULL) { @@ -806,10 +946,22 @@ CODESTARTsetModCnf continue; if(!strcmp(modpblk.descr[i].name, "timerequery")) { loadModConf->iTimeRequery = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "batchsize")) { + loadModConf->batchSize = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "schedulingpriority")) { loadModConf->iSchedPrio = (int) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "schedulingpolicy")) { loadModConf->pszSchedPolicy = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(modpblk.descr[i].name, "threads")) { + wrkrMax = (int) pvals[i].val.d.n; + if(wrkrMax > MAX_WRKR_THREADS) { + errmsg.LogError(0, RS_RET_PARAM_ERROR, "imudp: configured for %d" + "worker threads, but maximum permitted is %d", + wrkrMax, MAX_WRKR_THREADS); + loadModConf->wrkrMax = MAX_WRKR_THREADS; + } else { + loadModConf->wrkrMax = wrkrMax; + } } else { dbgprintf("imudp: program error, non-handled " "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); @@ -881,10 +1033,24 @@ ENDactivateCnfPrePrivDrop BEGINactivateCnf + int i; + int lenRcvBuf; CODESTARTactivateCnf /* caching various settings */ iMaxLine = glbl.GetMaxLine(); - CHKmalloc(pRcvBuf = MALLOC((iMaxLine + 1) * sizeof(char))); + lenRcvBuf = (iMaxLine + 1) * sizeof(char); +# ifdef HAVE_RECVMMSG + lenRcvBuf *= runModConf->batchSize; +# endif + for(i = 0 ; i < runModConf->wrkrMax ; ++i) { +# ifdef HAVE_RECVMMSG + CHKmalloc(wrkrInfo[i].recvmsg_iov = MALLOC(runModConf->batchSize * sizeof(struct iovec))); + CHKmalloc(wrkrInfo[i].recvmsg_mmh = MALLOC(runModConf->batchSize * sizeof(struct mmsghdr))); + CHKmalloc(wrkrInfo[i].frominet = MALLOC(runModConf->batchSize * sizeof(struct sockaddr_storage))); +# endif + CHKmalloc(wrkrInfo[i].pRcvBuf = MALLOC(lenRcvBuf)); + wrkrInfo[i].id = i; + } finalize_it: ENDactivateCnf @@ -896,20 +1062,34 @@ CODESTARTfreeCnf free(inst->pszBindPort); free(inst->pszBindAddr); free(inst->inputname); + free(inst->dfltTZ); del = inst; inst = inst->next; free(del); } ENDfreeCnf -/* This function is called to gather input. - * Note that sock must be non-NULL because otherwise we would not have - * indicated that we want to run (or we have a programming error ;)). -- rgerhards, 2008-10-02 - */ -BEGINrunInput -CODESTARTrunInput + +static void * +wrkr(void *myself) +{ + struct wrkrInfo_s *pWrkr = (struct wrkrInfo_s*) myself; +# if HAVE_PRCTL && defined PR_SET_NAME + uchar *pszDbgHdr; +# endif + uchar thrdName[32]; + + snprintf((char*)thrdName, sizeof(thrdName), "imudp(w%d)", pWrkr->id); +# if HAVE_PRCTL && defined PR_SET_NAME + /* set thread name - we ignore if the call fails, has no harsh consequences... */ + if(prctl(PR_SET_NAME, thrdName, 0, 0, 0) != 0) { + DBGPRINTF("prctl failed, not setting thread name for '%s'\n", thrdName); + } +# endif + dbgOutputTID((char*)thrdName); + /* Note well: the setting of scheduling parameters will not work - * when we dropped privileges (if the user is not sufficently + * when we dropped privileges (if the user is not sufficiently * privileged, of course). Howerver, we can't change the * scheduling params in PrePrivDrop(), as at that point our thread * is not yet created. So at least as an interim solution, we do @@ -917,7 +1097,51 @@ CODESTARTrunInput * privileges within the same instance. */ setSchedParams(runModConf); - iRet = rcvMainLoop(pThrd); + + /* support statistics gathering */ + statsobj.Construct(&(pWrkr->stats)); + statsobj.SetName(pWrkr->stats, thrdName); + STATSCOUNTER_INIT(pWrkr->ctrCall_recvmmsg, pWrkr->mutCtrCall_recvmmsg); + statsobj.AddCounter(pWrkr->stats, UCHAR_CONSTANT("called.recvmmsg"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pWrkr->ctrCall_recvmmsg)); + STATSCOUNTER_INIT(pWrkr->ctrCall_recvmsg, pWrkr->mutCtrCall_recvmsg); + statsobj.AddCounter(pWrkr->stats, UCHAR_CONSTANT("called.recvmsg"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pWrkr->ctrCall_recvmsg)); + STATSCOUNTER_INIT(pWrkr->ctrMsgsRcvd, pWrkr->mutCtrMsgsRcvd); + statsobj.AddCounter(pWrkr->stats, UCHAR_CONSTANT("msgs.received"), + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pWrkr->ctrMsgsRcvd)); + statsobj.ConstructFinalize(pWrkr->stats); + + rcvMainLoop(pWrkr); + + /* cleanup */ + return NULL; +} + +/* This function is called to gather input. + * In essence, it just starts the pool of workers. To save resources, + * we run one of the workers on our own thread -- otherwise that thread would + * just idle around and wait for the workers to finish. + */ +BEGINrunInput + int i; + pthread_attr_t wrkrThrdAttr; +CODESTARTrunInput + pthread_attr_init(&wrkrThrdAttr); + pthread_attr_setstacksize(&wrkrThrdAttr, 4096*1024); + for(i = 0 ; i < runModConf->wrkrMax - 1 ; ++i) { + wrkrInfo[i].pThrd = pThrd; + pthread_create(&wrkrInfo[i].tid, &wrkrThrdAttr, wrkr, &(wrkrInfo[i])); + } + pthread_attr_destroy(&wrkrThrdAttr); + + wrkrInfo[i].pThrd = pThrd; + wrkrInfo[i].id = i; + wrkr(&wrkrInfo[i]); + + for(i = 0 ; i < runModConf->wrkrMax - 1 ; ++i) { + pthread_join(wrkrInfo[i].tid, NULL); + } ENDrunInput @@ -931,6 +1155,7 @@ ENDwillRun BEGINafterRun struct lstn_s *lstn, *lstnDel; + int i; CODESTARTafterRun /* do cleanup here */ net.clearAllowedSenders((uchar*)"UDP"); @@ -944,9 +1169,13 @@ CODESTARTafterRun free(lstnDel); } lcnfRoot = lcnfLast = NULL; - if(pRcvBuf != NULL) { - free(pRcvBuf); - pRcvBuf = NULL; + for(i = 0 ; i < runModConf->wrkrMax ; ++i) { +# ifdef HAVE_RECVMMSG + free(wrkrInfo[i].recvmsg_iov); + free(wrkrInfo[i].recvmsg_mmh); + free(wrkrInfo[i].frominet); +# endif + free(wrkrInfo[i].pRcvBuf); } ENDafterRun @@ -1007,6 +1236,11 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(ruleset, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); + DBGPRINTF("imudp: version %s initializing\n", VERSION); +# ifdef HAVE_RECVMMSG + DBGPRINTF("imdup: support for recvmmsg() present\n"); +# endif + /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputudpserverbindruleset", 0, eCmdHdlrGetWord, NULL, &cs.pszBindRuleset, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index df504dd6..a53ce159 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -62,7 +62,6 @@ MODULE_TYPE_NOKEEP MODULE_CNFNAME("imuxsock") /* defines */ -#define MAXFUNIX 50 #ifndef _PATH_LOG #ifdef BSD #define _PATH_LOG "/var/run/log" @@ -148,7 +147,7 @@ typedef struct lstn_s { sbool bUseSysTimeStamp; /* use timestamp from system (instead of from message) */ sbool bUnlink; /* unlink&re-create socket at start and end of processing */ } lstn_t; -static lstn_t listeners[MAXFUNIX]; +static lstn_t *listeners; static prop_t *pLocalHostIP = NULL; /* there is only one global IP for all internally-generated messages */ static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ @@ -156,7 +155,8 @@ static int startIndexUxLocalSockets; /* process fd from that index on (used to * suppress local logging. rgerhards 2005-08-01 * read-only after startup */ -static int nfd = 1; /* number of Unix sockets open / read-only after startup */ +static int nfd = 1; /* number of active unix sockets (socket 0 is always reserved for the system + socket, even if it is not enabled. */ static int sd_fds = 0; /* number of systemd activated sockets */ #define DFLT_bCreatePath 0 @@ -360,12 +360,7 @@ finalize_it: } -/* add an additional listen socket. Socket names are added - * until the array is filled up. It is never reset, only at - * module unload. - * TODO: we should change the array to a list so that we - * can support any number of listen socket names. - * rgerhards, 2007-12-20 +/* add an additional listen socket. * added capability to specify hostname for socket -- rgerhards, 2008-08-01 */ static rsRetVal @@ -373,53 +368,50 @@ addListner(instanceConf_t *inst) { DEFiRet; - if(nfd < MAXFUNIX) { - if(*inst->sockName == ':') { - listeners[nfd].bParseHost = 1; - } else { - listeners[nfd].bParseHost = 0; - } - if(inst->pLogHostName == NULL) { - listeners[nfd].hostName = NULL; - } else { - CHKiRet(prop.Construct(&(listeners[nfd].hostName))); - CHKiRet(prop.SetString(listeners[nfd].hostName, inst->pLogHostName, ustrlen(inst->pLogHostName))); - CHKiRet(prop.ConstructFinalize(listeners[nfd].hostName)); - } - if(inst->ratelimitInterval > 0) { - if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, - (void(*)(void*))ratelimitDestruct)) == NULL) { - /* in this case, we simply turn off rate-limiting */ - DBGPRINTF("imuxsock: turning off rate limiting because we could not " - "create hash table\n"); - inst->ratelimitInterval = 0; - } + if(*inst->sockName == ':') { + listeners[nfd].bParseHost = 1; + } else { + listeners[nfd].bParseHost = 0; + } + if(inst->pLogHostName == NULL) { + listeners[nfd].hostName = NULL; + } else { + CHKiRet(prop.Construct(&(listeners[nfd].hostName))); + CHKiRet(prop.SetString(listeners[nfd].hostName, inst->pLogHostName, ustrlen(inst->pLogHostName))); + CHKiRet(prop.ConstructFinalize(listeners[nfd].hostName)); + } + if(inst->ratelimitInterval > 0) { + if((listeners[nfd].ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, + (void(*)(void*))ratelimitDestruct)) == NULL) { + /* in this case, we simply turn off rate-limiting */ + DBGPRINTF("imuxsock: turning off rate limiting because we could not " + "create hash table\n"); + inst->ratelimitInterval = 0; } - listeners[nfd].ratelimitInterval = inst->ratelimitInterval; - listeners[nfd].ratelimitBurst = inst->ratelimitBurst; - listeners[nfd].ratelimitSev = inst->ratelimitSeverity; - listeners[nfd].flowCtl = inst->bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; - listeners[nfd].flags = inst->bIgnoreTimestamp ? IGNDATE : NOFLAG; - listeners[nfd].bCreatePath = inst->bCreatePath; - listeners[nfd].sockName = ustrdup(inst->sockName); - listeners[nfd].bUseCreds = (inst->bDiscardOwnMsgs || inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate || inst->bUseSysTimeStamp) ? 1 : 0; - listeners[nfd].bAnnotate = inst->bAnnotate; - listeners[nfd].bParseTrusted = inst->bParseTrusted; - listeners[nfd].bDiscardOwnMsgs = inst->bDiscardOwnMsgs; - listeners[nfd].bUnlink = inst->bUnlink; - listeners[nfd].bWritePid = inst->bWritePid; - listeners[nfd].bUseSysTimeStamp = inst->bUseSysTimeStamp; - CHKiRet(ratelimitNew(&listeners[nfd].dflt_ratelimiter, "imuxsock", NULL)); - ratelimitSetLinuxLike(listeners[nfd].dflt_ratelimiter, - listeners[nfd].ratelimitInterval, - listeners[nfd].ratelimitBurst); - ratelimitSetSeverity(listeners[nfd].dflt_ratelimiter, - listeners[nfd].ratelimitSev); - nfd++; } else { - errmsg.LogError(0, NO_ERRCODE, "Out of unix socket name descriptors, ignoring %s\n", - inst->sockName); + listeners[nfd].ht = NULL; } + listeners[nfd].ratelimitInterval = inst->ratelimitInterval; + listeners[nfd].ratelimitBurst = inst->ratelimitBurst; + listeners[nfd].ratelimitSev = inst->ratelimitSeverity; + listeners[nfd].flowCtl = inst->bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; + listeners[nfd].flags = inst->bIgnoreTimestamp ? IGNDATE : NOFLAG; + listeners[nfd].bCreatePath = inst->bCreatePath; + listeners[nfd].sockName = ustrdup(inst->sockName); + listeners[nfd].bUseCreds = (inst->bDiscardOwnMsgs || inst->bWritePid || inst->ratelimitInterval || inst->bAnnotate || inst->bUseSysTimeStamp) ? 1 : 0; + listeners[nfd].bAnnotate = inst->bAnnotate; + listeners[nfd].bParseTrusted = inst->bParseTrusted; + listeners[nfd].bDiscardOwnMsgs = inst->bDiscardOwnMsgs; + listeners[nfd].bUnlink = inst->bUnlink; + listeners[nfd].bWritePid = inst->bWritePid; + listeners[nfd].bUseSysTimeStamp = inst->bUseSysTimeStamp; + CHKiRet(ratelimitNew(&listeners[nfd].dflt_ratelimiter, "imuxsock", NULL)); + ratelimitSetLinuxLike(listeners[nfd].dflt_ratelimiter, + listeners[nfd].ratelimitInterval, + listeners[nfd].ratelimitBurst); + ratelimitSetSeverity(listeners[nfd].dflt_ratelimiter, + listeners[nfd].ratelimitSev); + nfd++; finalize_it: RETiRet; @@ -1286,10 +1278,28 @@ ENDcheckCnf BEGINactivateCnfPrePrivDrop instanceConf_t *inst; + int nLstn; + int i; CODESTARTactivateCnfPrePrivDrop runModConf = pModConf; + /* we first calculate the number of listeners so that we can + * appropriately size the listener array. Note that we will + * always allocate memory for the system log socket. + */ + nLstn = 0; for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { - addListner(inst); + ++nLstn; + } + if(nLstn > 0) { + DBGPRINTF("imuxsock: allocating memory for %d addtl listeners\n", nLstn); + CHKmalloc(listeners = realloc(listeners, (1+nLstn)*sizeof(lstn_t))); + for(i = 1 ; i < nLstn ; ++i) { + listeners[i].sockName = NULL; + listeners[i].fd = -1; + } + for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { + addListner(inst); + } } CHKiRet(activateListeners()); finalize_it: @@ -1329,6 +1339,8 @@ BEGINrunInput #endif CODESTARTrunInput + if(runModConf->bOmitLocalLogging && nfd == 1) + ABORT_FINALIZE(RS_RET_OK); /* this is an endless loop - it is terminated when the thread is * signalled to do so. This, however, is handled by the framework, * right into the sleep below. @@ -1419,6 +1431,7 @@ ENDafterRun BEGINmodExit CODESTARTmodExit + free(listeners); if(pInputName != NULL) prop.Destruct(&pInputName); @@ -1481,7 +1494,6 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a BEGINmodInit() - int i; CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr @@ -1512,6 +1524,7 @@ CODEmodInit_QueryRegCFSLineHdlr pLocalHostIP = glbl.GetLocalHostIP(); /* init system log socket settings */ + CHKmalloc(listeners = malloc(sizeof(lstn_t))); listeners[0].flags = IGNDATE; listeners[0].sockName = UCHAR_CONSTANT(_PATH_LOG); listeners[0].hostName = NULL; @@ -1533,12 +1546,6 @@ CODEmodInit_QueryRegCFSLineHdlr listeners[0].ratelimitInterval = 0; } - /* initialize socket names */ - for(i = 1 ; i < MAXFUNIX ; ++i) { - listeners[i].sockName = NULL; - listeners[i].fd = -1; - } - /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensocketignoremsgtimestamp", 0, eCmdHdlrBinary, NULL, &cs.bIgnoreTimestamp, STD_LOADABLE_MODULE_ID)); @@ -1598,13 +1605,13 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(statsobj.SetName(modStats, UCHAR_CONSTANT("imuxsock"))); STATSCOUNTER_INIT(ctrSubmit, mutCtrSubmit); CHKiRet(statsobj.AddCounter(modStats, UCHAR_CONSTANT("submitted"), - ctrType_IntCtr, &ctrSubmit)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &ctrSubmit)); STATSCOUNTER_INIT(ctrLostRatelimit, mutCtrLostRatelimit); CHKiRet(statsobj.AddCounter(modStats, UCHAR_CONSTANT("ratelimit.discarded"), - ctrType_IntCtr, &ctrLostRatelimit)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &ctrLostRatelimit)); STATSCOUNTER_INIT(ctrNumRatelimiters, mutCtrNumRatelimiters); CHKiRet(statsobj.AddCounter(modStats, UCHAR_CONSTANT("ratelimit.numratelimiters"), - ctrType_IntCtr, &ctrNumRatelimiters)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &ctrNumRatelimiters)); CHKiRet(statsobj.ConstructFinalize(modStats)); ENDmodInit diff --git a/plugins/mmanon/mmanon.c b/plugins/mmanon/mmanon.c index 16a4f34b..28797807 100644 --- a/plugins/mmanon/mmanon.c +++ b/plugins/mmanon/mmanon.c @@ -71,6 +71,10 @@ typedef struct _instanceData { } ipv4; } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + struct modConfData_s { rsconf_t *pConf; /* our overall config object */ }; @@ -119,6 +123,10 @@ BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -130,6 +138,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + static inline void setInstParamDefaults(instanceData *pData) { @@ -358,7 +371,7 @@ CODESTARTdoAction lenMsg = getMSGLen(pMsg); msg = getMSG(pMsg); for(i = 0 ; i < lenMsg ; ++i) { - anonip(pData, msg, &lenMsg, &i); + anonip(pWrkrData->pData, msg, &lenMsg, &i); } if(lenMsg != getMSGLen(pMsg)) setMSGLen(pMsg, lenMsg); @@ -387,6 +400,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES ENDqueryEtryPt diff --git a/plugins/mmcount/Makefile.am b/plugins/mmcount/Makefile.am new file mode 100644 index 00000000..9c8c99db --- /dev/null +++ b/plugins/mmcount/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmcount.la + +mmcount_la_SOURCES = mmcount.c +mmcount_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmcount_la_LDFLAGS = -module -avoid-version +mmcount_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmcount/mmcount.c b/plugins/mmcount/mmcount.c new file mode 100644 index 00000000..56a4de55 --- /dev/null +++ b/plugins/mmcount/mmcount.c @@ -0,0 +1,342 @@ +/* mmcount.c + * count messages by priority or json property of given app-name. + * + * Copyright 2013 Red Hat Inc. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include <json/json.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include "hashtable.h" + +#define JSON_COUNT_NAME "!mmcount" +#define SEVERITY_COUNT 8 + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmcount") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +typedef struct _instanceData { + char *pszAppName; + int severity[SEVERITY_COUNT]; + char *pszKey; + char *pszValue; + int valueCounter; + struct hashtable *ht; +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "appname", eCmdHdlrGetWord, 0 }, + { "key", eCmdHdlrGetWord, 0 }, + { "value", eCmdHdlrGetWord, 0 }, +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + int i; + + pData->pszAppName = NULL; + for (i = 0; i < SEVERITY_COUNT; i++) + pData->severity[i] = 0; + pData->pszKey = NULL; + pData->pszValue = NULL; + pData->valueCounter = 0; + pData->ht = NULL; +} + +static unsigned int +hash_from_key_fn(void *k) +{ + return *(unsigned int *)k; +} + +static int +key_equals_fn(void *k1, void *k2) +{ + return (*(unsigned int *)k1 == *(unsigned int *)k2); +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmcount)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "appname")) { + pData->pszAppName = es_str2cstr(pvals[i].val.d.estr, NULL); + continue; + } + if(!strcmp(actpblk.descr[i].name, "key")) { + pData->pszKey = es_str2cstr(pvals[i].val.d.estr, NULL); + continue; + } + if(!strcmp(actpblk.descr[i].name, "value")) { + pData->pszValue = es_str2cstr(pvals[i].val.d.estr, NULL); + continue; + } + dbgprintf("mmcount: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + + if(pData->pszAppName == NULL) { + dbgprintf("mmcount: action requires a appname"); + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + if(pData->pszKey != NULL && pData->pszValue == NULL) { + if(NULL == (pData->ht = create_hashtable(100, hash_from_key_fn, key_equals_fn, NULL))) { + DBGPRINTF("mmcount: error creating hash table!\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + } +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +static int * +getCounter(struct hashtable *ht, char *str) { + unsigned int key; + int *pCounter; + unsigned int *pKey; + + /* we dont store str as key, instead we store hash of the str + as key to reduce memory usage */ + key = hash_from_string(str); + pCounter = hashtable_search(ht, &key); + if(pCounter) { + return pCounter; + } + + /* counter is not found for the str, so add new entry and + return the counter */ + if(NULL == (pKey = (unsigned int*)malloc(sizeof(unsigned int)))) { + DBGPRINTF("mmcount: memory allocation for key failed\n"); + return NULL; + } + *pKey = key; + + if(NULL == (pCounter = (int*)malloc(sizeof(int)))) { + DBGPRINTF("mmcount: memory allocation for value failed\n"); + free(pKey); + return NULL; + } + *pCounter = 0; + + if(!hashtable_insert(ht, pKey, pCounter)) { + DBGPRINTF("mmcount: inserting element into hashtable failed\n"); + free(pKey); + free(pCounter); + return NULL; + } + return pCounter; +} + +BEGINdoAction + msg_t *pMsg; + char *appname; + struct json_object *json = NULL; + es_str_t *estr = NULL; + struct json_object *keyjson = NULL; + char *pszValue; + int *pCounter; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + appname = getAPPNAME(pMsg, LOCK_MUTEX); + + if(0 != strcmp(appname, pData->pszAppName)) { + /* we are not working for this appname. nothing to do */ + ABORT_FINALIZE(RS_RET_OK); + } + + if(!pData->pszKey) { + /* no key given for count, so we count severity */ + if(pMsg->iSeverity <= SEVERITY_COUNT) { + pData->severity[pMsg->iSeverity]++; + json = json_object_new_int(pData->severity[pMsg->iSeverity]); + } + ABORT_FINALIZE(RS_RET_OK); + } + + /* key is given, so get the property json */ + estr = es_newStrFromBuf(pData->pszKey, strlen(pData->pszKey)); + if(msgGetCEEPropJSON(pMsg, estr, &keyjson) != RS_RET_OK) { + /* key not found in the message. nothing to do */ + ABORT_FINALIZE(RS_RET_OK); + } + + /* key found, so get the value */ + pszValue = (char*)json_object_get_string(keyjson); + + if(pData->pszValue) { + /* value also given for count */ + if(!strcmp(pszValue, pData->pszValue)) { + /* count for (value and key and appname) matched */ + pData->valueCounter++; + json = json_object_new_int(pData->valueCounter); + } + ABORT_FINALIZE(RS_RET_OK); + } + + /* value is not given, so we count for each value of given key */ + pCounter = getCounter(pData->ht, pszValue); + if(pCounter) { + (*pCounter)++; + json = json_object_new_int(*pCounter); + } +finalize_it: + if(estr) { + es_deleteStr(estr); + } + + if(json) { + msgAddJSON(pMsg, (uchar *)JSON_COUNT_NAME, json); + } +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmcount:", sizeof(":mmcount:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmcount supports only v6+ config format, use: " + "action(type=\"mmcount\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmcount: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmfields/Makefile.am b/plugins/mmfields/Makefile.am new file mode 100644 index 00000000..08170d52 --- /dev/null +++ b/plugins/mmfields/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmfields.la + +mmfields_la_SOURCES = mmfields.c +mmfields_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmfields_la_LDFLAGS = -module -avoid-version +mmfields_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmfields/mmfields.c b/plugins/mmfields/mmfields.c new file mode 100644 index 00000000..c408a6c9 --- /dev/null +++ b/plugins/mmfields/mmfields.c @@ -0,0 +1,287 @@ +/* mmfields.c + * Parse all fields of the message into structured data inside the + * JSON tree. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmfields") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +/* define operation modes we have */ +#define SIMPLE_MODE 0 /* just overwrite */ +#define REWRITE_MODE 1 /* rewrite IP address, canoninized */ +typedef struct _instanceData { + char separator; + uchar *jsonRoot; /**< container where to store fields */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "separator", eCmdHdlrGetChar, 0 }, + { "jsonroot", eCmdHdlrString, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance + free(pData->jsonRoot); +ENDfreeInstance + +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->separator = ','; + pData->jsonRoot = NULL; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmfields)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "separator")) { + pData->separator = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else if(!strcmp(actpblk.descr[i].name, "jsonroot")) { + pData->jsonRoot = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("mmfields: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + if(pData->jsonRoot == NULL) { + CHKmalloc(pData->jsonRoot = (uchar*) strdup("!")); + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +static inline rsRetVal +extractField(instanceData *pData, uchar *msgtext, int lenMsg, int *curridx, uchar *fieldbuf) +{ + int i, j; + DEFiRet; + i = *curridx; + j = 0; + while(i < lenMsg && msgtext[i] != pData->separator) { + fieldbuf[j++] = msgtext[i++]; + } + fieldbuf[j] = '\0'; + if(i < lenMsg) + ++i; + *curridx = i; + + RETiRet; +} + + +static inline rsRetVal +parse_fields(instanceData *pData, msg_t *pMsg, uchar *msgtext, int lenMsg) +{ + uchar fieldbuf[32*1024]; + uchar fieldname[512]; + struct json_object *json; + struct json_object *jval; + int field; + uchar *buf; + int currIdx = 0; + DEFiRet; + + if(lenMsg < (int) sizeof(fieldbuf)) { + buf = fieldbuf; + } else { + CHKmalloc(buf = malloc(lenMsg+1)); + } + + json = json_object_new_object(); + if(json == NULL) { + ABORT_FINALIZE(RS_RET_ERR); + } + field = 1; + while(currIdx < lenMsg) { + CHKiRet(extractField(pData, msgtext, lenMsg, &currIdx, buf)); + DBGPRINTF("mmfields: field %d: '%s'\n", field, buf); + snprintf((char*)fieldname, sizeof(fieldname), "f%d", field); + fieldname[sizeof(fieldname)-1] = '\0'; + jval = json_object_new_string((char*)fieldbuf); + json_object_object_add(json, (char*)fieldname, jval); + field++; + } + msgAddJSON(pMsg, pData->jsonRoot, json); +finalize_it: + RETiRet; +} + + +BEGINdoAction + msg_t *pMsg; + uchar *msg; + int lenMsg; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + lenMsg = getMSGLen(pMsg); + msg = getMSG(pMsg); + CHKiRet(parse_fields(pWrkrData->pData, pMsg, msg, lenMsg)); +finalize_it: +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmfields:", sizeof(":mmfields:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmfields supports only v6+ config format, use: " + "action(type=\"mmfields\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmfields: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmjsonparse/mmjsonparse.c b/plugins/mmjsonparse/mmjsonparse.c index b16aef0e..9c0ab88f 100644 --- a/plugins/mmjsonparse/mmjsonparse.c +++ b/plugins/mmjsonparse/mmjsonparse.c @@ -58,9 +58,15 @@ DEFobjCurrIf(errmsg); DEF_OMOD_STATIC_DATA typedef struct _instanceData { - struct json_tokener *tokener; + int dummy; /* not needed, but some compilers do not support empty structs */ + /* REMOVE dummy when real data items are to be added! */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; + struct json_tokener *tokener; +} wrkrInstanceData_t; + struct modConfData_s { rsconf_t *pConf; /* our overall config object */ }; @@ -94,14 +100,18 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance - pData->tokener = json_tokener_new(); - if(pData->tokener == NULL) { +ENDcreateInstance + +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + pWrkrData->tokener = json_tokener_new(); + if(pWrkrData->tokener == NULL) { errmsg.LogError(0, RS_RET_ERR, "error: could not create json " - "tokener, cannot activate action"); + "tokener, cannot activate instance"); ABORT_FINALIZE(RS_RET_ERR); } finalize_it: -ENDcreateInstance +ENDcreateWrkrInstance BEGINisCompatibleWithFeature @@ -111,10 +121,14 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - if(pData->tokener != NULL) - json_tokener_free(pData->tokener); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + if(pWrkrData->tokener != NULL) + json_tokener_free(pWrkrData->tokener); +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -128,28 +142,28 @@ ENDtryResume static rsRetVal -processJSON(instanceData *pData, msg_t *pMsg, char *buf, size_t lenBuf) +processJSON(wrkrInstanceData_t *pWrkrData, msg_t *pMsg, char *buf, size_t lenBuf) { struct json_object *json; const char *errMsg; DEFiRet; - assert(pData->tokener != NULL); + assert(pWrkrData->tokener != NULL); DBGPRINTF("mmjsonparse: toParse: '%s'\n", buf); - json_tokener_reset(pData->tokener); + json_tokener_reset(pWrkrData->tokener); - json = json_tokener_parse_ex(pData->tokener, buf, lenBuf); + json = json_tokener_parse_ex(pWrkrData->tokener, buf, lenBuf); if(Debug) { errMsg = NULL; if(json == NULL) { enum json_tokener_error err; - err = pData->tokener->err; + err = pWrkrData->tokener->err; if(err != json_tokener_continue) errMsg = json_tokener_errors[err]; else errMsg = "Unterminated input"; - } else if((size_t)pData->tokener->char_offset < lenBuf) + } else if((size_t)pWrkrData->tokener->char_offset < lenBuf) errMsg = "Extra characters after JSON object"; else if(!json_object_is_type(json, json_type_object)) errMsg = "JSON value is not an object"; @@ -159,7 +173,7 @@ processJSON(instanceData *pData, msg_t *pMsg, char *buf, size_t lenBuf) } } if(json == NULL - || ((size_t)pData->tokener->char_offset < lenBuf) + || ((size_t)pWrkrData->tokener->char_offset < lenBuf) || (!json_object_is_type(json, json_type_object))) { ABORT_FINALIZE(RS_RET_NO_CEE_MSG); } @@ -194,7 +208,7 @@ CODESTARTdoAction ABORT_FINALIZE(RS_RET_NO_CEE_MSG); } buf += LEN_COOKIE; - CHKiRet(processJSON(pData, pMsg, (char*) buf, strlen((char*)buf))); + CHKiRet(processJSON(pWrkrData, pMsg, (char*) buf, strlen((char*)buf))); bSuccess = 1; finalize_it: if(iRet == RS_RET_NO_CEE_MSG) { @@ -257,6 +271,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES ENDqueryEtryPt diff --git a/plugins/mmnormalize/mmnormalize.c b/plugins/mmnormalize/mmnormalize.c index 7e25824a..f82836b1 100644 --- a/plugins/mmnormalize/mmnormalize.c +++ b/plugins/mmnormalize/mmnormalize.c @@ -9,7 +9,7 @@ * * File begun on 2010-01-01 by RGerhards * - * Copyright 2010-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2010-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -70,6 +70,10 @@ typedef struct _instanceData { ee_ctx ctxee; /**< context to be used for libee */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { uchar *rulebase; /**< name of normalization rulebase to use */ int bUseRawMsg; /**< use %rawmsg% instead of %msg% */ @@ -139,6 +143,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINbeginCnfLoad CODESTARTbeginCnfLoad loadModConf = pModConf; @@ -181,6 +190,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("mmnormalize\n"); @@ -207,14 +221,14 @@ CODESTARTdoAction * requires changes to the libraries. For now, we accept message * duplication. -- rgerhards, 2010-12-01 */ - if(pData->bUseRawMsg) { + if(pWrkrData->pData->bUseRawMsg) { getRawMsg(pMsg, &buf, &len); } else { buf = getMSG(pMsg); len = getMSGLen(pMsg); } str = es_newStrFromCStr((char*)buf, len); - r = ln_normalize(pData->ctxln, str, &event); + r = ln_normalize(pWrkrData->pData->ctxln, str, &event); if(r != 0) { DBGPRINTF("error %d during ln_normalize\n", r); MsgSetParseSuccess(pMsg, 0); @@ -338,6 +352,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt diff --git a/plugins/mmpstrucdata/Makefile.am b/plugins/mmpstrucdata/Makefile.am new file mode 100644 index 00000000..090150ab --- /dev/null +++ b/plugins/mmpstrucdata/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmpstrucdata.la + +mmpstrucdata_la_SOURCES = mmpstrucdata.c +mmpstrucdata_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmpstrucdata_la_LDFLAGS = -module -avoid-version +mmpstrucdata_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmpstrucdata/mmpstrucdata.c b/plugins/mmpstrucdata/mmpstrucdata.c new file mode 100644 index 00000000..e9e36b27 --- /dev/null +++ b/plugins/mmpstrucdata/mmpstrucdata.c @@ -0,0 +1,413 @@ +/* mmpstrucdata.c + * Parse all fields of the message into structured data inside the + * JSON tree. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmpstrucdata") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +typedef struct _instanceData { + uchar *jsonRoot; /**< container where to store fields */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "jsonroot", eCmdHdlrString, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance + free(pData->jsonRoot); +ENDfreeInstance + +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->jsonRoot = NULL; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmpstrucdata)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "jsonroot")) { + pData->jsonRoot = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("mmpstrucdata: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + if(pData->jsonRoot == NULL) { + CHKmalloc(pData->jsonRoot = (uchar*) strdup("!")); + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +static inline rsRetVal +parsePARAM_VALUE(uchar *sdbuf, int lenbuf, int *curridx, uchar *fieldbuf) +{ + int i, j; + DEFiRet; +dbgprintf("DDDD: parsePARAM_VALUE\n"); + i = *curridx; + j = 0; + while(i < lenbuf && sdbuf[i] != '"') { + if(sdbuf[i] == '\\') { + if(++i == lenbuf) { + fieldbuf[j++] = '\\'; + } else { + if(sdbuf[i] == '"') { + fieldbuf[j++] = '"'; + } else if(sdbuf[i] == '\\') { + fieldbuf[j++] = '\\'; + } else if(sdbuf[i] == ']') { + fieldbuf[j++] = '"'; + } else { + fieldbuf[j++] = '\\'; + fieldbuf[j++] = sdbuf[i]; + } + ++i; + } + } else { + fieldbuf[j++] = sdbuf[i++]; + } + } + fieldbuf[j] = '\0'; + *curridx = i; +dbgprintf("DDDD: parsePARAM_VALUE: '%s'\n", fieldbuf); + RETiRet; +} + + +static rsRetVal +parseSD_NAME(uchar *sdbuf, int lenbuf, int *curridx, uchar *namebuf) +{ + int i, j; + DEFiRet; +dbgprintf("DDDD: parseSD_NAME %s\n", sdbuf+*curridx); + i = *curridx; + for(j = 0 ; i < lenbuf && j < 32; ++j) { + if( sdbuf[i] == '=' || sdbuf[i] == '"' + || sdbuf[i] == ']' || sdbuf[i] == ' ') + break; + namebuf[j] = sdbuf[i++]; + } + namebuf[j] = '\0'; +dbgprintf("DDDD: parseSD_NAME, NAME: '%s'\n", namebuf); + *curridx = i; + RETiRet; +} + + +static inline rsRetVal +parseSD_PARAM(instanceData *pData, uchar *sdbuf, int lenbuf, int *curridx, struct json_object *jroot) +{ + int i; + uchar pName[33]; + uchar pVal[32*1024]; + struct json_object *jval; + DEFiRet; +dbgprintf("DDDD: parseSD_PARAM %s\n", sdbuf+*curridx); + + i = *curridx; + CHKiRet(parseSD_NAME(sdbuf, lenbuf, &i, pName)); + if(sdbuf[i] != '=') { + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; + if(sdbuf[i] != '"') { + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; + CHKiRet(parsePARAM_VALUE(sdbuf, lenbuf, &i, pVal)); + if(sdbuf[i] != '"') { + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; + + jval = json_object_new_string((char*)pVal); + json_object_object_add(jroot, (char*)pName, jval); + + *curridx = i; +finalize_it: + RETiRet; +} + + +static inline rsRetVal +parseSD_ELEMENT(instanceData *pData, uchar *sdbuf, int lenbuf, int *curridx, struct json_object *jroot) +{ + int i; + uchar sd_id[33]; + struct json_object *json; + DEFiRet; +dbgprintf("DDDD: parseSD_ELEMENT: %s\n", sdbuf+*curridx); + + i = *curridx; + if(sdbuf[i] != '[') { + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; /* eat '[' */ + + CHKiRet(parseSD_NAME(sdbuf, lenbuf, &i, sd_id)); + json = json_object_new_object(); + + while(i < lenbuf) { + if(sdbuf[i] == ']') { + break; + } else if(sdbuf[i] != ' ') { + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; + while(i < lenbuf && sdbuf[i] == ' ') + ++i; + CHKiRet(parseSD_PARAM(pData, sdbuf, lenbuf, &i, json)); +dbgprintf("DDDD: done parseSD_PARAM, in loop, i:%d, lenbuf:%d, rest: %s\n", i, lenbuf, sdbuf+i); + } + + if(sdbuf[i] != ']') { + DBGPRINTF("mmpstrucdata: SD-ELEMENT does not terminate with " + "']': '%s'\n", sdbuf+i); + ABORT_FINALIZE(RS_RET_STRUC_DATA_INVLD); + } + ++i; /* eat ']' */ + *curridx = i; + json_object_object_add(jroot, (char*)sd_id, json); +dbgprintf("DDDD: SD_ELEMENT: json: '%s'\n", json_object_get_string(json)); +dbgprintf("DDDD: SD_ELEMENT: jroot '%s'\n", json_object_get_string(json)); +finalize_it: +dbgprintf("DDDD: parseSD_ELEMENT iRet:%d, i:%d, *curridx:%d\n", iRet, i, *curridx); + RETiRet; +} + +static inline rsRetVal +parse_sd(instanceData *pData, msg_t *pMsg) +{ +#if 0 + uchar fieldbuf[32*1024]; + uchar fieldname[512]; + struct json_object *json; + struct json_object *jval; + int field; + uchar *buf; +#endif + struct json_object *json, *jroot; + uchar *sdbuf; + int lenbuf; + int i = 0; + DEFiRet; + +#if 0 + if(lenMsg < (int) sizeof(fieldbuf)) { + buf = fieldbuf; + } else { + CHKmalloc(buf = malloc(lenMsg+1)); + } +#endif + +dbgprintf("DDDD: parse_sd\n"); + json = json_object_new_object(); + if(json == NULL) { + ABORT_FINALIZE(RS_RET_ERR); + } + MsgGetStructuredData(pMsg, &sdbuf,&lenbuf); + while(i < lenbuf) { + CHKiRet(parseSD_ELEMENT(pData, sdbuf, lenbuf, &i, json)); +dbgprintf("DDDD: parse_sd, i:%d\n", i); + } +dbgprintf("DDDD: json: '%s'\n", json_object_get_string(json)); + + jroot = json_object_new_object(); + if(jroot == NULL) { + ABORT_FINALIZE(RS_RET_ERR); + } + json_object_object_add(jroot, "RFC5424-SD", json); + msgAddJSON(pMsg, pData->jsonRoot, jroot); +finalize_it: + RETiRet; +} + + +BEGINdoAction + msg_t *pMsg; +CODESTARTdoAction +dbgprintf("DDDD: enter mmpstrucdata\n"); + pMsg = (msg_t*) ppString[0]; + if(!MsgHasStructuredData(pMsg)) { + DBGPRINTF("mmpstrucdata: message does not have structured data\n"); + FINALIZE; + } +dbgprintf("DDDD: parse mmpstrucdata\n"); + /* don't check return code - we never want rsyslog to retry + * or suspend this action! + */ + parse_sd(pWrkrData->pData, pMsg); +dbgprintf("DDDD: done parse mmpstrucdata\n"); +finalize_it: +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmpstrucdata:", sizeof(":mmpstrucdata:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmpstrucdata supports only v6+ config format, use: " + "action(type=\"mmpstrucdata\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmpstrucdata: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmrfc5424addhmac/Makefile.am b/plugins/mmrfc5424addhmac/Makefile.am new file mode 100644 index 00000000..6567def0 --- /dev/null +++ b/plugins/mmrfc5424addhmac/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmrfc5424addhmac.la + +mmrfc5424addhmac_la_SOURCES = mmrfc5424addhmac.c +mmrfc5424addhmac_la_CPPFLAGS = $(OPENSSL_CFLAGS) $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmrfc5424addhmac_la_LDFLAGS = -module -avoid-version +mmrfc5424addhmac_la_LIBADD = $(OPENSSL_LIBS) + +EXTRA_DIST = diff --git a/plugins/mmrfc5424addhmac/mmrfc5424addhmac.c b/plugins/mmrfc5424addhmac/mmrfc5424addhmac.c new file mode 100644 index 00000000..a7aea9b4 --- /dev/null +++ b/plugins/mmrfc5424addhmac/mmrfc5424addhmac.c @@ -0,0 +1,379 @@ +/* mmrfc5424addhmac.c + * custom module: add hmac to RFC5424 messages + * + * Note on important design decision: This module is fully self-contained. + * Most importantly, it does not rely on mmpstrucdata to populate the + * structured data portion of the messages JSON. There are two reasons + * for this: + * 1. robustness + * - this guard against misconfiguration + * - it permits us to be more liberal in regard to malformed + * structured data + * - it permits us to handle border-cases (like duplicate + * SD-IDs) with much less complexity + * 2. performance + * With being "on the spot" of what we need we can reduce memory + * reads and writes. This is a considerable save if the JSON representation + * is not otherwise needed. + * + * Note that the recommended calling sequence if both of these modules + * are used is + * + * 1. mmrfc5424addhmac + * 2. mmpstrucdata + * + * This sequence permits mmpstrucdata to pick up the modifications we + * made in this module here. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include <openssl/hmac.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmrfc5424addhmac") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +typedef struct _instanceData { + uchar *key; + int16_t keylen; /* cached length of key, to avoid recomputation */ + uchar *sdid; /* SD-ID to be used to persist the hmac */ + int16_t sdidLen; + const EVP_MD *algo; +} instanceData; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "key", eCmdHdlrString, 1 }, + { "hashfunction", eCmdHdlrString, 1 }, + { "sd_id", eCmdHdlrGetWord, 1 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->key = NULL; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + char *ciphername; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmrfc5424addhmac)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "replacementchar")) { + pData->key = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + pData->keylen = es_strlen(pvals[i].val.d.estr); + } else if(!strcmp(actpblk.descr[i].name, "hashfunction")) { + ciphername = es_str2cstr(pvals[i].val.d.estr, NULL); + pData->algo = EVP_get_digestbyname(ciphername); + if(pData->algo == NULL) { + errmsg.LogError(0, RS_RET_CRY_INVLD_ALGO, + "hashFunction '%s' unknown to openssl - " + "cannot continue", ciphername); + free(ciphername); + ABORT_FINALIZE(RS_RET_CRY_INVLD_ALGO); + } + free(ciphername); + } else if(!strcmp(actpblk.descr[i].name, "sd_id")) { + pData->sdid = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + pData->sdidLen = es_strlen(pvals[i].val.d.estr); + } else { + dbgprintf("mmrfc5424addhmac: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +/* turn the binary data in bin of length len into a + * printable hex string. "print" must be 2*len+1 (for \0) + */ +static inline void +hexify(uchar *bin, int len, uchar *print) +{ + static const char hexchars[16] = + {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'}; + int iSrc, iDst; + + for(iSrc = iDst = 0 ; iSrc < len ; ++iSrc) { + print[iDst++] = hexchars[bin[iSrc]>>4]; + print[iDst++] = hexchars[bin[iSrc]&0x0f]; + } + print[iDst] = '\0'; +} + + +/* skip to end of current SD-ID. This function can be improved + * in regard to fully parsing based on RFC5424, HOWEVER, this would + * also reduce performance. So we consider the current implementation + * to be superior. + */ +static inline void +skipSDID(uchar *sdbuf, int sdlen, int *rootIdx) +{ + int i; + i = *rootIdx; + while(i < sdlen) { + if(sdbuf[i] == ']') { + if(i > *rootIdx && sdbuf[i-1] == '\\') { + ; /* escaped, nothing to do! */ + } else { + ++i; /* eat ']' */ + break; + } + } + ++i; + } + *rootIdx = i; +} + +static inline void +getSDID(uchar *sdbuf, int sdlen, int *rootIdx, uchar *sdid) +{ + int i, j; + i = *rootIdx; + j = 0; + + if(sdbuf[i] != '[') { + ++i; + goto done; + } + + ++i; + while(i < sdlen && sdbuf[i] != '=' && sdbuf[i] != ' ' + && sdbuf[i] != ']' && sdbuf[i] != '"') { + sdid[j++] = sdbuf[i++]; + } +done: + sdid[j] = '\0'; + *rootIdx = i; +} + +/* check if "our" hmac is already present */ +static inline sbool +isHmacPresent(instanceData *pData, msg_t *pMsg) +{ + uchar *sdbuf; + rs_size_t sdlen; + sbool found; + int i; + uchar sdid[33]; /* RFC-based size limit */ + + MsgGetStructuredData(pMsg, &sdbuf, &sdlen); + found = 0; + + if(sdbuf[0] == '-') /* RFC: struc data is empty! */ + goto done; + + i = 0; + while(i < sdlen && !found) { + getSDID(sdbuf, sdlen, &i, sdid); + if(!strcmp((char*)pData->sdid, (char*)sdid)) { + found = 1; + break; + } + skipSDID(sdbuf, sdlen, &i); + } + +done: + return found; +} + +static inline rsRetVal +hashMsg(instanceData *pData, msg_t *pMsg) +{ + uchar *pRawMsg; + int lenRawMsg; + uchar *sdbuf; + rs_size_t sdlen; + unsigned int hashlen; + uchar hash[EVP_MAX_MD_SIZE]; + uchar hashPrintable[2*EVP_MAX_MD_SIZE+1]; + uchar newsd[64*1024]; /* we assume this is sufficient... */ + int lenNewsd; + DEFiRet; + + MsgGetStructuredData(pMsg, &sdbuf, &sdlen); + getRawMsg(pMsg, &pRawMsg, &lenRawMsg); + HMAC(pData->algo, pData->key, pData->keylen, + pRawMsg, lenRawMsg, hash, &hashlen); + hexify(hash, hashlen, hashPrintable); + lenNewsd = snprintf((char*)newsd, sizeof(newsd), "[%s hash=\"%s\"]", + (char*)pData->sdid, (char*)hashPrintable); + MsgAddToStructuredData(pMsg, newsd, lenNewsd); + RETiRet; +} + + +BEGINdoAction + msg_t *pMsg; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + if( msgGetProtocolVersion(pMsg) == MSG_RFC5424_PROTOCOL + && !isHmacPresent(pData, pMsg)) { + hashMsg(pData, pMsg); + } else { + if(Debug) { + uchar *pRawMsg; + int lenRawMsg; + getRawMsg(pMsg, &pRawMsg, &lenRawMsg); + dbgprintf("mmrfc5424addhmac: non-rfc5424 or HMAC already " + "present: %.256s\n", pRawMsg); + } + } +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmrfc5424addhmac:", sizeof(":mmrfc5424addhmac:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmrfc5424addhmac supports only v6+ config format, use: " + "action(type=\"mmrfc5424addhmac\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); + EVP_cleanup(); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmrfc5424addhmac: module compiled with rsyslog version %s.\n", VERSION); + OpenSSL_add_all_digests(); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmsequence/Makefile.am b/plugins/mmsequence/Makefile.am new file mode 100644 index 00000000..543d6d84 --- /dev/null +++ b/plugins/mmsequence/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmsequence.la + +mmsequence_la_SOURCES = mmsequence.c +mmsequence_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmsequence_la_LDFLAGS = -module -avoid-version +mmsequence_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmsequence/mmsequence.c b/plugins/mmsequence/mmsequence.c new file mode 100644 index 00000000..609338c2 --- /dev/null +++ b/plugins/mmsequence/mmsequence.c @@ -0,0 +1,420 @@ +/* mmsequence.c + * Generate a number based on some sequence. + * + * Copyright 2013 pavel@levshin.spb.ru. + * + * Based on: mmcount.c + * Copyright 2013 Red Hat Inc. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include <time.h> +#include <limits.h> +#include <json/json.h> +#include <pthread.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include "hashtable.h" + +#define JSON_VAR_NAME "$!mmsequence" + +enum mmSequenceModes { + mmSequenceRandom, + mmSequencePerInstance, + mmSequencePerKey +}; + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmsequence") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* config variables */ + +typedef struct _instanceData { + enum mmSequenceModes mode; + int valueFrom; + int valueTo; + int step; + unsigned int seed; + int value; + char *pszKey; + char *pszVar; +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "mode", eCmdHdlrGetWord, 0 }, + { "from", eCmdHdlrNonNegInt, 0 }, + { "to", eCmdHdlrPositiveInt, 0 }, + { "step", eCmdHdlrNonNegInt, 0 }, + { "key", eCmdHdlrGetWord, 0 }, + { "var", eCmdHdlrGetWord, 0 }, +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +/* table for key-counter pairs */ +static struct hashtable *ght; +static pthread_mutex_t ght_mutex = PTHREAD_MUTEX_INITIALIZER; + +static pthread_mutex_t inst_mutex = PTHREAD_MUTEX_INITIALIZER; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->mode = mmSequencePerInstance; + pData->valueFrom = 0; + pData->valueTo = INT_MAX; + pData->step = 1; + pData->pszKey = ""; + pData->pszVar = JSON_VAR_NAME; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; + char *cstr; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmsequence)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "mode")) { + if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"random", + sizeof("random")-1)) { + pData->mode = mmSequenceRandom; + } else if (!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"instance", + sizeof("instance")-1)) { + pData->mode = mmSequencePerInstance; + } else if (!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"key", + sizeof("key")-1)) { + pData->mode = mmSequencePerKey; + } else { + cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_INVLD_MODE, + "mmsequence: invalid mode '%s' - ignored", + cstr); + free(cstr); + } + continue; + } + if(!strcmp(actpblk.descr[i].name, "from")) { + pData->valueFrom = pvals[i].val.d.n; + continue; + } + if(!strcmp(actpblk.descr[i].name, "to")) { + pData->valueTo = pvals[i].val.d.n; + continue; + } + if(!strcmp(actpblk.descr[i].name, "step")) { + pData->step = pvals[i].val.d.n; + continue; + } + if(!strcmp(actpblk.descr[i].name, "key")) { + pData->pszKey = es_str2cstr(pvals[i].val.d.estr, NULL); + continue; + } + if(!strcmp(actpblk.descr[i].name, "var")) { + cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + if (strlen(cstr) < 3) { + errmsg.LogError(0, RS_RET_VALUE_NOT_SUPPORTED, + "mmsequence: valid variable name should be at least " + "3 symbols long, got %s", cstr); + free(cstr); + } else if (cstr[0] != '$') { + errmsg.LogError(0, RS_RET_VALUE_NOT_SUPPORTED, + "mmsequence: valid variable name should start with $," + "got %s", cstr); + free(cstr); + } else { + pData->pszVar = cstr; + } + continue; + } + dbgprintf("mmsequence: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + switch(pData->mode) { + case mmSequenceRandom: + pData->seed = (unsigned int)(intptr_t)pData ^ (unsigned int)time(NULL); + break; + case mmSequencePerInstance: + pData->value = pData->valueTo; + break; + case mmSequencePerKey: + if (pthread_mutex_lock(&ght_mutex)) { + DBGPRINTF("mmsequence: mutex lock has failed!\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + if (ght == NULL) { + if(NULL == (ght = create_hashtable(100, hash_from_string, key_equals_string, NULL))) { + pthread_mutex_unlock(&ght_mutex); + DBGPRINTF("mmsequence: error creating hash table!\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + } + pthread_mutex_unlock(&ght_mutex); + break; + default: + errmsg.LogError(0, RS_RET_INVLD_MODE, + "mmsequence: this mode is not currently implemented"); + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +static int * +getCounter(struct hashtable *ht, char *str, int initial) { + int *pCounter; + char *pStr; + + pCounter = hashtable_search(ht, str); + if(pCounter) { + return pCounter; + } + + /* counter is not found for the str, so add new entry and + return the counter */ + if(NULL == (pStr = strdup(str))) { + DBGPRINTF("mmsequence: memory allocation for key failed\n"); + return NULL; + } + + if(NULL == (pCounter = (int*)malloc(sizeof(*pCounter)))) { + DBGPRINTF("mmsequence: memory allocation for value failed\n"); + free(pStr); + return NULL; + } + *pCounter = initial; + + if(!hashtable_insert(ht, pStr, pCounter)) { + DBGPRINTF("mmsequence: inserting element into hashtable failed\n"); + free(pStr); + free(pCounter); + return NULL; + } + return pCounter; +} + + +BEGINdoAction + msg_t *pMsg; + struct json_object *json; + int val = 0; + int *pCounter; + instanceData *pData; +CODESTARTdoAction + pData = pWrkrData->pData; + pMsg = (msg_t*) ppString[0]; + + switch(pData->mode) { + case mmSequenceRandom: + val = pData->valueFrom + (rand_r(&pData->seed) % + (pData->valueTo - pData->valueFrom)); + break; + case mmSequencePerInstance: + if (!pthread_mutex_lock(&inst_mutex)) { + pCounter = getCounter(ght, pData->pszKey, pData->valueTo); + if (pData->value >= pData->valueTo - pData->step) { + pData->value = pData->valueFrom; + } else { + pData->value += pData->step; + } + val = pData->value; + pthread_mutex_unlock(&inst_mutex); + } else { + errmsg.LogError(0, RS_RET_ERR, + "mmsequence: mutex lock has failed!"); + } + break; + case mmSequencePerKey: + if (!pthread_mutex_lock(&ght_mutex)) { + pCounter = getCounter(ght, pData->pszKey, pData->valueTo); + if(pCounter) { + if (*pCounter >= pData->valueTo - pData->step + || *pCounter < pData->valueFrom ) { + *pCounter = pData->valueFrom; + } else { + *pCounter += pData->step; + } + val = *pCounter; + } else { + errmsg.LogError(0, RS_RET_NOT_FOUND, + "mmsequence: unable to fetch the counter from hash"); + } + pthread_mutex_unlock(&ght_mutex); + } else { + errmsg.LogError(0, RS_RET_ERR, + "mmsequence: mutex lock has failed!"); + } + + break; + default: + errmsg.LogError(0, RS_RET_NOT_IMPLEMENTED, + "mmsequence: this mode is not currently implemented"); + } + + /* finalize_it: */ + json = json_object_new_int(val); + if (json == NULL) { + errmsg.LogError(0, RS_RET_OBJ_CREATION_FAILED, + "mmsequence: unable to create JSON"); + } else if (RS_RET_OK != msgAddJSON(pMsg, (uchar *)pData->pszVar + 1, json)) { + errmsg.LogError(0, RS_RET_OBJ_CREATION_FAILED, + "mmsequence: unable to pass out the value"); + json_object_put(json); + } +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmsequence:", sizeof(":mmsequence:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmsequence supports only v6+ config format, use: " + "action(type=\"mmsequence\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmsequence: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/mmutf8fix/Makefile.am b/plugins/mmutf8fix/Makefile.am new file mode 100644 index 00000000..2c0f283a --- /dev/null +++ b/plugins/mmutf8fix/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = mmutf8fix.la + +mmutf8fix_la_SOURCES = mmutf8fix.c +mmutf8fix_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +mmutf8fix_la_LDFLAGS = -module -avoid-version +mmutf8fix_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/mmutf8fix/mmutf8fix.c b/plugins/mmutf8fix/mmutf8fix.c new file mode 100644 index 00000000..351bb129 --- /dev/null +++ b/plugins/mmutf8fix/mmutf8fix.c @@ -0,0 +1,333 @@ +/* mmutf8fix.c + * fix invalid UTF8 sequences. This is begun as a very simple replacer + * of non-control characters, and actually breaks some UTF-8 encoding + * right now. If the module turns out to be useful, it should be enhanced + * to support modes that really detect invalid UTF8. In the longer term + * it could also be evolved into an any-charset-to-UTF8 converter. But + * first let's see if it really gets into widespread enough use. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include <stdint.h> +#include "conf.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT +MODULE_TYPE_NOKEEP +MODULE_CNFNAME("mmutf8fix") + + +DEFobjCurrIf(errmsg); +DEF_OMOD_STATIC_DATA + +/* define operation modes we have */ +#define MODE_CC 0 /* just fix control characters */ +#define MODE_UTF8 1 /* do real UTF-8 fixing */ + +/* config variables */ +typedef struct _instanceData { + uchar replChar; + uint8_t mode; /* operations mode */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + +struct modConfData_s { + rsconf_t *pConf; /* our overall config object */ +}; +static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ +static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ + + +/* tables for interfacing with the v6 config system */ +/* action (instance) parameters */ +static struct cnfparamdescr actpdescr[] = { + { "mode", eCmdHdlrGetWord, 0 }, + { "replacementchar", eCmdHdlrGetChar, 0 } +}; +static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, + sizeof(actpdescr)/sizeof(struct cnfparamdescr), + actpdescr + }; + +BEGINbeginCnfLoad +CODESTARTbeginCnfLoad + loadModConf = pModConf; + pModConf->pConf = pConf; +ENDbeginCnfLoad + +BEGINendCnfLoad +CODESTARTendCnfLoad +ENDendCnfLoad + +BEGINcheckCnf +CODESTARTcheckCnf +ENDcheckCnf + +BEGINactivateCnf +CODESTARTactivateCnf + runModConf = pModConf; +ENDactivateCnf + +BEGINfreeCnf +CODESTARTfreeCnf +ENDfreeCnf + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + +static inline void +setInstParamDefaults(instanceData *pData) +{ + pData->mode = MODE_UTF8; + pData->replChar = ' '; +} + +BEGINnewActInst + struct cnfparamvals *pvals; + int i; +CODESTARTnewActInst + DBGPRINTF("newActInst (mmutf8fix)\n"); + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + + CODE_STD_STRING_REQUESTnewActInst(1) + CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + CHKiRet(createInstance(&pData)); + setInstParamDefaults(pData); + + for(i = 0 ; i < actpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(actpblk.descr[i].name, "mode")) { + if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"utf-8", + sizeof("utf-8")-1)) { + pData->mode = MODE_UTF8; + } else if(!es_strbufcmp(pvals[i].val.d.estr, (uchar*)"controlcharacters", + sizeof("controlcharacters")-1)) { + pData->mode = MODE_CC; + } else { + char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + errmsg.LogError(0, RS_RET_INVLD_MODE, + "mmutf8fix: invalid mode '%s' - ignored", + cstr); + free(cstr); + } + } else if(!strcmp(actpblk.descr[i].name, "replacementchar")) { + pData->replChar = es_getBufAddr(pvals[i].val.d.estr)[0]; + } else { + dbgprintf("mmutf8fix: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); + } + } + +CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); +ENDnewActInst + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + + +static inline void +doCC(instanceData *pData, uchar *msg, int lenMsg) +{ + int i; + + for(i = 0 ; i < lenMsg ; ++i) { + if(msg[i] < 32 || msg[i] > 126) { + msg[i] = pData->replChar; + } + } +} + +/* fix an invalid multibyte sequence */ +static inline void +fixInvldMBSeq(instanceData *pData, uchar *msg, int lenMsg, int strtIdx, int *endIdx, int8_t seqLen) +{ + int i; + + *endIdx = strtIdx + seqLen; + if(*endIdx > lenMsg) + *endIdx = lenMsg; + for(i = strtIdx ; i < *endIdx ; ++i) + msg[i] = pData->replChar; +} + +static inline void +doUTF8(instanceData *pData, uchar *msg, int lenMsg) +{ + uchar c; + int8_t seqLen, bytesLeft = 0; + uint32_t codepoint; + int strtIdx, endIdx; + int i; + + for(i = 0 ; i < lenMsg ; ++i) { + c = msg[i]; + if(bytesLeft) { + if((c & 0xc0) != 0x80) { + /* sequence invalid, invalidate all bytes */ + fixInvldMBSeq(pData, msg, lenMsg, strtIdx, &endIdx, + seqLen); + i = endIdx - 1; + bytesLeft = 0; + } else { + codepoint = (codepoint << 6) | (c & 0x3f); + --bytesLeft; + if(bytesLeft == 0) { + /* too-large codepoint? */ + if(codepoint > 0x10FFFF) { + fixInvldMBSeq(pData, msg, lenMsg, + strtIdx, &endIdx, + seqLen); + } + } + } + } else { + if((c & 0x80) == 0) { + /* 1-byte sequence, US-ASCII */ + ; /* nothing to do, all well */ + } else if((c & 0xe0) == 0xc0) { + /* 2-byte sequence */ + strtIdx = i; + seqLen = bytesLeft = 1; + codepoint = c & 0x1f; + } else if((c & 0xf0) == 0xe0) { + /* 3-byte sequence */ + strtIdx = i; + seqLen = bytesLeft = 2; + codepoint = c & 0x0f; + } else if((c & 0xf8) == 0xf0) { + /* 4-byte sequence */ + strtIdx = i; + seqLen = bytesLeft = 3; + codepoint = c & 0x07; + } else { /* invalid (5&6 byte forbidden by RFC3629) */ + msg[i] = pData->replChar; + } + if(i+bytesLeft >= lenMsg) { + int dummy = lenMsg; + /* invalid, as rest of message cannot contain full char */ + fixInvldMBSeq(pData, msg, lenMsg, strtIdx, &dummy, seqLen); + i = lenMsg - 1; + } + } + } +} + +BEGINdoAction + msg_t *pMsg; + uchar *msg; + int lenMsg; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + lenMsg = getMSGLen(pMsg); + msg = getMSG(pMsg); + if(pWrkrData->pData->mode == MODE_CC) { + doCC(pWrkrData->pData, msg, lenMsg); + } else { + doUTF8(pWrkrData->pData, msg, lenMsg); + } +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + if(strncmp((char*) p, ":mmutf8fix:", sizeof(":mmutf8fix:") - 1)) { + errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, + "mmutf8fix supports only v6+ config format, use: " + "action(type=\"mmutf8fix\" ...)"); + } + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES +CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_STD_CONF2_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + DBGPRINTF("mmutf8fix: module compiled with rsyslog version %s.\n", VERSION); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit diff --git a/plugins/omelasticsearch/README b/plugins/omelasticsearch/README index 9021bc0e..b8bf4151 100644 --- a/plugins/omelasticsearch/README +++ b/plugins/omelasticsearch/README @@ -1,3 +1,7 @@ +How to access ElasticSearch on local machine (for testing): +=========================================================== +see: https://github.com/mobz/elasticsearch-head + How to produce an error: ======================== It's quite easy to get 400, if you put a wrong mapping to your diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c index b82968d0..b878050d 100644 --- a/plugins/omelasticsearch/omelasticsearch.c +++ b/plugins/omelasticsearch/omelasticsearch.c @@ -69,8 +69,8 @@ STATSCOUNTER_DEF(indexESFail, mutIndexESFail) typedef struct curl_slist HEADER; typedef struct _instanceData { int port; - int replyLen; int fdErrFile; /* error file fd or -1 if not open */ + pthread_mutex_t mutErrFile; uchar *server; uchar *uid; uchar *pwd; @@ -80,24 +80,29 @@ typedef struct _instanceData { uchar *tplName; uchar *timeout; uchar *bulkId; - uchar *restURL; /* last used URL for error reporting */ uchar *errorFile; - char *reply; sbool dynSrchIdx; sbool dynSrchType; sbool dynParent; sbool dynBulkId; sbool bulkmode; sbool asyncRepl; +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; + int replyLen; + char *reply; + CURL *curlHandle; /* libcurl session handle */ + HEADER *postHeader; /* json POST request info */ + uchar *restURL; /* last used URL for error reporting */ struct { es_str_t *data; int nmemb; /* number of messages in batch (for statistics counting) */ uchar *currTpl1; uchar *currTpl2; } batch; - CURL *curlHandle; /* libcurl session handle */ - HEADER *postHeader; /* json POST request info */ -} instanceData; +} wrkrInstanceData_t; /* tables for interfacing with the v6 config system */ @@ -117,7 +122,7 @@ static struct cnfparamdescr actpdescr[] = { { "asyncrepl", eCmdHdlrBinary, 0 }, { "timeout", eCmdHdlrGetWord, 0 }, { "errorfile", eCmdHdlrGetWord, 0 }, - { "template", eCmdHdlrGetWord, 1 }, + { "template", eCmdHdlrGetWord, 0 }, { "dynbulkid", eCmdHdlrBinary, 0 }, { "bulkid", eCmdHdlrGetWord, 0 }, }; @@ -127,12 +132,32 @@ static struct cnfparamblk actpblk = actpdescr }; +static rsRetVal curlSetup(wrkrInstanceData_t *pWrkrData, instanceData *pData); + BEGINcreateInstance CODESTARTcreateInstance - pData->restURL = NULL; pData->fdErrFile = -1; + pthread_mutex_init(&pData->mutErrFile, NULL); ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +dbgprintf("omelasticsearch: createWrkrInstance\n"); + pWrkrData->restURL = NULL; + if(pData->bulkmode) { + pWrkrData->batch.currTpl1 = NULL; + pWrkrData->batch.currTpl2 = NULL; + if((pWrkrData->batch.data = es_newStr(1024)) == NULL) { + DBGPRINTF("omelasticsearch: error creating batch string " + "turned off bulk mode\n"); + pData->bulkmode = 0; /* at least it works */ + } + } + CHKiRet(curlSetup(pWrkrData, pWrkrData->pData)); +finalize_it: +dbgprintf("DDDD: createWrkrInstance,pData %p/%p, pWrkrData %p\n", pData, pWrkrData->pData, pWrkrData); +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -141,16 +166,9 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - if (pData->postHeader) { - curl_slist_free_all(pData->postHeader); - pData->postHeader = NULL; - } - if (pData->curlHandle) { - curl_easy_cleanup(pData->curlHandle); - pData->curlHandle = NULL; - } if(pData->fdErrFile != -1) close(pData->fdErrFile); + pthread_mutex_destroy(&pData->mutErrFile); free(pData->server); free(pData->uid); free(pData->pwd); @@ -159,11 +177,23 @@ CODESTARTfreeInstance free(pData->parent); free(pData->tplName); free(pData->timeout); - free(pData->restURL); free(pData->errorFile); free(pData->bulkId); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + if(pWrkrData->postHeader) { + curl_slist_free_all(pWrkrData->postHeader); + pWrkrData->postHeader = NULL; + } + if(pWrkrData->curlHandle) { + curl_easy_cleanup(pWrkrData->curlHandle); + pWrkrData->curlHandle = NULL; + } + free(pWrkrData->restURL); +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("omelasticsearch\n"); @@ -211,7 +241,7 @@ setBaseURL(instanceData *pData, es_str_t **url) static inline rsRetVal -checkConn(instanceData *pData) +checkConn(wrkrInstanceData_t *pWrkrData) { es_str_t *url; CURL *curl = NULL; @@ -219,7 +249,7 @@ checkConn(instanceData *pData) char *cstr; DEFiRet; - setBaseURL(pData, &url); + setBaseURL(pWrkrData->pData, &url); curl = curl_easy_init(); if(curl == NULL) { DBGPRINTF("omelasticsearch: checkConn() curl_easy_init() failed\n"); @@ -235,16 +265,16 @@ checkConn(instanceData *pData) curl_easy_setopt(curl, CURLOPT_URL, cstr); free(cstr); - pData->reply = NULL; - pData->replyLen = 0; - curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); + pWrkrData->reply = NULL; + pWrkrData->replyLen = 0; + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pWrkrData); res = curl_easy_perform(curl); if(res != CURLE_OK) { DBGPRINTF("omelasticsearch: checkConn() curl_easy_perform() " "failed: %s\n", curl_easy_strerror(res)); ABORT_FINALIZE(RS_RET_SUSPENDED); } - free(pData->reply); + free(pWrkrData->reply); DBGPRINTF("omelasticsearch: checkConn() completed with success\n"); finalize_it: @@ -257,7 +287,7 @@ finalize_it: BEGINtryResume CODESTARTtryResume DBGPRINTF("omelasticsearch: tryResume called\n"); - iRet = checkConn(pData); + iRet = checkConn(pWrkrData); ENDtryResume @@ -330,7 +360,7 @@ getIndexTypeAndParent(instanceData *pData, uchar **tpls, static rsRetVal -setCurlURL(instanceData *pData, uchar **tpls) +setCurlURL(wrkrInstanceData_t *pWrkrData, instanceData *pData, uchar **tpls) { char authBuf[1024]; uchar *searchIndex; @@ -368,11 +398,11 @@ setCurlURL(instanceData *pData, uchar **tpls) if(r == 0) r = es_addBuf(&url, (char*)parent, ustrlen(parent)); } - free(pData->restURL); - pData->restURL = (uchar*)es_str2cstr(url, NULL); - curl_easy_setopt(pData->curlHandle, CURLOPT_URL, pData->restURL); + free(pWrkrData->restURL); + pWrkrData->restURL = (uchar*)es_str2cstr(url, NULL); + curl_easy_setopt(pWrkrData->curlHandle, CURLOPT_URL, pWrkrData->restURL); es_deleteStr(url); - DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", pData->restURL); + DBGPRINTF("omelasticsearch: using REST URL: '%s'\n", pWrkrData->restURL); if(pData->uid != NULL) { rLocal = snprintf(authBuf, sizeof(authBuf), "%s:%s", pData->uid, @@ -383,8 +413,8 @@ setCurlURL(instanceData *pData, uchar **tpls) rLocal); ABORT_FINALIZE(RS_RET_ERR); } - curl_easy_setopt(pData->curlHandle, CURLOPT_USERPWD, authBuf); - curl_easy_setopt(pData->curlHandle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); + curl_easy_setopt(pWrkrData->curlHandle, CURLOPT_USERPWD, authBuf); + curl_easy_setopt(pWrkrData->curlHandle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); } finalize_it: RETiRet; @@ -396,7 +426,7 @@ finalize_it: * index changes. */ static rsRetVal -buildBatch(instanceData *pData, uchar *message, uchar **tpls) +buildBatch(wrkrInstanceData_t *pWrkrData, uchar *message, uchar **tpls) { int length = strlen((char *)message); int r; @@ -411,29 +441,29 @@ buildBatch(instanceData *pData, uchar *message, uchar **tpls) # define META_ID "\", \"_id\":\"" # define META_END "\"}}\n" - getIndexTypeAndParent(pData, tpls, &searchIndex, &searchType, &parent, &bulkId); - r = es_addBuf(&pData->batch.data, META_STRT, sizeof(META_STRT)-1); - if(r == 0) r = es_addBuf(&pData->batch.data, (char*)searchIndex, + getIndexTypeAndParent(pWrkrData->pData, tpls, &searchIndex, &searchType, &parent, &bulkId); + r = es_addBuf(&pWrkrData->batch.data, META_STRT, sizeof(META_STRT)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)searchIndex, ustrlen(searchIndex)); - if(r == 0) r = es_addBuf(&pData->batch.data, META_TYPE, sizeof(META_TYPE)-1); - if(r == 0) r = es_addBuf(&pData->batch.data, (char*)searchType, + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, META_TYPE, sizeof(META_TYPE)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)searchType, ustrlen(searchType)); if(parent != NULL) { - if(r == 0) r = es_addBuf(&pData->batch.data, META_PARENT, sizeof(META_PARENT)-1); - if(r == 0) r = es_addBuf(&pData->batch.data, (char*)parent, ustrlen(parent)); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, META_PARENT, sizeof(META_PARENT)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)parent, ustrlen(parent)); } if(bulkId != NULL) { - if(r == 0) r = es_addBuf(&pData->batch.data, META_ID, sizeof(META_ID)-1); - if(r == 0) r = es_addBuf(&pData->batch.data, (char*)bulkId, ustrlen(bulkId)); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, META_ID, sizeof(META_ID)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)bulkId, ustrlen(bulkId)); } - if(r == 0) r = es_addBuf(&pData->batch.data, META_END, sizeof(META_END)-1); - if(r == 0) r = es_addBuf(&pData->batch.data, (char*)message, length); - if(r == 0) r = es_addBuf(&pData->batch.data, "\n", sizeof("\n")-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, META_END, sizeof(META_END)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)message, length); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, "\n", sizeof("\n")-1); if(r != 0) { DBGPRINTF("omelasticsearch: growing batch failed with code %d\n", r); ABORT_FINALIZE(RS_RET_ERR); } - ++pData->batch.nmemb; + ++pWrkrData->batch.nmemb; iRet = RS_RET_DEFER_COMMIT; finalize_it: @@ -446,7 +476,7 @@ finalize_it: * needs to be closed, HUP must be sent. */ static inline rsRetVal -writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) +writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) { char *rendered = NULL; cJSON *errRoot; @@ -454,6 +484,7 @@ writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) cJSON *replyRoot = *pReplyRoot; size_t toWrite; ssize_t wrRet; + sbool bMutLocked = 0; char errStr[1024]; DEFiRet; @@ -463,6 +494,9 @@ writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) FINALIZE; } + pthread_mutex_lock(&pData->mutErrFile); + bMutLocked = 1; + if(pData->fdErrFile == -1) { pData->fdErrFile = open((char*)pData->errorFile, O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE|O_CLOEXEC, @@ -474,7 +508,7 @@ writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) } } if((req=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); - cJSON_AddItemToObject(req, "url", cJSON_CreateString((char*)pData->restURL)); + cJSON_AddItemToObject(req, "url", cJSON_CreateString((char*)pWrkrData->restURL)); cJSON_AddItemToObject(req, "postdata", cJSON_CreateString((char*)reqmsg)); if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); @@ -495,13 +529,15 @@ writeDataError(instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) *pReplyRoot = NULL; /* tell caller not to delete once again! */ finalize_it: + if(bMutLocked) + pthread_mutex_unlock(&pData->mutErrFile); free(rendered); RETiRet; } static inline rsRetVal -checkResultBulkmode(instanceData *pData, cJSON *root) +checkResultBulkmode(wrkrInstanceData_t *pWrkrData, cJSON *root) { int i; int numitems; @@ -515,7 +551,7 @@ checkResultBulkmode(instanceData *pData, cJSON *root) if(items == NULL || items->type != cJSON_Array) { DBGPRINTF("omelasticsearch: error in elasticsearch reply: " "bulkmode insert does not return array, reply is: %s\n", - pData->reply); + pWrkrData->reply); ABORT_FINALIZE(RS_RET_DATAFAIL); } numitems = cJSON_GetArraySize(items); @@ -547,20 +583,20 @@ finalize_it: static inline rsRetVal -checkResult(instanceData *pData, uchar *reqmsg) +checkResult(wrkrInstanceData_t *pWrkrData, uchar *reqmsg) { cJSON *root; cJSON *ok; DEFiRet; - root = cJSON_Parse(pData->reply); + root = cJSON_Parse(pWrkrData->reply); if(root == NULL) { DBGPRINTF("omelasticsearch: could not parse JSON result \n"); ABORT_FINALIZE(RS_RET_ERR); } - if(pData->bulkmode) { - iRet = checkResultBulkmode(pData, root); + if(pWrkrData->pData->bulkmode) { + iRet = checkResultBulkmode(pWrkrData, root); } else { ok = cJSON_GetObjectItem(root, "ok"); if(ok == NULL || ok->type != cJSON_True) { @@ -572,7 +608,7 @@ checkResult(instanceData *pData, uchar *reqmsg) * these in any case. */ if(iRet == RS_RET_DATAFAIL) { - writeDataError(pData, &root, reqmsg); + writeDataError(pWrkrData, pWrkrData->pData, &root, reqmsg); iRet = RS_RET_OK; /* we have handled the problem! */ } @@ -587,19 +623,19 @@ finalize_it: static rsRetVal -curlPost(instanceData *pData, uchar *message, int msglen, uchar **tpls, int nmsgs) +curlPost(wrkrInstanceData_t *pWrkrData, uchar *message, int msglen, uchar **tpls, int nmsgs) { CURLcode code; - CURL *curl = pData->curlHandle; + CURL *curl = pWrkrData->curlHandle; DEFiRet; - pData->reply = NULL; - pData->replyLen = 0; + pWrkrData->reply = NULL; + pWrkrData->replyLen = 0; - if(pData->dynSrchIdx || pData->dynSrchType || pData->dynParent) - CHKiRet(setCurlURL(pData, tpls)); + if(pWrkrData->pData->dynSrchIdx || pWrkrData->pData->dynSrchType || pWrkrData->pData->dynParent) + CHKiRet(setCurlURL(pWrkrData, pWrkrData->pData, tpls)); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, pData); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, pWrkrData); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, (char *)message); curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, msglen); code = curl_easy_perform(curl); @@ -608,7 +644,7 @@ curlPost(instanceData *pData, uchar *message, int msglen, uchar **tpls, int nmsg case CURLE_COULDNT_RESOLVE_PROXY: case CURLE_COULDNT_CONNECT: case CURLE_WRITE_ERROR: - STATSCOUNTER_INC(indexHTTPReqFail, mutHTTPReqFail); + STATSCOUNTER_INC(indexHTTPReqFail, mutIndexHTTPReqFail); indexHTTPFail += nmsgs; DBGPRINTF("omelasticsearch: we are suspending ourselfs due " "to failure %lld of curl_easy_perform()\n", @@ -618,27 +654,27 @@ curlPost(instanceData *pData, uchar *message, int msglen, uchar **tpls, int nmsg break; } - DBGPRINTF("omelasticsearch: pData replyLen = '%d'\n", pData->replyLen); - if (pData->replyLen > 0) { - pData->reply[pData->replyLen] = '\0'; /* Append 0 Byte if replyLen is above 0 - byte has been reserved in malloc */ + DBGPRINTF("omelasticsearch: pWrkrData replyLen = '%d'\n", pWrkrData->replyLen); + if(pWrkrData->replyLen > 0) { + pWrkrData->reply[pWrkrData->replyLen] = '\0'; /* Append 0 Byte if replyLen is above 0 - byte has been reserved in malloc */ } - DBGPRINTF("omelasticsearch: pData reply: '%s'\n", pData->reply); + DBGPRINTF("omelasticsearch: pWrkrData reply: '%s'\n", pWrkrData->reply); - CHKiRet(checkResult(pData, message)); + CHKiRet(checkResult(pWrkrData, message)); finalize_it: - free(pData->reply); + free(pWrkrData->reply); RETiRet; } BEGINbeginTransaction CODESTARTbeginTransaction -dbgprintf("omelasticsearch: beginTransaction\n"); - if(!pData->bulkmode) { +dbgprintf("omelasticsearch: beginTransaction, pWrkrData %p, pData %p\n", pWrkrData, pWrkrData->pData); + if(!pWrkrData->pData->bulkmode) { FINALIZE; } - es_emptyStr(pData->batch.data); - pData->batch.nmemb = 0; + es_emptyStr(pWrkrData->batch.data); + pWrkrData->batch.nmemb = 0; finalize_it: ENDbeginTransaction @@ -646,14 +682,14 @@ ENDbeginTransaction BEGINdoAction CODESTARTdoAction STATSCOUNTER_INC(indexSubmit, mutIndexSubmit); - if(pData->bulkmode) { - CHKiRet(buildBatch(pData, ppString[0], ppString)); + if(pWrkrData->pData->bulkmode) { + CHKiRet(buildBatch(pWrkrData, ppString[0], ppString)); } else { - CHKiRet(curlPost(pData, ppString[0], strlen((char*)ppString[0]), + CHKiRet(curlPost(pWrkrData, ppString[0], strlen((char*)ppString[0]), ppString, 1)); } finalize_it: -dbgprintf("omelasticsearch: result doAction: %d (bulkmode %d)\n", iRet, pData->bulkmode); +dbgprintf("omelasticsearch: result doAction: %d (bulkmode %d)\n", iRet, pWrkrData->pData->bulkmode); ENDdoAction @@ -662,13 +698,13 @@ BEGINendTransaction CODESTARTendTransaction dbgprintf("omelasticsearch: endTransaction init\n"); /* End Transaction only if batch data is not empty */ - if (pData->batch.data != NULL ) { - cstr = es_str2cstr(pData->batch.data, NULL); + if (pWrkrData->batch.data != NULL ) { + cstr = es_str2cstr(pWrkrData->batch.data, NULL); dbgprintf("omelasticsearch: endTransaction, batch: '%s'\n", cstr); - CHKiRet(curlPost(pData, (uchar*) cstr, strlen(cstr), NULL, pData->batch.nmemb)); + CHKiRet(curlPost(pWrkrData, (uchar*) cstr, strlen(cstr), NULL, pWrkrData->batch.nmemb)); } else - dbgprintf("omelasticsearch: endTransaction, pData->batch.data is NULL, nothing to send. \n"); + dbgprintf("omelasticsearch: endTransaction, pWrkrData->batch.data is NULL, nothing to send. \n"); finalize_it: free(cstr); dbgprintf("omelasticsearch: endTransaction done with %d\n", iRet); @@ -679,24 +715,24 @@ size_t curlResult(void *ptr, size_t size, size_t nmemb, void *userdata) { char *p = (char *)ptr; - instanceData *pData = (instanceData*) userdata; + wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t*) userdata; char *buf; size_t newlen; - newlen = pData->replyLen + size*nmemb; - if((buf = realloc(pData->reply, newlen + 1)) == NULL) { + newlen = pWrkrData->replyLen + size*nmemb; + if((buf = realloc(pWrkrData->reply, newlen + 1)) == NULL) { DBGPRINTF("omelasticsearch: realloc failed in curlResult\n"); return 0; /* abort due to failure */ } - memcpy(buf+pData->replyLen, p, size*nmemb); - pData->replyLen = newlen; - pData->reply = buf; + memcpy(buf+pWrkrData->replyLen, p, size*nmemb); + pWrkrData->replyLen = newlen; + pWrkrData->reply = buf; return size*nmemb; } static rsRetVal -curlSetup(instanceData *pData) +curlSetup(wrkrInstanceData_t *pWrkrData, instanceData *pData) { HEADER *header; CURL *handle; @@ -712,13 +748,13 @@ curlSetup(instanceData *pData) curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, curlResult); curl_easy_setopt(handle, CURLOPT_POST, 1); - pData->curlHandle = handle; - pData->postHeader = header; + pWrkrData->curlHandle = handle; + pWrkrData->postHeader = header; if( pData->bulkmode || (pData->dynSrchIdx == 0 && pData->dynSrchType == 0 && pData->dynParent == 0)) { /* in this case, we know no tpls are involved in the request-->NULL OK! */ - setCurlURL(pData, NULL); + setCurlURL(pWrkrData, pData, NULL); } if(Debug) { @@ -838,16 +874,6 @@ CODESTARTnewActInst ABORT_FINALIZE(RS_RET_CONFIG_ERROR); } - if(pData->bulkmode) { - pData->batch.currTpl1 = NULL; - pData->batch.currTpl2 = NULL; - if((pData->batch.data = es_newStr(1024)) == NULL) { - DBGPRINTF("omelasticsearch: error creating batch string " - "turned off bulk mode\n"); - pData->bulkmode = 0; /* at least it works */ - } - } - iNumTpls = 1; if(pData->dynSrchIdx) ++iNumTpls; if(pData->dynSrchType) ++iNumTpls; @@ -939,9 +965,6 @@ CODESTARTnewActInst pData->searchIndex = (uchar*) strdup("system"); if(pData->searchType == NULL) pData->searchType = (uchar*) strdup("events"); - - CHKiRet(curlSetup(pData)); - CODE_STD_FINALIZERnewActInst cnfparamvalsDestruct(pvals, &actpblk); ENDnewActInst @@ -979,6 +1002,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_doHUP @@ -1001,18 +1025,18 @@ CODEmodInit_QueryRegCFSLineHdlr /* support statistics gathering */ CHKiRet(statsobj.Construct(&indexStats)); CHKiRet(statsobj.SetName(indexStats, (uchar *)"omelasticsearch")); - STATSCOUNTER_INIT(indexSubmit, mutCtrIndexSubmit); + STATSCOUNTER_INIT(indexSubmit, mutIndexSubmit); CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"submitted", - ctrType_IntCtr, &indexSubmit)); - STATSCOUNTER_INIT(indexHTTPFail, mutCtrIndexHTTPFail); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexSubmit)); + STATSCOUNTER_INIT(indexHTTPFail, mutIndexHTTPFail); CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"failed.http", - ctrType_IntCtr, &indexHTTPFail)); - STATSCOUNTER_INIT(indexHTTPReqFail, mutCtrIndexHTTPReqFail); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexHTTPFail)); + STATSCOUNTER_INIT(indexHTTPReqFail, mutIndexHTTPReqFail); CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"failed.httprequests", - ctrType_IntCtr, &indexHTTPReqFail)); - STATSCOUNTER_INIT(indexESFail, mutCtrIndexESFail); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexHTTPReqFail)); + STATSCOUNTER_INIT(indexESFail, mutIndexESFail); CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"failed.es", - ctrType_IntCtr, &indexESFail)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexESFail)); CHKiRet(statsobj.ConstructFinalize(indexStats)); ENDmodInit diff --git a/plugins/omjournal/omjournal.c b/plugins/omjournal/omjournal.c index 160c369d..82fd7bfb 100644 --- a/plugins/omjournal/omjournal.c +++ b/plugins/omjournal/omjournal.c @@ -56,6 +56,10 @@ DEF_OMOD_STATIC_DATA typedef struct _instanceData { } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + struct modConfData_s { rsconf_t *pConf; /* our overall config object */ }; @@ -91,6 +95,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature ENDisCompatibleWithFeature @@ -101,6 +110,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINnewActInst CODESTARTnewActInst /* Note: we currently do not have any parameters, so we do not need @@ -172,6 +186,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES ENDqueryEtryPt diff --git a/plugins/omlibdbi/omlibdbi.c b/plugins/omlibdbi/omlibdbi.c index 3beba4f0..c203b4aa 100644 --- a/plugins/omlibdbi/omlibdbi.c +++ b/plugins/omlibdbi/omlibdbi.c @@ -50,6 +50,9 @@ #include "errmsg.h" #include "conf.h" +#undef HAVE_DBI_TXSUPP +#warning transaction support disabled in v8 -- TODO: reenable + MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP MODULE_CNFNAME("omlibdbi") @@ -73,6 +76,10 @@ typedef struct _instanceData { int txSupport; /* transaction support */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { uchar *dbiDrvrDir; /* global: where do the dbi drivers reside? */ uchar *drvrName; /* driver to use */ @@ -94,6 +101,8 @@ static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current l static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config parameters permitted? */ +static pthread_mutex_t mutDoAct = PTHREAD_MUTEX_INITIALIZER; + /* tables for interfacing with the v6 config system */ /* module-global parameters */ @@ -157,6 +166,10 @@ BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -187,6 +200,9 @@ CODESTARTfreeInstance free(pData->dbName); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -326,16 +342,16 @@ finalize_it: BEGINtryResume CODESTARTtryResume - if(pData->conn == NULL) { - iRet = initConn(pData, 1); + if(pWrkrData->pData->conn == NULL) { + iRet = initConn(pWrkrData->pData, 1); } ENDtryResume /* transaction support 2013-03 */ BEGINbeginTransaction CODESTARTbeginTransaction - if(pData->conn == NULL) { - CHKiRet(initConn(pData, 0)); + if(pWrkrData->pData->conn == NULL) { + CHKiRet(initConn(pWrkrData->pData, 0)); } # if HAVE_DBI_TXSUPP if (pData->txSupport == 1) { @@ -355,13 +371,15 @@ ENDbeginTransaction BEGINdoAction CODESTARTdoAction - CHKiRet(writeDB(ppString[0], pData)); + pthread_mutex_lock(&mutDoAct); + CHKiRet(writeDB(ppString[0], pWrkrData->pData)); # if HAVE_DBI_TXSUPP if (pData->txSupport == 1) { iRet = RS_RET_DEFER_COMMIT; } # endif finalize_it: + pthread_mutex_unlock(&mutDoAct); ENDdoAction /* transaction support 2013-03 */ @@ -552,6 +570,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES diff --git a/plugins/ommail/ommail.c b/plugins/ommail/ommail.c index 0a781e10..910b371a 100644 --- a/plugins/ommail/ommail.c +++ b/plugins/ommail/ommail.c @@ -13,7 +13,7 @@ * * File begun on 2008-04-04 by RGerhards * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -82,13 +82,21 @@ typedef struct _instanceData { uchar *pszSrvPort; uchar *pszFrom; toRcpt_t *lstRcpt; + } smtp; + } md; /* mode-specific data */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; + union { + struct { char RcvBuf[1024]; /* buffer for receiving server responses */ size_t lenRcvBuf; size_t iRcvBuf; /* current index into the rcvBuf (buf empty if iRcvBuf == lenRcvBuf) */ int sock; /* socket to this server (most important when we do multiple msgs per mail) */ } smtp; } md; /* mode-specific data */ -} instanceData; +} wrkrInstanceData_t; typedef struct configSettings_s { toRcpt_t *lstRcpt; @@ -112,7 +120,7 @@ ENDinitConfVars /* forward definitions (as few as possible) */ static rsRetVal Send(int sock, char *msg, size_t len); -static rsRetVal readResponse(instanceData *pData, int *piState, int iExpected); +static rsRetVal readResponse(wrkrInstanceData_t *pWrkrData, int *piState, int iExpected); /* helpers for handling the recipient lists */ @@ -163,24 +171,22 @@ finalize_it: * iStatusToCheck < 0 means no checking should happen */ static rsRetVal -WriteRcpts(instanceData *pData, uchar *pszOp, size_t lenOp, int iStatusToCheck) +WriteRcpts(wrkrInstanceData_t *pWrkrData, uchar *pszOp, size_t lenOp, int iStatusToCheck) { toRcpt_t *pRcpt; int iState; DEFiRet; - assert(pData != NULL); - assert(pszOp != NULL); assert(lenOp != 0); - for(pRcpt = pData->md.smtp.lstRcpt ; pRcpt != NULL ; pRcpt = pRcpt->pNext) { + for(pRcpt = pWrkrData->pData->md.smtp.lstRcpt ; pRcpt != NULL ; pRcpt = pRcpt->pNext) { dbgprintf("Sending '%s: <%s>'\n", pszOp, pRcpt->pszTo); - CHKiRet(Send(pData->md.smtp.sock, (char*)pszOp, lenOp)); - CHKiRet(Send(pData->md.smtp.sock, ":<", sizeof(":<") - 1)); - CHKiRet(Send(pData->md.smtp.sock, (char*)pRcpt->pszTo, strlen((char*)pRcpt->pszTo))); - CHKiRet(Send(pData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)pszOp, lenOp)); + CHKiRet(Send(pWrkrData->md.smtp.sock, ":<", sizeof(":<") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)pRcpt->pszTo, strlen((char*)pRcpt->pszTo))); + CHKiRet(Send(pWrkrData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); if(iStatusToCheck >= 0) - CHKiRet(readResponse(pData, &iState, iStatusToCheck)); + CHKiRet(readResponse(pWrkrData, &iState, iStatusToCheck)); } finalize_it: @@ -193,6 +199,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -203,17 +214,19 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance if(pData->iMode == 0) { - if(pData->md.smtp.pszSrv != NULL) - free(pData->md.smtp.pszSrv); - if(pData->md.smtp.pszSrvPort != NULL) - free(pData->md.smtp.pszSrvPort); - if(pData->md.smtp.pszFrom != NULL) - free(pData->md.smtp.pszFrom); + free(pData->md.smtp.pszSrv); + free(pData->md.smtp.pszSrvPort); + free(pData->md.smtp.pszFrom); lstRcptDestruct(pData->md.smtp.lstRcpt); } ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo printf("mail"); /* TODO: extend! */ @@ -229,16 +242,16 @@ ENDdbgPrintInstInfo * rgerhards, 2008-04-04 */ static rsRetVal -getRcvChar(instanceData *pData, char *pC) +getRcvChar(wrkrInstanceData_t *pWrkrData, char *pC) { DEFiRet; ssize_t lenBuf; - assert(pData != NULL); - if(pData->md.smtp.iRcvBuf == pData->md.smtp.lenRcvBuf) { /* buffer empty? */ + if(pWrkrData->md.smtp.iRcvBuf == pWrkrData->md.smtp.lenRcvBuf) { /* buffer empty? */ /* yes, we need to read the next server response */ do { - lenBuf = recv(pData->md.smtp.sock, pData->md.smtp.RcvBuf, sizeof(pData->md.smtp.RcvBuf), 0); + lenBuf = recv(pWrkrData->md.smtp.sock, pWrkrData->md.smtp.RcvBuf, + sizeof(pWrkrData->md.smtp.RcvBuf), 0); if(lenBuf == 0) { ABORT_FINALIZE(RS_RET_NO_MORE_DATA); } else if(lenBuf < 0) { @@ -247,15 +260,15 @@ getRcvChar(instanceData *pData, char *pC) } } else { /* good read */ - pData->md.smtp.iRcvBuf = 0; - pData->md.smtp.lenRcvBuf = lenBuf; + pWrkrData->md.smtp.iRcvBuf = 0; + pWrkrData->md.smtp.lenRcvBuf = lenBuf; } } while(lenBuf < 1); } /* when we reach this point, we have a non-empty buffer */ - *pC = pData->md.smtp.RcvBuf[pData->md.smtp.iRcvBuf++]; + *pC = pWrkrData->md.smtp.RcvBuf[pWrkrData->md.smtp.iRcvBuf++]; finalize_it: RETiRet; @@ -266,14 +279,14 @@ finalize_it: * rgerhards, 2008-04-08 */ static rsRetVal -serverDisconnect(instanceData *pData) +serverDisconnect(wrkrInstanceData_t *pWrkrData) { DEFiRet; - assert(pData != NULL); + assert(pWrkrData != NULL); - if(pData->md.smtp.sock != -1) { - close(pData->md.smtp.sock); - pData->md.smtp.sock = -1; + if(pWrkrData->md.smtp.sock != -1) { + close(pWrkrData->md.smtp.sock); + pWrkrData->md.smtp.sock = -1; } RETiRet; @@ -284,16 +297,17 @@ serverDisconnect(instanceData *pData) * rgerhards, 2008-04-04 */ static rsRetVal -serverConnect(instanceData *pData) +serverConnect(wrkrInstanceData_t *pWrkrData) { struct addrinfo *res = NULL; struct addrinfo hints; char *smtpPort; char *smtpSrv; char errStr[1024]; - + instanceData *pData; DEFiRet; - assert(pData != NULL); + + pData = pWrkrData->pData; if(pData->md.smtp.pszSrv == NULL) smtpSrv = "127.0.0.1"; @@ -313,12 +327,12 @@ serverConnect(instanceData *pData) ABORT_FINALIZE(RS_RET_IO_ERROR); } - if((pData->md.smtp.sock = socket(res->ai_family, res->ai_socktype, res->ai_protocol)) == -1) { + if((pWrkrData->md.smtp.sock = socket(res->ai_family, res->ai_socktype, res->ai_protocol)) == -1) { dbgprintf("couldn't create send socket, reason %s", rs_strerror_r(errno, errStr, sizeof(errStr))); ABORT_FINALIZE(RS_RET_IO_ERROR); } - if(connect(pData->md.smtp.sock, res->ai_addr, res->ai_addrlen) != 0) { + if(connect(pWrkrData->md.smtp.sock, res->ai_addr, res->ai_addrlen) != 0) { dbgprintf("create tcp connection failed, reason %s", rs_strerror_r(errno, errStr, sizeof(errStr))); ABORT_FINALIZE(RS_RET_IO_ERROR); } @@ -328,9 +342,9 @@ finalize_it: freeaddrinfo(res); if(iRet != RS_RET_OK) { - if(pData->md.smtp.sock != -1) { - close(pData->md.smtp.sock); - pData->md.smtp.sock = -1; + if(pWrkrData->md.smtp.sock != -1) { + close(pWrkrData->md.smtp.sock); + pWrkrData->md.smtp.sock = -1; } } @@ -374,7 +388,7 @@ finalize_it: * The body is special in that we must escape a leading dot inside a line */ static rsRetVal -bodySend(instanceData *pData, char *msg, size_t len) +bodySend(wrkrInstanceData_t *pWrkrData, char *msg, size_t len) { DEFiRet; char szBuf[2048]; @@ -383,12 +397,12 @@ bodySend(instanceData *pData, char *msg, size_t len) int bHadCR = 0; int bInStartOfLine = 1; - assert(pData != NULL); + assert(pWrkrData != NULL); assert(msg != NULL); for(iSrc = 0 ; iSrc < len ; ++iSrc) { if(iBuf >= sizeof(szBuf) - 1) { /* one is reserved for our extra dot */ - CHKiRet(Send(pData->md.smtp.sock, szBuf, iBuf)); + CHKiRet(Send(pWrkrData->md.smtp.sock, szBuf, iBuf)); iBuf = 0; } szBuf[iBuf++] = msg[iSrc]; @@ -413,7 +427,7 @@ bodySend(instanceData *pData, char *msg, size_t len) } if(iBuf > 0) { /* incomplete buffer to send (the *usual* case)? */ - CHKiRet(Send(pData->md.smtp.sock, szBuf, iBuf)); + CHKiRet(Send(pWrkrData->md.smtp.sock, szBuf, iBuf)); } finalize_it: @@ -424,17 +438,17 @@ finalize_it: /* read response line from server */ static rsRetVal -readResponseLn(instanceData *pData, char *pLn, size_t lenLn) +readResponseLn(wrkrInstanceData_t *pWrkrData, char *pLn, size_t lenLn) { DEFiRet; size_t i = 0; char c; - assert(pData != NULL); + assert(pWrkrData != NULL); assert(pLn != NULL); do { - CHKiRet(getRcvChar(pData, &c)); + CHKiRet(getRcvChar(pWrkrData, &c)); if(c == '\n') break; if(i < (lenLn - 1)) /* if line is too long, we simply discard the rest */ @@ -453,18 +467,18 @@ finalize_it: * rgerhards, 2008-04-07 */ static rsRetVal -readResponse(instanceData *pData, int *piState, int iExpected) +readResponse(wrkrInstanceData_t *pWrkrData, int *piState, int iExpected) { DEFiRet; int bCont; char buf[128]; - assert(pData != NULL); + assert(pWrkrData != NULL); assert(piState != NULL); bCont = 1; do { - CHKiRet(readResponseLn(pData, buf, sizeof(buf))); + CHKiRet(readResponseLn(pWrkrData, buf, sizeof(buf))); /* note: the code below is not 100% clean as we may have received less than 4 characters. * However, as we have a fixed size this will not create a vulnerability. An error will * also most likely be generated, so it is quite acceptable IMHO -- rgerhards, 2008-04-08 @@ -506,64 +520,65 @@ mkSMTPTimestamp(uchar *pszBuf, size_t lenBuf) * rgerhards, 2008-04-04 */ static rsRetVal -sendSMTP(instanceData *pData, uchar *body, uchar *subject) +sendSMTP(wrkrInstanceData_t *pWrkrData, uchar *body, uchar *subject) { DEFiRet; int iState; /* SMTP state */ + instanceData *pData; uchar szDateBuf[64]; - assert(pData != NULL); + pData = pWrkrData->pData; - CHKiRet(serverConnect(pData)); - CHKiRet(readResponse(pData, &iState, 220)); + CHKiRet(serverConnect(pWrkrData)); + CHKiRet(readResponse(pWrkrData, &iState, 220)); - CHKiRet(Send(pData->md.smtp.sock, "HELO ", 5)); - CHKiRet(Send(pData->md.smtp.sock, (char*)glbl.GetLocalHostName(), strlen((char*)glbl.GetLocalHostName()))); - CHKiRet(Send(pData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); - CHKiRet(readResponse(pData, &iState, 250)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "HELO ", 5)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)glbl.GetLocalHostName(), strlen((char*)glbl.GetLocalHostName()))); + CHKiRet(Send(pWrkrData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); + CHKiRet(readResponse(pWrkrData, &iState, 250)); - CHKiRet(Send(pData->md.smtp.sock, "MAIL FROM:<", sizeof("MAIL FROM:<") - 1)); - CHKiRet(Send(pData->md.smtp.sock, (char*)pData->md.smtp.pszFrom, strlen((char*)pData->md.smtp.pszFrom))); - CHKiRet(Send(pData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); - CHKiRet(readResponse(pData, &iState, 250)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "MAIL FROM:<", sizeof("MAIL FROM:<") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)pData->md.smtp.pszFrom, strlen((char*)pData->md.smtp.pszFrom))); + CHKiRet(Send(pWrkrData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); + CHKiRet(readResponse(pWrkrData, &iState, 250)); - CHKiRet(WriteRcpts(pData, (uchar*)"RCPT TO", sizeof("RCPT TO") - 1, 250)); + CHKiRet(WriteRcpts(pWrkrData, (uchar*)"RCPT TO", sizeof("RCPT TO") - 1, 250)); - CHKiRet(Send(pData->md.smtp.sock, "DATA\r\n", sizeof("DATA\r\n") - 1)); - CHKiRet(readResponse(pData, &iState, 354)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "DATA\r\n", sizeof("DATA\r\n") - 1)); + CHKiRet(readResponse(pWrkrData, &iState, 354)); /* now come the data part */ /* header */ mkSMTPTimestamp(szDateBuf, sizeof(szDateBuf)); - CHKiRet(Send(pData->md.smtp.sock, (char*)szDateBuf, strlen((char*)szDateBuf))); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)szDateBuf, strlen((char*)szDateBuf))); - CHKiRet(Send(pData->md.smtp.sock, "From: <", sizeof("From: <") - 1)); - CHKiRet(Send(pData->md.smtp.sock, (char*)pData->md.smtp.pszFrom, strlen((char*)pData->md.smtp.pszFrom))); - CHKiRet(Send(pData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "From: <", sizeof("From: <") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)pData->md.smtp.pszFrom, strlen((char*)pData->md.smtp.pszFrom))); + CHKiRet(Send(pWrkrData->md.smtp.sock, ">\r\n", sizeof(">\r\n") - 1)); - CHKiRet(WriteRcpts(pData, (uchar*)"To", sizeof("To") - 1, -1)); + CHKiRet(WriteRcpts(pWrkrData, (uchar*)"To", sizeof("To") - 1, -1)); - CHKiRet(Send(pData->md.smtp.sock, "Subject: ", sizeof("Subject: ") - 1)); - CHKiRet(Send(pData->md.smtp.sock, (char*)subject, strlen((char*)subject))); - CHKiRet(Send(pData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "Subject: ", sizeof("Subject: ") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, (char*)subject, strlen((char*)subject))); + CHKiRet(Send(pWrkrData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); - CHKiRet(Send(pData->md.smtp.sock, "X-Mailer: rsyslog-immail\r\n", sizeof("x-mailer: rsyslog-immail\r\n") - 1)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "X-Mailer: rsyslog-immail\r\n", sizeof("x-mailer: rsyslog-immail\r\n") - 1)); - CHKiRet(Send(pData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); /* indicate end of header */ + CHKiRet(Send(pWrkrData->md.smtp.sock, "\r\n", sizeof("\r\n") - 1)); /* indicate end of header */ /* body */ if(pData->bEnableBody) - CHKiRet(bodySend(pData, (char*)body, strlen((char*) body))); + CHKiRet(bodySend(pWrkrData, (char*)body, strlen((char*) body))); /* end of data, back to envelope transaction */ - CHKiRet(Send(pData->md.smtp.sock, "\r\n.\r\n", sizeof("\r\n.\r\n") - 1)); - CHKiRet(readResponse(pData, &iState, 250)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "\r\n.\r\n", sizeof("\r\n.\r\n") - 1)); + CHKiRet(readResponse(pWrkrData, &iState, 250)); - CHKiRet(Send(pData->md.smtp.sock, "QUIT\r\n", sizeof("QUIT\r\n") - 1)); - CHKiRet(readResponse(pData, &iState, 221)); + CHKiRet(Send(pWrkrData->md.smtp.sock, "QUIT\r\n", sizeof("QUIT\r\n") - 1)); + CHKiRet(readResponse(pWrkrData, &iState, 221)); /* we are finished, a new connection is created for each request, so let's close it now */ - CHKiRet(serverDisconnect(pData)); + CHKiRet(serverDisconnect(pWrkrData)); finalize_it: RETiRet; @@ -583,8 +598,8 @@ finalize_it: */ BEGINtryResume CODESTARTtryResume - CHKiRet(serverConnect(pData)); - CHKiRet(serverDisconnect(pData)); /* if we fail, we will never reach this line */ + CHKiRet(serverConnect(pWrkrData)); + CHKiRet(serverDisconnect(pWrkrData)); /* if we fail, we will never reach this line */ finalize_it: if(iRet == RS_RET_IO_ERROR) iRet = RS_RET_SUSPENDED; @@ -593,17 +608,14 @@ ENDtryResume BEGINdoAction CODESTARTdoAction - dbgprintf(" Mail\n"); + DBGPRINTF(" Mail\n"); - /* forward */ - if(pData->bHaveSubject) - iRet = sendSMTP(pData, ppString[0], ppString[1]); - else - iRet = sendSMTP(pData, ppString[0], (uchar*)"message from rsyslog"); + iRet = sendSMTP(pWrkrData, ppString[0], + (pWrkrData->pData->bHaveSubject) ? + ppString[1] : (uchar*)"message from rsyslog"); if(iRet != RS_RET_OK) { - /* error! */ - dbgprintf("error sending mail, suspending\n"); + DBGPRINTF("error sending mail, suspending\n"); iRet = RS_RET_SUSPENDED; } ENDdoAction @@ -689,6 +701,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/ommongodb/ommongodb.c b/plugins/ommongodb/ommongodb.c index ecfd2518..09f19768 100644 --- a/plugins/ommongodb/ommongodb.c +++ b/plugins/ommongodb/ommongodb.c @@ -4,7 +4,7 @@ * mongodb C interface is crap. Obtain the library here: * https://github.com/algernon/libmongo-client * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -71,6 +71,10 @@ typedef struct _instanceData { int bErrMsgPermitted; /* only one errmsg permitted per connection */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + /* tables for interfacing with the v6 config system */ /* action (instance) parameters */ @@ -89,10 +93,16 @@ static struct cnfparamblk actpblk = actpdescr }; +static pthread_mutex_t mutDoAct = PTHREAD_MUTEX_INITIALIZER; + BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature /* use this to specify if select features are supported by this @@ -126,6 +136,10 @@ CODESTARTfreeInstance free(pData->tplName); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -235,12 +249,18 @@ getDefaultBSON(msg_t *pMsg) int severity, facil; gint64 ts_gen, ts_rcv; /* timestamps: generated, received */ int secfrac; - - procid = MsgGetProp(pMsg, NULL, PROP_PROGRAMNAME, NULL, &procid_len, &procid_free, NULL); - tag = MsgGetProp(pMsg, NULL, PROP_SYSLOGTAG, NULL, &tag_len, &tag_free, NULL); - pid = MsgGetProp(pMsg, NULL, PROP_PROCID, NULL, &pid_len, &pid_free, NULL); - sys = MsgGetProp(pMsg, NULL, PROP_HOSTNAME, NULL, &sys_len, &sys_free, NULL); - msg = MsgGetProp(pMsg, NULL, PROP_MSG, NULL, &msg_len, &msg_free, NULL); + msgPropDescr_t cProp; /* we use internal implementation knowledge... */ + + cProp.id = PROP_PROGRAMNAME; + procid = MsgGetProp(pMsg, NULL, &cProp, &procid_len, &procid_free, NULL); + cProp.id = PROP_SYSLOGTAG; + tag = MsgGetProp(pMsg, NULL, &cProp, &tag_len, &tag_free, NULL); + cProp.id = PROP_PROCID; + pid = MsgGetProp(pMsg, NULL, &cProp, &pid_len, &pid_free, NULL); + cProp.id = PROP_HOSTNAME; + sys = MsgGetProp(pMsg, NULL, &cProp, &sys_len, &sys_free, NULL); + cProp.id = PROP_MSG; + msg = MsgGetProp(pMsg, NULL, &cProp, &msg_len, &msg_free, NULL); // TODO: move to datetime? Refactor in any case! rgerhards, 2012-03-30 ts_gen = (gint64) datetime.syslogTime2time_t(&pMsg->tTIMESTAMP) * 1000; /* ms! */ @@ -311,8 +331,11 @@ BSONAppendJSONObject(bson *doc, const gchar *name, struct json_object *json) case json_type_int: { int64_t i; - /* FIXME: the future version will have get_int64 */ +#ifdef HAVE_JSON_OBJECT_NEW_INT64 + i = json_object_get_int64(json); +#else /* HAVE_JSON_OBJECT_NEW_INT64 */ i = json_object_get_int(json); +#endif /* HAVE_JSON_OBJECT_NEW_INT64 */ if (i >= INT32_MIN && i <= INT32_MAX) return bson_append_int32(doc, name, i); else @@ -413,14 +436,17 @@ error: BEGINtryResume CODESTARTtryResume - if(pData->conn == NULL) { - iRet = initMongoDB(pData, 1); + if(pWrkrData->pData->conn == NULL) { + iRet = initMongoDB(pWrkrData->pData, 1); } ENDtryResume BEGINdoAction bson *doc = NULL; + instanceData *pData; CODESTARTdoAction + pthread_mutex_lock(&mutDoAct); + pData = pWrkrData->pData; /* see if we are ready to proceed */ if(pData->conn == NULL) { CHKiRet(initMongoDB(pData, 0)); @@ -445,6 +471,7 @@ CODESTARTdoAction } finalize_it: + pthread_mutex_unlock(&mutDoAct); if(doc != NULL) bson_free(doc); ENDdoAction @@ -551,6 +578,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt diff --git a/plugins/ommysql/ommysql.c b/plugins/ommysql/ommysql.c index 2dfa29de..c004d1c6 100644 --- a/plugins/ommysql/ommysql.c +++ b/plugins/ommysql/ommysql.c @@ -6,7 +6,7 @@ * * File begun on 2007-07-20 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * Licensed under the Apache License, Version 2.0 (the "License"); @@ -55,18 +55,22 @@ DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) typedef struct _instanceData { - MYSQL *f_hmysql; /* handle to MySQL */ - char f_dbsrv[MAXHOSTNAMELEN+1]; /* IP or hostname of DB server*/ - unsigned int f_dbsrvPort; /* port of MySQL server */ - char f_dbname[_DB_MAXDBLEN+1]; /* DB name */ - char f_dbuid[_DB_MAXUNAMELEN+1]; /* DB user */ - char f_dbpwd[_DB_MAXPWDLEN+1]; /* DB user's password */ - unsigned uLastMySQLErrno; /* last errno returned by MySQL or 0 if all is well */ - uchar * f_configfile; /* MySQL Client Configuration File */ - uchar * f_configsection; /* MySQL Client Configuration Section */ - uchar *tplName; /* format template to use */ + char dbsrv[MAXHOSTNAMELEN+1]; /* IP or hostname of DB server*/ + unsigned int dbsrvPort; /* port of MySQL server */ + char dbname[_DB_MAXDBLEN+1]; /* DB name */ + char dbuid[_DB_MAXUNAMELEN+1]; /* DB user */ + char dbpwd[_DB_MAXPWDLEN+1]; /* DB user's password */ + uchar *configfile; /* MySQL Client Configuration File */ + uchar *configsection; /* MySQL Client Configuration Section */ + uchar *tplName; /* format template to use */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; + MYSQL *hmysql; /* handle to MySQL */ + unsigned uLastMySQLErrno; /* last errno returned by MySQL or 0 if all is well */ +} wrkrInstanceData_t; + typedef struct configSettings_s { int iSrvPort; /* database server port */ uchar *pszMySQLConfigFile; /* MySQL Client Configuration File */ @@ -104,6 +108,12 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + pWrkrData->hmysql = NULL; +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -115,33 +125,28 @@ ENDisCompatibleWithFeature * MySQL connection. * Initially added 2004-10-28 */ -static void closeMySQL(instanceData *pData) +static void closeMySQL(wrkrInstanceData_t *pWrkrData) { - ASSERT(pData != NULL); - - if(pData->f_hmysql != NULL) { /* just to be on the safe side... */ - mysql_close(pData->f_hmysql); - pData->f_hmysql = NULL; - } - if(pData->f_configfile!=NULL){ - free(pData->f_configfile); - pData->f_configfile=NULL; - } - if(pData->f_configsection!=NULL){ - free(pData->f_configsection); - pData->f_configsection=NULL; + if(pWrkrData->hmysql != NULL) { /* just to be on the safe side... */ + mysql_close(pWrkrData->hmysql); + pWrkrData->hmysql = NULL; } } BEGINfreeInstance CODESTARTfreeInstance - free(pData->f_configfile); - free(pData->f_configsection); + free(pData->configfile); + free(pData->configsection); free(pData->tplName); - closeMySQL(pData); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + closeMySQL(pWrkrData); +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo /* nothing special here */ @@ -152,25 +157,23 @@ ENDdbgPrintInstInfo * We check if we have a valid MySQL handle. If not, we simply * report an error, but can not be specific. RGerhards, 2007-01-30 */ -static void reportDBError(instanceData *pData, int bSilent) +static void reportDBError(wrkrInstanceData_t *pWrkrData, int bSilent) { char errMsg[512]; unsigned uMySQLErrno; - ASSERT(pData != NULL); - /* output log message */ errno = 0; - if(pData->f_hmysql == NULL) { + if(pWrkrData->hmysql == NULL) { errmsg.LogError(0, NO_ERRCODE, "unknown DB error occured - could not obtain MySQL handle"); } else { /* we can ask mysql for the error description... */ - uMySQLErrno = mysql_errno(pData->f_hmysql); + uMySQLErrno = mysql_errno(pWrkrData->hmysql); snprintf(errMsg, sizeof(errMsg)/sizeof(char), "db error (%d): %s\n", uMySQLErrno, - mysql_error(pData->f_hmysql)); - if(bSilent || uMySQLErrno == pData->uLastMySQLErrno) + mysql_error(pWrkrData->hmysql)); + if(bSilent || uMySQLErrno == pWrkrData->uLastMySQLErrno) dbgprintf("mysql, DBError(silent): %s\n", errMsg); else { - pData->uLastMySQLErrno = uMySQLErrno; + pWrkrData->uLastMySQLErrno = uMySQLErrno; errmsg.LogError(0, NO_ERRCODE, "%s", errMsg); } } @@ -183,25 +186,26 @@ static void reportDBError(instanceData *pData, int bSilent) * MySQL connection. * Initially added 2004-10-28 mmeckelein */ -static rsRetVal initMySQL(instanceData *pData, int bSilent) +static rsRetVal initMySQL(wrkrInstanceData_t *pWrkrData, int bSilent) { + instanceData *pData; DEFiRet; - ASSERT(pData != NULL); - ASSERT(pData->f_hmysql == NULL); - pData->f_hmysql = mysql_init(NULL); - if(pData->f_hmysql == NULL) { + ASSERT(pWrkrData->hmysql == NULL); + pData = pWrkrData->pData; + pWrkrData->hmysql = mysql_init(NULL); + if(pWrkrData->hmysql == NULL) { errmsg.LogError(0, RS_RET_SUSPENDED, "can not initialize MySQL handle"); iRet = RS_RET_SUSPENDED; } else { /* we could get the handle, now on with work... */ - mysql_options(pData->f_hmysql,MYSQL_READ_DEFAULT_GROUP,((pData->f_configsection!=NULL)?(char*)pData->f_configsection:"client")); - if(pData->f_configfile!=NULL){ + mysql_options(pWrkrData->hmysql,MYSQL_READ_DEFAULT_GROUP,((pData->configsection!=NULL)?(char*)pData->configsection:"client")); + if(pData->configfile!=NULL){ FILE * fp; - fp=fopen((char*)pData->f_configfile,"r"); + fp=fopen((char*)pData->configfile,"r"); int err=errno; if(fp==NULL){ char msg[512]; - snprintf(msg,sizeof(msg)/sizeof(char),"Could not open '%s' for reading",pData->f_configfile); + snprintf(msg,sizeof(msg)/sizeof(char),"Could not open '%s' for reading",pData->configfile); if(bSilent) { char errStr[512]; rs_strerror_r(err, errStr, sizeof(errStr)); @@ -210,17 +214,17 @@ static rsRetVal initMySQL(instanceData *pData, int bSilent) errmsg.LogError(err,NO_ERRCODE,"mysql configuration error: %s\n",msg); } else { fclose(fp); - mysql_options(pData->f_hmysql,MYSQL_READ_DEFAULT_FILE,pData->f_configfile); + mysql_options(pWrkrData->hmysql,MYSQL_READ_DEFAULT_FILE,pData->configfile); } } /* Connect to database */ - if(mysql_real_connect(pData->f_hmysql, pData->f_dbsrv, pData->f_dbuid, - pData->f_dbpwd, pData->f_dbname, pData->f_dbsrvPort, NULL, 0) == NULL) { - reportDBError(pData, bSilent); - closeMySQL(pData); /* ignore any error we may get */ + if(mysql_real_connect(pWrkrData->hmysql, pData->dbsrv, pData->dbuid, + pData->dbpwd, pData->dbname, pData->dbsrvPort, NULL, 0) == NULL) { + reportDBError(pWrkrData, bSilent); + closeMySQL(pWrkrData); /* ignore any error we may get */ ABORT_FINALIZE(RS_RET_SUSPENDED); } - mysql_autocommit(pData->f_hmysql, 0); + mysql_autocommit(pWrkrData->hmysql, 0); } finalize_it: @@ -232,35 +236,32 @@ finalize_it: * to an established MySQL session. * Initially added 2004-10-28 mmeckelein */ -rsRetVal writeMySQL(uchar *psz, instanceData *pData) +rsRetVal writeMySQL(wrkrInstanceData_t *pWrkrData, uchar *psz) { DEFiRet; - ASSERT(psz != NULL); - ASSERT(pData != NULL); - /* see if we are ready to proceed */ - if(pData->f_hmysql == NULL) { - CHKiRet(initMySQL(pData, 0)); + if(pWrkrData->hmysql == NULL) { + CHKiRet(initMySQL(pWrkrData, 0)); } /* try insert */ - if(mysql_query(pData->f_hmysql, (char*)psz)) { + if(mysql_query(pWrkrData->hmysql, (char*)psz)) { /* error occured, try to re-init connection and retry */ - closeMySQL(pData); /* close the current handle */ - CHKiRet(initMySQL(pData, 0)); /* try to re-open */ - if(mysql_query(pData->f_hmysql, (char*)psz)) { /* re-try insert */ + closeMySQL(pWrkrData); /* close the current handle */ + CHKiRet(initMySQL(pWrkrData, 0)); /* try to re-open */ + if(mysql_query(pWrkrData->hmysql, (char*)psz)) { /* re-try insert */ /* we failed, giving up for now */ - reportDBError(pData, 0); - closeMySQL(pData); /* free ressources */ + reportDBError(pWrkrData, 0); + closeMySQL(pWrkrData); /* free ressources */ ABORT_FINALIZE(RS_RET_SUSPENDED); } } finalize_it: if(iRet == RS_RET_OK) { - pData->uLastMySQLErrno = 0; /* reset error for error supression */ + pWrkrData->uLastMySQLErrno = 0; /* reset error for error supression */ } RETiRet; @@ -269,28 +270,28 @@ finalize_it: BEGINtryResume CODESTARTtryResume - if(pData->f_hmysql == NULL) { - iRet = initMySQL(pData, 1); + if(pWrkrData->hmysql == NULL) { + iRet = initMySQL(pWrkrData, 1); } ENDtryResume BEGINbeginTransaction CODESTARTbeginTransaction - CHKiRet(writeMySQL((uchar*)"START TRANSACTION", pData)); + CHKiRet(writeMySQL(pWrkrData, (uchar*)"START TRANSACTION")); finalize_it: ENDbeginTransaction BEGINdoAction CODESTARTdoAction dbgprintf("\n"); - CHKiRet(writeMySQL(ppString[0], pData)); + CHKiRet(writeMySQL(pWrkrData, ppString[0])); iRet = RS_RET_DEFER_COMMIT; finalize_it: ENDdoAction BEGINendTransaction CODESTARTendTransaction - if (mysql_commit(pData->f_hmysql) != 0) { + if(mysql_commit(pWrkrData->hmysql) != 0) { dbgprintf("mysql server error: transaction not committed\n"); iRet = RS_RET_SUSPENDED; } @@ -300,11 +301,10 @@ ENDendTransaction static inline void setInstParamDefaults(instanceData *pData) { - pData->f_dbsrvPort = 0; - pData->f_configfile = NULL; - pData->f_configsection = NULL; + pData->dbsrvPort = 0; + pData->configfile = NULL; + pData->configsection = NULL; pData->tplName = NULL; - pData->f_hmysql = NULL; /* initialize, but connect only on first message (important for queued mode!) */ } @@ -329,26 +329,26 @@ CODESTARTnewActInst continue; if(!strcmp(actpblk.descr[i].name, "server")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); - strncpy(pData->f_dbsrv, cstr, sizeof(pData->f_dbsrv)); + strncpy(pData->dbsrv, cstr, sizeof(pData->dbsrv)); free(cstr); } else if(!strcmp(actpblk.descr[i].name, "serverport")) { - pData->f_dbsrvPort = (int) pvals[i].val.d.n, NULL; + pData->dbsrvPort = (int) pvals[i].val.d.n, NULL; } else if(!strcmp(actpblk.descr[i].name, "db")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); - strncpy(pData->f_dbname, cstr, sizeof(pData->f_dbname)); + strncpy(pData->dbname, cstr, sizeof(pData->dbname)); free(cstr); } else if(!strcmp(actpblk.descr[i].name, "uid")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); - strncpy(pData->f_dbuid, cstr, sizeof(pData->f_dbuid)); + strncpy(pData->dbuid, cstr, sizeof(pData->dbuid)); free(cstr); } else if(!strcmp(actpblk.descr[i].name, "pwd")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); - strncpy(pData->f_dbpwd, cstr, sizeof(pData->f_dbpwd)); + strncpy(pData->dbpwd, cstr, sizeof(pData->dbpwd)); free(cstr); } else if(!strcmp(actpblk.descr[i].name, "mysqlconfig.file")) { - pData->f_configfile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + pData->configfile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "mysqlconfig.section")) { - pData->f_configsection = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + pData->configsection = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { @@ -399,19 +399,19 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) * Now we read the MySQL connection properties * and verify that the properties are valid. */ - if(getSubString(&p, pData->f_dbsrv, MAXHOSTNAMELEN+1, ',')) + if(getSubString(&p, pData->dbsrv, MAXHOSTNAMELEN+1, ',')) iMySQLPropErr++; - if(*pData->f_dbsrv == '\0') + if(*pData->dbsrv == '\0') iMySQLPropErr++; - if(getSubString(&p, pData->f_dbname, _DB_MAXDBLEN+1, ',')) + if(getSubString(&p, pData->dbname, _DB_MAXDBLEN+1, ',')) iMySQLPropErr++; - if(*pData->f_dbname == '\0') + if(*pData->dbname == '\0') iMySQLPropErr++; - if(getSubString(&p, pData->f_dbuid, _DB_MAXUNAMELEN+1, ',')) + if(getSubString(&p, pData->dbuid, _DB_MAXUNAMELEN+1, ',')) iMySQLPropErr++; - if(*pData->f_dbuid == '\0') + if(*pData->dbuid == '\0') iMySQLPropErr++; - if(getSubString(&p, pData->f_dbpwd, _DB_MAXPWDLEN+1, ';')) + if(getSubString(&p, pData->dbpwd, _DB_MAXPWDLEN+1, ';')) iMySQLPropErr++; /* now check for template * We specify that the SQL option must be present in the template. @@ -431,10 +431,9 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) errmsg.LogError(0, RS_RET_INVALID_PARAMS, "Trouble with MySQL connection properties. -MySQL logging disabled"); ABORT_FINALIZE(RS_RET_INVALID_PARAMS); } else { - pData->f_dbsrvPort = (unsigned) cs.iSrvPort; /* set configured port */ - pData->f_configfile = cs.pszMySQLConfigFile; - pData->f_configsection = cs.pszMySQLConfigSection; - pData->f_hmysql = NULL; /* initialize, but connect only on first message (important for queued mode!) */ + pData->dbsrvPort = (unsigned) cs.iSrvPort; /* set configured port */ + pData->configfile = cs.pszMySQLConfigFile; + pData->configsection = cs.pszMySQLConfigSection; } CODE_STD_FINALIZERparseSelectorAct @@ -454,6 +453,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt @@ -492,7 +492,7 @@ CODEmodInit_QueryRegCFSLineHdlr mysql_server_init(0, NULL, NULL) # endif ) { - errmsg.LogError(0, NO_ERRCODE, "ommysql: mysql_server_init() failed, plugin " + errmsg.LogError(0, NO_ERRCODE, "ommysql: intializing mysql client failed, plugin " "can not run"); ABORT_FINALIZE(RS_RET_ERR); } diff --git a/plugins/ompgsql/ompgsql.c b/plugins/ompgsql/ompgsql.c index 11f346f6..87599484 100644 --- a/plugins/ompgsql/ompgsql.c +++ b/plugins/ompgsql/ompgsql.c @@ -6,7 +6,7 @@ * * File begun on 2007-10-18 by sur5r (converted from ommysql.c) * - * Copyright 2007, 2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007, 2013 Rainer Gerhards and Adiscon GmbH. * * The following link my be useful for the not-so-postgres literate * when setting up a test environment (on Fedora): @@ -66,11 +66,17 @@ typedef struct _instanceData { ConnStatusType eLastPgSQLStatus; /* last status from postgres */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { EMPTY_STRUCT } configSettings_t; static configSettings_t __attribute__((unused)) cs; +static pthread_mutex_t mutDoAct = PTHREAD_MUTEX_INITIALIZER; + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars ENDinitConfVars @@ -82,6 +88,10 @@ BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -108,6 +118,9 @@ CODESTARTfreeInstance closePgSQL(pData); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -244,8 +257,8 @@ finalize_it: BEGINtryResume CODESTARTtryResume - if(pData->f_hpgsql == NULL) { - iRet = initPgSQL(pData, 1); + if(pWrkrData->pData->f_hpgsql == NULL) { + iRet = initPgSQL(pWrkrData->pData, 1); if(iRet == RS_RET_OK) { /* the code above seems not to actually connect to the database. As such, we do a * dummy statement (a pointless select...) to verify the connection and return @@ -253,7 +266,7 @@ CODESTARTtryResume * PostgreSQL expert, so any patch that does the desired result in a more * intelligent way is highly welcome. -- rgerhards, 2009-12-16 */ - iRet = writePgSQL((uchar*)"select 'a' as a", pData); + iRet = writePgSQL((uchar*)"select 'a' as a", pWrkrData->pData); } } @@ -263,23 +276,25 @@ ENDtryResume BEGINbeginTransaction CODESTARTbeginTransaction dbgprintf("ompgsql: beginTransaction\n"); - iRet = writePgSQL((uchar*) "begin", pData); /* TODO: make user-configurable */ + iRet = writePgSQL((uchar*) "begin", pWrkrData->pData); /* TODO: make user-configurable */ ENDbeginTransaction BEGINdoAction CODESTARTdoAction + pthread_mutex_lock(&mutDoAct); dbgprintf("\n"); - CHKiRet(writePgSQL(ppString[0], pData)); + CHKiRet(writePgSQL(ppString[0], pWrkrData->pData)); if(bCoreSupportsBatching) iRet = RS_RET_DEFER_COMMIT; finalize_it: + pthread_mutex_unlock(&mutDoAct); ENDdoAction BEGINendTransaction CODESTARTendTransaction - iRet = writePgSQL((uchar*) "commit;", pData); /* TODO: make user-configurable */ + iRet = writePgSQL((uchar*) "commit;", pWrkrData->pData); /* TODO: make user-configurable */ dbgprintf("ompgsql: endTransaction\n"); ENDendTransaction @@ -361,6 +376,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ENDqueryEtryPt @@ -372,6 +388,11 @@ INITLegCnfVars CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); + +# warning: transaction support missing for v8 + bCoreSupportsBatching= 0; + DBGPRINTF("ompgsql: transactions are not yet supported on v8\n"); + DBGPRINTF("ompgsql: module compiled with rsyslog version %s.\n", VERSION); DBGPRINTF("ompgsql: %susing transactional output interface.\n", bCoreSupportsBatching ? "" : "not "); ENDmodInit diff --git a/plugins/omprog/omprog.c b/plugins/omprog/omprog.c index d821ff16..1060b5c3 100644 --- a/plugins/omprog/omprog.c +++ b/plugins/omprog/omprog.c @@ -6,7 +6,7 @@ * * File begun on 2009-04-01 by RGerhards * - * Copyright 2009-2012 Adiscon GmbH. + * Copyright 2009-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -35,6 +35,7 @@ #include <errno.h> #include <unistd.h> #include <wait.h> +#include <pthread.h> #include "conf.h" #include "syslogd-types.h" #include "srUtils.h" @@ -54,12 +55,19 @@ DEFobjCurrIf(errmsg) typedef struct _instanceData { uchar *szBinary; /* name of binary to call */ + char **aParams; /* Optional Parameters for binary command */ uchar *tplName; /* assigned output template */ - pid_t pid; /* pid of currently running process */ - int fdPipe; /* file descriptor to write to */ + pid_t pid; /* pid of currently running process */ + int fdPipe; /* file descriptor to write to */ int bIsRunning; /* is binary currently running? 0-no, 1-yes */ + int iParams; /* Holds the count of parameters if set*/ + pthread_mutex_t mut; /* make sure only one instance is active */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { uchar *szBinary; /* name of binary to call */ } configSettings_t; @@ -87,8 +95,13 @@ ENDinitConfVars BEGINcreateInstance CODESTARTcreateInstance + pthread_mutex_init(&pData->mut, NULL); ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -98,11 +111,23 @@ ENDisCompatibleWithFeature BEGINfreeInstance + int i; CODESTARTfreeInstance + pthread_mutex_destroy(&pData->mut); if(pData->szBinary != NULL) free(pData->szBinary); + if(pData->aParams != NULL) { + for (i = 0; i < pData->iParams; i++) { + free(pData->aParams[i]); + } + free(pData->aParams); + } ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -120,10 +145,9 @@ ENDtryResume static void execBinary(instanceData *pData, int fdStdin) { - int i; + int i, iRet; struct sigaction sigAct; sigset_t set; - char *newargv[] = { NULL }; char *newenviron[] = { NULL }; assert(pData != NULL); @@ -135,7 +159,7 @@ static void execBinary(instanceData *pData, int fdStdin) * gets some more widespread use... */ } - //fclose(stdout); + /*fclose(stdout);*/ /* we close all file handles as we fork soon * Is there a better way to do this? - mail me! rgerhards@adiscon.com @@ -157,11 +181,11 @@ static void execBinary(instanceData *pData, int fdStdin) alarm(0); /* finally exec child */ - execve((char*)pData->szBinary, newargv, newenviron); - /* switch to? - execlp((char*)program, (char*) program, (char*)arg, NULL); - */ - + iRet = execve((char*)pData->szBinary, pData->aParams, newenviron); + if (iRet == -1) { + dbgprintf("omprog: failed to execute binary '%s' with return code: %d\n", pData->szBinary, errno); + } + /* we should never reach this point, but if we do, we terminate */ exit(1); } @@ -183,7 +207,7 @@ openPipe(instanceData *pData) ABORT_FINALIZE(RS_RET_ERR_CREAT_PIPE); } - DBGPRINTF("executing program '%s'\n", pData->szBinary); + DBGPRINTF("omprog: executing program '%s' with '%d' parameters\n", pData->szBinary, pData->iParams); /* NO OUTPUT AFTER FORK! */ @@ -201,7 +225,7 @@ openPipe(instanceData *pData) /*NO CODE HERE - WILL NEVER BE REACHED!*/ } - DBGPRINTF("child has pid %d\n", (int) cpid); + DBGPRINTF("omprog: child has pid %d\n", (int) cpid); pData->fdPipe = pipefd[1]; pData->pid = cpid; close(pipefd[0]); @@ -226,11 +250,11 @@ cleanup(instanceData *pData) ret = waitpid(pData->pid, &status, 0); if(ret != pData->pid) { /* if waitpid() fails, we can not do much - try to ignore it... */ - DBGPRINTF("waitpid() returned state %d[%s], future malfunction may happen\n", ret, + DBGPRINTF("omprog: waitpid() returned state %d[%s], future malfunction may happen\n", ret, rs_strerror_r(errno, errStr, sizeof(errStr))); } else { /* check if we should print out some diagnostic information */ - DBGPRINTF("waitpid status return for program '%s': %2.2x\n", + DBGPRINTF("omprog: waitpid status return for program '%s': %2.2x\n", pData->szBinary, status); if(WIFEXITED(status)) { errmsg.LogError(0, NO_ERRCODE, "program '%s' exited normally, state %d", @@ -279,22 +303,21 @@ writePipe(instanceData *pData, uchar *szMsg) lenWrite = strlen((char*)szMsg); writeOffset = 0; - do - { + do { lenWritten = write(pData->fdPipe, ((char*)szMsg)+writeOffset, lenWrite); if(lenWritten == -1) { switch(errno) { - case EPIPE: - DBGPRINTF("Program '%s' terminated, trying to restart\n", - pData->szBinary); - CHKiRet(cleanup(pData)); - CHKiRet(tryRestart(pData)); - break; - default: - DBGPRINTF("error %d writing to pipe: %s\n", errno, - rs_strerror_r(errno, errStr, sizeof(errStr))); - ABORT_FINALIZE(RS_RET_ERR_WRITE_PIPE); - break; + case EPIPE: + DBGPRINTF("omprog: Program '%s' terminated, trying to restart\n", + pData->szBinary); + CHKiRet(cleanup(pData)); + CHKiRet(tryRestart(pData)); + break; + default: + DBGPRINTF("omprog: error %d writing to pipe: %s\n", errno, + rs_strerror_r(errno, errStr, sizeof(errStr))); + ABORT_FINALIZE(RS_RET_ERR_WRITE_PIPE); + break; } } else { writeOffset += lenWritten; @@ -308,7 +331,10 @@ finalize_it: BEGINdoAction + instanceData *pData; CODESTARTdoAction + pData = pWrkrData->pData; + pthread_mutex_lock(&pData->mut); if(pData->bIsRunning == 0) { openPipe(pData); } @@ -317,6 +343,7 @@ CODESTARTdoAction if(iRet != RS_RET_OK) iRet = RS_RET_SUSPENDED; + pthread_mutex_unlock(&pData->mut); ENDdoAction @@ -324,13 +351,23 @@ static inline void setInstParamDefaults(instanceData *pData) { pData->szBinary = NULL; + pData->aParams = NULL; + pData->iParams = 0; pData->fdPipe = -1; pData->bIsRunning = 0; } BEGINnewActInst struct cnfparamvals *pvals; + sbool bInQuotes; int i; + int iPrm; + unsigned char *c; + es_size_t iCnt; + es_size_t iStr; + es_str_t *estrBinary; + es_str_t *estrParams; + es_str_t *estrTmp; CODESTARTnewActInst if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); @@ -344,12 +381,85 @@ CODESTARTnewActInst if(!pvals[i].bUsed) continue; if(!strcmp(actpblk.descr[i].name, "binary")) { - pData->szBinary = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + estrBinary = pvals[i].val.d.estr; + estrParams = NULL; + + /* Search for space */ + c = es_getBufAddr(pvals[i].val.d.estr); + iCnt = 0; + while(iCnt < es_strlen(pvals[i].val.d.estr) ) { + if (c[iCnt] == ' ') { + /* Split binary name from parameters */ + estrBinary = es_newStrFromSubStr ( pvals[i].val.d.estr, 0, iCnt ); + estrParams = es_newStrFromSubStr ( pvals[i].val.d.estr, iCnt+1, es_strlen(pvals[i].val.d.estr)); + break; + } + iCnt++; + } + /* Assign binary and params */ + pData->szBinary = (uchar*)es_str2cstr(estrBinary, NULL); + dbgprintf("omprog: szBinary = '%s'\n", pData->szBinary); + /* Check for Params! */ + if (estrParams != NULL) { + dbgprintf("omprog: szParams = '%s'\n", es_str2cstr(estrParams, NULL) ); + + /* Count parameters if set */ + c = es_getBufAddr(estrParams); /* Reset to beginning */ + pData->iParams = 2; /* Set default to 2, first parameter for binary and second parameter at least from config*/ + iCnt = 0; + while(iCnt < es_strlen(estrParams) ) { + if (c[iCnt] == ' ' && c[iCnt-1] != '\\') + pData->iParams++; + iCnt++; + } + dbgprintf("omprog: iParams = '%d'\n", pData->iParams); + + /* Create argv Array */ + CHKmalloc(pData->aParams = malloc( (pData->iParams+1) * sizeof(char*))); /* One more for first param */ + + /* Second Loop, create parameter array*/ + c = es_getBufAddr(estrParams); /* Reset to beginning */ + iCnt = iStr = iPrm = 0; + estrTmp = NULL; + bInQuotes = FALSE; + /* Set first parameter to binary */ + pData->aParams[iPrm] = strdup((char*)pData->szBinary); + dbgprintf("omprog: Param (%d): '%s'\n", iPrm, pData->aParams[iPrm]); + iPrm++; + while(iCnt < es_strlen(estrParams) ) { + if ( c[iCnt] == ' ' && !bInQuotes ) { + /* Copy into Param Array! */ + estrTmp = es_newStrFromSubStr( estrParams, iStr, iCnt-iStr); + } + else if ( iCnt+1 >= es_strlen(estrParams) ) { + /* Copy rest of string into Param Array! */ + estrTmp = es_newStrFromSubStr( estrParams, iStr, iCnt-iStr+1); + } + else if (c[iCnt] == '"') { + /* switch inQuotes Mode */ + bInQuotes = !bInQuotes; + } + + if ( estrTmp != NULL ) { + pData->aParams[iPrm] = es_str2cstr(estrTmp, NULL); + iStr = iCnt+1; /* Set new start */ + dbgprintf("omprog: Param (%d): '%s'\n", iPrm, pData->aParams[iPrm]); + es_deleteStr( estrTmp ); + estrTmp = NULL; + iPrm++; + } + + /*Next char*/ + iCnt++; + } + /* NULL last parameter! */ + pData->aParams[iPrm] = NULL; + + } } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { - dbgprintf("omprog: program error, non-handled " - "param '%s'\n", actpblk.descr[i].name); + dbgprintf("omprog: program error, non-handled param '%s'\n", actpblk.descr[i].name); } } @@ -405,6 +515,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c index 3e355464..3b1584e2 100644 --- a/plugins/omrelp/omrelp.c +++ b/plugins/omrelp/omrelp.c @@ -2,8 +2,17 @@ * * This is the implementation of the RELP output module. * - * NOTE: read comments in module-template.h to understand how this file - * works! + * Note that when multiple action workers are activated, we currently + * also create multiple actions. This may be the source of some mild + * message loss (!) if the worker instance is shut down while the + * connection to the remote system is in retry state. + * TODO: think if we should implement a mode where we do NOT + * support multiple action worker instances. This would be + * slower, but not have this loss opportunity. But it should + * definitely be optional and by default off due to the + * performance implications (and given the fact that message + * loss is pretty unlikely in usual cases). + * * * File begun on 2008-03-13 by RGerhards * @@ -55,29 +64,62 @@ DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) +#define DFLT_ENABLE_TLS 0 +#define DFLT_ENABLE_TLSZIP 0 + static relpEngine_t *pRelpEngine; /* our relp engine */ typedef struct _instanceData { uchar *target; uchar *port; - int bInitialConnect; /* is this the initial connection request of our module? (0-no, 1-yes) */ - int bIsConnected; /* currently connected to server? 0 - no, 1 - yes */ + int sizeWindow; /**< the RELP window size - 0=use default */ unsigned timeout; - relpClt_t *pRelpClt; /* relp client for this instance */ + unsigned rebindInterval; + sbool bEnableTLS; + sbool bEnableTLSZip; + sbool bHadAuthFail; /**< set on auth failure, will cause retry to disable action */ + uchar *pristring; /* GnuTLS priority string (NULL if not to be provided) */ + uchar *authmode; + uchar *caCertFile; + uchar *myCertFile; + uchar *myPrivKeyFile; uchar *tplName; + struct { + int nmemb; + uchar **name; + } permittedPeers; } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; + int bInitialConnect; /* is this the initial connection request of our module? (0-no, 1-yes) */ + int bIsConnected; /* currently connected to server? 0 - no, 1 - yes */ + relpClt_t *pRelpClt; /* relp client for this instance */ + unsigned nSent; /* number msgs sent - for rebind support */ +} wrkrInstanceData_t; + typedef struct configSettings_s { EMPTY_STRUCT } configSettings_t; static configSettings_t __attribute__((unused)) cs; +static rsRetVal doCreateRelpClient(wrkrInstanceData_t *pWrkrData); /* tables for interfacing with the v6 config system */ /* action (instance) parameters */ static struct cnfparamdescr actpdescr[] = { { "target", eCmdHdlrGetWord, 1 }, + { "tls", eCmdHdlrBinary, 0 }, + { "tls.compression", eCmdHdlrBinary, 0 }, + { "tls.prioritystring", eCmdHdlrString, 0 }, + { "tls.cacert", eCmdHdlrString, 0 }, + { "tls.mycert", eCmdHdlrString, 0 }, + { "tls.myprivkey", eCmdHdlrString, 0 }, + { "tls.authmode", eCmdHdlrString, 0 }, + { "tls.permittedpeer", eCmdHdlrArray, 0 }, { "port", eCmdHdlrGetWord, 0 }, + { "rebindinterval", eCmdHdlrInt, 0 }, + { "windowsize", eCmdHdlrInt, 0 }, { "timeout", eCmdHdlrInt, 0 }, { "template", eCmdHdlrGetWord, 0 } }; @@ -104,34 +146,126 @@ static uchar *getRelpPt(instanceData *pData) return(pData->port); } -static inline rsRetVal -doCreateRelpClient(instanceData *pData) +static void +onErr(void *pUsr, char *objinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) { + wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t*) pUsr; + errmsg.LogError(0, RS_RET_RELP_AUTH_FAIL, "omrelp[%s:%s]: error '%s', object " + " '%s' - action may not work as intended", + pWrkrData->pData->target, pWrkrData->pData->port, errmesg, objinfo); +} + +static void +onGenericErr(char *objinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) +{ + errmsg.LogError(0, RS_RET_RELP_ERR, "omrelp: librelp error '%s', object " + "'%s' - action may not work as intended", + errmesg, objinfo); +} + +static void +onAuthErr(void *pUsr, char *authinfo, char* errmesg, __attribute__((unused)) relpRetVal errcode) +{ + instanceData *pData = ((wrkrInstanceData_t*) pUsr)->pData; + errmsg.LogError(0, RS_RET_RELP_AUTH_FAIL, "omrelp[%s:%s]: authentication error '%s', peer " + "is '%s' - DISABLING action", pData->target, pData->port, errmesg, authinfo); + pData->bHadAuthFail = 1; +} + +static rsRetVal +doCreateRelpClient(wrkrInstanceData_t *pWrkrData) +{ + int i; + instanceData *pData; DEFiRet; - if(relpEngineCltConstruct(pRelpEngine, &pData->pRelpClt) != RELP_RET_OK) + + pData = pWrkrData->pData; + if(relpEngineCltConstruct(pRelpEngine, &pWrkrData->pRelpClt) != RELP_RET_OK) ABORT_FINALIZE(RS_RET_RELP_ERR); - if(relpCltSetTimeout(pData->pRelpClt, pData->timeout) != RELP_RET_OK) + if(relpCltSetTimeout(pWrkrData->pRelpClt, pData->timeout) != RELP_RET_OK) ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetWindowSize(pWrkrData->pRelpClt, pData->sizeWindow) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetUsrPtr(pWrkrData->pRelpClt, pWrkrData) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(pData->bEnableTLS) { + if(relpCltEnableTLS(pWrkrData->pRelpClt) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(pData->bEnableTLSZip) { + if(relpCltEnableTLSZip(pWrkrData->pRelpClt) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + } + if(relpCltSetGnuTLSPriString(pWrkrData->pRelpClt, (char*) pData->pristring) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetAuthMode(pWrkrData->pRelpClt, (char*) pData->authmode) != RELP_RET_OK) { + errmsg.LogError(0, RS_RET_RELP_ERR, + "omrelp: invalid auth mode '%s'\n", pData->authmode); + ABORT_FINALIZE(RS_RET_RELP_ERR); + } + if(relpCltSetCACert(pWrkrData->pRelpClt, (char*) pData->caCertFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetOwnCert(pWrkrData->pRelpClt, (char*) pData->myCertFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + if(relpCltSetPrivKey(pWrkrData->pRelpClt, (char*) pData->myPrivKeyFile) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + for(i = 0 ; i < pData->permittedPeers.nmemb ; ++i) { + relpCltAddPermittedPeer(pWrkrData->pRelpClt, (char*)pData->permittedPeers.name[i]); + } + } + if(glbl.GetSourceIPofLocalClient() == NULL) { /* ar Do we have a client IP set? */ + if(relpCltSetClientIP(pWrkrData->pRelpClt, glbl.GetSourceIPofLocalClient()) != RELP_RET_OK) + ABORT_FINALIZE(RS_RET_RELP_ERR); + } + pWrkrData->bInitialConnect = 1; + pWrkrData->nSent = 0; finalize_it: RETiRet; } - BEGINcreateInstance CODESTARTcreateInstance - pData->bInitialConnect = 1; + pData->sizeWindow = 0; pData->timeout = 90; + pData->rebindInterval = 0; + pData->bEnableTLS = DFLT_ENABLE_TLS; + pData->bEnableTLSZip = DFLT_ENABLE_TLSZIP; + pData->bHadAuthFail = 0; + pData->pristring = NULL; + pData->authmode = NULL; + pData->caCertFile = NULL; + pData->myCertFile = NULL; + pData->myPrivKeyFile = NULL; + pData->permittedPeers.nmemb = 0; ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + pWrkrData->pRelpClt = NULL; + iRet = doCreateRelpClient(pWrkrData); +ENDcreateWrkrInstance + BEGINfreeInstance + int i; CODESTARTfreeInstance - if(pData->pRelpClt != NULL) - relpEngineCltDestruct(pRelpEngine, &pData->pRelpClt); free(pData->target); free(pData->port); free(pData->tplName); + free(pData->pristring); + free(pData->authmode); + free(pData->caCertFile); + free(pData->myCertFile); + free(pData->myPrivKeyFile); + for(i = 0 ; i < pData->permittedPeers.nmemb ; ++i) { + free(pData->permittedPeers.name[i]); + } ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + if(pWrkrData->pRelpClt != NULL) + relpEngineCltDestruct(pRelpEngine, &pWrkrData->pRelpClt); +ENDfreeWrkrInstance + static inline void setInstParamDefaults(instanceData *pData) { @@ -139,12 +273,22 @@ setInstParamDefaults(instanceData *pData) pData->port = NULL; pData->tplName = NULL; pData->timeout = 90; + pData->sizeWindow = 0; + pData->rebindInterval = 0; + pData->bEnableTLS = DFLT_ENABLE_TLS; + pData->bEnableTLSZip = DFLT_ENABLE_TLSZIP; + pData->pristring = NULL; + pData->authmode = NULL; + pData->caCertFile = NULL; + pData->myCertFile = NULL; + pData->myPrivKeyFile = NULL; + pData->permittedPeers.nmemb = 0; } BEGINnewActInst struct cnfparamvals *pvals; - int i; + int i,j; CODESTARTnewActInst if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); @@ -164,6 +308,31 @@ CODESTARTnewActInst pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "timeout")) { pData->timeout = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "rebindinterval")) { + pData->rebindInterval = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "windowsize")) { + pData->sizeWindow = (int) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "tls")) { + pData->bEnableTLS = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "tls.compression")) { + pData->bEnableTLSZip = (unsigned) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "tls.prioritystring")) { + pData->pristring = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "tls.cacert")) { + pData->caCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "tls.mycert")) { + pData->myCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "tls.myprivkey")) { + pData->myPrivKeyFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "tls.authmode")) { + pData->authmode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "tls.permittedpeer")) { + pData->permittedPeers.nmemb = pvals[i].val.d.ar->nmemb; + CHKmalloc(pData->permittedPeers.name = + malloc(sizeof(uchar*) * pData->permittedPeers.nmemb)); + for(j = 0 ; j < pvals[i].val.d.ar->nmemb ; ++j) { + pData->permittedPeers.name[j] = (uchar*)es_str2cstr(pvals[i].val.d.ar->arr[j], NULL); + } } else { dbgprintf("omrelp: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); @@ -176,8 +345,6 @@ CODESTARTnewActInst "RSYSLOG_ForwardFormat" : (char*)pData->tplName), OMSR_NO_RQD_TPL_OPTS)); - CHKiRet(doCreateRelpClient(pData)); - CODE_STD_FINALIZERnewActInst if(pvals != NULL) cnfparamvalsDestruct(pvals, &actpblk); @@ -205,22 +372,23 @@ ENDdbgPrintInstInfo /* try to connect to server * rgerhards, 2008-03-21 */ -static rsRetVal doConnect(instanceData *pData) +static rsRetVal doConnect(wrkrInstanceData_t *pWrkrData) { DEFiRet; - if(pData->bInitialConnect) { - iRet = relpCltConnect(pData->pRelpClt, glbl.GetDefPFFamily(), pData->port, pData->target); + if(pWrkrData->bInitialConnect) { + iRet = relpCltConnect(pWrkrData->pRelpClt, glbl.GetDefPFFamily(), + pWrkrData->pData->port, pWrkrData->pData->target); if(iRet == RELP_RET_OK) - pData->bInitialConnect = 0; + pWrkrData->bInitialConnect = 0; } else { - iRet = relpCltReconnect(pData->pRelpClt); + iRet = relpCltReconnect(pWrkrData->pRelpClt); } if(iRet == RELP_RET_OK) { - pData->bIsConnected = 1; + pWrkrData->bIsConnected = 1; } else { - pData->bIsConnected = 0; + pWrkrData->bIsConnected = 0; iRet = RS_RET_SUSPENDED; } @@ -230,40 +398,87 @@ static rsRetVal doConnect(instanceData *pData) BEGINtryResume CODESTARTtryResume - iRet = doConnect(pData); + if(pWrkrData->pData->bHadAuthFail) { + ABORT_FINALIZE(RS_RET_DISABLE_ACTION); + } + iRet = doConnect(pWrkrData); +finalize_it: ENDtryResume +static inline rsRetVal +doRebind(wrkrInstanceData_t *pWrkrData) +{ + DEFiRet; + DBGPRINTF("omrelp: destructing relp client due to rebindInterval\n"); + CHKiRet(relpEngineCltDestruct(pRelpEngine, &pWrkrData->pRelpClt)); + pWrkrData->bIsConnected = 0; + CHKiRet(doCreateRelpClient(pWrkrData)); +finalize_it: + RETiRet; +} + +BEGINbeginTransaction +CODESTARTbeginTransaction +dbgprintf("omrelp: beginTransaction\n"); + if(!pWrkrData->bIsConnected) { + CHKiRet(doConnect(pWrkrData)); + } + relpCltHintBurstBegin(pWrkrData->pRelpClt); +finalize_it: +ENDbeginTransaction BEGINdoAction uchar *pMsg; /* temporary buffering */ size_t lenMsg; relpRetVal ret; + instanceData *pData; CODESTARTdoAction + pData = pWrkrData->pData; dbgprintf(" %s:%s/RELP\n", pData->target, getRelpPt(pData)); - if(!pData->bIsConnected) { - CHKiRet(doConnect(pData)); + if(!pWrkrData->bIsConnected) { + CHKiRet(doConnect(pWrkrData)); } pMsg = ppString[0]; lenMsg = strlen((char*) pMsg); /* TODO: don't we get this? */ - /* TODO: think about handling oversize messages! */ + /* we need to truncate oversize msgs - no way around that... */ if((int) lenMsg > glbl.GetMaxLine()) lenMsg = glbl.GetMaxLine(); /* forward */ - ret = relpCltSendSyslog(pData->pRelpClt, (uchar*) pMsg, lenMsg); + ret = relpCltSendSyslog(pWrkrData->pRelpClt, (uchar*) pMsg, lenMsg); if(ret != RELP_RET_OK) { /* error! */ dbgprintf("error forwarding via relp, suspending\n"); - iRet = RS_RET_SUSPENDED; + ABORT_FINALIZE(RS_RET_SUSPENDED); } + if(pData->rebindInterval != 0 && + (++pWrkrData->nSent >= pData->rebindInterval)) { + doRebind(pWrkrData); + } finalize_it: + if(pData->bHadAuthFail) + iRet = RS_RET_DISABLE_ACTION; + if(iRet == RS_RET_OK) { + /* we mimic non-commit, as otherwise our endTransaction handler + * will not get called. While this is not 100% correct, the worst + * that can happen is some message duplication, something that + * rsyslog generally accepts and prefers over message loss. + */ + iRet = RS_RET_PREVIOUS_COMMITTED; + } ENDdoAction +BEGINendTransaction +CODESTARTendTransaction + dbgprintf("omrelp: endTransaction\n"); + relpCltHintBurstEnd(pWrkrData->pRelpClt); +ENDendTransaction + BEGINparseSelectorAct uchar *q; int i; @@ -329,7 +544,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) ++p; } - /* TODO: make this if go away! */ if(*p == ';') { *p = '\0'; /* trick to obtain hostname (later)! */ CHKmalloc(pData->target = ustrdup(q)); @@ -341,8 +555,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* process template */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) "RSYSLOG_ForwardFormat")); - CHKiRet(doCreateRelpClient(pData)); - CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -360,8 +572,10 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES CODEqueryEtryPt_SetShutdownImmdtPtr ENDqueryEtryPt @@ -374,12 +588,12 @@ CODEmodInit_QueryRegCFSLineHdlr /* create our relp engine */ CHKiRet(relpEngineConstruct(&pRelpEngine)); CHKiRet(relpEngineSetDbgprint(pRelpEngine, dbgprintf)); + CHKiRet(relpEngineSetOnAuthErr(pRelpEngine, onAuthErr)); + CHKiRet(relpEngineSetOnGenericErr(pRelpEngine, onGenericErr)); + CHKiRet(relpEngineSetOnErr(pRelpEngine, onErr)); CHKiRet(relpEngineSetEnableCmd(pRelpEngine, (uchar*) "syslog", eRelpCmdState_Required)); /* tell which objects we need */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); ENDmodInit - -/* vim:set ai: - */ diff --git a/plugins/omruleset/omruleset.c b/plugins/omruleset/omruleset.c index 11765507..73419915 100644 --- a/plugins/omruleset/omruleset.c +++ b/plugins/omruleset/omruleset.c @@ -10,7 +10,7 @@ * * File begun on 2009-11-02 by RGerhards * - * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -70,6 +70,10 @@ typedef struct _instanceData { uchar *pszRulesetName; /* primarily for debugging/display purposes */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { ruleset_t *pRuleset; /* ruleset to enqueue message to (NULL = Default, not recommended) */ uchar *pszRulesetName; @@ -87,11 +91,21 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature ENDisCompatibleWithFeature +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINfreeInstance CODESTARTfreeInstance free(pData->pszRulesetName); @@ -117,9 +131,9 @@ BEGINdoAction CODESTARTdoAction CHKmalloc(pMsg = MsgDup((msg_t*) ppString[0])); DBGPRINTF(":omruleset: forwarding message %p to ruleset %s[%p]\n", pMsg, - (char*) pData->pszRulesetName, pData->pRuleset); + (char*) pWrkrData->pData->pszRulesetName, pWrkrData->pData->pRuleset); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); - MsgSetRuleset(pMsg, pData->pRuleset); + MsgSetRuleset(pMsg, pWrkrData->pData->pRuleset); /* Note: we intentionally use submitMsg2() here, as we process messages * that were already run through the rate-limiter. So it is (at least) * questionable if they were rate-limited again. @@ -199,6 +213,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omsnmp/omsnmp.c b/plugins/omsnmp/omsnmp.c index 42d1de6b..02862c94 100644 --- a/plugins/omsnmp/omsnmp.c +++ b/plugins/omsnmp/omsnmp.c @@ -2,7 +2,7 @@ * * This module sends an snmp trap. * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -74,15 +74,19 @@ typedef struct _instanceData { * http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt * http://www.adiscon.org/download/ADISCON-MIB.txt */ - int iPort; /* Target Port */ - int iSNMPVersion; /* SNMP Version to use */ - int iTrapType; /* Snmp TrapType or GenericType */ - int iSpecificType; /* Snmp Specific Type */ + int iPort; /* Target Port */ + int iSNMPVersion; /* SNMP Version to use */ + int iTrapType; /* Snmp TrapType or GenericType */ + int iSpecificType; /* Snmp Specific Type */ - netsnmp_session *snmpsession; /* Holds to SNMP Session, NULL if not initialized */ - uchar *tplName; /* format template to use */ + uchar *tplName; /* format template to use */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; + netsnmp_session *snmpsession; /* Holds to SNMP Session, NULL if not initialized */ +} wrkrInstanceData_t; + typedef struct configSettings_s { uchar* pszTransport; /* default transport */ uchar* pszTarget; @@ -147,6 +151,10 @@ BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + pWrkrData->snmpsession = NULL; +ENDcreateWrkrInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -171,14 +179,16 @@ ENDisCompatibleWithFeature /* Exit SNMP Session * alorbach, 2008-02-12 */ -static rsRetVal omsnmp_exitSession(instanceData *pData) +static rsRetVal +omsnmp_exitSession(wrkrInstanceData_t *pWrkrData) { DEFiRet; - if(pData->snmpsession != NULL) { - dbgprintf( "omsnmp_exitSession: Clearing Session to '%s' on Port = '%d'\n", pData->szTarget, pData->iPort); - snmp_close(pData->snmpsession); - pData->snmpsession = NULL; + if(pWrkrData->snmpsession != NULL) { + DBGPRINTF("omsnmp_exitSession: Clearing Session to '%s' on Port = '%d'\n", + pWrkrData->pData->szTarget, pWrkrData->pData->iPort); + snmp_close(pWrkrData->snmpsession); + pWrkrData->snmpsession = NULL; } RETiRet; @@ -187,15 +197,19 @@ static rsRetVal omsnmp_exitSession(instanceData *pData) /* Init SNMP Session * alorbach, 2008-02-12 */ -static rsRetVal omsnmp_initSession(instanceData *pData) +static rsRetVal +omsnmp_initSession(wrkrInstanceData_t *pWrkrData) { netsnmp_session session; + instanceData *pData; char szTargetAndPort[MAXHOSTNAMELEN+128]; /* work buffer for specifying a full target and port string */ DEFiRet; /* should not happen, but if session is not cleared yet - we do it now! */ - if (pData->snmpsession != NULL) - omsnmp_exitSession(pData); + if (pWrkrData->snmpsession != NULL) + omsnmp_exitSession(pWrkrData); + + pData = pWrkrData->pData; snprintf((char*)szTargetAndPort, sizeof(szTargetAndPort), "%s:%s:%d", (pData->szTransport == NULL) ? "udp" : (char*)pData->szTransport, @@ -217,8 +231,8 @@ static rsRetVal omsnmp_initSession(instanceData *pData) session.community_len = strlen((char*) session.community); } - pData->snmpsession = snmp_open(&session); - if (pData->snmpsession == NULL) { + pWrkrData->snmpsession = snmp_open(&session); + if (pWrkrData->snmpsession == NULL) { errmsg.LogError(0, RS_RET_SUSPENDED, "omsnmp_initSession: snmp_open to host '%s' on Port '%d' failed\n", pData->szTarget, pData->iPort); /* Stay suspended */ iRet = RS_RET_SUSPENDED; @@ -227,7 +241,7 @@ static rsRetVal omsnmp_initSession(instanceData *pData) RETiRet; } -static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) +static rsRetVal omsnmp_sendsnmp(wrkrInstanceData_t *pWrkrData, uchar *psz) { DEFiRet; @@ -239,10 +253,12 @@ static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) int status; char *trap = NULL; const char *strErr = NULL; + instanceData *pData; + pData = pWrkrData->pData; /* Init SNMP Session if necessary */ - if (pData->snmpsession == NULL) { - CHKiRet(omsnmp_initSession(pData)); + if (pWrkrData->snmpsession == NULL) { + CHKiRet(omsnmp_initSession(pWrkrData)); } /* String should not be NULL */ @@ -250,7 +266,7 @@ static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) dbgprintf( "omsnmp_sendsnmp: ENTER - Syslogmessage = '%s'\n", (char*)psz); /* If SNMP Version1 is configured !*/ - if(pData->snmpsession->version == SNMP_VERSION_1) { + if(pWrkrData->snmpsession->version == SNMP_VERSION_1) { pdu = snmp_pdu_create(SNMP_MSG_TRAP); /* Set enterprise */ @@ -275,7 +291,7 @@ static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) pdu->time = get_uptime(); } /* If SNMP Version2c is configured !*/ - else if (pData->snmpsession->version == SNMP_VERSION_2c) + else if (pWrkrData->snmpsession->version == SNMP_VERSION_2c) { long sysuptime; char csysuptime[20]; @@ -320,15 +336,15 @@ static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) } /* Send the TRAP */ - status = snmp_send(pData->snmpsession, pdu) == 0; + status = snmp_send(pWrkrData->snmpsession, pdu) == 0; if (status) { /* Debug Output! */ - int iErrorCode = pData->snmpsession->s_snmp_errno; + int iErrorCode = pWrkrData->snmpsession->s_snmp_errno; errmsg.LogError(0, RS_RET_SUSPENDED, "omsnmp_sendsnmp: snmp_send failed error '%d', Description='%s'\n", iErrorCode*(-1), api_errors[iErrorCode*(-1)]); /* Clear Session */ - omsnmp_exitSession(pData); + omsnmp_exitSession(pWrkrData); ABORT_FINALIZE(RS_RET_SUSPENDED); } @@ -347,7 +363,7 @@ finalize_it: BEGINtryResume CODESTARTtryResume - iRet = omsnmp_initSession(pData); + iRet = omsnmp_initSession(pWrkrData); ENDtryResume BEGINdoAction @@ -358,19 +374,20 @@ CODESTARTdoAction } /* This will generate and send the SNMP Trap */ - iRet = omsnmp_sendsnmp(pData, ppString[0]); + iRet = omsnmp_sendsnmp(pWrkrData, ppString[0]); finalize_it: ENDdoAction BEGINfreeInstance CODESTARTfreeInstance - /* free snmp Session here */ - omsnmp_exitSession(pData); - free(pData->tplName); free(pData->szTarget); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + omsnmp_exitSession(pWrkrData); +ENDfreeWrkrInstance static inline void setInstParamDefaults(instanceData *pData) @@ -499,9 +516,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* Set some defaults in the NetSNMP library */ netsnmp_ds_set_int(NETSNMP_DS_LIBRARY_ID, NETSNMP_DS_LIB_DEFAULT_PORT, pData->iPort ); - - /* Init Session Pointer */ - pData->snmpsession = NULL; CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -545,6 +559,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt diff --git a/plugins/omstdout/omstdout.c b/plugins/omstdout/omstdout.c index a84a7593..210b0165 100644 --- a/plugins/omstdout/omstdout.c +++ b/plugins/omstdout/omstdout.c @@ -6,7 +6,7 @@ * * File begun on 2009-03-19 by RGerhards * - * Copyright 2009-2012 Adiscon GmbH. + * Copyright 2009-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -60,6 +60,10 @@ typedef struct _instanceData { int bEnsureLFEnding; /* ensure that a linefeed is written at the end of EACH record (test aid for nettester) */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { int bUseArrayInterface; /* shall action use array instead of string template interface? */ int bEnsureLFEnding; /* shall action use array instead of string template interface? */ @@ -76,6 +80,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -88,6 +97,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo ENDdbgPrintInstInfo @@ -107,7 +121,7 @@ BEGINdoAction size_t len; int r; CODESTARTdoAction - if(pData->bUseArrayInterface) { + if(pWrkrData->pData->bUseArrayInterface) { /* if we use array passing, we need to put together a string * ourselves. At this point, please keep in mind that omstdout is * primarily a testing aid. Other modules may do different processing @@ -145,7 +159,7 @@ CODESTARTdoAction DBGPRINTF("omstdout: error %d writing to stdout[%d]: %s\n", r, len, toWrite); } - if(pData->bEnsureLFEnding && toWrite[len-1] != '\n') { + if(pWrkrData->pData->bEnsureLFEnding && toWrite[len-1] != '\n') { if((r = write(1, "\n", 1)) != 1) { /* write missing LF */ DBGPRINTF("omstdout: error %d writing \\n to stdout\n", r); @@ -186,6 +200,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omtesting/omtesting.c b/plugins/omtesting/omtesting.c index c9f1e06b..2cc1159e 100644 --- a/plugins/omtesting/omtesting.c +++ b/plugins/omtesting/omtesting.c @@ -22,7 +22,7 @@ * NOTE: read comments in module-template.h to understand how this file * works! * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -63,7 +63,6 @@ MODULE_CNFNAME("omtesting") */ DEF_OMOD_STATIC_DATA - typedef struct _instanceData { enum { MD_SLEEP, MD_FAIL, MD_RANDFAIL, MD_ALWAYS_SUSPEND } mode; @@ -76,6 +75,10 @@ typedef struct _instanceData { int iCurrRetries; } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { int bEchoStdout; /* echo non-failed messages to stdout */ } configSettings_t; @@ -93,6 +96,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("Action delays rule by %d second(s) and %d millisecond(s)\n", @@ -170,11 +178,11 @@ static rsRetVal doRandFail(void) BEGINtryResume CODESTARTtryResume dbgprintf("omtesting tryResume() called\n"); - switch(pData->mode) { + switch(pWrkrData->pData->mode) { case MD_SLEEP: break; case MD_FAIL: - iRet = doFailOnResume(pData); + iRet = doFailOnResume(pWrkrData->pData); break; case MD_RANDFAIL: iRet = doRandFail(); @@ -187,8 +195,10 @@ ENDtryResume BEGINdoAction + instanceData *pData; CODESTARTdoAction dbgprintf("omtesting received msg '%s'\n", ppString[0]); + pData = pWrkrData->pData; switch(pData->mode) { case MD_SLEEP: iRet = doSleep(pData); @@ -220,6 +230,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINparseSelectorAct int i; uchar szBuf[1024]; @@ -313,6 +328,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES ENDqueryEtryPt diff --git a/plugins/omudpspoof/omudpspoof.c b/plugins/omudpspoof/omudpspoof.c index c80f0e57..ad3508c8 100644 --- a/plugins/omudpspoof/omudpspoof.c +++ b/plugins/omudpspoof/omudpspoof.c @@ -98,15 +98,19 @@ typedef struct _instanceData { uchar *port; uchar *sourceTpl; int mtu; - int *pSockArray; /* sockets to use for UDP */ - struct addrinfo *f_addr; - u_short sourcePort; u_short sourcePortStart; /* for sorce port iteration */ u_short sourcePortEnd; int bReportLibnetInitErr; /* help prevent multiple error messages on init err */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; libnet_t *libnet_handle; + u_short sourcePort; + int *pSockArray; /* sockets to use for UDP */ + struct addrinfo *f_addr; char errbuf[LIBNET_ERRBUF_SIZE]; -} instanceData; +} wrkrInstanceData_t; #define DFLT_SOURCE_PORT_START 32000 #define DFLT_SOURCE_PORT_END 42000 @@ -172,7 +176,7 @@ ENDinitConfVars pthread_mutex_t mutLibnet; /* forward definitions */ -static rsRetVal doTryResume(instanceData *pData); +static rsRetVal doTryResume(wrkrInstanceData_t *pWrkrData); /* this function gets the default template. It coordinates action between @@ -217,15 +221,14 @@ finalize_it: * rgerhards, 2009-05-29 */ static rsRetVal -closeUDPSockets(instanceData *pData) +closeUDPSockets(wrkrInstanceData_t *pWrkrData) { DEFiRet; - assert(pData != NULL); - if(pData->pSockArray != NULL) { - net.closeUDPListenSockets(pData->pSockArray); - pData->pSockArray = NULL; - freeaddrinfo(pData->f_addr); - pData->f_addr = NULL; + if(pWrkrData->pSockArray != NULL) { + net.closeUDPListenSockets(pWrkrData->pSockArray); + pWrkrData->pSockArray = NULL; + freeaddrinfo(pWrkrData->f_addr); + pWrkrData->f_addr = NULL; } RETiRet; } @@ -310,12 +313,17 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance - pData->libnet_handle = NULL; pData->mtu = 1500; pData->bReportLibnetInitErr = 1; ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + pWrkrData->libnet_handle = NULL; + pWrkrData->sourcePort = pData->sourcePortStart; +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -326,15 +334,19 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance /* final cleanup */ - closeUDPSockets(pData); free(pData->tplName); free(pData->port); free(pData->host); free(pData->sourceTpl); - if(pData->libnet_handle != NULL) - libnet_destroy(pData->libnet_handle); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + closeUDPSockets(pWrkrData); + if(pWrkrData->libnet_handle != NULL) + libnet_destroy(pWrkrData->libnet_handle); +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -348,11 +360,12 @@ ENDdbgPrintInstInfo * rgehards, 2007-12-20 */ static inline rsRetVal -UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) +UDPSend(wrkrInstanceData_t *pWrkrData, uchar *pszSourcename, char *msg, size_t len) { struct addrinfo *r; int lsent = 0; int bSendSuccess; + instanceData *pData; struct sockaddr_in *tempaddr,source_ip; libnet_ptag_t ip, ipo; libnet_ptag_t udp; @@ -363,9 +376,10 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) unsigned maxPktLen, pktLen; DEFiRet; - if(pData->pSockArray == NULL) { - CHKiRet(doTryResume(pData)); + if(pWrkrData->pSockArray == NULL) { + CHKiRet(doTryResume(pWrkrData)); } + pData = pWrkrData->pData; if(len > 65528) { DBGPRINTF("omudpspoof: msg with length %d truncated to 64k: '%.768s'\n", @@ -374,8 +388,8 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) } ip = ipo = udp = 0; - if(pData->sourcePort++ >= pData->sourcePortEnd){ - pData->sourcePort = pData->sourcePortStart; + if(pWrkrData->sourcePort++ >= pData->sourcePortEnd){ + pWrkrData->sourcePort = pData->sourcePortStart; } inet_pton(AF_INET, (char*)pszSourcename, &(source_ip.sin_addr)); @@ -383,7 +397,7 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) bSendSuccess = RSFALSE; d_pthread_mutex_lock(&mutLibnet); bNeedUnlock = 1; - for (r = pData->f_addr; r && bSendSuccess == RSFALSE ; r = r->ai_next) { + for (r = pWrkrData->f_addr; r && bSendSuccess == RSFALSE ; r = r->ai_next) { tempaddr = (struct sockaddr_in *)r->ai_addr; /* Getting max payload size (must be multiple of 8) */ maxPktLen = (pData->mtu - LIBNET_IPV4_H) & ~0x07; @@ -400,19 +414,19 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) } DBGPRINTF("omudpspoof: stage 1: MF:%d, hdrOffs %d, pktLen %d\n", (hdrOffs & IP_MF) >> 13, (hdrOffs & 0x1FFF) << 3, pktLen); - libnet_clear_packet(pData->libnet_handle); + libnet_clear_packet(pWrkrData->libnet_handle); /* note: libnet does need ports in host order NOT in network byte order! -- rgerhards, 2009-11-12 */ udp = libnet_build_udp( - ntohs(pData->sourcePort),/* source port */ + ntohs(pWrkrData->sourcePort),/* source port */ ntohs(tempaddr->sin_port),/* destination port */ pktLen+LIBNET_UDP_H, /* packet length */ 0, /* checksum */ (u_char*)msg, /* payload */ pktLen, /* payload size */ - pData->libnet_handle, /* libnet handle */ + pWrkrData->libnet_handle, /* libnet handle */ udp); /* libnet id */ if (udp == -1) { - DBGPRINTF("omudpspoof: can't build UDP header: %s\n", libnet_geterror(pData->libnet_handle)); + DBGPRINTF("omudpspoof: can't build UDP header: %s\n", libnet_geterror(pWrkrData->libnet_handle)); } ip = libnet_build_ipv4( @@ -427,22 +441,22 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) tempaddr->sin_addr.s_addr, NULL, /* payload */ 0, /* payload size */ - pData->libnet_handle, /* libnet handle */ + pWrkrData->libnet_handle, /* libnet handle */ ip); /* libnet id */ if (ip == -1) { - DBGPRINTF("omudpspoof: can't build IP header: %s\n", libnet_geterror(pData->libnet_handle)); + DBGPRINTF("omudpspoof: can't build IP header: %s\n", libnet_geterror(pWrkrData->libnet_handle)); } /* Write it to the wire. */ - lsent = libnet_write(pData->libnet_handle); + lsent = libnet_write(pWrkrData->libnet_handle); if(lsent != (int) (LIBNET_IPV4_H+LIBNET_UDP_H+pktLen)) { /* note: access to fd is a libnet internal. If a newer version of libnet does * not expose that member, we should simply remove it. However, while it is there * it is useful for consolidating with strace output. */ DBGPRINTF("omudpspoof: write error (total len %d): pktLen %d, sent %d, fd %d: %s\n", - len, LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, lsent, pData->libnet_handle->fd, - libnet_geterror(pData->libnet_handle)); + len, LIBNET_IPV4_H+LIBNET_UDP_H+pktLen, lsent, pWrkrData->libnet_handle->fd, + libnet_geterror(pWrkrData->libnet_handle)); if(lsent != -1) { bSendSuccess = RSTRUE; } @@ -452,7 +466,7 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) msgOffs += pktLen; /* We need to get rid of the UDP header to build the other fragments */ - libnet_clear_packet(pData->libnet_handle); + libnet_clear_packet(pWrkrData->libnet_handle); ip = LIBNET_PTAG_INITIALIZER; while(len > msgOffs ) { /* loop until all payload is sent */ /* check if there will be more fragments */ @@ -481,16 +495,16 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) tempaddr->sin_addr.s_addr, (uint8_t*)(msg+msgOffs), /* payload */ pktLen, /* payload size */ - pData->libnet_handle, /* libnet handle */ + pWrkrData->libnet_handle, /* libnet handle */ ip); /* libnet id */ if (ip == -1) { - DBGPRINTF("omudpspoof: can't build IP fragment header: %s\n", libnet_geterror(pData->libnet_handle)); + DBGPRINTF("omudpspoof: can't build IP fragment header: %s\n", libnet_geterror(pWrkrData->libnet_handle)); } /* Write it to the wire. */ - lsent = libnet_write(pData->libnet_handle); + lsent = libnet_write(pWrkrData->libnet_handle); if(lsent != (int) (LIBNET_IPV4_H+pktLen)) { DBGPRINTF("omudpspoof: fragment write error len %d, sent %d: %s\n", - LIBNET_IPV4_H+LIBNET_UDP_H+len, lsent, libnet_geterror(pData->libnet_handle)); + LIBNET_IPV4_H+LIBNET_UDP_H+len, lsent, libnet_geterror(pWrkrData->libnet_handle)); bSendSuccess = RSFALSE; continue; } @@ -500,9 +514,9 @@ UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) finalize_it: if(iRet != RS_RET_OK) { - if(pData->libnet_handle != NULL) { - libnet_destroy(pData->libnet_handle); - pData->libnet_handle = NULL; + if(pWrkrData->libnet_handle != NULL) { + libnet_destroy(pWrkrData->libnet_handle); + pWrkrData->libnet_handle = NULL; } } if(bNeedUnlock) { @@ -515,29 +529,28 @@ finalize_it: /* try to resume connection if it is not ready * rgerhards, 2007-08-02 */ -static rsRetVal doTryResume(instanceData *pData) +static rsRetVal doTryResume(wrkrInstanceData_t *pWrkrData) { int iErr; struct addrinfo *res; struct addrinfo hints; + instanceData *pData; DEFiRet; - if(pData->pSockArray != NULL) + if(pWrkrData->pSockArray != NULL) FINALIZE; + pData = pWrkrData->pData; - if(pData->host == NULL) - ABORT_FINALIZE(RS_RET_DISABLE_ACTION); - - if(pData->libnet_handle == NULL) { + if(pWrkrData->libnet_handle == NULL) { /* Initialize the libnet library. Root priviledges are required. * this initializes a IPv4 socket to use for forging UDP packets. */ - pData->libnet_handle = libnet_init( + pWrkrData->libnet_handle = libnet_init( LIBNET_RAW4, /* injection type */ NULL, /* network interface */ - pData->errbuf); /* errbuf */ + pWrkrData->errbuf); /* errbuf */ - if(pData->libnet_handle == NULL) { + if(pWrkrData->libnet_handle == NULL) { if(pData->bReportLibnetInitErr) { errmsg.LogError(0, RS_RET_ERR_LIBNET_INIT, "omudpsoof: error " "initializing libnet - are you running as root?"); @@ -562,17 +575,16 @@ static rsRetVal doTryResume(instanceData *pData) ABORT_FINALIZE(RS_RET_SUSPENDED); } DBGPRINTF("%s found, resuming.\n", pData->host); - pData->f_addr = res; - pData->pSockArray = net.create_udp_socket((uchar*)pData->host, NULL, 0); + pWrkrData->f_addr = res; + pWrkrData->pSockArray = net.create_udp_socket((uchar*)pData->host, NULL, 0, 0); finalize_it: if(iRet != RS_RET_OK) { - if(pData->f_addr != NULL) { - freeaddrinfo(pData->f_addr); - pData->f_addr = NULL; + if(pWrkrData->f_addr != NULL) { + freeaddrinfo(pWrkrData->f_addr); + pWrkrData->f_addr = NULL; } - if(iRet != RS_RET_DISABLE_ACTION) - iRet = RS_RET_SUSPENDED; + iRet = RS_RET_SUSPENDED; } RETiRet; @@ -581,7 +593,7 @@ finalize_it: BEGINtryResume CODESTARTtryResume - iRet = doTryResume(pData); + iRet = doTryResume(pWrkrData); ENDtryResume BEGINdoAction @@ -589,10 +601,10 @@ BEGINdoAction unsigned l; int iMaxLine; CODESTARTdoAction - CHKiRet(doTryResume(pData)); + CHKiRet(doTryResume(pWrkrData)); - DBGPRINTF(" %s:%s/omudpspoof, src '%s', msg strt '%.256s'\n", pData->host, - getFwdPt(pData), ppString[1], ppString[0]); + DBGPRINTF(" %s:%s/omudpspoof, src '%s', msg strt '%.256s'\n", pWrkrData->pData->host, + getFwdPt(pWrkrData->pData), ppString[1], ppString[0]); iMaxLine = glbl.GetMaxLine(); psz = (char*) ppString[0]; @@ -600,7 +612,7 @@ CODESTARTdoAction if((int) l > iMaxLine) l = iMaxLine; - CHKiRet(UDPSend(pData, ppString[1], psz, l)); + CHKiRet(UDPSend(pWrkrData, ppString[1], psz, l)); finalize_it: ENDdoAction @@ -664,7 +676,6 @@ CODESTARTnewActInst } } CODE_STD_STRING_REQUESTnewActInst(2) - pData->sourcePort = pData->sourcePortStart; tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); @@ -703,7 +714,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(2) else CHKmalloc(pData->port = ustrdup(cs.pszTargetPort)); CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(sourceTpl), OMSR_NO_RQD_TPL_OPTS)); - pData->sourcePort = pData->sourcePortStart = cs.iSourcePortStart; + pData->sourcePortStart = cs.iSourcePortStart; pData->sourcePortEnd = cs.iSourcePortEnd; /* process template */ @@ -744,6 +755,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES diff --git a/plugins/omuxsock/omuxsock.c b/plugins/omuxsock/omuxsock.c index 583b9f94..da4e8e94 100644 --- a/plugins/omuxsock/omuxsock.c +++ b/plugins/omuxsock/omuxsock.c @@ -4,7 +4,7 @@ * NOTE: read comments in module-template.h to understand how this file * works! * - * Copyright 2010-2012 Adiscon GmbH. + * Copyright 2010-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -60,10 +60,14 @@ typedef struct _instanceData { permittedPeers_t *pPermPeers; uchar *sockName; int sock; - int bIsConnected; /* are we connected to remote host? 0 - no, 1 - yes, UDP means addr resolved */ struct sockaddr_un addr; } instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + /* config data */ typedef struct configSettings_s { uchar *tplName; /* name of the default template to use */ @@ -90,6 +94,7 @@ static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current l static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ +static pthread_mutex_t mutDoAct = PTHREAD_MUTEX_INITIALIZER; BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars @@ -147,7 +152,6 @@ closeSocket(instanceData *pData) close(pData->sock); pData->sock = INVLD_SOCK; } -pData->bIsConnected = 0; // TODO: remove this variable altogether RETiRet; } @@ -224,6 +228,10 @@ CODESTARTcreateInstance pData->sock = INVLD_SOCK; ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature @@ -239,6 +247,10 @@ CODESTARTfreeInstance free(pData->sockName); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo @@ -332,7 +344,7 @@ static rsRetVal doTryResume(instanceData *pData) BEGINtryResume CODESTARTtryResume - iRet = doTryResume(pData); + iRet = doTryResume(pWrkrData->pData); ENDtryResume BEGINdoAction @@ -340,20 +352,22 @@ BEGINdoAction register unsigned l; int iMaxLine; CODESTARTdoAction - CHKiRet(doTryResume(pData)); + pthread_mutex_lock(&mutDoAct); + CHKiRet(doTryResume(pWrkrData->pData)); iMaxLine = glbl.GetMaxLine(); - DBGPRINTF(" omuxsock:%s\n", pData->sockName); + DBGPRINTF(" omuxsock:%s\n", pWrkrData->pData->sockName); psz = (char*) ppString[0]; l = strlen((char*) psz); if((int) l > iMaxLine) l = iMaxLine; - CHKiRet(sendMsg(pData, psz, l)); + CHKiRet(sendMsg(pWrkrData->pData, psz, l)); finalize_it: + pthread_mutex_unlock(&mutDoAct); ENDdoAction @@ -413,6 +427,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES ENDqueryEtryPt diff --git a/plugins/pmlastmsg/pmlastmsg.c b/plugins/pmlastmsg/pmlastmsg.c index a290c446..8c475fbb 100644 --- a/plugins/pmlastmsg/pmlastmsg.c +++ b/plugins/pmlastmsg/pmlastmsg.c @@ -132,7 +132,7 @@ dbgprintf("wrong closing text!\n"); */ DBGPRINTF("pmlastmsg detected a \"last message repeated n times\" message\n"); - setProtocolVersion(pMsg, 0); + setProtocolVersion(pMsg, MSG_LEGACY_PROTOCOL); memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); MsgSetMSGoffs(pMsg, pMsg->offAfterPRI); /* we don't have a header! */ MsgSetTAG(pMsg, (uchar*)"", 0); diff --git a/plugins/pmrfc3164sd/pmrfc3164sd.c b/plugins/pmrfc3164sd/pmrfc3164sd.c index de5805bc..0ed4a56c 100644 --- a/plugins/pmrfc3164sd/pmrfc3164sd.c +++ b/plugins/pmrfc3164sd/pmrfc3164sd.c @@ -177,7 +177,7 @@ CODESTARTparse assert(pMsg->pszRawMsg != NULL); lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; /* note: offAfterPRI is already the number of PRI chars (do not add one!) */ p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ - setProtocolVersion(pMsg, 0); + setProtocolVersion(pMsg, MSG_LEGACY_PROTOCOL); /* Check to see if msg contains a timestamp. We start by assuming * that the message timestamp is the time of reception (which we diff --git a/runtime/Makefile.am b/runtime/Makefile.am index 34384bea..230be69e 100644 --- a/runtime/Makefile.am +++ b/runtime/Makefile.am @@ -69,6 +69,8 @@ librsyslog_la_SOURCES = \ prop.h \ ratelimit.c \ ratelimit.h \ + lookup.c \ + lookup.h \ cfsysline.c \ cfsysline.h \ sd-daemon.c \ diff --git a/runtime/batch.h b/runtime/batch.h index 2ec07670..b0aa8574 100644 --- a/runtime/batch.h +++ b/runtime/batch.h @@ -46,17 +46,6 @@ typedef unsigned char batch_state_t; */ struct batch_obj_s { msg_t *pMsg; - /* work variables for action processing; these are reused for each action (or block of - * actions) - */ - sbool bPrevWasSuspended; - /* following are caches to save allocs if not absolutely necessary */ - uchar *staticActStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /**< for strings */ - /* a cache to save malloc(), if not absolutely necessary */ - void *staticActParams[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /**< for anything else */ - size_t staticLenStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; - /* and the same for the message length (if used) */ - /* end action work variables */ }; /* the batch @@ -77,11 +66,7 @@ struct batch_s { int maxElem; /* maximum number of elements that this batch supports */ int nElem; /* actual number of element in this entry */ int nElemDeq; /* actual number of elements dequeued (and thus to be deleted) - see comment above! */ - int iDoneUpTo; /* all messages below this index have state other than RDY */ qDeqID deqID; /* ID of dequeue operation that generated this batch */ - int *pbShutdownImmediate;/* end processing of this batch immediately if set to 1 */ - sbool *active; /* which messages are active for processing, NULL=all */ - sbool bSingleRuleset; /* do all msgs of this batch use a single ruleset? */ batch_obj_t *pElem; /* batch elements */ batch_state_t *eltState;/* state (array!) for individual objects. NOTE: we have moved this out of batch_obj_t because we @@ -93,24 +78,6 @@ struct batch_s { }; -/* some inline functions (we may move this off to an object .. or not) */ -static inline void -batchSetSingleRuleset(batch_t *pBatch, sbool val) { - pBatch->bSingleRuleset = val; -} - -/* get the batches ruleset (if we have a single ruleset) */ -static inline ruleset_t* -batchGetRuleset(batch_t *pBatch) { - return (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; -} - -/* get the ruleset of a specifc element of the batch (index not verified!) */ -static inline ruleset_t* -batchElemGetRuleset(batch_t *pBatch, int i) { - return pBatch->pElem[i].pMsg->pRuleset; -} - /* get number of msgs for this batch */ static inline int batchNumMsgs(batch_t *pBatch) { @@ -134,8 +101,7 @@ batchSetElemState(batch_t *pBatch, int i, batch_state_t newState) { */ static inline int batchIsValidElem(batch_t *pBatch, int i) { - return( (pBatch->eltState[i] != BATCH_STATE_DISC) - && (pBatch->active == NULL || pBatch->active[i])); + return(pBatch->eltState[i] != BATCH_STATE_DISC); } @@ -152,7 +118,7 @@ batchFree(batch_t *pBatch) { /* staticActParams MUST be freed immediately (if required), * so we do not need to do that! */ - free(pBatch->pElem[i].staticActStrings[j]); + //TODO: do this in wti! free(pBatch->pElem[i].staticActStrings[j]); } } free(pBatch->pElem); @@ -167,11 +133,9 @@ batchFree(batch_t *pBatch) { static inline rsRetVal batchInit(batch_t *pBatch, int maxElem) { DEFiRet; - pBatch->iDoneUpTo = 0; pBatch->maxElem = maxElem; CHKmalloc(pBatch->pElem = calloc((size_t)maxElem, sizeof(batch_obj_t))); CHKmalloc(pBatch->eltState = calloc((size_t)maxElem, sizeof(batch_state_t))); - // TODO: replace calloc by inidividual writes? finalize_it: RETiRet; } diff --git a/runtime/conf.c b/runtime/conf.c index c01715cb..83931bca 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -486,111 +486,6 @@ rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]) } -/* Helper to cfline(). This function takes the filter part of a property - * based filter and decodes it. It processes the line up to the beginning - * of the action part. A pointer to that beginnig is passed back to the caller. - * rgerhards 2005-09-15 - */ -rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt) -{ - rsParsObj *pPars; - cstr_t *pCSCompOp; - cstr_t *pCSPropName; - rsRetVal iRet; - int iOffset; /* for compare operations */ - - ASSERT(pline != NULL); - - dbgprintf("Decoding property-based filter '%s'\n", pline); - - /* create parser object starting with line string without leading colon */ - if((iRet = rsParsConstructFromSz(&pPars, pline+1)) != RS_RET_OK) { - errmsg.LogError(0, iRet, "Error %d constructing parser object - ignoring selector", iRet); - return(iRet); - } - - /* read property */ - iRet = parsDelimCStr(pPars, &pCSPropName, ',', 1, 1, 1); - if(iRet != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - iRet = propNameToID(pCSPropName, &stmt->d.s_propfilt.propID); - if(iRet != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - if(stmt->d.s_propfilt.propID == PROP_CEE) { - /* in CEE case, we need to preserve the actual property name */ - if((stmt->d.s_propfilt.propName = - es_newStrFromBuf((char*)cstrGetSzStrNoNULL(pCSPropName)+2, cstrLen(pCSPropName)-2)) == NULL) { - cstrDestruct(&pCSPropName); - return(RS_RET_ERR); - } - } - cstrDestruct(&pCSPropName); - - /* read operation */ - iRet = parsDelimCStr(pPars, &pCSCompOp, ',', 1, 1, 1); - if(iRet != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d compare operation property - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - - /* we now first check if the condition is to be negated. To do so, we first - * must make sure we have at least one char in the param and then check the - * first one. - * rgerhards, 2005-09-26 - */ - if(rsCStrLen(pCSCompOp) > 0) { - if(*rsCStrGetBufBeg(pCSCompOp) == '!') { - stmt->d.s_propfilt.isNegated = 1; - iOffset = 1; /* ignore '!' */ - } else { - stmt->d.s_propfilt.isNegated = 0; - iOffset = 0; - } - } else { - stmt->d.s_propfilt.isNegated = 0; - iOffset = 0; - } - - if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "contains", 8)) { - stmt->d.s_propfilt.operation = FIOP_CONTAINS; - } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isequal", 7)) { - stmt->d.s_propfilt.operation = FIOP_ISEQUAL; - } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "isempty", 7)) { - stmt->d.s_propfilt.operation = FIOP_ISEMPTY; - } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (uchar*) "startswith", 10)) { - stmt->d.s_propfilt.operation = FIOP_STARTSWITH; - } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "regex", 5)) { - stmt->d.s_propfilt.operation = FIOP_REGEX; - } else if(!rsCStrOffsetSzStrCmp(pCSCompOp, iOffset, (unsigned char*) "ereregex", 8)) { - stmt->d.s_propfilt.operation = FIOP_EREREGEX; - } else { - errmsg.LogError(0, NO_ERRCODE, "error: invalid compare operation '%s' - ignoring selector", - (char*) rsCStrGetSzStrNoNULL(pCSCompOp)); - return(RS_RET_ERR); - } - rsCStrDestruct(&pCSCompOp); /* no longer needed */ - - if(stmt->d.s_propfilt.operation != FIOP_ISEMPTY) { - /* read compare value */ - iRet = parsQuotedCStr(pPars, &stmt->d.s_propfilt.pCSCompValue); - if(iRet != RS_RET_OK) { - errmsg.LogError(0, iRet, "error %d compare value property - ignoring selector", iRet); - rsParsDestruct(pPars); - return(iRet); - } - } - - return rsParsDestruct(pPars); -} - - /* process the action part of a selector line * rgerhards, 2007-08-01 */ @@ -623,12 +518,10 @@ rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction) bHadWarning = 1; iRet = RS_RET_OK; } - if(iRet == RS_RET_OK || iRet == RS_RET_SUSPENDED) { - if((iRet = addAction(&pAction, pMod, pModData, pOMSR, NULL, NULL, - (iRet == RS_RET_SUSPENDED)? 1 : 0)) == RS_RET_OK) { + if(iRet == RS_RET_OK) { + if((iRet = addAction(&pAction, pMod, pModData, pOMSR, NULL, NULL)) == RS_RET_OK) { /* here check if the module is compatible with select features * (currently, we have no such features!) */ - pAction->eState = ACT_STATE_RDY; /* action is enabled */ conf->actions.nbrActions++; /* one more active action! */ } break; diff --git a/runtime/conf.h b/runtime/conf.h index a1bb51ad..cb956678 100644 --- a/runtime/conf.h +++ b/runtime/conf.h @@ -63,7 +63,6 @@ rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEn rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); rsRetVal DecodePRIFilter(uchar *pline, uchar pmask[]); -rsRetVal DecodePropFilter(uchar *pline, struct cnfstmt *stmt); rsRetVal cflineDoAction(rsconf_t *conf, uchar **p, action_t **ppAction); extern EHostnameCmpMode eDfltHostnameCmpMode; extern cstr_t *pDfltHostnameCmp; diff --git a/runtime/cryprov.h b/runtime/cryprov.h index 005b33f7..2742a4a5 100644 --- a/runtime/cryprov.h +++ b/runtime/cryprov.h @@ -24,14 +24,25 @@ #ifndef INCLUDED_CRYPROV_H #define INCLUDED_CRYPROV_H +/* we unfortunately need to have two different param names depending on the + * context in which parameters are set. Other than (re/over)engineering the core + * interface, we just define some values to keep track of that. + */ +#define CRYPROV_PARAMTYPE_REGULAR 0 +#define CRYPROV_PARAMTYPE_DISK 1 + /* interface */ BEGINinterface(cryprov) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Construct)(void *ppThis); - rsRetVal (*SetCnfParam)(void *ppThis, struct nvlst *lst); + rsRetVal (*SetCnfParam)(void *ppThis, struct nvlst *lst, int paramType); rsRetVal (*Destruct)(void *ppThis); - rsRetVal (*OnFileOpen)(void *pThis, uchar *fn, void *pFileInstData); + rsRetVal (*OnFileOpen)(void *pThis, uchar *fn, void *pFileInstData, char openMode); rsRetVal (*Encrypt)(void *pFileInstData, uchar *buf, size_t *lenBuf); + rsRetVal (*Decrypt)(void *pFileInstData, uchar *buf, size_t *lenBuf); rsRetVal (*OnFileClose)(void *pFileInstData, off64_t offsLogfile); + rsRetVal (*DeleteStateFiles)(uchar *logfn); + rsRetVal (*GetBytesLeftInBlock)(void *pFileInstData, ssize_t *left); + void (*SetDeleteOnClose)(void *pFileInstData, int val); ENDinterface(cryprov) -#define cryprovCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define cryprovCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ #endif /* #ifndef INCLUDED_CRYPROV_H */ diff --git a/runtime/datetime.c b/runtime/datetime.c index 841ff625..3d50238c 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -626,6 +626,15 @@ finalize_it: RETiRet; } +void +applyDfltTZ(struct syslogTime *pTime, char *tz) +{ + pTime->OffsetMode = tz[0]; + pTime->OffsetHour = (tz[1] - '0') * 10 + (tz[2] - '0'); + pTime->OffsetMinute = (tz[4] - '0') * 10 + (tz[5] - '0'); + +} + /******************************************************************* * END CODE-LIBLOGGING * *******************************************************************/ diff --git a/runtime/datetime.h b/runtime/datetime.h index 9f3611e1..72c3a97f 100644 --- a/runtime/datetime.h +++ b/runtime/datetime.h @@ -62,5 +62,6 @@ ENDinterface(datetime) /* prototypes */ PROTOTYPEObj(datetime); +void applyDfltTZ(struct syslogTime *pTime, char *tz); #endif /* #ifndef INCLUDED_DATETIME_H */ diff --git a/runtime/glbl.c b/runtime/glbl.c index 41d56c2c..e1d4e92d 100644 --- a/runtime/glbl.c +++ b/runtime/glbl.c @@ -61,6 +61,7 @@ DEFobjCurrIf(net) * For this object, these variables are obviously what makes the "meat" of the * class... */ +static struct cnfobj *mainqCnfObj = NULL;/* main queue object, to be used later in startup sequence */ static uchar *pszWorkDir = NULL; static int bOptimizeUniProc = 1; /* enable uniprocessor optimizations */ static int bParseHOSTNAMEandTAG = 1; /* parser modification (based on startup params!) */ @@ -91,6 +92,7 @@ static DEF_ATOMIC_HELPER_MUT(mutTerminateInputs); #ifdef USE_UNLIMITED_SELECT static int iFdSetSize = howmany(FD_SETSIZE, __NFDBITS) * sizeof (fd_mask); /* size of select() bitmask in bytes */ #endif +static uchar *SourceIPofLocalClient = NULL; /* [ar] Source IP for local client to be used on multihomed host */ /* tables for interfacing with the v6 config system */ @@ -138,6 +140,7 @@ static dataType Get##nameFunc(void) \ SIMP_PROP(ParseHOSTNAMEandTAG, bParseHOSTNAMEandTAG, int) SIMP_PROP(OptimizeUniProc, bOptimizeUniProc, int) SIMP_PROP(PreserveFQDN, bPreserveFQDN, int) +SIMP_PROP(mainqCnfObj, mainqCnfObj, struct cnfobj *) SIMP_PROP(MaxLine, iMaxLine, int) SIMP_PROP(DefPFFamily, iDefPFFamily, int) /* note that in the future we may check the family argument */ SIMP_PROP(DropMalPTRMsgs, bDropMalPTRMsgs, int) @@ -506,6 +509,23 @@ GetDfltNetstrmDrvrCertFile(void) } +/* [ar] Source IP for local client to be used on multihomed host */ +static rsRetVal +SetSourceIPofLocalClient(uchar *newname) +{ + if(SourceIPofLocalClient != NULL) { + free(SourceIPofLocalClient); } + SourceIPofLocalClient = newname; + return RS_RET_OK; +} + +static uchar* +GetSourceIPofLocalClient(void) +{ + return(SourceIPofLocalClient); +} + + /* queryInterface function * rgerhards, 2008-02-21 */ @@ -526,6 +546,8 @@ CODESTARTobjQueryInterface(glbl) pIf->GetLocalHostIP = GetLocalHostIP; pIf->SetGlobalInputTermination = SetGlobalInputTermination; pIf->GetGlobalInputTermState = GetGlobalInputTermState; + pIf->GetSourceIPofLocalClient = GetSourceIPofLocalClient; /* [ar] */ + pIf->SetSourceIPofLocalClient = SetSourceIPofLocalClient; /* [ar] */ #define SIMP_PROP(name) \ pIf->Get##name = Get##name; \ pIf->Set##name = Set##name; @@ -537,6 +559,7 @@ CODESTARTobjQueryInterface(glbl) SIMP_PROP(DropMalPTRMsgs); SIMP_PROP(Option_DisallowWarning); SIMP_PROP(DisableDNS); + SIMP_PROP(mainqCnfObj); SIMP_PROP(LocalFQDNName) SIMP_PROP(LocalHostName) SIMP_PROP(LocalDomain) @@ -587,6 +610,8 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a void glblPrepCnf(void) { + free(mainqCnfObj); + mainqCnfObj = NULL; free(cnfparamvals); cnfparamvals = NULL; } @@ -604,6 +629,34 @@ glblProcessCnf(struct cnfobj *o) cnfparamsPrint(¶mblk, cnfparamvals); } +/* Set mainq parameters. Note that when this is not called, we'll use the + * legacy parameter config. mainq parameters can only be set once. + */ +void +glblProcessMainQCnf(struct cnfobj *o) +{ + if(mainqCnfObj == NULL) { + mainqCnfObj = o; + } else { + errmsg.LogError(0, RS_RET_ERR, "main_queue() object can only be specified " + "once - all but first ignored\n"); + } +} + +/* destruct the main q cnf object after it is no longer needed. This is + * also used to do some final checks. + */ +void +glblDestructMainqCnfObj() +{ + /* Only destruct if not NULL! */ + if (mainqCnfObj != NULL) { + nvlstChkUnused(mainqCnfObj->nvlst); + } + cnfobjDestruct(mainqCnfObj); + mainqCnfObj = NULL; +} + void glblDoneLoadCnf(void) { diff --git a/runtime/glbl.h b/runtime/glbl.h index e95e48f7..4b1cd9f8 100644 --- a/runtime/glbl.h +++ b/runtime/glbl.h @@ -52,6 +52,7 @@ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ SIMP_PROP(Option_DisallowWarning, int) SIMP_PROP(DisableDNS, int) SIMP_PROP(LocalFQDNName, uchar*) + SIMP_PROP(mainqCnfObj, struct cnfobj*) SIMP_PROP(LocalHostName, uchar*) SIMP_PROP(LocalDomain, uchar*) SIMP_PROP(StripDomains, char**) @@ -81,6 +82,8 @@ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ /* next change is v9! */ /* v8 - 2012-03-21 */ prop_t* (*GetLocalHostIP)(void); + uchar* (*GetSourceIPofLocalClient)(void); /* [ar] */ + rsRetVal (*SetSourceIPofLocalClient)(uchar*); /* [ar] */ #undef SIMP_PROP ENDinterface(glbl) #define glblCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ @@ -94,6 +97,8 @@ static inline void glblSetOurPid(pid_t pid) { glbl_ourpid = pid; } void glblPrepCnf(void); void glblProcessCnf(struct cnfobj *o); +void glblProcessMainQCnf(struct cnfobj *o); +void glblDestructMainqCnfObj(); void glblDoneLoadCnf(void); #endif /* #ifndef GLBL_H_INCLUDED */ diff --git a/runtime/libgcry.c b/runtime/libgcry.c index 51c10af4..4772cf47 100644 --- a/runtime/libgcry.c +++ b/runtime/libgcry.c @@ -49,8 +49,12 @@ #include <errno.h> #include "rsyslog.h" +#include "srUtils.h" #include "libgcry.h" +#define READBUF_SIZE 4096 /* size of the read buffer */ + +static rsRetVal rsgcryBlkBegin(gcryfile gf); static rsRetVal eiWriteRec(gcryfile gf, char *recHdr, size_t lenRecHdr, char *buf, size_t lenBuf) @@ -90,19 +94,66 @@ finalize_it: RETiRet; } +static rsRetVal +eiRead(gcryfile gf) +{ + ssize_t nRead; + DEFiRet; + + if(gf->readBuf == NULL) { + CHKmalloc(gf->readBuf = malloc(READBUF_SIZE)); + } + + nRead = read(gf->fd, gf->readBuf, READBUF_SIZE); + if(nRead <= 0) { /* TODO: provide specific EOF case? */ + ABORT_FINALIZE(RS_RET_ERR); + } + gf->readBufMaxIdx = (int16_t) nRead; + gf->readBufIdx = 0; + +finalize_it: + RETiRet; +} + + +/* returns EOF on any kind of error */ +static int +eiReadChar(gcryfile gf) +{ + int c; + + if(gf->readBufIdx >= gf->readBufMaxIdx) { + if(eiRead(gf) != RS_RET_OK) { + c = EOF; + goto finalize_it; + } + } + c = gf->readBuf[gf->readBufIdx++]; +finalize_it: + return c; +} + static rsRetVal eiCheckFiletype(gcryfile gf) { char hdrBuf[128]; size_t toRead, didRead; + sbool bNeedClose = 0; DEFiRet; - CHKiRet(eiOpenRead(gf)); + if(gf->fd == -1) { + bNeedClose = 1; + CHKiRet(eiOpenRead(gf)); + } + if(Debug) memset(hdrBuf, 0, sizeof(hdrBuf)); /* for dbgprintf below! */ toRead = sizeof("FILETYPE:")-1 + sizeof(RSGCRY_FILETYPE_NAME)-1 + 1; didRead = read(gf->fd, hdrBuf, toRead); - close(gf->fd); + if(bNeedClose) { + close(gf->fd); + gf->fd = -1; + } DBGPRINTF("eiCheckFiletype read %d bytes: '%s'\n", didRead, hdrBuf); if( didRead != toRead || strncmp(hdrBuf, "FILETYPE:" RSGCRY_FILETYPE_NAME "\n", toRead)) @@ -111,6 +162,98 @@ finalize_it: RETiRet; } +/* rectype/value must be EIF_MAX_*_LEN+1 long! + * returns 0 on success or something else on error/EOF + */ +static rsRetVal +eiGetRecord(gcryfile gf, char *rectype, char *value) +{ + unsigned short i, j; + int c; + DEFiRet; + + c = eiReadChar(gf); + if(c == EOF) { ABORT_FINALIZE(RS_RET_NO_DATA); } + for(i = 0 ; i < EIF_MAX_RECTYPE_LEN ; ++i) { + if(c == ':' || c == EOF) + break; + rectype[i] = c; + c = eiReadChar(gf); + } + if(c != ':') { ABORT_FINALIZE(RS_RET_ERR); } + rectype[i] = '\0'; + j = 0; + for(++i ; i < EIF_MAX_VALUE_LEN ; ++i, ++j) { + c = eiReadChar(gf); + if(c == '\n' || c == EOF) + break; + value[j] = c; + } + if(c != '\n') { ABORT_FINALIZE(RS_RET_ERR); } + value[j] = '\0'; +finalize_it: + RETiRet; +} + +static rsRetVal +eiGetIV(gcryfile gf, uchar *iv, size_t leniv) +{ + char rectype[EIF_MAX_RECTYPE_LEN+1]; + char value[EIF_MAX_VALUE_LEN+1]; + size_t valueLen; + unsigned short i, j; + unsigned char nibble; + DEFiRet; + + CHKiRet(eiGetRecord(gf, rectype, value)); + if(strcmp(rectype, "IV")) { + DBGPRINTF("no IV record found when expected, record type " + "seen is '%s'\n", rectype); + ABORT_FINALIZE(RS_RET_ERR); + } + valueLen = strlen(value); + if(valueLen/2 != leniv) { + DBGPRINTF("length of IV is %d, expected %d\n", + valueLen/2, leniv); + ABORT_FINALIZE(RS_RET_ERR); + } + + for(i = j = 0 ; i < valueLen ; ++i) { + if(value[i] >= '0' && value[i] <= '9') + nibble = value[i] - '0'; + else if(value[i] >= 'a' && value[i] <= 'f') + nibble = value[i] - 'a' + 10; + else { + DBGPRINTF("invalid IV '%s'\n", value); + ABORT_FINALIZE(RS_RET_ERR); + } + if(i % 2 == 0) + iv[j] = nibble << 4; + else + iv[j++] |= nibble; + } +finalize_it: + RETiRet; +} + +static rsRetVal +eiGetEND(gcryfile gf, off64_t *offs) +{ + char rectype[EIF_MAX_RECTYPE_LEN+1]; + char value[EIF_MAX_VALUE_LEN+1]; + DEFiRet; + + CHKiRet(eiGetRecord(gf, rectype, value)); + if(strcmp(rectype, "END")) { + DBGPRINTF("no END record found when expected, record type " + "seen is '%s'\n", rectype); + ABORT_FINALIZE(RS_RET_ERR); + } + *offs = atoll(value); +finalize_it: + RETiRet; +} + static rsRetVal eiOpenAppend(gcryfile gf) { @@ -177,13 +320,55 @@ eiClose(gcryfile gf, off64_t offsLogfile) size_t len; if(gf->fd == -1) return; - /* 2^64 is 20 digits, so the snprintf buffer is large enough */ - len = snprintf(offs, sizeof(offs), "%lld", offsLogfile); - eiWriteRec(gf, "END:", 4, offs, len); + if(gf->openMode == 'w') { + /* 2^64 is 20 digits, so the snprintf buffer is large enough */ + len = snprintf(offs, sizeof(offs), "%lld", offsLogfile); + eiWriteRec(gf, "END:", 4, offs, len); + } + gcry_cipher_close(gf->chd); + free(gf->readBuf); close(gf->fd); + gf->fd = -1; DBGPRINTF("encryption info file %s: closed\n", gf->eiName); } +/* this returns the number of bytes left inside the block or -1, if the block + * size is unbounded. The function automatically handles end-of-block and begins + * to read the next block in this case. + */ +rsRetVal +gcryfileGetBytesLeftInBlock(gcryfile gf, ssize_t *left) +{ + DEFiRet; + if(gf->bytesToBlkEnd == 0) { + DBGPRINTF("libgcry: end of current crypto block\n"); + gcry_cipher_close(gf->chd); + CHKiRet(rsgcryBlkBegin(gf)); + } + *left = gf->bytesToBlkEnd; +finalize_it: + // TODO: remove once this code is sufficiently well-proven + DBGPRINTF("gcryfileGetBytesLeftInBlock returns %lld, iRet %d\n", (long long) *left, iRet); + RETiRet; +} + +/* this is a special functon for use by the rsyslog disk queue subsystem. It + * needs to have the capability to delete state when a queue file is rolled + * over. This simply generates the file name and deletes it. It must take care + * of "all" state files, which currently happens to be a single one. + */ +rsRetVal +gcryfileDeleteState(uchar *logfn) +{ + char fn[MAXFNAME+1]; + DEFiRet; + snprintf(fn, sizeof(fn), "%s%s", logfn, ENCINFO_SUFFIX); + fn[MAXFNAME] = '\0'; /* be on save side */ + DBGPRINTF("crypto provider deletes state file '%s' on request\n", fn); + unlink(fn); + RETiRet; +} + static rsRetVal gcryfileConstruct(gcryctx ctx, gcryfile *pgf, uchar *logfn) { @@ -193,6 +378,7 @@ gcryfileConstruct(gcryctx ctx, gcryfile *pgf, uchar *logfn) CHKmalloc(gf = calloc(1, sizeof(struct gcryfile_s))); gf->ctx = ctx; + gf->fd = -1; snprintf(fn, sizeof(fn), "%s%s", logfn, ENCINFO_SUFFIX); fn[MAXFNAME] = '\0'; /* be on save side */ gf->eiName = (uchar*) strdup(fn); @@ -219,7 +405,12 @@ gcryfileDestruct(gcryfile gf, off64_t offsLogfile) if(gf == NULL) goto done; + DBGPRINTF("libgcry: close file %s\n", gf->eiName); eiClose(gf, offsLogfile); + if(gf->bDeleteOnClose) { + DBGPRINTF("unlink file '%s' due to bDeleteOnClose set\n", gf->eiName); + unlink((char*)gf->eiName); + } free(gf->eiName); free(gf); done: return r; @@ -246,13 +437,13 @@ addPadding(gcryfile pF, uchar *buf, size_t *plen) } static inline void -removePadding(char *buf, size_t *plen) +removePadding(uchar *buf, size_t *plen) { unsigned len = (unsigned) *plen; unsigned iSrc, iDst; - char *frstNUL; + uchar *frstNUL; - frstNUL = strchr(buf, 0x00); + frstNUL = (uchar*)strchr((char*)buf, 0x00); if(frstNUL == NULL) goto done; iDst = iSrc = frstNUL - buf; @@ -343,53 +534,123 @@ seedIV(gcryfile gf, uchar **iv) } } -rsRetVal -rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname) +static inline rsRetVal +readIV(gcryfile gf, uchar **iv) { - gcry_error_t gcryError; - gcryfile gf = NULL; - uchar *iv = NULL; + rsRetVal localRet; DEFiRet; - CHKiRet(gcryfileConstruct(ctx, &gf, fname)); + if(gf->fd == -1) { + while(gf->fd == -1) { + localRet = eiOpenRead(gf); + if(localRet == RS_RET_EI_NO_EXISTS) { + /* wait until it is created */ + srSleep(0, 10000); + } else { + CHKiRet(localRet); + } + } + CHKiRet(eiCheckFiletype(gf)); + } + *iv = malloc(gf->blkLength); /* do NOT zero-out! */ + CHKiRet(eiGetIV(gf, *iv, (size_t) gf->blkLength)); +finalize_it: + RETiRet; +} + +/* this tries to read the END record. HOWEVER, no such record may be + * present, which is the case if we handle a currently-written to queue + * file. On the other hand, the queue file may contain multiple blocks. So + * what we do is try to see if there is a block end or not - and set the + * status accordingly. Note that once we found no end-of-block, we will never + * retry. This is because that case can never happen under current queue + * implementations. -- gerhards, 2013-05-16 + */ +static inline rsRetVal +readBlkEnd(gcryfile gf) +{ + off64_t blkEnd; + DEFiRet; + + iRet = eiGetEND(gf, &blkEnd); + if(iRet == RS_RET_OK) { + gf->bytesToBlkEnd = (ssize_t) blkEnd; + } else if(iRet == RS_RET_NO_DATA) { + gf->bytesToBlkEnd = -1; + } else { + FINALIZE; + } + +finalize_it: + RETiRet; +} - gf->blkLength = gcry_cipher_get_algo_blklen(ctx->algo); - gcryError = gcry_cipher_open(&gf->chd, ctx->algo, ctx->mode, 0); +/* Read the block begin metadata and set our state variables accordingly. Can also + * be used to init the first block in write case. + */ +static rsRetVal +rsgcryBlkBegin(gcryfile gf) +{ + gcry_error_t gcryError; + uchar *iv = NULL; + DEFiRet; + + gcryError = gcry_cipher_open(&gf->chd, gf->ctx->algo, gf->ctx->mode, 0); if (gcryError) { - dbgprintf("gcry_cipher_open failed: %s/%s\n", - gcry_strsource(gcryError), - gcry_strerror(gcryError)); + DBGPRINTF("gcry_cipher_open failed: %s/%s\n", + gcry_strsource(gcryError), gcry_strerror(gcryError)); ABORT_FINALIZE(RS_RET_ERR); } gcryError = gcry_cipher_setkey(gf->chd, gf->ctx->key, gf->ctx->keyLen); if (gcryError) { - dbgprintf("gcry_cipher_setkey failed: %s/%s\n", - gcry_strsource(gcryError), - gcry_strerror(gcryError)); + DBGPRINTF("gcry_cipher_setkey failed: %s/%s\n", + gcry_strsource(gcryError), gcry_strerror(gcryError)); ABORT_FINALIZE(RS_RET_ERR); } - seedIV(gf, &iv); + if(gf->openMode == 'r') { + readIV(gf, &iv); + readBlkEnd(gf); + } else { + seedIV(gf, &iv); + } + gcryError = gcry_cipher_setiv(gf->chd, iv, gf->blkLength); if (gcryError) { - dbgprintf("gcry_cipher_setiv failed: %s/%s\n", - gcry_strsource(gcryError), - gcry_strerror(gcryError)); + DBGPRINTF("gcry_cipher_setiv failed: %s/%s\n", + gcry_strsource(gcryError), gcry_strerror(gcryError)); ABORT_FINALIZE(RS_RET_ERR); } - CHKiRet(eiOpenAppend(gf)); - CHKiRet(eiWriteIV(gf, iv)); - *pgf = gf; + + if(gf->openMode == 'w') { + CHKiRet(eiOpenAppend(gf)); + CHKiRet(eiWriteIV(gf, iv)); + } finalize_it: free(iv); + RETiRet; +} + +rsRetVal +rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname, char openMode) +{ + gcryfile gf = NULL; + DEFiRet; + + CHKiRet(gcryfileConstruct(ctx, &gf, fname)); + gf->openMode = openMode; + gf->blkLength = gcry_cipher_get_algo_blklen(ctx->algo); + CHKiRet(rsgcryBlkBegin(gf)); + *pgf = gf; +finalize_it: if(iRet != RS_RET_OK && gf != NULL) gcryfileDestruct(gf, -1); RETiRet; } -int +rsRetVal rsgcryEncrypt(gcryfile pF, uchar *buf, size_t *len) { int gcryError; @@ -410,6 +671,35 @@ finalize_it: RETiRet; } +/* TODO: handle multiple blocks + * test-read END record; if present, store offset, else unbounded (current active block) + * when decrypting, check if bound is reached. If yes, split into two blocks, get new IV for + * second one. + */ +rsRetVal +rsgcryDecrypt(gcryfile pF, uchar *buf, size_t *len) +{ + gcry_error_t gcryError; + DEFiRet; + + if(pF->bytesToBlkEnd != -1) + pF->bytesToBlkEnd -= *len; + gcryError = gcry_cipher_decrypt(pF->chd, buf, *len, NULL, 0); + if(gcryError) { + DBGPRINTF("gcry_cipher_decrypt failed: %s/%s\n", + gcry_strsource(gcryError), + gcry_strerror(gcryError)); + ABORT_FINALIZE(RS_RET_ERR); + } + removePadding(buf, len); + // TODO: remove dbgprintf once things are sufficently stable -- rgerhards, 2013-05-16 + dbgprintf("libgcry: decrypted, bytesToBlkEnd %lld, buffer is now '%50.50s'\n", (long long) pF->bytesToBlkEnd, buf); + +finalize_it: + RETiRet; +} + + /* module-init dummy for potential later use */ int diff --git a/runtime/libgcry.h b/runtime/libgcry.h index 692ce408..ae5a6735 100644 --- a/runtime/libgcry.h +++ b/runtime/libgcry.h @@ -38,7 +38,15 @@ struct gcryfile_s { size_t blkLength; /* size of low-level crypto block */ uchar *eiName; /* name of .encinfo file */ int fd; /* descriptor of .encinfo file (-1 if not open) */ + char openMode; /* 'r': read, 'w': write */ gcryctx ctx; + uchar *readBuf; + int16_t readBufIdx; + int16_t readBufMaxIdx; + int8_t bDeleteOnClose; /* for queue support, similar to stream subsys */ + ssize_t bytesToBlkEnd; /* number of bytes remaining in current crypto block + -1 means -> no end (still being writen to, queue files), + 0 means -> end of block, new one must be started. */ }; int gcryGetKeyFromFile(char *fn, char **key, unsigned *keylen); @@ -50,9 +58,12 @@ rsRetVal rsgcrySetAlgo(gcryctx ctx, uchar *modename); gcryctx gcryCtxNew(void); void rsgcryCtxDel(gcryctx ctx); int gcryfileDestruct(gcryfile gf, off64_t offsLogfile); -rsRetVal rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname); -int rsgcryEncrypt(gcryfile pF, uchar *buf, size_t *len); +rsRetVal rsgcryInitCrypt(gcryctx ctx, gcryfile *pgf, uchar *fname, char openMode); +rsRetVal rsgcryEncrypt(gcryfile pF, uchar *buf, size_t *len); +rsRetVal rsgcryDecrypt(gcryfile pF, uchar *buf, size_t *len); int gcryGetKeyFromProg(char *cmd, char **key, unsigned *keylen); +rsRetVal gcryfileDeleteState(uchar *fn); +rsRetVal gcryfileGetBytesLeftInBlock(gcryfile gf, ssize_t *left); /* error states */ #define RSGCRYE_EI_OPEN 1 /* error opening .encinfo file */ @@ -63,6 +74,14 @@ int gcryGetKeyFromProg(char *cmd, char **key, unsigned *keylen); #define RSGCRY_FILETYPE_NAME "rsyslog-enrcyption-info" #define ENCINFO_SUFFIX ".encinfo" +/* Note: gf may validly be NULL, e.g. if file has not yet been opened! */ +static inline void +gcryfileSetDeleteOnClose(gcryfile gf, int val) +{ + if(gf != NULL) + gf->bDeleteOnClose = val; +} + static inline int rsgcryAlgoname2Algo(char *algoname) { if(!strcmp((char*)algoname, "3DES")) return GCRY_CIPHER_3DES; diff --git a/runtime/lmcry_gcry.c b/runtime/lmcry_gcry.c index 0a9b94bc..9a0c0072 100644 --- a/runtime/lmcry_gcry.c +++ b/runtime/lmcry_gcry.c @@ -43,17 +43,30 @@ DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) /* tables for interfacing with the v6 config system */ -static struct cnfparamdescr cnfpdescr[] = { +static struct cnfparamdescr cnfpdescrRegular[] = { { "cry.key", eCmdHdlrGetWord, 0 }, { "cry.keyfile", eCmdHdlrGetWord, 0 }, { "cry.keyprogram", eCmdHdlrGetWord, 0 }, { "cry.mode", eCmdHdlrGetWord, 0 }, /* CBC, ECB, etc */ { "cry.algo", eCmdHdlrGetWord, 0 } }; -static struct cnfparamblk pblk = +static struct cnfparamblk pblkRegular = { CNFPARAMBLK_VERSION, - sizeof(cnfpdescr)/sizeof(struct cnfparamdescr), - cnfpdescr + sizeof(cnfpdescrRegular)/sizeof(struct cnfparamdescr), + cnfpdescrRegular + }; + +static struct cnfparamdescr cnfpdescrQueue[] = { + { "queue.cry.key", eCmdHdlrGetWord, 0 }, + { "queue.cry.keyfile", eCmdHdlrGetWord, 0 }, + { "queue.cry.keyprogram", eCmdHdlrGetWord, 0 }, + { "queue.cry.mode", eCmdHdlrGetWord, 0 }, /* CBC, ECB, etc */ + { "queue.cry.algo", eCmdHdlrGetWord, 0 } +}; +static struct cnfparamblk pblkQueue = + { CNFPARAMBLK_VERSION, + sizeof(cnfpdescrQueue)/sizeof(struct cnfparamdescr), + cnfpdescrQueue }; @@ -85,7 +98,7 @@ ENDobjDestruct(lmcry_gcry) * Defaults are expected to have been set during construction. */ static rsRetVal -SetCnfParam(void *pT, struct nvlst *lst) +SetCnfParam(void *pT, struct nvlst *lst, int paramType) { lmcry_gcry_t *pThis = (lmcry_gcry_t*) pT; int i, r; @@ -97,34 +110,41 @@ SetCnfParam(void *pT, struct nvlst *lst) uchar *mode = NULL; int nKeys; /* number of keys (actually methods) specified */ struct cnfparamvals *pvals; + struct cnfparamblk *pblk; DEFiRet; + pblk = (paramType == CRYPROV_PARAMTYPE_REGULAR ) ? &pblkRegular : &pblkQueue; nKeys = 0; - pvals = nvlstGetParams(lst, &pblk, NULL); + pvals = nvlstGetParams(lst, pblk, NULL); if(Debug) { dbgprintf("param blk in lmcry_gcry:\n"); - cnfparamsPrint(&pblk, pvals); + cnfparamsPrint(pblk, pvals); } - for(i = 0 ; i < pblk.nParams ; ++i) { + for(i = 0 ; i < pblk->nParams ; ++i) { if(!pvals[i].bUsed) continue; - if(!strcmp(pblk.descr[i].name, "cry.key")) { + if(!strcmp(pblk->descr[i].name, "cry.key") || + !strcmp(pblk->descr[i].name, "queue.cry.key")) { key = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); ++nKeys; - } else if(!strcmp(pblk.descr[i].name, "cry.keyfile")) { + } else if(!strcmp(pblk->descr[i].name, "cry.keyfile") || + !strcmp(pblk->descr[i].name, "queue.cry.keyfile")) { keyfile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++nKeys; - } else if(!strcmp(pblk.descr[i].name, "cry.keyprogram")) { + } else if(!strcmp(pblk->descr[i].name, "cry.keyprogram") || + !strcmp(pblk->descr[i].name, "queue.cry.keyprogram")) { keyprogram = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++nKeys; - } else if(!strcmp(pblk.descr[i].name, "cry.mode")) { + } else if(!strcmp(pblk->descr[i].name, "cry.mode") || + !strcmp(pblk->descr[i].name, "queue.cry.mode")) { mode = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); - } else if(!strcmp(pblk.descr[i].name, "cry.algo")) { + } else if(!strcmp(pblk->descr[i].name, "cry.algo") || + !strcmp(pblk->descr[i].name, "queue.cry.algo")) { algo = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else { DBGPRINTF("lmcry_gcry: program error, non-handled " - "param '%s'\n", pblk.descr[i].name); + "param '%s'\n", pblk->descr[i].name); } } if(algo != NULL) { @@ -177,7 +197,7 @@ SetCnfParam(void *pT, struct nvlst *lst) ABORT_FINALIZE(RS_RET_INVALID_PARAMS); } - cnfparamvalsDestruct(pvals, &pblk); + cnfparamvalsDestruct(pvals, pblk); if(key != NULL) { memset(key, 0, strlen((char*)key)); free(key); @@ -189,15 +209,33 @@ finalize_it: RETiRet; } +static void +SetDeleteOnClose(void *pF, int val) +{ + gcryfileSetDeleteOnClose(pF, val); +} + +static rsRetVal +GetBytesLeftInBlock(void *pF, ssize_t *left) +{ + return gcryfileGetBytesLeftInBlock((gcryfile) pF, left); +} static rsRetVal -OnFileOpen(void *pT, uchar *fn, void *pGF) +DeleteStateFiles(uchar *logfn) +{ + return gcryfileDeleteState(logfn); +} + +static rsRetVal +OnFileOpen(void *pT, uchar *fn, void *pGF, char openMode) { lmcry_gcry_t *pThis = (lmcry_gcry_t*) pT; gcryfile *pgf = (gcryfile*) pGF; DEFiRet; + DBGPRINTF("lmcry_gcry: open file '%s', mode '%c'\n", fn, openMode); - CHKiRet(rsgcryInitCrypt(pThis->ctx, pgf, fn)); + CHKiRet(rsgcryInitCrypt(pThis->ctx, pgf, fn, openMode)); finalize_it: /* TODO: enable this error message (need to cleanup loop first ;)) errmsg.LogError(0, iRet, "Encryption Provider" @@ -207,6 +245,16 @@ finalize_it: } static rsRetVal +Decrypt(void *pF, uchar *rec, size_t *lenRec) +{ + DEFiRet; + iRet = rsgcryDecrypt(pF, rec, lenRec); + + RETiRet; +} + + +static rsRetVal Encrypt(void *pF, uchar *rec, size_t *lenRec) { DEFiRet; @@ -231,10 +279,14 @@ CODESTARTobjQueryInterface(lmcry_gcry) } pIf->Construct = (rsRetVal(*)(void*)) lmcry_gcryConstruct; pIf->SetCnfParam = SetCnfParam; + pIf->SetDeleteOnClose = SetDeleteOnClose; pIf->Destruct = (rsRetVal(*)(void*)) lmcry_gcryDestruct; pIf->OnFileOpen = OnFileOpen; pIf->Encrypt = Encrypt; + pIf->Decrypt = Decrypt; pIf->OnFileClose = OnFileClose; + pIf->DeleteStateFiles = DeleteStateFiles; + pIf->GetBytesLeftInBlock = GetBytesLeftInBlock; finalize_it: ENDobjQueryInterface(lmcry_gcry) diff --git a/runtime/lookup.c b/runtime/lookup.c new file mode 100644 index 00000000..bc3b1a94 --- /dev/null +++ b/runtime/lookup.c @@ -0,0 +1,375 @@ +/* lookup.c + * Support for lookup tables in RainerScript. + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <json/json.h> +#include <json/json.h> +#include <assert.h> + +#include "rsyslog.h" +#include "srUtils.h" +#include "errmsg.h" +#include "lookup.h" +#include "msg.h" +#include "rsconf.h" +#include "dirty.h" +#include "unicode-helper.h" + +/* definitions for objects we access */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) + +/* forward definitions */ +static rsRetVal lookupReadFile(lookup_t *pThis); + +/* static data */ +/* tables for interfacing with the v6 config system (as far as we need to) */ +static struct cnfparamdescr modpdescr[] = { + { "name", eCmdHdlrString, CNFPARAM_REQUIRED }, + { "file", eCmdHdlrString, CNFPARAM_REQUIRED } +}; +static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, + sizeof(modpdescr)/sizeof(struct cnfparamdescr), + modpdescr + }; + + +/* create a new lookup table object AND include it in our list of + * lookup tables. + */ +rsRetVal +lookupNew(lookup_t **ppThis) +{ + lookup_t *pThis = NULL; + DEFiRet; + + CHKmalloc(pThis = malloc(sizeof(lookup_t))); + pthread_rwlock_init(&pThis->rwlock, NULL); + pThis->name = NULL; + + if(loadConf->lu_tabs.root == NULL) { + loadConf->lu_tabs.root = pThis; + pThis->next = NULL; + } else { + pThis->next = loadConf->lu_tabs.last; + } + loadConf->lu_tabs.last = pThis; + + *ppThis = pThis; +finalize_it: + if(iRet != RS_RET_OK) { + free(pThis); + } + RETiRet; +} +void +lookupDestruct(lookup_t *pThis) +{ + pthread_rwlock_destroy(&pThis->rwlock); + free(pThis->name); + free(pThis); +} + +void +lookupInitCnf(lookup_tables_t *lu_tabs) +{ + lu_tabs->root = NULL; + lu_tabs->last = NULL; +} + + +/* comparison function for qsort() and string array compare + * this is for the string lookup table type + */ +static int +qs_arrcmp_strtab(const void *s1, const void *s2) +{ + return ustrcmp(((lookup_string_tab_etry_t*)s1)->key, ((lookup_string_tab_etry_t*)s2)->key); +} +/* comparison function for bsearch() and string array compare + * this is for the string lookup table type + */ +static int +bs_arrcmp_strtab(const void *s1, const void *s2) +{ + return strcmp((char*)s1, (char*)((lookup_string_tab_etry_t*)s2)->key); +} + +rsRetVal +lookupBuildTable(lookup_t *pThis, struct json_object *jroot) +{ + struct json_object *jversion, *jnomatch, *jtype, *jtab; + struct json_object *jrow, *jindex, *jvalue; + uint32_t i; + uint32_t maxStrSize; + DEFiRet; + + jversion = json_object_object_get(jroot, "version"); + jnomatch = json_object_object_get(jroot, "nomatch"); + jtype = json_object_object_get(jroot, "type"); + jtab = json_object_object_get(jroot, "table"); + pThis->nmemb = json_object_array_length(jtab); + CHKmalloc(pThis->d.strtab = malloc(pThis->nmemb * sizeof(lookup_string_tab_etry_t))); + + maxStrSize = 0; + for(i = 0 ; i < pThis->nmemb ; ++i) { + jrow = json_object_array_get_idx(jtab, i); + jindex = json_object_object_get(jrow, "index"); + jvalue = json_object_object_get(jrow, "value"); + CHKmalloc(pThis->d.strtab[i].key = (uchar*) strdup(json_object_get_string(jindex))); + CHKmalloc(pThis->d.strtab[i].val = (uchar*) strdup(json_object_get_string(jvalue))); + maxStrSize += ustrlen(pThis->d.strtab[i].val); + } + + qsort(pThis->d.strtab, pThis->nmemb, sizeof(lookup_string_tab_etry_t), qs_arrcmp_strtab); +dbgprintf("DDDD: table loaded (max size %u):\n", maxStrSize); +for(i = 0 ; i < pThis->nmemb ; ++i) + dbgprintf("key: '%s', val: '%s'\n", pThis->d.strtab[i].key, pThis->d.strtab[i].val); + +finalize_it: + RETiRet; +} + + +/* find a lookup table. This is a naive O(n) algo, but this really + * doesn't matter as it is called only a few times during config + * load. The function returns either a pointer to the requested + * table or NULL, if not found. + */ +lookup_t * +lookupFindTable(uchar *name) +{ + lookup_t *curr; + + for(curr = loadConf->lu_tabs.root ; curr != NULL ; curr = curr->next) { + if(!ustrcmp(curr->name, name)) + break; + } + return curr; +} + + +/* this reloads a lookup table. This is done while the engine is running, + * as such the function must ensure proper locking and proper order of + * operations (so that nothing can interfere). If the table cannot be loaded, + * the old table is continued to be used. + */ +static rsRetVal +lookupReload(lookup_t *pThis) +{ + uint32_t i; + lookup_t newlu; /* dummy to be able to use support functions without + affecting current settings. */ + DEFiRet; + + DBGPRINTF("reload requested for lookup table '%s'\n", pThis->name); + memset(&newlu, 0, sizeof(newlu)); + CHKmalloc(newlu.name = ustrdup(pThis->name)); + CHKmalloc(newlu.filename = ustrdup(pThis->filename)); + CHKiRet(lookupReadFile(&newlu)); + /* all went well, copy over data members */ + pthread_rwlock_wrlock(&pThis->rwlock); + for(i = 0 ; i < pThis->nmemb ; ++i) { + free(pThis->d.strtab[i].key), /* we don't care about exec order of frees */ + free(pThis->d.strtab[i].val); + } + free(pThis->d.strtab); + pThis->d.strtab = newlu.d.strtab; /* hand table AND ALL STRINGS over! */ + pthread_rwlock_unlock(&pThis->rwlock); + errmsg.LogError(0, RS_RET_OK, "lookup table '%s' reloaded from file '%s'", + pThis->name, pThis->filename); +finalize_it: + free(newlu.name); + free(newlu.filename); + RETiRet; +} + + +/* reload all lookup tables on HUP */ +void +lookupDoHUP() +{ + lookup_t *lu; + for(lu = loadConf->lu_tabs.root ; lu != NULL ; lu = lu->next) { + lookupReload(lu); + } +} + + +/* returns either a pointer to the value (read only!) or NULL + * if either the key could not be found or an error occured. + * Note that an estr_t object is returned. The caller is + * responsible for freeing it. + */ +es_str_t * +lookupKey_estr(lookup_t *pThis, uchar *key) +{ + lookup_string_tab_etry_t *etry; + char *r; + es_str_t *estr; + + pthread_rwlock_rdlock(&pThis->rwlock); + etry = bsearch(key, pThis->d.strtab, pThis->nmemb, sizeof(lookup_string_tab_etry_t), bs_arrcmp_strtab); + if(etry == NULL) { + r = ""; // TODO: use set default + } else { + r = (char*)etry->val; + } + estr = es_newStrFromCStr(r, strlen(r)); + pthread_rwlock_unlock(&pThis->rwlock); + return estr; +} + + +/* note: widely-deployed json_c 0.9 does NOT support incremental + * parsing. In order to keep compatible with e.g. Ubuntu 12.04LTS, + * we read the file into one big memory buffer and parse it at once. + * While this is not very elegant, it will not pose any real issue + * for "reasonable" lookup tables (and "unreasonably" large ones + * will probably have other issues as well...). + */ +static rsRetVal +lookupReadFile(lookup_t *pThis) +{ + struct json_tokener *tokener = NULL; + struct json_object *json = NULL; + int eno = errno; + char errStr[1024]; + char *iobuf = NULL; + int fd; + ssize_t nread; + struct stat sb; + DEFiRet; + + + if(stat((char*)pThis->filename, &sb) == -1) { + eno = errno; + errmsg.LogError(0, RS_RET_FILE_NOT_FOUND, + "lookup table file '%s' stat failed: %s", + pThis->filename, rs_strerror_r(eno, errStr, sizeof(errStr))); + ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); + } + + CHKmalloc(iobuf = malloc(sb.st_size)); + + if((fd = open((const char*) pThis->filename, O_RDONLY)) == -1) { + eno = errno; + errmsg.LogError(0, RS_RET_FILE_NOT_FOUND, + "lookup table file '%s' could not be opened: %s", + pThis->filename, rs_strerror_r(eno, errStr, sizeof(errStr))); + ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); + } + + tokener = json_tokener_new(); + nread = read(fd, iobuf, sb.st_size); + if(nread != (ssize_t) sb.st_size) { + eno = errno; + errmsg.LogError(0, RS_RET_READ_ERR, + "lookup table file '%s' read error: %s", + pThis->filename, rs_strerror_r(eno, errStr, sizeof(errStr))); + ABORT_FINALIZE(RS_RET_READ_ERR); + } + + json = json_tokener_parse_ex(tokener, iobuf, sb.st_size); + if(json == NULL) { + errmsg.LogError(0, RS_RET_JSON_PARSE_ERR, + "lookup table file '%s' json parsing error", + pThis->filename); + ABORT_FINALIZE(RS_RET_JSON_PARSE_ERR); + } + free(iobuf); /* early free to sever resources*/ + iobuf = NULL; /* make sure no double-free */ + + /* got json object, now populate our own in-memory structure */ + CHKiRet(lookupBuildTable(pThis, json)); + +finalize_it: + free(iobuf); + if(tokener != NULL) + json_tokener_free(tokener); + if(json != NULL) + json_object_put(json); + RETiRet; +} + + +rsRetVal +lookupProcessCnf(struct cnfobj *o) +{ + struct cnfparamvals *pvals; + lookup_t *lu; + short i; + DEFiRet; + + pvals = nvlstGetParams(o->nvlst, &modpblk, NULL); + if(pvals == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); + } + DBGPRINTF("lookupProcessCnf params:\n"); + cnfparamsPrint(&modpblk, pvals); + + CHKiRet(lookupNew(&lu)); + + for(i = 0 ; i < modpblk.nParams ; ++i) { + if(!pvals[i].bUsed) + continue; + if(!strcmp(modpblk.descr[i].name, "file")) { + CHKmalloc(lu->filename = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL)); + } else if(!strcmp(modpblk.descr[i].name, "name")) { + CHKmalloc(lu->name = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL)); + } else { + dbgprintf("lookup_table: program error, non-handled " + "param '%s'\n", modpblk.descr[i].name); + } + } + CHKiRet(lookupReadFile(lu)); + DBGPRINTF("lookup table '%s' loaded from file '%s'\n", lu->name, lu->filename); + +finalize_it: + cnfparamvalsDestruct(pvals, &modpblk); + RETiRet; +} + +void +lookupClassExit(void) +{ + objRelease(glbl, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); +} + +rsRetVal +lookupClassInit(void) +{ + DEFiRet; + CHKiRet(objGetObjInterface(&obj)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +finalize_it: + RETiRet; +} diff --git a/runtime/lookup.h b/runtime/lookup.h new file mode 100644 index 00000000..c478d679 --- /dev/null +++ b/runtime/lookup.h @@ -0,0 +1,57 @@ +/* header for lookup.c + * + * Copyright 2013 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * -or- + * see COPYING.ASL20 in the source distribution + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef INCLUDED_LOOKUP_H +#define INCLUDED_LOOKUP_H +#include <libestr.h> + +struct lookup_tables_s { + lookup_t *root; /* the root of the template list */ + lookup_t *last; /* points to the last element of the template list */ +}; + +struct lookup_string_tab_etry_s { + uchar *key; + uchar *val; +}; + +/* a single lookup table */ +struct lookup_s { + pthread_rwlock_t rwlock; /* protect us in case of dynamic reloads */ + uchar *name; + uchar *filename; + uint32_t nmemb; + union { + lookup_string_tab_etry_t *strtab; + } d; + lookup_t *next; +}; + +/* prototypes */ +void lookupInitCnf(lookup_tables_t *lu_tabs); +rsRetVal lookupProcessCnf(struct cnfobj *o); +lookup_t *lookupFindTable(uchar *name); +es_str_t * lookupKey_estr(lookup_t *pThis, uchar *key); +void lookupDestruct(lookup_t *pThis); +void lookupClassExit(void); +void lookupDoHUP(); +rsRetVal lookupClassInit(void); + +#endif /* #ifndef INCLUDED_LOOKUP_H */ diff --git a/runtime/module-template.h b/runtime/module-template.h index 8a958f90..f65aaf0f 100644 --- a/runtime/module-template.h +++ b/runtime/module-template.h @@ -175,6 +175,44 @@ static rsRetVal freeInstance(void* pModData)\ RETiRet;\ } +/* createWrkrInstance() + */ +#define BEGINcreateWrkrInstance \ +static rsRetVal createWrkrInstance(wrkrInstanceData_t **ppWrkrData, instanceData *pData)\ + {\ + DEFiRet; /* store error code here */\ + wrkrInstanceData_t *pWrkrData; /* use this to point to data elements */ + +#define CODESTARTcreateWrkrInstance \ + if((pWrkrData = calloc(1, sizeof(wrkrInstanceData_t))) == NULL) {\ + *ppWrkrData = NULL;\ + ENDfunc \ + return RS_RET_OUT_OF_MEMORY;\ + } \ + pWrkrData->pData = pData; + +#define ENDcreateWrkrInstance \ + *ppWrkrData = pWrkrData;\ + RETiRet;\ +} + +/* freeWrkrInstance */ +#define BEGINfreeWrkrInstance \ +static rsRetVal freeWrkrInstance(void* pd)\ +{\ + DEFiRet;\ + wrkrInstanceData_t *pWrkrData; + +#define CODESTARTfreeWrkrInstance \ + pWrkrData = (wrkrInstanceData_t*) pd; + +#define ENDfreeWrkrInstance \ + if(pWrkrData != NULL)\ + free(pWrkrData); /* we need to free this in any case */\ + RETiRet;\ +} + + /* isCompatibleWithFeature() */ #define BEGINisCompatibleWithFeature \ @@ -194,7 +232,7 @@ static rsRetVal isCompatibleWithFeature(syslogFeature __attribute__((unused)) eF * introduced in v4.3.3 -- rgerhards, 2009-04-27 */ #define BEGINbeginTransaction \ -static rsRetVal beginTransaction(instanceData __attribute__((unused)) *pData)\ +static rsRetVal beginTransaction(wrkrInstanceData_t __attribute__((unused)) *pWrkrData)\ {\ DEFiRet; @@ -209,7 +247,7 @@ static rsRetVal beginTransaction(instanceData __attribute__((unused)) *pData)\ * introduced in v4.3.3 -- rgerhards, 2009-04-27 */ #define BEGINendTransaction \ -static rsRetVal endTransaction(instanceData __attribute__((unused)) *pData)\ +static rsRetVal endTransaction(wrkrInstanceData_t __attribute__((unused)) *pWrkrData)\ {\ DEFiRet; @@ -223,7 +261,7 @@ static rsRetVal endTransaction(instanceData __attribute__((unused)) *pData)\ /* doAction() */ #define BEGINdoAction \ -static rsRetVal doAction(uchar __attribute__((unused)) **ppString, unsigned __attribute__((unused)) iMsgOpts, instanceData __attribute__((unused)) *pData)\ +static rsRetVal doAction(uchar __attribute__((unused)) **ppString, unsigned __attribute__((unused)) iMsgOpts, wrkrInstanceData_t __attribute__((unused)) *pWrkrData)\ {\ DEFiRet; @@ -382,12 +420,12 @@ static rsRetVal newInpInst(struct nvlst *lst)\ * rgerhard, 2007-08-02 */ #define BEGINtryResume \ -static rsRetVal tryResume(instanceData __attribute__((unused)) *pData)\ +static rsRetVal tryResume(wrkrInstanceData_t __attribute__((unused)) *pWrkrData)\ {\ DEFiRet; #define CODESTARTtryResume \ - assert(pData != NULL); + assert(pWrkrData != NULL); #define ENDtryResume \ RETiRet;\ @@ -467,6 +505,13 @@ static rsRetVal queryEtryPt(uchar *name, rsRetVal (**pEtryPoint)())\ *pEtryPoint = tryResume;\ } +/* standard queries for output module interface in rsyslog v8+ */ +#define CODEqueryEtryPt_STD_OMOD8_QUERIES \ + else if(!strcmp((char*) name, "createWrkrInstance")) {\ + *pEtryPoint = createWrkrInstance;\ + } else if(!strcmp((char*) name, "freeWrkrInstance")) {\ + *pEtryPoint = freeWrkrInstance;\ + } /* the following definition is queryEtryPt block that must be added * if an output module supports the transactional interface. diff --git a/runtime/modules.c b/runtime/modules.c index 56606306..52096082 100644 --- a/runtime/modules.c +++ b/runtime/modules.c @@ -652,6 +652,9 @@ doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_ CHKiRet((*pNew->modQueryEtryPt)((uchar*)"doAction", &pNew->mod.om.doAction)); CHKiRet((*pNew->modQueryEtryPt)((uchar*)"parseSelectorAct", &pNew->mod.om.parseSelectorAct)); CHKiRet((*pNew->modQueryEtryPt)((uchar*)"tryResume", &pNew->tryResume)); + CHKiRet((*pNew->modQueryEtryPt)((uchar*)"createWrkrInstance", &pNew->mod.om.createWrkrInstance)); + CHKiRet((*pNew->modQueryEtryPt)((uchar*)"freeWrkrInstance", &pNew->mod.om.freeWrkrInstance)); + /* try load optional interfaces */ localRet = (*pNew->modQueryEtryPt)((uchar*)"doHUP", &pNew->doHUP); if(localRet != RS_RET_OK && localRet != RS_RET_MODULE_ENTRY_POINT_NOT_FOUND) diff --git a/runtime/modules.h b/runtime/modules.h index 64644be2..a0637ac9 100644 --- a/runtime/modules.h +++ b/runtime/modules.h @@ -122,10 +122,6 @@ struct modInfo_s { rsRetVal (*activateCnf)(void*Cnf); /* make provided config the running conf */ rsRetVal (*freeCnf)(void*Cnf); /* end v2 config system specific */ - /* below: create an instance of this module. Most importantly the module - * can allocate instance memory in this call. - */ - rsRetVal (*createInstance)(); union { struct {/* data for input modules */ /* TODO: remove? */rsRetVal (*willRun)(void); /* check if the current config will be able to run*/ @@ -143,6 +139,8 @@ struct modInfo_s { rsRetVal (*parseSelectorAct)(uchar**, void**,omodStringRequest_t**); rsRetVal (*newActInst)(uchar *modName, struct nvlst *lst, void **, omodStringRequest_t **); rsRetVal (*SetShutdownImmdtPtr)(void *pData, void *pPtr); + rsRetVal (*createWrkrInstance)(void*ppWrkrData, void*pData); + rsRetVal (*freeWrkrInstance)(void*pWrkrData); } om; struct { /* data for library modules */ char dummy; diff --git a/runtime/msg.c b/runtime/msg.c index 10ecf48a..5df4e031 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -65,6 +65,13 @@ #include "net.h" #include "var.h" #include "rsconf.h" +#include "parserif.h" + +/* TODO: move the global variable root to the config object - had no time to to it + * right now before vacation -- rgerhards, 2013-07-22 + */ +static pthread_rwlock_t glblVars_rwlock; +struct json_object *global_var_root = NULL; /* static data */ DEFobjStaticHelpers @@ -317,7 +324,7 @@ static pthread_mutex_t mutTrimCtr; /* mutex to handle malloc trim */ /* some forward declarations */ static int getAPPNAMELen(msg_t *pM, sbool bLockMutex); -static rsRetVal jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate); +static rsRetVal jsonPathFindParent(struct json_object *jroot, uchar *name, uchar *leaf, struct json_object **parent, int bCreate); static uchar * jsonPathGetLeaf(uchar *name, int lenName); static struct json_object *jsonDeepCopy(struct json_object *src); @@ -377,13 +384,6 @@ MsgSetRulesetByName(msg_t *pMsg, cstr_t *rulesetName) rulesetGetRuleset(runConf, &(pMsg->pRuleset), rsCStrGetSzStrNoNULL(rulesetName)); } - -static inline int getProtocolVersion(msg_t *pM) -{ - return(pM->iProtocolVersion); -} - - /* do a DNS reverse resolution, if not already done, reflect status * rgerhards, 2009-11-16 */ @@ -452,14 +452,12 @@ getRcvFromIP(msg_t *pM) } -/* map a property name (C string) to a property ID */ +/* map a property name (string) to a property ID */ rsRetVal -propNameStrToID(uchar *pName, propid_t *pPropID) +propNameToID(uchar *pName, propid_t *pPropID) { DEFiRet; - assert(pName != NULL); - /* sometimes there are aliases to the original MonitoWare * property names. These come after || in the ifs below. */ if(!strcmp((char*) pName, "msg")) { @@ -534,13 +532,18 @@ propNameStrToID(uchar *pName, propid_t *pPropID) *pPropID = PROP_SYS_MYHOSTNAME; } else if(!strcmp((char*) pName, "$!all-json")) { *pPropID = PROP_CEE_ALL_JSON; - } else if(!strncmp((char*) pName, "$!", 2)) { - *pPropID = PROP_CEE; } else if(!strcmp((char*) pName, "$bom")) { *pPropID = PROP_SYS_BOM; } else if(!strcmp((char*) pName, "$uptime")) { *pPropID = PROP_SYS_UPTIME; + } else if(!strncmp((char*) pName, "$!", 2) || pName[0] == '!') { + *pPropID = PROP_CEE; + } else if(!strncmp((char*) pName, "$.", 2) || pName[0] == '.') { + *pPropID = PROP_LOCAL_VAR; + } else if(!strncmp((char*) pName, "$/", 2) || pName[0] == '/') { + *pPropID = PROP_GLOBAL_VAR; } else { + DBGPRINTF("PROP_INVALID for name '%s'\n", pName); *pPropID = PROP_INVALID; iRet = RS_RET_VAR_NOT_FOUND; } @@ -549,21 +552,6 @@ propNameStrToID(uchar *pName, propid_t *pPropID) } -/* map a property name (string) to a property ID */ -rsRetVal -propNameToID(cstr_t *pCSPropName, propid_t *pPropID) -{ - uchar *pName; - DEFiRet; - - assert(pCSPropName != NULL); - assert(pPropID != NULL); - pName = rsCStrGetSzStrNoNULL(pCSPropName); - iRet = propNameStrToID(pName, pPropID); - RETiRet; -} - - /* map a property ID to a name string (useful for displaying) */ uchar *propIDToName(propid_t propID) { @@ -634,6 +622,8 @@ uchar *propIDToName(propid_t propID) return UCHAR_CONSTANT("$MYHOSTNAME"); case PROP_CEE: return UCHAR_CONSTANT("*CEE-based property*"); + case PROP_LOCAL_VAR: + return UCHAR_CONSTANT("*LOCAL_VARIABLE*"); case PROP_CEE_ALL_JSON: return UCHAR_CONSTANT("$!all-json"); case PROP_SYS_BOM: @@ -699,7 +689,7 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->pszTIMESTAMP3339 = NULL; pM->pszTIMESTAMP_MySQL = NULL; pM->pszTIMESTAMP_PgSQL = NULL; - pM->pCSStrucData = NULL; + pM->pszStrucData = NULL; pM->pCSAPPNAME = NULL; pM->pCSPROCID = NULL; pM->pCSMSGID = NULL; @@ -708,6 +698,8 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) pM->rcvFrom.pRcvFrom = NULL; pM->pRuleset = NULL; pM->json = NULL; + pM->localvars = NULL; + pM->dfltTZ[0] = '\0'; memset(&pM->tRcvdAt, 0, sizeof(pM->tRcvdAt)); memset(&pM->tTIMESTAMP, 0, sizeof(pM->tTIMESTAMP)); pM->TAG.pszTAG = NULL; @@ -841,10 +833,9 @@ CODESTARTobjDestruct(msg) free(pThis->pszRcvdAt_PgSQL); free(pThis->pszTIMESTAMP_MySQL); free(pThis->pszTIMESTAMP_PgSQL); + free(pThis->pszStrucData); if(pThis->iLenPROGNAME >= CONF_PROGNAME_BUFSIZE) free(pThis->PROGNAME.ptr); - if(pThis->pCSStrucData != NULL) - rsCStrDestruct(&pThis->pCSStrucData); if(pThis->pCSAPPNAME != NULL) rsCStrDestruct(&pThis->pCSAPPNAME); if(pThis->pCSPROCID != NULL) @@ -853,6 +844,8 @@ CODESTARTobjDestruct(msg) rsCStrDestruct(&pThis->pCSMSGID); if(pThis->json != NULL) json_object_put(pThis->json); + if(pThis->localvars != NULL) + json_object_put(pThis->localvars); if(pThis->pszUUID != NULL) free(pThis->pszUUID); # ifndef HAVE_ATOMIC_BUILTINS @@ -994,14 +987,21 @@ msg_t* MsgDup(msg_t* pOld) tmpCOPYSZ(HOSTNAME); } } + if(pOld->pszStrucData == NULL) { + pNew->pszStrucData = NULL; + } else { + pNew->pszStrucData = (uchar*)strdup((char*)pOld->pszStrucData); + pNew->lenStrucData = pOld->lenStrucData; + } - tmpCOPYCSTR(StrucData); tmpCOPYCSTR(APPNAME); tmpCOPYCSTR(PROCID); tmpCOPYCSTR(MSGID); if(pOld->json != NULL) pNew->json = jsonDeepCopy(pOld->json); + if(pOld->localvars != NULL) + pNew->localvars = jsonDeepCopy(pOld->localvars); /* we do not copy all other cache properties, as we do not even know * if they are needed once again. So we let them re-create if needed. @@ -1056,12 +1056,17 @@ static rsRetVal MsgSerialize(msg_t *pThis, strm_t *pStrm) CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFrom"), PROPTYPE_PSZ, (void*) psz)); psz = getRcvFromIP(pThis); CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFromIP"), PROPTYPE_PSZ, (void*) psz)); + psz = pThis->pszStrucData; + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvStrucData"), PROPTYPE_PSZ, (void*) psz)); if(pThis->json != NULL) { psz = (uchar*) json_object_get_string(pThis->json); CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("json"), PROPTYPE_PSZ, (void*) psz)); } + if(pThis->localvars != NULL) { + psz = (uchar*) json_object_get_string(pThis->localvars); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("localvars"), PROPTYPE_PSZ, (void*) psz)); + } - objSerializePTR(pStrm, pCSStrucData, CSTR); objSerializePTR(pStrm, pCSAPPNAME, CSTR); objSerializePTR(pStrm, pCSPROCID, CSTR); objSerializePTR(pStrm, pCSMSGID, CSTR); @@ -1202,7 +1207,15 @@ MsgDeserialize(msg_t *pMsg, strm_t *pStrm) reinitVar(pVar); CHKiRet(objDeserializeProperty(pVar, pStrm)); } - if(isProp("pCSStrucData")) { + if(isProp("localvars")) { + tokener = json_tokener_new(); + pMsg->localvars = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pVar->val.pStr), + cstrLen(pVar->val.pStr)); + json_tokener_free(tokener); + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } + if(isProp("pszStrucData")) { MsgSetStructuredData(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); reinitVar(pVar); CHKiRet(objDeserializeProperty(pVar, pStrm)); @@ -1292,7 +1305,7 @@ static rsRetVal aquirePROCIDFromTAG(msg_t *pM) if(pM->pCSPROCID != NULL) return RS_RET_OK; /* we are already done ;) */ - if(getProtocolVersion(pM) != 0) + if(msgGetProtocolVersion(pM) != 0) return RS_RET_OK; /* we can only emulate if we have legacy format */ pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); @@ -1975,7 +1988,7 @@ static inline void tryEmulateTAG(msg_t *pM, sbool bLockMutex) return; /* done, no need to emulate */ } - if(getProtocolVersion(pM) == 1) { + if(msgGetProtocolVersion(pM) == 1) { if(!strcmp(getPROCID(pM, MUTEX_ALREADY_LOCKED), "-")) { /* no process ID, use APP-NAME only */ MsgSetTAG(pM, (uchar*) getAPPNAME(pM, MUTEX_ALREADY_LOCKED), getAPPNAMELen(pM, MUTEX_ALREADY_LOCKED)); @@ -2075,42 +2088,27 @@ rsRetVal MsgSetStructuredData(msg_t *pMsg, char* pszStrucData) { DEFiRet; ISOBJ_TYPE_assert(pMsg, msg); - if(pMsg->pCSStrucData == NULL) { - /* we need to obtain the object first */ - CHKiRet(rsCStrConstruct(&pMsg->pCSStrucData)); - } - /* if we reach this point, we have the object */ - iRet = rsCStrSetSzStr(pMsg->pCSStrucData, (uchar*) pszStrucData); - + free(pMsg->pszStrucData); + CHKmalloc(pMsg->pszStrucData = (uchar*)strdup(pszStrucData)); + pMsg->lenStrucData = strlen(pszStrucData); finalize_it: RETiRet; } -/* get the length of the "STRUCTURED-DATA" sz string - * rgerhards, 2005-11-24 - */ -#if 0 /* This method is currently not called, be we like to preserve it */ -static int getStructuredDataLen(msg_t *pM) -{ - return (pM->pCSStrucData == NULL) ? 1 : rsCStrLen(pM->pCSStrucData); -} -#endif - -/* get the "STRUCTURED-DATA" as sz string - * rgerhards, 2005-11-24 - */ -static inline char *getStructuredData(msg_t *pM) +/* get the "STRUCTURED-DATA" as sz string, including length */ +void +MsgGetStructuredData(msg_t *pM, uchar **pBuf, rs_size_t *len) { - uchar *pszRet; - MsgLock(pM); - if(pM->pCSStrucData == NULL) - pszRet = UCHAR_CONSTANT("-"); - else - pszRet = rsCStrGetSzStrNoNULL(pM->pCSStrucData); + if(pM->pszStrucData == NULL) { + *pBuf = UCHAR_CONSTANT("-"), + *len = 1; + } else { + *pBuf = pM->pszStrucData, + *len = pM->lenStrucData; + } MsgUnlock(pM); - return (char*) pszRet; } /* get the "programname" as sz string @@ -2145,7 +2143,7 @@ static void tryEmulateAPPNAME(msg_t *pM) if(pM->pCSAPPNAME != NULL) return; /* we are already done */ - if(getProtocolVersion(pM) == 0) { + if(msgGetProtocolVersion(pM) == 0) { /* only then it makes sense to emulate */ MsgSetAPPNAME(pM, (char*)getProgramName(pM, MUTEX_ALREADY_LOCKED)); } @@ -2215,6 +2213,15 @@ void MsgSetInputName(msg_t *pThis, prop_t *inputName) pThis->pInputName = inputName; } +/* Set default TZ. Note that at most 7 chars are set, as we would + * otherwise overrun our buffer! + */ +void MsgSetDfltTZ(msg_t *pThis, char *tz) +{ + strncpy(pThis->dfltTZ, tz, 7); + pThis->dfltTZ[7] = '\0'; /* ensure 0-Term in case of overflow! */ +} + /* Set the pfrominet socket store, so that we can obtain the peer at some * later time. Note that we do not check if pRcvFrom is already set, so this @@ -2237,7 +2244,6 @@ finalize_it: RETiRet; } - /* rgerhards 2008-09-10: set RcvFrom name in msg object. This calls AddRef() * on the property, because this must be done in all current cases and there * is no case expected where this may not be necessary. @@ -2463,11 +2469,17 @@ typedef enum ENOWType { NOW_NOW, NOW_YEAR, NOW_MONTH, NOW_DAY, NOW_HOUR, NOW_HHO static uchar *getNOW(eNOWType eNow, struct syslogTime *t) { uchar *pBuf; + struct syslogTime tt; if((pBuf = (uchar*) MALLOC(sizeof(uchar) * tmpBUFSIZE)) == NULL) { return NULL; } + if(t == NULL) { /* can happen if called via script engine */ + datetime.getCurrTime(&tt, NULL); + t = &tt; + } + if(t->year == 0) { /* not yet set! */ datetime.getCurrTime(t, NULL); } @@ -2510,12 +2522,12 @@ static uchar *getNOW(eNOWType eNow, struct syslogTime *t) #undef tmpBUFSIZE /* clean up */ -/* Get a CEE-Property as string value*/ +/* Get a JSON-Property as string value (used for various types of JSON-based vars) */ rsRetVal -getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed) +getJSONPropVal(msg_t *pMsg, msgPropDescr_t *pProp, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed) { - uchar *name = NULL; uchar *leaf; + struct json_object *jroot; struct json_object *parent; struct json_object *field; DEFiRet; @@ -2523,15 +2535,26 @@ getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, un if(*pbMustBeFreed) free(*pRes); *pRes = NULL; - // TODO: mutex? - if(pM->json == NULL) goto finalize_it; - if(!es_strbufcmp(propName, (uchar*)"!", 1)) { - field = pM->json; + if(pProp->id == PROP_CEE) { + jroot = pMsg->json; + } else if(pProp->id == PROP_LOCAL_VAR) { + jroot = pMsg->localvars; + } else if(pProp->id == PROP_GLOBAL_VAR) { + pthread_rwlock_rdlock(&glblVars_rwlock); + jroot = global_var_root; } else { - name = (uchar*)es_str2cstr(propName, NULL); - leaf = jsonPathGetLeaf(name, ustrlen(name)); - CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + DBGPRINTF("msgGetJSONPropVal; invalid property id %d\n", + pProp->id); + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + if(jroot == NULL) goto finalize_it; + + if(!strcmp((char*)pProp->name, "!")) { + field = jroot; + } else { + leaf = jsonPathGetLeaf(pProp->name, pProp->nameLen); + CHKiRet(jsonPathFindParent(jroot, pProp->name, leaf, &parent, 1)); field = json_object_object_get(parent, (char*)leaf); } if(field != NULL) { @@ -2541,7 +2564,8 @@ getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, un } finalize_it: - free(name); + if(pProp->id == PROP_GLOBAL_VAR) + pthread_rwlock_unlock(&glblVars_rwlock); if(*pRes == NULL) { /* could not find any value, so set it to empty */ *pRes = (unsigned char*)""; @@ -2551,35 +2575,47 @@ finalize_it: } -/* Get a CEE-Property as native json object - */ +/* Get a JSON-based-variable as native json object */ rsRetVal -msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson) +msgGetJSONPropJSON(msg_t *pMsg, msgPropDescr_t *pProp, struct json_object **pjson) { - uchar *name = NULL; + struct json_object *jroot; uchar *leaf; struct json_object *parent; DEFiRet; - // TODO: mutex? - if(pM->json == NULL) { + if(pProp->id == PROP_CEE) { + jroot = pMsg->json; + } else if(pProp->id == PROP_LOCAL_VAR) { + jroot = pMsg->localvars; + } else if(pProp->id == PROP_GLOBAL_VAR) { + pthread_rwlock_rdlock(&glblVars_rwlock); + jroot = global_var_root; + } else { + DBGPRINTF("msgGetJSONPropJSON; invalid property id %d\n", + pProp->id); + ABORT_FINALIZE(RS_RET_NOT_FOUND); + } + if(jroot == NULL) { + DBGPRINTF("msgGetJSONPropJSON; jroot empty for property %s\n", + pProp->name); ABORT_FINALIZE(RS_RET_NOT_FOUND); } - if(!es_strbufcmp(propName, (uchar*)"!", 1)) { - *pjson = pM->json; + if(!strcmp((char*)pProp->name, "!")) { + *pjson = jroot; FINALIZE; } - name = (uchar*)es_str2cstr(propName, NULL); - leaf = jsonPathGetLeaf(name, ustrlen(name)); - CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + leaf = jsonPathGetLeaf(pProp->name, pProp->nameLen); + CHKiRet(jsonPathFindParent(jroot, pProp->name, leaf, &parent, 1)); *pjson = json_object_object_get(parent, (char*)leaf); if(*pjson == NULL) { ABORT_FINALIZE(RS_RET_NOT_FOUND); } finalize_it: - free(name); + if(pProp->id == PROP_GLOBAL_VAR) + pthread_rwlock_unlock(&glblVars_rwlock); RETiRet; } @@ -2784,7 +2820,7 @@ finalize_it: *pPropLen = sizeof("**OUT OF MEMORY**") - 1; \ return(UCHAR_CONSTANT("**OUT OF MEMORY**"));} uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - propid_t propid, es_str_t *propName, rs_size_t *pPropLen, + msgPropDescr_t *pProp, rs_size_t *pPropLen, unsigned short *pbMustBeFreed, struct syslogTime *ttNow) { uchar *pRes; /* result pointer */ @@ -2807,7 +2843,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, *pbMustBeFreed = 0; - switch(propid) { + switch(pProp->id) { case PROP_MSG: pRes = getMSG(pMsg); bufLen = getMSGLen(pMsg); @@ -2880,7 +2916,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, pRes = (uchar*)getProtocolVersionString(pMsg); break; case PROP_STRUCTURED_DATA: - pRes = (uchar*)getStructuredData(pMsg); + MsgGetStructuredData(pMsg, &pRes, &bufLen); break; case PROP_APP_NAME: pRes = (uchar*)getAPPNAME(pMsg, LOCK_MUTEX); @@ -2979,7 +3015,9 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } break; case PROP_CEE: - getCEEPropVal(pMsg, propName, &pRes, &bufLen, pbMustBeFreed); + case PROP_LOCAL_VAR: + case PROP_GLOBAL_VAR: + getJSONPropVal(pMsg, pProp, &pRes, &bufLen, pbMustBeFreed); break; case PROP_SYS_BOM: if(*pbMustBeFreed == 1) @@ -3040,7 +3078,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* there is no point in continuing, we may even otherwise render the * error message unreadable. rgerhards, 2007-07-10 */ - dbgprintf("invalid property id: '%d'\n", propid); + dbgprintf("invalid property id: '%d'\n", pProp->id); *pbMustBeFreed = 0; *pPropLen = sizeof("**INVALID PROPERTY NAME**") - 1; return UCHAR_CONSTANT("**INVALID PROPERTY NAME**"); @@ -3648,66 +3686,6 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } -/* The function returns a cee variable suitable for use with RainerScript. - * Note: caller must free the returned string. - * Note that we need to do a lot of conversions between es_str_t and cstr -- this will go away once - * we have moved larger parts of rsyslog to es_str_t. Acceptable for the moment, especially as we intend - * to rewrite the script engine as well! - * rgerhards, 2010-12-03 - */ -es_str_t* -msgGetCEEVarNew(msg_t *pMsg, char *name) -{ - uchar *leaf; - char *val; - es_str_t *estr = NULL; - struct json_object *json, *parent; - - ISOBJ_TYPE_assert(pMsg, msg); - - if(pMsg->json == NULL) { - estr = es_newStr(1); - goto done; - } - leaf = jsonPathGetLeaf((uchar*)name, strlen(name)); - if(jsonPathFindParent(pMsg, (uchar*)name, leaf, &parent, 1) != RS_RET_OK) { - estr = es_newStr(1); - goto done; - } - json = json_object_object_get(parent, (char*)leaf); - val = (char*)json_object_get_string(json); - estr = es_newStrFromCStr(val, strlen(val)); -done: - return estr; -} - - -/* Return an es_str_t for given message property. - */ -es_str_t* -msgGetMsgVarNew(msg_t *pThis, uchar *name) -{ - rs_size_t propLen; - uchar *pszProp = NULL; - propid_t propid; - unsigned short bMustBeFreed = 0; - es_str_t *estr; - - ISOBJ_TYPE_assert(pThis, msg); - - /* always call MsgGetProp() without a template specifier */ - /* TODO: optimize propNameToID() call -- rgerhards, 2009-06-26 */ - propNameStrToID(name, &propid); - pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, NULL, &propLen, &bMustBeFreed, NULL); - - estr = es_newStrFromCStr((char*)pszProp, propLen); - if(bMustBeFreed) - free(pszProp); - - return estr; -} - - /* This function can be used as a generic way to set properties. * We have to handle a lot of legacy, so our return value is not always * 100% correct (called functions do not always provide one, should @@ -3758,7 +3736,7 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) prop.Destruct(&propRcvFrom); } else if(isProp("pszHOSTNAME")) { MsgSetHOSTNAME(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr)); - } else if(isProp("pCSStrucData")) { + } else if(isProp("pszStrucData")) { MsgSetStructuredData(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); } else if(isProp("pCSAPPNAME")) { MsgSetAPPNAME(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); @@ -3809,16 +3787,22 @@ static uchar * jsonPathGetLeaf(uchar *name, int lenName) { int i; - for(i = lenName ; name[i] != '!' && i >= 0 ; --i) - /* just skip */; - if(name[i] == '!') + for(i = lenName ; i >= 0 ; --i) + if(i == 0) { + if(name[0] == '!' || name[0] == '.' || name[0] == '/') + break; + } else { + if(name[i] == '!') + break; + } + if(name[i] == '!' || name[i] == '.' || name[i] == '/') ++i; return name + i; } static rsRetVal -jsonPathFindNext(struct json_object *root, uchar **name, uchar *leaf, +jsonPathFindNext(struct json_object *root, uchar *namestart, uchar **name, uchar *leaf, struct json_object **found, int bCreate) { uchar namebuf[1024]; @@ -3827,9 +3811,9 @@ jsonPathFindNext(struct json_object *root, uchar **name, uchar *leaf, uchar *p = *name; DEFiRet; - if(*p == '!') + if(*p == '!' || (*name == namestart && (*p == '.' || *p == '/'))) ++p; - for(i = 0 ; *p && *p != '!' && p != leaf && i < sizeof(namebuf)-1 ; ++i, ++p) + for(i = 0 ; *p && !(p == namestart && (*p == '.' || *p == '/')) && *p != '!' && p != leaf && i < sizeof(namebuf)-1 ; ++i, ++p) namebuf[i] = *p; if(i > 0) { namebuf[i] = '\0'; @@ -3853,12 +3837,14 @@ finalize_it: } static rsRetVal -jsonPathFindParent(msg_t *pM, uchar *name, uchar *leaf, struct json_object **parent, int bCreate) +jsonPathFindParent(struct json_object *jroot, uchar *name, uchar *leaf, struct json_object **parent, int bCreate) { + uchar *namestart; DEFiRet; - *parent = pM->json; + namestart = name; + *parent = jroot; while(name < leaf-1) { - jsonPathFindNext(*parent, &name, leaf, parent, bCreate); + jsonPathFindNext(*parent, namestart, &name, leaf, parent, bCreate); } RETiRet; } @@ -3885,31 +3871,28 @@ DBGPRINTF("AAAA jsonMerge adds '%s'\n", it.key); /* find a JSON structure element (field or container doesn't matter). */ rsRetVal -jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres) +jsonFind(struct json_object *jroot, msgPropDescr_t *pProp, struct json_object **jsonres) { - uchar *name = NULL; uchar *leaf; struct json_object *parent; struct json_object *field; DEFiRet; - if(pM->json == NULL) { + if(jroot == NULL) { field = NULL; goto finalize_it; } - if(!es_strbufcmp(propName, (uchar*)"!", 1)) { - field = pM->json; + if(!strcmp((char*)pProp->name, "!")) { + field = jroot; } else { - name = (uchar*)es_str2cstr(propName, NULL); - leaf = jsonPathGetLeaf(name, ustrlen(name)); - CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 0)); + leaf = jsonPathGetLeaf(pProp->name, pProp->nameLen); + CHKiRet(jsonPathFindParent(jroot, pProp->name, leaf, &parent, 0)); field = json_object_object_get(parent, (char*)leaf); } *jsonres = field; finalize_it: - free(name); RETiRet; } @@ -3917,23 +3900,33 @@ rsRetVal msgAddJSON(msg_t *pM, uchar *name, struct json_object *json) { /* TODO: error checks! This is a quick&dirty PoC! */ + struct json_object **pjroot; struct json_object *parent, *leafnode; uchar *leaf; DEFiRet; MsgLock(pM); - if(name[0] == '!' && name[1] == '\0') { - if(pM->json == NULL) - pM->json = json; + if(name[0] == '!') { + pjroot = &pM->json; + } else if(name[0] == '.') { + pjroot = &pM->localvars; + } else { /* globl var */ + pthread_rwlock_wrlock(&glblVars_rwlock); + pjroot = &global_var_root; + } + + if(name[1] == '\0') { /* full tree? */ + if(*pjroot == NULL) + *pjroot = json; else - CHKiRet(jsonMerge(pM->json, json)); + CHKiRet(jsonMerge(*pjroot, json)); } else { - if(pM->json == NULL) { + if(*pjroot == NULL) { /* now we need a root obj */ - pM->json = json_object_new_object(); + *pjroot = json_object_new_object(); } leaf = jsonPathGetLeaf(name, ustrlen(name)); - CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + CHKiRet(jsonPathFindParent(*pjroot, name, leaf, &parent, 1)); if (json_object_get_type(parent) != json_type_object) { DBGPRINTF("msgAddJSON: not a container in json path," "name is '%s'\n", name); @@ -3945,7 +3938,7 @@ msgAddJSON(msg_t *pM, uchar *name, struct json_object *json) json_object_object_add(parent, (char*)leaf, json); } else { if(json_object_get_type(json) == json_type_object) { - CHKiRet(jsonMerge(pM->json, json)); + CHKiRet(jsonMerge(*pjroot, json)); } else { //dbgprintf("AAAA: leafnode already exists, type is %d, update with %d\n", (int)json_object_get_type(leafnode), (int)json_object_get_type(json)); /* TODO: improve the code below, however, the current @@ -3971,33 +3964,52 @@ msgAddJSON(msg_t *pM, uchar *name, struct json_object *json) } finalize_it: + if(name[0] == '/') + pthread_rwlock_unlock(&glblVars_rwlock); MsgUnlock(pM); RETiRet; } + rsRetVal msgDelJSON(msg_t *pM, uchar *name) { + struct json_object **jroot; struct json_object *parent, *leafnode; uchar *leaf; DEFiRet; dbgprintf("AAAA: unset variable '%s'\n", name); MsgLock(pM); - if(name[0] == '!' && name[1] == '\0') { - /* strange, but I think we should permit this. After all, + + if(name[0] == '!') { + jroot = &pM->json; + } else if(name[0] == '.') { + jroot = &pM->localvars; + } else { /* globl var */ + pthread_rwlock_wrlock(&glblVars_rwlock); + jroot = &global_var_root; + } + if(jroot == NULL) { + DBGPRINTF("msgDelJSONVar; jroot empty in unset for property %s\n", + name); + FINALIZE; + } + + if(name[1] == '\0') { + /* full tree! Strange, but I think we should permit this. After all, * we trust rsyslog.conf to be written by the admin. */ DBGPRINTF("unsetting JSON root object\n"); - json_object_put(pM->json); - pM->json = NULL; + json_object_put(*jroot); + *jroot = NULL; } else { - if(pM->json == NULL) { + if(*jroot == NULL) { /* now we need a root obj */ - pM->json = json_object_new_object(); + *jroot = json_object_new_object(); } leaf = jsonPathGetLeaf(name, ustrlen(name)); - CHKiRet(jsonPathFindParent(pM, name, leaf, &parent, 1)); + CHKiRet(jsonPathFindParent(*jroot, name, leaf, &parent, 1)); leafnode = json_object_object_get(parent, (char*)leaf); DBGPRINTF("AAAA: unset found JSON value path '%s', " "leaf '%s', leafnode %p\n", name, leaf, leafnode); if(leafnode == NULL) { @@ -4012,6 +4024,8 @@ DBGPRINTF("AAAA: unset found JSON value path '%s', " "leaf '%s', leafnode %p\n", } finalize_it: + if(name[0] == '/') + pthread_rwlock_unlock(&glblVars_rwlock); MsgUnlock(pM); RETiRet; } @@ -4033,7 +4047,11 @@ jsonDeepCopy(struct json_object *src) dst = json_object_new_double(json_object_get_double(src)); break; case json_type_int: +#ifdef HAVE_JSON_OBJECT_NEW_INT64 + dst = json_object_new_int64(json_object_get_int64(src)); +#else /* HAVE_JSON_OBJECT_NEW_INT64 */ dst = json_object_new_int(json_object_get_int(src)); +#endif /* HAVE_JSON_OBJECT_NEW_INT64 */ break; case json_type_string: dst = json_object_new_string(json_object_get_string(src)); @@ -4076,7 +4094,11 @@ msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *v) free(cstr); break; case 'N':/* number (integer) */ +#ifdef HAVE_JSON_OBJECT_NEW_INT64 + json = json_object_new_int64(v->d.n); +#else /* HAVE_JSON_OBJECT_NEW_INT64 */ json = json_object_new_int((int) v->d.n); +#endif /* HAVE_JSON_OBJECT_NEW_INT64 */ break; case 'J':/* native JSON */ json = jsonDeepCopy(v->d.json); @@ -4085,11 +4107,76 @@ msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *v) v->datatype); ABORT_FINALIZE(RS_RET_ERR); } - msgAddJSON(pMsg, varname+1, json); + + msgAddJSON(pMsg, varname, json); +finalize_it: + RETiRet; +} + +rsRetVal +MsgAddToStructuredData(msg_t *pMsg, uchar *toadd, rs_size_t len) +{ + uchar *newptr; + rs_size_t newlen; + DEFiRet; + newlen = (pMsg->pszStrucData[0] == '-') ? len : pMsg->lenStrucData + len; + CHKmalloc(newptr = (uchar*) realloc(pMsg->pszStrucData, newlen+1)); + pMsg->pszStrucData = newptr; + if(pMsg->pszStrucData[0] == '-') { /* empty? */ + memcpy(pMsg->pszStrucData, toadd, len); + } else { + memcpy(pMsg->pszStrucData+pMsg->lenStrucData, toadd, len); + } + pMsg->pszStrucData[newlen] = '\0'; + pMsg->lenStrucData = newlen; finalize_it: RETiRet; } + +/* Fill a message propert description. Space must already be alloced + * by the caller. This is for efficiency, as we expect this to happen + * as part of a larger structure alloc. + * Note that CEE/LOCAL_VAR properties can come in either as + * "$!xx"/"$.xx" or "!xx"/".xx" - we will unify them here. + */ +rsRetVal +msgPropDescrFill(msgPropDescr_t *pProp, uchar *name, int nameLen) +{ + propid_t id; + int offs; + DEFiRet; + if(propNameToID(name, &id) != RS_RET_OK) { + parser_errmsg("invalid property '%s'", name); + ABORT_FINALIZE(RS_RET_INVLD_PROP); + } + if(id == PROP_CEE || id == PROP_LOCAL_VAR || id == PROP_GLOBAL_VAR) { + /* in these cases, we need the field name for later processing */ + /* normalize name: remove $ if present */ + offs = (name[0] == '$') ? 1 : 0; + pProp->name = ustrdup(name + offs); + pProp->nameLen = nameLen - offs; + /* we patch the root name, so that support functions do not need to + * check for different root chars. */ + pProp->name[0] = '!'; + } + pProp->id = id; +finalize_it: + RETiRet; +} + +void +msgPropDescrDestruct(msgPropDescr_t *pProp) +{ + if(pProp != NULL) { + if(pProp->id == PROP_CEE || + pProp->id == PROP_LOCAL_VAR || + pProp->id == PROP_GLOBAL_VAR) + free(pProp->name); + } +} + + /* dummy */ rsRetVal msgQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } @@ -4098,6 +4185,8 @@ rsRetVal msgQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } * rgerhards, 2008-01-04 */ BEGINObjClassInit(msg, 1, OBJ_IS_CORE_MODULE) + pthread_rwlock_init(&glblVars_rwlock, NULL); + /* request objects we use */ CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); diff --git a/runtime/msg.h b/runtime/msg.h index e7babdbb..a2392a20 100644 --- a/runtime/msg.h +++ b/runtime/msg.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-13 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -30,6 +30,7 @@ #include <pthread.h> #include <libestr.h> +#include <stdint.h> #include <json.h> #include "obj.h" #include "syslogd-types.h" @@ -85,7 +86,8 @@ struct msg { char *pszTIMESTAMP3339; /* TIMESTAMP as RFC3339 formatted string (32 charcters at most) */ char *pszTIMESTAMP_MySQL;/* TIMESTAMP as MySQL formatted string (always 14 charcters) */ char *pszTIMESTAMP_PgSQL;/* TIMESTAMP as PgSQL formatted string (always 21 characters) */ - cstr_t *pCSStrucData; /* STRUCTURED-DATA */ + uchar *pszStrucData; /* STRUCTURED-DATA */ + uint16_t lenStrucData; /* (cached) length of STRUCTURED-DATA */ cstr_t *pCSAPPNAME; /* APP-NAME */ cstr_t *pCSPROCID; /* PROCID */ cstr_t *pCSMSGID; /* MSGID */ @@ -107,6 +109,7 @@ struct msg { struct syslogTime tRcvdAt;/* time the message entered this program */ struct syslogTime tTIMESTAMP;/* (parsed) value of the timestamp */ struct json_object *json; + struct json_object *localvars; /* some fixed-size buffers to save malloc()/free() for frequently used fields (from the default templates) */ uchar szRawMsg[CONF_RAWMSG_BUFSIZE]; /* most messages are small, and these are stored here (without malloc/free!) */ uchar szHOSTNAME[CONF_HOSTNAME_BUFSIZE]; @@ -124,7 +127,8 @@ struct msg { char pszRcvdAt_SecFrac[7]; /* same as above. Both are fractional seconds for their respective timestamp */ char pszTIMESTAMP_Unix[12]; /* almost as small as a pointer! */ char pszRcvdAt_Unix[12]; - uchar *pszUUID; /* The message's UUID */ + char dfltTZ[8]; /* 7 chars max, less overhead than ptr! */ + uchar *pszUUID; /* The message's UUID */ }; @@ -141,6 +145,9 @@ struct msg { #define NEEDS_ACLCHK_U 0x080 /* check UDP ACLs after DNS resolution has been done in main queue consumer */ #define NO_PRI_IN_RAW 0x100 /* rawmsg does not include a PRI (Solaris!), but PRI is already set correctly in the msg object */ +/* (syslog) protocol types */ +#define MSG_LEGACY_PROTOCOL 0 +#define MSG_RFC5424_PROTOCOL 1 /* function prototypes */ @@ -154,6 +161,7 @@ msg_t* MsgDup(msg_t* pOld); msg_t *MsgAddRef(msg_t *pM); void setProtocolVersion(msg_t *pM, int iNewVersion); void MsgSetInputName(msg_t *pMsg, prop_t*); +void MsgSetDfltTZ(msg_t *pThis, char *tz); rsRetVal MsgSetAPPNAME(msg_t *pMsg, char* pszAPPNAME); rsRetVal MsgSetPROCID(msg_t *pMsg, char* pszPROCID); rsRetVal MsgSetMSGID(msg_t *pMsg, char* pszMSGID); @@ -162,6 +170,8 @@ void MsgSetTAG(msg_t *pMsg, uchar* pszBuf, size_t lenBuf); void MsgSetRuleset(msg_t *pMsg, ruleset_t*); rsRetVal MsgSetFlowControlType(msg_t *pMsg, flowControl_t eFlowCtl); rsRetVal MsgSetStructuredData(msg_t *pMsg, char* pszStrucData); +rsRetVal MsgAddToStructuredData(msg_t *pMsg, uchar *toadd, rs_size_t len); +void MsgGetStructuredData(msg_t *pM, uchar **pBuf, rs_size_t *len); rsRetVal msgSetFromSockinfo(msg_t *pThis, struct sockaddr_storage *sa); void MsgSetRcvFrom(msg_t *pMsg, prop_t*); void MsgSetRcvFromStr(msg_t *pMsg, uchar* pszRcvFrom, int, prop_t **); @@ -173,20 +183,14 @@ void MsgSetMSGoffs(msg_t *pMsg, short offs); void MsgSetRawMsgWOSize(msg_t *pMsg, char* pszRawMsg); void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg, size_t lenMsg); rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG); -uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - propid_t propid, es_str_t *propName, +uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, msgPropDescr_t *pProp, rs_size_t *pPropLen, unsigned short *pbMustBeFreed, struct syslogTime *ttNow); -rsRetVal msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar); -es_str_t* msgGetMsgVarNew(msg_t *pThis, uchar *name); uchar *getRcvFrom(msg_t *pM); void getTAG(msg_t *pM, uchar **ppBuf, int *piLen); char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt); char *getPRI(msg_t *pMsg); void getRawMsg(msg_t *pM, uchar **pBuf, int *piLen); -rsRetVal msgGetCEEVar(msg_t *pThis, cstr_t *propName, var_t **ppVar); -es_str_t* msgGetCEEVarNew(msg_t *pMsg, char *name); rsRetVal msgAddJSON(msg_t *pM, uchar *name, struct json_object *json); -rsRetVal getCEEPropVal(msg_t *pM, es_str_t *propName, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed); rsRetVal MsgGetSeverity(msg_t *pThis, int *piSeverity); rsRetVal MsgDeserialize(msg_t *pMsg, strm_t *pStrm); @@ -202,18 +206,29 @@ char *getHOSTNAME(msg_t *pM); int getHOSTNAMELen(msg_t *pM); uchar *getProgramName(msg_t *pM, sbool bLockMutex); uchar *getRcvFrom(msg_t *pM); -rsRetVal propNameToID(cstr_t *pCSPropName, propid_t *pPropID); +rsRetVal propNameToID(uchar *pName, propid_t *pPropID); uchar *propIDToName(propid_t propID); -rsRetVal msgGetCEEPropJSON(msg_t *pM, es_str_t *propName, struct json_object **pjson); +rsRetVal msgGetJSONPropJSON(msg_t *pMsg, msgPropDescr_t *pProp, struct json_object **pjson); +rsRetVal getJSONPropVal(msg_t *pMsg, msgPropDescr_t *pProp, uchar **pRes, rs_size_t *buflen, unsigned short *pbMustBeFreed); rsRetVal msgSetJSONFromVar(msg_t *pMsg, uchar *varname, struct var *var); rsRetVal msgDelJSON(msg_t *pMsg, uchar *varname); -rsRetVal jsonFind(msg_t *pM, es_str_t *propName, struct json_object **jsonres); +rsRetVal jsonFind(struct json_object *jroot, msgPropDescr_t *pProp, struct json_object **jsonres); -static inline rsRetVal -msgUnsetJSON(msg_t *pMsg, uchar *varname) { - return msgDelJSON(pMsg, varname+1); +rsRetVal msgPropDescrFill(msgPropDescr_t *pProp, uchar *name, int nameLen); +void msgPropDescrDestruct(msgPropDescr_t *pProp); + +static inline int +msgGetProtocolVersion(msg_t *pM) +{ + return(pM->iProtocolVersion); } +/* returns non-zero if the message has structured data, 0 otherwise */ +static inline sbool +MsgHasStructuredData(msg_t *pM) +{ + return (pM->pszStrucData == NULL) ? 0 : 1; +} /* ------------------------------ some inline functions ------------------------------ */ diff --git a/runtime/net.c b/runtime/net.c index 13391cc0..7c180b19 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -1189,12 +1189,16 @@ void closeUDPListenSockets(int *pSockArr) * hostname and/or pszPort may be NULL, but not both! * bIsServer indicates if a server socket should be created * 1 - server, 0 - client + * param rcvbuf indicates desired rcvbuf size; 0 means OS default */ -int *create_udp_socket(uchar *hostname, uchar *pszPort, int bIsServer) +int *create_udp_socket(uchar *hostname, uchar *pszPort, int bIsServer, int rcvbuf) { struct addrinfo hints, *res, *r; int error, maxs, *s, *socks, on = 1; int sockflags; + int actrcvbuf; + socklen_t optlen; + char errStr[1024]; assert(!((pszPort == NULL) && (hostname == NULL))); memset(&hints, 0, sizeof(hints)); @@ -1297,6 +1301,35 @@ int *create_udp_socket(uchar *hostname, uchar *pszPort, int bIsServer) continue; } + if(rcvbuf != 0) { +# if defined(SO_RCVBUFFORCE) + if(setsockopt(*s, SOL_SOCKET, SO_RCVBUFFORCE, &rcvbuf, sizeof(rcvbuf)) < 0) +# endif + { + /* if we fail, try to do it the regular way. Experiments show that at + * least some platforms do not return an error here, but silently set + * it to the max permitted value. So we do our error check a bit + * differently by querying the size below. + */ + setsockopt(*s, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof(rcvbuf)); + } + } + + if(Debug || rcvbuf != 0) { + optlen = sizeof(actrcvbuf); + if(getsockopt(*s, SOL_SOCKET, SO_RCVBUF, &actrcvbuf, &optlen) == 0) { + dbgprintf("socket %d, actual rcvbuf size %d\n", *s, actrcvbuf); + if(rcvbuf != 0 && actrcvbuf/2 != rcvbuf) { + errmsg.LogError(errno, NO_ERRCODE, + "cannot set rcvbuf size %d for socket %d, value now is %d", + rcvbuf, *s, actrcvbuf/2); + } + } else { + dbgprintf("could not obtain rcvbuf size for socket %d: %s\n", + *s, rs_strerror_r(errno, errStr, sizeof(errStr))); + } + } + if(bIsServer) { /* rgerhards, 2007-06-22: if we run on a kernel that does not support * the IPV6_V6ONLY socket option, we need to use a work-around. On such diff --git a/runtime/net.h b/runtime/net.h index b196116b..d7a7b512 100644 --- a/runtime/net.h +++ b/runtime/net.h @@ -137,7 +137,7 @@ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ void (*PrintAllowedSenders)(int iListToPrint); void (*clearAllowedSenders)(uchar*); void (*debugListenInfo)(int fd, char *type); - int *(*create_udp_socket)(uchar *hostname, uchar *LogPort, int bIsServer); + int *(*create_udp_socket)(uchar *hostname, uchar *LogPort, int bIsServer, int rcvbuf); void (*closeUDPListenSockets)(int *finet); int (*isAllowedSender)(uchar *pszType, struct sockaddr *pFrom, const char *pszFromHost); /* deprecated! */ rsRetVal (*getLocalHostname)(uchar**); diff --git a/runtime/nspoll.c b/runtime/nspoll.c index a936b255..43631f4e 100644 --- a/runtime/nspoll.c +++ b/runtime/nspoll.c @@ -66,7 +66,6 @@ loadDrvr(nspoll_t *pThis) uchar szDrvrName[48]; /* 48 shall be large enough */ pBaseDrvrName = pThis->pBaseDrvrName; - if(pBaseDrvrName == NULL) /* if no drvr name is set, use system default */ pBaseDrvrName = glbl.GetDfltNetstrmDrvr(); if(snprintf((char*)szDrvrName, sizeof(szDrvrName), "lmnsdpoll_%s", pBaseDrvrName) == sizeof(szDrvrName)) ABORT_FINALIZE(RS_RET_DRVRNAME_TOO_LONG); @@ -138,6 +137,29 @@ Wait(nspoll_t *pThis, int timeout, int *numEntries, nsd_epworkset_t workset[]) { } +/* set the base driver name. If the driver name + * is set to NULL, the previously set name is deleted but + * no name set again (which results in the system default being + * used)-- rgerhards, 2008-05-05 + */ +static rsRetVal +SetDrvrName(nspoll_t *pThis, uchar *pszName) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, netstrms); + if(pThis->pBaseDrvrName != NULL) { + free(pThis->pBaseDrvrName); + pThis->pBaseDrvrName = NULL; + } + + if(pszName != NULL) { + CHKmalloc(pThis->pBaseDrvrName = (uchar*) strdup((char*) pszName)); + } +finalize_it: + RETiRet; +} + + /* semantics like the epoll_ctl() function, does the same thing. * rgerhards, 2009-11-18 */ @@ -164,6 +186,7 @@ CODESTARTobjQueryInterface(nspoll) */ pIf->Construct = nspollConstruct; pIf->ConstructFinalize = ConstructFinalize; + pIf->SetDrvrName = SetDrvrName; pIf->Destruct = nspollDestruct; pIf->Wait = Wait; pIf->Ctl = Ctl; diff --git a/runtime/nspoll.h b/runtime/nspoll.h index 037f6c38..3a6e060c 100644 --- a/runtime/nspoll.h +++ b/runtime/nspoll.h @@ -53,8 +53,10 @@ BEGINinterface(nspoll) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Wait)(nspoll_t *pNsdpoll, int timeout, int *numEntries, nsd_epworkset_t workset[]); rsRetVal (*Ctl)(nspoll_t *pNsdpoll, netstrm_t *pStrm, int id, void *pUsr, int mode, int op); rsRetVal (*IsEPollSupported)(void); /* static method */ + /* v3 - 2013-09-17 by rgerhards */ + rsRetVal (*SetDrvrName)(nspoll_t *pThis, uchar *name); ENDinterface(nspoll) -#define nspollCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define nspollCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface change in v2 is that wait supports multiple return objects */ /* prototypes */ diff --git a/runtime/nssel.c b/runtime/nssel.c index 751dae9b..6ca0f262 100644 --- a/runtime/nssel.c +++ b/runtime/nssel.c @@ -127,6 +127,29 @@ finalize_it: } +/* set the base driver name. If the driver name + * is set to NULL, the previously set name is deleted but + * no name set again (which results in the system default being + * used)-- rgerhards, 2008-05-05 + */ +static rsRetVal +SetDrvrName(nssel_t *pThis, uchar *pszName) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, netstrms); + if(pThis->pBaseDrvrName != NULL) { + free(pThis->pBaseDrvrName); + pThis->pBaseDrvrName = NULL; + } + + if(pszName != NULL) { + CHKmalloc(pThis->pBaseDrvrName = (uchar*) strdup((char*) pszName)); + } +finalize_it: + RETiRet; +} + + /* Add a stream object to the current select() set. * Note that a single stream may have multiple "sockets" if * it is a listener. If so, all of them are begin added. @@ -195,6 +218,7 @@ CODESTARTobjQueryInterface(nssel) pIf->Construct = nsselConstruct; pIf->ConstructFinalize = ConstructFinalize; pIf->Destruct = nsselDestruct; + pIf->SetDrvrName = SetDrvrName; pIf->Add = Add; pIf->Wait = Wait; pIf->IsReady = IsReady; diff --git a/runtime/nssel.h b/runtime/nssel.h index d7f4fcd3..6131d9b4 100644 --- a/runtime/nssel.h +++ b/runtime/nssel.h @@ -42,8 +42,10 @@ BEGINinterface(nssel) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Add)(nssel_t *pThis, netstrm_t *pStrm, nsdsel_waitOp_t waitOp); rsRetVal (*Wait)(nssel_t *pThis, int *pNumReady); rsRetVal (*IsReady)(nssel_t *pThis, netstrm_t *pStrm, nsdsel_waitOp_t waitOp, int *pbIsReady, int *piNumReady); + /* v2 - 2013-09-17 by rgerhards */ + rsRetVal (*SetDrvrName)(nssel_t *pThis, uchar *name); ENDinterface(nssel) -#define nsselCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define nsselCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ /* prototypes */ PROTOTYPEObj(nssel); diff --git a/runtime/queue.c b/runtime/queue.c index 073c6823..8fb734e7 100644 --- a/runtime/queue.c +++ b/runtime/queue.c @@ -12,7 +12,7 @@ * function names - this makes it really hard to read and does not provide much * benefit, at least I (now) think so... * - * Copyright 2008-2011 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -85,7 +85,6 @@ static rsRetVal qqueueMultiEnqObjDirect(qqueue_t *pThis, multi_submit_t *pMultiS static rsRetVal qAddDirect(qqueue_t *pThis, msg_t *pMsg); static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qConstructDirect(qqueue_t __attribute__((unused)) *pThis); -static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis); static rsRetVal qDestructDisk(qqueue_t *pThis); /* some constants for queuePersist () */ @@ -118,6 +117,7 @@ static struct cnfparamdescr cnfpdescr[] = { { "queue.dequeueslowdown", eCmdHdlrInt, 0 }, { "queue.dequeuetimebegin", eCmdHdlrInt, 0 }, { "queue.dequeuetimeend", eCmdHdlrInt, 0 }, + { "queue.cry.provider", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk pblk = { CNFPARAMBLK_VERSION, @@ -350,16 +350,15 @@ qqueueAdviseMaxWorkers(qqueue_t *pThis) if(pThis->bIsDA && getLogicalQueueSize(pThis) >= pThis->iHighWtrMrk) { DBGOPRINT((obj_t*) pThis, "(re)activating DA worker\n"); wtpAdviseMaxWorkers(pThis->pWtpDA, 1); /* disk queues have always one worker */ + } + if(getLogicalQueueSize(pThis) == 0) { + iMaxWorkers = 0; + } else if(pThis->qType == QUEUETYPE_DISK || pThis->iMinMsgsPerWrkr == 0) { + iMaxWorkers = 1; } else { - if(getLogicalQueueSize(pThis) == 0) { - iMaxWorkers = 0; - } else if(pThis->qType == QUEUETYPE_DISK || pThis->iMinMsgsPerWrkr == 0) { - iMaxWorkers = 1; - } else { - iMaxWorkers = getLogicalQueueSize(pThis) / pThis->iMinMsgsPerWrkr + 1; - } - wtpAdviseMaxWorkers(pThis->pWtpReg, iMaxWorkers); + iMaxWorkers = getLogicalQueueSize(pThis) / pThis->iMinMsgsPerWrkr + 1; } + wtpAdviseMaxWorkers(pThis->pWtpReg, iMaxWorkers); } RETiRet; @@ -483,7 +482,6 @@ InitDA(qqueue_t *pThis, int bLockMutex) CHKiRet(wtpSetpfDoWork (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, void *pWti)) ConsumerDA)); CHKiRet(wtpSetpfObjProcessed (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, wti_t *pWti)) batchProcessed)); CHKiRet(wtpSetpmutUsr (pThis->pWtpDA, pThis->mut)); - CHKiRet(wtpSetpcondBusy (pThis->pWtpDA, &pThis->notEmpty)); CHKiRet(wtpSetiNumWorkerThreads (pThis->pWtpDA, 1)); CHKiRet(wtpSettoWrkShutdown (pThis->pWtpDA, pThis->toWrkShutdown)); CHKiRet(wtpSetpUsr (pThis->pWtpDA, pThis)); @@ -706,9 +704,13 @@ queueSwitchToEmergencyMode(qqueue_t *pThis, rsRetVal initiatingError) pThis->qType = QUEUETYPE_DIRECT; pThis->qConstruct = qConstructDirect; pThis->qDestruct = qDestructDirect; + /* these entry points shall not be used in direct mode + * To catch program errors, make us abort if that happens! + * rgerhards, 2013-11-05 + */ pThis->qAdd = qAddDirect; - pThis->qDel = qDelDirect; pThis->MultiEnq = qqueueMultiEnqObjDirect; + pThis->qDel = NULL; if(pThis->pqParent != NULL) { DBGOPRINT((obj_t*) pThis, "DA queue is in emergency mode, disabling DA in parent\n"); pThis->pqParent->bIsDA = 0; @@ -776,11 +778,19 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) (rsRetVal(*)(obj_t*,void*))qqueueLoadPersStrmInfoFixup, pThis)); CHKiRet(obj.Deserialize(&pThis->tVars.disk.pReadDel, (uchar*) "strm", psQIF, (rsRetVal(*)(obj_t*,void*))qqueueLoadPersStrmInfoFixup, pThis)); - /* create a duplicate for the read "pointer". */ CHKiRet(strm.Dup(pThis->tVars.disk.pReadDel, &pThis->tVars.disk.pReadDeq)); CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pReadDeq, 0)); /* deq must NOT delete the files! */ CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pReadDeq)); + /* if we use a crypto provider, we need to amend the objects with it's info */ + if(pThis->useCryprov) { + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pWrite, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pWrite, pThis->cryprovData)); + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pReadDeq, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pReadDeq, pThis->cryprovData)); + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pReadDel, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pReadDel, pThis->cryprovData)); + } CHKiRet(strm.SeekCurrOffs(pThis->tVars.disk.pWrite)); CHKiRet(strm.SeekCurrOffs(pThis->tVars.disk.pReadDel)); @@ -834,6 +844,10 @@ static rsRetVal qConstructDisk(qqueue_t *pThis) CHKiRet(strm.SetiMaxFiles(pThis->tVars.disk.pWrite, 10000000)); CHKiRet(strm.SettOperationsMode(pThis->tVars.disk.pWrite, STREAMMODE_WRITE)); CHKiRet(strm.SetsType(pThis->tVars.disk.pWrite, STREAMTYPE_FILE_CIRCULAR)); + if(pThis->useCryprov) { + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pWrite, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pWrite, pThis->cryprovData)); + } CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pWrite)); CHKiRet(strm.Construct(&pThis->tVars.disk.pReadDeq)); @@ -842,6 +856,10 @@ static rsRetVal qConstructDisk(qqueue_t *pThis) CHKiRet(strm.SetiMaxFiles(pThis->tVars.disk.pReadDeq, 10000000)); CHKiRet(strm.SettOperationsMode(pThis->tVars.disk.pReadDeq, STREAMMODE_READ)); CHKiRet(strm.SetsType(pThis->tVars.disk.pReadDeq, STREAMTYPE_FILE_CIRCULAR)); + if(pThis->useCryprov) { + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pReadDeq, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pReadDeq, pThis->cryprovData)); + } CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pReadDeq)); CHKiRet(strm.Construct(&pThis->tVars.disk.pReadDel)); @@ -851,6 +869,10 @@ static rsRetVal qConstructDisk(qqueue_t *pThis) CHKiRet(strm.SetiMaxFiles(pThis->tVars.disk.pReadDel, 10000000)); CHKiRet(strm.SettOperationsMode(pThis->tVars.disk.pReadDel, STREAMMODE_READ)); CHKiRet(strm.SetsType(pThis->tVars.disk.pReadDel, STREAMTYPE_FILE_CIRCULAR)); + if(pThis->useCryprov) { + CHKiRet(strm.Setcryprov(pThis->tVars.disk.pReadDel, &pThis->cryprov)); + CHKiRet(strm.SetcryprovData(pThis->tVars.disk.pReadDel, pThis->cryprovData)); + } CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pReadDel)); CHKiRet(strm.SetFName(pThis->tVars.disk.pWrite, pThis->pszFilePrefix, pThis->lenFilePrefix)); @@ -938,13 +960,11 @@ static rsRetVal qDestructDirect(qqueue_t __attribute__((unused)) *pThis) return RS_RET_OK; } -static rsRetVal qAddDirect(qqueue_t *pThis, msg_t* pMsg) +static rsRetVal qAddDirectWithWti(qqueue_t *pThis, msg_t* pMsg, wti_t *pWti) { batch_t singleBatch; batch_obj_t batchObj; batch_state_t batchState = BATCH_STATE_RDY; - sbool active = 1; - int i; DEFiRet; //TODO: init batchObj (states _OK and new fields -- CHECK) @@ -964,46 +984,29 @@ static rsRetVal qAddDirect(qqueue_t *pThis, msg_t* pMsg) singleBatch.nElem = 1; /* there always is only one in direct mode */ singleBatch.pElem = &batchObj; singleBatch.eltState = &batchState; - singleBatch.active = &active; - iRet = pThis->pConsumer(pThis->pAction, &singleBatch, &pThis->bShutdownImmediate); - /* delete the batch string params: TODO: create its own "class" for this */ - for(i = 0 ; i < CONF_OMOD_NUMSTRINGS_MAXSIZE ; ++i) { - free(batchObj.staticActStrings[i]); - } + iRet = pThis->pConsumer(pThis->pAction, &singleBatch, pWti); msgDestruct(&pMsg); RETiRet; } -/* "enqueue" a batch in direct mode. This is a shortcut which saves all the overhead - * otherwise incured. -- rgerhards, ~2010-06-23 +/* this is called if we do not have a pWti. This currently only happens + * when we are called from a main queue in direct mode. If so, we need + * to obtain a dummy pWti. */ -rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch) +static rsRetVal +qAddDirect(qqueue_t *pThis, msg_t* pMsg) { + wti_t *pWti; DEFiRet; - ASSERT(pThis != NULL); - - /* calling the consumer is quite different here than it is from a worker thread */ - /* we need to provide the consumer's return value back to the caller because in direct - * mode the consumer probably has a lot to convey (which get's lost in the other modes - * because they are asynchronous. But direct mode is deliberately synchronous. - * rgerhards, 2008-02-12 - * We use our knowledge about the batch_t structure below, but without that, we - * pay a too-large performance toll... -- rgerhards, 2009-04-22 - */ - iRet = pThis->pConsumer(pThis->pAction, pBatch, NULL); - + pWti = wtiGetDummy(); + pWti->pbShutdownImmediate = &pThis->bShutdownImmediate; + iRet = qAddDirectWithWti(pThis, pMsg, pWti); RETiRet; } -static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis) -{ - return RS_RET_OK; -} - - /* --------------- end type-specific handlers -------------------- */ @@ -1298,7 +1301,7 @@ finalize_it: * to modify some parameters before the queue is actually started. */ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThreads, - int iMaxQueueSize, rsRetVal (*pConsumer)(void*, batch_t*,int*)) + int iMaxQueueSize, rsRetVal (*pConsumer)(void*, batch_t*, wti_t*)) { DEFiRet; qqueue_t *pThis; @@ -1321,6 +1324,7 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->iMaxFileSize = 1024 * 1024; /* default is 1 MiB */ pThis->iQueueSize = 0; pThis->nLogDeq = 0; + pThis->useCryprov = 0; pThis->iMaxQueueSize = iMaxQueueSize; pThis->pConsumer = pConsumer; pThis->iNumWorkerThreads = iWorkerThreads; @@ -1856,7 +1860,8 @@ ConsumerReg(qqueue_t *pThis, wti_t *pWti) pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &iCancelStateSave); - CHKiRet(pThis->pConsumer(pThis->pAction, &pWti->batch, &pThis->bShutdownImmediate)); + pWti->pbShutdownImmediate = &pThis->bShutdownImmediate; + CHKiRet(pThis->pConsumer(pThis->pAction, &pWti->batch, pWti)); /* we now need to check if we should deliberately delay processing a bit * and, if so, do that. -- rgerhards, 2008-01-30 @@ -2078,9 +2083,13 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ case QUEUETYPE_DIRECT: pThis->qConstruct = qConstructDirect; pThis->qDestruct = qDestructDirect; + /* these entry points shall not be used in direct mode + * To catch program errors, make us abort if that happens! + * rgerhards, 2013-11-05 + */ pThis->qAdd = qAddDirect; - pThis->qDel = qDelDirect; pThis->MultiEnq = qqueueMultiEnqObjDirect; + pThis->qDel = NULL; break; } @@ -2118,7 +2127,6 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ pthread_mutex_init(&pThis->mutThrdMgmt, NULL); pthread_cond_init (&pThis->notFull, NULL); - pthread_cond_init (&pThis->notEmpty, NULL); pthread_cond_init (&pThis->belowFullDlyWtrMrk, NULL); pthread_cond_init (&pThis->belowLightDlyWtrMrk, NULL); @@ -2136,11 +2144,13 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ } DBGOPRINT((obj_t*) pThis, "type %d, enq-only %d, disk assisted %d, maxFileSz %lld, maxQSize %d, lqsize %d, pqsize %d, child %d, " - "full delay %d, light delay %d, deq batch size %d starting, high wtrrmrk %d, low wtrmrk %d\n", + "full delay %d, light delay %d, deq batch size %d starting, high wtrmrk %d, low wtrmrk %d\n" + "max wrkr %d, min msgs f. wrkr %d\n", pThis->qType, pThis->bEnqOnly, pThis->bIsDA, pThis->iMaxFileSize, pThis->iMaxQueueSize, getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis), pThis->pqParent == NULL ? 0 : 1, pThis->iFullDlyMrk, pThis->iLightDlyMrk, - pThis->iDeqBatchSize, pThis->iHighWtrMrk, pThis->iLowWtrMrk); + pThis->iDeqBatchSize, pThis->iHighWtrMrk, pThis->iLowWtrMrk, + pThis->iNumWorkerThreads, pThis->iMinMsgsPerWrkr); pThis->bQueueStarted = 1; if(pThis->qType == QUEUETYPE_DIRECT) @@ -2157,7 +2167,6 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ CHKiRet(wtpSetpfDoWork (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, void *pWti)) ConsumerReg)); CHKiRet(wtpSetpfObjProcessed (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, wti_t *pWti)) batchProcessed)); CHKiRet(wtpSetpmutUsr (pThis->pWtpReg, pThis->mut)); - CHKiRet(wtpSetpcondBusy (pThis->pWtpReg, &pThis->notEmpty)); CHKiRet(wtpSetiNumWorkerThreads (pThis->pWtpReg, pThis->iNumWorkerThreads)); CHKiRet(wtpSettoWrkShutdown (pThis->pWtpReg, pThis->toWrkShutdown)); CHKiRet(wtpSetpUsr (pThis->pWtpReg, pThis)); @@ -2181,26 +2190,26 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ /* we need to save the queue size, as the stats module initializes it to 0! */ /* iQueueSize is a dual-use counter: no init, no mutex! */ CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("size"), - ctrType_Int, &pThis->iQueueSize)); + ctrType_Int, CTR_FLAG_NONE, &pThis->iQueueSize)); STATSCOUNTER_INIT(pThis->ctrEnqueued, pThis->mutCtrEnqueued); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("enqueued"), - ctrType_IntCtr, &pThis->ctrEnqueued)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrEnqueued)); STATSCOUNTER_INIT(pThis->ctrFull, pThis->mutCtrFull); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("full"), - ctrType_IntCtr, &pThis->ctrFull)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrFull)); STATSCOUNTER_INIT(pThis->ctrFDscrd, pThis->mutCtrFDscrd); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("discarded.full"), - ctrType_IntCtr, &pThis->ctrFDscrd)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrFDscrd)); STATSCOUNTER_INIT(pThis->ctrNFDscrd, pThis->mutCtrNFDscrd); CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("discarded.nf"), - ctrType_IntCtr, &pThis->ctrNFDscrd)); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &pThis->ctrNFDscrd)); pThis->ctrMaxqsize = 0; /* no mutex needed, thus no init call */ CHKiRet(statsobj.AddCounter(pThis->statsobj, UCHAR_CONSTANT("maxqsize"), - ctrType_Int, &pThis->ctrMaxqsize)); + ctrType_Int, CTR_FLAG_NONE, &pThis->ctrMaxqsize)); CHKiRet(statsobj.ConstructFinalize(pThis->statsobj)); @@ -2422,7 +2431,6 @@ CODESTARTobjDestruct(qqueue) } pthread_mutex_destroy(&pThis->mutThrdMgmt); pthread_cond_destroy(&pThis->notFull); - pthread_cond_destroy(&pThis->notEmpty); pthread_cond_destroy(&pThis->belowFullDlyWtrMrk); pthread_cond_destroy(&pThis->belowLightDlyWtrMrk); @@ -2435,6 +2443,13 @@ CODESTARTobjDestruct(qqueue) free(pThis->pszFilePrefix); free(pThis->pszSpoolDir); + if(pThis->useCryprov) { + pThis->cryprov.Destruct(&pThis->cryprovData); + obj.ReleaseObj(__FILE__, pThis->cryprovNameFull+2, pThis->cryprovNameFull, + (void*) &pThis->cryprov); + free(pThis->cryprovName); + free(pThis->cryprovNameFull); + } /* some queues do not provide stats and thus have no statsobj! */ if(pThis->statsobj != NULL) @@ -2656,13 +2671,14 @@ static rsRetVal qqueueMultiEnqObjDirect(qqueue_t *pThis, multi_submit_t *pMultiSub) { int i; + wti_t *pWti; DEFiRet; - ISOBJ_TYPE_assert(pThis, qqueue); - assert(pMultiSub != NULL); + pWti = wtiGetDummy(); + pWti->pbShutdownImmediate = &pThis->bShutdownImmediate; for(i = 0 ; i < pMultiSub->nElem ; ++i) { - CHKiRet(qAddDirect(pThis, (void*)pMultiSub->ppMsgs[i])); + CHKiRet(qAddDirectWithWti(pThis, (void*)pMultiSub->ppMsgs[i], pWti)); } finalize_it: @@ -2671,22 +2687,7 @@ finalize_it: /* ------------------------------ END multi-enqueue functions ------------------------------ */ -/* enqueue a new user data element in direct mode - * NOTE/TODO: This is a TESTER/EXPERIEMENTAL, to be changed to better - * code later on (like multi submit!) 2010-06-10 - * Enqueues the new element and awakes worker thread. - */ -rsRetVal -qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg) -{ - DEFiRet; - ISOBJ_TYPE_assert(pThis, qqueue); - iRet = qAddDirect(pThis, pMsg); - RETiRet; -} - - -/* enqueue a new user data element +/* enqueue a new user data element * Enqueues the new element and awakes worker thread. */ rsRetVal @@ -2720,27 +2721,67 @@ finalize_it: } -/* take v6 config list and extract the queue params out of it. Hand the - * param values back to the caller. Caller is responsible for destructing - * them when no longer needed. Caller can use this param block to configure - * all parameters for a newly created queue with one call to qqueueSetParams(). - * rgerhards, 2011-07-22 +/* are any queue params set at all? 1 - yes, 0 - no + * We need to evaluate the param block for this function, which is somewhat + * inefficient. HOWEVER, this is only done during config load, so we really + * don't care... -- rgerhards, 2013-05-10 */ -rsRetVal -qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals) +int +queueCnfParamsSet(struct nvlst *lst) { - *ppvals = nvlstGetParams(lst, &pblk, NULL); - return RS_RET_OK; + int r; + struct cnfparamvals *pvals; + + pvals = nvlstGetParams(lst, &pblk, NULL); + r = cnfparamvalsIsSet(&pblk, pvals); + cnfparamvalsDestruct(pvals, &pblk); + return r; } -/* are any queue params set at all? 1 - yes, 0 - no */ -int -queueCnfParamsSet(struct cnfparamvals *pvals) +static inline rsRetVal +initCryprov(qqueue_t *pThis, struct nvlst *lst) { - return cnfparamvalsIsSet(&pblk, pvals); -} + uchar szDrvrName[1024]; + DEFiRet; + if(snprintf((char*)szDrvrName, sizeof(szDrvrName), "lmcry_%s", pThis->cryprovName) + == sizeof(szDrvrName)) { + errmsg.LogError(0, RS_RET_ERR, "queue: crypto provider " + "name is too long: '%s' - encryption disabled", + pThis->cryprovName); + ABORT_FINALIZE(RS_RET_ERR); + } + pThis->cryprovNameFull = ustrdup(szDrvrName); + + pThis->cryprov.ifVersion = cryprovCURR_IF_VERSION; + /* The pDrvrName+2 below is a hack to obtain the object name. It + * safes us to have yet another variable with the name without "lm" in + * front of it. If we change the module load interface, we may re-think + * about this hack, but for the time being it is efficient and clean enough. + */ + if(obj.UseObj(__FILE__, szDrvrName, szDrvrName, (void*) &pThis->cryprov) + != RS_RET_OK) { + errmsg.LogError(0, RS_RET_LOAD_ERROR, "queue: could not load " + "crypto provider '%s' - encryption disabled", + szDrvrName); + ABORT_FINALIZE(RS_RET_CRYPROV_ERR); + } + + if(pThis->cryprov.Construct(&pThis->cryprovData) != RS_RET_OK) { + errmsg.LogError(0, RS_RET_CRYPROV_ERR, "queue: error constructing " + "crypto provider %s dataset - encryption disabled", + szDrvrName); + ABORT_FINALIZE(RS_RET_CRYPROV_ERR); + } + CHKiRet(pThis->cryprov.SetCnfParam(pThis->cryprovData, lst, CRYPROV_PARAMTYPE_DISK)); + + dbgprintf("loaded crypto provider %s, data instance at %p\n", + szDrvrName, pThis->cryprovData); + pThis->useCryprov = 1; +finalize_it: + RETiRet; +} /* apply all params from param block to queue. Must be called before * finalizing. This supports the v6 config system. Defaults were already @@ -2748,15 +2789,24 @@ queueCnfParamsSet(struct cnfparamvals *pvals) * function. */ rsRetVal -qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals) +qqueueApplyCnfParam(qqueue_t *pThis, struct nvlst *lst) { int i; + struct cnfparamvals *pvals; + + pvals = nvlstGetParams(lst, &pblk, NULL); + if(Debug) { + dbgprintf("queue param blk:\n"); + cnfparamsPrint(&pblk, pvals); + } for(i = 0 ; i < pblk.nParams ; ++i) { if(!pvals[i].bUsed) continue; if(!strcmp(pblk.descr[i].name, "queue.filename")) { pThis->pszFilePrefix = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); pThis->lenFilePrefix = es_strlen(pvals[i].val.d.estr); + } else if(!strcmp(pblk.descr[i].name, "queue.cry.provider")) { + pThis->cryprovName = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(pblk.descr[i].name, "queue.size")) { pThis->iMaxQueueSize = pvals[i].val.d.n; } else if(!strcmp(pblk.descr[i].name, "queue.dequeuebatchsize")) { @@ -2808,12 +2858,27 @@ qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals) "param '%s'\n", pblk.descr[i].name); } } - if(pThis->qType == QUEUETYPE_DISK && pThis->pszFilePrefix == NULL) { - errmsg.LogError(0, RS_RET_QUEUE_DISK_NO_FN, "error on queue '%s', disk mode selected, but " - "no queue file name given; queue type changed to 'linkedList'", + if(pThis->qType == QUEUETYPE_DISK) { + if(pThis->pszFilePrefix == NULL) { + errmsg.LogError(0, RS_RET_QUEUE_DISK_NO_FN, "error on queue '%s', disk mode selected, but " + "no queue file name given; queue type changed to 'linkedList'", + obj.GetName((obj_t*) pThis)); + pThis->qType = QUEUETYPE_LINKEDLIST; + } + } + + if(pThis->pszFilePrefix == NULL && pThis->cryprovName != NULL) { + errmsg.LogError(0, RS_RET_QUEUE_CRY_DISK_ONLY, "error on queue '%s', crypto provider can " + "only be set for disk or disk assisted queue - ignored", obj.GetName((obj_t*) pThis)); - pThis->qType = QUEUETYPE_LINKEDLIST; + free(pThis->cryprovName); + pThis->cryprovName = NULL; + } + + if(pThis->cryprovName != NULL) { + initCryprov(pThis, lst); } + cnfparamvalsDestruct(pvals, &pblk); return RS_RET_OK; } diff --git a/runtime/queue.h b/runtime/queue.h index 886fac8d..86107d24 100644 --- a/runtime/queue.h +++ b/runtime/queue.h @@ -30,6 +30,7 @@ #include "batch.h" #include "stream.h" #include "statsobj.h" +#include "cryprov.h" /* support for the toDelete list */ typedef struct toDeleteLst_s toDeleteLst_t; @@ -102,11 +103,10 @@ struct queue_s { * the user really wanted...). -- rgerhards, 2008-04-02 */ /* end dequeue time window */ - rsRetVal (*pConsumer)(void *,batch_t*,int*); /* user-supplied consumer function for dequeued messages */ + rsRetVal (*pConsumer)(void *,batch_t*, wti_t*); /* user-supplied consumer function for dequeued messages */ /* calling interface for pConsumer: arg1 is the global user pointer from this structure, arg2 is the * user pointer array that was dequeued (actual sample: for actions, arg1 is the pAction and arg2 - * is pointer to an array of message message pointers), arg3 is a pointer to an interger which is zero - * during normal operations and one if the consumer must urgently shut down. + * is pointer to an array of message message pointers) */ /* type-specific handlers (set during construction) */ rsRetVal (*qConstruct)(struct queue_s *pThis); @@ -121,7 +121,7 @@ struct queue_s { /* synchronization variables */ pthread_mutex_t mutThrdMgmt; /* mutex for the queue's thread management */ pthread_mutex_t *mut; /* mutex for enqueing and dequeueing messages */ - pthread_cond_t notFull, notEmpty; + pthread_cond_t notFull; pthread_cond_t belowFullDlyWtrMrk; /* below eFLOWCTL_FULL_DELAY watermark */ pthread_cond_t belowLightDlyWtrMrk; /* below eFLOWCTL_FULL_DELAY watermark */ int bThrdStateChanged; /* at least one thread state has changed if 1 */ @@ -168,6 +168,11 @@ struct queue_s { strm_t *pReadDel; /* current file for deleting */ } disk; } tVars; + sbool useCryprov; /* quicker than checkig ptr (1 vs 8 bytes!) */ + uchar *cryprovName; /* crypto provider to use */ + cryprov_if_t cryprov; /* ptr to crypto provider interface */ + void *cryprovData; /* opaque data ptr for provider use */ + uchar *cryprovNameFull;/* full internal crypto provider name */ DEF_ATOMIC_HELPER_MUT(mutQueueSize); DEF_ATOMIC_HELPER_MUT(mutLogDeq); /* for statistics subsystem */ @@ -189,17 +194,14 @@ struct queue_s { /* prototypes */ rsRetVal qqueueDestruct(qqueue_t **ppThis); -rsRetVal qqueueEnqMsgDirect(qqueue_t *pThis, msg_t *pMsg); rsRetVal qqueueEnqMsg(qqueue_t *pThis, flowControl_t flwCtlType, msg_t *pMsg); rsRetVal qqueueStart(qqueue_t *pThis); rsRetVal qqueueSetMaxFileSize(qqueue_t *pThis, size_t iMaxFileSize); rsRetVal qqueueSetFilePrefix(qqueue_t *pThis, uchar *pszPrefix, size_t iLenPrefix); rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThreads, - int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*, int*)); -rsRetVal qqueueEnqObjDirectBatch(qqueue_t *pThis, batch_t *pBatch); -rsRetVal qqueueDoCnfParams(struct nvlst *lst, struct cnfparamvals **ppvals); -int queueCnfParamsSet(struct cnfparamvals *pvals); -rsRetVal qqueueApplyCnfParam(qqueue_t *pThis, struct cnfparamvals *pvals); + int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*, wti_t *)); +int queueCnfParamsSet(struct nvlst *lst); +rsRetVal qqueueApplyCnfParam(qqueue_t *pThis, struct nvlst *lst); void qqueueSetDefaultsRulesetQueue(qqueue_t *pThis); void qqueueSetDefaultsActionQueue(qqueue_t *pThis); void qqueueDbgPrint(qqueue_t *pThis); diff --git a/runtime/rsconf.c b/runtime/rsconf.c index d8b81f1b..f2feb989 100644 --- a/runtime/rsconf.c +++ b/runtime/rsconf.c @@ -2,7 +2,7 @@ * * Module begun 2011-04-19 by Rainer Gerhards * - * Copyright 2011-2012 Adiscon GmbH. + * Copyright 2011-2013 Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -83,7 +83,7 @@ rsconf_t *runConf = NULL;/* the currently running config */ rsconf_t *loadConf = NULL;/* the config currently being loaded (no concurrent config load supported!) */ /* hardcoded standard templates (used for defaults) */ -static uchar template_DebugFormat[] = "\"Debug line with all properties:\nFROMHOST: '%FROMHOST%', fromhost-ip: '%fromhost-ip%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nescaped msg: '%msg:::drop-cc%'\ninputname: %inputname% rawmsg: '%rawmsg%'\n\n\""; +static uchar template_DebugFormat[] = "\"Debug line with all properties:\nFROMHOST: '%FROMHOST%', fromhost-ip: '%fromhost-ip%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nescaped msg: '%msg:::drop-cc%'\ninputname: %inputname% rawmsg: '%rawmsg%'\n$!:%$!%\n$.:%$.%\n$/:%$/%\n\n\""; static uchar template_SyslogProtocol23Format[] = "\"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n\""; static uchar template_TraditionalFileFormat[] = "=RSYSLOG_TraditionalFileFormat"; static uchar template_FileFormat[] = "=RSYSLOG_FileFormat"; @@ -124,15 +124,16 @@ BEGINobjConstruct(rsconf) /* be sure to specify the object type also in END macr pThis->templates.last = NULL; pThis->templates.lastStatic = NULL; pThis->actions.nbrActions = 0; + lookupInitCnf(&pThis->lu_tabs); CHKiRet(llInit(&pThis->rulesets.llRulesets, rulesetDestructForLinkedList, rulesetKeyDestruct, strcasecmp)); /* queue params */ - pThis->globals.mainQ.iMainMsgQueueSize = 10000; - pThis->globals.mainQ.iMainMsgQHighWtrMark = 8000; - pThis->globals.mainQ.iMainMsgQLowWtrMark = 2000; - pThis->globals.mainQ.iMainMsgQDiscardMark = 9800; + pThis->globals.mainQ.iMainMsgQueueSize = 100000; + pThis->globals.mainQ.iMainMsgQHighWtrMark = 80000; + pThis->globals.mainQ.iMainMsgQLowWtrMark = 20000; + pThis->globals.mainQ.iMainMsgQDiscardMark = 98000; pThis->globals.mainQ.iMainMsgQDiscardSeverity = 8; - pThis->globals.mainQ.iMainMsgQueueNumWorkers = 1; + pThis->globals.mainQ.iMainMsgQueueNumWorkers = 2; pThis->globals.mainQ.MainMsgQueType = QUEUETYPE_FIXED_ARRAY; pThis->globals.mainQ.pszMainMsgQFName = NULL; pThis->globals.mainQ.iMainMsgQueMaxFileSize = 1024*1024; @@ -142,10 +143,10 @@ BEGINobjConstruct(rsconf) /* be sure to specify the object type also in END macr pThis->globals.mainQ.iMainMsgQtoActShutdown = 1000; pThis->globals.mainQ.iMainMsgQtoEnq = 2000; pThis->globals.mainQ.iMainMsgQtoWrkShutdown = 60000; - pThis->globals.mainQ.iMainMsgQWrkMinMsgs = 100; + pThis->globals.mainQ.iMainMsgQWrkMinMsgs = 40000; pThis->globals.mainQ.iMainMsgQDeqSlowdown = 0; pThis->globals.mainQ.iMainMsgQueMaxDiskSpace = 0; - pThis->globals.mainQ.iMainMsgQueDeqBatchSize = 32; + pThis->globals.mainQ.iMainMsgQueDeqBatchSize = 256; pThis->globals.mainQ.bMainMsgQSaveOnShutdown = 1; pThis->globals.mainQ.iMainMsgQueueDeqtWinFromHr = 0; pThis->globals.mainQ.iMainMsgQueueDeqtWinToHr = 25; @@ -253,92 +254,6 @@ CODESTARTobjDebugPrint(rsconf) ENDobjDebugPrint(rsconf) -/* This function returns the current date in different - * variants. It is used to construct the $NOW series of - * system properties. The returned buffer must be freed - * by the caller when no longer needed. If the function - * can not allocate memory, it returns a NULL pointer. - * TODO: this was taken from msg.c and we should consolidate it with the code - * there. This is especially important when we increase the number of system - * variables (what we definitely want to do). - */ -typedef enum ENOWType { NOW_NOW, NOW_YEAR, NOW_MONTH, NOW_DAY, NOW_HOUR, NOW_MINUTE } eNOWType; -static rsRetVal -getNOW(eNOWType eNow, es_str_t **estr) -{ - DEFiRet; - uchar szBuf[16]; - struct syslogTime t; - es_size_t len; - - datetime.getCurrTime(&t, NULL); - switch(eNow) { - case NOW_NOW: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), - "%4.4d-%2.2d-%2.2d", t.year, t.month, t.day); - break; - case NOW_YEAR: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%4.4d", t.year); - break; - case NOW_MONTH: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.month); - break; - case NOW_DAY: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.day); - break; - case NOW_HOUR: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.hour); - break; - case NOW_MINUTE: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "%2.2d", t.minute); - break; - default: - len = snprintf((char*) szBuf, sizeof(szBuf)/sizeof(uchar), "*invld eNow*"); - break; - } - - /* now create a string object out of it and hand that over to the var */ - *estr = es_newStrFromCStr((char*)szBuf, len); - - RETiRet; -} - - - -static inline es_str_t * -getSysVar(char *name) -{ - es_str_t *estr = NULL; - rsRetVal iRet = RS_RET_OK; - - if(!strcmp(name, "now")) { - CHKiRet(getNOW(NOW_NOW, &estr)); - } else if(!strcmp(name, "year")) { - CHKiRet(getNOW(NOW_YEAR, &estr)); - } else if(!strcmp(name, "month")) { - CHKiRet(getNOW(NOW_MONTH, &estr)); - } else if(!strcmp(name, "day")) { - CHKiRet(getNOW(NOW_DAY, &estr)); - } else if(!strcmp(name, "hour")) { - CHKiRet(getNOW(NOW_HOUR, &estr)); - } else if(!strcmp(name, "minute")) { - CHKiRet(getNOW(NOW_MINUTE, &estr)); - } else if(!strcmp(name, "myhostname")) { - char *hn = (char*)glbl.GetLocalHostName(); - estr = es_newStrFromCStr(hn, strlen(hn)); - } else { - ABORT_FINALIZE(RS_RET_SYSVAR_NOT_FOUND); - } -finalize_it: - if(iRet != RS_RET_OK) { - dbgprintf("getSysVar error iRet %d\n", iRet); - if(estr == NULL) - estr = es_newStrFromCStr("*ERROR*", sizeof("*ERROR*") - 1); - } - return estr; -} - - /* Process input() objects */ rsRetVal inputProcessCnf(struct cnfobj *o) @@ -377,6 +292,21 @@ finalize_it: extern int yylineno; void +parser_warnmsg(char *fmt, ...) +{ + va_list ap; + char errBuf[1024]; + + va_start(ap, fmt); + if(vsnprintf(errBuf, sizeof(errBuf), fmt, ap) == sizeof(errBuf)) + errBuf[sizeof(errBuf)-1] = '\0'; + errmsg.LogError(0, RS_RET_CONF_PARSE_WARNING, + "warning during parsing file %s, on or before line %d: %s", + cnfcurrfn, yylineno, errBuf); + va_end(ap); +} + +void parser_errmsg(char *fmt, ...) { va_list ap; @@ -399,6 +329,7 @@ yyerror(char *s) } void cnfDoObj(struct cnfobj *o) { + int bDestructObj = 1; int bChkUnuse = 1; dbgprintf("cnf:global:obj: "); @@ -407,12 +338,19 @@ void cnfDoObj(struct cnfobj *o) case CNFOBJ_GLOBAL: glblProcessCnf(o); break; + case CNFOBJ_MAINQ: + glblProcessMainQCnf(o); + bDestructObj = 0; + break; case CNFOBJ_MODULE: modulesProcessCnf(o); break; case CNFOBJ_INPUT: inputProcessCnf(o); break; + case CNFOBJ_LOOKUP_TABLE: + lookupProcessCnf(o); + break; case CNFOBJ_TPL: if(tplProcessCnf(o) != RS_RET_OK) parser_errmsg("error processing template object"); @@ -430,9 +368,11 @@ void cnfDoObj(struct cnfobj *o) o->objType); break; } - if(bChkUnuse) - nvlstChkUnused(o->nvlst); - cnfobjDestruct(o); + if(bDestructObj) { + if(bChkUnuse) + nvlstChkUnused(o->nvlst); + cnfobjDestruct(o); + } } void cnfDoScript(struct cnfstmt *script) @@ -468,30 +408,6 @@ void cnfDoBSDHost(char *ln) "solution (Block '%s')", ln); free(ln); } - -es_str_t* -cnfGetVar(char *name, void *usrptr) -{ - es_str_t *estr; - if(name[0] == '$') { - if(name[1] == '$') - estr = getSysVar(name+2); - else if(name[1] == '!') - estr = msgGetCEEVarNew((msg_t*) usrptr, name+2); - else - estr = msgGetMsgVarNew((msg_t*) usrptr, (uchar*)name+1); - } else { /* if this happens, we have a program logic error */ - estr = es_newStrFromCStr("err: var must start with $", - strlen("err: var must start with $")); - } - if(Debug) { - char *s; - s = es_str2cstr(estr, NULL); - dbgprintf("rainerscript: var '%s': '%s'\n", name, s); - free(s); - } - return estr; -} /*------------------------------ end interface to flex/bison parser ------------------------------*/ @@ -757,9 +673,18 @@ startInputModules(void) static inline rsRetVal activateMainQueue() { + struct cnfobj *mainqCnfObj; DEFiRet; + + mainqCnfObj = glbl.GetmainqCnfObj(); + DBGPRINTF("activateMainQueue: mainq cnf obj ptr is %p\n", mainqCnfObj); /* create message queue */ - CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"), NULL)) { + iRet = createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"), + (mainqCnfObj == NULL) ? NULL : mainqCnfObj->nvlst); + if(iRet == RS_RET_OK) { + iRet = startMainQueue(pMsgQueue); + } + if(iRet != RS_RET_OK) { /* no queue is fatal, we need to give up in that case... */ fprintf(stderr, "fatal error %d: could not create message queue - rsyslogd can not run!\n", iRet); FINALIZE; @@ -768,6 +693,7 @@ activateMainQueue() bHaveMainQueue = (ourConf->globals.mainQ.MainMsgQueType == QUEUETYPE_DIRECT) ? 0 : 1; DBGPRINTF("Main processing queue is initialized and running\n"); finalize_it: + glblDestructMainqCnfObj(); RETiRet; } @@ -821,6 +747,7 @@ activate(rsconf_t *cnf) tellModulesActivateConfig(); startInputModules(); CHKiRet(activateActions()); + CHKiRet(activateRulesetQueues()); CHKiRet(activateMainQueue()); /* finally let the inputs run... */ runInputModules(); @@ -1298,6 +1225,7 @@ ourConf = loadConf; // TODO: remove, once ourConf is gone! ABORT_FINALIZE(RS_RET_NO_ACTIONS); } tellLexEndParsing(); + DBGPRINTF("Number of actions in this configuration: %d\n", iActionNbr); rulesetOptimizeAll(loadConf); tellCoreConfigLoadDone(); diff --git a/runtime/rsconf.h b/runtime/rsconf.h index 484fec8c..894c0d12 100644 --- a/runtime/rsconf.h +++ b/runtime/rsconf.h @@ -25,6 +25,7 @@ #include "linkedlist.h" #include "queue.h" +#include "lookup.h" /* --- configuration objects (the plan is to have ALL upper layers in this file) --- */ @@ -143,6 +144,7 @@ struct rsconf_s { globals_t globals; defaults_t defaults; templates_t templates; + lookup_tables_t lu_tabs; outchannels_t och; actions_t actions; rulesets_t rulesets; diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index 047dfa9b..53c7855a 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -74,6 +74,7 @@ #include "prop.h" #include "ruleset.h" #include "parser.h" +#include "lookup.h" #include "strgen.h" #include "statsobj.h" #include "atomic.h" @@ -186,6 +187,8 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(strgenClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "rsconf"; CHKiRet(rsconfClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "lookup"; + CHKiRet(lookupClassInit()); /* dummy "classes" */ if(ppErrObj != NULL) *ppErrObj = "str"; diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index e62ba867..e0331da7 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -49,6 +49,7 @@ #define CONF_PROGNAME_BUFSIZE 16 #define CONF_HOSTNAME_BUFSIZE 32 #define CONF_PROP_BUFSIZE 16 /* should be close to sizeof(ptr) or lighly above it */ +#define CONF_IPARAMS_BUFSIZE 16 /* initial size of iparams array in wti (is automatically extended) */ #define CONF_MIN_SIZE_FOR_COMPRESS 60 /* config param: minimum message size to try compression. The smaller * the message, the less likely is any compression gain. We check for * gain before we submit the message. But to do so we still need to @@ -102,50 +103,6 @@ #define _PATH_CONSOLE "/dev/console" #endif -/* properties are now encoded as (tiny) integers. I do not use an enum as I would like - * to keep the memory footprint small (and thus cache hits high). - * rgerhards, 2009-06-26 - */ -typedef uintTiny propid_t; -#define PROP_INVALID 0 -#define PROP_MSG 1 -#define PROP_TIMESTAMP 2 -#define PROP_HOSTNAME 3 -#define PROP_SYSLOGTAG 4 -#define PROP_RAWMSG 5 -#define PROP_INPUTNAME 6 -#define PROP_FROMHOST 7 -#define PROP_FROMHOST_IP 8 -#define PROP_PRI 9 -#define PROP_PRI_TEXT 10 -#define PROP_IUT 11 -#define PROP_SYSLOGFACILITY 12 -#define PROP_SYSLOGFACILITY_TEXT 13 -#define PROP_SYSLOGSEVERITY 14 -#define PROP_SYSLOGSEVERITY_TEXT 15 -#define PROP_TIMEGENERATED 16 -#define PROP_PROGRAMNAME 17 -#define PROP_PROTOCOL_VERSION 18 -#define PROP_STRUCTURED_DATA 19 -#define PROP_APP_NAME 20 -#define PROP_PROCID 21 -#define PROP_MSGID 22 -#define PROP_PARSESUCCESS 23 -#define PROP_SYS_NOW 150 -#define PROP_SYS_YEAR 151 -#define PROP_SYS_MONTH 152 -#define PROP_SYS_DAY 153 -#define PROP_SYS_HOUR 154 -#define PROP_SYS_HHOUR 155 -#define PROP_SYS_QHOUR 156 -#define PROP_SYS_MINUTE 157 -#define PROP_SYS_MYHOSTNAME 158 -#define PROP_CEE 200 -#define PROP_CEE_ALL_JSON 201 -#define PROP_SYS_BOM 159 -#define PROP_SYS_UPTIME 160 -#define PROP_UUID 161 - /* The error codes below are orginally "borrowed" from * liblogging. As such, we reserve values up to -2999 @@ -399,7 +356,7 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_RULESET_EXISTS = -2306,/**< ruleset already exists */ RS_RET_DEPRECATED = -2307,/**< deprecated functionality is used */ RS_RET_DS_PROP_SEQ_ERR = -2308,/**< property sequence error deserializing object */ - RS_RET_TPL_INVLD_PROP = -2309,/**< property name error in template (unknown name) */ + RS_RET_INVLD_PROP = -2309,/**< property name error (unknown name) */ RS_RET_NO_RULEBASE = -2310,/**< mmnormalize: rulebase can not be found or otherwise invalid */ RS_RET_INVLD_MODE = -2311,/**< invalid mode specified in configuration */ RS_RET_INVLD_ANON_BITS = -2312,/**< mmanon: invalid number of bits to anonymize specified */ @@ -416,6 +373,16 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_CA_CERT_MISSING = -2329,/**< a CA cert is missing where one is required (e.g. TLS) */ RS_RET_CERT_MISSING = -2330,/**< a cert is missing where one is required (e.g. TLS) */ RS_RET_CERTKEY_MISSING = -2331,/**< a cert (private) key is missing where one is required (e.g. TLS) */ + RS_RET_STRUC_DATA_INVLD = -2349,/**< structured data is malformed */ + + /* up to 2350 reserved for 7.4 */ + RS_RET_QUEUE_CRY_DISK_ONLY = -2351,/**< crypto provider only supported for disk-associated queues */ + RS_RET_NO_DATA = -2352,/**< file has no data; more a state than a real error */ + RS_RET_RELP_AUTH_FAIL = -2353,/**< RELP peer authentication failed */ + RS_RET_ERR_UDPSEND = -2354,/**< sending msg via UDP failed */ + RS_RET_LAST_ERRREPORT = -2355,/**< module does not emit more error messages as limit is reached */ + RS_RET_READ_ERR = -2356,/**< read error occured (file i/o) */ + RS_RET_CONF_PARSE_WARNING = -2357,/**< warning parsing config file */ /* RainerScript error messages (range 1000.. 1999) */ RS_RET_SYSVAR_NOT_FOUND = 1001, /**< system variable could not be found (maybe misspelled) */ @@ -434,7 +401,12 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth * Be sure to call the to-be-returned variable always "iRet" and * the function finalizer always "finalize_it". */ -#define CHKiRet(code) if((iRet = code) != RS_RET_OK) goto finalize_it +#if HAVE_BUILTIN_EXCEPT +# define CHKiRet(code) if(__builtin_expect(((iRet = code) != RS_RET_OK), 0)) goto finalize_it +#else +# define CHKiRet(code) if((iRet = code) != RS_RET_OK) goto finalize_it +#endif + /* macro below is to be used if we need our own handling, eg for cleanup */ #define CHKiRet_Hdlr(code) if((iRet = code) != RS_RET_OK) /* macro below is to handle failing malloc/calloc/strdup... which we almost always handle in the same way... */ diff --git a/runtime/ruleset.c b/runtime/ruleset.c index 1afb4039..e217b3e0 100644 --- a/runtime/ruleset.c +++ b/runtime/ruleset.c @@ -48,6 +48,7 @@ #include "rainerscript.h" #include "srUtils.h" #include "modules.h" +#include "wti.h" #include "dirty.h" /* for main ruleset queue creation */ /* static data */ @@ -67,8 +68,8 @@ static struct cnfparamblk rspblk = }; /* forward definitions */ -static rsRetVal processBatch(batch_t *pBatch); -static rsRetVal scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active); +static rsRetVal processBatch(batch_t *pBatch, wti_t *pWti); +static void scriptExec(struct cnfstmt *root, msg_t *pMsg, wti_t *pWti); /* ---------- linked-list key handling functions (ruleset) ---------- */ @@ -160,250 +161,112 @@ finalize_it: RETiRet; } - -/* This function is similar to processBatch(), but works on a batch that - * contains rules from multiple rulesets. In this case, we can not push - * the whole batch through the ruleset. Instead, we examine it and - * partition it into sub-rulesets which we then push through the system. - * rgerhards, 2010-06-15 - */ -static inline rsRetVal -processBatchMultiRuleset(batch_t *pBatch) +/* driver to iterate over all rulesets */ +DEFFUNC_llExecFunc(doActivateRulesetQueues) { - ruleset_t *currRuleset; - batch_t snglRuleBatch; - int i; - int iStart; /* start index of partial batch */ - int iNew; /* index for new (temporary) batch */ - int bHaveUnprocessed; /* do we (still) have unprocessed entries? (loop term predicate) */ DEFiRet; - - do { - bHaveUnprocessed = 0; - /* search for first unprocessed element */ - for(iStart = 0 ; iStart < pBatch->nElem && pBatch->eltState[iStart] == BATCH_STATE_DISC ; ++iStart) - /* just search, no action */; - if(iStart == pBatch->nElem) - break; /* everything processed */ - - /* prepare temporary batch */ - CHKiRet(batchInit(&snglRuleBatch, pBatch->nElem)); - snglRuleBatch.pbShutdownImmediate = pBatch->pbShutdownImmediate; - currRuleset = batchElemGetRuleset(pBatch, iStart); - iNew = 0; - for(i = iStart ; i < pBatch->nElem ; ++i) { - if(batchElemGetRuleset(pBatch, i) == currRuleset) { - /* for performance reasons, we copy only those members that we actually need */ - snglRuleBatch.pElem[iNew].pMsg = pBatch->pElem[i].pMsg; - snglRuleBatch.eltState[iNew] = pBatch->eltState[i]; - ++iNew; - /* We indicate the element also as done, so it will not be processed again */ - pBatch->eltState[i] = BATCH_STATE_DISC; - } else { - bHaveUnprocessed = 1; - } - } - snglRuleBatch.nElem = iNew; /* was left just right by the for loop */ - batchSetSingleRuleset(&snglRuleBatch, 1); - /* process temp batch */ - processBatch(&snglRuleBatch); - batchFree(&snglRuleBatch); - } while(bHaveUnprocessed == 1); - -finalize_it: + ruleset_t* pThis = (ruleset_t*) pData; + dbgprintf("Activating Ruleset Queue[%p] for Ruleset %s\n", + pThis->pQueue, pThis->pszName); + if(pThis->pQueue != NULL) + startMainQueue(pThis->pQueue); RETiRet; } - -/* return a new "active" structure for the batch. Free with freeActive(). */ -static inline sbool *newActive(batch_t *pBatch) +/* activate all ruleset queues */ +rsRetVal +activateRulesetQueues() { - return malloc(sizeof(sbool) * batchNumMsgs(pBatch)); - -} -static inline void freeActive(sbool *active) { free(active); } + DEFiRet; + llExecFunc(&(runConf->rulesets.llRulesets), doActivateRulesetQueues, NULL); -/* for details, see scriptExec() header comment! */ -/* call action for all messages with filter on */ -static rsRetVal -execAct(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) -{ - DEFiRet; -dbgprintf("RRRR: execAct [%s]: batch of %d elements, active %p\n", modGetName(stmt->d.act->pMod), batchNumMsgs(pBatch), active); - pBatch->active = active; - stmt->d.act->submitToActQ(stmt->d.act, pBatch); RETiRet; } -static rsRetVal -execSet(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) + +static void +execAct(struct cnfstmt *stmt, msg_t *pMsg, wti_t *pWti) { - int i; - struct var result; - DEFiRet; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( pBatch->eltState[i] != BATCH_STATE_DISC - && (active == NULL || active[i])) { - cnfexprEval(stmt->d.s_set.expr, &result, pBatch->pElem[i].pMsg); - msgSetJSONFromVar(pBatch->pElem[i].pMsg, stmt->d.s_set.varname, - &result); - varDelete(&result); - } + if(stmt->d.act->bDisabled) { + DBGPRINTF("action %d died, do NOT execute\n", stmt->d.act->iActionNbr); + goto done; } - RETiRet; + + DBGPRINTF("executing action %d\n", stmt->d.act->iActionNbr); + stmt->d.act->submitToActQ(stmt->d.act, pWti, pMsg); +done: return; } -static rsRetVal -execUnset(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +static void +execSet(struct cnfstmt *stmt, msg_t *pMsg) { - int i; - DEFiRet; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( pBatch->eltState[i] != BATCH_STATE_DISC - && (active == NULL || active[i])) { - msgUnsetJSON(pBatch->pElem[i].pMsg, stmt->d.s_unset.varname); - } - } - RETiRet; + struct var result; + cnfexprEval(stmt->d.s_set.expr, &result, pMsg); + msgSetJSONFromVar(pMsg, stmt->d.s_set.varname, &result); + varDelete(&result); } -/* for details, see scriptExec() header comment! */ -/* "stop" simply discards the filtered items - it's just a (hopefully more intuitive - * shortcut for users. - */ -static rsRetVal -execStop(batch_t *pBatch, sbool *active) +static void +execUnset(struct cnfstmt *stmt, msg_t *pMsg) { - int i; - DEFiRet; - for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pBatch->pbShutdownImmediate) ; ++i) { - if( pBatch->eltState[i] != BATCH_STATE_DISC - && (active == NULL || active[i])) { - pBatch->eltState[i] = BATCH_STATE_DISC; - } - } - RETiRet; + msgDelJSON(pMsg, stmt->d.s_unset.varname); } + static rsRetVal -execCall(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +execCall(struct cnfstmt *stmt, msg_t *pMsg, wti_t *pWti) { - msg_t *pMsg; - int i; DEFiRet; if(stmt->d.s_call.ruleset == NULL) { - scriptExec(stmt->d.s_call.stmt, pBatch, active); + scriptExec(stmt->d.s_call.stmt, pMsg, pWti); } else { - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - CHKmalloc(pMsg = MsgDup((msg_t*) pBatch->pElem[i].pMsg)); - DBGPRINTF("CALL: forwarding message %d to async ruleset %p\n", - i, stmt->d.s_call.ruleset->pQueue); - MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); - MsgSetRuleset(pMsg, stmt->d.s_call.ruleset); - /* Note: we intentionally use submitMsg2() here, as we process messages - * that were already run through the rate-limiter. - */ - submitMsg2(pMsg); - } + CHKmalloc(pMsg = MsgDup((msg_t*) pMsg)); + DBGPRINTF("CALL: forwarding message to async ruleset %p\n", + stmt->d.s_call.ruleset->pQueue); + MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); + MsgSetRuleset(pMsg, stmt->d.s_call.ruleset); + /* Note: we intentionally use submitMsg2() here, as we process messages + * that were already run through the rate-limiter. + */ + submitMsg2(pMsg); } finalize_it: RETiRet; } -/* for details, see scriptExec() header comment! */ -// save current filter, evaluate new one -// perform then (if any message) -// if ELSE given: -// set new filter, inverted -// perform else (if any messages) -static rsRetVal -execIf(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +static void +execIf(struct cnfstmt *stmt, msg_t *pMsg, wti_t *pWti) { - sbool *newAct; - int i; sbool bRet; - sbool allInactive = 1; - DEFiRet; - newAct = newActive(pBatch); - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(*(pBatch->pbShutdownImmediate)) - FINALIZE; - if(pBatch->eltState[i] == BATCH_STATE_DISC) - continue; /* will be ignored in any case */ - if(active == NULL || active[i]) { - bRet = cnfexprEvalBool(stmt->d.s_if.expr, pBatch->pElem[i].pMsg); - allInactive = 0; - } else - bRet = 0; - newAct[i] = bRet; - DBGPRINTF("batch: item %d: expr eval: %d\n", i, bRet); - } - - if(allInactive) { - DBGPRINTF("execIf: all batch elements are inactive, holding execution\n"); - freeActive(newAct); - FINALIZE; - } - - if(stmt->d.s_if.t_then != NULL) { - scriptExec(stmt->d.s_if.t_then, pBatch, newAct); - } - if(stmt->d.s_if.t_else != NULL) { - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(*(pBatch->pbShutdownImmediate)) - FINALIZE; - if(pBatch->eltState[i] != BATCH_STATE_DISC - && (active == NULL || active[i])) - newAct[i] = !newAct[i]; - } - scriptExec(stmt->d.s_if.t_else, pBatch, newAct); + bRet = cnfexprEvalBool(stmt->d.s_if.expr, pMsg); + DBGPRINTF("if condition result is %d\n", bRet); + if(bRet) { + if(stmt->d.s_if.t_then != NULL) + scriptExec(stmt->d.s_if.t_then, pMsg, pWti); + } else { + if(stmt->d.s_if.t_else != NULL) + scriptExec(stmt->d.s_if.t_else, pMsg, pWti); } - freeActive(newAct); -finalize_it: - RETiRet; } -/* for details, see scriptExec() header comment! */ static void -execPRIFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +execPRIFILT(struct cnfstmt *stmt, msg_t *pMsg, wti_t *pWti) { - sbool *newAct; - msg_t *pMsg; int bRet; - int i; - newAct = newActive(pBatch); - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(*(pBatch->pbShutdownImmediate)) - return; - if(pBatch->eltState[i] == BATCH_STATE_DISC) - continue; /* will be ignored in any case */ - pMsg = pBatch->pElem[i].pMsg; - if(active == NULL || active[i]) { - if( (stmt->d.s_prifilt.pmask[pMsg->iFacility] == TABLE_NOPRI) || - ((stmt->d.s_prifilt.pmask[pMsg->iFacility] - & (1<<pMsg->iSeverity)) == 0) ) - bRet = 0; - else - bRet = 1; - } else - bRet = 0; - newAct[i] = bRet; - DBGPRINTF("batch: item %d PRIFILT %d\n", i, newAct[i]); - } - - if(stmt->d.s_prifilt.t_then != NULL) { - scriptExec(stmt->d.s_prifilt.t_then, pBatch, newAct); - } - if(stmt->d.s_prifilt.t_else != NULL) { - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(*(pBatch->pbShutdownImmediate)) - return; - if(pBatch->eltState[i] != BATCH_STATE_DISC - && (active == NULL || active[i])) - newAct[i] = !newAct[i]; - } - scriptExec(stmt->d.s_prifilt.t_else, pBatch, newAct); + if( (stmt->d.s_prifilt.pmask[pMsg->iFacility] == TABLE_NOPRI) || + ((stmt->d.s_prifilt.pmask[pMsg->iFacility] + & (1<<pMsg->iSeverity)) == 0) ) + bRet = 0; + else + bRet = 1; + + DBGPRINTF("PRIFILT condition result is %d\n", bRet); + if(bRet) { + if(stmt->d.s_prifilt.t_then != NULL) + scriptExec(stmt->d.s_prifilt.t_then, pMsg, pWti); + } else { + if(stmt->d.s_prifilt.t_else != NULL) + scriptExec(stmt->d.s_prifilt.t_else, pMsg, pWti); } - freeActive(newAct); } @@ -416,12 +279,11 @@ evalPROPFILT(struct cnfstmt *stmt, msg_t *pMsg) int bRet = 0; rs_size_t propLen; - if(stmt->d.s_propfilt.propID == PROP_INVALID) + if(stmt->d.s_propfilt.prop.id == PROP_INVALID) goto done; - pszPropVal = MsgGetProp(pMsg, NULL, stmt->d.s_propfilt.propID, - stmt->d.s_propfilt.propName, &propLen, - &pbMustBeFreed, NULL); + pszPropVal = MsgGetProp(pMsg, NULL, &stmt->d.s_propfilt.prop, + &propLen, &pbMustBeFreed, NULL); /* Now do the compares (short list currently ;)) */ switch(stmt->d.s_propfilt.operation ) { @@ -465,15 +327,18 @@ evalPROPFILT(struct cnfstmt *stmt, msg_t *pMsg) bRet = (bRet == 1) ? 0 : 1; if(Debug) { - char *cstr; - if(stmt->d.s_propfilt.propID == PROP_CEE) { - cstr = es_str2cstr(stmt->d.s_propfilt.propName, NULL); + if(stmt->d.s_propfilt.prop.id == PROP_CEE) { DBGPRINTF("Filter: check for CEE property '%s' (value '%s') ", - cstr, pszPropVal); - free(cstr); + stmt->d.s_propfilt.prop.name, pszPropVal); + } else if(stmt->d.s_propfilt.prop.id == PROP_LOCAL_VAR) { + DBGPRINTF("Filter: check for local var '%s' (value '%s') ", + stmt->d.s_propfilt.prop.name, pszPropVal); + } else if(stmt->d.s_propfilt.prop.id == PROP_GLOBAL_VAR) { + DBGPRINTF("Filter: check for global var '%s' (value '%s') ", + stmt->d.s_propfilt.prop.name, pszPropVal); } else { DBGPRINTF("Filter: check for property '%s' (value '%s') ", - propIDToName(stmt->d.s_propfilt.propID), pszPropVal); + propIDToName(stmt->d.s_propfilt.prop.id), pszPropVal); } if(stmt->d.s_propfilt.isNegated) DBGPRINTF("NOT "); @@ -496,79 +361,63 @@ done: return bRet; } -/* for details, see scriptExec() header comment! */ static void -execPROPFILT(struct cnfstmt *stmt, batch_t *pBatch, sbool *active) +execPROPFILT(struct cnfstmt *stmt, msg_t *pMsg, wti_t *pWti) { - sbool *thenAct; sbool bRet; - int i; - thenAct = newActive(pBatch); - for(i = 0 ; i < batchNumMsgs(pBatch) ; ++i) { - if(*(pBatch->pbShutdownImmediate)) - return; - if(pBatch->eltState[i] == BATCH_STATE_DISC) - continue; /* will be ignored in any case */ - if(active == NULL || active[i]) { - bRet = evalPROPFILT(stmt, pBatch->pElem[i].pMsg); - } else - bRet = 0; - thenAct[i] = bRet; - DBGPRINTF("batch: item %d PROPFILT %d\n", i, thenAct[i]); - } - scriptExec(stmt->d.s_propfilt.t_then, pBatch, thenAct); - freeActive(thenAct); + bRet = evalPROPFILT(stmt, pMsg); + DBGPRINTF("PROPFILT condition result is %d\n", bRet); + if(bRet) + scriptExec(stmt->d.s_propfilt.t_then, pMsg, pWti); } /* The rainerscript execution engine. It is debatable if that would be better * contained in grammer/rainerscript.c, HOWEVER, that file focusses primarily * on the parsing and object creation part. So as an actual executor, it is * better suited here. - * param active: if NULL, all messages are active (to be processed), if non-null - * this is an array of the same size as the batch. If 1, the message - * is to be processed, otherwise not. - * NOTE: this function must receive batches which contain a single ruleset ONLY! * rgerhards, 2012-09-04 */ -static rsRetVal -scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active) +static void +scriptExec(struct cnfstmt *root, msg_t *pMsg, wti_t *pWti) { - DEFiRet; struct cnfstmt *stmt; for(stmt = root ; stmt != NULL ; stmt = stmt->next) { + if(*pWti->pbShutdownImmediate) { + DBGPRINTF("scriptExec: ShutdownImmediate set, " + "force terminating\n"); + goto done; + } if(Debug) { - dbgprintf("scriptExec: batch of %d elements, active %p, active[0]:%d\n", - batchNumMsgs(pBatch), active, (active == NULL ? 1 : active[0])); cnfstmtPrintOnly(stmt, 2, 0); } switch(stmt->nodetype) { case S_NOP: break; case S_STOP: - execStop(pBatch, active); + goto done; break; case S_ACT: - execAct(stmt, pBatch, active); + execAct(stmt, pMsg, pWti); break; case S_SET: - execSet(stmt, pBatch, active); + execSet(stmt, pMsg); break; case S_UNSET: - execUnset(stmt, pBatch, active); + execUnset(stmt, pMsg); break; case S_CALL: - execCall(stmt, pBatch, active); + execCall(stmt, pMsg, pWti); break; case S_IF: - execIf(stmt, pBatch, active); + execIf(stmt, pMsg, pWti); break; case S_PRIFILT: - execPRIFILT(stmt, pBatch, active); + execPRIFILT(stmt, pMsg, pWti); break; case S_PROPFILT: - execPROPFILT(stmt, pBatch, active); + execPROPFILT(stmt, pMsg, pWti); break; default: dbgprintf("error: unknown stmt type %u during exec\n", @@ -576,36 +425,42 @@ scriptExec(struct cnfstmt *root, batch_t *pBatch, sbool *active) break; } } - RETiRet; +done: return; } /* Process (consume) a batch of messages. Calls the actions configured. - * If the whole batch uses a singel ruleset, we can process the batch as - * a whole. Otherwise, we need to process it slower, on a message-by-message - * basis (what can be optimized to a per-ruleset basis) - * rgerhards, 2005-10-13 + * This is called by MAIN queues. */ static rsRetVal -processBatch(batch_t *pBatch) +processBatch(batch_t *pBatch, wti_t *pWti) { - ruleset_t *pThis; + int i; + msg_t *pMsg; + ruleset_t *pRuleset; DEFiRet; - assert(pBatch != NULL); - - DBGPRINTF("processBatch: batch of %d elements must be processed\n", pBatch->nElem); - if(pBatch->bSingleRuleset) { - pThis = batchGetRuleset(pBatch); - if(pThis == NULL) - pThis = ourConf->rulesets.pDflt; - ISOBJ_TYPE_assert(pThis, ruleset); - CHKiRet(scriptExec(pThis->root, pBatch, NULL)); - } else { - CHKiRet(processBatchMultiRuleset(pBatch)); + + DBGPRINTF("processBATCH: batch of %d elements must be processed\n", pBatch->nElem); + + wtiResetExecState(pWti, pBatch); + + /* execution phase */ + for(i = 0 ; i < batchNumMsgs(pBatch) && !*(pWti->pbShutdownImmediate) ; ++i) { + pMsg = pBatch->pElem[i].pMsg; + DBGPRINTF("processBATCH: next msg %d: %.128s\n", i, pMsg->pszRawMsg); + pRuleset = (pMsg->pRuleset == NULL) ? ourConf->rulesets.pDflt : pMsg->pRuleset; + scriptExec(pRuleset->root, pMsg, pWti); + // TODO: think if we need a return state of scriptExec - most probably + // the answer is "no", as we need to process the batch in any case! + // TODO: we must refactor this! flag messages as committed + batchSetElemState(pBatch, i, BATCH_STATE_COMM); } -finalize_it: - DBGPRINTF("ruleset.ProcessMsg() returns %d\n", iRet); + /* commit phase */ + dbgprintf("END batch execution phase, entering to commit phase\n"); + actionCommitAllDirect(pWti); + + DBGPRINTF("processBATCH: batch of %d elements has been processed\n", pBatch->nElem); RETiRet; } @@ -952,7 +807,6 @@ rsRetVal rulesetProcessCnf(struct cnfobj *o) { struct cnfparamvals *pvals; - struct cnfparamvals *queueParams; rsRetVal localRet; uchar *rsName = NULL; uchar *parserName; @@ -998,11 +852,10 @@ rulesetProcessCnf(struct cnfobj *o) } /* pick up ruleset queue parameters */ - qqueueDoCnfParams(o->nvlst, &queueParams); - if(queueCnfParamsSet(queueParams)) { + if(queueCnfParamsSet(o->nvlst)) { rsname = (pRuleset->pszName == NULL) ? (uchar*) "[ruleset]" : pRuleset->pszName; DBGPRINTF("adding a ruleset-specific \"main\" queue for ruleset '%s'\n", rsname); - CHKiRet(createMainQueue(&pRuleset->pQueue, rsname, queueParams)); + CHKiRet(createMainQueue(&pRuleset->pQueue, rsname, o->nvlst)); } finalize_it: diff --git a/runtime/ruleset.h b/runtime/ruleset.h index 64fe92ff..d3dfd664 100644 --- a/runtime/ruleset.h +++ b/runtime/ruleset.h @@ -2,7 +2,7 @@ * * This implements rulesets within rsyslog. * - * Copyright 2009-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -46,7 +46,7 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ rsRetVal (*Destruct)(ruleset_t **ppThis); rsRetVal (*DestructAllActions)(rsconf_t *conf); rsRetVal (*SetName)(ruleset_t *pThis, uchar *pszName); - rsRetVal (*ProcessBatch)(batch_t*); + rsRetVal (*ProcessBatch)(batch_t*, wti_t *); rsRetVal (*GetRuleset)(rsconf_t *conf, ruleset_t **ppThis, uchar*); rsRetVal (*SetDefaultRuleset)(rsconf_t *conf, uchar*); rsRetVal (*SetCurrRuleset)(rsconf_t *conf, uchar*); @@ -64,8 +64,9 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ /* AddRule() removed */ /*TODO:REMOVE*/rsRetVal (*IterateAllActions)(rsconf_t *conf, rsRetVal (*pFunc)(void*, void*), void* pParam); void (*AddScript)(ruleset_t *pThis, struct cnfstmt *script); + /* v8: changed processBatch interface */ ENDinterface(ruleset) -#define rulesetCURR_IF_VERSION 7 /* increment whenever you change the interface structure! */ +#define rulesetCURR_IF_VERSION 8 /* increment whenever you change the interface structure! */ /* prototypes */ @@ -104,6 +105,7 @@ rulesetHasQueue(ruleset_t *pRuleset) rsRetVal rulesetGetRuleset(rsconf_t *conf, ruleset_t **ppRuleset, uchar *pszName); rsRetVal rulesetOptimizeAll(rsconf_t *conf); rsRetVal rulesetProcessCnf(struct cnfobj *o); +rsRetVal activateRulesetQueues(void); /* Set a current rule set to already-known pointer */ static inline void diff --git a/runtime/srutils.c b/runtime/srutils.c index 6a509b4a..8eb2459c 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -86,6 +86,7 @@ syslogName_t syslogFacNames[] = { {"mark", LOG_MARK}, /* INTERNAL */ {"news", LOG_NEWS}, {"security", LOG_AUTH}, /* DEPRECATED */ + {"bsd_security", (13<<3) }, /* BSD-specific, unfortunatly with duplicate name... */ {"syslog", LOG_SYSLOG}, {"user", LOG_USER}, {"uucp", LOG_UUCP}, @@ -95,6 +96,7 @@ syslogName_t syslogFacNames[] = { #if defined(LOG_AUDIT) {"audit", LOG_AUDIT}, #endif + {"console", (14 << 3)}, /* BSD-specific priority */ {"local0", LOG_LOCAL0}, {"local1", LOG_LOCAL1}, {"local2", LOG_LOCAL2}, diff --git a/runtime/statsobj.c b/runtime/statsobj.c index 25275616..edac7d48 100644 --- a/runtime/statsobj.c +++ b/runtime/statsobj.c @@ -142,7 +142,7 @@ finalize_it: * is called. */ static rsRetVal -addCounter(statsobj_t *pThis, uchar *ctrName, statsCtrType_t ctrType, void *pCtr) +addCounter(statsobj_t *pThis, uchar *ctrName, statsCtrType_t ctrType, int8_t flags, void *pCtr) { ctr_t *ctr; DEFiRet; @@ -151,6 +151,7 @@ addCounter(statsobj_t *pThis, uchar *ctrName, statsCtrType_t ctrType, void *pCtr ctr->next = NULL; ctr->prev = NULL; CHKmalloc(ctr->name = ustrdup(ctrName)); + ctr->flags = flags; ctr->ctrType = ctrType; switch(ctrType) { case ctrType_IntCtr: @@ -166,9 +167,24 @@ finalize_it: RETiRet; } +static inline void +resetResettableCtr(ctr_t *pCtr, int8_t bResetCtrs) +{ + if(bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)) { + switch(pCtr->ctrType) { + case ctrType_IntCtr: + *(pCtr->val.pIntCtr) = 0; + break; + case ctrType_Int: + *(pCtr->val.pInt) = 0; + break; + } + } +} + /* get all the object's countes together as CEE. */ static rsRetVal -getStatsLineCEE(statsobj_t *pThis, cstr_t **ppcstr, int cee_cookie) +getStatsLineCEE(statsobj_t *pThis, cstr_t **ppcstr, int cee_cookie, int8_t bResetCtrs) { cstr_t *pcstr; ctr_t *pCtr; @@ -209,7 +225,7 @@ getStatsLineCEE(statsobj_t *pThis, cstr_t **ppcstr, int cee_cookie) } else { cstrAppendChar(pcstr, '}'); } - + resetResettableCtr(pCtr, bResetCtrs); } pthread_mutex_unlock(&pThis->mutCtr); @@ -223,7 +239,7 @@ finalize_it: /* get all the object's countes together with object name as one line. */ static rsRetVal -getStatsLine(statsobj_t *pThis, cstr_t **ppcstr) +getStatsLine(statsobj_t *pThis, cstr_t **ppcstr, int8_t bResetCtrs) { cstr_t *pcstr; ctr_t *pCtr; @@ -247,6 +263,7 @@ getStatsLine(statsobj_t *pThis, cstr_t **ppcstr) break; } cstrAppendChar(pcstr, ' '); + resetResettableCtr(pCtr, bResetCtrs); } pthread_mutex_unlock(&pThis->mutCtr); @@ -265,7 +282,7 @@ finalize_it: * line. If the callback reports an error, processing is stopped. */ static rsRetVal -getAllStatsLines(rsRetVal(*cb)(void*, cstr_t*), void *usrptr, statsFmtType_t fmt) +getAllStatsLines(rsRetVal(*cb)(void*, cstr_t*), void *usrptr, statsFmtType_t fmt, int8_t bResetCtrs) { statsobj_t *o; cstr_t *cstr; @@ -274,13 +291,13 @@ getAllStatsLines(rsRetVal(*cb)(void*, cstr_t*), void *usrptr, statsFmtType_t fmt for(o = objRoot ; o != NULL ; o = o->next) { switch(fmt) { case statsFmt_Legacy: - CHKiRet(getStatsLine(o, &cstr)); + CHKiRet(getStatsLine(o, &cstr, bResetCtrs)); break; case statsFmt_CEE: - CHKiRet(getStatsLineCEE(o, &cstr, 1)); + CHKiRet(getStatsLineCEE(o, &cstr, 1, bResetCtrs)); break; case statsFmt_JSON: - CHKiRet(getStatsLineCEE(o, &cstr, 0)); + CHKiRet(getStatsLineCEE(o, &cstr, 0, bResetCtrs)); break; } CHKiRet(cb(usrptr, cstr)); @@ -348,7 +365,7 @@ CODESTARTobjQueryInterface(statsobj) pIf->Destruct = statsobjDestruct; pIf->DebugPrint = statsobjDebugPrint; pIf->SetName = setName; - pIf->GetStatsLine = getStatsLine; + //pIf->GetStatsLine = getStatsLine; pIf->GetAllStatsLines = getAllStatsLines; pIf->AddCounter = addCounter; pIf->EnableStats = enableStats; diff --git a/runtime/statsobj.h b/runtime/statsobj.h index 14b33215..d56485de 100644 --- a/runtime/statsobj.h +++ b/runtime/statsobj.h @@ -50,6 +50,9 @@ typedef enum statsFmtType_e { statsFmt_CEE } statsFmtType_t; +/* counter flags */ +#define CTR_FLAG_NONE 0 +#define CTR_FLAG_RESETTABLE 1 /* helper entity, the counter */ typedef struct ctr_s { @@ -59,6 +62,7 @@ typedef struct ctr_s { intctr_t *pIntCtr; int *pInt; } val; + int8_t flags; struct ctr_s *next, *prev; } ctr_t; @@ -82,15 +86,17 @@ BEGINinterface(statsobj) /* name must also be changed in ENDinterface macro! */ rsRetVal (*ConstructFinalize)(statsobj_t *pThis); rsRetVal (*Destruct)(statsobj_t **ppThis); rsRetVal (*SetName)(statsobj_t *pThis, uchar *name); - rsRetVal (*GetStatsLine)(statsobj_t *pThis, cstr_t **ppcstr); - rsRetVal (*GetAllStatsLines)(rsRetVal(*cb)(void*, cstr_t*), void *usrptr, statsFmtType_t fmt); - rsRetVal (*AddCounter)(statsobj_t *pThis, uchar *ctrName, statsCtrType_t ctrType, void *pCtr); + //rsRetVal (*GetStatsLine)(statsobj_t *pThis, cstr_t **ppcstr); + rsRetVal (*GetAllStatsLines)(rsRetVal(*cb)(void*, cstr_t*), void *usrptr, statsFmtType_t fmt, int8_t bResetCtr); + rsRetVal (*AddCounter)(statsobj_t *pThis, uchar *ctrName, statsCtrType_t ctrType, int8_t flags, void *pCtr); rsRetVal (*EnableStats)(void); ENDinterface(statsobj) -#define statsobjCURR_IF_VERSION 10 /* increment whenever you change the interface structure! */ +#define statsobjCURR_IF_VERSION 11 /* increment whenever you change the interface structure! */ /* Changes * v2-v9 rserved for future use in "older" version branches * v10, 2012-04-01: GetAllStatsLines got fmt parameter + * v11, 2013-09-07: - add "flags" to AddCounter API + * - GetAllStatsLines got parameter telling if ctrs shall be reset */ diff --git a/runtime/stream.c b/runtime/stream.c index 94fc0ca7..b35d6a11 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -256,7 +256,9 @@ doPhysOpen(strm_t *pThis) if(pThis->cryprov != NULL) { CHKiRet(pThis->cryprov->OnFileOpen(pThis->cryprovData, - pThis->pszCurrFName, &pThis->cryprovFileData)); + pThis->pszCurrFName, &pThis->cryprovFileData, + (pThis->tOperationsMode == STREAMMODE_READ) ? 'r' : 'w')); + pThis->cryprov->SetDeleteOnClose(pThis->cryprovFileData, pThis->bDeleteOnClose); } finalize_it: RETiRet; @@ -404,6 +406,12 @@ static rsRetVal strmCloseFile(strm_t *pThis) } } + /* if we have a signature provider, we must make sure that the crypto + * state files are opened and proper close processing happens. */ + if(pThis->cryprov != NULL && pThis->fd == -1) { + strmOpenFile(pThis); + } + /* the file may already be closed (or never have opened), so guard * against this. -- rgerhards, 2010-03-19 */ @@ -550,11 +558,14 @@ finalize_it: * rgerhards, 2008-02-13 */ static rsRetVal -strmReadBuf(strm_t *pThis) +strmReadBuf(strm_t *pThis, int *padBytes) { DEFiRet; int bRun; long iLenRead; + size_t actualDataLen; + size_t toRead; + ssize_t bytesLeft; ISOBJ_TYPE_assert(pThis, strm); /* We need to try read at least twice because we may run into EOF and need to switch files. */ @@ -565,13 +576,35 @@ strmReadBuf(strm_t *pThis) * rgerhards, 2008-02-13 */ CHKiRet(strmOpenFile(pThis)); - iLenRead = read(pThis->fd, pThis->pIOBuf, pThis->sIOBufSize); + if(pThis->cryprov == NULL) { + toRead = pThis->sIOBufSize; + } else { + CHKiRet(pThis->cryprov->GetBytesLeftInBlock(pThis->cryprovFileData, &bytesLeft)); + if(bytesLeft == -1 || bytesLeft > (ssize_t) pThis->sIOBufSize) { + toRead = pThis->sIOBufSize; + } else { + toRead = (size_t) bytesLeft; + } + } + iLenRead = read(pThis->fd, pThis->pIOBuf, toRead); DBGOPRINT((obj_t*) pThis, "file %d read %ld bytes\n", pThis->fd, iLenRead); + /* end crypto */ if(iLenRead == 0) { CHKiRet(strmHandleEOF(pThis)); } else if(iLenRead < 0) ABORT_FINALIZE(RS_RET_IO_ERROR); else { /* good read */ + /* here we place our crypto interface */ + if(pThis->cryprov != NULL) { + actualDataLen = iLenRead; + pThis->cryprov->Decrypt(pThis->cryprovFileData, pThis->pIOBuf, &actualDataLen); + *padBytes = iLenRead - actualDataLen; + iLenRead = actualDataLen; + DBGOPRINT((obj_t*) pThis, "encrypted file %d pad bytes %d, actual " + "data %ld\n", pThis->fd, *padBytes, iLenRead); + } else { + *padBytes = 0; + } pThis->iBufPtrMax = iLenRead; bRun = 0; /* exit loop */ } @@ -593,6 +626,7 @@ finalize_it: */ static rsRetVal strmReadChar(strm_t *pThis, uchar *pC) { + int padBytes = 0; /* in crypto mode, we may have some padding (non-data) bytes */ DEFiRet; ASSERT(pThis != NULL); @@ -608,8 +642,9 @@ static rsRetVal strmReadChar(strm_t *pThis, uchar *pC) /* do we need to obtain a new buffer? */ if(pThis->iBufPtr >= pThis->iBufPtrMax) { - CHKiRet(strmReadBuf(pThis)); + CHKiRet(strmReadBuf(pThis, &padBytes)); } + pThis->iCurrOffs += padBytes; /* if we reach this point, we have data available in the buffer */ @@ -645,7 +680,7 @@ static rsRetVal strmUnreadChar(strm_t *pThis, uchar c) * destruction of the returned CStr object! -- dlang 2010-12-13 */ static rsRetVal -strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) +strmReadLine(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF) { /* mode = 0 single line mode (equivalent to ReadLine) * mode = 1 LFLF mode (paragraph, blank line between entries) @@ -655,6 +690,7 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) uchar c; uchar finished; rsRetVal readCharRet; + sbool bPrevWasNL; DEFiRet; ASSERT(pThis != NULL); @@ -680,18 +716,25 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) CHKiRet(cstrFinalize(*ppCStr)); } else if(mode == 1) { finished=0; + bPrevWasNL = 0; while(finished == 0){ if(c != '\n') { CHKiRet(cstrAppendChar(*ppCStr, c)); CHKiRet(strmReadChar(pThis, &c)); + bPrevWasNL = 0; } else { if ((((*ppCStr)->iStrLen) > 0) ){ - if ((*ppCStr)->pBuf[(*ppCStr)->iStrLen -1 ] == '\n'){ - rsCStrTruncate(*ppCStr,1); /* remove the prior newline */ + if(bPrevWasNL) { + rsCStrTruncate(*ppCStr, (bEscapeLF) ? 4 : 1); /* remove the prior newline */ finished=1; } else { - CHKiRet(cstrAppendChar(*ppCStr, c)); + if(bEscapeLF) { + CHKiRet(rsCStrAppendStrWithLen(*ppCStr, (uchar*)"#012", sizeof("#012")-1)); + } else { + CHKiRet(cstrAppendChar(*ppCStr, c)); + } CHKiRet(strmReadChar(pThis, &c)); + bPrevWasNL = 1; } } else { finished=1; /* this is a blank line, a \n with nothing since the last complete record */ @@ -702,6 +745,7 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) } else if(mode == 2) { /* indented follow-up lines */ finished=0; + bPrevWasNL = 0; while(finished == 0){ if ((*ppCStr)->iStrLen == 0){ if(c != '\n') { @@ -712,22 +756,31 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, int mode) finished=1; /* this is a blank line, a \n with nothing since the last complete record */ } } else { - if ((*ppCStr)->pBuf[(*ppCStr)->iStrLen -1 ] != '\n'){ - /* not the first character after a newline, add it to the buffer */ - CHKiRet(cstrAppendChar(*ppCStr, c)); - CHKiRet(strmReadChar(pThis, &c)); - } else { + if(bPrevWasNL) { if ((c == ' ') || (c == '\t')){ CHKiRet(cstrAppendChar(*ppCStr, c)); CHKiRet(strmReadChar(pThis, &c)); + bPrevWasNL = 0; } else { /* clean things up by putting the character we just read back into * the input buffer and removing the LF character that is currently at the * end of the output string */ CHKiRet(strmUnreadChar(pThis, c)); - rsCStrTruncate(*ppCStr,1); + rsCStrTruncate(*ppCStr, (bEscapeLF) ? 4 : 1); finished=1; } + } else { /* not the first character after a newline, add it to the buffer */ + if(c == '\n') { + bPrevWasNL = 1; + if(bEscapeLF) { + CHKiRet(rsCStrAppendStrWithLen(*ppCStr, (uchar*)"#012", sizeof("#012")-1)); + } else { + CHKiRet(cstrAppendChar(*ppCStr, c)); + } + } else { + CHKiRet(cstrAppendChar(*ppCStr, c)); + } + CHKiRet(strmReadChar(pThis, &c)); } } } @@ -1454,6 +1507,8 @@ strmMultiFileSeek(strm_t *pThis, int FNum, off64_t offs, off64_t *bytesDel) "deleting '%s' (%lld bytes)\n", pThis->iCurrFNum, FNum, pThis->pszCurrFName, (long long) *bytesDel); unlink((char*)pThis->pszCurrFName); + if(pThis->cryprov != NULL) + pThis->cryprov->DeleteStateFiles(pThis->pszCurrFName); free(pThis->pszCurrFName); pThis->pszCurrFName = NULL; pThis->iCurrFNum = FNum; @@ -1467,17 +1522,31 @@ finalize_it: } - /* seek to current offset. This is primarily a helper to readjust the OS file * pointer after a strm object has been deserialized. */ static rsRetVal strmSeekCurrOffs(strm_t *pThis) { + off64_t targetOffs; + uchar c; DEFiRet; ISOBJ_TYPE_assert(pThis, strm); - iRet = strmSeek(pThis, pThis->iCurrOffs); + if(pThis->cryprov == NULL || pThis->tOperationsMode != STREAMMODE_READ) { + iRet = strmSeek(pThis, pThis->iCurrOffs); + FINALIZE; + } + + /* As the cryprov may use CBC or similiar things, we need to read skip data */ + targetOffs = pThis->iCurrOffs; + pThis->iCurrOffs = 0; + DBGOPRINT((obj_t*) pThis, "encrypted, doing skip read of %lld bytes\n", + (long long) targetOffs); + while(targetOffs != pThis->iCurrOffs) { + CHKiRet(strmReadChar(pThis, &c)); + } +finalize_it: RETiRet; } @@ -1604,7 +1673,6 @@ finalize_it: /* property set methods */ /* simple ones first */ -DEFpropSetMeth(strm, bDeleteOnClose, int) DEFpropSetMeth(strm, iMaxFileSize, int64) DEFpropSetMeth(strm, iFileNumDigits, int) DEFpropSetMeth(strm, tOperationsMode, int) @@ -1620,6 +1688,15 @@ DEFpropSetMeth(strm, pszSizeLimitCmd, uchar*) DEFpropSetMeth(strm, cryprov, cryprov_if_t*) DEFpropSetMeth(strm, cryprovData, void*) +static rsRetVal strmSetbDeleteOnClose(strm_t *pThis, int val) +{ + pThis->bDeleteOnClose = val; + if(pThis->cryprov != NULL) { + pThis->cryprov->SetDeleteOnClose(pThis->cryprovFileData, pThis->bDeleteOnClose); + } + return RS_RET_OK; +} + static rsRetVal strmSetiMaxFiles(strm_t *pThis, int iNewVal) { pThis->iMaxFiles = iNewVal; diff --git a/runtime/stream.h b/runtime/stream.h index 4f4a4301..092d3226 100644 --- a/runtime/stream.h +++ b/runtime/stream.h @@ -66,6 +66,7 @@ #define STREAM_H_INCLUDED #include <pthread.h> +#include <stdint.h> #include "obj-types.h" #include "glbl.h" #include "stream.h" @@ -188,7 +189,7 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ INTERFACEpropSetMeth(strm, iFlushInterval, int); INTERFACEpropSetMeth(strm, pszSizeLimitCmd, uchar*); /* v6 added */ - rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, int mode); + rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF); /* v7 added 2012-09-14 */ INTERFACEpropSetMeth(strm, bVeryReliableZip, int); /* v8 added 2013-03-21 */ @@ -197,7 +198,8 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ INTERFACEpropSetMeth(strm, cryprov, cryprov_if_t*); INTERFACEpropSetMeth(strm, cryprovData, void*); ENDinterface(strm) -#define strmCURR_IF_VERSION 9 /* increment whenever you change the interface structure! */ +#define strmCURR_IF_VERSION 10 /* increment whenever you change the interface structure! */ +/* V10, 2013-09-10: added new parameter bEscapeLF, changed mode to uint8_t (rgerhards) */ static inline int strmGetCurrFileNum(strm_t *pStrm) { diff --git a/runtime/stringbuf.c b/runtime/stringbuf.c index 13f38710..97bfed40 100644 --- a/runtime/stringbuf.c +++ b/runtime/stringbuf.c @@ -528,26 +528,6 @@ rsRetVal rsCStrTruncate(cstr_t *pThis, size_t nTrunc) /* Trim trailing whitespace from a given string */ -rsRetVal rsCStrTrimTrailingWhiteSpace(cstr_t *pThis) -{ - register int i; - register uchar *pC; - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - - i = pThis->iStrLen; - pC = pThis->pBuf + i - 1; - while(i > 0 && isspace((int)*pC)) { - --pC; - --i; - } - /* i now is the new string length! */ - pThis->iStrLen = i; - - return RS_RET_OK; -} - -/* Trim trailing whitespace from a given string - */ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) { register int i; @@ -563,8 +543,10 @@ rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) --i; } /* i now is the new string length! */ - pThis->iStrLen = i; - pThis->pBuf[pThis->iStrLen] = '\0'; /* we always have this space */ + if(i != (int) pThis->iStrLen) { + pThis->iStrLen = i; + pThis->pBuf[pThis->iStrLen] = '\0'; /* we always have this space */ + } done: return RS_RET_OK; } diff --git a/runtime/stringbuf.h b/runtime/stringbuf.h index d0502a5b..2df48ab4 100644 --- a/runtime/stringbuf.h +++ b/runtime/stringbuf.h @@ -155,7 +155,6 @@ static inline uchar* cstrGetSzStrNoNULL(cstr_t *pThis) */ rsRetVal rsCStrTruncate(cstr_t *pThis, size_t nTrunc); -rsRetVal rsCStrTrimTrailingWhiteSpace(cstr_t *pThis); rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis); /** diff --git a/runtime/typedefs.h b/runtime/typedefs.h index d3f68b4a..2efd7d65 100644 --- a/runtime/typedefs.h +++ b/runtime/typedefs.h @@ -25,6 +25,7 @@ */ #ifndef INCLUDED_TYPEDEFS_H #define INCLUDED_TYPEDEFS_H +#include <stdint.h> #if defined(__FreeBSD__) #include <sys/types.h> #endif @@ -63,6 +64,7 @@ typedef struct nsdsel_ptcp_s nsdsel_ptcp_t; typedef struct nsdsel_gtls_s nsdsel_gtls_t; typedef struct nsdpoll_ptcp_s nsdpoll_ptcp_t; typedef struct wti_s wti_t; +typedef struct msgPropDescr_s msgPropDescr_t; typedef struct msg msg_t; typedef struct queue_s qqueue_t; typedef struct prop_s prop_t; @@ -100,6 +102,9 @@ typedef struct outchannels_s outchannels_t; typedef struct modConfData_s modConfData_t; typedef struct instanceConf_s instanceConf_t; typedef struct ratelimit_s ratelimit_t; +typedef struct lookup_string_tab_etry_s lookup_string_tab_etry_t; +typedef struct lookup_tables_s lookup_tables_t; +typedef struct lookup_s lookup_t; typedef struct action_s action_t; typedef int rs_size_t; /* we do never need more than 2Gig strings, signed permits to * use -1 as a special flag. */ @@ -159,6 +164,53 @@ typedef enum { typedef off_t off64_t; #endif + +/* properties are now encoded as (tiny) integers. I do not use an enum as I would like + * to keep the memory footprint small (and thus cache hits high). + * rgerhards, 2009-06-26 + */ +typedef uintTiny propid_t; +#define PROP_INVALID 0 +#define PROP_MSG 1 +#define PROP_TIMESTAMP 2 +#define PROP_HOSTNAME 3 +#define PROP_SYSLOGTAG 4 +#define PROP_RAWMSG 5 +#define PROP_INPUTNAME 6 +#define PROP_FROMHOST 7 +#define PROP_FROMHOST_IP 8 +#define PROP_PRI 9 +#define PROP_PRI_TEXT 10 +#define PROP_IUT 11 +#define PROP_SYSLOGFACILITY 12 +#define PROP_SYSLOGFACILITY_TEXT 13 +#define PROP_SYSLOGSEVERITY 14 +#define PROP_SYSLOGSEVERITY_TEXT 15 +#define PROP_TIMEGENERATED 16 +#define PROP_PROGRAMNAME 17 +#define PROP_PROTOCOL_VERSION 18 +#define PROP_STRUCTURED_DATA 19 +#define PROP_APP_NAME 20 +#define PROP_PROCID 21 +#define PROP_MSGID 22 +#define PROP_PARSESUCCESS 23 +#define PROP_SYS_NOW 150 +#define PROP_SYS_YEAR 151 +#define PROP_SYS_MONTH 152 +#define PROP_SYS_DAY 153 +#define PROP_SYS_HOUR 154 +#define PROP_SYS_HHOUR 155 +#define PROP_SYS_QHOUR 156 +#define PROP_SYS_MINUTE 157 +#define PROP_SYS_MYHOSTNAME 158 +#define PROP_SYS_BOM 159 +#define PROP_SYS_UPTIME 160 +#define PROP_UUID 161 +#define PROP_CEE 200 +#define PROP_CEE_ALL_JSON 201 +#define PROP_LOCAL_VAR 202 +#define PROP_GLOBAL_VAR 203 + /* types of configuration handlers */ typedef enum cslCmdHdlrType { @@ -209,6 +261,13 @@ struct multi_submit_s { msg_t **ppMsgs; }; +/* the following structure is a helper to describe a message property */ +struct msgPropDescr_s { + propid_t id; + uchar *name; /* name and lenName are only set for dynamic */ + int nameLen; /* properties (JSON) */ +}; + #endif /* multi-include protection */ /* vim:set ai: */ diff --git a/runtime/wti.c b/runtime/wti.c index f91fb5a9..a9154670 100644 --- a/runtime/wti.c +++ b/runtime/wti.c @@ -44,12 +44,15 @@ #include "wti.h" #include "obj.h" #include "glbl.h" +#include "action.h" #include "atomic.h" /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) +pthread_key_t thrd_wti_key; + /* forward-definitions */ /* methods */ @@ -171,6 +174,8 @@ BEGINobjDestruct(wti) /* be sure to specify the object type also in END and CODE CODESTARTobjDestruct(wti) /* actual destruction */ batchFree(&pThis->batch); + free(pThis->actWrkrInfo); + pthread_cond_destroy(&pThis->pcondBusy); DESTROY_ATOMIC_HELPER_MUT(pThis->mutIsRunning); free(pThis->pszDbgHdr); @@ -181,6 +186,7 @@ ENDobjDestruct(wti) */ BEGINobjConstruct(wti) /* be sure to specify the object type also in END macro! */ INIT_ATOMIC_HELPER_MUT(pThis->mutIsRunning); + pthread_cond_init(&pThis->pcondBusy, NULL); ENDobjConstruct(wti) @@ -195,11 +201,20 @@ wtiConstructFinalize(wti_t *pThis) ISOBJ_TYPE_assert(pThis, wti); - DBGPRINTF("%s: finalizing construction of worker instance data\n", wtiGetDbgHdr(pThis)); + DBGPRINTF("%s: finalizing construction of worker instance data (for %d actions)\n", + wtiGetDbgHdr(pThis), iActionNbr); /* initialize our thread instance descriptor (no concurrency here) */ pThis->bIsRunning = RSFALSE; + /* must use calloc as we need zero-init */ + CHKmalloc(pThis->actWrkrInfo = calloc(iActionNbr, sizeof(actWrkrInfo_t))); + + if(pThis->pWtp == NULL) { + dbgprintf("wtiConstructFinalize: pWtp not set, this may be intentional\n"); + FINALIZE; + } + /* we now alloc the array for user pointers. We obtain the max from the queue itself. */ CHKiRet(pThis->pWtp->pfGetDeqBatchSize(pThis->pWtp->pUsr, &iDeqBatchSize)); CHKiRet(batchInit(&pThis->batch, iDeqBatchSize)); @@ -249,10 +264,10 @@ doIdleProcessing(wti_t *pThis, wtp_t *pWtp, int *pbInactivityTOOccured) if(pThis->bAlwaysRunning) { /* never shut down any started worker */ - d_pthread_cond_wait(pWtp->pcondBusy, pWtp->pmutUsr); + d_pthread_cond_wait(&pThis->pcondBusy, pWtp->pmutUsr); } else { timeoutComp(&t, pWtp->toWrkShutdown);/* get absolute timeout */ - if(d_pthread_cond_timedwait(pWtp->pcondBusy, pWtp->pmutUsr, &t) != 0) { + if(d_pthread_cond_timedwait(&pThis->pcondBusy, pWtp->pmutUsr, &t) != 0) { DBGPRINTF("%s: inactivity timeout, worker terminating...\n", wtiGetDbgHdr(pThis)); *pbInactivityTOOccured = 1; /* indicate we had a timeout */ } @@ -277,6 +292,7 @@ wtiWorker(wti_t *pThis) rsRetVal localRet; rsRetVal terminateRet; int iCancelStateSave; + int i; DEFiRet; ISOBJ_TYPE_assert(pThis, wti); @@ -286,6 +302,7 @@ wtiWorker(wti_t *pThis) dbgSetThrdName(pThis->pszDbgHdr); pthread_cleanup_push(wtiWorkerCancelCleanup, pThis); pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); +dbgprintf("DDDD: wti %p: worker starting\n", pThis); /* now we have our identity, on to real processing */ while(1) { /* loop will be broken below - need to do mutex locks */ @@ -332,9 +349,20 @@ wtiWorker(wti_t *pThis) bInactivityTOOccured = 0; /* reset for next run */ } + DBGPRINTF("DDDD: wti %p: worker cleanup up action instances\n", pThis); + for(i = 0 ; i < iActionNbr ; ++i) { + dbgprintf("wti %p, action %d, ptr %p\n", pThis, i, pThis->actWrkrInfo[i].actWrkrData); + if(pThis->actWrkrInfo[i].actWrkrData != NULL) { + dbgprintf("DDDD: calling freeWrkrData!\n"); + pThis->actWrkrInfo[i].pAction->pMod->mod.om.freeWrkrInstance(pThis->actWrkrInfo[i].actWrkrData); + pThis->actWrkrInfo[i].actWrkrData = NULL; /* re-init for next activation */ + } + } + /* indicate termination */ pthread_cleanup_pop(0); /* remove cleanup handler */ pthread_setcancelstate(iCancelStateSave, NULL); +dbgprintf("DDDD: wti %p: worker exiting\n", pThis); RETiRet; } @@ -374,6 +402,33 @@ finalize_it: } +/* This function returns (and creates if necessary) a dummy wti suitable + * for use by the rule engine. It is intended to be used for direct-mode + * main queues (folks, don't do that!). Once created, data is stored in + * thread-specific storage. + * Note: we do NOT do error checking -- if this functions fails, all the + * rest will fail as well... (also, it will only fail under OOM, so...). + * Memleak: we leak pWti's when run in direct mode. However, this is only + * a cosmetic leak, as we need them until all inputs are terminated, + * what means essentially until rsyslog itself is terminated. So we + * don't care -- it's just not nice in valgrind, but that's it. + */ +wti_t * +wtiGetDummy(void) +{ + wti_t *pWti; + + pWti = (wti_t*) pthread_getspecific(thrd_wti_key); + if(pWti == NULL) { + wtiConstruct(&pWti); + wtiConstructFinalize(pWti); + if(pthread_setspecific(thrd_wti_key, pWti) != 0) { + DBGPRINTF("wtiGetDummy: error setspecific thrd_wti_key\n"); + } + } + return pWti; +} + /* dummy */ rsRetVal wtiQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } @@ -383,6 +438,7 @@ BEGINObjClassExit(wti, OBJ_IS_CORE_MODULE) /* CHANGE class also in END MACRO! */ CODESTARTObjClassExit(nsdsel_gtls) /* release objects we no longer need */ objRelease(glbl, CORE_COMPONENT); + pthread_key_delete(thrd_wti_key); ENDObjClassExit(wti) @@ -391,8 +447,14 @@ ENDObjClassExit(wti) * rgerhards, 2008-01-09 */ BEGINObjClassInit(wti, 1, OBJ_IS_CORE_MODULE) /* one is the object version (most important for persisting) */ + int r; /* request objects we use */ CHKiRet(objUse(glbl, CORE_COMPONENT)); + r = pthread_key_create(&thrd_wti_key, NULL); + if(r != 0) { + dbgprintf("wti.c: pthread_key_create failed\n"); + iRet = RS_RET_ERR; + } ENDObjClassInit(wti) /* vi:set ai: diff --git a/runtime/wti.h b/runtime/wti.h index 014251f0..adc7897c 100644 --- a/runtime/wti.h +++ b/runtime/wti.h @@ -1,6 +1,6 @@ /* Definition of the worker thread instance (wti) class. * - * Copyright 2008-2012 Adiscon GmbH. + * Copyright 2008-2013 Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -26,18 +26,67 @@ #include "wtp.h" #include "obj.h" #include "batch.h" +#include "action.h" +#define ACT_STATE_RDY 0 /* action ready, waiting for new transaction */ +#define ACT_STATE_ITX 1 /* transaction active, waiting for new data or commit */ +#define ACT_STATE_COMM 2 /* transaction finished (a transient state) */ +#define ACT_STATE_RTRY 3 /* failure occured, trying to restablish ready state */ +#define ACT_STATE_SUSP 4 /* suspended due to failure (return fail until timeout expired) */ +/* note: 3 bit bit field --> highest value is 7! */ + +/* The following structure defines immutable parameters which need to + * be passed as action parameters. Note that the current implementation + * does NOT focus on performance, but on a simple PoC in order to get + * things going. TODO: Once it works, revisit this code and think about + * an array implementation. We also need to support other passing modes + * as well. -- gerhards, 2013-11-04 + */ +typedef struct actWrkrIParams { + int msgFlags; + /* following are caches to save allocs if not absolutely necessary */ + uchar *staticActStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /**< for strings */ + /* a cache to save malloc(), if not absolutely necessary */ + unsigned staticLenStrings[CONF_OMOD_NUMSTRINGS_MAXSIZE]; + /* and the same for the message length (if used) */ + void *staticActParams[CONF_OMOD_NUMSTRINGS_MAXSIZE]; +} actWrkrIParams_t; + +typedef struct actWrkrInfo { + action_t *pAction; + void *actWrkrData; + uint16_t uResumeOKinRow;/* number of times in a row that resume said OK with an immediate failure following */ + int iNbrResRtry; /* number of retries since last suspend */ + struct { + unsigned actState : 3; + } flags; + actWrkrIParams_t *iparams; + int currIParam; + int maxIParams; /* current max */ + void *staticActParams[CONF_OMOD_NUMSTRINGS_MAXSIZE]; /* for non-strings */ +} actWrkrInfo_t; + /* the worker thread instance class */ struct wti_s { BEGINobjInstance; pthread_t thrdID; /* thread ID */ int bIsRunning; /* is this thread currently running? (must be int for atomic op!) */ sbool bAlwaysRunning; /* should this thread always run? */ + int *pbShutdownImmediate;/* end processing of this batch immediately if set to 1 */ wtp_t *pWtp; /* my worker thread pool (important if only the work thread instance is passed! */ batch_t batch; /* pointer to an object array meaningful for current user pointer (e.g. queue pUsr data elemt) */ uchar *pszDbgHdr; /* header string for debug messages */ + actWrkrInfo_t *actWrkrInfo; /* *array* of action wrkr infos for all actions (sized for max nbr of actions in config!) */ + pthread_cond_t pcondBusy; /* condition to wake up the worker, protected by pmutUsr in wtp */ DEF_ATOMIC_HELPER_MUT(mutIsRunning); + struct { + uint8_t bPrevWasSuspended; + uint8_t bDoAutoCommit; /* do a commit after each message + * this is usually set for batches with 0 element, but may + * also be added as a user-selectable option (not implemented yet) + */ + } execState; /* state for the execution engine */ }; @@ -52,8 +101,96 @@ rsRetVal wtiSetAlwaysRunning(wti_t *pThis); rsRetVal wtiSetState(wti_t *pThis, sbool bNew); rsRetVal wtiWakeupThrd(wti_t *pThis); sbool wtiGetState(wti_t *pThis); +wti_t *wtiGetDummy(void); PROTOTYPEObjClassInit(wti); PROTOTYPEpropSetMeth(wti, pszDbgHdr, uchar*); PROTOTYPEpropSetMeth(wti, pWtp, wtp_t*); +static inline uint8_t +getActionStateByNbr(wti_t *pWti, int iActNbr) +{ + return((uint8_t) pWti->actWrkrInfo[iActNbr].flags.actState); +} + +static inline uint8_t +getActionState(wti_t *pWti, action_t *pAction) +{ + return((uint8_t) pWti->actWrkrInfo[pAction->iActionNbr].flags.actState); +} + +static inline void +setActionState(wti_t *pWti, action_t *pAction, uint8_t newState) +{ + pWti->actWrkrInfo[pAction->iActionNbr].flags.actState = newState; +} + +static inline uint16_t +getActionResumeInRow(wti_t *pWti, action_t *pAction) +{ + return(pWti->actWrkrInfo[pAction->iActionNbr].uResumeOKinRow); +} + +static inline void +setActionResumeInRow(wti_t *pWti, action_t *pAction, uint16_t val) +{ + pWti->actWrkrInfo[pAction->iActionNbr].uResumeOKinRow = val; +} + +static inline void +incActionResumeInRow(wti_t *pWti, action_t *pAction) +{ + pWti->actWrkrInfo[pAction->iActionNbr].uResumeOKinRow++; +} + +static inline int +getActionNbrResRtry(wti_t *pWti, action_t *pAction) +{ + return(pWti->actWrkrInfo[pAction->iActionNbr].iNbrResRtry); +} + +static inline void +setActionNbrResRtry(wti_t *pWti, action_t *pAction, uint16_t val) +{ + pWti->actWrkrInfo[pAction->iActionNbr].iNbrResRtry = val; +} + +static inline void +incActionNbrResRtry(wti_t *pWti, action_t *pAction) +{ + pWti->actWrkrInfo[pAction->iActionNbr].iNbrResRtry++; +} + +static inline rsRetVal +wtiNewIParam(wti_t *pWti, action_t *pAction, actWrkrIParams_t **piparams) +{ + actWrkrInfo_t *wrkrInfo; + actWrkrIParams_t *iparams; + int newMax; + DEFiRet; + + wrkrInfo = &(pWti->actWrkrInfo[pAction->iActionNbr]); + if(wrkrInfo->currIParam == wrkrInfo->maxIParams) { + /* we need to extend */ + dbgprintf("DDDD: extending iparams, max %d\n", wrkrInfo->maxIParams); + newMax = (wrkrInfo->maxIParams == 0) ? CONF_IPARAMS_BUFSIZE : 2 * wrkrInfo->maxIParams; + CHKmalloc(iparams = realloc(wrkrInfo->iparams, sizeof(actWrkrIParams_t) * newMax)); + wrkrInfo->iparams = iparams; + wrkrInfo->maxIParams = newMax; + } +dbgprintf("DDDD: adding param %d for action %d\n", wrkrInfo->currIParam, pAction->iActionNbr); + iparams = wrkrInfo->iparams + wrkrInfo->currIParam; + memset(iparams, 0, sizeof(actWrkrIParams_t)); + *piparams = iparams; + ++wrkrInfo->currIParam; + +finalize_it: + RETiRet; +} + +static inline void +wtiResetExecState(wti_t *pWti, batch_t *pBatch) +{ + pWti->execState.bPrevWasSuspended = 0; + pWti->execState.bDoAutoCommit = (batchNumMsgs(pBatch) == 1); +} #endif /* #ifndef WTI_H_INCLUDED */ diff --git a/runtime/wtp.c b/runtime/wtp.c index 19151e7c..66942e6b 100644 --- a/runtime/wtp.c +++ b/runtime/wtp.c @@ -8,7 +8,7 @@ * (and in the web doc set on http://www.rsyslog.com/doc). Be sure to read it * if you are getting aquainted to the object. * - * Copyright 2008,2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -91,6 +91,7 @@ BEGINobjConstruct(wtp) /* be sure to specify the object type also in END macro! pthread_cond_init(&pThis->condThrdTrm, NULL); pthread_attr_init(&pThis->attrThrd); /* Set thread scheduling policy to default */ +#warning do we need this any longer? I think it was a cure for an already fixed bug.. #ifdef HAVE_PTHREAD_SETSCHEDPARAM pthread_attr_setschedpolicy(&pThis->attrThrd, default_thr_sched_policy); pthread_attr_setschedparam(&pThis->attrThrd, &default_sched_param); @@ -121,7 +122,8 @@ wtpConstructFinalize(wtp_t *pThis) ISOBJ_TYPE_assert(pThis, wtp); - DBGPRINTF("%s: finalizing construction of worker thread pool\n", wtpGetDbgHdr(pThis)); + DBGPRINTF("%s: finalizing construction of worker thread pool (numworkerThreads %d)\n", + wtpGetDbgHdr(pThis), pThis->iNumWorkerThreads); /* alloc and construct workers - this can only be done in finalizer as we previously do * not know the max number of workers */ @@ -233,9 +235,9 @@ wtpShutdownAll(wtp_t *pThis, wtpState_t tShutdownCmd, struct timespec *ptTimeout /* lock mutex to prevent races (may otherwise happen during idle processing and such...) */ d_pthread_mutex_lock(pThis->pmutUsr); wtpSetState(pThis, tShutdownCmd); - pthread_cond_broadcast(pThis->pcondBusy); /* wake up all workers */ /* awake workers in retry loop */ for(i = 0 ; i < pThis->iNumWorkerThreads ; ++i) { + pthread_cond_signal(&pThis->pWrkr[i]->pcondBusy); wtiWakeupThrd(pThis->pWrkr[i]); } d_pthread_mutex_unlock(pThis->pmutUsr); @@ -455,7 +457,7 @@ wtpAdviseMaxWorkers(wtp_t *pThis, int nMaxWrkr) { DEFiRet; int nMissing; /* number workers missing to run */ - int i; + int i, nRunning; ISOBJ_TYPE_assert(pThis, wtp); @@ -475,7 +477,13 @@ wtpAdviseMaxWorkers(wtp_t *pThis, int nMaxWrkr) CHKiRet(wtpStartWrkr(pThis)); } } else { - pthread_cond_signal(pThis->pcondBusy); + /* we have needed number of workers, but they may be sleeping */ + for(i = 0, nRunning = 0; i < pThis->iNumWorkerThreads && nRunning < nMaxWrkr; ++i) { + if (wtiGetState(pThis->pWrkr[i]) != WRKTHRD_STOPPED) { + pthread_cond_signal(&pThis->pWrkr[i]->pcondBusy); + nRunning++; + } + } } @@ -490,7 +498,6 @@ DEFpropSetMeth(wtp, wtpState, wtpState_t) DEFpropSetMeth(wtp, iNumWorkerThreads, int) DEFpropSetMeth(wtp, pUsr, void*) DEFpropSetMethPTR(wtp, pmutUsr, pthread_mutex_t) -DEFpropSetMethPTR(wtp, pcondBusy, pthread_cond_t) DEFpropSetMethFP(wtp, pfChkStopWrkr, rsRetVal(*pVal)(void*, int)) DEFpropSetMethFP(wtp, pfRateLimiter, rsRetVal(*pVal)(void*)) DEFpropSetMethFP(wtp, pfGetDeqBatchSize, rsRetVal(*pVal)(void*, int*)) diff --git a/runtime/wtp.h b/runtime/wtp.h index 25992f7f..4bc284cb 100644 --- a/runtime/wtp.h +++ b/runtime/wtp.h @@ -56,7 +56,6 @@ struct wtp_s { void *pUsr; /* pointer to user object (in this case, the queue the wtp belongs to) */ pthread_attr_t attrThrd;/* attribute for new threads (created just once and cached here) */ pthread_mutex_t *pmutUsr; - pthread_cond_t *pcondBusy; /* condition the user will signal "busy again, keep runing" on (awakes worker) */ rsRetVal (*pfChkStopWrkr)(void *pUsr, int); rsRetVal (*pfGetDeqBatchSize)(void *pUsr, int*); /* obtains max dequeue count from queue config */ rsRetVal (*pfObjProcessed)(void *pUsr, wti_t *pWti); /* indicate user object is processed */ @@ -95,6 +94,5 @@ PROTOTYPEpropSetMeth(wtp, iMaxWorkerThreads, int); PROTOTYPEpropSetMeth(wtp, pUsr, void*); PROTOTYPEpropSetMeth(wtp, iNumWorkerThreads, int); PROTOTYPEpropSetMethPTR(wtp, pmutUsr, pthread_mutex_t); -PROTOTYPEpropSetMethPTR(wtp, pcondBusy, pthread_cond_t); #endif /* #ifndef WTP_H_INCLUDED */ diff --git a/tcps_sess.c b/tcps_sess.c index 5821e443..96bb6a4b 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -253,6 +253,8 @@ defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttG CHKiRet(msgConstructWithTime(&pMsg, stTime, ttGenTime)); MsgSetRawMsg(pMsg, (char*)pThis->pMsg, pThis->iMsg); MsgSetInputName(pMsg, pThis->pLstnInfo->pInputName); + if(pThis->pLstnInfo->dfltTZ != NULL) + MsgSetDfltTZ(pMsg, (char*) pThis->pLstnInfo->dfltTZ); MsgSetFlowControlType(pMsg, pThis->pSrv->bUseFlowControl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; @@ -134,6 +134,7 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort, int bSuppOctetFram) /* create entry */ CHKmalloc(pEntry = MALLOC(sizeof(tcpLstnPortList_t))); CHKmalloc(pEntry->pszPort = ustrdup(pszPort)); + strcpy((char*)pEntry->dfltTZ, (char*)pThis->dfltTZ); pEntry->pSrv = pThis; pEntry->pRuleset = pThis->pRuleset; pEntry->bSuppOctetFram = bSuppOctetFram; @@ -157,7 +158,7 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort, int bSuppOctetFram) ratelimitSetThreadSafe(pEntry->ratelimiter); STATSCOUNTER_INIT(pEntry->ctrSubmit, pEntry->mutCtrSubmit); CHKiRet(statsobj.AddCounter(pEntry->stats, UCHAR_CONSTANT("submitted"), - ctrType_IntCtr, &(pEntry->ctrSubmit))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pEntry->ctrSubmit))); CHKiRet(statsobj.ConstructFinalize(pEntry->stats)); finalize_it: @@ -743,7 +744,8 @@ RunSelect(tcpsrv_t *pThis, nsd_epworkset_t workset[], size_t sizeWorkset) pthread_cleanup_push(RunCancelCleanup, (void*) &pSel); while(1) { CHKiRet(nssel.Construct(&pSel)); - // TODO: set driver + if(pThis->pszDrvrName != NULL) + CHKiRet(nssel.SetDrvrName(pSel, pThis->pszDrvrName)); CHKiRet(nssel.ConstructFinalize(pSel)); /* Add the TCP listen sockets to the list of read descriptors. */ @@ -859,7 +861,8 @@ Run(tcpsrv_t *pThis) * to prevent us from leaking anything. -- rgerhards, 20080-04-24 */ if((localRet = nspoll.Construct(&pPoll)) == RS_RET_OK) { - // TODO: set driver + if(pThis->pszDrvrName != NULL) + CHKiRet(nspoll.SetDrvrName(pPoll, pThis->pszDrvrName)); localRet = nspoll.ConstructFinalize(pPoll); } if(localRet != RS_RET_OK) { @@ -916,9 +919,11 @@ BEGINobjConstruct(tcpsrv) /* be sure to specify the object type also in END macr pThis->addtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; pThis->bDisableLFDelim = 0; pThis->OnMsgReceive = NULL; + pThis->dfltTZ[0] = '\0'; pThis->ratelimitInterval = 0; pThis->ratelimitBurst = 10000; pThis->bUseFlowControl = 1; + pThis->pszDrvrName = NULL; ENDobjConstruct(tcpsrv) @@ -931,12 +936,13 @@ tcpsrvConstructFinalize(tcpsrv_t *pThis) /* prepare network stream subsystem */ CHKiRet(netstrms.Construct(&pThis->pNS)); + if(pThis->pszDrvrName != NULL) + CHKiRet(netstrms.SetDrvrName(pThis->pNS, pThis->pszDrvrName)); CHKiRet(netstrms.SetDrvrMode(pThis->pNS, pThis->iDrvrMode)); if(pThis->pszDrvrAuthMode != NULL) CHKiRet(netstrms.SetDrvrAuthMode(pThis->pNS, pThis->pszDrvrAuthMode)); if(pThis->pPermPeers != NULL) CHKiRet(netstrms.SetDrvrPermPeers(pThis->pNS, pThis->pPermPeers)); - // TODO: set driver! CHKiRet(netstrms.ConstructFinalize(pThis->pNS)); /* set up listeners */ @@ -965,6 +971,7 @@ CODESTARTobjDestruct(tcpsrv) if(pThis->pNS != NULL) netstrms.Destruct(&pThis->pNS); + free(pThis->pszDrvrName); free(pThis->pszDrvrAuthMode); free(pThis->ppLstn); free(pThis->ppLstnPort); @@ -1109,6 +1116,15 @@ SetAddtlFrameDelim(tcpsrv_t *pThis, int iDelim) } +static rsRetVal +SetDfltTZ(tcpsrv_t *pThis, uchar *tz) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, tcpsrv); + strcpy((char*)pThis->dfltTZ, (char*)tz); + RETiRet; +} + /* Set the input name to use -- rgerhards, 2008-12-10 */ static rsRetVal SetInputName(tcpsrv_t *pThis, uchar *name) @@ -1173,6 +1189,16 @@ SetDrvrMode(tcpsrv_t *pThis, int iMode) RETiRet; } +static rsRetVal +SetDrvrName(tcpsrv_t *pThis, uchar *name) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, tcpsrv); + free(pThis->pszDrvrName); + CHKmalloc(pThis->pszDrvrName = ustrdup(name)); +finalize_it: + RETiRet; +} /* set the driver authentication mode -- rgerhards, 2008-05-19 */ static rsRetVal @@ -1268,6 +1294,7 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetKeepAlive = SetKeepAlive; pIf->SetUsrP = SetUsrP; pIf->SetInputName = SetInputName; + pIf->SetDfltTZ = SetDfltTZ; pIf->SetAddtlFrameDelim = SetAddtlFrameDelim; pIf->SetbDisableLFDelim = SetbDisableLFDelim; pIf->SetSessMax = SetSessMax; @@ -1275,6 +1302,7 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetLstnMax = SetLstnMax; pIf->SetDrvrMode = SetDrvrMode; pIf->SetDrvrAuthMode = SetDrvrAuthMode; + pIf->SetDrvrName = SetDrvrName; pIf->SetDrvrPermPeers = SetDrvrPermPeers; pIf->SetCBIsPermittedHost = SetCBIsPermittedHost; pIf->SetCBOpenLstnSocks = SetCBOpenLstnSocks; @@ -43,6 +43,7 @@ struct tcpLstnPortList_s { statsobj_t *stats; /**< associated stats object */ sbool bSuppOctetFram; /**< do we support octect-counted framing? (if no->legay only!)*/ ratelimit_t *ratelimiter; + uchar dfltTZ[8]; /**< default TZ if none in timestamp; '\0' =No Default */ STATSCOUNTER_DEF(ctrSubmit, mutCtrSubmit) tcpLstnPortList_t *pNext; /**< next port or NULL */ }; @@ -56,6 +57,7 @@ struct tcpsrv_s { netstrms_t *pNS; /**< pointer to network stream subsystem */ int iDrvrMode; /**< mode of the stream driver to use */ uchar *pszDrvrAuthMode; /**< auth mode of the stream driver to use */ + uchar *pszDrvrName; /**< name of stream driver to use */ uchar *pszInputName; /**< value to be used as input name */ ruleset_t *pRuleset; /**< ruleset to bind to */ permittedPeers_t *pPermPeers;/**< driver's permitted peers */ @@ -67,6 +69,7 @@ struct tcpsrv_s { tcpLstnPortList_t **ppLstnPort; /**< pointer to relevant listen port description */ int iLstnMax; /**< max number of listeners supported */ int iSessMax; /**< max number of sessions supported */ + uchar dfltTZ[8]; /**< default TZ if none in timestamp; '\0' =No Default */ tcpLstnPortList_t *pLstnPorts; /**< head pointer for listen ports */ int addtlFrameDelim; /**< additional frame delimiter for plain TCP syslog framing (e.g. to handle NetScreen) */ @@ -110,7 +113,6 @@ BEGINinterface(tcpsrv) /* name must also be changed in ENDinterface macro! */ rsRetVal (*ConstructFinalize)(tcpsrv_t __attribute__((unused)) *pThis); rsRetVal (*Destruct)(tcpsrv_t **ppThis); rsRetVal (*configureTCPListen)(tcpsrv_t*, uchar *pszPort, int bSuppOctetFram); - //rsRetVal (*SessAccept)(tcpsrv_t *pThis, tcpLstnPortList_t*, tcps_sess_t **ppSess, netstrm_t *pStrm); rsRetVal (*create_tcp_socket)(tcpsrv_t *pThis); rsRetVal (*Run)(tcpsrv_t *pThis); /* set methods */ @@ -147,8 +149,12 @@ BEGINinterface(tcpsrv) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetKeepAlive)(tcpsrv_t*, int); /* added v13 -- rgerhards, 2012-10-15 */ rsRetVal (*SetLinuxLikeRatelimiters)(tcpsrv_t *pThis, int interval, int burst); + /* added v14 -- rgerhards, 2013-07-28 */ + rsRetVal (*SetDfltTZ)(tcpsrv_t *pThis, uchar *dfltTZ); + /* added v15 -- rgerhards, 2013-09-17 */ + rsRetVal (*SetDrvrName)(tcpsrv_t *pThis, uchar *pszName); ENDinterface(tcpsrv) -#define tcpsrvCURR_IF_VERSION 13 /* increment whenever you change the interface structure! */ +#define tcpsrvCURR_IF_VERSION 15 /* increment whenever you change the interface structure! */ /* change for v4: * - SetAddtlFrameDelim() added -- rgerhards, 2008-12-10 * - SetInputName() added -- rgerhards, 2008-12-10 @@ -1,7 +1,7 @@ /* This is the template processing code of rsyslog. * begun 2004-11-17 rgerhards * - * Copyright 2004-2012 Rainer Gerhards and Adiscon + * Copyright 2004-2013 Rainer Gerhards and Adiscon * * This file is part of rsyslog. * @@ -163,13 +163,13 @@ tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf, FINALIZE; } - if(pTpl->subtree != NULL) { + if(pTpl->bHaveSubtree) { /* only a single CEE subtree must be provided */ /* note: we could optimize the code below, however, this is * not worth the effort, as this passing mode is not expected * in subtree mode and so most probably only used for debug & test. */ - getCEEPropVal(pMsg, pTpl->subtree, &pVal, &iLenVal, &bMustBeFreed); + getJSONPropVal(pMsg, &pTpl->subtree, &pVal, &iLenVal, &bMustBeFreed); if(iLenVal >= (rs_size_t)*pLenBuf) /* we reserve one char for the final \0! */ CHKiRet(ExtendBuf(ppBuf, pLenBuf, iLenVal + 1)); memcpy(*ppBuf, pVal, iLenVal+1); @@ -193,9 +193,8 @@ tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf, iLenVal = pTpe->data.constant.iLenConstant; bMustBeFreed = 0; } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &iLenVal, - &bMustBeFreed, ttNow); + pVal = (uchar*) MsgGetProp(pMsg, pTpe, &pTpe->data.field.msgProp, + &iLenVal, &bMustBeFreed, ttNow); /* we now need to check if we should use SQL option. In this case, * we must go over the generated string and escape '\'' characters. * rgerhards, 2005-09-22: the option values below look somewhat misplaced, @@ -264,12 +263,12 @@ tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime assert(pMsg != NULL); assert(ppArr != NULL); - if(pTpl->subtree) { + if(pTpl->bHaveSubtree) { /* Note: this mode is untested, as there is no official plugin * using array passing, so I simply could not test it. */ CHKmalloc(pArr = calloc(2, sizeof(uchar*))); - getCEEPropVal(pMsg, pTpl->subtree, &pVal, &propLen, &bMustBeFreed); + getJSONPropVal(pMsg, &pTpl->subtree, &pVal, &propLen, &bMustBeFreed); if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ pArr[0] = pVal; /* ... so we can use it! */ } else { @@ -290,9 +289,8 @@ tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr, struct syslogTime if(pTpe->eEntryType == CONSTANT) { CHKmalloc(pArr[iArr] = (uchar*)strdup((char*) pTpe->data.constant.pConstant)); } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, - &bMustBeFreed, ttNow); + pVal = (uchar*) MsgGetProp(pMsg, pTpe, &pTpe->data.field.msgProp, + &propLen, &bMustBeFreed, ttNow); if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ pArr[iArr] = pVal; /* ... so we can use it! */ } else { @@ -326,8 +324,8 @@ tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson, struct rsRetVal localRet; DEFiRet; - if(pTpl->subtree != NULL){ - localRet = jsonFind(pMsg, pTpl->subtree, pjson); + if(pTpl->bHaveSubtree){ + localRet = jsonFind(pMsg->json, &pTpl->subtree, pjson); if(*pjson == NULL) { /* we need to have a root object! */ *pjson = json_object_new_object(); @@ -345,21 +343,22 @@ tplToJSON(struct template *pTpl, msg_t *pMsg, struct json_object **pjson, struct jsonf = json_object_new_string((char*) pTpe->data.constant.pConstant); json_object_object_add(json, (char*)pTpe->fieldName, jsonf); } else if(pTpe->eEntryType == FIELD) { - if(pTpe->data.field.propid == PROP_CEE) { - localRet = msgGetCEEPropJSON(pMsg, pTpe->data.field.propName, &jsonf); + if(pTpe->data.field.msgProp.id == PROP_CEE || + pTpe->data.field.msgProp.id == PROP_LOCAL_VAR || + pTpe->data.field.msgProp.id == PROP_GLOBAL_VAR ) { + localRet = msgGetJSONPropJSON(pMsg, &pTpe->data.field.msgProp, &jsonf); if(localRet == RS_RET_OK) { json_object_object_add(json, (char*)pTpe->fieldName, json_object_get(jsonf)); } else { - DBGPRINTF("tplToJSON: error %d looking up property\n", - localRet); + DBGPRINTF("tplToJSON: error %d looking up property %s\n", + localRet, pTpe->fieldName); if(pTpe->data.field.options.bMandatory) { json_object_object_add(json, (char*)pTpe->fieldName, NULL); } } } else { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, - pTpe->data.field.propName, &propLen, - &bMustBeFreed, ttNow); + pVal = (uchar*) MsgGetProp(pMsg, pTpe, &pTpe->data.field.msgProp, + &propLen, &bMustBeFreed, ttNow); if(pTpe->data.field.options.bMandatory || propLen > 0) { jsonf = json_object_new_string_len((char*)pVal, propLen); json_object_object_add(json, (char*)pTpe->fieldName, jsonf); @@ -755,7 +754,7 @@ static rsRetVal do_Parameter(uchar **pp, struct template *pTpl) { uchar *p; - cstr_t *pStrProp; + cstr_t *pStrProp = NULL; cstr_t *pStrField = NULL; struct templateEntry *pTpe; int iNum; /* to compute numbers */ @@ -785,19 +784,8 @@ do_Parameter(uchar **pp, struct template *pTpl) /* got the name */ cstrFinalize(pStrProp); - if(propNameToID(pStrProp, &pTpe->data.field.propid) != RS_RET_OK) { - errmsg.LogError(0, RS_RET_TPL_INVLD_PROP, "template '%s': invalid parameter '%s'", - pTpl->pszName, cstrGetSzStrNoNULL(pStrProp)); - cstrDestruct(&pStrProp); - ABORT_FINALIZE(RS_RET_TPL_INVLD_PROP); - } - if(pTpe->data.field.propid == PROP_CEE) { - /* in CEE case, we need to preserve the actual property name */ - if((pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(pStrProp)+1, cstrLen(pStrProp)-1)) == NULL) { - cstrDestruct(&pStrProp); - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - } + CHKiRet(msgPropDescrFill(&pTpe->data.field.msgProp, cstrGetSzStrNoNULL(pStrProp), + cstrLen(pStrProp))); /* Check frompos, if it has an R, then topos should be a regex */ if(*p == ':') { @@ -1094,8 +1082,8 @@ do_Parameter(uchar **pp, struct template *pTpl) /* save field name - if none was given, use the property name instead */ if(pStrField == NULL) { - if(pTpe->data.field.propid == PROP_CEE) { - /* in CEE case, we remove "$!" from the fieldname - it's just our indicator */ + if(pTpe->data.field.msgProp.id == PROP_CEE || pTpe->data.field.msgProp.id == PROP_LOCAL_VAR) { + /* in CEE case, we remove "$!"/"$." from the fieldname - it's just our indicator */ pTpe->fieldName = ustrdup(cstrGetSzStrNoNULL(pStrProp)+2); pTpe->lenFieldName = cstrLen(pStrProp)-2; } else { @@ -1111,10 +1099,11 @@ do_Parameter(uchar **pp, struct template *pTpl) DBGPRINTF("template/do_Parameter: fieldName is NULL!\n"); ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } - cstrDestruct(&pStrProp); if(*p) ++p; /* eat '%' */ *pp = p; finalize_it: + if(pStrProp != NULL) + cstrDestruct(&pStrProp); RETiRet; } @@ -1575,12 +1564,8 @@ createPropertyTpe(struct template *pTpl, struct cnfobj *o) /* apply */ CHKmalloc(pTpe = tpeConstruct(pTpl)); pTpe->eEntryType = FIELD; - CHKiRet(propNameToID(name, &pTpe->data.field.propid)); - if(pTpe->data.field.propid == PROP_CEE) { - /* in CEE case, we need to preserve the actual property name */ - pTpe->data.field.propName = es_newStrFromCStr((char*)cstrGetSzStrNoNULL(name)+1, - cstrLen(name)-1); - } + CHKiRet(msgPropDescrFill(&pTpe->data.field.msgProp, cstrGetSzStrNoNULL(name), + cstrLen(name))); pTpe->data.field.options.bDropLastLF = droplastlf; pTpe->data.field.options.bSPIffNo1stSP = spifno1stsp; pTpe->data.field.options.bMandatory = mandatory; @@ -1713,8 +1698,9 @@ tplProcessCnf(struct cnfobj *o) char *name = NULL; uchar *tplStr = NULL; uchar *plugin = NULL; - es_str_t *subtree = NULL; uchar *p; + msgPropDescr_t subtree; + sbool bHaveSubtree = 0; enum { T_STRING, T_PLUGIN, T_LIST, T_SUBTREE } tplType = T_STRING; /* init just to keep compiler happy: mandatory parameter */ int i; @@ -1761,10 +1747,11 @@ tplProcessCnf(struct cnfobj *o) free(name); /* overall assigned */ ABORT_FINALIZE(RS_RET_ERR); } else { - /* TODO: unify strings! */ - char *cstr = es_str2cstr(pvals[i].val.d.estr, NULL); - subtree = es_newStrFromBuf(cstr+1, es_strlen(pvals[i].val.d.estr)-1); + uchar *cstr; + cstr = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + CHKiRet(msgPropDescrFill(&subtree, cstr, ustrlen(cstr))); free(cstr); + bHaveSubtree = 1; } } else if(!strcmp(pblk.descr[i].name, "plugin")) { plugin = (uchar*) es_str2cstr(pvals[i].val.d.estr, NULL); @@ -1807,7 +1794,7 @@ tplProcessCnf(struct cnfobj *o) } } - if(subtree == NULL) { + if(!bHaveSubtree) { if(tplType == T_SUBTREE) { errmsg.LogError(0, RS_RET_ERR, "template '%s' of type subtree needs " "subtree parameter", name); @@ -1877,7 +1864,8 @@ tplProcessCnf(struct cnfobj *o) break; case T_LIST: createListTpl(pTpl, o); break; - case T_SUBTREE: pTpl->subtree = subtree; + case T_SUBTREE: memcpy(&pTpl->subtree, &subtree, sizeof(msgPropDescr_t)); + pTpl->bHaveSubtree = 1; break; } @@ -1969,9 +1957,8 @@ void tplDeleteAll(rsconf_t *conf) regexp.regfree(&(pTpeDel->data.field.re)); } } - if(pTpeDel->data.field.propName != NULL) - es_deleteStr(pTpeDel->data.field.propName); #endif + msgPropDescrDestruct(&pTpeDel->data.field.msgProp); break; } free(pTpeDel->fieldName); @@ -1981,8 +1968,8 @@ void tplDeleteAll(rsconf_t *conf) pTplDel = pTpl; pTpl = pTpl->pNext; free(pTplDel->pszName); - if(pTplDel->subtree != NULL) - es_deleteStr(pTplDel->subtree); + if(pTplDel->bHaveSubtree) + msgPropDescrDestruct(&pTplDel->subtree); free(pTplDel); } ENDfunc @@ -2029,9 +2016,8 @@ void tplDeleteNew(rsconf_t *conf) regexp.regfree(&(pTpeDel->data.field.re)); } } - if(pTpeDel->data.field.propName != NULL) - es_deleteStr(pTpeDel->data.field.propName); #endif + msgPropDescrDestruct(&pTpeDel->data.field.msgProp); break; } /*dbgprintf("\n");*/ @@ -2040,8 +2026,8 @@ void tplDeleteNew(rsconf_t *conf) pTplDel = pTpl; pTpl = pTpl->pNext; free(pTplDel->pszName); - if(pTplDel->subtree != NULL) - es_deleteStr(pTplDel->subtree); + if(pTplDel->bHaveSubtree) + msgPropDescrDestruct(&pTplDel->subtree); free(pTplDel); } ENDfunc @@ -2083,11 +2069,13 @@ void tplPrintList(rsconf_t *conf) pTpe->data.constant.pConstant); break; case FIELD: - dbgprintf("(FIELD), value: '%d' ", pTpe->data.field.propid); - if(pTpe->data.field.propid == PROP_CEE) { - char *cstr = es_str2cstr(pTpe->data.field.propName, NULL); - dbgprintf("[EE-Property: '%s'] ", cstr); - free(cstr); + dbgprintf("(FIELD), value: '%d' ", pTpe->data.field.msgProp.id); + if(pTpe->data.field.msgProp.id == PROP_CEE) { + dbgprintf("[EE-Property: '%s'] ", pTpe->data.field.msgProp.name); + } else if(pTpe->data.field.msgProp.id == PROP_LOCAL_VAR) { + dbgprintf("[Local Var: '%s'] ", pTpe->data.field.msgProp.name); + //} else if(pTpe->data.field.propid == PROP_GLOBAL_VAR) { + // dbgprintf("[Global Var: '%s'] ", pTpe->data.field.propName); } switch(pTpe->data.field.eDateFormat) { case tplFmtDefault: @@ -40,7 +40,8 @@ struct template { char *pszName; int iLenName; rsRetVal (*pStrgen)(msg_t*, uchar**, size_t *); - es_str_t *subtree; /* subtree name for subtree-type templates */ + sbool bHaveSubtree; + msgPropDescr_t subtree; /* subtree property name for subtree-type templates */ int tpenElements; /* number of elements in templateEntry list */ struct templateEntry *pEntryRoot; struct templateEntry *pEntryLast; @@ -79,7 +80,7 @@ struct templateEntry { int iLenConstant; /* its length */ } constant; struct { - propid_t propid; /* property to be used */ + msgPropDescr_t msgProp; /* property to be used */ unsigned iFromPos; /* for partial strings only chars from this position ... */ unsigned iToPos; /* up to that one... */ unsigned iFieldNr; /* for field extraction: field to extract */ @@ -103,7 +104,6 @@ struct templateEntry { int field_expand; /* use multiple instances of the field delimiter as a single one? */ #endif - es_str_t *propName; /**< property name (currently being used for CEE only) */ enum tplFormatTypes eDateFormat; enum tplFormatCaseConvTypes eCaseConv; diff --git a/tests/Makefile.am b/tests/Makefile.am index b339e797..6fa4d96f 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -14,6 +14,7 @@ TESTS += \ diskqueue.sh \ diskqueue-fsync.sh \ rulesetmultiqueue.sh \ + rulesetmultiqueue-v6.sh \ manytcp.sh \ rsf_getenv.sh \ imtcp_conndrop.sh \ diff --git a/tests/daqueue-persist.sh b/tests/daqueue-persist.sh index feb2a347..0781a7dc 100755 --- a/tests/daqueue-persist.sh +++ b/tests/daqueue-persist.sh @@ -2,6 +2,7 @@ # to carry out multiple tests with different queue modes # added 2009-05-27 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== echo \[daqueue-persist.sh\]: test data persisting at shutdown source $srcdir/daqueue-persist-drvr.sh LinkedList source $srcdir/daqueue-persist-drvr.sh FixedArray diff --git a/tests/diskqueue.sh b/tests/diskqueue.sh index b871e9eb..853a836a 100755 --- a/tests/diskqueue.sh +++ b/tests/diskqueue.sh @@ -5,6 +5,7 @@ # added 2009-04-17 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: +echo =============================================================================== echo \[diskqueue.sh\]: testing queue disk-only mode # uncomment for debugging support: #export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" diff --git a/tests/rulesetmultiqueue-v6.sh b/tests/rulesetmultiqueue-v6.sh new file mode 100755 index 00000000..21166fe1 --- /dev/null +++ b/tests/rulesetmultiqueue-v6.sh @@ -0,0 +1,33 @@ +# Test for disk-only queue mode with v6+ config +# This tests defines three rulesets, each one with its own queue. Then, it +# sends data to them and checks the outcome. Note that we do need to +# use some custom code as the test driver framework does not (yet?) +# support multi-output-file operations. +# added 2013-11-14 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[rulesetmultiqueu.sh\]: testing multiple queues via rulesets +source $srcdir/diag.sh init +rm -f rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log +source $srcdir/diag.sh startup rulesetmultiqueue-v6.conf +source $srcdir/diag.sh wait-startup +# now fill the three files (a bit sequentially, but they should +# still get their share of concurrency - to increase the chance +# we use three connections per set). +source $srcdir/diag.sh tcpflood -c3 -p13514 -m20000 -i0 +source $srcdir/diag.sh tcpflood -c3 -p13515 -m20000 -i20000 +source $srcdir/diag.sh tcpflood -c3 -p13516 -m20000 -i40000 + +# in this version of the imdiag, we do not have the capability to poll +# all queues for emptyness. So we do a sleep in the hopes that this will +# sufficiently drain the queues. This is race, but the best we currently +# can do... - rgerhards, 2009-11-05 +sleep 2 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +# now consolidate all logs into a single one so that we can use the +# regular check logic +cat rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 59999 +rm -f rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log +source $srcdir/diag.sh exit diff --git a/tests/testsuites/rulesetmultiqueue-v6.conf b/tests/testsuites/rulesetmultiqueue-v6.conf new file mode 100644 index 00000000..3aeaa337 --- /dev/null +++ b/tests/testsuites/rulesetmultiqueue-v6.conf @@ -0,0 +1,34 @@ +# Test for multiple ruleset queues (see .sh file for details) +# rgerhards, 2009-10-30 +$IncludeConfig diag-common.conf +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 + +# general definition +$template outfmt,"%msg:F,58:2%\n" + +# create the individual rulesets +$template dynfile1,"rsyslog.out1.log" # trick to use relative path names! +ruleset(name="file1" queue.type="linkedList") { + :msg, contains, "msgnum:" ?dynfile1;outfmt +} + +$template dynfile2,"rsyslog.out2.log" # trick to use relative path names! +ruleset(name="file2" queue.type="linkedList") { + :msg, contains, "msgnum:" ?dynfile2;outfmt +} + +$template dynfile3,"rsyslog.out3.log" # trick to use relative path names! +ruleset(name="file3" queue.type="linkedList") { + :msg, contains, "msgnum:" ?dynfile3;outfmt +} + +# start listeners and bind them to rulesets +$InputTCPServerBindRuleset file1 +$InputTCPServerRun 13514 + +$InputTCPServerBindRuleset file2 +$InputTCPServerRun 13515 + +$InputTCPServerBindRuleset file3 +$InputTCPServerRun 13516 diff --git a/tools/omdiscard.c b/tools/omdiscard.c index 15c6ea82..a76bcc33 100644 --- a/tools/omdiscard.c +++ b/tools/omdiscard.c @@ -6,7 +6,7 @@ * * File begun on 2007-07-24 by RGerhards * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -49,6 +49,10 @@ typedef struct _instanceData { EMPTY_STRUCT } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + /* we do not need a createInstance()! BEGINcreateInstance CODESTARTcreateInstance @@ -56,6 +60,11 @@ ENDcreateInstance */ +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo /* do nothing */ @@ -87,6 +96,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINparseSelectorAct CODESTARTparseSelectorAct CODE_STD_STRING_REQUESTparseSelectorAct(0) @@ -114,6 +128,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES ENDqueryEtryPt diff --git a/tools/omfile.c b/tools/omfile.c index ba9f7f70..1ee1026c 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -133,6 +133,7 @@ typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; typedef struct _instanceData { + pthread_mutex_t mutWrite; /* guard against multiple instances writing to single file */ uchar *f_fname; /* file or template name (display only) */ uchar *tplName; /* name of assigned template */ strm_t *pStrm; /* our output stream */ @@ -155,7 +156,6 @@ typedef struct _instanceData { uchar *cryprovName; /* crypto provider */ uchar *cryprovNameFull;/* full internal crypto provider name */ void *cryprovData; /* opaque data ptr for provider use */ - void *cryprovFileData;/* opaque data ptr for file instance */ cryprov_if_t cryprov; /* ptr to crypto provider interface */ sbool useCryprov; /* quicker than checkig ptr (1 vs 8 bytes!) */ int iCurrElt; /* currently active cache element (-1 = none) */ @@ -182,6 +182,20 @@ typedef struct _instanceData { STATSCOUNTER_DEF(ctrMax, mutCtrMax); } instanceData; +/* to build a linked list for temporary storage of lines while we cannot commit */ +typedef struct linebuf { + uchar *filename; /* for dynafiles, make go away */ + uchar *ln; + unsigned iMsgOpts; + struct linebuf *pNext; +} linebuf_t; + +typedef struct wrkrInstanceData { + instanceData *pData; + linebuf_t *pRoot; + linebuf_t *pLast; +} wrkrInstanceData_t; + typedef struct configSettings_s { int iDynaFileCacheSize; /* max cache for dynamic files */ @@ -207,6 +221,8 @@ uchar *pszFileDfltTplName; /* name of the default template to use */ struct modConfData_s { rsconf_t *pConf; /* our overall config object */ uchar *tplName; /* default template */ + int fCreateMode; /* default mode to use when creating files */ + int fDirCreateMode; /* default mode to use when creating files */ }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ @@ -216,6 +232,8 @@ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current ex /* module-global parameters */ static struct cnfparamdescr modpdescr[] = { { "template", eCmdHdlrGetWord, 0 }, + { "dircreatemode", eCmdHdlrFileCreateMode, 0 }, + { "filecreatemode", eCmdHdlrFileCreateMode, 0 } }; static struct cnfparamblk modpblk = { CNFPARAMBLK_VERSION, @@ -783,7 +801,7 @@ finalize_it: /* rgerhards 2004-11-11: write to a file output. */ static rsRetVal -writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) +writeFile(instanceData *pData, linebuf_t *linebuf) { DEFiRet; @@ -793,7 +811,7 @@ writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) * check if it still is ok or a new file needs to be created */ if(pData->bDynamicName) { - CHKiRet(prepareDynFile(pData, ppString[1], iMsgOpts)); + CHKiRet(prepareDynFile(pData, linebuf->filename, linebuf->iMsgOpts)); } else { /* "regular", non-dynafile */ if(pData->pStrm == NULL) { CHKiRet(prepareFile(pData, pData->f_fname)); @@ -803,7 +821,7 @@ writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) } } - CHKiRet(doWrite(pData, ppString[0], strlen(CHAR_CONVERT(ppString[0])))); + CHKiRet(doWrite(pData, linebuf->ln, ustrlen(linebuf->ln))); finalize_it: RETiRet; @@ -815,6 +833,8 @@ CODESTARTbeginCnfLoad loadModConf = pModConf; pModConf->pConf = pConf; pModConf->tplName = NULL; + pModConf->fCreateMode = 0644; + pModConf->fDirCreateMode = 0700; ENDbeginCnfLoad BEGINsetModCnf @@ -843,6 +863,10 @@ CODESTARTsetModCnf "was already set via legacy directive - may lead to inconsistent " "results."); } + } else if(!strcmp(modpblk.descr[i].name, "dircreatemode")) { + loadModConf->fDirCreateMode = (int) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "filecreatemode")) { + loadModConf->fCreateMode = (int) pvals[i].val.d.n; } else { dbgprintf("omfile: program error, non-handled " "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); @@ -879,9 +903,15 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance pData->pStrm = NULL; + pthread_mutex_init(&pData->mutWrite, NULL); ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINfreeInstance CODESTARTfreeInstance free(pData->tplName); @@ -904,9 +934,15 @@ CODESTARTfreeInstance free(pData->cryprovName); free(pData->cryprovNameFull); } + pthread_mutex_destroy(&pData->mutWrite); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINtryResume CODESTARTtryResume ENDtryResume @@ -917,8 +953,70 @@ CODESTARTbeginTransaction ENDbeginTransaction +static rsRetVal +bufferLine(wrkrInstanceData_t *pWrkrData, uchar *filename, uchar *line) +{ + linebuf_t *lb; + DEFiRet; + + CHKmalloc(lb = (linebuf_t*) malloc(sizeof(linebuf_t))); +dbgprintf("DDDD: filename '%s'\n", filename); + CHKmalloc(lb->filename = ustrdup(filename)); + CHKmalloc(lb->ln = ustrdup(line)); + lb->pNext = NULL; + if(pWrkrData->pRoot == NULL) { + pWrkrData->pRoot = pWrkrData->pLast = lb; + } else { + pWrkrData->pLast->pNext = lb; + pWrkrData->pLast = lb; + } +finalize_it: + RETiRet; +} + +static void +submitCachedLines(wrkrInstanceData_t *pWrkrData, instanceData *pData) +{ + linebuf_t *curr, *todel; + +dbgprintf("omfile: waiting on write lock (pWrkrData %p)\n", pWrkrData); + pthread_mutex_lock(&pData->mutWrite); +dbgprintf("omfile: aquired write lock (pWrkrData %p)\n", pWrkrData); + + for(curr = pWrkrData->pRoot ; curr != NULL ; ) { + DBGPRINTF("omfile: file to log to: %s\n", curr->filename); + DBGPRINTF("omfile: start of data: '%.128s'\n", curr->ln); + STATSCOUNTER_INC(pData->ctrRequests, pData->mutCtrRequests); + writeFile(pData, curr); + + todel = curr; + curr = curr->pNext; + free(todel->filename); + free(todel->ln); + free(todel); + } + pthread_mutex_unlock(&pData->mutWrite); +dbgprintf("omfile: free write lock (pWrkrData %p)\n", pWrkrData); + pWrkrData->pRoot = NULL; +} + + +BEGINdoAction + instanceData *pData; +CODESTARTdoAction + pData = pWrkrData->pData; +dbgprintf("DDDD: bDynName %d, filename '%s'\n", pData->bDynamicName, ppString[1]); + iRet = bufferLine(pWrkrData, (pData->bDynamicName) ? ppString[1] : pData->f_fname, + ppString[0]); + if(iRet == RS_RET_OK) + iRet = RS_RET_DEFER_COMMIT; +ENDdoAction + BEGINendTransaction + instanceData *pData; CODESTARTendTransaction + pData = pWrkrData->pData; + submitCachedLines(pWrkrData, pData); /* Note: pStrm may be NULL if there was an error opening the stream */ if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { /* if we have an async writer, it controls the flush via @@ -932,21 +1030,6 @@ finalize_it: ENDendTransaction -BEGINdoAction -CODESTARTdoAction - DBGPRINTF("file to log to: %s\n", - (pData->bDynamicName) ? ppString[1] : pData->f_fname); - DBGPRINTF("omfile: start of data: '%.128s'\n", ppString[0]); - STATSCOUNTER_INC(pData->ctrRequests, pData->mutCtrRequests); - CHKiRet(writeFile(ppString, iMsgOpts, pData)); - if(!bCoreSupportsBatching && pData->bFlushOnTXEnd) { - CHKiRet(strm.Flush(pData->pStrm)); - } -finalize_it: - if(iRet == RS_RET_OK) - iRet = RS_RET_DEFER_COMMIT; -ENDdoAction - static inline void setInstParamDefaults(instanceData *pData) @@ -959,8 +1042,8 @@ setInstParamDefaults(instanceData *pData) pData->dirGID = -1; pData->bFailOnChown = 1; pData->iDynaFileCacheSize = 10; - pData->fCreateMode = 0644; - pData->fDirCreateMode = 0700; + pData->fCreateMode = loadModConf->fCreateMode; + pData->fDirCreateMode = loadModConf->fDirCreateMode; pData->bCreateDirs = 1; pData->bSyncFile = 0; pData->iZipLevel = 0; @@ -993,19 +1076,19 @@ setupInstStatsCtrs(instanceData *pData) CHKiRet(statsobj.SetName(pData->stats, ctrName)); STATSCOUNTER_INIT(pData->ctrRequests, pData->mutCtrRequests); CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("requests"), - ctrType_IntCtr, &(pData->ctrRequests))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pData->ctrRequests))); STATSCOUNTER_INIT(pData->ctrLevel0, pData->mutCtrLevel0); CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("level0"), - ctrType_IntCtr, &(pData->ctrLevel0))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pData->ctrLevel0))); STATSCOUNTER_INIT(pData->ctrMiss, pData->mutCtrMiss); CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("missed"), - ctrType_IntCtr, &(pData->ctrMiss))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pData->ctrMiss))); STATSCOUNTER_INIT(pData->ctrEvict, pData->mutCtrEvict); CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("evicted"), - ctrType_IntCtr, &(pData->ctrEvict))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pData->ctrEvict))); STATSCOUNTER_INIT(pData->ctrMax, pData->mutCtrMax); CHKiRet(statsobj.AddCounter(pData->stats, UCHAR_CONSTANT("maxused"), - ctrType_IntCtr, &(pData->ctrMax))); + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &(pData->ctrMax))); CHKiRet(statsobj.ConstructFinalize(pData->stats)); finalize_it: @@ -1089,7 +1172,7 @@ initCryprov(instanceData *pData, struct nvlst *lst) szDrvrName); ABORT_FINALIZE(RS_RET_CRYPROV_ERR); } - CHKiRet(pData->cryprov.SetCnfParam(pData->cryprovData, lst)); + CHKiRet(pData->cryprov.SetCnfParam(pData->cryprovData, lst, CRYPROV_PARAMTYPE_REGULAR)); dbgprintf("loaded crypto provider %s, data instance at %p\n", szDrvrName, pData->cryprovData); @@ -1327,6 +1410,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a BEGINdoHUP CODESTARTdoHUP + pthread_mutex_lock(&pData->mutWrite); if(pData->bDynamicName) { dynaFileFreeCacheEntries(pData); } else { @@ -1334,6 +1418,7 @@ CODESTARTdoHUP closeFile(pData); } } + pthread_mutex_unlock(&pData->mutWrite); ENDdoHUP @@ -1349,6 +1434,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES diff --git a/tools/omfwd.c b/tools/omfwd.c index 129392d2..8cf306ed 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -4,7 +4,7 @@ * NOTE: read comments in module-template.h to understand how this file * works! * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -21,9 +21,6 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * - * TODO v6 config: - * - permitted peer *list* */ #include "config.h" #include "rsyslog.h" @@ -39,6 +36,7 @@ #include <errno.h> #include <ctype.h> #include <unistd.h> +#include <stdint.h> #ifdef USE_NETZIP #include <zlib.h> #endif @@ -74,32 +72,51 @@ DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) DEFobjCurrIf(tcpclt) + +/* some local constants (just) for better readybility */ +#define IS_FLUSH 1 +#define NO_FLUSH 0 + typedef struct _instanceData { uchar *tplName; /* name of assigned template */ - netstrms_t *pNS; /* netstream subsystem */ - netstrm_t *pNetstrm; /* our output netstream */ uchar *pszStrmDrvr; uchar *pszStrmDrvrAuthMode; permittedPeers_t *pPermPeers; int iStrmDrvrMode; char *target; - int *pSockArray; /* sockets to use for UDP */ - int bIsConnected; /* are we connected to remote host? 0 - no, 1 - yes, UDP means addr resolved */ - struct addrinfo *f_addr; int compressionLevel; /* 0 - no compression, else level for zlib */ char *port; int protocol; int iRebindInterval; /* rebind interval */ - int nXmit; /* number of transmissions since last (re-)bind */ # define FORW_UDP 0 # define FORW_TCP 1 /* following fields for TCP-based delivery */ TCPFRAMINGMODE tcp_framing; int bResendLastOnRecon; /* should the last message be re-sent on a successful reconnect? */ +# define COMPRESS_NEVER 0 +# define COMPRESS_SINGLE_MSG 1 /* old, single-message compression */ + /* all other settings are for stream-compression */ +# define COMPRESS_STREAM_ALWAYS 2 + uint8_t compressionMode; + int errsToReport; /* max number of errors to report (per instance) */ + sbool strmCompFlushOnTxEnd; /* flush stream compression on transaction end? */ +} instanceData; + +typedef struct wrkrInstanceData { + instanceData *pData; + netstrms_t *pNS; /* netstream subsystem */ + netstrm_t *pNetstrm; /* our output netstream */ + struct addrinfo *f_addr; + int *pSockArray; /* sockets to use for UDP */ + int bIsConnected; /* are we connected to remote host? 0 - no, 1 - yes, UDP means addr resolved */ + int nXmit; /* number of transmissions since last (re-)bind */ tcpclt_t *pTCPClt; /* our tcpclt object */ + sbool bzInitDone; /* did we do an init of zstrm already? */ + z_stream zstrm; /* zip stream to use for tcp compression */ uchar sndBuf[16*1024]; /* this is intensionally fixed -- see no good reason to make configurable */ unsigned offsSndBuf; /* next free spot in send buffer */ -} instanceData; + int errsToReport; /* (remaining) number of errors to report */ +} wrkrInstanceData_t; /* config data */ typedef struct configSettings_s { @@ -132,6 +149,9 @@ static struct cnfparamdescr actpdescr[] = { { "protocol", eCmdHdlrGetWord, 0 }, { "tcp_framing", eCmdHdlrGetWord, 0 }, { "ziplevel", eCmdHdlrInt, 0 }, + { "compression.mode", eCmdHdlrGetWord, 0 }, + { "compression.stream.flushontxend", eCmdHdlrBinary, 0 }, + { "maxerrormessages", eCmdHdlrInt, 0 }, { "rebindinterval", eCmdHdlrInt, 0 }, { "streamdriver", eCmdHdlrGetWord, 0 }, { "streamdrivermode", eCmdHdlrInt, 0 }, @@ -155,6 +175,9 @@ static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current l static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current exec process */ +static rsRetVal initTCP(wrkrInstanceData_t *pWrkrData); + + BEGINinitConfVars /* (re)set config variables to default values */ CODESTARTinitConfVars cs.pszTplName = NULL; /* name of the default template to use */ @@ -168,7 +191,8 @@ CODESTARTinitConfVars ENDinitConfVars -static rsRetVal doTryResume(instanceData *pData); +static rsRetVal doTryResume(wrkrInstanceData_t *); +static rsRetVal doZipFinish(wrkrInstanceData_t *); /* this function gets the default template. It coordinates action between * old-style and new-style configuration parts. @@ -212,17 +236,16 @@ finalize_it: * rgerhards, 2009-05-29 */ static rsRetVal -closeUDPSockets(instanceData *pData) +closeUDPSockets(wrkrInstanceData_t *pWrkrData) { DEFiRet; - assert(pData != NULL); - if(pData->pSockArray != NULL) { - net.closeUDPListenSockets(pData->pSockArray); - pData->pSockArray = NULL; - freeaddrinfo(pData->f_addr); - pData->f_addr = NULL; + if(pWrkrData->pSockArray != NULL) { + net.closeUDPListenSockets(pWrkrData->pSockArray); + pWrkrData->pSockArray = NULL; + freeaddrinfo(pWrkrData->f_addr); + pWrkrData->f_addr = NULL; } -pData->bIsConnected = 0; // TODO: remove this variable altogether +pWrkrData->bIsConnected = 0; // TODO: remove this variable altogether RETiRet; } @@ -233,17 +256,17 @@ pData->bIsConnected = 0; // TODO: remove this variable altogether * rgerhards, 2008-06-04 * Note that we DO NOT discard the current buffer contents * (if any). This permits us to save data between sessions. In - * the wort case, some duplication occurs, but we do not + * the worst case, some duplication occurs, but we do not * loose data. */ static inline void -DestructTCPInstanceData(instanceData *pData) +DestructTCPInstanceData(wrkrInstanceData_t *pWrkrData) { - assert(pData != NULL); - if(pData->pNetstrm != NULL) - netstrm.Destruct(&pData->pNetstrm); - if(pData->pNS != NULL) - netstrms.Destruct(&pData->pNS); + doZipFinish(pWrkrData); + if(pWrkrData->pNetstrm != NULL) + netstrm.Destruct(&pWrkrData->pNetstrm); + if(pWrkrData->pNS != NULL) + netstrms.Destruct(&pWrkrData->pNS); } @@ -314,10 +337,25 @@ ENDfreeCnf BEGINcreateInstance CODESTARTcreateInstance - pData->offsSndBuf = 0; + pData->errsToReport = 5; + if(cs.pszStrmDrvr != NULL) + CHKmalloc(pData->pszStrmDrvr = (uchar*)strdup((char*)cs.pszStrmDrvr)); + if(cs.pszStrmDrvrAuthMode != NULL) + CHKmalloc(pData->pszStrmDrvrAuthMode = + (uchar*)strdup((char*)cs.pszStrmDrvrAuthMode)); +finalize_it: ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance + dbgprintf("DDDD: createWrkrInstance: pWrkrData %p\n", pWrkrData); + pWrkrData->offsSndBuf = 0; + pWrkrData->errsToReport = pData->errsToReport; + iRet = initTCP(pWrkrData); +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -327,22 +365,25 @@ ENDisCompatibleWithFeature BEGINfreeInstance CODESTARTfreeInstance - /* final cleanup */ - DestructTCPInstanceData(pData); - closeUDPSockets(pData); - - if(pData->protocol == FORW_TCP) { - tcpclt.Destruct(&pData->pTCPClt); - } - - free(pData->port); - free(pData->target); free(pData->pszStrmDrvr); free(pData->pszStrmDrvrAuthMode); + free(pData->port); + free(pData->target); net.DestructPermittedPeers(&pData->pPermPeers); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance + DestructTCPInstanceData(pWrkrData); + closeUDPSockets(pWrkrData); + + if(pWrkrData->pData->protocol == FORW_TCP) { + tcpclt.Destruct(&pWrkrData->pTCPClt); + } +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("%s", pData->target); @@ -352,25 +393,27 @@ ENDdbgPrintInstInfo /* Send a message via UDP * rgehards, 2007-12-20 */ -static rsRetVal UDPSend(instanceData *pData, char *msg, size_t len) +static rsRetVal UDPSend(wrkrInstanceData_t *pWrkrData, char *msg, size_t len) { DEFiRet; struct addrinfo *r; int i; unsigned lsent = 0; - int bSendSuccess; + sbool bSendSuccess; + int lasterrno; + char errStr[1024]; - if(pData->iRebindInterval && (pData->nXmit++ % pData->iRebindInterval == 0)) { + if(pWrkrData->pData->iRebindInterval && (pWrkrData->nXmit++ % pWrkrData->pData->iRebindInterval == 0)) { dbgprintf("omfwd dropping UDP 'connection' (as configured)\n"); - pData->nXmit = 1; /* else we have an addtl wrap at 2^31-1 */ - CHKiRet(closeUDPSockets(pData)); + pWrkrData->nXmit = 1; /* else we have an addtl wrap at 2^31-1 */ + CHKiRet(closeUDPSockets(pWrkrData)); } - if(pData->pSockArray == NULL) { - CHKiRet(doTryResume(pData)); + if(pWrkrData->pSockArray == NULL) { + CHKiRet(doTryResume(pWrkrData)); } - if(pData->pSockArray != NULL) { + if(pWrkrData->pSockArray != NULL) { /* we need to track if we have success sending to the remote * peer. Success is indicated by at least one sendto() call * succeeding. We track this be bSendSuccess. We can not simply @@ -379,25 +422,37 @@ static rsRetVal UDPSend(instanceData *pData, char *msg, size_t len) * the sendto() succeeded. -- rgerhards, 2007-06-22 */ bSendSuccess = RSFALSE; - for (r = pData->f_addr; r; r = r->ai_next) { - for (i = 0; i < *pData->pSockArray; i++) { - lsent = sendto(pData->pSockArray[i+1], msg, len, 0, r->ai_addr, r->ai_addrlen); + for (r = pWrkrData->f_addr; r; r = r->ai_next) { + for (i = 0; i < *pWrkrData->pSockArray; i++) { + lsent = sendto(pWrkrData->pSockArray[i+1], msg, len, 0, r->ai_addr, r->ai_addrlen); if (lsent == len) { bSendSuccess = RSTRUE; break; } else { - int eno = errno; - char errStr[1024]; - dbgprintf("sendto() error: %d = %s.\n", - eno, rs_strerror_r(eno, errStr, sizeof(errStr))); + lasterrno = errno; + DBGPRINTF("sendto() error: %d = %s.\n", + lasterrno, + rs_strerror_r(lasterrno, errStr, sizeof(errStr))); } } if (lsent == len && !send_to_all) break; } /* finished looping */ - if (bSendSuccess == RSFALSE) { + if(bSendSuccess == RSFALSE) { dbgprintf("error forwarding via udp, suspending\n"); + if(pWrkrData->errsToReport > 0) { + rs_strerror_r(lasterrno, errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR_UDPSEND, "omfwd: error sending " + "via udp: %s", errStr); + if(pWrkrData->errsToReport == 1) { + errmsg.LogError(0, RS_RET_LAST_ERRREPORT, "omfwd: " + "max number of error message emitted " + "- further messages will be " + "suppressed"); + } + --pWrkrData->errsToReport; + } iRet = RS_RET_SUSPENDED; } } @@ -423,24 +478,19 @@ finalize_it: /* CODE FOR SENDING TCP MESSAGES */ - -/* Send a buffer via TCP. Usually, this is used to send the current - * send buffer, but if a message is larger than the buffer, we need to - * have the capability to send the message buffer directly. - * rgerhards, 2011-04-04 - */ static rsRetVal -TCPSendBuf(instanceData *pData, uchar *buf, unsigned len) +TCPSendBufUncompressed(wrkrInstanceData_t *pWrkrData, uchar *buf, unsigned len) { DEFiRet; unsigned alreadySent; ssize_t lenSend; alreadySent = 0; - CHKiRet(netstrm.CheckConnection(pData->pNetstrm)); /* hack for plain tcp syslog - see ptcp driver for details */ + CHKiRet(netstrm.CheckConnection(pWrkrData->pNetstrm)); /* hack for plain tcp syslog - see ptcp driver for details */ + while(alreadySent != len) { lenSend = len - alreadySent; - CHKiRet(netstrm.Send(pData->pNetstrm, buf+alreadySent, &lenSend)); + CHKiRet(netstrm.Send(pWrkrData->pNetstrm, buf+alreadySent, &lenSend)); DBGPRINTF("omfwd: TCP sent %ld bytes, requested %u\n", (long) lenSend, len - alreadySent); alreadySent += lenSend; } @@ -449,38 +499,136 @@ finalize_it: if(iRet != RS_RET_OK) { /* error! */ dbgprintf("TCPSendBuf error %d, destruct TCP Connection!\n", iRet); - DestructTCPInstanceData(pData); + DestructTCPInstanceData(pWrkrData); iRet = RS_RET_SUSPENDED; } RETiRet; } +static rsRetVal +TCPSendBufCompressed(wrkrInstanceData_t *pWrkrData, uchar *buf, unsigned len, sbool bIsFlush) +{ + int zRet; /* zlib return state */ + unsigned outavail; + uchar zipBuf[32*1024]; + int op; + DEFiRet; + + if(!pWrkrData->bzInitDone) { + /* allocate deflate state */ + pWrkrData->zstrm.zalloc = Z_NULL; + pWrkrData->zstrm.zfree = Z_NULL; + pWrkrData->zstrm.opaque = Z_NULL; + /* see note in file header for the params we use with deflateInit2() */ + zRet = deflateInit(&pWrkrData->zstrm, 9); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateInit()\n", zRet); + ABORT_FINALIZE(RS_RET_ZLIB_ERR); + } + pWrkrData->bzInitDone = RSTRUE; + } + + /* now doing the compression */ + pWrkrData->zstrm.next_in = (Bytef*) buf; + pWrkrData->zstrm.avail_in = len; + if(pWrkrData->pData->strmCompFlushOnTxEnd && bIsFlush) + op = Z_SYNC_FLUSH; + else + op = Z_NO_FLUSH; + /* run deflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("omfwd: in deflate() loop, avail_in %d, total_in %ld, isFlush %d\n", pWrkrData->zstrm.avail_in, pWrkrData->zstrm.total_in, bIsFlush); + pWrkrData->zstrm.avail_out = sizeof(zipBuf); + pWrkrData->zstrm.next_out = zipBuf; + zRet = deflate(&pWrkrData->zstrm, op); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pWrkrData->zstrm.avail_out); + outavail = sizeof(zipBuf) - pWrkrData->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(TCPSendBufUncompressed(pWrkrData, zipBuf, outavail)); + } + } while (pWrkrData->zstrm.avail_out == 0); + +finalize_it: + RETiRet; +} + +static rsRetVal +TCPSendBuf(wrkrInstanceData_t *pWrkrData, uchar *buf, unsigned len, sbool bIsFlush) +{ + DEFiRet; + if(pWrkrData->pData->compressionMode >= COMPRESS_STREAM_ALWAYS) + iRet = TCPSendBufCompressed(pWrkrData, buf, len, bIsFlush); + else + iRet = TCPSendBufUncompressed(pWrkrData, buf, len); + RETiRet; +} + +/* finish zlib buffer, to be called before closing the ZIP file (if + * running in stream mode). + */ +static rsRetVal +doZipFinish(wrkrInstanceData_t *pWrkrData) +{ + int zRet; /* zlib return state */ + DEFiRet; + unsigned outavail; + uchar zipBuf[32*1024]; + + if(!pWrkrData->bzInitDone) + goto done; + +// TODO: can we get this into a single common function? +dbgprintf("DDDD: in doZipFinish()\n"); + pWrkrData->zstrm.avail_in = 0; + /* run deflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", pWrkrData->zstrm.avail_in, pWrkrData->zstrm.total_in); + pWrkrData->zstrm.avail_out = sizeof(zipBuf); + pWrkrData->zstrm.next_out = zipBuf; + zRet = deflate(&pWrkrData->zstrm, Z_FINISH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, pWrkrData->zstrm.avail_out); + outavail = sizeof(zipBuf) - pWrkrData->zstrm.avail_out; + if(outavail != 0) { + CHKiRet(TCPSendBufUncompressed(pWrkrData, zipBuf, outavail)); + } + } while (pWrkrData->zstrm.avail_out == 0); + +finalize_it: + zRet = deflateEnd(&pWrkrData->zstrm); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); + } + + pWrkrData->bzInitDone = 0; +done: RETiRet; +} + /* Add frame to send buffer (or send, if requried) */ static rsRetVal TCPSendFrame(void *pvData, char *msg, size_t len) { DEFiRet; - instanceData *pData = (instanceData *) pvData; + wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t *) pvData; DBGPRINTF("omfwd: add %u bytes to send buffer (curr offs %u)\n", - (unsigned) len, pData->offsSndBuf); - if(pData->offsSndBuf != 0 && pData->offsSndBuf + len >= sizeof(pData->sndBuf)) { + (unsigned) len, pWrkrData->offsSndBuf); + if(pWrkrData->offsSndBuf != 0 && pWrkrData->offsSndBuf + len >= sizeof(pWrkrData->sndBuf)) { /* no buffer space left, need to commit previous records */ - CHKiRet(TCPSendBuf(pData, pData->sndBuf, pData->offsSndBuf)); - pData->offsSndBuf = 0; + CHKiRet(TCPSendBuf(pWrkrData, pWrkrData->sndBuf, pWrkrData->offsSndBuf, NO_FLUSH)); + pWrkrData->offsSndBuf = 0; iRet = RS_RET_PREVIOUS_COMMITTED; } /* check if the message is too large to fit into buffer */ - if(len > sizeof(pData->sndBuf)) { - CHKiRet(TCPSendBuf(pData, (uchar*)msg, len)); + if(len > sizeof(pWrkrData->sndBuf)) { + CHKiRet(TCPSendBuf(pWrkrData, (uchar*)msg, len, NO_FLUSH)); ABORT_FINALIZE(RS_RET_OK); /* committed everything so far */ } /* we now know the buffer has enough free space */ - memcpy(pData->sndBuf + pData->offsSndBuf, msg, len); - pData->offsSndBuf += len; + memcpy(pWrkrData->sndBuf + pWrkrData->offsSndBuf, msg, len); + pWrkrData->offsSndBuf += len; iRet = RS_RET_DEFER_COMMIT; finalize_it: @@ -495,11 +643,10 @@ finalize_it: static rsRetVal TCPSendPrepRetry(void *pvData) { DEFiRet; - instanceData *pData = (instanceData *) pvData; -dbgprintf("TCPSendPrepRetry performs a DestructTCPInstanceData\n"); + wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t *) pvData; - assert(pData != NULL); - DestructTCPInstanceData(pData); + assert(pWrkrData != NULL); + DestructTCPInstanceData(pWrkrData); RETiRet; } @@ -510,36 +657,39 @@ dbgprintf("TCPSendPrepRetry performs a DestructTCPInstanceData\n"); static rsRetVal TCPSendInit(void *pvData) { DEFiRet; - instanceData *pData = (instanceData *) pvData; + wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t *) pvData; + instanceData *pData; + + assert(pWrkrData != NULL); + pData = pWrkrData->pData; - assert(pData != NULL); - if(pData->pNetstrm == NULL) { + if(pWrkrData->pNetstrm == NULL) { dbgprintf("TCPSendInit CREATE\n"); - CHKiRet(netstrms.Construct(&pData->pNS)); + CHKiRet(netstrms.Construct(&pWrkrData->pNS)); /* the stream driver must be set before the object is finalized! */ - CHKiRet(netstrms.SetDrvrName(pData->pNS, pData->pszStrmDrvr)); - CHKiRet(netstrms.ConstructFinalize(pData->pNS)); + CHKiRet(netstrms.SetDrvrName(pWrkrData->pNS, pData->pszStrmDrvr)); + CHKiRet(netstrms.ConstructFinalize(pWrkrData->pNS)); /* now create the actual stream and connect to the server */ - CHKiRet(netstrms.CreateStrm(pData->pNS, &pData->pNetstrm)); - CHKiRet(netstrm.ConstructFinalize(pData->pNetstrm)); - CHKiRet(netstrm.SetDrvrMode(pData->pNetstrm, pData->iStrmDrvrMode)); + CHKiRet(netstrms.CreateStrm(pWrkrData->pNS, &pWrkrData->pNetstrm)); + CHKiRet(netstrm.ConstructFinalize(pWrkrData->pNetstrm)); + CHKiRet(netstrm.SetDrvrMode(pWrkrData->pNetstrm, pData->iStrmDrvrMode)); /* now set optional params, but only if they were actually configured */ if(pData->pszStrmDrvrAuthMode != NULL) { - CHKiRet(netstrm.SetDrvrAuthMode(pData->pNetstrm, pData->pszStrmDrvrAuthMode)); + CHKiRet(netstrm.SetDrvrAuthMode(pWrkrData->pNetstrm, pData->pszStrmDrvrAuthMode)); } if(pData->pPermPeers != NULL) { - CHKiRet(netstrm.SetDrvrPermPeers(pData->pNetstrm, pData->pPermPeers)); + CHKiRet(netstrm.SetDrvrPermPeers(pWrkrData->pNetstrm, pData->pPermPeers)); } /* params set, now connect */ - CHKiRet(netstrm.Connect(pData->pNetstrm, glbl.GetDefPFFamily(), + CHKiRet(netstrm.Connect(pWrkrData->pNetstrm, glbl.GetDefPFFamily(), (uchar*)pData->port, (uchar*)pData->target)); } finalize_it: if(iRet != RS_RET_OK) { dbgprintf("TCPSendInit FAILED with %d.\n", iRet); - DestructTCPInstanceData(pData); + DestructTCPInstanceData(pWrkrData); } RETiRet; @@ -549,15 +699,17 @@ finalize_it: /* try to resume connection if it is not ready * rgerhards, 2007-08-02 */ -static rsRetVal doTryResume(instanceData *pData) +static rsRetVal doTryResume(wrkrInstanceData_t *pWrkrData) { int iErr; struct addrinfo *res; struct addrinfo hints; + instanceData *pData; DEFiRet; - if(pData->bIsConnected) + if(pWrkrData->bIsConnected) FINALIZE; + pData = pWrkrData->pData; /* The remote address is not yet known and needs to be obtained */ dbgprintf(" %s\n", pData->target); @@ -573,20 +725,20 @@ static rsRetVal doTryResume(instanceData *pData) ABORT_FINALIZE(RS_RET_SUSPENDED); } dbgprintf("%s found, resuming.\n", pData->target); - pData->f_addr = res; - pData->bIsConnected = 1; - if(pData->pSockArray == NULL) { - pData->pSockArray = net.create_udp_socket((uchar*)pData->target, NULL, 0); + pWrkrData->f_addr = res; + pWrkrData->bIsConnected = 1; + if(pWrkrData->pSockArray == NULL) { + pWrkrData->pSockArray = net.create_udp_socket((uchar*)pData->target, NULL, 0, 0); } } else { - CHKiRet(TCPSendInit((void*)pData)); + CHKiRet(TCPSendInit((void*)pWrkrData)); } finalize_it: if(iRet != RS_RET_OK) { - if(pData->f_addr != NULL) { - freeaddrinfo(pData->f_addr); - pData->f_addr = NULL; + if(pWrkrData->f_addr != NULL) { + freeaddrinfo(pWrkrData->f_addr); + pWrkrData->f_addr = NULL; } iRet = RS_RET_SUSPENDED; } @@ -597,7 +749,8 @@ finalize_it: BEGINtryResume CODESTARTtryResume - iRet = doTryResume(pData); + dbgprintf("DDDD: tryResume: pWrkrData %p\n", pWrkrData); + iRet = doTryResume(pWrkrData); ENDtryResume @@ -614,8 +767,11 @@ BEGINdoAction # ifdef USE_NETZIP Bytef *out = NULL; /* for compression */ # endif + instanceData *pData; CODESTARTdoAction - CHKiRet(doTryResume(pData)); + dbgprintf("DDDD: doAction: pWrkrData %p\n", pWrkrData); + pData = pWrkrData->pData; + CHKiRet(doTryResume(pWrkrData)); iMaxLine = glbl.GetMaxLine(); @@ -636,11 +792,10 @@ CODESTARTdoAction * hard-coded but this may be changed to a config parameter. * rgerhards, 2006-11-30 */ - if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { + if(pData->compressionMode == COMPRESS_SINGLE_MSG && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { uLongf destLen = iMaxLine + iMaxLine/100 +12; /* recommended value from zlib doc */ uLong srcLen = l; int ret; - /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ CHKmalloc(out = (Bytef*) MALLOC(destLen)); out[0] = 'z'; out[1] = '\0'; @@ -669,16 +824,19 @@ CODESTARTdoAction if(pData->protocol == FORW_UDP) { /* forward via UDP */ - CHKiRet(UDPSend(pData, psz, l)); + CHKiRet(UDPSend(pWrkrData, psz, l)); } else { /* forward via TCP */ - iRet = tcpclt.Send(pData->pTCPClt, pData, psz, l); + iRet = tcpclt.Send(pWrkrData->pTCPClt, pWrkrData, psz, l); if(iRet != RS_RET_OK && iRet != RS_RET_DEFER_COMMIT && iRet != RS_RET_PREVIOUS_COMMITTED) { /* error! */ dbgprintf("error forwarding via tcp, suspending\n"); - DestructTCPInstanceData(pData); + DestructTCPInstanceData(pWrkrData); iRet = RS_RET_SUSPENDED; } + if(pData->compressionMode >= COMPRESS_STREAM_ALWAYS && pData->strmCompFlushOnTxEnd) + /* mimic not committed, as we need the EndTx entry point to be called */ + iRet = RS_RET_DEFER_COMMIT; } finalize_it: # ifdef USE_NETZIP @@ -689,10 +847,10 @@ ENDdoAction BEGINendTransaction CODESTARTendTransaction -dbgprintf("omfwd: endTransaction, offsSndBuf %u\n", pData->offsSndBuf); - if(pData->offsSndBuf != 0) { - iRet = TCPSendBuf(pData, pData->sndBuf, pData->offsSndBuf); - pData->offsSndBuf = 0; +dbgprintf("omfwd: endTransaction, offsSndBuf %u\n", pWrkrData->offsSndBuf); + if(pWrkrData->offsSndBuf != 0) { + iRet = TCPSendBuf(pWrkrData, pWrkrData->sndBuf, pWrkrData->offsSndBuf, IS_FLUSH); + pWrkrData->offsSndBuf = 0; } ENDendTransaction @@ -719,25 +877,22 @@ finalize_it: * created. */ static rsRetVal -initTCP(instanceData *pData) +initTCP(wrkrInstanceData_t *pWrkrData) { + instanceData *pData; DEFiRet; + + pData = pWrkrData->pData; if(pData->protocol == FORW_TCP) { /* create our tcpclt */ - CHKiRet(tcpclt.Construct(&pData->pTCPClt)); - CHKiRet(tcpclt.SetResendLastOnRecon(pData->pTCPClt, pData->bResendLastOnRecon)); + CHKiRet(tcpclt.Construct(&pWrkrData->pTCPClt)); + CHKiRet(tcpclt.SetResendLastOnRecon(pWrkrData->pTCPClt, pData->bResendLastOnRecon)); /* and set callbacks */ - CHKiRet(tcpclt.SetSendInit(pData->pTCPClt, TCPSendInit)); - CHKiRet(tcpclt.SetSendFrame(pData->pTCPClt, TCPSendFrame)); - CHKiRet(tcpclt.SetSendPrepRetry(pData->pTCPClt, TCPSendPrepRetry)); - CHKiRet(tcpclt.SetFraming(pData->pTCPClt, pData->tcp_framing)); - CHKiRet(tcpclt.SetRebindInterval(pData->pTCPClt, pData->iRebindInterval)); - pData->iStrmDrvrMode = cs.iStrmDrvrMode; - if(cs.pszStrmDrvr != NULL) - CHKmalloc(pData->pszStrmDrvr = (uchar*)strdup((char*)cs.pszStrmDrvr)); - if(cs.pszStrmDrvrAuthMode != NULL) - CHKmalloc(pData->pszStrmDrvrAuthMode = - (uchar*)strdup((char*)cs.pszStrmDrvrAuthMode)); + CHKiRet(tcpclt.SetSendInit(pWrkrData->pTCPClt, TCPSendInit)); + CHKiRet(tcpclt.SetSendFrame(pWrkrData->pTCPClt, TCPSendFrame)); + CHKiRet(tcpclt.SetSendPrepRetry(pWrkrData->pTCPClt, TCPSendPrepRetry)); + CHKiRet(tcpclt.SetFraming(pWrkrData->pTCPClt, pData->tcp_framing)); + CHKiRet(tcpclt.SetRebindInterval(pWrkrData->pTCPClt, pData->iRebindInterval)); } finalize_it: RETiRet; @@ -756,14 +911,19 @@ setInstParamDefaults(instanceData *pData) pData->iRebindInterval = 0; pData->bResendLastOnRecon = 0; pData->pPermPeers = NULL; - pData->compressionLevel = 0; + pData->compressionLevel = 9; + pData->strmCompFlushOnTxEnd = 1; + pData->compressionMode = COMPRESS_NEVER; + pData->errsToReport = 5; } BEGINnewActInst struct cnfparamvals *pvals; uchar *tplToUse; + char *cstr; int i; rsRetVal localRet; + int complevel = -1; CODESTARTnewActInst DBGPRINTF("newActInst (omfwd)\n"); @@ -860,9 +1020,10 @@ CODESTARTnewActInst free(str); } else if(!strcmp(actpblk.descr[i].name, "ziplevel")) { # ifdef USE_NETZIP - int complevel = pvals[i].val.d.n; + complevel = pvals[i].val.d.n; if(complevel >= 0 && complevel <= 10) { pData->compressionLevel = complevel; + pData->compressionMode = COMPRESS_SINGLE_MSG; } else { errmsg.LogError(0, NO_ERRCODE, "Invalid ziplevel %d specified in " "forwardig action - NOT turning on compression.", @@ -872,21 +1033,50 @@ CODESTARTnewActInst errmsg.LogError(0, NO_ERRCODE, "Compression requested, but rsyslogd is not compiled " "with compression support - request ignored."); # endif /* #ifdef USE_NETZIP */ + } else if(!strcmp(actpblk.descr[i].name, "maxerrormessages")) { + pData->errsToReport = (int) pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "resendlastmsgonreconnect")) { pData->bResendLastOnRecon = (int) pvals[i].val.d.n; } else if(!strcmp(actpblk.descr[i].name, "template")) { pData->tplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "compression.stream.flushontxend")) { + pData->strmCompFlushOnTxEnd = (sbool) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "compression.mode")) { + cstr = es_str2cstr(pvals[i].val.d.estr, NULL); + if(!strcasecmp(cstr, "stream:always")) { + pData->compressionMode = COMPRESS_STREAM_ALWAYS; + } else if(!strcasecmp(cstr, "none")) { + pData->compressionMode = COMPRESS_NEVER; + } else if(!strcasecmp(cstr, "single")) { + pData->compressionMode = COMPRESS_SINGLE_MSG; + } else { + errmsg.LogError(0, RS_RET_PARAM_ERROR, "omfwd: invalid value for 'compression.mode' " + "parameter (given is '%s')", cstr); + free(cstr); + ABORT_FINALIZE(RS_RET_PARAM_ERROR); + } + free(cstr); } else { DBGPRINTF("omfwd: program error, non-handled " "param '%s'\n", actpblk.descr[i].name); } } + + if(complevel != -1) { + pData->compressionLevel = complevel; + if(pData->compressionMode == COMPRESS_NEVER) { + /* to keep compatible with pre-7.3.11, only setting the + * compresion level means old-style single-message mode. + */ + pData->compressionMode = COMPRESS_SINGLE_MSG; + } + } + CODE_STD_STRING_REQUESTnewActInst(1) tplToUse = ustrdup((pData->tplName == NULL) ? getDfltTpl() : pData->tplName); CHKiRet(OMSRsetEntry(*ppOMSR, 0, tplToUse, OMSR_NO_RQD_TPL_OPTS)); - CHKiRet(initTCP(pData)); CODE_STD_FINALIZERnewActInst cnfparamvalsDestruct(pvals, &actpblk); ENDnewActInst @@ -948,6 +1138,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) iLevel = *p - '0'; ++p; /* eat */ pData->compressionLevel = iLevel; + pData->compressionMode = COMPRESS_SINGLE_MSG; } else { errmsg.LogError(0, NO_ERRCODE, "Invalid compression level '%c' specified in " "forwardig action - NOT turning on compression.", @@ -1025,7 +1216,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) while(*p && *p != ';' && *p != '#' && !isspace((int) *p)) ++p; /*JUST SKIP*/ - /* TODO: make this if go away! */ if(*p == ';' || *p == '#' || isspace(*p)) { uchar cTmp = *p; *p = '\0'; /* trick to obtain hostname (later)! */ @@ -1055,7 +1245,6 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) cs.pPermPeers = NULL; } } - CHKiRet(initTCP(pData)); CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -1091,6 +1280,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES diff --git a/tools/ompipe.c b/tools/ompipe.c index df8066b1..79f3ae84 100644 --- a/tools/ompipe.c +++ b/tools/ompipe.c @@ -12,7 +12,7 @@ * NOTE: read comments in module-template.h to understand how this pipe * works! * - * Copyright 2007-2012 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2013 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -69,9 +69,14 @@ typedef struct _instanceData { uchar *pipe; /* pipe or template name (display only) */ uchar *tplName; /* format template to use */ short fd; /* pipe descriptor for (current) pipe */ + pthread_mutex_t mutWrite; /* guard against multiple instances writing to same pipe */ sbool bHadError; /* did we already have/report an error on this pipe? */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { EMPTY_STRUCT } configSettings_t; @@ -276,25 +281,42 @@ CODESTARTcreateInstance pData->pipe = NULL; pData->fd = -1; pData->bHadError = 0; + pthread_mutex_init(&pData->mutWrite, NULL); ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINfreeInstance CODESTARTfreeInstance + pthread_mutex_destroy(&pData->mutWrite); free(pData->pipe); if(pData->fd != -1) close(pData->fd); ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINtryResume CODESTARTtryResume ENDtryResume BEGINdoAction + instanceData *pData; CODESTARTdoAction + pData = pWrkrData->pData; DBGPRINTF(" (%s)\n", pData->pipe); + /* this module is single-threaded by nature */ + pthread_mutex_lock(&pData->mutWrite); iRet = writePipe(ppString, pData); + pthread_mutex_unlock(&pData->mutWrite); ENDdoAction @@ -390,6 +412,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_doHUP CODEqueryEtryPt_STD_CONF2_QUERIES CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES diff --git a/tools/omshell.c b/tools/omshell.c index ac62fa62..ad6e979f 100644 --- a/tools/omshell.c +++ b/tools/omshell.c @@ -19,7 +19,7 @@ * of the "old" message code without any modifications. However, it * helps to have things at the right place one we go to the meat of it. * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -63,11 +63,20 @@ typedef struct _instanceData { uchar progName[MAXFNAME]; /* program to execute */ } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + BEGINcreateInstance CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -80,6 +89,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo printf("%s", pData->progName); @@ -92,13 +106,9 @@ ENDtryResume BEGINdoAction CODESTARTdoAction - /* TODO: using pData->progName is not clean from the point of - * modularization. We'll change that as we go ahead with modularization. - * rgerhards, 2007-07-20 - */ dbgprintf("\n"); - if(execProg((uchar*) pData->progName, 1, ppString[0]) == 0) - errmsg.LogError(0, NO_ERRCODE, "Executing program '%s' failed", (char*)pData->progName); + if(execProg((uchar*) pWrkrData->pData->progName, 1, ppString[0]) == 0) + errmsg.LogError(0, NO_ERRCODE, "Executing program '%s' failed", (char*)pWrkrData->pData->progName); ENDdoAction @@ -139,6 +149,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES ENDqueryEtryPt diff --git a/tools/omusrmsg.c b/tools/omusrmsg.c index f4cc4094..5d0b088f 100644 --- a/tools/omusrmsg.c +++ b/tools/omusrmsg.c @@ -8,7 +8,7 @@ * File begun on 2007-07-20 by RGerhards (extracted from syslogd.c, which at the * time of the fork from sysklogd was under BSD license) * - * Copyright 2007-2012 Adiscon GmbH. + * Copyright 2007-2013 Adiscon GmbH. * * This file is part of rsyslog. * @@ -87,6 +87,10 @@ typedef struct _instanceData { uchar *tplName; } instanceData; +typedef struct wrkrInstanceData { + instanceData *pData; +} wrkrInstanceData_t; + typedef struct configSettings_s { EMPTY_STRUCT } configSettings_t; @@ -115,6 +119,11 @@ CODESTARTcreateInstance ENDcreateInstance +BEGINcreateWrkrInstance +CODESTARTcreateWrkrInstance +ENDcreateWrkrInstance + + BEGINisCompatibleWithFeature CODESTARTisCompatibleWithFeature if(eFeat == sFEATURERepeatedMsgReduction) @@ -128,6 +137,11 @@ CODESTARTfreeInstance ENDfreeInstance +BEGINfreeWrkrInstance +CODESTARTfreeWrkrInstance +ENDfreeWrkrInstance + + BEGINdbgPrintInstInfo register int i; CODESTARTdbgPrintInstInfo @@ -276,7 +290,7 @@ ENDtryResume BEGINdoAction CODESTARTdoAction dbgprintf("\n"); - iRet = wallmsg(ppString[0], pData); + iRet = wallmsg(ppString[0], pWrkrData->pData); ENDdoAction @@ -435,6 +449,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_STD_OMOD8_QUERIES CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ENDqueryEtryPt diff --git a/tools/pmrfc3164.c b/tools/pmrfc3164.c index 5dfa74f0..25964702 100644 --- a/tools/pmrfc3164.c +++ b/tools/pmrfc3164.c @@ -84,7 +84,7 @@ CODESTARTparse assert(pMsg->pszRawMsg != NULL); lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; /* note: offAfterPRI is already the number of PRI chars (do not add one!) */ p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ - setProtocolVersion(pMsg, 0); + setProtocolVersion(pMsg, MSG_LEGACY_PROTOCOL); /* Check to see if msg contains a timestamp. We start by assuming * that the message timestamp is the time of reception (which we @@ -95,6 +95,8 @@ CODESTARTparse if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { /* we are done - parse pointer is moved by ParseTIMESTAMP3339 */; } else if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { + if(pMsg->dfltTZ[0] != '\0') + applyDfltTZ(&pMsg->tTIMESTAMP, pMsg->dfltTZ); /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; } else if(*p2parse == ' ' && lenMsg > 1) { /* try to see if it is slighly malformed - HP procurve seems to do that sometimes */ ++p2parse; /* move over space */ diff --git a/tools/pmrfc5424.c b/tools/pmrfc5424.c index 9b5c6165..8e9510f3 100644 --- a/tools/pmrfc5424.c +++ b/tools/pmrfc5424.c @@ -227,7 +227,7 @@ CODESTARTparse ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE); } DBGPRINTF("Message has RFC5424/syslog-protocol format.\n"); - setProtocolVersion(pMsg, 1); + setProtocolVersion(pMsg, MSG_RFC5424_PROTOCOL); p2parse += 2; lenMsg -= 2; diff --git a/tools/rsyslogd.8 b/tools/rsyslogd.8 index 620006f2..ac732b88 100644 --- a/tools/rsyslogd.8 +++ b/tools/rsyslogd.8 @@ -191,6 +191,10 @@ is specified and the host logging resolves to satu.infodrom.north.de no domain would be cut, you will have to specify two domains like: .BR "\-s north.de:infodrom.north.de" . .TP +.BI "\-S ip_address" "local client source IP" +rsyslogd uses ip_address as local client address while connecting +to remote logserver. Currently used by omrelp only and only with tcp. +.TP .BI "\-u " "userlevel" This is a "catch all" option for some very seldomly-used user settings. The "userlevel" variable selects multiple things. Add the specific values diff --git a/tools/syslogd.c b/tools/syslogd.c index a8a733d6..052a9329 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -497,24 +497,19 @@ finalize_it: * rgerhards, 2010-06-09 */ static inline rsRetVal -preprocessBatch(batch_t *pBatch) { +preprocessBatch(batch_t *pBatch, int *pbShutdownImmediate) { prop_t *ip; prop_t *fqdn; prop_t *localName; prop_t *propFromHost = NULL; prop_t *propFromHostIP = NULL; - int bSingleRuleset; - ruleset_t *batchRuleset; /* the ruleset used for all message inside the batch, if there is a single one */ int bIsPermitted; msg_t *pMsg; int i; rsRetVal localRet; DEFiRet; - bSingleRuleset = 1; - batchRuleset = (pBatch->nElem > 0) ? pBatch->pElem[0].pMsg->pRuleset : NULL; - - for(i = 0 ; i < pBatch->nElem && !*(pBatch->pbShutdownImmediate) ; i++) { + for(i = 0 ; i < pBatch->nElem && !*pbShutdownImmediate ; i++) { pMsg = pBatch->pElem[i].pMsg; if((pMsg->msgFlags & NEEDS_ACLCHK_U) != 0) { DBGPRINTF("msgConsumer: UDP ACL must be checked for message (hostname-based)\n"); @@ -539,12 +534,8 @@ preprocessBatch(batch_t *pBatch) { pBatch->eltState[i] = BATCH_STATE_DISC; } } - if(pMsg->pRuleset != batchRuleset) - bSingleRuleset = 0; } - batchSetSingleRuleset(pBatch, bSingleRuleset); - finalize_it: if(propFromHost != NULL) prop.Destruct(&propFromHost); @@ -560,17 +551,16 @@ finalize_it: * for the main queue. */ static rsRetVal -msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch, int *pbShutdownImmediate) +msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch, wti_t *pWti) { DEFiRet; assert(pBatch != NULL); - pBatch->pbShutdownImmediate = pbShutdownImmediate; /* TODO: move this to batch creation! */ - preprocessBatch(pBatch); - ruleset.ProcessBatch(pBatch); + preprocessBatch(pBatch, pWti->pbShutdownImmediate); + ruleset.ProcessBatch(pBatch, pWti); //TODO: the BATCH_STATE_COMM must be set somewhere down the road, but we //do not have this yet and so we emulate -- 2010-06-10 int i; - for(i = 0 ; i < pBatch->nElem && !*pbShutdownImmediate ; i++) { + for(i = 0 ; i < pBatch->nElem && !*pWti->pbShutdownImmediate ; i++) { pBatch->eltState[i] = BATCH_STATE_COMM; } RETiRet; @@ -1056,7 +1046,7 @@ finalize_it: * the time being (remember that we want to restructure config processing at large!). * rgerhards, 2009-10-27 */ -rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfparamvals *queueParams) +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct nvlst *lst) { struct queuefilenames_s *qfn; uchar *qfname = NULL; @@ -1072,7 +1062,7 @@ rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfpara /* name our main queue object (it's not fatal if it fails...) */ obj.SetName((obj_t*) (*ppQueue), pszQueueName); - if(queueParams == NULL) { /* use legacy parameters? */ + if(lst == NULL) { /* use legacy parameters? */ /* ... set some properties ... */ # define setQPROP(func, directive, data) \ CHKiRet_Hdlr(func(*ppQueue, data)) { \ @@ -1129,11 +1119,16 @@ rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName, struct cnfpara # undef setQPROPstr } else { /* use new style config! */ qqueueSetDefaultsRulesetQueue(*ppQueue); - qqueueApplyCnfParam(*ppQueue, queueParams); + qqueueApplyCnfParam(*ppQueue, lst); } + RETiRet; +} - /* ... and finally start the queue! */ - CHKiRet_Hdlr(qqueueStart(*ppQueue)) { +rsRetVal +startMainQueue(qqueue_t *pQueue) +{ + DEFiRet; + CHKiRet_Hdlr(qqueueStart(pQueue)) { /* no queue is fatal, we need to give up in that case... */ errmsg.LogError(0, iRet, "could not start (ruleset) main message queue"); \ } @@ -1253,6 +1248,7 @@ doHUP(void) queryLocalHostname(); /* re-read our name */ ruleset.IterateAllActions(ourConf, doHUPActions, NULL); + lookupDoHUP(); } @@ -1342,6 +1338,11 @@ static void printVersion(void) #else printf("\tuuid support:\t\t\t\tNo\n"); #endif +#ifdef HAVE_JSON_OBJECT_NEW_INT64 + printf("\tNumber of Bits in RainerScript integers: 64\n"); +#else + printf("\tNumber of Bits in RainerScript integers: 32 (due to too-old json-c lib)\n"); +#endif printf("\nSee http://www.rsyslog.com for more information.\n"); } @@ -1772,7 +1773,7 @@ int realMain(int argc, char **argv) * of other options, we do this during the inital option processing. * rgerhards, 2008-04-04 */ - while((ch = getopt(argc, argv, "46a:Ac:dDef:g:hi:l:m:M:nN:op:qQr::s:t:T:u:vwx")) != EOF) { + while((ch = getopt(argc, argv, "46a:Ac:dDef:g:hi:l:m:M:nN:op:qQr::s:S:t:T:u:vwx")) != EOF) { switch((char)ch) { case '4': case '6': @@ -1790,6 +1791,7 @@ int realMain(int argc, char **argv) case 'q': /* add hostname if DNS resolving has failed */ case 'Q': /* dont resolve hostnames in ACL to IPs */ case 's': + case 'S': /* Source IP for local client to be used on multihomed host */ case 'T': /* chroot on startup (primarily for testing) */ case 'u': /* misc user settings */ case 'w': /* disable disallowed host warnings */ @@ -1881,6 +1883,13 @@ int realMain(int argc, char **argv) case 'a': fprintf(stderr, "rsyslogd: error -a is no longer supported, use module imuxsock instead"); break; + case 'S': /* Source IP for local client to be used on multihomed host */ + if(glbl.GetSourceIPofLocalClient() != NULL) { + fprintf (stderr, "rsyslogd: Only one -S argument allowed, the first one is taken.\n"); + } else { + glbl.SetSourceIPofLocalClient((uchar*)arg); + } + break; case 'f': /* configuration file */ ConfFile = (uchar*) arg; break; |