From 6a45fa0c79e1b20fca72ad853bbec6d4470b8245 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Wed, 15 Feb 2006 09:15:27 +0000 Subject: doc updates for 1.12.2 --- doc/bugs.html | 5 - doc/features.html | 2 +- doc/rsyslog_php_syslog_ng.html | 296 ++++++++++++++++++++--------------------- doc/status.html | 80 +++++------ 4 files changed, 189 insertions(+), 194 deletions(-) diff --git a/doc/bugs.html b/doc/bugs.html index 617af9f8..1903ba5b 100644 --- a/doc/bugs.html +++ b/doc/bugs.html @@ -14,11 +14,6 @@ both.

This list has last been updated on 2005-09-26 by Rainer Gerhards.

rsyslogd

multi-threading and BSD

Multi-threading does currently not work reliably on BSD, at least -not upon program termination. It looks like this is related with the -way BSD handles signals. I am currently looking at it. In the mean -time, compile BSD builds without threading support (the default).

REPEATED LOG LINES

If multiple log lines with the exact same content are received, the duplicates are NOT suppressed. This is done by sysklogd diff --git a/doc/features.html b/doc/features.html index 58d6f0a2..3dba7a54 100644 --- a/doc/features.html +++ b/doc/features.html @@ -26,7 +26,7 @@ is going on, you can also subscribe to the syslog-protocol compliant messages (it is volatile because standardization is currently underway and this is a proof-of-concept implementation to aid this effort) +

ability to limit the allowed network senders

powerful BSD-style hostname and program name blocks for easy multi-host support

multi-threaded - currently experimental

very experimental and volatile support for syslog-protocol compliant messages (it is volatile because standardization is currently underway and this is a proof-of-concept implementation to aid this effort)

Upcoming Features

Using php-syslog-ng with rsyslog

Written by - Rainer - Gerhards (2005-08-04)

Abstract

In this paper, I describe how to use -php-syslog-ng with -rsyslogd. Php-syslog-ng is a -popular web interface to syslog data. Its name stem from the fact that it -usually picks up its data from a database created by -syslog-ng and some -helper scripts. However, there is nothing syslog-ng specific in the database. -With rsyslogd's high customizability, it is easy to write to a syslog-ng like -schema. I will tell you how to do this, enabling you to use php-syslog-ng as a -front-end for rsyslogd - or save the hassle with syslog-ng database -configuration and simply go ahead and use rsyslogd instead.

Overall System Setup

The setup is pretty straightforward. Basically, php-syslog-ng's interface to -the syslogd is the database. We use the schema that php-syslog-ng expects and -make rsyslogd write to it in its format. Because of this, php-syslog-ng does not -even know there is no syslog-ng present.

Setting up the system

For php-syslog-ng, you can follow its usual setup instructions. Just skip any -steps refering to configure syslog-ng. Make sure you create the database schema -in MySQL. As of this writing, the expected schema can be created via this script:

- CREATE DATABASE syslog - ! - USE syslog - ! - CREATE TABLE logs ( - host varchar(32) default NULL, - facility varchar(10) default NULL, - priority varchar(10) default NULL, - level varchar(10) default NULL, - tag varchar(10) default NULL, - date date default NULL, - time time default NULL, - program varchar(15) default NULL, - msg text, - seq int(10) unsigned NOT NULL auto_increment, - PRIMARY KEY (seq), - KEY host (host), - KEY seq (seq), - KEY program (program), - KEY time (time), - KEY date (date), - KEY priority (priority), - KEY facility (facility) - ) TYPE=MyISAM; -

Please note that at the time you are reading this paper, the schema might have changed. -Check for any differences. As we customize rsyslogd to the schema, it is vital -to have the correct one. If this paper is outdated, -let me know so that I can fix it.

Once this schema is created, we simply instruct rsyslogd to store received -data in it. I wont go into too much detail here. If you are interested in some -more details, you might find my paper "Writing -syslog messages to MySQL" worth reading. For this article, we simply modify -rsyslog.conf so that it writes to the database. -That is easy. Just these two lines are needed:

- $template syslog-ng,"insert into logs(host, facility, priority, tag, date, - time, msg) values ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, - %syslogtag%', '%timereported:::date-mysql%', '%timereported:::date-mysql%', - '%msg%')", SQL - *.* >mysql-server,syslog,user,pass;syslog-ng - -

These are just two lines. I have color-coded them so that you see what -belongs together (the colors have no other meaning). The green line is the -actual SQL statement being used to take care of the syslog-ng schema. Rsyslogd -allows you to fully control the statement sent to the database. This allows you -to write to any database format, including your homegrown one (if you so desire). -Please note that there is a small inefficiency in our current usage: the - '%timereported:::date-mysql%' -property is used for both the time and the date (if you wonder about what all -these funny characters mean, see the rsyslogd -property replacer manual) . We could have extracted just the date and time -parts of the respective properties. However, this is more complicated and also -adds processing time to rsyslogd's processing (substrings must be extracted). So we take a full mysql-formatted timestamp and supply it to MySQL. The sql engine in turn -discards the unneeded part. It works pretty well. As of my understanding, the -inefficiency of discarding the unneeded part in MySQL is lower than the -effciency gain from using the full timestamp in rsyslogd. So it is most probably -the best solution.

Please note that rsyslogd knows two different timestamp properties: one is -timereported, used here. It is the timestamp from the message itself. Sometimes -that is a good choice, in other cases not. It depends on your environment. The other one is the timegenerated -property. This is the time when rsyslogd received the message. For obvious -reasons, that timestamp is consistent, even when your devices are in multiple -time zones or their clocks are off. However, it is not "the real thing". It's -your choice which one you prefer. If you prefer timegenerated ... simply use it -;)

The line in red tells rsyslogd which messages to log and where to store it. -The "*.*" selects all messages. You can use standard syslog selector line filters here if -you do not like to see everything in your database. The ">" tells -rsyslogd that a MySQL connection -must be established. Then, "mysql-server" is the name or IP address of the -server machine, "syslog" is the database name (default from the schema) and "user" -and "pass" are the logon credentials. Use a user with low privileges, insert into the -logs table is sufficient. "syslog-ng" is the template name and tells rsyslogd to -use the SQL statement shown above.

Once you have made the changes, all you need to do is reload (or HUP) -rsyslogd. Then, you should see syslog messages flow into your database - and -show up in php-syslog-ng.

Conclusion

With minumal effort, you can use php-syslog-ng together with rsyslogd. For -those unfamiliar with syslog-ng, this configuration is probably easier to set up -then switching to syslog-ng. For existing rsyslogd users, php-syslog-ng might be a nice -add-on to their logging infrastructure.

Please note that the MonitorWare family (to which rsyslog belongs) also -offers a web-interface: phpLogCon. At the time of this writing, phpLogCon's code -is by far not as clean as I would like it to be. Also the user-interface is -definitely not as intutive as pp-syslog-ng. From a functionality point of view, -however, I think it already is a bit ahead. So you might -consider using it. I have set up a demo server., -You can have a peek at it -without installing anything.

Feedback Requested

I would appreciate feedback on this paper. If you have additional ideas, -comments or find bugs, please -let me know.

References and Additional Material

php-syslog-ng

Revision History

2005-08-04 * - Rainer Gerhards * - initial version created

Copyright

Permission is granted to copy, distribute and/or modify this document under -the terms of the GNU Free Documentation License, Version 1.2 or any later -version published by the Free Software Foundation; with no Invariant Sections, -no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be -viewed at -http://www.gnu.org/copyleft/fdl.html.

Using php-syslog-ng with rsyslog

Written by + Rainer + Gerhards (2005-08-04)

Abstract

In this paper, I describe how to use +php-syslog-ng with +rsyslogd. Php-syslog-ng is a +popular web interface to syslog data. Its name stem from the fact that it +usually picks up its data from a database created by +syslog-ng and some +helper scripts. However, there is nothing syslog-ng specific in the database. +With rsyslogd's high customizability, it is easy to write to a syslog-ng like +schema. I will tell you how to do this, enabling you to use php-syslog-ng as a +front-end for rsyslogd - or save the hassle with syslog-ng database +configuration and simply go ahead and use rsyslogd instead.

Overall System Setup

The setup is pretty straightforward. Basically, php-syslog-ng's interface to +the syslogd is the database. We use the schema that php-syslog-ng expects and +make rsyslogd write to it in its format. Because of this, php-syslog-ng does not +even know there is no syslog-ng present.

Setting up the system

For php-syslog-ng, you can follow its usual setup instructions. Just skip any +steps refering to configure syslog-ng. Make sure you create the database schema +in MySQL. As of this writing, the expected schema can be created via this script:

+ CREATE DATABASE syslog + ! + USE syslog + ! + CREATE TABLE logs ( + host varchar(32) default NULL, + facility varchar(10) default NULL, + priority varchar(10) default NULL, + level varchar(10) default NULL, + tag varchar(10) default NULL, + date date default NULL, + time time default NULL, + program varchar(15) default NULL, + msg text, + seq int(10) unsigned NOT NULL auto_increment, + PRIMARY KEY (seq), + KEY host (host), + KEY seq (seq), + KEY program (program), + KEY time (time), + KEY date (date), + KEY priority (priority), + KEY facility (facility) + ) TYPE=MyISAM; +

Please note that at the time you are reading this paper, the schema might have changed. +Check for any differences. As we customize rsyslogd to the schema, it is vital +to have the correct one. If this paper is outdated, +let me know so that I can fix it.

Once this schema is created, we simply instruct rsyslogd to store received +data in it. I wont go into too much detail here. If you are interested in some +more details, you might find my paper "Writing +syslog messages to MySQL" worth reading. For this article, we simply modify +rsyslog.conf so that it writes to the database. +That is easy. Just these two lines are needed:

+ $template syslog-ng,"insert into logs(host, facility, priority, tag, date, + time, msg) values ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, + '%syslogtag%', '%timereported:::date-mysql%', '%timereported:::date-mysql%', + '%msg%')", SQL + *.* >mysql-server,syslog,user,pass;syslog-ng + +

These are just two lines. I have color-coded them so that you see what +belongs together (the colors have no other meaning). The green line is the +actual SQL statement being used to take care of the syslog-ng schema. Rsyslogd +allows you to fully control the statement sent to the database. This allows you +to write to any database format, including your homegrown one (if you so desire). +Please note that there is a small inefficiency in our current usage: the + '%timereported:::date-mysql%' +property is used for both the time and the date (if you wonder about what all +these funny characters mean, see the rsyslogd +property replacer manual) . We could have extracted just the date and time +parts of the respective properties. However, this is more complicated and also +adds processing time to rsyslogd's processing (substrings must be extracted). So we take a full mysql-formatted timestamp and supply it to MySQL. The sql engine in turn +discards the unneeded part. It works pretty well. As of my understanding, the +inefficiency of discarding the unneeded part in MySQL is lower than the +effciency gain from using the full timestamp in rsyslogd. So it is most probably +the best solution.

Please note that rsyslogd knows two different timestamp properties: one is +timereported, used here. It is the timestamp from the message itself. Sometimes +that is a good choice, in other cases not. It depends on your environment. The other one is the timegenerated +property. This is the time when rsyslogd received the message. For obvious +reasons, that timestamp is consistent, even when your devices are in multiple +time zones or their clocks are off. However, it is not "the real thing". It's +your choice which one you prefer. If you prefer timegenerated ... simply use it +;)

The line in red tells rsyslogd which messages to log and where to store it. +The "*.*" selects all messages. You can use standard syslog selector line filters here if +you do not like to see everything in your database. The ">" tells +rsyslogd that a MySQL connection +must be established. Then, "mysql-server" is the name or IP address of the +server machine, "syslog" is the database name (default from the schema) and "user" +and "pass" are the logon credentials. Use a user with low privileges, insert into the +logs table is sufficient. "syslog-ng" is the template name and tells rsyslogd to +use the SQL statement shown above.

Once you have made the changes, all you need to do is reload (or HUP) +rsyslogd. Then, you should see syslog messages flow into your database - and +show up in php-syslog-ng.

Conclusion

With minumal effort, you can use php-syslog-ng together with rsyslogd. For +those unfamiliar with syslog-ng, this configuration is probably easier to set up +then switching to syslog-ng. For existing rsyslogd users, php-syslog-ng might be a nice +add-on to their logging infrastructure.

Please note that the MonitorWare family (to which rsyslog belongs) also +offers a web-interface: phpLogCon. At the time of this writing, phpLogCon's code +is by far not as clean as I would like it to be. Also the user-interface is +definitely not as intutive as pp-syslog-ng. From a functionality point of view, +however, I think it already is a bit ahead. So you might +consider using it. I have set up a demo server., +You can have a peek at it +without installing anything.

Feedback Requested

I would appreciate feedback on this paper. If you have additional ideas, +comments or find bugs, please +let me know.

References and Additional Material

php-syslog-ng

Revision History

2005-08-04 * + Rainer Gerhards * + initial version created

Copyright

Permission is granted to copy, distribute and/or modify this document under +the terms of the GNU Free Documentation License, Version 1.2 or any later +version published by the Free Software Foundation; with no Invariant Sections, +no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be +viewed at +http://www.gnu.org/copyleft/fdl.html.

rsyslog status page

This page reflects the status as of 2005-11-23.

Current Releases

development: 1.12.1 - change log - -download

stable: 1.0.3 - change log - -download

(How are versions named?)

Do NOT use versions prior to 1.10.1 or 1.0.1, -because they contain a SQL injection vulnerability (read -security advisory).

Platforms

Linux [tested on Red Hat & Debian]
BSD [tested on -FreeBSD and a quick Test on NetBSD, 0.8.4 only]

Additional information

Currently supported features are now listed on the rsyslog features page.

The rsyslog home page is www.rsyslog.com.
Mailing list info can be found at - http://lists.adiscon.com/rsyslog.
The change log can be found at - - http://www.rsyslog.com/Topic4.phtml.
You may also find Rainer's - syslog blog an interesting read.

The project was initiated in 2004 by -Rainer Gerhards -and is currently being maintained by him. See the history -page for more background information.

rsyslog status page

This page reflects the status as of 2006-02-15.

Current Releases

development: 1.12.2 - change log - +download

stable: 1.0.4 - change log - +download

(How are versions named?)

Do NOT use versions prior to 1.10.1 or 1.0.1, +because they contain a SQL injection vulnerability (read +security advisory).

Platforms

Linux [tested on Red Hat & Debian]
BSD [tested on +FreeBSD and a quick Test on NetBSD, 0.8.4 only]

Additional information

Currently supported features are now listed on the rsyslog features page.

The rsyslog home page is www.rsyslog.com.
Mailing list info can be found at + http://lists.adiscon.com/rsyslog.
The change log can be found at + + http://www.rsyslog.com/Topic4.phtml.
You may also find Rainer's + syslog blog an interesting read.

The project was initiated in 2004 by +Rainer Gerhards +and is currently being maintained by him. See the history +page for more background information.