What is LoggerFS?
LoggerFS is a FUSE-based virtual file system written in C++ using the FUSEXX C++ bindings. It seamlessly passes log data through the file system and directly into a database. Unlike existing log parsers, which often run periodically and scan the entire file for changes, LoggerFS takes a unique approach by masking the database backend with a filesystem frontend. When log lines are appended to a virtual file on the LoggerFS file system, lines that match a user-specified regex pattern are stored in a user-defined database. No need to poll the log file to monitor changes, simply prepare a target database, configure the file system using easy-to-read XML files, mount, and go!
Read more to find out how to use this exciting new log management solution!
How LoggerFS Works: A Quick Overview
It doesn't take long to get LoggerFS up and running, although it may require some planning, depending on how many and what format your log files are in. Before you begin, make sure you've installed and configured either PostgreSQL or MySQL. LoggerFS, when started, parses it's configuration files (
schemas.xml and
logs.xml) to determine what virtual log files and database connections it needs to create, so be sure to have your respective SQL daemon started and pre-configured to accept connections as define in the
logs.xml file. You start (that is, mount) LoggerFS filesystems just like other FUSE based filesystems:
# loggerfs /var/log/loggerfs
Now you must tell your applications to use the LoggerFS virtual log files. Note that in order for this to work, you must mount the LoggerFS filesystem before you start any services you wish to log to LoggerFS virtual files! Altering the log file paths of your services should be fairly straightforward; refer to the service in question's man pages or documentation for how to do this.
Once you've configured your services, start them up and monitor your SQL server for traffic. Be sure to perform an action that generates log data for your daemon, such as visiting the website on your Apache server. If all goes well, you should see rows being inserted into the tables you defined in logs.xml!
Quick Installation Methods
Download the appropriate package for your architecture/distribution from the LoggerFS SourceForge Page.
The debian packages can be installed by running the following as root:
# dpkg -i <package-name>.deb && apt-get -f install
Or download the .tar.gz'ed source code and...
# cd loggerfs-0.5
# ./configure --with-mysql
# make && make install
LoggerFS is now installed and ready to go!
Configuring LoggerFS: A Quick Rundown
LoggerFS is a remarkably simple but unique approach to managing log files. The log files themselves are defined in the logs.xml file. Each <log> entry contains the following properties:
- A <location> relative to where LoggerFS is mounted.
- The <uid> of the file (should probably be owned by the service that will log to it, ie, apache, mysql, squid, etc - can be either the UNIX account name or the numeric UID).
- The <gid> of the file (again, either UNIX account name or numeric GID).
- The four-digit octal <permissions> of the file (note that LoggerFS virtual files are WRITE only, ie, 0200, or 0220, etc).
- A <schemas> which links to a specific regex pattern defined in the schemas.xml file (more on this later)
- A <database-software>that tells LoggerFS what kind of database you're using
- A <database> that points LoggerFS to the database your logs will be stored in
- A corresponding <table> line points LoggerFS to the table of the above database
- A <server> that the database is stored on
- The <port> the database server is listening on.
- A <username> that LoggerFS connects to to the database as
- and a corresponding <password> to identify with.
By adding multiple <log> entries, multiple virtual log files will be created. At this time, creating subdirectories and additional files isn't possible, but may be added in a future release, allowing for the entire /var/log directory structure to be replicated in LoggerFS. For example, here are two log file entries included with LoggerFS, to match the apache_access.log and auth.log log files found on many systems:
<!-- logs.xml -->
<logs>
<!-- Sample PostgreSQL log files -->
<log>
<location>apache_access.log</location>
<uid>apache</uid>
<gid>apache</gid>
<permissions>0220</permissions>
<schemas>apache_combined,http_common</schemas>
<database-software>pgsql</database-software>
<database>loggerfs</database>
<table>apache</table>
<server>localhost</server>
<port>5432</port>
<username>postgres</username>
<password></password>
</log>
<log>
<location>auth.log</location>
<schemas>all</schemas>
<database-software>pgsql</database-software>
<database>loggerfs</database>
<table>authlog</table>
<server>localhost</server>
<username></username>
<password></password>
</log>
<!-- Sample MySQL log files -->
<log>
<location>syslog</location>
<schemas>all</schemas>
<database-software>mysql</database-software>
<database>loggerfs</database>
<table>syslog</table>
<server>localhost</server>
<username>root</username>
<password></password>
</log>
</logs>
Code 1: The logs.xml file
As stated before, each log file uses a specific
<schema>, which defines a regular expression pattern you wish to match against. LoggerFS virtual log files only support one schema at this time, but that should be enough for most applications. In the
schemas.xml configuration file, each schema contains a regex pattern and corresponding table columns to store the regex matches in. Below are some example schemas provided with LoggerFS. The first is a catch-all schema, aptly named [b]all, followed by expressions to match Apache's Combined and Common log formats. Squid, PostgreSQL, and Syslog schemas are also shown below:
<!-- schemas.xml -->
<schemas>
<schema>
<name>all</name>
<regex>(.*)</regex>
<columns>text</columns>
</schema>
<schema>
<name>apache_combined</name>
<regex>(\S+) (\S+) (\S+) \[.+\] "(.+)"+ (\S+) (\S+) "(\S+)" "(.+)"$</regex>
<columns>dst_ip,logname,authname,request,http_status,bytes,referer,user_agent</columns>
</schema>
<schema>
<name>http_common</name>
<regex>(\S+) (\S+) (\S+) \[.+\] "(.+)"+ (\S+) (\S+)$</regex>
<columns>dst_ip,logname,authname,request,http_status,bytes</columns>
</schema>
<schema>
<name>squid_native</name>
<regex>^\S+ +(\d+) (\S+) (\S+)\/(\d+) (\d+) (\S+ \S+) (\S+) (\S+)\/(\S+) (\S+)$</regex>
<columns>elapsed,src_ip,result_code,http_status,bytes,request,authname,hierarchy_code,dst_ip,type</columns>
</schema>
<schema>
<name>postgresql</name>
<regex>(\S+): *(.*)$</regex>
<columns>event,message</columns>
</schema>
<schema>
<name>syslog</name>
<regex>^\S+ *\d{1,2} \d+:\d+:\d+ (\S+) ([a-zA-Z\-\/\.]+).*?: *(.*)$</regex>
<columns>hostname,process,message</columns>
</schema>
</schemas>
Code 2: The schemas.xml file
Troubleshooting LoggerFS
I admit, it can be frustrating getting LoggerFS to work properly because it has a lot of things that need to "just work" in order for LoggerFS to function properly. Below are some things to consider the first time you deploy LoggerFS:
- Check that your database server is listening on the correct IP address and port (hint: use the netstat -natp command as root). Recall that LoggerFS currently only supports the default port (3306 for MySQL and 5432 for PostgreSQL).
- Be sure that database authentication is working properly, as specified in your logs.xml file. Try it out on the command line to make sure you can authenticate with the SQL server.
- Double check that you've specified the database schema correctly, as defined in the schemas.xml file.
- Test some lines of your log files against the regular expression you defined in your schemas.xmlp file. Use online tools such as RegexTester to verify your expressions work properly.
- Make sure that whatever daemon or server you have that's outputting your logs has permission to write to the virtual log file; this includes thing like user/group ownership and file permissions (must have write privileges, obviously).
- Make sure that LoggerFS is mounted ;)
- As root, try cat'ing a file that matches your regex into the LoggerFS virtual file. Monitor your SQL server to make sure that the rows are being inserted (or just run top and watch your CPU activity). This is a simple and efficient way to test and benchmark your system.
Be sure to read the follow up to this story: Configuring a LAMP Server to utilize LoggerFS!
If you need additional help, please ask in the comments below.
Nice
I have found two interesting sources and would like to give the benefit of my experience to you.
I am tuning my pc by the best software for free, with the file search engine Fileshunt.com and Filesfinds.com May be you have your own experience and could give some useful sites too. Because this two social sites help me much.
"Check that your database
"Check that your database server is listening on the correct IP address and port (hint: use the netstat -natp command as root). Recall that LoggerFS currently only supports the default port (3306 for MySQL and 5432 for PostgreSQL)." for sale site
Is that the default port for PostgreSQL?
work
QuickStarters are controversial because they use up RAM even when the application isn't open. For this reason, QuickStarters can actually decrease performance as a whole. For example, if six applications load a QuickStarter into the System Tray at 50 megabytes per piece, that is 300 megabytes of RAM that could otherwise be used for working applications. If the user only has 256 megabytes of RAM, this will cause running applications to page to virtual memory and drastically reduce performance. scarpe lavoro
Post new comment