The Open Source Journal

In this follow up to LoggerFS: A Revolutionary Approach to Logging, I will guide you through a complete installation of LoggerFS on a RedHat Enterprise Linux 5 LAMP server running stock RHEL5 software to store Apache combined log format logs, including the installation of FUSE and creation of the MySQL LoggerFS database.

Requirements

Throughout this guide, I will assume you have the following:

1. A RedHat Enterprise Linux 5 Server or Workstation with an up-to-date kernel (at the time of this writing, 2.6.18-8.1.14.el5 is the latest kernel)
2. Root access to said system
3. The latest available PHP, MySQL, and Apache packages available to RHEL5 via the yum interface. Other versions will probably work, but for the sake of consistency I will emphasize the versions listed below and their dependencies:
   
   1. http
   2. mysql-server
   3. php
4. and finally, I will assume that you've previously configured (or can configure on your own after installation) MySQL and Apache.

Before you begin...

And have installed the necessary packages for your LAMP server:

The following packages will be required to build LoggerFS:

[root@localhost ~] # yum -y install boost-devel make gcc-c++ pkgconfig libxml2-devel

Finally it might also be a good idea to get your system fully up-to-date if you haven't already done so:

[root@localhost ~] # yum -y update

Installing the FUSE Module

Before we can use LoggerFS, we need to get the FUSE module installed. Since LoggerFS is based on FUSE, you'll need to also install the FUSE libraries and FUSE development files, so that you can compile LoggerFS. Use the links below to download them from rpm.pbone.net

FUSE Kernel Modules [x86_64] [i386]
FUSE Development [x86_64] [i386]
FUSE Libraries [x86_64] [i386]

Check that the kernel module loads properly:

Downloading and Installing LoggerFS

Download LoggerFS (the latest version as of this writing is 0.5) from the [url=http://sourceforge.net/project/platformdownload.php?group_id=200449]LoggerFS SourceForge Page[/h2].

Compilation and installation should be familiar to those with prior Linux experience. Be sure to include the proper flags for configure; you'll need to use --with-mysql for MySQL support, but if you wish you can also specific --with-postgresql for PostgreSQL support:

Once installation is completed (if it fails, make sure you've installed mysql, boost-devel, make, gcc-c++, and libxml2-devel!) you can begin configuring LoggerFS for your system!

Preparing Apache for LoggerFS

For your first installation, you may also wish to move the error_log (ie, set ErrorLog /tmp/error_log in httpd.conf) file to a temporary location, or else Apache will fail to start:

Preparing MySQL for LoggerFS

Since LoggerFS will automatically create the table and columns, we only need to instruct MySQL to create a database and user with permissions on that database. We'll name our database myApacheLogs and create a user to insert data into it named apachelogger and use s3cr3tp4zzw0rd as the password. The SQL statements below will do this for us:

Now start MySQL by running # /etc/init.d/mysqld start- it has to be running before you can use LoggerFS!

Configuring LoggerFS

As you may recall, there are only two configuration files for LoggerFS: logs.xml and schemas.xml. When using the --prefix=/ configure option, these logs are stored in /etc/loggerfs; otherwise they will be placed in /usr/local/etc/loggerfs. Throughout this tutorial I'll expect them to be installed in /etc/loggerfs.

Let's create these files from scratch for practice. Either delete or zero the existing configuration files:

We'll start with logs.xml. RHEL5 stores Apache log files in /var/log/httpd and all Log paths in httpd.conf are relative to that path, so we'll set the log file location to simply access_log. Apache by default (at least in RHEL5) uses combined format, so we'll call our schema apache_combined. Since the apache server will need permission to write to these log files, we'll trust that the user apache and anyone in the apache group with write permissions by settings <permissions> to 0220. Obviously we're using MySQL for our LAMP server, so we'll say MySQL is listening on localhost and on the default port (3306). Finally, we set the database, username, and password to match what we created when setting up MySQL:

Onto schemas.xml. When creating this file, it's important to consider the data you're trying to log; for example, here's an apache line in Combined Format:

First, we identify what parts of the line are important to us; what do we want to log? Then we write our regular expression out and give each match a column name. For our LAMP server, the following will suffice as a regular expression:

Now we'll name each match of the regular expression. These names will become the column names of the accessLogs table (specified above in logs.xml).

We've already named this schema apache_combined in the logs.xml, so we have everything we need to create our schemas.xml file:

Running and Testing LoggerFS

Be sure to delete any existing logs from /var/log/httpd or LoggerFS will complain. The command to mount a LoggerFS file system is not unlike that for other FUSE-based file systems:

Now that apache is running, just generate some traffic on your server! You can view what kind of traffic you're experiencing with the following command:

# mysql -e 'select * from accessLogs limit 5' -u apachelogger -Ps3cr3tp4zzw0rd -p myApacheLogs

If you see a bunch of text spit out that looks vaguely like apache log data, you're done!