[ Top | Up | Prev | Next | Map | Index ]

Readme for analog3.32

Choosing a logfile

The basic command for selecting a logfile is

LOGFILE logfilename

or just to put the logfile name on the command line without any arguments, e.g., analog logfilename. A - sign or the word stdin is interpreted as standard input: this is useful on Unix systems for constructing pipes. All logfiles must be within your computer's file system (on disk, or at least mounted under Unix, or on a mapped drive under NT) -- analog won't use FTP or HTTP to fetch them from the internet. In the Mac version, you can also analyse a particular single logfile by dragging it onto the analog icon.

You can have several LOGFILE commands. You can include wildcards in the logfile name (but not necessarily in the directory name: this is system-dependent), and you can use a list of logfiles separated by commas (without spaces). So the following commands would tell analog to read logfile1, c:\logs\logfile2, and all files ending in .log:

LOGFILE logfile1,*.log
LOGFILE c:\logs\logfile2

Or if you were on a Mac, you might use something like

LOGFILE "Hard Drive:Internet Applications:Analog:Logs:*"

The LOGFILE commands are cumulative, except that any logfiles on the command line or in user-specified configuration files override any in the default configuration file, and are themselves overridden by any in the mandatory configuration file. There is also the special command

LOGFILE none

which erases the list of logfiles specified so far.

Analog knows about several different types of logfile. By default it will attempt to see if your logfile is of one of the types it knows about, based on the first line. The types it can usually diagnose are the common log format, the NCSA combined format, referrer log and browser log, the W3 extended log format, the Microsoft IIS format, the Netscape format, the WebSTAR format and the WebSite format. Examples of all these formats are given at the end of this section. If you have debugging on, analog will report what type of logfile it thinks yours is.

If your logfile is not in one of the standard formats, you will probably still be OK, because it is possible to tell analog about other formats using a LOGFORMAT command. This is explained in the next section. But most users don't ever need to know about this because they have logfiles in a standard format. So the best thing to do is just to try analysing your logfile and see if analog will understand it. If it does, you don't need to worry about LOGFORMATs.

If analog can't understand your logfile, it will warn you that it can't detect the format, or possibly that it found a lot of corrupt lines. There are basically four reasons why this might happen:

Since analog tries to deduce the format based on the first line of the logfile, it could just be that the first line is corrupt. In this case, you could tell analog the format, or you could just fix the first line.
For the same reason, if the format changes midway through the log, analog will count the remaining lines as corrupt. In this case, you will find that your report contains a partial analysis but with a large number of corrupt lines too. You will need to give analog two LOGFORMAT commands to tell it about the two different formats.
Some log formats are not very well designed and analog can't analyse them reliably. In this case it will give up rather than risk doing a bad job, usually with a helpful message. So if you believe that your logfile is in a standard format, but analog still can't analyse it, you should read the notes on all the built-in formats below where some common problems with those formats are described.
Finally, some logfiles really aren't in one of the standard formats. In this case you will need to read the next section and learn how to tell analog about your format.

There's also a second argument to the logfile command, which specifies a prefix to add to all the filenames in that logfile. This is useful if you've got several different servers or virtual hosts, when the same filename may occur on each of the servers. The argument can contain a %v, and the name of the virtual host will then be inserted at that point. For example,

LOGFILE log1,log2 http://www.%v.mydomain.com

would translate a filename /file.html with virtual host host1 in log1 or log2 to http://www.host1.mydomain.com/file.html. If you are using the second argument to the LOGFILE command, you will probably want to use the SUBDIR command as well.

If %v is included in the argument and the logfile line doesn't have a virtual host, that line will be marked as corrupt. If VHOSTLOWMEM 3 is specified, the %v's will not be translated and will just appear as %v in the output.

It is often convenient to store logfiles compressed to save disk space. Analog on the Mac can read logfiles compressed using gzip. And analog on Unix, Win32, and VMS 7.0 and above can read compressed logfiles provided that you use an UNCOMPRESS command to say how to uncompress them. You need to supply the types of file that you want to uncompress in a comma-separated list, together with the name of a command that will uncompress the files to standard output (rather than to a file). For example, on Unix you might use

UNCOMPRESS *.gz,*.Z  /usr/bin/gzcat

whereas on Windows NT, you might use

UNCOMPRESS *.gz "c:\Program Files\gzip\gzip -cd"

and on VMS, it could be

UNCOMPRESS *.LOG-GZ;*  "gunzip -c"

This would be a suitable command to include in the default configuration file.

If analog determines when it starts to uncompress a logfile that that file isn't wanted for the analysis, two undesirable things can happen. Either the program might pause until the logfile is fully uncompressed, or there might be a "broken pipe" error reported. This is system dependent, and out of analog's control.

Logfile formats

Here is a summary of the various logfile formats which analog knows about. To illustrate them, I have used the same (fictional) request as it might be recorded in the different formats.

The common logfile format is written by most servers. Its lines look like

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243

Some versions of Microsoft software have a buggy version of this with an extra quote mark before the HTTP like this:

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ "HTTP/1.0" 200 1243

Analog will understand these, but (as with any two formats) it will reject lines if the format changes half way through.

The NCSA referrer log looks like

[25/Dec/1998:17:45:35] http://www.site.com/ -> /~sret1/

and the browser (or agent) log looks like

[25/Dec/1998:17:45:35] Mozilla/2.0 (X11; I; HP-UX A.09.05)

In the referrer log, the date can be omitted.

The NCSA combined log is the same as the common log, except that it has the referrer and browser on the end in quotes, like this:

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200 1243
"http://www.site.com/" "Mozilla/2.0 (X11; I; HP-UX A.09.05)"

except all one line. If you are using the Apache server, you can generate this with the mod_log_config module, using the command

LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\""

It is usually better to use the combined log than separate logs, because it stores more information in less space.

The Microsoft IIS logfile looks like

192.64.25.41, -, 25/12/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
30, 303, 1243, 200, 0, GET, /~sret1/, -,

(except all on one line). However, the format is extremely badly designed, in that the date follows local conventions: in other words, in North America the above example would have the date 12/25/98 instead. Analog will diagnose which form the logfile is in if possible: but if both the date and the month are at most 12, there is no way to tell which format it is. In this case, it will advise you to use the command LOGFORMAT MICROSOFT-NA for North American date format, or LOGFORMAT MICROSOFT-INT for international date format. In some countries, the date will not be in either of these formats, in which case you need to write your own LOGFORMAT command.

There are also various third-party extensions to the Microsoft format to include, for example, the browser and referrer. But they all do it in different ways, so analog can't automatically diagnose them, and again, you need to write a LOGFORMAT command for them.

The WebSite format looks like

12/25/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178

(except all on one line, and with the fields separated by tabs). It suffers from the same problem with ambiguous dates as the IIS logfile (above), so again you might have to use LOGFORMAT WEBSITE-NA or LOGFORMAT WEBSITE-INT, or even have to write your own LOGFORMAT command.

The W3 extended log, the Netscape log, and the WebSTAR log can be recognised because they must include at or near the top a line telling analog what format to expect on subsequent lines. (They may also contain later lines changing the format). If the header line is missing, analog won't be able to interpret the subsequent lines and so won't be able to analyse the logfile. In this case, you will have to either replace the missing header or use a LOGFORMAT command to tell analog your format.

If analog finds that the header line is corrupt, it will usually tell you what was wrong with it. Here are two common problems. First, the header line musn't contain the same item twice, even under two different names. (This is because analog doesn't know which one you want to use.) If it does contain the same item twice, you will have to use a LOGFORMAT command to tell analog which one you want to ignore.

Secondly, you're not allowed the time without the date or vice versa -- in particular, having the date just at the top of the logfile is not sufficient; you must have it on each line. Microsoft servers produce extended logs with the date only at the top. But if the date changes during the logfile, the server doesn't then write a new date line. For this reason analog can't analyse such logfiles safely. There are some programs on the helper applications page to put the date on each line. If you already have such a logfile you might want to use one of these programs, but they have to assume that the date doesn't change during the logfile, so it would be safer to tell your server to log in a better format in future.

The extended log is described at http://www.w3.org/TR/WD-logfile.html. Its header line looks like

#Fields: date time cs-uri

In the rest of the logfile, the fields can be separated by spaces or tabs. There is also Microsoft's attempt at the extended format -- unfortunately they didn't read the spec., so they didn't enclose the browser and referrer in quotes, and they replaced spaces in the browser name with +'s. Extended logs always record the time in GMT, so you will probably need to use a LOGTIMEOFFSET command to convert to your local timezone.

The WebSTAR file has a header line like

!!LOG_FORMAT DATE TIME RESULT URL BYTES_SENT HOSTNAME

In the rest of the logfile, the fields are separated by tabs. Some other Mac servers also use the WebSTAR format, or something looking like it. Analog will understand these too.

Finally, the Netscape header line looks like

format=%Ses->client.ip% [%SYSDATE%] "%Req->reqpb.clf-request%"
%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%

Stephen Turner
Personal e-mail: analog-author@lists.isite.net
Need help with analog? Subscribe to the analog-help mailing list

[ Top | Up | Prev | Next | Map | Index ]