Logstats Rewriter

Sambar Server Documentation

Log Rewriter
Pro Server Only

The logstats rewriter is a pattern substitution engine that is designed to simplify the manipulation of entries in a log file. The config/logstats.ini file can have an additional [rewrite] section that the logstats modifier engine reads to rewrite individual log lines before processing. In the [rewrite] section the user can specify a field of the log file to examine, and a wildcard pattern to be matched. When a line in the log file matches a pattern in the rewrite section, the engine will perform the specified action. This functionality is only available with the Sambar Server Pro release..

Template Processor Actions
The template processor can perform the following actions on a log line:

Action Description

Drop Loglines Drop any log lines matching the comparison.

Cut Data Use only the data matching the comparison string, and remove all other data from the field being compared against.

Strip Data Strip out any data that matches the comparison string.

Substitute Data Replace the data matching the comparison string with data specified by the user.

If/Then Substitute If there is a match for the comparison string, replace data matching the qualification with data string.

Global If/Then Substitute If there is a match for the comparison string, replace all string elements matching the qualification with data string.

Action	Description
Drop Loglines	Drop any log lines matching the comparison.
Cut Data	Use only the data matching the comparison string, and remove all other data from the field being compared against.
Strip Data	Strip out any data that matches the comparison string.
Substitute Data	Replace the data matching the comparison string with data specified by the user.
If/Then Substitute	If there is a match for the comparison string, replace data matching the qualification with data string.
Global If/Then Substitute	If there is a match for the comparison string, replace *all* string elements matching the qualification with data string.

Template Processor Fields
The following is a list of fields that the logstats rewriter can operate over:

Field	Description
REQUEST	The URL request.
HOST	The client the request was received from.
USER	The username of the client.
REFERER	The referer associated with the request.
AGENT	The agent (browser) the client performed the request with.
STATUS	The HTTP server response status (i.e. 200, 401, 500)

Wildcard Characters

Wildcard characters can be used as placeholders for other characters when you are specifying a pattern to be matched. You can use the following characters:

Character Usage Example

* Matches any number of characters. It can be used as the first or last character in the character string. /*/index.htm finds: /test/index.htm and /pubs/index.htm

$ Matches the end of the string. The $ symbol can only be found at the end of the wildcard pattern. /doc$ finds: /doc but not /documentation or /docking

^ Matches the beginning of the string. The ^ symbol can only be found at the front of the wildcard pattern. ^/doc finds: /doc and /docking but not /foo/doc

? Matches any single character. /b?ll finds: /ball, /bell, and /bill

# Matches any single numeric character. /doc# finds: /doc1 and /doc2 but not /docs

[ ] Matches any single character within the brackets. /b[ae]ll finds: /ball and /bell but not /bill

! Matches any character not in the brackets. /b[!ae]ll finds: /bill and /bull but not /bell

- Matches any one of a range of characters. You must specify the range in ascending order (A to Z, not Z to A). /b[a-c]d finds: /bad, /bbd, and /bcd

\ Matches any single "special" character above (the literal). /b\*d finds the literal string: /b*d

Character	Usage	Example
*	Matches any number of characters. It can be used as the first or last character in the character string.	/*/index.htm finds: /test/index.htm and /pubs/index.htm
$	Matches the end of the string. The $ symbol can only be found at the end of the wildcard pattern.	/doc$ finds: /doc but not /documentation or /docking
^	Matches the beginning of the string. The ^ symbol can only be found at the front of the wildcard pattern.	^/doc finds: /doc and /docking but not /foo/doc
?	Matches any single character.	/b?ll finds: /ball, /bell, and /bill
#	Matches any single numeric character.	/doc# finds: /doc1 and /doc2 but not /docs
[ ]	Matches any single character within the brackets.	/b[ae]ll finds: /ball and /bell but not /bill
!	Matches any character not in the brackets.	/b[!ae]ll finds: /bill and /bull but not /bell
-	Matches any one of a range of characters. You must specify the range in ascending order (A to Z, not Z to A).	/b[a-c]d finds: /bad, /bbd, and /bcd
\	Matches any single "special" character above (the literal).	/b\d finds the literal string: /bd

Example Comparisons

Example 1.

DROP = REQUEST = .gif$ DROP = HOST = 200.0.0.1 DROP = HOST = 192.0.0.*

The first line of this example will cause any log line that has a requested URL ending with ".gif" to be dropped from the file. The effect of the second and third lines is that any log line containing a client host with IP address 200.0.0.1, or an IP address starting with 192.0.0, will be dropped.

Example 2.

CUT = AGENT = MSIE*;

The first line of this example will examine the AGENT field of the log line, looking for the string "MSIE". It will then match from "MSIE" to the first occurence of a ";". Everything between these two will be preserved, everything else will be removed.

So if the AGENT field was: MSIE=foobar; Windows95; Version=4
it would be transformed to: MSIE=foobar;

Example 3.

STRIP = REFERER = ^http://www.foobar.com

This example would search the REFERER field to see if it started with the string "http://www.foobar.com". If it did, the string would be removed. The rest of the REFERER field would remain intact.

Thus the REFERER field: http://www.foobar.com/products/bassomatic.html
would be transformed to: /products/bassomatic.html
This would cause all referrals from www.foobar.com to be treated as if they were internal referrals from within your site.

Example 4.

SUBSTITUTE = REFERER = http://www.foo1.com* http://www.foo.com

This example would search the REFERER field, and replace all occurences of "http://www.foo1.com" with "http://www.foo.com"

Example 5.

IFTHEN = REQUEST = ^/products ? REQUEST = image0357 bassomatic

This example will check to see if the REQUEST field starts with the string "/products". If it does, the first occurence of the string "image0357" will be replaced with the string "bassomatic".

[rewrite] Configuration

The following sample file illustrates the format of a [rewrite] initialization section of the config/logstats.ini.

#
# REWRITE
#
# Valid Comparators:    HOST USER REQUEST REFERER AGENT STATUS
#
# Notes:                All comparisons are case-insensitive.
#
[rewrite]
drop = REQUEST = .gif$
drop = HOST = 200.200.0.0

cut = AGENT = MSIE*;

strip = REQUEST = index
strip = REQUEST = index.html$
strip = REFERER = ^http://www.foobar.com/

substitute = REQUEST = /RealMedia/* /RealMedia/ad.gif

ifthen = REQUEST = ^/docs ? REQUEST = docs help

global = REFERER = find ? REFERER = & /