Log File Formats
Summary works with logs from the following servers (and many others):
Apache - when set to produce NCSA Common or NCSA Combined format logs
Apple Personal Web Server
Apple Share IP - version 5.0 or higher
Boulevard
FileMaker Pro - brief format only
MacHTTP
Microsoft IIS - in all log modes. With Microsoft format optionally adding:
FlashLog from Maximized Software
(http://www.maximized.com/products/flashstats/flashlog.htm)
WebTrends (www.webtrends.com) when cookie support is on
Microsoft Personal Web Server for Macintosh
NCSA httpd
Netscape (including FastTrack, Commerce, Enterprise, and some others)
QuidProQuo
Rumpus - version 1.2 or higher, earlier versions with some limitations
Web Server 4D
WebSTAR - version 1.2.1 or higher
WebTen - see Apache comments
WN
Summary will automatically recognize the following log formats:
MacHTTP
Microsoft IIS Version 3 and Version 4
NCSA Common or Common Log File Format
NCSA Combined (sometimes called NCSA Extended)
Netscape
WebSTAR (used by several others: QuidProQuo, Web Server 4D, etc.)
W3C Extended Log Format (ExLF)
WN Verbose
Many servers are configurable to produce log files in several different formats, and some formats have a long list of fields that can be included individually. There is often a trade off between large log files with lots of information and smaller log files that don't tell you everything but actually fit on your hard disk. The following comments will help you choose between the different options, and tell you how to best configure your server for Summary.
Comments on specific servers and log formats
AppleShare IP
AppleShare IP, version 5.0 and higher, writes a log file in WebSTAR format. The log file is stored in System Folder, Preferences, AppleshareIP Preferences, HTTP Logs, HTTP Log. The log format is not configurable and AppleShare does not log referrer or agent information so reports that depend on those fields will be blank.
Apache
Apache has a highly configurable log format which, unfortunately, is not self documenting. That means that Summary can not automatically determine the format of an Apache log file unless it is in one of the standard formats. The two most common formats used with Apache are NCSA Common and NCSA Combined. I highly recommend that you configure Apache to produce NCSA Combined format logs. The Apache configuration command for this format is:
LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\""
When running multiple virtual servers with a single copy of Apache it is common practice to log each virtual server into it's own log file. This is important because NCSA Combined format logs don't contain the virtual server name. It is impossible to determine which log entry goes with which virtual server when using a single log file in NCSA Combined format with entries from multipule virtual servers.
FileMaker Pro
Summary can read FileMaker Pro web logs if they are in brief format. You will need to specify a "User log format definition" (entered on the Miscellaneous configuration page) of:
DATE-MDY TIME-12 HOST URI
Log-FM
Log-FM is not officially supported but you might want to try the following "User log format definition" (entered on the Miscellaneous configuration page):
DATE-MDY TIME-24 SKIP HOST URI BYTES SKIP AGENT METHOD SKIP SKIP REFERER TRAN-TIME-TICKS SKIP SKIP STATUS SERVER EOL
Microsoft IIS
IIS Version 3.x supports Common Log Format and Microsoft Extended Format. Neither one provides referrer or agent information. Summary supports FlashLog Format, using a server plugin from Maximized Software (http://www.maximized.com/products/flashstats/flashlog.htm) which adds the referrer and agent information to the end of the line. Summary also supports the WebTrends modified format (www.webtrends.com) if cookie logging is enabled.
IIS Version 4.x supports Common Log Format, Microsoft Extended Format, and W3C Extended Format (ExLF). We recommend using the W3C ExLF format since that allows you to customize the tokens appearing in the log file. See the discussion of W3C ExLF for more information about the various tokens.
Microsoft Personal Web Server for Macintosh
Microsoft Personal Web Server for Macintosh produces logs compatible with WebSTAR format containing the following log items:
DATE TIME CS-IP HOSTNAME URL RESULT USER BYTES_SENT
Older versions are not completely compatible with WebSTAR format. If your log can't be read, try the following custom log format:
SKIP DAY MONTH-NAME YEAR TIME-24 SKIP HOST HOST URI STATUS USER BYTES
NCSA Common Log Format
This is a very common format, supported by many servers. Unfortunately it does not provide referrer, agent, transfer time, server name, or cookie information, which disables many reports.
NCSA Combined
This is a reasonably common format, supported by many servers. It does not provide transfer time, server name, or cookie information, which disables some reports.
If you have Apache or WebTen you can also add transfer time in seconds and optionally cookie to the end of the NCSA combined format. The Apache and WebTen command to get NCSA Combined logs is:
LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\""
To add transfer time to the end:
LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %T"
To add transfer time and cookie to the end (should all be on one line):
LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %T \"%{Cookie}i\""
QuidProQuo format
QuidProQuo supports NCSA Common log format and QuidProQuo formats. There is an error in their Common Log Format support that makes it not conform to the specification, at least through version 2.1.2. Use this custom format string to parse QuidProQuo logs in Common Log Format:
HOST SKIP SKIP USER SKIP DATE-CLF FULL-REQUEST CODE BYTES EOL
QuidProQuo native format is very similar to WebSTAR format. Summary fully supports QuidProQuo format. See the WebSTAR format description below for additional comments.
WebSTAR format
The WebSTAR server's log format is highly configurable. Summary supports WebSTAR using Common Log Format (CLF) logs and with Extended Log Format (ExLF) logs but we recommend using WebSTAR Log Format (WLF).
When configuring your log format there are three issues to keep in mind. The more information you put in the log file, the more Summary will be able to report to you. At the same time, the more information you put in the larger the log files will become, eventually filling your hard disk. Finally, some of the log tokens have become obsolete and have been replaced by newer tokens. The older tokens will still work but are not recommended due to various limitations.
Summary requires the following tokens in WebSTAR format logs:
| DATE |
Date of request. |
| URL |
The requested item. Same as CS-URI and CS-URI-STEM. |
The following fields are very highly recommended:
| TIME |
Time of request. Required for the Hourly, Time of Day, and Gaps in Service reports. |
| HOSTNAME |
Name or IP address of the requesting computer. You can use C-IP if you always leave DNS lookups off in WebSTAR but it is slightly larger. You can use C-IP and C-DNS together in that order to keep all available information even though Summary won't take advantage of it and your log will be larger. CS-IP and CS-HOST together in that order work but they are not recommended. Required for the Top Level Domain, Domain, Host, Visits per Host, Hits per Visit, Pages per Visit, Bytes per Visit, Source, Destination, Path, and Reloads reports and the Unique Hosts and Visits columns in various reports. |
| BYTES |
Bytes sent, same as BYTES_SENT. Required for the Bytes per Visit, Requests by Bytes, Peak Hours, Peak Days, by File Type, Transfer Size, Transfer Time, and Connection Speed reports and all of the Bytes related columns in various reports. |
| SC-STATUS |
Result code, the same as STATUS. This provides just slightly more information than RESULT (which is also acceptable). CS-STATUS will work but it is not recommended. Required for the Bad Links and Failed Requests reports and the Errors column in various reports. |
| REFERRER |
Site and page that referred them to us. Slightly shorter than CS(REFERER), which also works. This field will increase the size of log files substantially. Required for the Domains, Referrers, Search Words, Search Phrases, Full Referrers, New Referrers, Local Referrers, Source, and Destination reports. |
The following fields provide additional information for Summary, which enables additional reports. You can decide if they are worth it. Listed from most interesting to least interesting overall, although that is partly a personal preference.
| AGENT |
Browser making the request. Slightly shorter than CS(USER-AGENT) which also works. This field will noticeably increase the size of the log file. It provides information for the Browser, Platform, Agent, and Web Robots reports. |
| USER |
Authenticated user name entered into a name and password dialog when some portion of the site is restricted. Provides information for the Auth User report. |
| TRANSFER_TIME |
time to send data in 1/60ths of seconds. This field provides information for the Connection Speed report. Avoid TIME_TAKEN which is sometime in hours:minutes:seconds, sometimes in seconds, and sometimes in 1/60ths of seconds which can cause inaccurate results. |
| CS(HOST) |
The name of the server the user sent the request to, the same as HOST and HOSTFIELD. This field provides information for the Virtual Server report and can be useful in filtering virtual domains. |
| METHOD |
The method from the request header, GET, PUT, etc. Same as CS-METHOD and slightly shorter than CS(METHOD). This field provides information for the Method report. Fairly technical. |
| SEARCH_ARGS |
CGI arguments. Same as CS-URI-QUERY. This field provides information for the CGI Arguments report, which must also be enabled in the Summary configuration. The value of this report will depend on your use of CGI and plug-in arguments. |
| CS(COOKIE) |
Any cookies sent by the browser, the same as COOKIE. This field provides information for the Cookie report, which must also be enabled in the Summary configuration. Not used by most sites. |
There are a few other fields that WebSTAR supports, which might be of some use to someone, but Summary doesn't use them:
| FROM |
almost always empty, used to be e-mail address of user but privacy concerns caused browsers to stop sending this field. Occasionally filled in by web robots. |
| CONNECTION_ID |
The internal WebSTAR id number associated with this connection. I can't imagine ever using this. |
| PATH_ARGS |
Portion of the request after a '$' character. This is a WebSTAR specific feature, designed to make programming CGI code easier but hardly ever used. |
W3C Extended Log Format (ExLF)
This is a highly configurable format, but not all servers allow all of the options.
Summary requires the following tokens in the log file:
| DATE |
Date of request. |
| CS-URI |
The requested item, essentially the same as CS-URI-STEM which also works. |
The following fields are very highly recommended:
| TIME |
Time of request. Required for the Hourly, Time of Day, and Gaps in Service reports. |
| C-IP |
Client IP address. Use along with C-DNS if you have DNS lookups turned on, as long as C-IP appears first. CS-IP and CS-HOST can be used together in that order instead but they are obsolete and not recommended. Required for the Top Level Domain, Domain, Host, Visits per Host, Hits per Visit, Pages per Visit, Bytes per Visit, Source, Destination, Path, and Reloads reports and the Unique Hosts and Visits columns in various reports. |
| BYTES |
Bytes sent. Same as SC-BYTES. Required for the Bytes per Visit, Requests by Bytes, Peak Hours, Peak Days, by File Type, Transfer Size, Transfer Time, and Connection Speed reports and all of the Bytes related columns in various reports. |
| SC-STATUS |
Result code. CS-STATUS will work but it is obsolete and is not recommended. Required for the Bad Links and Failed Requests reports and the Errors column in various reports. |
| CS(REFERER) |
Site and page that referred them to us. This field will increase the size of log files substantially. Required for the Domains, Referrers, Search Words, Search Phrases, Full Referrers, New Referrers, Local Referrers, Source, and Destination reports. |
-
The following fields provide additional information for Summary, which enables additional reports. You can decide if they are worth it. Listed from most interesting to least interesting overall, although that is partly a personal preference.
| CS(USER-AGENT) |
Browser making the request. This field will noticeably increase the size of the log file. It provides information for the Browser, Platform, Agent, and Web Robots reports. |
| CS-USERNAME |
Authenticated user name entered into a name and password dialog when some portion of the site is restricted. Provides information for the Auth User report. |
| TIME_TAKEN (WebSTAR) or TIME-TAKEN (Microsoft) |
time to send data. This field provides information for the Connection Speed report. |
| CS(HOST) |
The name of the server the user sent the request to. You can also use S-IP or S-DNS instead if they are available. This field provides information for the Virtual Server report and can be useful in filtering virtual domains. |
| CS-METHOD |
The method from the request header, GET, PUT, etc. Slightly shorter than CS(METHOD). This field provides information for the Method report. Fairly technical. |
| CS-URI-QUERY |
CGI arguments. This field provides information for the CGI Arguments report, which must also be enabled in the Summary configuration. The value of this report will depend on your use of CGI and plug-in arguments. |
| CS(COOKIE) |
Any cookies sent by the browser. This field provides information for the Cookie report, which must also be enabled in the Summary configuration. Not used by most sites. |
-
There are many fields that ExLF supports, which might be of some use to someone, but Summary doesn't use them. Here are a few of them:
| CS-FROM |
almost always empty, used to be e-mail address of user but privacy concerns caused browsers to stop sending this field. Occasionally filled in by web robots. |
| CS-VERSION |
The HTTP protocol version number. |
| CS-BYTES |
The number of bytes in the request sent by the client. |
User Specified Log Formats
If your server produces logs with a format not listed here, you may be able to configure Summary to read the log file by specifying the format manually. You specify your log format by making a string with the following tokens corresponding to the order of the fields in your log file:
| DATE-CLF |
Full date/time in Common Log File format |
| DATE-DMY |
Day, month, year |
| DATE-MDY |
Month, day, year |
| DATE-YMD |
Year, month, day |
| UNIX-TIME |
The number of seconds since 1970 |
| TIME-24 |
Hour, minute, second |
| TIME-12 |
Hour, minute, second, AM/PM |
| YEAR |
Two or four digit year number |
| MONTH |
One or two digit month number |
| MONTH-NAME |
Three character month name |
| DAY |
One or two digit day of the month |
| HOUR |
One or two digit hour of the day |
| MINUTE |
One or two digit minute of the hour |
| SECOND |
One or two digit second of the minute |
| FULL-REQUEST |
The original request line |
| HOST |
Host name |
| URI |
The requested resource, with optional '?' portion |
| URI-QUERY |
The '?' portion of the request |
| STATUS |
Three digit HTTP response code |
| WEBSTAR-RESULT |
WebSTAR four character response code |
| BYTES |
Number of bytes transferred |
| TRAN-TIME-SECS |
The transfer time in seconds |
| TRAN-TIME-TICKS |
The transfer time in 1/60ths of a second |
| TRAN-TIME-MILLI |
The transfer time in milliseconds |
| TRAN-TIME-HMS |
The transfer time in HH:MM:SS, or 1/60ths |
| REFERER |
The referer from HTTP header |
| AGENT |
The agent from HTTP header |
| USER |
The user name from authorization |
| METHOD |
The HTTP request method |
| SERVER |
The server name, often from HTTP host field |
| MAY-SERVER |
The server name, often from HTTP host field. This will not override a preceeding SERVER value. |
| COOKIE |
The HTTP cookie field |
| SKIP |
Skip to the next field |
| IF-EOL |
If at EOL, return valid entry, else continue |
| MUST-EOL |
Must be at the end of the line |
| EOL |
Skip to the end of the line, must be last if used |
| W3SVC |
Must match "W3SVC" |
| FIXUP |
Fix FlashLog parsing to WebTrends layout |
| WN |
Check for CLF, Combined, or WN Verbose log |
| CHAR |
Skip one character of input |
-
Summary will automatically parse the log file into tokens, handle quoted strings, and find fields separated by a space, comma and a space, or tabs.
For example NCSA Common Log Format and NCSA Combined format would be specified with the single string:
HOST SKIP USER DATE-CLF FULL-REQUEST CODE BYTES IF-EOL REFERER AGENT EOL
|
.dmg/Shareware World/Comms & Internet/Summary 1.4.1.sea/Summary 1.4.1/Documentation/art/sumblock.gif)