ProfitPress Mega CDROM2 Shareware Freeware (MSDOS)(1992)(Eng)

home *** CD-ROM | disk | FTP | other *** search

/ ProfitPress Mega CDROM2 …eeware (MSDOS)(1992)(Eng) / ProfitPress-MegaCDROM2.B6I / BBS / NETMAIL / GMD_310.ZIP / GMD.DOC < prev next >

Wrap

Text File | 1991-11-03 | 31.6 KB | 742 lines

**************************************************************************** ** ** ** Note: If you are upgrading from 3.0x be sure to read section 9.0. ** ** ** **************************************************************************** Grunged Message Detect (GMD) Utility ====================================== Version 3.10 ============== November 3, 1991 ================== Table of Contents =================== 1.0 Warning! 2.0 General Description 3.0 Reasons to Run GMD 4.0 Definition of a "Grunged" Message 5.0 Quick Start 6.0 Operation 6.1 GMD 6.2 GMD-Sort 6.3 GMD-Rpt 7.0 GMD Configuration File 7.1 General Operation Section 7.2 Set Definitions Section 7.3 Default Message Standards Section 7.4 Exception Areas Sections 7.5 What You Should Configure 8.0 Runtime Considerations 8.1 DOS 8.2 OS/2 8.3 Other Operating Systems 9.0 Upgrade Notes 10.0 Licensing Restrictions 11.0 Credits 12.0 History of Changes 1.0 Warning! ============= GMD is a very powerful program. Used properly, GMD can detect many types of grunged messages and some types of duplicate messages. Used improperly, GMD is capable of deleting valid echo mail, generating false warning messages, and violating Policy in general. Thus you should be very careful in setting it up and in using it. If you haven't read and understood FTS-0001 and FTS-0004 then you should not attempt to configure or use GMD. To protect you, me and the rest of the Net, GMD will not delete an echo mail message without trying to send an information message to the feed who sent you the packet with the bad message. Until you are sure you have your configuration correct, you should use GMD in the test mode by using the "-t" command line switch. 2.0 General Description ======================== GMD's main purpose is to detect grunged messages. In the process of doing this, GMD is also able to identify certain types of duplicate messages and non-standard messages. GMD operates directly on incoming packets, as described in FTS-0001 and FTS-0004. GMD is very configurable. Indeed, it depends entirely on a configuration file to tell it what parameters to check the messages for, and what to do when it finds a "bad" message. If you have any questions about GMD, or want to discuss any problems you encountered while running GMD, please use the "GMD" Echo, if possible. "GMD" is a Backbone echo in Zone 1. It is the best place to get GMD information and help. 3.0 Reasons to Run GMD ======================= Grunged messages can cause problems for some mail processing programs. For example, they can cause certain message renumbering programs to lockup. Since message renumbering is usually a part of nightly processing, your system is exposed to locking up while you are not around. When you discover the lockup, you are faced with the task of finding the grunged message, deleting it and then completing the night's processing. It is not uncommon for this recovery process to last several hours at an inopportune time. Deleting grunged messages and duplicate messages spares your users from seeing them and protects other boards that you feed echo mail. Informing the feed who sent you the packet with the bad messages allows him to correct the problem or to trace it back to it's source. GMD is most effective when used close to the source of the problem. Thus we encourage using GMD at the Net level, in addition to the Region and Zone levels. 4.0 Definition of a "Grunged" Message ====================================== What is a "grunged" message? This is a fuzzy question so it gets a fuzzy answer. A grunged message is a message which does not reasonably conform to FTSC-0001 and FTSC-0004 standards. It turns out that such strict conformance is not desired because so many messages fail to strictly conform to the standards (e.g., strict conformance to the standard requires two spaces between the year and time in the date/time stamp field of the message - it is fairly common to have only one space). Thus strict conformance would result in detecting messages otherwise generally considered acceptable. One of the classic symptoms of a grunged message is non-ASCII characters throughout the message. Fortunately, this condition is easy to detect in a message's control fields, especially the date field which is somewhat rigidly defined. Another form of grunging, related to a message becoming an unwanted duplicate, results in additional control fields being appended to the end of the message. Although not actually a grunged message, sometimes old messages will get scanned out again, resulting in duplicates which are too old for conventional dupe checking techniques to detect. GMD is capable of detecting these sorts of problem messages. 5.0 Quick Start ================ There is no "quick start" for GMD. We feel that due to the nature of the program you should read this file in its entirety before running GMD. 6.0 Operation ============== 6.1 GMD -------- GMD scans incoming packets. It should be run after unpacking incoming bundles, perhaps with a program like PolyXArc, but before importing or tossing the mail. GMD tests each message for conformance with the criteria specified in the configuration file. When GMD detects a non-conforming message it can be set to do nothing or to perform any combination of the following: 1) Generate an "information" net mail message to the feed who sent you the packet with the bad messages, informing him of the problem. 2) Delete the non-conforming message from the incoming packet. 3) Make a "save" copy of the non-conforming message for a local "bad" area. GMD only operates on the incoming packets. The original incoming packets maintain their time/date, even if messages have been deleted from them. Any messages that GMD generates are put into new packets, also to be processed by your regular mail program. GMD does not work with your BBS's message base. Thus it does not care how your messages are actually stored on your system, or if they are stored at all (passthrough). For security reasons, some mail processors will not accept the messages which GMD generates since they appear to be coming from that very system itself. In this case GMD should be configured to generate it's messages as if it is coming from a Point off your system, rather than your system itself. This address is set by the System.xxx parameters in the Default Message Standards Section of the configuration file. Whatever you set the System.xxx address to, you will need to make sure that your mail processor will accept routed net mail (the "information" messages) and echo mail (the "save" messages) from this address. GMD puts its FTSC product code (A3) into the header of the packets as it inspects or generates them. GMD will not process a packet which has its own product code. Thus GMD will not process the same incoming packet twice, or any packet which it generated. GMD can be called with two optional command line parameters: -c Specifies a configuration file other than the default of GMD.Cfg in the current directory, or the directory where GMD.Exe resides. -t Specifies "test mode" operation. Used for testing GMD's setup. Inhibits the "i" and "d" commands. Only the "s" command is functional. This allows you to test your GMD setup without disturbing the mail flow or generating information messages. GMD should be configured to ignore incoming mail from other systems which you know are also running GMD. This helps to avoid sending out redundant "information" messages. When you set up GMD you should inform all of your echo mail connections that you are running it, and you should ask them to let you know if they are, too. You should then list these addresses in the Ignore_Pkt_From parameter in the General Operation Section of the configuration file. You should always list your echo mail feed, even if he doesn't run GMD. Please see section 7.1 for more information. We realize that this is not a foolproof scheme, although it works for most cases. To cover some of the other cases, GMD also puts a "GMD: xxx" hidden line in any message which causes it to send out an information message. The "xxx" is the node who sends out the information message and puts the GMD hidden line in. Other systems running GMD check for the GMD hidden line before sending out information messages themselves. The presence of a GMD hidden line will not stop GMD from deleting a message if this is called for. There is a limit to the number of information messages that GMD will send out due to bad messages in any given packet. This helps in those cases that every message in a packet is bad. 6.2 GMD-Sort ------------- GMD generates a statistical database when it operates. The GMD-Sort program can be used to sort the database, prior to generating reports from it with the GMD-Rpt program. The database is sorted and replaced. This does not affect how GMD itself operates. It merely changes the order of the items in the report. To use GMD-Sort, call it with the name of the statistical database and optionally with one or more of the sort keys. The sort keys must be separated by at least one space. Each may be preceded by a "+" (default) or "-", which causes the sort to happen in ascending or descending order for that key. The default sort order, if you don't use any of the keys, is "+zone +net +node +point". The available sort keys are: zone Zone number net Net number node Node number point Point number length Number of messages too long to Number of errors in To name field from Number of errors in From name field subject Number of errors in Subject field datetime Number of errors in Date/Time field body Number of errors in body of message gated Number of errors in Gated Origin line origin Number of errors in origin line seenby Number of errors in Seen-By line path Number of errors in Path line endseq Number of bad packet endings info Number of info messages sent block Number of statistically suppressed messages delete Number of messages deleted uninf Number of uninformed deletes save Number of messages saved 6.3 GMD-Rpt ------------ GMD generates a statistical database when it operates. The GMD-Rpt program can be used to generate a report from it. To use GMD-Rpt, call it with the name of the statistical database. The report will be sent to the console. Redirection can be used to send the report wherever you desire. For example, to send it to a file called GMD.Rpt: GMD-Rpt GMD.Dat >GMD.Rpt GMD-Rpt can be called with the optional "-s" switch. This will cause it to only produce the summary. 7.0 GMD Configuration File =========================== The configuration file contains all the parameters that control GMD. The GMD distribution file contains a sample configuration file, called Sample.Cfg. Blank lines and any part of a line following a ";" are considered comments, and are ignored by GMD. The configuration file is divided into four major sections: General General system information Sets Defines legal zones and character sets Default Defines the default settings for all echo areas Area Exceptions for groups of echo areas Each of these sections starts with appropriate keyword, immediately followed by a ":". Each section is made up of a series of parameters. These parameters can be in any order. A parameter consists of a key word, a "=", and the parameter's value(s). For example: Section_1: parameter_1 = 12 parameter_2 = 34, "abc", y parameter_3 = "John Doe" Any string which contains embedded spaces or other punctuation, must be quoted. If in doubt - quote it, as it won't hurt. Any filespec can include a path in addition to the file's name. Most parameters test for a condition. These parameters are followed by an "ids" command(s) which tells GMD what to do if the condition is not met. The "ids" commands are: i Inform the feed (via net mail) about the problem d Delete the bad message from the incoming packet s Save a copy of the bad message Any combination (including none if no action is desired) of the "ids" commands may be used, except that if the "d" command is used, then the "i" command will also be executed, even if you do not specify it. Some parameters only allow the "is" commands, and not the "d" command. If a message fails more than one condition, then the action that GMD takes is the "or" of the individual conditions' command settings. For example, if a message fails 3 condition tests, yet only one of these conditions specifies that a information message be sent, it will be sent. GMD permits include files anywhere a token (a string of characters which GMD recognizes as a unit) can appear. The syntax is @filespec (or "@filespec"). Include files can be nested. Included files can be handy for separate, but similar, configuration files. They can also be used to input lists of various sorts (echo areas, names, search strings). 7.1 General Operation Section ------------------------------ This section contains general information about your system and how GMD should operate. GMD logs both to the screen and to a file. You can specify the level of detail you want for each. Each level includes the information in the lower levels. LogLevel Char Information Logged -------- ---- --------------------- 0 ! Errors 1 * Summary 2 + Packets 3 : Bad Messages, brief 4 # Bad Messages, verbose 5 - Good Messages, brief GMD generates some summary statistics about the numbers and types of non-conforming messages. This information is in the logs and can also be sent to you in a net mail message. The only disk directory that GMD needs to know about is the one which contains the incoming packets. This is usually your inbound mail area. GMD does not care what type of message base your system uses. Any messages that GMD itself generates will be put in the form of incoming packets. Thus your system treats them like any other incoming mail. This means that GMD also does not care what type of outbound area(s) your system uses. When GMD "saves" a copy of a non-conforming message, it does this by sending a copy of the message to a "bad" echo area which you can specify. This allows you to look at the messages later. GMD is capable of generating a database which contains statistical information about the number and types of errors encountered for each node. This database has two uses. It can be used to limit the number of information messages sent because of a given node. Also it serves as input to the GMD-Sort and GMD-Rpt reporting programs. The database is cleared by deleting the file. A typical use would be to generate a weekly report, then delete the database. To prevent your GMD from sending out information messages which have already been sent out by another node running GMD, you can instruct GMD to "ignore" packets coming in from certain Nodes. In fact, GMD will continue to inspect these packets, but will not generate any information messages or save copies, unless a delete is also called for, in which case it processes the packet normally. This allows for the possibility of a message being grunged after it has been processed by a node running GMD. You should always list your echo mail feed, even if he doesn't run GMD. 7.2 Set Definitions Section ---------------------------- This section contains definitions of sets which are used in the Default and Area Sections. Sets are used to specify character sets and zone numbers, for example. You can name them whatever you want to and you can have as many as you need. Sets make it possible to specify any combination of characters or numbers which you wish to allow. For example: Just basic, printing, ASCII characters. Or you might want to allow some (the alphabetic), but not all, of the hi-bit characters, too. 7.3 Default Message Standards Section -------------------------------------- This section contains the default settings used for the echo areas. These set the parameters which determine whether a message conforms or not. These settings can be overridden for specific echoes by using the Area Section. Most of these parameters test conditions, hence use the "ids" commands. These commands determine what GMD does if a non-conforming message is found. Many of the parameters allow you to relax or disable GMD's conformance checking of various aspects of the message specifications. At present, there are so many non-conforming messages that we felt this was necessary until the majority of the responsible programs were fixed. Please see section 6.1 for a discussion about how to set the System.xxx parameters. GMD uses two template files to build the information messages that it sends. One template file is used for messages which are passed and another template file is used for messages which are deleted. There are a number of macros available which cause GMD to insert the appropriate information about the offending message. These are the macros which can be used in the template files (note the trailing "."): &To. The non-conforming message's "to" field &From. The non-conforming message's "from" field &Subject. The non-conforming message's "subject" field &Date. The non-conforming message's "date" field &Flags. The non-conforming message's flags &Why. The reason(s) that the message didn't conform &Area. The echo area that the non-conforming message was in &Body. The non-conforming message's body (up to the origin line) &Routing. The non-conforming message's control fields (from the origin line on) &SysOp. The name specified in System.SysOp Any other text in the template files is used in the information message "as is". The GMD distribution file contains two sample template files. Sample.Pas is an example of the Info_Message.Pass_File and Sample.Del is an example of the Info_Message.Delete_File. Some parameters require sets to represent lists of acceptable characters. Minimum character sets are OR'ed with the character sets that you use to prevent leaving out basic characters. The minimum character set is 32 (space) to 126 (~) for the following parameters: To.Alphabet, From.Alphabet, Subject.Alphabet, Gated_Origin.Alphabet and Origin.Alphabet. In addition to 32 to 126, the minimum character set for the Body.Alphabet parameter also includes 1 (control A), 9 (tab), 10 (line feed) and 141 (soft carriage return). 7.4 Exception Areas Sections ----------------------------- These are optional sections which contain exceptions to the defaults listed in the Default Section. Any setting listed there can be overridden for specific echoes. An echo mail area can only be listed in one Exception Area Section. The only required parameter in an Area Section is "Echoes". It lists the echo mail areas for which the exceptions which follow it apply. Typical uses of Exception Areas include: A Node in more than one Network who has to cope with different message standards, or a Node or software developer who wants to do more stringent testing on certain test areas. 7.5 What You Should Configure ------------------------------ It is very important to configure these parameters to match your system: Mail.Inbound Ignore_Pkt_From System.Domain System.Zone System.Net System.Node System.Point System.SysOp System.Private_Net (if needed) These parameters may also be configured to suit your style of operation: Log.Screen_Level Log.Level Log.File Log.Error_Stats Mail.Error_Stats Mail.Bad_Area Info_Message.Kill_Sent Stat.File Stat.Max_Nr_To_Node Max_Nr_Per_Packet Ignore_Pkt_To (if needed) Ignore_Echo_List (if needed) Info_Message.Pass_File Info_Message.Delete_File Info_Message.Pass_Subject Info_Message.Delete_Subject 8.0 Runtime Considerations =========================== GMD returns the following errorlevels: 255 Compiler run time error 3 Configuration file error 2 Bad message deleted, information message sent 1 Bad message passed, information message sent 0 No bad messages or information messages sent GMD has been assigned FTSC product code "A3". 8.1 DOS -------- Use the DOS version on 8086/8 CPUs. The OS2 version runs a bit faster, but it can only be used on 80286, and above, CPUs. GMD requires about 350K of free memory to run. It is not overlaid, and does not use EMS or any other type of memory. GMD running on a 25 MHz 80486, under DesqView, takes about 6 minutes to process 10,000 messages. This is about one full day's mail for the Zone 1 Backbone as of November 1991. 8.2 OS/2 --------- Use the OS2 version of the programs. They are bound in the dual mode, thus work in either protected or real mode. 8.3 Other Operating Systems ---------------------------- We would be happy to work with anyone who is interested in porting GMD to other operating systems. Contact the author. 9.0 Upgrade Notes ================== There are some changes between version 3.0x and 3.10 which you need to take into account when upgrading. There were quite a few changes made to the configuration file. We recommend that you start over with the enclosed sample configuration file. The format of the statistical database has changed. You should delete the existing database before running the new GMD. GMD now returns more, and different, errorlevels than it used to. If you use the errorlevel, you will have adjust for the new ones. 10.0 Licensing Restrictions ============================ There are some simple restrictions associated with using the programs. You use these programs at your own risk. The author does not warrant that the programs work as described herein or that they are even suitable for any particular purpose. This documentation is intended to describe how the programs work in the author's environment. The author does not claim that they will work the same way in your environment or, if they do work, that they will continue to work. Furthermore, it is up to you to decide if these programs are suitable for any particular purpose on your system. The author freely releases the executable versions of these programs to the public domain. The author desires no fees for the use of these programs and does not support these programs. These programs compile and link using Borland C++ and Microsoft C 6.0 with no errors. The source codes for GMD and related programs are not generally available. However, if you have a legitimate need, and are willing to sign a non-disclosure agreement, contact the author. 11.0 Credits ============= David Troendle, 1:396/5, is the author of GMD and related programs. As time permits, he is willing to answer questions about techniques used and how these program generally work. John Souvestre, 1:396/1, is responsible for the testing and documenting of GMD and related programs. He also serves as the interface to the author, when David is busy otherwise. We would like to thank the many people who helped in one way or another with GMD's development. Thanks to the following people who supplied helpful coding ideas or actual code: Chris Irwin Jeffrey Nonken George Peace Thanks to the beta test team, past and present (*): Steve Ahola Dean Lachan * Martin Belcke * Paul Marwick * Bruce Berna * David Nugent * Bruce Bodger * Roscoe Primrose * Bill Bolton Mike Ratledge Rod Bowman * Bob Rakov * Larry Carter * Bob Satti * Tim Carter * Robert Soubie * Steve Cross * Graham Stair * Tony Davis * Mort Sternheim * Fabian Gordon * Roy Timberman * Jim Greely * John Valentyn * Miles Hoover * Tony Wagner * Mark Howard * Adrian Walker * Dave James * Matt Whelan * John Johnson * Ken Wilson Felix Kasza John Woodward * Thanks to the GMD Echo Moderator, Steve Shapiro. 12.0 History of Changes ======================== Version Date Changes, Modifications, Additions, etc. ------- -------- --------------------------------------- 1.00 12/29/90 Original release 1.01 2/16/91 Maintenance release, a few small fixes 2.00 2/28/91 Added features mainly for Hub usage - Uses echotoss log - Uses and updates lastread pointer - Optional passthrough area processing - Configurable log detail level - Returns an errorlevel 3.00 9/12/91 Complete redesign and rewrite! - Processes incoming packets directly - Many new parameters - Parameters now in configuration file - Ability to send information messages - Statistical database and reporting - "Delete" command disabled for now 3.01 9/17/91 Maintenance release - Fixed Ignore_Pkt_From logic - Added Seen_By.Node_Sort_OK 3.10 11/03/91 Miscellaneous additions and changes - "Delete" command enabled, but no longer allowed in some places - Fixed Ignore_Pkt_To logic - Send information messages to feed, not originator - Added Max_Nr_Per_Packet - Added GMD kludge line - Improved log formats - Improved report format - Added more errorlevel returns - Added Ignore_Echo_List - Fixed a file handling problem which caused trouble on certain LANs - OS/2 version - End -