Chip Hitware 7

home *** CD-ROM | disk | FTP | other *** search

/ Chip Hitware 7 / Chip_Hitware_Vol_07.iso / chiphit7 / online / 95newbot / newsbot.doc < prev next >

Wrap

Text File | 1996-06-20 | 30.8 KB | 635 lines

SBNews: News Robot Utility (C) Scott M Baker ------------------------------------------------------------------------------- Note: This program is still in the early beta stages. It hasn't been very extensively tested and it's very possible that you may experience trouble. Please report any problems to me at smbaker@primenet.com. I variously refer to this program under the names "NewsBot" and "SBNews" throughout the document. You can for the most part consider them to be the same. Eventually, I plan on including extra utilities in this package, "SBNews" will be the name of the full package and "NewsBot" will be the name of the News Robot utility within the package. ------------------------------------------------------------------------------- Table of Contents * Purpose * General Information * System Requirements * Distribution Information * Features * Files Included * Installation * Command Line Parameters * JPEG Viewer * Configuration * Statistics * Unfinished Stuff/Planned Enhancements * Acknowledgements * Registration * How to Contact the Author * Revision History ------------------------------------------------------------------------------- Purpose: The purpose of this program is to automatically download and uudecode files from newsgroups via a winsock connection. If you've found this program, then you probably know what this means, so I won't go into detail. Newsbot will attempt to piece together multi-part files. If you're new to binary newsgroups, then here are a few G-rated newsgroups that you might wish to try: alt.binaries.pictures.astro alt.binaries.pictures.animals alt.binaries.pictures.cartoons alt.binaries.pictures.fractals There are a wealth of adult-oriented newsgroups, but I'm not going to list them in this document. I have included a sample file, SAMPLE_G.TXT, which includes a listing of binary newsgroups. There are no descriptions, but you can probably find out from the newsgroup name what subject matter the newsgroup deals with. ------------------------------------------------------------------------------- Requirements: * Windows 3.1 or greater * Winsock connection capability (i.e. SLIP or PPP) Distribution: * 16-bit (Windows 3.1) Version: SBNEWSxx.ZIP (xx denotes version number) * 32-bit (Windows 95) Version: SBN32_xx.ZIP (xx denotes version number Features: * Unattended download - do other work (or go out to lunch) while downloads and uudecoding are all processed for you! * Automatic HTTP address logging -- NewsBot will keep track of any HTTP URL's that were found while downloading addresses. These will be saved to a convenient HTM file, HTTPGRAB.HTM. * Built in JPEG viewer. Allows you to see postage-stamp size images of the files you are downloading in progress. * Delete/View previously downloaded JPEG files. * Comprehensive logging capabilities - save the subject and from information for files that you might want to follow up on later. * Dupe Checkers -- avoid downloading the same files multiple times. Files Included: * NEWSBOT.EXE: Executable; what you need to run to get the program to work. * NEWSBOT.HTM: Documentation in HTML format (read with your browser) * NEWSBOT.DOC: Documentation in plain-ASCII format (read with any plain ascii text editor) * BWCC.DLL (16-bit) or BWCC32.DLL (32-bit): Borlands Custom-Control DLL library; used to support Newsbot's grey dialog boxes. * FILE_ID.DIZ: Brief description file; useful for bbs distribution * SAMPLE_G.TXT: Plain-ASCII listing of binary newsgroups * REGISTER.HTM: Registration info in HTML format * REGISTER.DOC: Registration info in plain-ASCII format * README.1ST: General information on documentation & archives ------------------------------------------------------------------------------- Installing and Using the Program: Here are some very quick notes on installing and using Newsbot: Installation: 1. Unzip the distribution archive into the directory of your choice. Running: 1. Run NEWSBOT.EXE -- this can be done in various ways - from the Win-95 command prompt, the Windows Explorer (and/or file manager), the Windows Run Program command, etc. 2. On your first session, you'll be prompted for a "News Host Name". This is the name of your news host, and should be given to you by your ISP (internet service provider). 3. On your first session, you'll be prompted for a "download path". This is the path where you want the files you download to be placed. 4. Once NewsBot is loaded, you'll want to add some newsgroups. This is done with the "Add" button. You'll need to know the names of the groups you want; you might want to consult SAMPLE_G.TXT for some sample newsgroup names. 5. Press the "CONNECT" button to start everything. All else should be mostly automatic from here on. Feel free to tinker around with the menu items ------------------------------------------------------------------------------- Command Line Parameters * "-i INI_FILE_NAME". Sets the name of the INI file to use. By default, newsbot uses the file newsbot.ini in your windows directory. Specifying a filename here will cause newsbot to use the file under that name in newsbot's home directory. For example "Newsbot -i news1.ini" would use the filename "news1.ini" in newsbot's home directory. * "-auto". Will automatically begin downloading news articles as soon as newsbot has loaded. Note: For unregistered users, about screen will still be displayed! * "-noabout". Omits display of the about dialog on startup of Newsbot. Note: Only supported in registered version! ------------------------------------------------------------------------------- Jpeg Viewer SBNews has a built in Jpeg viewer for viewing JPEG files offline. The JPEG viewer can be operated in several ways: 1. By double-clicking on a [RECEIVED] entry in Newsbot's log window 2. By single-clicking on one of the preview pictures on the right hand side of Newsbot's main window. 3. By using the FILE:VIEW JPEG option from Newsbot's pulldown menu. Once loaded, the JPEG viewer window will display the JPG file selected and include a listing of filenames and directories on the left-hand side of the window. This listing is provided to let you easilly select other files to view. Double clicking on a directory in the directory list will change to that directory and double clicking on a filename in the file list will display that file. The CHDIR and VIEW buttons perform the same functions, respectively. The "<" and ">" buttons will view the immediately previous and next files in the currently selected directory. You may use these to rapidly view a list of files that were downloaded overnight for example. The ">" button will activate a slideshow of the current directory. Each file will be displayed, there will be a slight delay, and the next file will be displayed. The FILE menu of the viewer has several options: * Open File: Allows you to open a new file using the windows common-dialog open method. * Delete File: Deletes the current file that you are viewing. * Save as BMP: Saves the current image as a BMP image, which may later be used as windows wallpaper, for example. * Set as Wallpaper: Saves the current file as a BMP and tells window to use it was wallpaper. The SlideShow menu has a few options as well: * Start/Stop: Starts and stops the slideshow, equivalent to the ">" button. * Set Interval: Sets the amount of delay between slideshow pictures. * "Normal Mode": Images will be displayed in the jpeg viewers window. * "Wallpaper Mode": Images will also be set as WallPaper. This sort-of turns your entire desktop into a slideshow display. You can minimize the JPEG viewer window and continue to do work while images display in the background as wallpaper. ------------------------------------------------------------------------------- Configuration Host Name: The host name is the name of the nntp host which NewsBot will connect to. If you don't already know this, then you may wish to contact your Internet Service Provider (ISP) or check an existing news program on your system for the name. Download Path: The base path where downloaded files will be placed. If you enable any of the download path expansion (see misc. preferences), then files may be downloaded into sub-directories of this base path. Miscellanious Preferences: Miscellanious preference options are located under the Configure menu under the heading Preferences. * Logging: Logging options control the creation of the newsbot.log file. The log may be useful so that you have more information about the files that you have downloaded. The log filename is "newsMMDD.log", where MM is the current month and DD is the current day. For example, the log for June 6 would be saved in "news0606.log". The log is plain-ascii and may be viewed with any text editor, or by using the dos "type" command. * * Log Enabled: if checked, then a log file will be written. Otherwise, no log will be written. * From user name: records the name of the person who posted the message * Subject: records the subject line of the message * Comments: records the first few non-file lines of the message. Useful if the sender has prefaced the message with some explanation of it's content. * Too Small: Messages that are below the minimum message line limit (see below). * Too Long: Messages that are above the maximum message line limit (see below). * Dupe File: Files caught by the dupe-file checker. * Dupe Subject: Messages caught by the dupe-subject checker. * Mime Headers: Really just debugging info for me.... * Delete old Logs: If checked, then any log files not equal to the current date will be deleted when newsbot is run. This minimizes wasted space of old log files laying around. * Path Expansion: These options will place files from different groups in different download directories. * None: Places all downloaded files in the same download directory. * Group Number: Places files in directories designated by their group numbers in the group list (i.e. the first group would go in .../group1/, the second in .../group2/, etc) * Group Name: Transforms the group name into a download path. For example, alt.binaries.pictures.astro would go into .../alt/binaries/pictures/astro/. * Case Conversion: These options will convert the case of the filename to a uniform format. Due to limitations in the 16-bit application model, filenames will probably be always upper case in the 16-bit executable. * None: Leave filename case as it appears over the modem. * Lower: Convert all filenames to lower case. * Upper: Convert all filenames to upper case. * Disconnect/Reconnect: The disconnect/reconnect system will automatically disconnect and reconnect the current nntp session in order to abort a message that is being transmitted. This is used to save time, rather than downloading entire messages which are not stored on disk. If you disable disconnect/reconnect, then Newsbot will receive the entire message, although the message will not be written to disk. You may toggle disconnect/reconnect on or off for several subcases: * No-Data: Messages with more non-encoded lines than the no-data threshold set below. * Dupe-File: Files caught by the dupe-file checker (not relavant to the dupe-subject checker, as the dupe-subject checker won't download the message in the first place) * No-Mask: Files which do not match an acceptable file mask. * Current-Delete: Files which were deleted during transfer by a user request. * Lockout-File: Files whose names are matched by the lockout filename system. * Message Line Limits: Newsbot can limit which messages are downloaded based on the length (number of lines) of the message. * Minimum Lines: Messages with fewer than the specified number of lines will not be downloaded. The rationale is that small messages do not contain any useful information (pictures are big!) and some efficiency may be obtained by not downloaded them. * Maximum Lines: Messages with greater than the specified number of lines will not be downloaded. The rationale is that really huge messages are a waste of time. It's usually the case of someone who scanned in a picture at too fine a resolution. * "No Data" Threshold: This specifies the number of lines that can be in a message when no attached files (eg images) can be found before the message is skipped. For example, if 500 lines are scanned, and no attached image is present, then the message is probably grunged, a misplaced part of a multipart encode, or some other useless data. Lockout Lists The lockout lists are used to lock out messages that contain certain text strings. For example, you may not wish to download messages written by a certain person, or messages whose title contains a certain string. There are several types of lockout lists supported: * Lockout Poster (i.e. Author): Applies to the "From:" line of the nntp message. Use this if you wish not to receive messages from a specific person. * Lockout Subject: Applies to the "Subject:" line of the nntp message. * Lockout FileName: Prevents specific filenames from being downloaded. All lockout lists are case insensitive (i.e. capitolization does not matter). All lockout lists support wildcard characters, * and ?. "*" is interpreted as "any sequence of zero or more characters" and "?" is interpreted as "any single character". Here are a few sample wildcard strings: * (Lockout-Poster) "joe@fubar.com" would refuse any messages from joe@fubar.com * (Lockout-Poster) "*@foobar.com" would refuse messages from ANYONE at fubar.com * (Lockout-Poster) "joe@*" would refuse messages from "joe" no matter where he posts from * (Lockout-Filename) "*.zip" would refuse any file that had an extention of ZIP * (Lockout-Filename) "fastcash*.*" would refuse any file which had "fastcash" as the first 8 characters. * (Lockout-Filename) "index-?.jpg" would refuse any file which had "index-", exactly one more character, and then ".jpg" as an extention. The is an additional toggle which lets you toggle between an "exact match" and "match any position in text". If it is checked on "exact match", then your pattern must match beginng at the first character of the text, if it is checked on "match any position", then your pattern will match if the pattern exists anywhere in the text. There is a TEST button on the lockout dialog which will let you enter a string and see if it matches anything. This is useful if your a little confused about the wildcard strings and want to make sure what you entered does actually do what you think it should. Authentication: The authentication options are used for news servers which require a user name and password to access the newsgroups. If your news server does not require a name and password, then you should leave this option alone. Authentication is located under the Configure menu. Acceptable File Masks Newsbot allows you to specify which types of files will be downloaded. Specifying "*.*" will enable any file to be downloaded, and this is the default. For example, if you only wanted to receive images, then you may wish to remove *.* and add in *.gif and *.jpg. If you only wanted archives, add in *.zip. etc. Dupe Checking Two different types of dupe checking are supported. They may be used independantly, or you can use both of them at the same time if you wish. By default, the dupe checkers maintain a list of approximately the most recent 2048 messages received. * Subject Dupe Checker: Remembers duplicates by keeping track of the subject of the message. Normally, the Dupe-Subject checker should be used with the "Consider 'From'" setting checked as enabled. Thus, if the same person posts multiple messages with the same subject, the message will be flagged as a dupe. This is useful for preventing download of crossposted messages (i.e. identical messages posted in multiple groups) * Filename Dupe Checker: Remembers duplicates by keeping track of the names of the files that were downloaded. This is very effective at weeding out dupes, but has the side effect of sometimes deleting messages that are not really duplicates. For example, there are a lot of files name "1.JPG" out there! Both of the two above mentioned dupe checkers have some options that control their behaviour: * Reject Duplicates: If checked, then dupe checking will be performed as stated above. If not checked, then no dupe checking will be performed. * Save dupe list: If checked, then dupe information will be saved from session to session. If not checked, then dupe information will only be kept for the current session. * Items to Keep: This is the number of items to "remember" for the dupe checker. A bigger number here will remember more messages (or files), but will also require more memory and/or disk space to hold to dupe checker information. * Consider "From": If enabled, requires the from fields of two messages to be indentical for the files to be considered duplicates. A good idea for the dupe subject checker, but probably not advisable for the dupe file checker. * Consider "Lines": If enabled, requires the number of lines of two messages to be within a 10% tolerance for the messages to be considered duplicates. For multi-part files, only the first message is considered. ------------------------------------------------------------------------------- Statistical Options There are a variety of options listed under the statistics menu item. Most of these simply return information that SBNews has collected while processing newsgroups. * Similar NewsGroups: Newsgroup headers contain an "Xref:" line which lists other newsgroups to which a specific article has been posted. Many times, a poster will "cross-post" an article to multiple groups which share the same interests. The Similar Newsgroups list will display all of the accumulated Xref's for the current area. * HTTP Grabber: The HTTP Grabber automatically keeps a list of any WWW addresses that are found in the text and/or subject lines of downloaded messages. In addition, a HTML file called HTTPGRAB.HTM will be written to the newsbot directory. You may load this file with your WWW browser and look up any references that SBNews found. * Dupe File List: The list that the dupe-checker keeps in memory to catch dupe files. * Dupe Subject List: The list that the dupe-checker keeps in memory to catch duplicate subject lines. ------------------------------------------------------------------------------- Unfinished Stuff This program is still largely unfinished. There's lots of "bells-and-whistles" that I'm planning on adding in the future. So don't dispair if a few features that you'd like are missing. Important things I'm hoping to add are: * Ability to download newsrc (the list of newsgroups) * Ability to specify search filters (i.e. only retrieve messages with xxx in the subject field) * Manual toggling of articles you want to download ------------------------------------------------------------------------------- Acknowledgements I would like to thank the following people for their help, support, or contributions to SBNews/Newsbot: * The many people who've helped me beta-test! * The Independent JPEG Group * The ISO Jpeg Standards Committee * Brian Kantor & Phil Lapsley (RFC977) * S. Barber (NNTP Extensions document) * The Papa winsock site * The Tucows winsock site * Lars Magne Ingebrigtsen, "Notes on News" ------------------------------------------------------------------------------- Registration SBNews/Newsbot is a shareware program and as such, you are only granted the right to operate it for a limited time to evaluate it's performance. Continued usage requires registration in the amount of $15.00. The registration policy is as follows: 1. Mail $15 (a personal check made out to "Scott M. Baker" is fine) to my US-Mail address. Include your EMAIL address with the check. 2. I will generate a registration code that corresponds to the email you included. I will email this registration code back to you at the address you provided. 3. Load up newsbot and select Register from the menu. Enter your email and the code I provided. The program will now be registered. 4. Your registration is good forever, and will be supported under ALL FUTURE VERSIONS of this program. Registration entitles you to use of the program (and all future upgrades) for as long as you wish. The 15-second shareware delay while loading will be removed. In addition, if I add additional utility programs to the SBNews package, your registration will entitle you to usage of them as well. Please do consider registering this shareware if you continue to use it. The programming is very involved and time consuming, and it really gives me a lot more incentive to add new features if I receive compensation for my work. ------------------------------------------------------------------------------- Contacting Me US-Mail: Scott M. Baker 2241 W Labriego Tucson, Az 85741 My Bulletin board: The Not-Yet-Named BBS (520) 544-4655 (USR Dual 14.4k) (520) 797-8573 (USR Sportster 28.8k) Email: smbaker@primenet.com My Homepage: http://www.primenet.com/~smbaker ------------------------------------------------------------------------------- Revision History * Version 1.0 * Initial public release * Version 1.1 * Added about dialog and shareware delay * Cleaned up (shrunk) main screen and reorganized information * Added context-sensitive hint text at bottom * Fixed problem with message pointers not being recorded properly * Version 1.2 * Added JPG viewer window (supported in 800x600 or above only) * Probably fixed a nasty stack overflow (Runtime Error 202) bug that was causing trouble with new users. * Version 1.3 * Made JPEG viewer more crash resistant * Increased JPEG viewer read buffer to deal better with images with large headers * Added a title to the main window * Finished registration code system * Version 1.4 * Added icon * Version 1.5 * Made sure to free gensock.dll on exit * Added Configure:Preferences dialog, many new preference options * Version 1.6 * Raised counter-over threshold from 250 to 1000 * Set update timer at 100ms instead of 250ms * Added Lockout Poster system * Revamped dupe checker for better reliability * Added similar newsgroups list * Raised maximum number of newsgroups from 32 to 128 * Added view and delete options to current and previous menus * Fixed problem with crashes on high-ascii characters in lockout poster list * Fixed possible bug with long header lines * Now handles nonexistant newsgroups without halting * Version 1.7 * Shareware delay now increments by 1 every day Newsbot is used * Newsbot now stops loading groups from INI file when first blank entry is found (speedup?) * Revamped the dupe message checker (again!) * Fixed bug which was causing unnecessary JPEG error 117 * Fixed problem with decoding lines that started with a double period * Version 1.8 * Added message when old BWCC found * Version 1.9 * Change "about" menu to "file" menu * Added open jpeg viewer option to file menu * Prevented crash if 0 messages in a group * Added HTTP URL Grabber * Fixed problem with long subject/from lines (i.e. >256 chars) * Double clicking on the log will bring up a viewer * Made sure files are written to proper directory * All uudecode filenames are normalized to lower case * Added dupe file checker * Revamped dupe subject checker (again!) * Version 2.0 * Added authentication support (AUTHINFO Original) * Ability to deal with SOME mime messages * Added acceptable file mask dialog * 32-bit version available * Incorporated gensock code into SBNews; Gensock.DLL no longer needed. * Fixed memory leak in jpeg viewer * Used suballocater to deal with strman hash table nodes * Added "add" and "refresh" options to similar groups dialog * Version 2.1 * Spiffed up Jpeg viewer window * Fixed up error handling on JPEG decoding routines * Fixed possible error with "Null" strings * Version 2.2 * Fixed problems with palette in 256 color mode * More tolerant of uuencoded lines with extra characters * Made small version of window even smaller * Dupe File, Delete Current, and No Mask signals will now disconnect/reconnect the session to keep from having to download entire message. * Several user-friendliness improvements * Included SAMPLE_G.TXT, a list of binary picture newsgroups * Added "-i [INIFILENAME]" command line parameter * Added Save as BMP option to JPEG viewer * Added Set Wallpaper option to JPEG viewer * Added normal vs Wallpaper mode toggle to JPEG viewer slideshow * If you drag & drop a JPG file into newsbot's main window, then Newsbot will launch a Jpeg viewer window to view the file. * Moved dupe checker preferences to their own menu item * Added "Items to keep" settings for dupe checkers * Fixed major distribution problem in hash function * Added Blank option to Current/Previous menus to blank out the current picture * Viewer enable defaulted to active * Fixed problem with long pathnames in jpeg viewer * Fallback when improper BWCC.DLL loaded * Version 2.3 * Fixed GPF on loadup of nonregistered users * Version 2.4 * Fixed problem with invalid (missing?) dates causing GPF * Made mime filename guesser stop at punctuation (except _ -) * Added color options to Jpeg viewer * Automatic restart after communications errors * Added automatic start command line option -auto * Added about box disable command line option -noabout * Fixed possible GDI leak in small viewer windows * Added wildcard (* and ?) support to lockout poster list * Cached hostname/ip address for reconnect * Added table of contents to documentation * Included seperate register.htm / register.doc * Automatic disconnect when all groups finished * Added Exit menu item to file menu * Version 2.5 * Fixed GPF when lockout poster list empty * Version 3.0 * Major redisign of internal header listing * Use linked list instead of array * Use hash table while downloading rather than linear search * Modification to header sort methods * Store from & subject in hash table * MSGID only stored for duration of current message; Needed to save wasted memory. * Major redesign of dupe checker routines * Store from & subject lines in same hash table as header listing to reduce memory requirements. * Allow FROM and LINES fields to be included in comparison. * FROM and LINES fields remembered and saved to disk * Addition of [Headers] button to group list: Manually configure header table * Added Received File statistics to statistics menu * Added Font configuration to config menu * Added help text for menu options * Added FROM and LINES toggles to dupe file checker * Added LINES toggle to dupe subject checker; Made FROM toggle-able * Revamped re search engine to support more than 4000 re nodes. * Added Lockout-Subject and Lockout-Filename * Made lockouts so they can be restricted to specific groups * Revamped dupe checker dialog to include edit and find/findnext features * Added stateset system to RE checker to improve speed in RE matchings * Log files now include date in filename to make deleting old logs easier * Delete old logs option added; Defaulted to on; gets rid of previous day(s) log files. * Added delete button to jpeg viewer to speed up deleting. * Added horizontal scrollbars to main dialog list boxes * Added autosave configuration * Added Save Preferences command * Rewrote ctime function to tolerate bad dates * Added Auto-Encryption to downloader * Added Auto-Decryption to viewer * Added Encrypt.exe and Decrypt.exe