Source Code 1994 March/Source_Code_CD-ROM_Walnut_Creek_March_1994.iso/compsrcs/misc/volume36/slurp/part01

Text File | 1993-03-11 | 51.8 KB | 2,006 lines

Newsgroups: comp.sources.misc
From: [e-mail] (Stephen Hebditch)
Subject: v36i013:  slurp - A passive NNTP transfer client, v1.05, Part01/02
Message-ID: <csm-v36i013=[e-mail]>
X-Md4-Signature: 889cd05f55838480418775b06acc6d05
Date: Fri, 12 Mar 1993 16:59:13 GMT
Approved: [e-mail]

Submitted-by: [e-mail] (Stephen Hebditch)
Posting-number: Volume 36, Issue 13
Archive-name: slurp/part01
Environment: ANSI-C, UNIX, !SCO

Slurp is an advanced passive NNTP client for UNIX. It will connect to a
remote NNTP server and retrieve articles in a specified set of Usenet
newsgroups that have arrived after a particular date (typically the
last time it was invoked) for processing by your local news system. It
replaces nntpxfer from the NNTP 1.5.11 reference implementation and
nntpget from the INN distribution.
-----------
#! /bin/sh
# This is a shell archive.  Remove anything before this line, then feed it
# into a shell via "sh file" or similar.  To overwrite existing files,
# type "sh file -c".
# Contents:  README HISTORY articles.c newnews.c slurp.c
# Wrapped by kent@sparky on Fri Mar 12 10:55:33 1993
PATH=/bin:/usr/bin:/usr/ucb:/usr/local/bin:/usr/lbin ; export PATH
echo If this archive is complete, you will see the following message:
echo '          "shar: End of archive 1 (of 2)."'
if test -f 'README' -a "${1}" != "-c" ; then
  echo shar: Will not clobber existing file \"'README'\"
else
  echo shar: Extracting \"'README'\" \(8586 characters\)
  sed "s/^X//" >'README' <<'END_OF_FILE'
X                SLURP - An advanced passive NNTP client
X                =======================================
X
X                             Version 1.05
X                      Written by Stephen Hebditch
X                     (C) 1992/3 TQM Communications
X
X                                 ------
X
XSlurp is an advanced passive NNTP client for UNIX. It will connect to a
Xremote NNTP server and retrieve articles in a specified set of Usenet
Xnewsgroups that have arrived after a particular date (typically the
Xlast time it was invoked) for processing by your local news system. It
Xreplaces nntpxfer from the NNTP 1.5.11 reference implementation and
Xnntpget from the INN distribution.
X
X
XThe Feature List.
X
X- No restriction on the number of newsgroups that can be retrieved in a
X  session. The client does a series of NEWNEWS requests if necessary to
X  build up a list of articles which need to be retrieved from the
X  server.
X
X- Compatible with C News and INN.
X
X- No lost articles due to unsynchronised clocks. By default, the time
X  for the next NEWNEWS is set by a call to the tcp time service on the
X  remote machine. This fixes a problem in nntpxfer and nntpget where
X  articles can be missed if the clock on the server is behind that on
X  the client.
X
X- Faster article throughput. If configured to do so, slurp stacks up
X  the next article request before the current article is received,
X  decreasing the total article transfer time by around 30%. Unlike
X  nntpget, article batches can be written to the rnews spool directory
X  for processing with 'rnews -U' later, rather than being offered to
X  innd in turn. When used with C News, batches are written to the
X  in.coming directory. This provides a big speedup over nntpxfer which
X  created a new pipe to rnews for each article, lowering the transfer
X  speed and increasing the load on the client machine. These speed
X  improvements are of most use to the growing number of people
X  retrieving news over a dialup SLIP/PPP connection.
X
X- Error recovery. In the event of an error occurring while articles are
X  being fetched, the message IDs of unretrieved articles are dumped to
X  a file. The next session can then retrieve the remaining articles
X  without having to restart from the previous start time.
X
X- Easy configuration.
X
X  slurp.sys defines which groups you wish to get from which server in a
X  similar way to the normal C News sys file, e.g.
X
X    news.demon.co.uk:demon.*,!demon.msdos.announce,uk.*,\
X    !uk.net.maps/world,uk
X
X  slurp.<hostname> contains the time to be used for the NEWNEWS request
X  at each server.
X
X  The newsgroups list, start time and distributions can be overridden
X  on the command line. There are also settings to enable debugging
X  output, prevent writing the time for the next NEWNEWS to
X  slurp.<hostname>, issue a 'MODE READER' command to switch innd to
X  nnrpd, not dump unfetched message IDs in the event of an error, and
X  use the local time for the next NEWNEWS rather than the time at the
X  server.
X
X- Support for the NNTP simple authorisation protocol.
X
X- Logging to syslog of the number of new, missing and duplicate
X  articles at the end of a session plus, if required, the transfer
X  speed of the article fetching stage.
X
X- Independent of the NNTP 1.5.11 code.
X
XSlurp was written in ANSI C under Dell's SVR4/386, but ports easily to
XSunOs 4.x, Linux, BSDI BSD/386, HPUX and most other recent versions of
XUNIX.
X
X
X
XCONTENTS
X--------
X
XIn this archive you should have the following files:-
X
XHISTORY        How we got from there to here.
XMakefile    The Makefile (yes, really!)
XREADME        What you're reading now
Xarticles.c    Routines for reading articles and batching them up
Xconf.h        Compilation configuration file
Xfakesyslog.c    Fake version of syslog for those without
Xfakesyslog.h    Definitions needed for the above
Xhistory.c    Routines to open and interrogate the news history file
Xmisc.c        Miscellaneous routines
Xnewnews.c    Routines to retrieve new message ids from the server
Xnntp.h        Definitions of nntp response codes
Xslurp.1        Slurp manual page
Xslurp.c        Main program
Xslurp.h        Definitions used by everything
Xslurp.sys    Configuration file for groups to take from a server
Xsockets.c    Server interface routines
Xspace.c        Determines space available on news disk
Xtime.c        Retrieves current time at remote server
X
X
X
XCOMPILATION AND INSTALLATION
X----------------------------
X
X1. Edit conf.h to reflect your reflect your local configuration. This
Xfile contains full details about each item that you may need to change.
X
X2. Edit Makefile. If you have not installed dbz.h in /usr/include then
Xyou will need to add -Ipathname to CFLAGS so that the compiler does a
Xsearch of the directory where it can be found. LIBS will need to
Xcontain entries to pick up the dbz library and possibly the sockets
Xlibrary depending on your system. If you don't have syslog then you
Xwill need to uncomment the FAKESRC and FAKEOBJ lines.
X
X3. Type 'make' and hopefully you will get the slurp executable.
X
X4. Copy slurp to wherever it is that you store your news binaries.
X
X5. Copy slurp.sys to whichever location you selected for it in conf.h.
X
X6. Copy the manual page to wherever you keep your manual pages.
X
X
X
XQUICK START
X-----------
X
XThe slurp manual page contains full details on how to operate slurp. As
Xa quick start all you need to do is:-
X
X1. Edit slurp.sys to set the hostname of the server you wish to connect
Xto for retrieving articles and the groups you wish to receive.
X
X2. Edit slurp.<hostname> and enter a single line containing the time a
Xcouple of days previously, say, in 'YYMMDD HHMMSS' format.
X
X3. Run 'slurp <hostname>' and sit back while the articles flood into
Xyour local news system.
X
XIf you are running C News, then the next time that newsrun is invoked
Xfrom cron, the articles in the in.coming directory will be added to the
Xnews system.
X
XIf you are running INN and do not have RNEWS defined, then it will be
Xnecessary to run 'rnews -U' to have the files from the rnews spool
Xdirectory processed by the news system. If RNEWS is defined then they
Xwill be piped to innd as they arrive - doing this will lower the speed
Xat which articles are transferred from the remote server.
X
XPatches for the NNTP 1.5.11 server are available on request which
Ximplement caching of article locations during a NEWNEWS so that a
Xfurther access of the history file is not required when an article is
Xretrieved. I hope to have an equivalent patch available for INN soon.
X
X
X
XCONDITIONS OF USE
X-----------------
X
XThe Slurp package and individual code as detailed is Copyright 1992/3
XTQM Communications, BCM Box 225, London, WC1N 3XX. All rights reserved.
X
XThis package may be freely distributed provided the following
Xconditions are met. The files detailed in the contents section of this
Xfile must all be included. No charge, other than basic online charges
Xor media costs, may be levied. You are permitted to make alterations to
Xthe code for your own personal use, but please feed back any bug fixes
Xor enhancements to us so they may be included in any future releases.
X
X
X
XDISCLAIMER
X----------
X
XTQM Communications makes no warranty of any kind in respect to this
Xdocumentation and the software described. The user assumes any risk as
Xto the quality and performance of this product. In no event will TQM
XCommunications be liable for direct, indirect, incidental or
Xconsequential damages arising from any defect in the performance and
Xuse of this product.
X
XAll trademarks acknowledged.
X
X
X
XACKNOWLEDGEMENTS
X----------------
X
XThe original inspiration for slurp came from nntpxfer, written by Brian
XKantor. The article batching code is inspired by batch.c in the NNTP
X1.5.11 package, written by Stan Barber and 'based on work by Henry
XSpencer and Geoff Collyer'. The fakesyslog code is taken from the NNTP
X1.5.11 package where it is credited to John Robert LoVerso and Paul
XMcKenny. The space code to determine whether there is enough space on
Xthe news disk was originally taken from the NNTP 1.5.11 package where
Xit is credited to Stan Barber, Tad Guy, Chris Jepeway and Tom Lane.
XRegular expression expansion courtesy of Nick Lai. Network knowledge
Xthanks to W. Richard Stevens, author of the most readable UNIX books
Xaround. For bug fixing, porting and enhancement ideas, thanks go to Tim
XRylance, Pete Bentley, Stefan Schwarz, David Clunie and David Boyce.
X
X
X
XCONTACT INFORMATION
X-------------------
X
XAll comments, bug reports, bug fixes, etc. should be sent to
[e-mail]. I'm also available for consultancy work if
Xyou want help in getting news and mail working.
X
X
X---- END OF DOCUMENTATION
END_OF_FILE
  if test 8586 -ne `wc -c <'README'`; then
    echo shar: \"'README'\" unpacked with wrong size!
  fi
  # end of 'README'
fi
if test -f 'HISTORY' -a "${1}" != "-c" ; then
  echo shar: Will not clobber existing file \"'HISTORY'\"
else
  echo shar: Extracting \"'HISTORY'\" \(3564 characters\)
  sed "s/^X//" >'HISTORY' <<'END_OF_FILE'
XSLURP HISTORY
X-------------
X
XV1.05 - 1 March 1993
X
X    Completely cleaned up space.c. Added proper support for space
X    checking in SVR4.
X
X    Modified read_sys so that it is possible to have lines in the
X    slurp.sys file which are longer than BUFSIZ.
X
X    Check unretrieved message IDs in the slurp.<hostname> file start
X    and end with chevrons.
X
X    Moved sublist in the command line arguments to being specified
X    after the hostname with a slash separating them, to make things
X    more consistent.
X
X    A few more minor tidy-ups, in particular portability has been
X    improved in a few places.
X
X
XV1.04 - 14 February 1993
X
X    Malloc message id separately from mnode structure as the previous
X    method seemed to cause problems on some architectures.
X
X    Now handles article lines of >= 511 bytes without overwriting
X    memory.
X
X    Support for simple authorisation protocol.
X
X    Support for INN 'MODE READER' command.
X
X    read_sys rewritten to allow command line flags to be specified for
X    servers, support having more than one set of newsgroups to be
X    retrieved from the same server, and provide the username and
X    password for the simple authorisation protocol.
X
X    New choice of piping articles to rnews (possibly in one batch only)
X    or writing article batches to the incoming news / rnews spool
X    directory. This will provide a speed-up for users of INN.
X
X    Replaced slurp.tim file with slurp.<hostname> files, solving
X    locking problems and providing support for the enhancement below.
X
X    If an error occurs during the article fetching stage, then dump the
X    message ids of articles not yet retrieved to slurp.<hostname>.
X    These can then be picked up in the next session, without having to
X    restart the NEWNEWS back at the previous time.
X
X    No requirement for slurp.<hostname> to exist as slurp.tim used to
X    have to be if you wanted a time written out.
X
X    Re-arranged the command line options.
X
X
XV1.03 - 19 December 1992
X
X    Added extra compilation configuration option for machines that
X    don't have strerror().
X
X    If a temporary file was opened for reading a long article into
X    which wouldn't fit in memory, then it was flushed at the end of
X    read_article. Unfortunately, the test wasn't done correctly so if a
X    long article was once read in, it would keep flushing the now
X    invalid file descriptor on future calls. Fixed by removing this
X    altogether as the subsequent rewind would do the job anyway.
X
X    Open syslog *before* we start doing things that might want to write
X    to it... Connected, articles retrieved and speed messages logged
X    as LOG_INFO instead of the previously wrong LOG_NOTICE.
X
X    When mallocing space for the article id tree, only malloc enough
X    room for the article id and left/right node pointers, not the whole
X    structure, saving oodles of memory.
X
X    Corrected the test for the maximum size of a NEWNEWS request line.
X
X    Various minor tidy-ups.
X
X    Added manual page and made various improvements to the
X    documentation.
X
X
XV1.02 - 7 December 1992
X
X    Corrected openlog calls so correct 4.2/4.3 version of syslog is
X    used.
X
X
XV1.01 - 6 December 1992
X
X    Fixed some problems with null-dereferencing of nn_distributions.
X    New slurp.tim not written if maximum number of messages that can
X    be transferred is hit.
X
X    When debugging switched on, print line being sent to server before
X    it is sent rather than after.
X
X    Added documentation on return codes.
X
X
XV1.00 - 2 December 1992
X
X    Initial release.
X
X---- END OF DOCUMENTATION
END_OF_FILE
  if test 3564 -ne `wc -c <'HISTORY'`; then
    echo shar: \"'HISTORY'\" unpacked with wrong size!
  fi
  # end of 'HISTORY'
fi
if test -f 'articles.c' -a "${1}" != "-c" ; then
  echo shar: Will not clobber existing file \"'articles.c'\"
else
  echo shar: Extracting \"'articles.c'\" \(9316 characters\)
  sed "s/^X//" >'articles.c' <<'END_OF_FILE'
X/*
X * articles - handle retrieval and batching of articles
X *
X * Copyright (C) 1992/93 Stephen Hebditch. All rights reserved.
X * TQM Communications, BCM Box 225, London, WC1N 3XX.
X * [e-mail]  [phone]
X *
X * See README for more information and disclaimers
X *
X * This file provides a set of routines to retrieve articles from the
X * remote NNTP server and add to a batch of articles being piped to
X * the local news system via rnews.
X *
X * $Id: articles.c,v 1.5 1993/03/01 17:36:51 root Exp $
X *
X * $Log: articles.c,v $
X * Revision 1.5  1993/03/01  17:36:51  root
X * Removed stray text after an endif.
X *
X * Revision 1.4  1993/02/14  14:44:25  root
X * Fixed problem with newline being added to article line buffer which
X * could already be at its maximum length.
X * Support for writing out batch files as well as piping batches to
X * rnews.
X * If BATCHSIZE is zero then keep pipe open to rnews indefinitely.
X * If error occurs then submit batch and dump unretrieved ids.
X *
X * Revision 1.3  1992/12/15
X * Minor tidy-ups, plus fixed flushing of tfp every time after it had
X * been opened once.
X *
X * Revision 1.1  1992/12/04
X * Print line before it is sent to server when debugging is on.
X *
X * Revision 1.0  1992/11/27
X * Adapted from nntpxfer-e code.
X * Pipe batches to rnews instead of creating in in.coming directory,
X * so can be used with INN.
X *
X */
X
X#include "slurp.h"
X
X
Xstatic struct mnode *last_article = NULL; /* last article requested */
Xstatic char artbuf [COPYSIZE];        /* temp storage for article in memory */
Xstatic char *endart = artbuf;        /* points to just past end of article */
Xstatic int incore = TRUE;            /* article in memory, not temp file */
Xstatic FILE *tfp = NULL;            /* temporary file descriptor */
X
Xstatic FILE *batchfp = NULL;        /* file descriptor of rnews pipe */
Xstatic size_t batchsize = 0;        /* size of current batch */
Xstatic char batchname [PATH_MAX];    /* name of current batch */
X
Xstatic void new_batch ();
Xstatic void read_article ();
Xstatic void batch_article ();
Xstatic void fetch_article ();
Xstatic void get_article ();
Xstatic void request_article (char *msgid);
Xstatic void traverse_tree ();
X
X
X/*
X * new_batch - Determines if there is enough room for the batch on the
X * disk containing the news spool directories. If there is, then a pipe
X * is opened to rnews and the batch variable initialised with the
X * details.
X */
X
X    static void
Xnew_batch ()
X    {
X    /* Make sure there is enough room for batch */
X#ifdef MINFREE
X    if (!space (MINFREE))
X        {
X        log_msg ("new_batch: Not enough space for incoming batch");
X        if ((root != NULL) && (!no_time_flag) && (!no_id_load_flag))
X            set_ntime ();
X        exit (5);
X        }
X#endif
X
X#ifdef RNEWS
X    /* Open a pipe to rnews for the batch */
X    if ((batchfp = popen (RNEWS, "w")) == NULL)
X        log_sys ("new_batch: Can't open pipe to %s", RNEWS);
X#else
X    /* Open a file in incoming news directory with temporary name */
X    (void) strcpy (batchname, BATCHNAME);
X    (void) mktemp (batchname);
X    if ((batchfp = fopen (batchname, "w")) == NULL)
X        log_sys ("new_batch: Can't open file %s", batchname);
X#endif
X    }
X
X
X/*
X * read_article - Read an article into artbuf or, if too large, into a
X * temporary file from the currently open NNTP server socket, reading up
X * to the end-of-article marker, a '.' on a single line. If it is stored
X * in memory, then incore will be TRUE, otherwise it will be FALSE and the
X * temporary file name will be stored in tempfile.
X */
X
X    static void
Xread_article ()
X    {
X    char *realline;
X    char line [NNTP_STRLEN];
X    int lines = 0;
X    int len;
X
X    incore = TRUE;
X    endart = artbuf;
X
X    /* Read in the article */
X    for (;;)
X        {
X        get_server (line, sizeof (line));
X
X        /* Dot on its own means article end reached */
X        if (!strcmp (line, "."))
X            break;
X
X        /* remove hidden dot if present */
X        realline = (line [0] == '.' ? line + 1 : line);
X
X        /* Article is currently stored in memory */
X        if (incore)
X            {
X            /* If no room in artbuf, open tempfile and copy article there */
X            len = strlen (realline);
X            if ((endart + len + 2 ) > (artbuf + sizeof (artbuf)))
X                {
X                if ((tfp = tmpfile ()) == NULL)
X                    log_sys ("read_article: Can't create temporary file");
X                (void) fwrite (artbuf, 1, endart - artbuf, tfp);
X                if (ferror (tfp))
X                    log_sys ("read_article: Can't write to tempfile");
X                (void) fputs (realline, tfp);
X                (void) putc ('\n', tfp);
X                if (ferror (tfp))
X                    log_sys ("read_article: Can't write to tempfile");
X                incore = FALSE;
X                }
X            else
X                {
X                /* If fits, append realline to artbuf at endart */
X                (void) strcpy (endart, realline);
X                endart += len;
X                *endart++ = '\n';
X                *endart = '\0';
X                }
X            }
X
X        /* Already writing article to temp file */
X        else
X            {
X            (void) fputs (realline, tfp);
X            (void) putc ('\n', tfp);
X            if (ferror (tfp))
X                log_sys ("read_article: Can't write to tempfile");
X            }
X
X        lines++;
X        }
X
X    /* Article successfully read in */
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %d lines\n", lines);
X    }
X
X
X/* batch_article - Append "#! rnews <count>" and the article from artbuf
X * or temporary file to the batch file.
X */
X
X    static void
Xbatch_article ()
X    {
X    size_t bytes = 0;
X    size_t size = 0;
X
X    /* Find article size */
X    if (incore)
X        size = endart - artbuf;
X    else
X        size = ftell (tfp);
X
X    totalsize += size;
X    batchsize += size;
X
X    /* Print the article header */
X    (void) fprintf (batchfp, "#! rnews %ld %s\n", (long) size, hostname);
X
X    /* Copy the article to the batch file */
X    if (incore)
X        {
X        (void) fwrite (artbuf, 1, size, batchfp);
X        if (ferror (batchfp))
X            log_sys ("batch_article: Can't write to batch");
X        }
X    else
X        {
X        rewind (tfp);
X        while ((bytes = fread (artbuf, 1, sizeof (artbuf), tfp)) > 0)
X            {
X            (void) fwrite (artbuf, 1, bytes, batchfp);
X            if (ferror (batchfp))
X                log_sys ("batch_article: Can't write to batch");
X            }
X        (void) fclose (tfp);
X
X        }
X    }
X
X
X/*
X * fetch_article - Retrieve an article from the currently open NNTP
X * server socket which has already been requested. The article is written
X * to the end of the current batch. If there is not already a batch
X * then a new pipe to rnews for the batch will be opened. If the current
X * batch is too large or has too many articles then the pipe will be
X * closed so that the batch may be submitted to the news system.
X */
X
X    static void
Xfetch_article ()
X    {
X    /* Open a new batch if required */
X    if (batchfp == NULL)
X        new_batch ();
X
X    /* Read in article */
X    read_article ();
X
X    /* Add it to the batch */
X    batch_article ();
X
X    /* Submit batch if ready */
X    if ((batchsize > BATCHSIZEMAX) && (BATCHSIZEMAX != 0))
X        enqueue_batch ();
X    }
X
X
X/*
X * get_article
X */
X
X    static void
Xget_article ()
X    {
X    char status [NNTP_STRLEN];
X
X    /* Read status line from server */
X    get_server (status, sizeof (status));
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s\n", status);
X
X    switch (atoi (status))
X        {
X        /* If missing, then add to missing list */
X        case ERR_NOART:
X            misart++;
X            newart--;
X            return;
X
X        /* If present, then fetch and add to batch */
X        case OK_ARTICLE:
X            fetch_article ();
X            break;
X
X        /* Otherwise must be a protocol error */
X        default:
X            log_msg ("get_article: NNTP protocol error: got '%s'", status);
X            if ((root != NULL) && (!no_time_flag) && (!no_id_load_flag))
X                set_ntime ();
X            exit (4);
X        }
X    }
X
X
X/*
X * request_article - Request an article with specified id from the server
X */
X
X    static void
Xrequest_article (char *msgid)
X    {
X    char request [NNTP_STRLEN];
X
X    (void) sprintf (request, "ARTICLE %s", msgid);
X    if (debug_flag)
X        (void) fprintf (stderr, "<- %s\n", request);
X    put_server (request);
X    }
X
X
X/*
X * traverse_tree - Traverse the tree requesting and getting each article
X */
X
X    static void
Xtraverse_tree (struct mnode *p)
X    {
X    if (p != NULL)
X        {
X        traverse_tree (p->left);
X        request_article (p->msgid);
X#ifdef SPEEDUP
X        if (last_article != NULL)
X            {
X            get_article ();
X            last_article->used = TRUE;
X            }
X        last_article = p;
X#else
X        get_article ();
X        p->used = TRUE;
X#endif /* SPEEDUP */
X        traverse_tree (p->right);
X        }
X    }
X
X
X/*
X * get_articles - Get the articles from the server whose message ids
X * were previously collected with do_newnews.
X */
X
X    void
Xget_articles ()
X    {
X    traverse_tree (root);
X#ifdef SPEEDUP
X    get_article ();
X    last_article->used = TRUE;
X#endif
X    }
X
X
X/*
X * enqueue_batch - Submit the batch to the new system by closing the
X * currently open pipe to rnews or renaming the temporary file in the
X * incoming news directory so it can be seen by the news system.
X */
X
X    void
Xenqueue_batch ()
X    {
X#ifndef RNEWS
X    char permname [PATH_MAX];
X    time_t now;
X#endif
X
X    /* Return if there is no currently open batch */
X    if (batchfp == NULL)
X        return;
X
X#ifdef RNEWS
X    /* Close the pipe to rnews */
X    if (pclose (batchfp))
X        log_sys ("enqueue_batch: Can't close pipe to %s", RNEWS);
X#else
X    /* Close the temporary file */
X    if (fclose (batchfp))
X        log_sys ("enqueue_batch: Can't close %s", batchname);
X
X    /* Rename it so it can be seen by news */
X    for (;;)
X        {
X        (void) sprintf (permname, "%ld.t", (long) time (&now));
X        if (link (batchname, permname) == 0)
X            break;
X        if (errno != EEXIST)
X            log_sys ("enqueue_batch: Error linking %s to %s", batchname, permname);
X        (void) sleep (2);
X        }
X    if (unlink (batchname))
X            log_sys ("enqueue_batch: Error unlinking %s", batchname);
X#endif /* RNEWS */
X
X    /* Reset the batch descriptor for a new batch */
X    batchfp = NULL;
X    batchsize = 0;
X    }
X
X/* END-OF-FILE */
END_OF_FILE
  if test 9316 -ne `wc -c <'articles.c'`; then
    echo shar: \"'articles.c'\" unpacked with wrong size!
  fi
  # end of 'articles.c'
fi
if test -f 'newnews.c' -a "${1}" != "-c" ; then
  echo shar: Will not clobber existing file \"'newnews.c'\"
else
  echo shar: Extracting \"'newnews.c'\" \(8980 characters\)
  sed "s/^X//" >'newnews.c' <<'END_OF_FILE'
X/*
X * newnews - Read in list of ids of new articles
X *
X * Copyright (C) 1992/93 Stephen Hebditch. All rights reserved.
X * TQM Communications, BCM Box 225, London, WC1N 3XX.
X * [e-mail]  [phone]
X *
X * See README for more information and disclaimers
X *
X * Using a previously initialised list of newsgroups, carries out a series
X * of NEWNEWS requests to the connected NNTP server, storing the message
X * ids of new articles in a binary tree in memory.
X *
X * $Id: newnews.c,v 1.5 1993/03/01 17:45:16 root Exp $
X *
X * $Log: newnews.c,v $
X * Revision 1.5  1993/03/01  17:45:16  root
X * Added cast to bzeroing of used_not_group_array.
X *
X * Revision 1.4  1993/02/14  14:55:41  root
X * Malloc msgid space separately from mnode.
X * Split-out process_id from do_newnews so it can be used in get_ntime
X * to load the tree with the unretrieved message ids.
X *
X * Revision 1.3  1992/12/14
X * Only malloc enough space for msgid, not whole mnode structure.
X * Minor tidy-ups.
X *
X * Revision 1.1  1992/12/06
X * Set no_time_flag if hit max no of messages
X * Print line before it is sent to server when debugging is on.
X * No longer need to handle null nn_distributions.
X *
X * Revision 1.0  1992/11/30
X * Transferred functions from slurp.c
X *
X */
X
X#include "slurp.h"
X
Xstatic int hit_max = FALSE;
X
Xstatic char **group_array;
Xstatic char **not_group_array;
Xstatic int  *used_not_group_array;
X
Xstatic int  groups_no;
Xstatic int  not_groups_no;
X
X
Xstatic void parse_groups ();
Xstatic int  add_id (char *msgid);
Xstatic void do_newnews (char *line);
Xstatic int  restreql (register char *w, register char *s);
Xstatic int  get_not_groups (char *group);
X
X
X/*
X * parse_groups - Turn list of groups into two arrays containing
X * pointers to groups to include and groups to exclude.
X */
X
X    static void
Xparse_groups ()
X    {
X    int i, got, not;
X    char *cp;
X
X    /* Calculate number of group entries */
X    for (i = 1, cp = nn_newsgroups; *cp != '\0'; cp++)
X        if (*cp == ',')
X            i++;
X
X    /* Malloc space for include and exclude group arrays */
X    if ((group_array = (char **) malloc (i * (sizeof (char *)))) == NULL)
X        log_sys ("parse_groups: malloc %d bytes", i * sizeof (char **));
X
X    if ((not_group_array = (char **) malloc (i * (sizeof (char *)))) == NULL)
X        log_sys ("parse_groups: malloc %d bytes", i * sizeof (char **));
X
X    if ((used_not_group_array = (int *) malloc (i * (sizeof (int)))) == NULL)
X        log_sys ("parse_groups: malloc %d bytes", i * sizeof (int));
X
X    /* Now start parsing the newsgroup list */
X    groups_no = 0;
X    not_groups_no = 0;
X    got = TRUE;
X    not = FALSE;
X
X    for (cp = nn_newsgroups; *cp != '\0'; ++cp)
X        {
X        if (*cp == '!')
X            got = FALSE;
X
X        if (got)
X            {
X            group_array [groups_no] = cp;
X            groups_no++;
X            got = FALSE;
X            }
X
X        if (not)
X            {
X            not_group_array [not_groups_no] = cp;
X            not_groups_no++;
X            not = FALSE;
X            }
X
X        if (*cp == ',')
X            {
X            *cp = '\0';
X            got = TRUE;
X            }
X
X        if (*cp == '!')
X            not = TRUE;
X        }
X    }
X
X
X/*
X * store_node - Store a new node in the binary tree
X */
X
X    static struct mnode *
Xstore_node (char *msgid)
X    {
X    struct mnode *node;
X
X    node = (struct mnode *) malloc (sizeof (struct mnode));
X    if (node == NULL)
X        log_sys ("add_id: malloc %d bytes", sizeof (struct mnode));
X    node->left = NULL;
X    node->right = NULL;
X    node->used = FALSE;
X    node->msgid = (char *) malloc (strlen (msgid) + sizeof (char));
X    if (node->msgid == NULL)
X        log_sys ("store_node: malloc %d bytes",
X                 strlen (msgid) + sizeof (char));
X    (void) strcpy (node->msgid, msgid);
X    entries++;
X    return (node);
X    }
X
X
X/*
X * add_id - Add a message id to the binary tree if not already present.
X * Returns -1 if the maximum number of entries in the tree has been
X * reached, 0 if the item is added okay, 1 if an entry with that
X * particular message id already exists.
X */
X
X    static int
Xadd_id (char *msgid)
X    {
X    struct mnode *current;
X    int test;
X
X    /* Test if hit the maximum number of entries in the cache */
X    if (entries >= MAXCACHE)
X        return (-1);
X
X    /* Handle the case when the tree is empty */
X    if (root == NULL) {
X        root = store_node (msgid);
X        return (0);
X    }
X
X    /* Search the tree for correct position to insert node */
X    current = root;
X
X    for (;;)
X        {
X        test = strcmp (msgid, current->msgid);
X        if (test < 0)
X            {
X            if (current->left == NULL)
X                {
X                current->left = store_node (msgid);
X                return (0);
X                }
X            else
X                current = current->left;
X            }
X        else if (test > 0)
X            {
X            if (current->right == NULL) {
X                current->right = store_node (msgid);
X                return (0);
X                }
X            else
X                current = current->right;
X            }
X        else
X            return (1);
X        }
X    }
X
X
X/*
X * process_id - Check if id already exists in local history file, if not
X * then add it to the message id tree if it isn't already in there.
X */
X
X    void
Xprocess_id (char *msgid)
X    {
X    char *cp;
X
X    /* Modify the message id appropriate to C-News history files */
X    cp = (char *) strchr (msgid, '@');
X    if (cp != NULL)
X        {
X        for (; *cp != '\0'; ++cp)
X            if (isupper (*cp))
X                *cp = tolower (*cp);
X        }
X
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s", msgid);
X
X    if (check_id (msgid))
X        {
X        switch (add_id (msgid))
X            {
X            case -1 :
X                hit_max = TRUE;
X                break;
X            case  0 :
X                newart++;
X                if (debug_flag)
X                    (void) fprintf (stderr, " new\n");
X                break;
X            default :
X                break;
X            }
X        }
X    else
X        {
X        dupart++;
X        if (debug_flag)
X            (void) fprintf (stderr, " dup\n");
X        }
X    }
X
X
X/*
X * do_newnews - Process a newnews for supplied list of groups, adding the
X * resultant data to the message id tree.
X */
X
X    static void
Xdo_newnews (char *line)
X    {
X    char buf [NNTP_STRLEN];
X
X    /* Create a full string to send to the server */
X    (void) sprintf (buf, "NEWNEWS %s %s GMT %s", line, nn_time,
X                    nn_distributions);
X
X    /* Do the actual NEWNEWS */
X    if (debug_flag)
X        (void) fprintf (stderr, "<- %s\n", buf);
X    put_server (buf);
X
X    /* Get the response and check it's okay */
X    get_server (buf, sizeof (buf));
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s\n", buf);
X    if (atoi (buf) != OK_NEWNEWS)
X        {
X        log_msg ("do_newnews: NNTP protocol error: got '%s'", buf);
X        exit (4);
X        }
X
X    /* Now get the data and stick it in the tree */
X    for (;;)
X        {
X        get_server (buf, sizeof (buf));
X        if (!strcmp (buf, "."))
X            break;
X
X        process_id (buf);
X        }
X    }
X
X
X/*
X * restreql -- A small regular expression string equivalence routine
X * purloined from nntp 1.5.11 which credits <[e-mail]>
X * for its creation. Returns 1 if the string pointed to by 's' matches
X * the asterisk-broadened regexp string pointed to by 'w', otherwise
X * returns 0.
X */
X
X    static int
Xrestreql (register char *w, register char *s)
X    {
X    while (*s && *w)
X        {
X        switch (*w)
X            {
X            case '*':
X                for (w++; *s; s++)
X                    if (restreql(w, s))
X                        return (1);
X                break;
X            default:
X                if (*w != *s)
X                    return (0);
X                w++, s++;
X                break;
X            }
X        }
X    if (*s)
X        return (0);
X    while (*w)
X        if (*w++ != '*')
X            return 0;
X
X    return (1);
X    }
X
X
X/*
X * get_not_groups - Add appropriate groups from the exclusion list to
X * a group that is to be requested from the server.
X */
X
X    static int
Xget_not_groups (char *group)
X    {
X    char matchgroups [NNTP_STRLEN];
X    int i;
X    size_t tlen;
X
X    matchgroups [0] = '\0';
X    tlen = strlen (group);
X    for (i = 0 ; i < not_groups_no ; i ++)
X        if (!used_not_group_array [i])
X            if (restreql (group, not_group_array [i]))
X                if ((strlen (matchgroups) + tlen + 3) < NNTP_STRLEN)
X                    {
X                    (void) strcat (matchgroups, ",!");
X                    (void) strcat (matchgroups, not_group_array [i]);
X                    used_not_group_array [i] = TRUE;
X                    }
X                else
X                    return (1);
X    (void) strcat (group, matchgroups);
X    return (0);
X    }
X
X
X/*
X * get_ids - Store in memory a tree of the message ids of new article at
X * the server which match the specified set of groups and distributions
X * for the currently connected host.
X */
X
X    void
Xget_ids ()
X    {
X    char line [NNTP_STRLEN];
X    char newgroups [NNTP_STRLEN];
X    int i, add_comma;
X
X    parse_groups ();
X
X    line [0] = '\0';
X    bzero ((char *) used_not_group_array, not_groups_no * sizeof (int));
X    add_comma = FALSE;
X
X    for (i = 0 ; i < groups_no ; i++)
X        {
X        (void) strcpy (newgroups, group_array [i]);
X
X        if (get_not_groups (newgroups))
X            {
X            log_msg ("get_ids: Not enough room in NNTP line for exclusion list %s",
X                     newgroups);
X            exit (2);
X            }
X
X        if ((strlen (line) + strlen (newgroups) +
X             strlen (nn_distributions) + (size_t) 31) > NNTP_STRLEN)
X            {
X            do_newnews (line);
X            line [0] = '\0';
X            bzero ((char *) used_not_group_array, not_groups_no * sizeof (int));
X            add_comma = FALSE;
X            }
X
X        if ((strlen (line) + strlen (newgroups) +
X             strlen (nn_distributions) + (size_t) 31) > NNTP_STRLEN)
X            {
X            log_msg ("get_ids: Not enough room in NNTP line for newsgroup %s",
X                     newgroups);
X            exit (2);
X            }
X        else
X            {
X            if (add_comma)
X                (void) strcat (line, ",");
X            else
X                add_comma = TRUE;
X            (void) strcat (line, newgroups);
X            }
X        }
X
X    do_newnews (line);
X
X    /* Report if couldn't fit everything in the tree */
X    if (hit_max)
X        {
X        log_msg ("Maximum limit of %d messages hit", MAXCACHE);
X        no_time_flag++;
X        }
X    }
X
X/* END-OF-FILE */
END_OF_FILE
  if test 8980 -ne `wc -c <'newnews.c'`; then
    echo shar: \"'newnews.c'\" unpacked with wrong size!
  fi
  # end of 'newnews.c'
fi
if test -f 'slurp.c' -a "${1}" != "-c" ; then
  echo shar: Will not clobber existing file \"'slurp.c'\"
else
  echo shar: Extracting \"'slurp.c'\" \(17145 characters\)
  sed "s/^X//" >'slurp.c' <<'END_OF_FILE'
X/*
X * slurp - a passive nntp news client
X *
X * Copyright (C) 1992/93 Stephen Hebditch. All rights reserved.
X * TQM Communications, BCM Box 225, London, WC1N 3XX.
X * [e-mail]  [phone]
X *
X * See README for more information and disclaimers
X *
X * This is the main routine for slurp together with the routines to
X * handle the configuration files and command line arguments.
X *
X *
X * $Id: slurp.c,v 1.5 1993/03/01 17:51:33 root Exp $
X *
X * $Log: slurp.c,v $
X * Revision 1.5  1993/03/01  17:51:33  root
X * read_sys can cope with lines longer than BUFSIZ.
X * report when attempting to load unretrieved message ids.
X * Move sublist to a parameter after the hostname, separate by a slash.
X * Changed some system error checking.
X *
X * Revision 1.4  1993/02/14  14:57:43  root
X * Added support for simple authorisation protocol.
X * Added support for INN's 'MODE READER' command.
X * Re-arranged command line options.
X * Rewrote read_sys and added flags and authorisation options to it.
X * Rewrote get_ntime and set_ntime to use a filename of slurp.<hostname>
X * instead of slurp.tim, solving lack of locking and allowing the file
X * to contain a list of unretrieved message ids on the lines following
X * the time.
X * Don't care if slurp.<hostname> doesn't exist already.
X * If RNEWS is not defined, then change to INDIR for writing out batch
X * files.
X *
X * Revision 1.3  1992/12/15
X * Open syslog *before* we start doing things that might write to it.
X * Informational messages logged as LOG_INFO.
X * Assorted minor tidy-ups.
X *
X * Revision 1.2  1992/12/07
X * Corrected test for 4.2/4.3 BSD syslog open.
X *
X * Revision 1.1  1992/12/06
X * Made no_time_flag global.
X * Fixed null dereferencing of nn_distributions.
X *
X * Revision 1.0  1992/08/07
X * Initial coding.
X *
X */
X
X#include "slurp.h"
X#include <sys/stat.h>
X
Xchar *hostname = NULL;
Xchar *pname;
X
Xint debug_flag = FALSE;
Xint no_time_flag = FALSE;
Xint no_id_load_flag = FALSE;
Xstatic int local_time_flag = FALSE;
Xstatic int mode_reader_flag = FALSE;
X
Xint  dupart = 0;
Xint  misart = 0;
Xint  newart = 0;
Xlong totalsize = 0;
X
Xchar *nn_newsgroups    = NULL;
Xchar *nn_time          = NULL;
Xchar *nn_distributions = NULL;
X
Xstatic char *sublist = NULL;
Xstatic char *ai_username = NULL;
Xstatic char *ai_password = NULL;
X
Xstruct mnode *root = NULL;
Xint entries = 0;
X
Xstatic long newdate, newtime;
X
X
X/*
X * test_time - Check NEWNEWS time string is in the right format (ish)
X */
X
X    static int
Xtest_time ()
X    {
X    return (!(isdigit (nn_time [0]) &&
X              isdigit (nn_time [1]) &&
X              isdigit (nn_time [2]) &&
X              isdigit (nn_time [3]) &&
X              isdigit (nn_time [4]) &&
X              isdigit (nn_time [5]) &&
X              isspace (nn_time [6]) &&
X              isdigit (nn_time [7]) &&
X              isdigit (nn_time [8]) &&
X              isdigit (nn_time [9]) &&
X              isdigit (nn_time [10]) &&
X              isdigit (nn_time [11]) &&
X              isdigit (nn_time [12])));
X    }
X
X
X/*
X * parse_args - Parse the command line arguments. Returns 1 if there is
X * an error, otherwise returns 0.
X */
X
X    static int
Xparse_args (int argc, char **argv)
X    {
X    int c;
X    extern int optind;
X    extern char *optarg;
X    char *pos;
X
X    while ((c = getopt (argc, argv, "a:g:t:dilrw")) != EOF)
X        switch (c)
X            {
X            case 'a':    /* Do an authinfo */
X                if (pos = strchr (optarg, '/'))
X                    {
X                    ai_username = optarg;
X                    *pos++ = '\0';
X                    ai_password = pos;
X                    }
X                else
X                    {
X                    (void) fprintf (stderr, "Invalid authinfo username/password");
X                    return (1);
X                    }
X                break;
X            case 'g':    /* Newsgroups list */
X                if (pos = strchr (optarg, '/'))
X                    {
X                    *pos++ = '\0';
X                    nn_distributions = pos;
X                    }
X                else
X                    nn_distributions = "";
X                nn_newsgroups = optarg;
X                no_time_flag++;
X                break;
X            case 't':    /* Start time */
X                nn_time = optarg;
X                break;
X            case 'd':    /* Debugging on */
X                debug_flag++;
X                break;
X            case 'i':    /* Don't load unprocessed ids */
X                no_id_load_flag++;
X                break;
X            case 'l':    /* Use local time */
X                local_time_flag++;
X                break;
X            case 'r':    /* Do a 'MODE READER' */
X                mode_reader_flag++;
X                break;
X            case 'w':    /* Don't set next time */
X                no_time_flag++;
X                break;
X            default:
X                return (1);
X            }
X
X    /* Get server name */
X    if (optind < argc)
X        {
X        hostname = argv [optind];
X        if (pos = strchr (hostname, '/'))
X            {
X            *pos++ = '\0';
X            sublist = pos;
X            }
X        }
X    else
X        {
X        (void) fprintf (stderr, "No server name supplied\n");
X        return (1);
X        }
X
X    /* If groups are specified, then must have a time */
X    if ((nn_newsgroups != NULL) && (nn_time == NULL))
X        {
X        (void) fprintf (stderr, "Time must be specified for -g option\n");
X        return (1);
X        }
X
X    /* Verify that the time is in something like the right format */
X    if (nn_time)
X        if (test_time ())
X            {
X            (void) fprintf (stderr, "Invalid time specification - should be 'YYMMDD HHMMSS'\n");
X            return (1);
X            }
X
X    return (0);
X    }
X
X
X/*
X * read_sys_line - Read a line from the slurp.sys file, skipping lines
X * which are blank or all comments, truncating lines at comments.
X * If the line has not yet all been read or the continued-on-next-line
X * token '\' is present, then returns 1; if eof then returns -1, otherwise
X * returns 0.
X */
X
X    static int
Xread_sys_line (char *line, int size, FILE *sysfp)
X    {
X    int status;
X    char *pos;
X
X    for (;;)
X        {
X        status = 0;
X
X        (void) fgets (line, size, sysfp);
X        if (feof (sysfp))
X            return (-1);
X        if (ferror (sysfp))
X            log_sys ("read_sys: Error reading %s", SYSFILE);
X
X        if (pos = strchr (line, '\n'))
X            *pos = '\0';
X        else
X            status = 1;
X
X        if (pos = strchr (line, '\\'))
X            {
X            *pos = '\0';
X            status = 1;
X            }
X
X        if (pos = strchr (line, '#'))
X            *pos = '\0';
X
X        if (strlen (line))
X            return (status);
X        }
X    }
X
X
X/*
X * read_sys - Read in the appropriate entry from the slurp.sys file
X * for the specified hostname. Stores the relevant newsgroups for that
X * host in nn_newsgroups and the relevant distribution in nn_distributions.
X * Returns 0 if an appropriate entry for the current host is found,
X * otherwise returns 1.
X */
X
X    static int
Xread_sys ()
X    {
X    FILE *sysfp;
X    char buf [BUFSIZ];
X    char searchname [BUFSIZ];
X    struct stat stbuf;
X    size_t tlen;
X    char *pos;
X    char *mbuf;
X    int status;
X    int object;
X
X    /* Attempt to open the sys file */
X    if ((sysfp = fopen (SYSFILE, "r")) == NULL)
X        log_sys ("read_sys: Error opening %s", SYSFILE);
X
X    /* Allocate memory block for storing the details */
X    if (fstat (fileno (sysfp), &stbuf) < 0)
X        log_sys ("read_sys: Can't fstat %s", SYSFILE);
X    if (stbuf.st_size == 0)
X        {
X        log_msg ("read_sys: Host %s not found in %s", hostname, SYSFILE);
X        return (1);
X        }
X    if ((mbuf = (char *) malloc (stbuf.st_size)) == NULL)
X        log_sys ("read_sys: malloc %d bytes", stbuf.st_size);
X
X    /* Create pattern to search for in the sys file */
X    (void) strcpy (searchname, hostname);
X    if (sublist)
X        {
X        (void) strcat (searchname, "/");
X        (void) strcat (searchname, sublist);
X        }
X    (void) strcat (searchname, ":");
X    tlen = strlen (searchname);
X
X    /* Read in file until we find hostname */
X    for (;;)
X        {
X        if ((status = read_sys_line (buf, sizeof (buf), sysfp)) == -1)
X            {
X            log_msg ("read_sys: Host %s not found in %s",
X                     hostname, SYSFILE);
X            return (1);
X            }
X        if (strncmp (buf, searchname, tlen) == 0)
X            break;
X        }
X
X    /* Strip off hostname stuff from front of line */
X    (void) strcpy (buf, buf + tlen);
X    object = 1;
X    nn_newsgroups = mbuf;
X
X    /* Loop through entry */
X    for (;;)
X        {
X        /* Currently adding newsgroups */
X        if (object == 1)
X            {
X            if (pos = strchr (buf, ':'))        /* Flags next */
X                {
X                *pos++ = '\0';
X                object = 3;
X                (void) strcpy (mbuf, buf);
X                mbuf += strlen (buf) + 1;
X                ai_username = mbuf;
X                (void) strcpy (buf, pos);
X                }
X            else if (pos = strchr (buf, '/'))    /* Distributions next */
X                {
X                *pos++ = '\0';
X                object = 2;
X                (void) strcpy (mbuf, buf);
X                mbuf += strlen (buf) + 1;
X                nn_distributions = mbuf;
X                (void) strcpy (buf, pos);
X                }
X            else
X                {
X                (void) strcpy (mbuf, buf);        /* Nothing else this line */
X                mbuf += strlen (buf);
X                }
X            }
X
X        /* Currently adding distributions */
X        if (object == 2)
X            {
X            if (pos = strchr (buf, ':'))        /* Flags next */
X                {
X                *pos++ = '\0';
X                object = 3;
X                (void) strcpy (mbuf, buf);
X                mbuf += strlen (buf) + 1;
X                ai_username = mbuf;
X                (void) strcpy (buf, pos);
X                }
X            else                                /* Nothing else this line */
X                {
X                (void) strcpy (mbuf, buf);
X                mbuf += strlen (buf);
X                }
X            }
X
X        /* Currently setting flags */
X        if (object == 3)
X            {
X            if (pos = strchr (buf, ':'))        /* authinfo user next */
X                *pos++ = '\0';
X            if (strchr (buf, 'i'))
X                no_id_load_flag++;
X            if (strchr (buf, 'l'))
X                local_time_flag++;
X            if (strchr (buf, 'r'))
X                mode_reader_flag++;
X            if (pos)
X                {
X                (void) strcpy (buf, pos);
X                object = 4;
X                }
X            }
X
X        /* Currently setting username */
X        if (object == 4)
X            {
X            if (pos = strchr (buf, '/'))        /* authinfo pass next */
X                {
X                *pos++ = '\0';
X                object = 5;
X                (void) strcpy (mbuf, buf);
X                mbuf += strlen (buf) + 1;
X                ai_password = mbuf;
X                (void) strcpy (buf, pos);
X                }
X            else
X                {
X                (void) strcpy (mbuf, buf);
X                break;
X                }
X            }
X
X        /* Currently setting password */
X        if (object == 5)
X            {
X            (void) strcpy (mbuf, buf);
X            mbuf += strlen (buf) +1;
X            }
X
X        if (status != 1)
X            break;
X
X        status = read_sys_line (buf, sizeof (buf), sysfp);
X        }
X
X    (void) fclose (sysfp);
X
X    if (nn_distributions == NULL)
X        nn_distributions = "";
X
X    return (0);
X    }
X
X
X/*
X * get_ntime - Get the start time for this NEWNEWS for system. Returns 0
X * if an appropriate entry for the current host is found, otherwise 1.
X */
X
X    static int
Xget_ntime ()
X    {
X    FILE *timefp;
X    char buf [BUFSIZ];
X    char filename [PATH_MAX];
X    char *pos;
X
X    /* Attempt to open the time file */
X    (void) strcpy (filename, TIMFILE);
X    (void) strcat (filename, hostname);
X    if (sublist)
X        {
X        (void) strcat (filename, ".");
X        (void) strcat (filename, sublist);
X        }
X    if ((timefp = fopen (filename, "r")) == NULL)
X        log_sys ("get_ntime: error opening %s", filename);
X
X    /* Read in the time and store it */
X    nn_time = (char *) malloc (14);
X    (void) fgets (nn_time, 14, timefp);
X    if (ferror (timefp))
X        log_sys ("get_ntime: Error reading %s", filename);
X
X    /* Return if time doesn't look ok */
X    if (test_time ())
X        return (1);
X
X    /* Load in any message ids following */
X    if (!no_id_load_flag)
X        {
X        if (debug_flag)
X            (void) fprintf (stderr, "Loading any unretrieved message IDs\n");
X        for (;;)
X            {
X            (void) fgets (buf, sizeof (buf), timefp);
X            if (feof (timefp))
X                break;
X            if (ferror (timefp))
X                log_sys ("get_ntime: Error reading %s", filename);
X            if (pos = strchr (buf, '\n'))
X                *pos = '\0';
X            if (strlen (buf))
X                if ((buf [0] == '<') && (buf [strlen (buf) - 1] == '>'))
X                    process_id (buf);
X            }
X        }
X
X    (void) fclose (timefp);
X    return (0);
X    }
X
X
X/*
X * write_tree - Traverse the tree writing out ids of articles that were
X * not successfully retrieved.
X */
X
X    static void
Xwrite_tree (struct mnode *p, FILE *timefp)
X    {
X    if (p != NULL)
X        {
X        write_tree (p->left, timefp);
X        if (!p->used)
X            {
X            (void) fprintf (timefp, "%s\n", p->msgid);
X            if (ferror (timefp))
X                log_sys ("write_tree: Error writing ids");
X            }
X        write_tree (p->right, timefp);
X        }
X    }
X
X
X/*
X * set_ntime - Set the start time for the next NEWNEWS for system
X */
X
X    void
Xset_ntime ()
X    {
X    FILE *timefp;
X    char filename [PATH_MAX];
X    char backup [PATH_MAX];
X
X    /* Copy the file to a backup */
X    (void) strcpy (filename, TIMFILE);
X    (void) strcat (filename, hostname);
X    if (sublist)
X        {
X        (void) strcat (filename, ".");
X        (void) strcat (filename, sublist);
X        }
X    (void) strcpy (backup, filename);
X    (void) strcat (backup, ".o");
X    if (rename (filename, backup))
X        if (errno != ENOENT)
X            {
X            log_ret ("set_ntime: Error renaming %s to %s", filename, backup);
X            exit (1);
X            }
X
X    /* Open new file */
X    if ((timefp = fopen (filename, "w")) == NULL)
X        {
X        log_ret ("get_ntime: Error opening %s", filename);
X        exit (1);
X        }
X
X    /* Write the new time for current host */
X    (void) fprintf (timefp, "%06ld %06ld\n", newdate, newtime);
X    if (ferror (timefp))
X        {
X        log_ret ("set_ntime: Error writing %s", filename);
X        exit (1);
X        }
X
X    /* Write out any message ids not read in */
X    write_tree (root, timefp);
X
X    (void) fclose (timefp);
X    }
X
X
X/*
X * do_authinfo - Check in the authinfo username and password with the
X * server.
X */
X
X    static void
Xdo_authinfo ()
X    {
X    char buf [NNTP_STRLEN];
X
X    /* Send the username to the server */
X    (void) sprintf (buf, "AUTHINFO USER %s", ai_username);
X    if (debug_flag)
X        (void) fprintf (stderr, "<- %s\n", buf);
X    put_server (buf);
X
X    /* Get the response and check it's okay */
X    get_server (buf, sizeof (buf));
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s\n", buf);
X    if (atoi (buf) != NEED_AUTHDATA)
X        {
X        log_msg ("do_authinfo: NNTP protocol error: got '%s'", buf);
X        exit (4);
X        }
X
X    /* Send the password to the server */
X    (void) sprintf (buf, "AUTHINFO PASS %s", ai_password);
X    if (debug_flag)
X        (void) fprintf (stderr, "<- %s\n", buf);
X    put_server (buf);
X
X    /* Get the response and check it's okay */
X    get_server (buf, sizeof (buf));
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s\n", buf);
X    if (atoi (buf) != OK_AUTH)
X        {
X        log_msg ("do_authinfo: NNTP protocol error: got '%s'", buf);
X        exit (4);
X        }
X    }
X
X
X/*
X * do_mode_reader - Send mode reader command to INN to switch to nnrpd
X * so we can do a NEWNEWS.
X */
X
X    static void
Xdo_mode_reader ()
X    {
X    char buf [NNTP_STRLEN];
X
X    /* Send the command to the server */
X    if (debug_flag)
X        (void) fprintf (stderr, "<- MODE reader\n");
X    put_server ("MODE READER");
X
X    /* Get the response and check it's okay */
X    get_server (buf, sizeof (buf));
X    if (debug_flag)
X        (void) fprintf (stderr, "-> %s\n", buf);
X    switch (atoi (buf))
X        {
X        case OK_CANPOST :
X        case OK_NOPOST :
X            break;
X        default :
X            log_msg ("do_authinfo: NNTP protocol error: got '%s'", buf);
X            exit (4);
X        }
X    }
X
X
X/*
X * MAIN PROCEDURE
X */
X
X    int
Xmain (int argc, char **argv)
X    {
X    int ret;
X    time_t clock, starttime, endtime;
X    struct tm *now;
X
X    /* Set the name of the program and parse the args */
X    pname = (pname = (char *) strrchr (argv [0], '/')) ? pname + 1 : argv [0];
X    if (parse_args (argc, argv))
X        {
X        (void) fprintf (stderr, "Usage: %s [-g newsgroups/distribution] [-t time] [-a username/password]\n", pname);
X        (void) fprintf (stderr, "       [-d] [-i] [-l] [-r] [-w] server[/sublist]\n");
X        exit (2);
X        }
X
X    /* Open syslog if required with appropriate BSD 4.2/4.3 call */
X#ifdef SYSLOG
X#ifdef LOG_AUTH
X    openlog(pname, LOG_PID, SYSLOG);
X#else
X    openlog(pname, LOG_PID);
X#endif
X#endif
X
X    /* If groups not supplied in args, then get from slurp.sys file */
X    if (nn_newsgroups == NULL)
X        if (read_sys ())
X            exit (2);
X
X    /* If start time not supplied in args, then get from slurp.tim file */
X    if (nn_time == NULL)
X        if (get_ntime ())
X            exit (2);
X
X    if (debug_flag)
X        {
X        (void) fprintf (stderr, "server: %s\n", hostname);
X        (void) fprintf (stderr, "time: %s\n", nn_time);
X        (void) fprintf (stderr, "newsgroups: '%s'\n", nn_newsgroups);
X        (void) fprintf (stderr, "distributions: '%s'\n", nn_distributions);
X        }
X
X    /* Unless don't write flag set, get time for next NEWNEWS */
X    if (!no_time_flag)
X        {
X        if (local_time_flag)
X            clock = time ((time_t *) 0);
X        else
X            if ((clock = server_time (hostname)) == 0)
X                exit (3);
X
X        now = gmtime (&clock);
X        newdate = (now->tm_year * 10000) +
X                 ((now->tm_mon + 1) * 100) +
X                   now->tm_mday;
X        newtime = (now->tm_hour * 10000) +
X                  (now->tm_min * 100) +
X                   now->tm_sec;
X        }
X
X    /* Open the history file */
X    if (open_history ())
X        log_sys ("Can't open history file %s", HISTORY_FILE);
X
X#ifndef RNEWS
X    /* Change to the incoming batch directory */
X    if (chdir (INDIR))
X        log_sys ("Can't change directory to %s", INDIR);
X#endif
X
X    /* Set up the connection to the server */
X    switch (ret = server_init (hostname))
X        {
X        case -1 :
X            exit (3);
X        case OK_CANPOST :
X        case OK_NOPOST :
X            break;
X        default :
X            log_msg ("Can't talk to %s: got response code %d", hostname, ret);
X            exit (4);
X        }
X
X    /* If authinfo details supplied, then use 'em */
X    if (ai_username)
X        do_authinfo ();
X
X    /* Switch INN to nnrpd instead of innd if needed */
X    if (mode_reader_flag)
X        do_mode_reader ();
X
X    /* Get a list of the new articles */
X    get_ids ();
X
X    /* Now get the actual articles */
X    starttime = time ((time_t *) 0);
X    if (entries > 0)
X        get_articles ();
X    endtime = time ((time_t *) 0);
X
X    /* Time to say goodbye */
X    close_server ();
X    close_history ();
X
X    /* Submit the remaining batch, if present */
X    enqueue_batch ();
X
X    /* do we want to update the timestamp file? */
X    if (!no_time_flag)
X        set_ntime ();
X
X#ifdef SYSLOG
X    if (!debug_flag)
X        syslog (LOG_INFO,"Processed %d new, %d duplicate, %d missing articles",
X                newart, dupart, misart);
X    else
X#endif
X        (void) fprintf (stderr, "Processed %d new, %d duplicate, %d missing articles\n",
X                        newart, dupart, misart);
X
X#ifdef SPEEDSTATS
X  #ifdef SYSLOG
X    if (!debug_flag)
X        syslog (LOG_INFO, "Average transfer speed %ld cps",
X                totalsize / (starttime == endtime ? 1 : endtime - starttime));
X    else
X  #endif
X        (void) fprintf (stderr, "Average transfer speed %ld cps\n",
X                totalsize / (starttime == endtime ? 1 : endtime - starttime));
X#endif
X
X    exit (0);
X    }
X
X/* END-OF-FILE */
END_OF_FILE
  if test 17145 -ne `wc -c <'slurp.c'`; then
    echo shar: \"'slurp.c'\" unpacked with wrong size!
  fi
  # end of 'slurp.c'
fi
echo shar: End of archive 1 \(of 2\).
cp /dev/null ark1isdone
MISSING=""
for I in 1 2 ; do
    if test ! -f ark${I}isdone ; then
    MISSING="${MISSING} ${I}"
    fi
done
if test "${MISSING}" = "" ; then
    echo You have unpacked both archives.
    rm -f ark[1-9]isdone
else
    echo You still must unpack the following archives:
    echo "        " ${MISSING}
fi
exit 0
exit 0 # Just in case...