Club Amiga de Montreal - CAM/CAM_CD_1.iso/files/441.lha/IP_Device/IP_Device.DOC

Text File | 1990-11-13 | 21.5 KB | 453 lines

                            The "IP:" Pipe Device
                            _____________________

                             -- Pete Goodeve --
                                August 1990

Overview
________

"IP:" is a handler that may be mounted in your system like any other
AmigaDOS device, to provide you with both standard "pipe" type facilities
and a number of features not available elsewhere.  In particular, as a
pipe, but unlike other pipes available for the Amiga, it has "unbuffered
pass-through"; in other words when a writer process sends a packet it
is immediately available to the reader.  (The manual description of the
1.3 pipe claims that it also does this, but in fact it does not -- the
reader gets no data until either the buffer is full or the writer closes
its end of the pipe.)  This feature makes it possible for processes to
actually interact via these pipes: sending a command line, for example.
will get an immediate response from the recipient.

In addition, an IP channel need not close when the program at either
end goes away.  It will remain open as long as there are ANY processes
(readers or writers) referencing it.  Normally it will send an end-of-file
to the reader (usually causing it to exit) when the last writer closes
its reference to the channel, but it is easy to inhibit this so that the
reader will remain active and waiting for input, whether or not there are
any writers, until a specific QUIT request is sent (see below).

Further, although only one reader may be attached at a time, you can have
multiple writer processes sending output to a single IP channel at the
same time -- a "Funnel", rather than a Pipe.

There is a special convention for opening both an input AND an output
channel with the same specification, so that you can for example attach
a Shell process to piped input and output instead of a window.  As IP
pipes don't buffer, this becomes a workable arrangement.

The IP device is actually based on the "PPIPC" protocol, and the "channel
names" are actually IPCPorts, so it provides a general mechanism for
programs that use this scheme to send text or other unstructured data to
other programs that don't know about PPIPC.  It also means that control
information may be sent (via IPCMessage) to an IP channel; the only
such that is publicly available at this time is the 'QUIT' message
mentioned above, but other capabilities can be provided when appropriate
programs are available.  (A separate program to send the QUIT is provided.)

Because all data is passed via publicly named IPCPorts, you can easily
interpose programs to extend performance in various ways.  Possibilities
(not actually written yet!) include things like a "tee", to make data
simultaneously available amongst several processes, and a "tank" that
would dynamically buffer data to decouple writer and reader more
completely (allowing a certain number of unread packets to accumulate
before blocking the writer).


Requirements and Installation
_____________________________

The IP device needs a number of elements to be set up for its correct
operation.  Most fundamentally it requires 'ppipc.library' to be present
in LIBS:, and of course it needs the handler code itself ('IP-Handler')
in the L: directory.  To allow it to be mounted, it will need a suitable
entry in DEVS:MountList, and for permanent installation you'll also want
a "Mount IP:" command in your Startup-Sequence.  For convenience, you will
probably want "IPCQuit" in C:.

This can all be done for you with minimum fuss -- but with your permission
at each step -- by the installation icon "Install_IP_Device".  This runs a
script that goes through all the above steps, keeping you informed and
asking permission at each stage.  Once you have installed the library, you
can start up the device without actually installing anything else, if you
like, by answering "NO" to the query regarding installation of the handler,
and then "YES" to the question about a temporary trial.

If you prefer to do it yourself by hand, "Install_IP_Device" is actually a
directory (only accessible through the Shell or CLI) containing all the
necessary pieces.  (It is completely self-contained, by the way; you can
copy the whole system to any desired location simply by dragging the icon.)

You don't really need to know anything about PPIPC to use this device,
but if you want full details (and I definitely encourage this!) the
complete package is on Fish Disk 290.

The code itself should run under any WorkBench from about 1.2 on (including
2.0 -- it has been tested there without encountering any problems).  The
installation script uses a couple of command options not available under
1.2 (and has not been tested there) so it may not work completely, but
probably should go most of the way; it is perfectly happy under 2.0.


Use as a standard Pipe
______________________

Once the IP device is installed and mounted as above, you use it just
like other AmigaDOS pipes.  Each channel (that is simultaneously open)
must have its own name, specified in the usual Amiga device:name format,
for example 'IP:xyz' or 'ip:MyChan'.  (These are not true "filenames"
of course.  In particular, there is no "directory structure"; '/'
separators have a different meaning -- see later.)  As distinct from
most AmigaDOS names, the name part is case-sensitive: 'xyz' and 'XYZ'
are different channels.  This is because the channels are actually
IPCPorts, where case is important.

To pass data between two processes, you simply open a particular channel
for writing by one and reading by the other.  Thus as a trivial example:

                RUN LIST >IP:xyz
                TYPE IP:xyz

or equivalently:

                RUN TYPE IP:xyz
                LIST >IP:xyz

Either reader or writer may be started first; it will block on its first
I/O request until the other end is ready.  (It is important that at least
the first command be invoked by RUN -- or of course from a separate CLI
window if you prefer -- because it will not continue until the connection
is made.)

Once made, the connection will remain open until the writer closes it,
at which point an end-of-file is sent to the reader, which can in turn
close its end.  This is the conventional sequence, but it may differ at
the option of either program.  If the reader chooses not to exit when
sent an EOF, the channel will remain open for further input from other
processes (further details in the next section).

On the other hand, the reader might terminate on some signal other than
EOF, in which case another reader could attach itself to the channel to
process further output from the writer.  Note, however, that any output
sent to a IP channel which has no current reader -- except when a
connection is being first established -- is DISCARDED; a write error is
actually also returned to the sender, but the majority of programs
unfortunately ignore this.


Use as a "Funnel"
________________

As mentioned you can have as many processes writing to an IP channel
as you want.  These can be either running in parallel or concatenated
sequentially, as appropriate; in the latter case you must inhibit automatic
closing of the channel as described below.  Data is passed by packet as
written by each sender (usually a line at a time, though some programs --
such as COPY -- may use packets of a different size);  if more than one
process is writing at once, packets from each will be interleaved in the
order that they arrive.

Writer processes may attach and detach themselves on a channel as they
require, without disturbing the reader, until the last writer closes,
at which time the reader gets an end-of-file, causing it too to close
(usually).  An EOF, by the way, is -- as normal in AmigaDOS -- an empty
data packet.

If you wish to keep the channel open on a semi-permanent basis, you can
suppress this automatic EOF.  To do so, you prefix the channel name that
you open for reading by "!":

                TYPE IP:!xyz

The exclamation mark is NOT a part of the channel name: writers still
normally access it as "IP:xyz"  (for writing, the mark has a different
meaning: "exclusive write" -- see below).

If you do open a reader in this mode, when you eventually want to close it
you will have to do so by some other method: either terminate it directly
somehow -- by some recognized command, say -- or explicitly cause an EOF by
sending the channel a QUIT message.  Using the IPCQuit command (a separate
program provided with the package), you specify the name of the channel(s)
you wish to terminate, WITHOUT the "IP:" prefix:

                IPCQUIT xyz



Exclusive Write Mode
____________________

You can prevent a writer process that intends to open a particular channel
from actually opening it if it already exists by specifying "exclusive
write".  The exclamation mark is again used to specify this:

                LIST >IP:!xyz

If ANY process (reader or writer) currently has this channel open, the
request will be rejected.  Note that this mode only affects the specific
request in which it is used.  Any later request that does not include
the "!" will connect normally.

A caution:  the channel is actually opened momentarily to make the check;
if there happens to be just a single reader (in non-funnel mode) on the
channel at that time, it will get an EOF as a result.  This scheme should
only be used where the write end is always opened first.

This mode will probably be of use mainly in things like ARexx scripts
that may have several simultaneous multitasking incarnations.  The script
can loop through a sequence of generated channel names until it successfully
opens one, which it then knows is unique and safe to use.


Bidirectional Data
__________________

There is an option for IP channel name specification by which you may
specify both input and output channel at the same time.  This is ONLY
useful for programs that normally expect a "CON:" window specification at
that point -- in other words it is really only useful for NEWCLI or
NEWSHELL.

The convention for this is to specify the input and output channel names
-- which must of course be different -- separated by a slash ("/")
character:

                NEWSHELL IP:inchan/outchan

The exclamation mark modifier may be used on either part, but it is
superfluous on the input side because the Shell and CLI ignore EOFs
on that channel.  In fact it is probably undesirable since it would
prevent programs running under the shell from receiving an EOF either!
"Exclusive Write" could well be useful on the output side, though, as
NEWSHELL will fail (with return code 20) if the connection fails, so
whatever script has invoked it could detect the condition.


Clogged Pipes
_____________

A word or two may be useful on what to do if a program at one end or other
of your pipe fails to behave as expected, and leaves the other end hanging.
In general all you'll need to restore normality is an available Shell/CLI
window from which to give a command or two.

If you start a reader successfully, but the writer process never gets
going, the reader will just keep waiting.  If you just made some typing
error in the writer command line, you can just try again with a corrected
line.  If you can't do this, and as long as you didn't open it as a
maintained "funnel" (with "!"), all it needs is some process to first open
a writer then close it, to send an EOF and let it terminate. A simple way
would be:

                ECHO >IP:xyz ""

More convenient still is to send a QUIT message, and this also works
for funnels as explained above:

                IPCQUIT xyz

If it's the reader side that fails, the writer will be the one to be
left hanging in the air, waiting for somewhere to send its output.  The
best way to clear this is to start some other process to take the output
from that channel.  You could either type it to the screen, or if it is
voluminous or non-text, send it off to NIL:

                TYPE IP:xyz to NIL:

Then of course there's the situation where you mistyped the channel name
in one of the two commands, so you have TWO processes going nowhere and
talking to no one.  The most direct solution is to clear one or both
processes as described above, but if you're really desperate -- or just
want to play -- you could simply start a THIRD process, to form a bridge
between the two loose ends, with a command like the following!

                TYPE IP:xyz to IP:xzz

Use TYPE rather than COPY, by the way, because the latter does some odd
things when it opens a destination -- checking for existence and so on --
with multiple opens and closes that will result in your destination channel
getting shut before it ever gets started.  For this reason you should also,
if you ever want to COPY to an IP channel, either start the COPY command
FIRST or use maintained funnel mode on the receiver.

One other problem that might crop up if you have two programs communicating
with each other by a pair of IP pipes is circular lock-up.  If one of the
two reads an input line from one pipe, processes it, and puts some output
on the other, it will then hang until that data is read by the other program
-- the one that is sending IT input.  Each program therefore must be sure
to read ALL the current output of the other at each cycle before attempting
to send any of its own, or the other end will never reach to point of
reading it, and there will be a lock up.  If this does occur, judicious
application of IPCQuit should break the lock.


Packet Management
_________________

As this is a DOS device, data is passed into, and taken out of, an IP
channel as standard DOS packets.  The (maximum) amount of data a packet can
contain is determined by the program doing the I/O.

When a packet is written to an IP channel, the whole packet contents is
always passed (as an IPCMessage) to the IPCPort acting as the rendezvous.
There is no copying at this stage -- just a pointer is passed; the sender
is suspended until the data has been accepted by the reader.

If the packet supplied by the reader to receive the data is large enough
to take it all, the data is copied to the receiving packet and the sender's
packet is released.  If the receiving packet is too small, as much data
as possible will be copied, and the sender will remain suspended until
further reads have dealt with all its data.

Symmetrically, of course, if a reader sends a packet (makes a read request)
to the IP device when there is no packet from a writer waiting, it will
be suspended until one arrives.


PPIPC Specifications
____________________

It is a little ironic that while one of the specific advantages claimed for
PPIPC is its concern for the structure of data, this application essentially
passes unformatted byte streams.  However this is of course dictated by
the expectations of the programs that will connect to the IP device.
The application does make good use of the "safe connection" aspect of
PPIPC, though.  This section describes the format of IPCMessages sent
and recognized by the IP device.  Knowledge of the PPIPC protocol will
be needed -- see Fish Disk 290.


IPCPorts:               Each IP device channel uses an IPCPort to pass
                        its data through. The name of a channel's port is
    the string following the "IP:" device identifier prefix (and remember
    case is significant).  Any other PPIPC-capable device can communicate
    with this port, without having to know that it is an IP channel
    (provided that its messages have suitable IDs).

    A request to open an IP channel for reading will attach that IP
    process to the port as a Server (therefore there may be only one
    reader at a time).  A request to write will attach it as a Client:
    the first connection attempt is made with LoadIPCPort(), so that if
    it is not already served and there is a Broker running, the appropriate
    startup action can be taken; if this fails, the connection is simply
    made with GetIPCPort(), BUT -- if there is no server -- the 'IPP_LOADING'
    flag is also set so that the first message sent will not bounce.  This
    assumes that eventually a server WILL arrive!

    When an AmigaDOS 'close file' request is sent, a writer will simply
    drop its port and quit.  The reader first 'shuts' the port, ensures
    that any unprocessed messages are returned with 'IPC_REJECT' set,
    and then quits.  The Server will detect the disappearance of its
    last Client and -- unless opened as a "funnel" -- will automatically
    send an EOF (null packet) to its reader process.


IPCMessages Sent:       An IP output connection itself will send only
                        one kind of message.  Its ipc_Id field is 'DATA'
    and it has one IPCItem whose ii_Id field is also 'DATA'.  ii_Size is
    set to the number of bytes in the packet received from the writer, and
    ii_Ptr points to that packet data buffer.  No assumptions are made as
    to the nature of the data block.  The message must of course be replied
    when processed by the reader.


IPCMessages Received:   The reader (Server) end can handle several other
                        message formats besides the one above.  It expects
    ipc_Id to be one of 'DATA', 'TEXT', 'CMND', or 'QUIT'.  (A fifth ID
    'SETP' is also recognized.  It is used to set up some other parameters
    that are used by an as yet unreleased application.  It will not be
    described further at the moment.)

    All messages are processed in the order that they arrive.  The first
    three message IDs are treated identically.  'QUIT' causes an EOF
    (null packet) to be sent to the reader.  Any data message should
    contain an item with one of three ii_Id's: 'DATA', 'TEXT', or 'LINE'.
    In the current implementation only the first item with one of these IDs
    will be found in a message -- any preceding irrelevant items, and
    anything following the first found, will be ignored.

    The contents of a 'DATA' item is passed to the reader's AmigaDOS packet
    exactly as received (broken into suitable pieces if it is too big).
    Its size is taken from the ii_Size field of the item.

    A 'TEXT' item is treated similarly, except that it is assumed to be
    a null-terminated (ASCII) string, and its size is calculated up to
    the first null byte.  (It is not allowed to exceed ii_Size, however.)
    The null byte is NOT copied to the DOS packet!  (Text length is
    indicated by the return value in the packet.)

    A 'LINE' is a null terminated ASCII line of text WITHOUT the
    terminating newline.  The missing newline is added to the packet
    before passing it on.

    Normally the Server will reply all messages that it receives, but
    a Client (this won't be an IP process naturally) may also send one that
    is not to be replied, IF it ensures that the following is true: a) the
    data being passed is in the user data area of the message itself, b)
    the ReplyPort field is NULL.  (For completeness, 'IPC_TRANSFER' should
    be set in ipc_Flags, but this is not currently checked.)  If an IP Server
    does receive a message with no replyport, it ASSUMES that it can safely
    delete it, and does so without further fuss.

                            + + + + +

Coda
____

I suspect many people's reaction to the IP Device may be "Cute... but so
what?".  Its utility may not be immediately obvious, but think about it
and you may find that it's exactly what you need.  It is just another
building block that in certain places will turn out to be a perfect fit.

Most straightforwardly you should be able to replace all use of the 1.3
pipes (provided that you don't rely on buffering to pass data from one
program to another under the same CLI).  I have been frustrated by the
erratic behaviour of the 1.3 pipes -- they will fail to open for no
apparent reason, or irretrievably lock up.  Any "lock-up" of the IP
device should be easily clearable (see "Clogged Pipes", above), and as
far as I can tell it behaves completely reliably.

In addition you get the non-buffered immediate response, and the ability to
concatenate output from more than one program, which are not available
from other pipes.  I haven't included any examples or demos here, because
although the IP dvice is now central to my environment, the scripts I
use are also tailored too much to my needs to be much use to others.
And, quite frankly, most of the appplications I envision for it I haven't
even tried yet.

One such use is as a link in a "SuperShell".  You could have a Rexx script
(or Tcl or something -- let's not restrict ourselves) that keeps an IP
channel open to a Shell process.  Commands would be examined by the script
and processed in some really intelligent fashion before being passed to the
Shell for execution.  (ARexx can of course send command strings to AmigaDOS,
but a new process is opened for each one, so you can't do something as
simple as change a directory.)

The "tee" and "tank" modules mentioned in the overview will add to the
capabilities when they arrive.  In the meantime, see what you can do
with the current ones.

If you have any comments, requests or suggestions, you can contact me
at the address below.

                            + + + + +

The IP Device and this manual are Copyright 1990 by Peter Goodeve.
There is no restriction on their non-commercial use or copying.
If you wish to incorporate the IP device in a commercial product or
distribution, please contact the author.  (By 'commercial', I mean
a distribution for which more than a nominal materials and copying
fee is charged.)

                            + + + + +

                                            Pete Goodeve
                                            3012 Deakin Street #D
                                            Berkeley, Calif. 94705
                                            U. S. A.

                                            net: [e-mail]