NetNews Usenet Archive 1992 #31/NN_1992_31.iso/spool/comp/unix/question/15128

Internet Message Format | 1993-01-03 | 3.8 KB

Path: sparky!uunet!pipex!bnr.co.uk!uknet!gdt!ccsis
From: [e-mail] (Icarus Sparry)
Newsgroups: comp.unix.questions
Subject: Re: two shellscripts
Message-ID: <[e-mail]>
Date: 3 Jan 93 17:39:41 GMT
References: <[e-mail]>
Organization: Bath University Computing Services, UK
Lines: 81

In the referenced article, [e-mail] writes:
>Both start with "netstat" doing an output once every hour.
>(Actually what it does is pump out two headers and a cumulative line, followed
>by the first of the hourly stats.  Then it sleeps for an hour before doing
>another.  I have to stop it with a "Ctrl-C".)

script 1 omitted but is netstat | egrep | awk
script 2 omitted, but is netstat | awk

>The first script does not work as I wanted; the second does.  The question is
>why.  By the way a modified version of the first script, leaving out the "awk"
>does work.

Let us call this modified version script 3.

>I suspected that it is because of the way this version of "netstat" works,
>ie, it doesn't flush its output, but what I can't figure out is what this has
>to do with how many pipes you have (as it appears here in this example).
>Can you explain this one to me?

Yes. You are partially correct. The problem is actually that 'egrep'
does not flush its output if the output is not going to a terminal.
This is the common behaviour of programs which use the 'stdio' buffered
I/O package, i.e. most Unix programs.

When a program first attempts to write to stdout, it attempts to decide
if the output is going to a terminal, or not. If it is, then it sets an
internal flag to say 'each time I see a newline character, then I
should flush my output  buffer'. It always flushes its buffer when the
buffer is full.

Now netstat 'knows' that it often produces information at long
intervals, so it sets its own buffering to 'each time I see a
newline...' always, but egrep does not.

So in script 1, 'netstat' sends its results to 'egrep' a line at a
time. 'egrep' sees that its output is to a pipe, so 'egrep' only sends
the results of its search to 'awk' every time the buffer is full. 'awk'
then sends them to the terminal a line at a time.

In script 2, 'netstat' sends its results to 'awk' a line at a time, and
'awk' sends them to the terminal a line at a time.

In script 3, 'netstat' sends its results to 'egrep' a line at a time.
'egrep' sees that its output is going to a terminal, so it sends its
output a line at a time.

You would have seen the same problem if you had redirect the output one
of the scripts to a file, and then used 'tail -f' to look at the file.

The 'fix' is to make 'egrep' send its output a line at a time. You
could change the source for 'egrep' (you can get the GNU source if you
don't already have a set of sources to hand), but this is not a general
solution.  It works in this case, but then you have the problem of what
to do if you want to feed the output into say 'tee' to keep a log file
i.e

netstat | egrep | awk | tee

Then 'awk' will no longer think it is sending its output to a terminal,
so buffers it, so you have to modify the source of 'awk' and so things
go on.

There is help at hand however. Dan Bernstein wrote a program called
'pty', which was posted to comp.sources.unix in volume 23. With this,
you are able to make a program believe that it is sending its output to
a terminal.  In order to do this, it uses the 'pty' devices from BSD
Unix. Therefore if you change your script into

netstat | pty egrep | awk

(with suitable options to pty), and it will work in the manner in which
you expect. If you later want to tee the output, then

netstat | pty egrep | pty awk | tee

will also work.

Moral, get 'pty' now if you have a unix with BSD pty devices. Available
from all major archive sites, e.g.
ftp.uu.net:usenet/comp.sources.unix/volume23/pty/part0[1-6].Z