Programmer 7500/MAX_PROGRAMMERS.iso/PASCAL/GIBSONTS.ZIP/TSR.TXT

Text File | 1987-07-15 | 18.5 KB | 362 lines

                                                     July 1987

                      The History and Technology
              of TSR Terminate and Stay Resident software

                                  by
                             Steve Gibson
                   InfoWorld TechTalk Columnist and
                  President of Gibson Research Corp.


                            HISTORY BEGINS
----------------------------------------------------------------------
Once upon a time a Big Blue company named IBM decided that it wanted
to make personal computers too.  For reasons known only to the Gods of
Boca Raton Florida, IBM decided to use the 8088 microcomputer from
Intel rather than Motorola's vastly "cleaner" 68000 (as is used in
Apple's macintosh).  Viewed in retrospect, this has turned out to be a
ridiculously expensive, i.e. WRONG, decision.  But anyway... IBM
journeyed to the West to get its software for this new machine.

IBM first visited the company who was then the acknowledged king of
operating systems, Digital Research, Inc.  DRI had developed CP/M, the
defacto standard operating system for the very popular 8080 and Z80
microcomputer systems.  IBM explained that it needed an operating
system for the new 8088 microprocessor in their new, blue, personal
computer and asked if DRI would please provide them with one.  DRI
said, quite truthfully, that they didn't have one.  Big blue must have
then asked if they would like to make one, to which DRI must have
again said: "No thanks."

IBM shrugged and headed North. . . . .

Though IBM may have only planned to ask Microsoft for an 8088 BASIC
language interpreter for their new machine, they found themselves in
Washington without an operating system.  Microsoft said they would be
pleased to provide IBM with both an 8088 operating system and the
BASIC language interpreter.  IBM said "Thank you" and said they needed
it very soon please.

But Microsoft didn't have an 8088 operating system either, and being
so very much smaller then than now, didn't have the resources to build
one from scratch by IBM's deadline.  So they found an even tinier
company called Seattle Computer who had built a little unpretentious
operating system for the 8088 and 8086 micros called 86-DOS.  It was
also known by some as QDOS.  QDOS was said to stand for Quick and
Dirty Operating System.

So Microsoft, then able to meet IBM's 8088 machine timetable,
delivered both an operating system (albeit one of unassuming parentage
and questionable virtue) and a BASIC interpreter.

Everything seemed fine until we all began really using the new
operating system.  We wanted additional and rather reasonable things
from MS-DOS and PC-DOS.  Things which had not been designed into 86-
DOS, like printing in the background while the user was doing other
things.  Or supporting RS-232 serial printers, which required routing
the printer data out through a serial port instead of the normal
parallel printer port.

To meet these reasonable demands Microsoft made a series of small
changes and additions to the guts of MS-DOS which allowed extra "add-
on" code to be "hung" onto the outside of MS-DOS and also provided for
limited communication between DOS and the add-on code.  Microsoft's
idea was that these MS-DOS limitations would be "fixed" by writing
some new DOS commands (in the above examples PRINT and MODE) which
would make use of special undocumented aspects of their latest version
operating system.

The problem was that MS/PC-DOS, still really old 86-DOS (QDOS) in
disguise, was never built to support multitasking.  This means that it
was never meant to serve more than a single master at a time.  A
command like PRINT, which can continue to operate even AFTER the user
has begun doing something else, could very easily confuse it, since
now there could be service requests coming into MS-DOS from two quite
different places at once.

Since this began making the entire operating system rather fragile and
weird, Microsoft's grand plan was to retain all these special hooks
and communications paths as a company secret.  They figured that as
long as they were the only ones to use these new MS-DOS secrets things
wouldn't get out of hand.

But Microsoft must have gravely underestimated the cleverness and
determination of the development community.  When the MODE command
proudly stated: "Resident portion installed" developer's eyebrows shot
up!  RESIDENT portion?  Hmmmmmm.  And if the PRINT command could
access the disk for file printing when we were messing around with a
spreadsheet, why couldn't we build a little resident notepad or two,
too?

But EVEN after these new resident goodies began appearing in
increasing numbers, Microsoft refused to "disclose" anything
whatsoever.  They said they "couldn't" document these things or they'd
be forced to make all FUTURE versions of the operating system support
them in upwardly compatible fashion.  Since these were awkward
"kludge" solutions to the real need for a true multitasking operating
system, Microsoft's delicate condition can be readily understood.

However, developers sensing massive market opportunities were not to
be stopped.  Debugging software which allowed programmers to peer
inside the still working machine sold like wildfire.  Using these
debuggers (like PERISCOPE, my personal favorite), developers probed
into the very heart of Microsoft's own PRINT and MODE commands,
unscrambling and deciphering the undocumented secrets of their
operation.  Using the same secrets Microsoft's own developers had
used, they learned how to give their OWN programs similar
capabilities.

In an old episode of Star Trek, Mr. Spock said, after capturing the
Romulan's cloaking device, "...military secrets are the most fleeting
of all, Commander."  This applies equally well to technological
secrets in general, and as we've seen, to Resident Software in
particular!

It's unfortunate that Microsoft did not have time back then in the
beginning to build their own DOS from scratch.  They might have done
it right.  But we're stuck with what we have today, and making the
most of it is certainly our best strategy.

It remains unfortunate, though certainly understandable, that
Microsoft isn't willing to disclose the details of MS-DOS's resident
software hooks and techniques.  This policy keeps such knowledge quite
secret, since even the third-party developers are reluctant to
disclose their own hard-won "insider's" knowledge and proprietary
techniques for forcing reliable multitasking behavior from good ol'
single tasking DOS.  It's quite interesting to speculate that perhaps
today even Microsoft does not KNOW how to write MS-DOS multitasking
applications as well and robustly as several of the larger commercial
resident software publishers!  The tricks required to make DOS
multitask are real nasty stomach-turners.

So... what are these tricks and techniques?  How DO resident programs
"pop up over" applications at the simple press of a "hot key"?



                  THE TECHNOLOGY OF RESIDENT SOFTWARE
----------------------------------------------------------------------

Since we've now know how we got to wherever it is we are, let's answer
the those questions posed above....

Two things are required for the functioning of TSR technology:  First,
the TSR program needs to somehow "hang around" well after any other
program has started to run.  Secondly, it has to "know" what's going
on in the system to be able to jump in and help out when called upon
for action.  It also needs some way of re-assuming control of the
machine when the time comes for it to take action.

The requirement of continuing RAM residency is satisfied by a service
provided by the operating system from which TSR's get their name:
Terminate and Stay Resident (TSR).

The main operating RAM memory of an IBM compatible PC is a contiguous
block beginning at address zero (called the bottom of RAM memory) and
extending "upward" for a maximum of 640K bytes (called the top of
memory).  Note that 640K is actually 655,360 bytes since "Ks" in
computerdom are really 1024 bytes, not 1000.  Since the very bottom of
this large contiguous block of RAM is reserved for system housekeeping
purposes, DOS is "booted" by copying it from disk into RAM just
"above" this low-RAM region.

Once DOS has been loaded, all the space "above" it, to the top of RAM,
is available for application program loading and use.  We'll call this
location just above DOS the LOAD POINT for application software.  So,
when a program is started, it is copied from disk into RAM starting at
this LOAD POINT, continuing upward until the entire program has been
loaded.  The memory available, and allocated, to the program for its
own data storage and manipulation extends from the end of itself to
the physical top of the RAM memory.

Most standard programs word processors, spreadsheets, or databases
have "control" of the computer until the user tells the program to
terminate.  After performing whatever cleanup and file saving might be
required, the program simply informs DOS that it is all finished by
saying the equivalent of "I'm all finished now, terminate me please".



Responding to this "Terminate" request from a program, DOS simply
"forgets" all about the program and whatever data it might have had in
RAM.  It displays the command line prompt and awaits another command.
The next program to run is loaded at the same LOAD POINT as before,
thus overwriting whatever terminated program was just running, and
implicitly making the same amount of memory available to this next
program.

Now consider the special case of the RESIDENT PROGRAM...

It begins innocently, just like any other application program, when
its name is typed at the DOS prompt.  DOS copies it from disk into the
user's RAM memory starting at the same LOAD POINT as any prior
application software, implicitly leaving the rest of RAM memory for
the program's own use until done, then hands control of the computer
system over to this program.

Now things start getting interesting!

The resident program will generally say hello to the user announcing
its intention to hang around for the duration.  Then it does a few
"non-standard things" to the system.  These are the very things
MICROSOFT wanted to keep all to itself.  These special "things" keep
the program from being ex-communicated from system activities after it
terminates.  Then, rather than simply saying "terminate me" to DOS, it
says: "I'm xxx bytes long.  Now terminate me but DON'T OVERWRITE ME."
Which is to say: Terminate me, but let me Stay Resident.

Upon receiving this request, DOS takes control from the program, moves
the software LOAD POINT UPWARD by those xxx bytes, and only THEN
returns the DOS prompt to the user.  Since the LOAD POINT has been
moved up by "xxx" bytes, all subsequent programs will be loaded AFTER
the programs which ASKED to remain resident, thus leaving it
undisturbed in the system's memory!

Additionally, since it's the resident program which tells DOS how big
it is, it is able to declare itself to be ANY LENGTH IT CHOOSES!  This
is how programs like SideKick, which are only 39K bytes long while on
the disk, can have a 50K notepad file and take up 89K bytes while in
RAM!  It effectively LIES to DOS about how big it is, so DOS
obligingly leaves it space for itself and for however much working
memory it wants!  This is the way resident programs are able to eat up
so much of our main RAM memory!  When this process is repeated many
time (as is frequently the case) a resident program "stack" is created
in main RAM memory.

So, we've now seen how programs are able to remain in memory after
being "officially" terminated.  The final question is:  How are these
"terminated" programs able to remain "aware" of the activities around
them?... and how are they able to regain control of the computer
whenever they see fit to do so?  In other words, how can a notepad or
other utility "pop up" ??  To answer these last questions let's step
back for a moment, reviewing a bit about how computer's are
controlled:






A computer's actions are directed by its reading of instructions which
are stored in its main memory.  These instructions may be temporary,
as in the case of programs loaded from disk into RAM, or permanent, as
in the case of instructions permanently wired into ROM chips.  In
either case, each instruction has a unique address which is just the
number of the byte it occupies in the machine's memory.

We cause the computer to perform a given task by having it read a
series of instructions which detail the task.  In a sense we CALL upon
those instructions whenever we need to exert their intended function.
Such a "CALLED" instruction sequence generally has a clever
instruction called RETURN which is able to send the computer back to
wherever it was called from!

This notion of FUNCTIONS which are being performed by instruction
sequences called upon in memory, and which will return to us when
finished, is very powerful.  IBM compatible computers come equipped
with a BIOS ROM which contains a host of very useful general purpose
functions which return control to their caller as soon as they have
performed their designated duty.  This is EXACTLY what that often
mentioned BIOS ROM is all about!  B.I.O.S. stands for "Basic Input
Output System", and consists of a collection of standard functions for
reading and writing to the system keyboard, screen, mass storage, and
many other standard devices.

Naturally, we must know the exact address of these "functional"
services in order to CALL them into service.  By convention, the
starting address of every built-in function in the IBM PC is located
in a predefined table of addresses located at the very start of the
computer's RAM memory.  Every function is numbered simply by the
location of its address in this table.  This means that in order to
call for any built-in IBM PC function we only need to know its
standard function number - from this we're able to calculate the
location in the table where that function's starting address can be
found.

There's even a clever computer instruction which lets you directly
call upon a function whose ADDRESS is stored at a specific location in
this table.  So in effect you can simply ask the computer to call
"standard function number 7".  In response to this function request
the computer will look into the 7th slot in the table for the address
of function number 7, and go there directly, returning to you when the
function has been performed.  In computer lingo, a place which
contains the address of something else is a POINTER or a VECTOR.

Even the electrical hardware of the computer is able to call upon
these standard functions.  In fact many of the standard functions in
the ROM BIOS are designed to operate hand-in-hand with the PC's
hardware.  For example, every time a keyboard key is either pressed or
released the standard function number 9, whose address is naturally in
location 9 of the Address Table, is called upon by the hardware.  This
function is called EVEN IF THE SOFTWARE WAS BUSY DOING SOMETHING ELSE
AT THE TIME!

This ability of the hardware to INTERRUPT whatever the software might
be doing at the time is called a "hardware interrupt".  Essentially
the keyboard hardware says to the processor:



"Hey there CPU, hold on a second, remember where you are and what you
are doing, now go perform the function whose address is in slot number
9, and when that function returns to you, go back to whatever is was
you were doing when I bugged you just like nothing ever happened."

This is EXACTLY how we are able to type ahead of our computers.  Even
if our spreadsheet is lost in thought recalculating our tax refund,
we're able to type ahead of it!  Each depression and release of a key
steals a fraction of a moment, a slice of time, away from the
spreadsheet's recalc.  The function invoked by the keyboard's hardware
interrupt places the key into a type-ahead buffer ... which the
spreadsheet will read from as soon as its good and ready ... then
returns control to the recalc as if nothing at all had happened.

Now imagine what would happen if some program (like our resident one)
were to point the POINTER stored in the 9th table location to
somewhere else, such as to ITSELF, prior to issuing the Terminate and
Stay Resident (TSR) request to DOS!

THEN the hardware interrupt occurring at every keyboard action would
turn control over to THIS TSR'd RESIDENT PROGRAM instead of to the
normal built-in keyboard handling function!  If the resident program
wasn't "interested" in that particular keyboard action or keystroke,
it could simply pass control along to the place where the POINTER HAD
BEEN POINTING BEFORE just as if it hadn't been evesdropping at all!
However, if the key action happened to be THIS resident program's
assigned "HOT KEY", it could retain control of the system as long as
it wished, before passing it along as if nothing at all had happened!

That's all there is to it!

As we've seen, the clever application of these various tricks and
techniques can significantly expand the power and flexibility of MS-
DOS and PC-DOS based personal computers.

There are those who feel that resident software is a makeshift
temporary solution which is not long for this world.  They say that
TSR programs will be eliminated by the presence of full multitasking
operating systems.  This author could not disagree more.  I believe
TSR programs will be here for the duration as necessary and viable
add-ons to otherwise incomplete solutions.

So there you have it.  The history and technology of TSR Resident
Software in a nutshell.  I hope this gives you a better feel for
what's going on under the hood of your machine!

Thanks for listening ... and REMEMBER, if you would like to receive
notification of other free information, software, and our future TSR
and utility products, simply pickup the phone or drop us a note or a
card.  We'd be pleased to keep you up to date.

------------------------------------------------------------------------
GIBSON RESEARCH CORPORATION, BOX 6024, IRVINE, CA  92716, [phone]
------------------------------------------------------------------------

Again, Thanks!

Steve Gibson.
                          <<< END OF FILE >>>