\documentstyle[11pt]{article}

\title{M2P, A Modula-2 Preprocessor}
\author{Dennis Brueni}
\date{30 July 1991}

\topmargin 0in
\oddsidemargin 0in
\evensidemargin 0in
\textheight 9in
\textwidth 6.5in

\begin{document}

\maketitle

%====================================================================

\section{Introduction}

This archive contains a preprocessor written in Modula-2 and intended
for Modula-2 programs\footnote{Actually, M2P actually will work with
about any text file}.  The purpose is to provide pre-compile time
control of source code in a portable manner, that is to allow the
same source to be used for several target machines, operating systems,
and/or compilers.  It is also very good for controlling the inclusion
or exclusion of such things as debugging code or nonessential features.

It will strip all comments from a file, except those containing directives,
which are usually marked with a '\$'.  eg.\ (*\$S+*) would not be stripped.
Comments containing directives must be at the first level, otherwise they
will also be stripped from the file.  Nested comments are handled properly.
M2P will also strip excess whitespace from a file, only leaving that which
is needed by the compiler to parse the input correctly.   

M2P tries not to interfere with the line spacing of the file, so that
error messages flagged by the compiler on the preprocessed file
correspond correctly to their location in the original file.  However, since
comments and whitespace are stripped, the horizontal position may be 
wrong for those compilers that do indicate the exact line position.

All preprocessor commands are prefixed by '@'.  This is chosen to aviod
conflict with any Oberon or Modula-2 lexicals \footnote{The new Modula-2
standard declares '@' as the 'required dereferencing operator', which 
causes a conflict for those people choosing to use '@' rather than '\^\,'.} 
(unlike the C-Preprocessor
where '\#' may be confused with not equal).  Preprocessor commands may be
given in free format (unlike most C-preprocessors where the '\#' must be
in the first column.)

%====================================================================

\section{Preprocessor Commands}

The following statements are recognized by the preprocessor.

\subsection{@DEFINE {\em identifier}}

        This defines an identifier with the value TRUE.  An identifier is
        associated with the boolean value FALSE if it is not @DEFINE'd.
        For example:
\begin{quote}\begin{verbatim}        
@DEFINE AMIGA
@DEFINE       THE
@DEFINE           BEST
@DEFINE                COMPUTER  
\end{verbatim}\end{quote}
                
\subsection{@UNDEF {\em identifier} }

        This cancels the most recent definition of the given identifier.
        Normally, this associates FALSE with the identifier, unless the
        identifier had been multiply defined earlier.  
        
\begin{quote}\begin{verbatim}
@UNDEF  MSDOS
@UNDEF  MAC
\end{verbatim}\end{quote}

\subsection{@INCLUDE 'filename' }

        The named file is preprocessed and fed into the output file.
        Caveat Emptor:  This takes care not to ruin line spacing by
        filtering the linefeeds and such out of the include file, so
        the resulting preprocessed file may contain some very long
        lines.  If your compiler cannot handle such long lines, then
        this may create a problem.  It is suggested that INCLUDE files
        only contain preprocessor directives (MACROS and DEFINES) to 
        aviod this problem.

\begin{quote}\begin{verbatim}
@INCLUDE "MACROS"
@INCLUDE "CopyRight"
\end{verbatim}\end{quote}

        The environment variable 'M2PInclude' may be used to set the
        search path for INCLUDE files.  Paths may also be specified
        from the command line by using the INCLUDE switch.
        
\begin{quote}\begin{verbatim}
> setenv M2PInclude "'DH0:M2/Macros/','DH0:M2/OldMacros/'"
> M2P INCLUDE "DH0:M2/NewMacros/" Args.mpp
\end{verbatim}\end{quote}
                
        M2P searchs include paths in the following order:
        
        \begin{enumerate}
            \item  Path(s) specified on the command line
            \item  The current directory
            \item  Path(s) specified in the environment varaiable M2PInclude
        \end{enumerate}
        
        Hence, in the above example, {\tt DH0:M2/NewMacros/} will be searched
        first, then the current directory, then {\tt DH0:M2/Macros/}, and
        finally {\tt DH0:M2/OldMacros/}.

\subsection{@MACRO {\em identifier} [ '(' \{ {\em arglist} \} ')' ] {\em definition} @ENDM}

        A macro definition.  There may be an arbitrary number of arguments
        separated by commas,
        which are used as local macros within the macro definition.  The
        opening (left) parenthesis for the arguments must immediately
        follow {\em identifier}, otherwise it will be treated 
        as part of the macro.
        
\begin{quote}\begin{verbatim}
@MACRO VectorSum(Array1,Array2,LO,HI,DestArray,i)
    @i:=@LO;
    WHILE @i <=@HI DO
       @DestArray[@i]:=@Array1[@i]+@Array2[@i];
       INC(@i);
    END;
@ENDM
\end{verbatim}\end{quote}
                
        Macro definitions may be nested arbitrary levels.  Realize that
        nested macro definitions will be evaluated lazily, and are not
        going to be treated as locals unless you explicitely place an
        @UNDEF at the end of the outer macro to undefine the local bindings.
        I caution against the use of nested macro definitions unless
        you are {\em very} sure about what you are doing.  

\begin{quote}\begin{verbatim}
@MACRO Ugly
   @MACRO LocalToUgly  a surfboard @ENDM
   @MACRO Sentence well known that life is@ENDM
   @MACRO Global        It @ENDM
   @Global is @Sentence sure @Global is @Sentence 
   @LocalToUgly, and that to @LocalToUgly @Global 
   @Sentence @Global.
   @UNDEF LocalToUgly
   @UNDEF Sentence
@ENDM
\end{verbatim}\end{quote}
                
        \begin{description}
        \item[Question:] What would {\tt @Ugly} evalute to?
        
        \item[Answer:]  It is well known that life is sure It is well known that
                life is a surfboard, and that to a surfboard It is well
                known that life is It.
        \end{description}

The symbol table of M2P is a hash table which uses a linked lists for each
bucket.  These linked lists behave somewhat like stacks, so when a symbol
is redefined, the most recent binding is the one always used.  When that
binding is removed, the previous binding will appear on the top of stack
and be used from that point on.  This allows one to define local macros
or flags within macros and then @UNDEF them at then end of the macro.  
However, whenever a flag is redefined, an error message is written to
standard output.

        Finally, it is not illegal to define a macro recursively, but it
        will take an infinite amount of time and space to expand.

\subsection{@ {\em identifier} [ '(' \{ {\em arglist} \} ')' ] }

        A macro instance.  The arguments are bound 
        as local macros and inserted
        into the previous macro definition which is expanded and placed
        in the output file.  The number of arguments used must be less than or
        equal to the number of arguments in the corresponding macro definition.
        An error will be flagged if more arguments are used.  Also, if the 
        argument used in the macro requires expanding, but was not specified,
        an error will be flagged.  Arguments are evaluated lazily 
        and not treated any differently than any other macro.
        {\em Be careful with this! } If
        you have a global defined having the same name as one of
        the arguments, and do not specify that argument, the global
        will be used in its place.  For example:
        
\begin{quote}\begin{verbatim} 
@MACRO c GOTCHA @ENDM
@MACRO a(b,c) (@b,@c) @ENDM

@c                      ==> GOTCHA
@a(one,two)             ==> (one,two)
@a(one)                 ==> (one,GOTCHA)
\end{verbatim}\end{quote}
                
\subsection{@LINE and @SPACE}

        When a macro is recorded, line feeds and carriage returns are
        stripped from the macro, so that when it is later inserted in
        a file, it will not change the line spacing of the file, (ie.
        it is inserted entirely on one line).  However, sometimes this
        may not be desirable because linefeeds in the file may have meaning
        and cannot be stipped (eg. assembly language source).  To solve
        this shortcoming, @LINE can be used to insert a line feed into
        the output file.  and @SPACE can be used to insert a single space
        Of course, the use of @LINE will destroy the line numbers of 
        the output file as it corresponds to the input file.  For example:
        
\begin{quote}\begin{verbatim}
@MACRO Length(Loop,Areg,Dreg,Scratch)
   @IF NOT @Scratch THEN
      @MACRO Scratch D0 @ENDM
   @END
   @SPACE move.w   @Dreg,@Scratch  @LINE
   @Loop  tst.w    (@Areg)+        @LINE
   @SPACE dbeq     @Scratch,@Loop  @LINE
   @SPACE sub.w    @Scratch,@Dreg  @LINE
@ENDM
\end{verbatim}\end{quote}
                
        An instance of this macro will always produce four 
       lines of code, throwing the line count off by four.
                
\subsection{@@}

        A way to get an @ sign into your output file.
        
\subsection{@(, @), and @,}

        Built-in macros for '(', ')', and comma, respectively.
        These above macros are meant to be
        used only in weird situations using macro arguments.  Normally
        it is unnecessary to use them to get ',' '(' or ')'.  The wierd
        situations pop up when you want to place a comma or an unbalanced
        number of parenthesis in a macro argument.  For example:
        
\begin{quote}\begin{verbatim}
@MACRO identity(a) @a @ENDM }

@identity (argument)    ==>  argument
@identity ((((5))))     ==>  (((5)))
@identity ((5@,6@,7))   ==>  (5,6,7)
@identity (@)@,@()      ==>  ),(
\end{verbatim}\end{quote}
                     
\subsection{@IF {\em expression} @THEN {\em statements} 
           \{ @ELSIF {\em expression} @THEN {\em statements} \}
           [ @ELSE {\em statements} ]
           @END}

        The IF-THEN-ELSIF-ELSE-END clause.  The form of this statment is
        identical to the Modula IF-THEN-ELSE-ELSIF-END statements, with
        the exception that the keywords are preceded by a @.  The expression
        is a boolean expression made up of boolean flags 
        (identifiers defined with @DEFINE or @MACRO), 
        AND(\&), OR(|) and, NOT(\~\,).  Either the MODULA or OBERON
        syntaxes are accepted for AND, OR, and NOT.  Precedence is given
        by first parenthesis, then NOT, then AND, then OR, as usual.
        The @ before THEN is optional as it is not absolutely required
        for parsing the statement.

\begin{quote}\begin{verbatim}
@IF    AMIGA @THEN "The Best"
@ELSIF MSDOS @THEN What can I say
@ELSIF MAC   @THEN pity, a real pity.
@END

@IF Debug1 THEN
   @IF Debug2 THEN
      @IF Debug3 THEN
         This is really fun!!!
      @ELSE
         @IF Debug6 THEN
            @IF Debug7 THEN
               @IF Debug8 THEN
                  @IF Debug9 THEN
                     @IF Debug10 THEN
                        We could do this all day!!!
                     @END
                  @END
               @ELSE
                  Couldn't we?
               @END
            @END
         @END
      @END
   @END
@ELSIF Debug2 | Debug3 | Debug4&Debug5 | Debug6 THEN
   Whew, there's no end to the possibilities
@ELSIF Debug7&~Debug8&~Debug9 THEN
   When will this nut stop?
@ELSE
   Here, I suppose.
@END
\end{verbatim}\end{quote}

\subsection{@STRIP and @NOSTRIP}

The {\tt @STRIP} and {\tt @NOSTRIP} directives are used 
to control whether M2P should strip comments
or not, respectively.  Comments not containing directives
within {\tt @MACRO} definition statements are always stripped.

%====================================================================

\section{Using M2P}

If M2P is invoked with no arguments it will give a usage message in the
form of a regular expression giving the format of the command line
parameters.

M2P will accept as many as 128 parameters.
\begin{verbatim}
> M2P T1.mpp(A B C) T2.dpp T3.lpp(DEBUG) T4.app TO test5.a68
\end{verbatim}
This will first invoke M2P on T1.mpp.  The output file will be
automatically translated to T1.mod, and the symbols A, B, and C will
be defined as true.  The symbol table will be cleared, and T2.dpp will
be processed and output will go to T2.def.  T3.lpp will be processed
with DEBUG defined and the output will go to T3.lsp.  Finally, T4.app
will be processed and the output will specifically be sent to test5.a68.
Without the TO clause, the output would have went to T4.asm.

If an input file is given as -- then the standard input will be used.
Likewise if an output file is given as --, the output will go to the
standard output and extraneous messages will be suppressed.  This makes
it possible to use M2P with redirection and concurrent pipes.

The following rules are used for automatic translation of filename extensions:

\begin{center}
\begin{tabular}{|l|l|l|}  
\hline
\hline
{\em Input extension} & {\em Output extension} & {\em Example File Type}\\
\hline
        .mpp    &     .mod            & Modula-2 \\
        .dpp    &     .def            & "    \\
        .cpp    &     .c              & C \\
        .hpp    &     .h              & " \\
        .app    &     .asm            & Assembly \\
        .lpp    &     .lsp            & Lisp \\
        .tpp    &     .txt            & text \\
        .ppp    &     .pas            & pascal \\
        .opp    &     .mod            & Oberon \\
        .fpp    &     .for            & Fortran \\
        .spp    &     .scm            & Scheme \\
\hline
\hline
\end{tabular} 
\end{center}
       
Although I included the .cpp to .c and .hpp to .h translations for C 
source code, M2P will not always work properly with C source code since
certain legal syntaxes in C look like comments in Modula-2.  It is 
recommended to turn off comment stripping (using {\tt @NOSTRIP}) if
using M2P on C source code.

M2P reads the M2PFlags environment variable and places all flags seen
there on the symbol table defined as true.  ENV: must be assigned to
prevent a requester from popping up asking to (on the Amiga) "Please
Insert Disk ENV:"  See your 1.3 enhancer manual for more information.  If
ENV: is assigned, but M2PFlags is not defined, it is ignored.

The following options are also accepted by M2P as command line arguments.

\begin{description}
\item{INCLUDE {\em path}}  Adds {\em path} to M2P's searchpath for INCLUDE files.
\item{STRIP}   Instructs M2P to strip comments.  (default)
\item{NOSTRIP} Instructs M2P not to strip comments.
\end{description}

M2P can also be ran from the Amiga Workbench.  It will open its own
window and assign the standard input and output channels to that window.  

%====================================================================

\section{Memory Requirements}

M2P requires about 45K statically at runtime.  However, as computations
progress, and large macros are defined, memory is dynamically consumed
in 4K chunks.  Typically, for a large file M2P may need around 80K total.
The M2P executable requires less than 30K.

%====================================================================

\section{Error Messages}

\begin{itemize}

\item   Strings may not be broken across line boundaries.

        A LF,CR,FF, or VT control charactor was found inside a string.
        This may also happen if there is loose apostrophe inside a
        comment containing directives.

\item   Comment not closed with *)

        End of file was reached before M2P could locate the closing *)
        of a comment.
        
\item   Token is too long, limit is 255 charactors

        You have a string or an identifier which is ridiculously long.

\item   Filename (string) expected following @INCLUDE

        Recall the syntax:  @INCLUDE "filename"

\item   Unrecognized Preprocessor statement

        Something was found following an @ which did not form a valid
        preprocessor statement.

\item   Identifier must be defined as a macro

        An attempt was made to use @<id> as a macro, but <id> is 
        not a macro.

\item   WARNING: Symbol has already been defined

        An attempt has been made to bind a symbol to a macro definition,
        or @DEFINE it, but that symbol was previously defined.  The new
        definition will be stacked on top of the old definition.

\item   WARNING: Symbol has not been defined

        An attempt was made to use @<id> as a macro, but <id> is 
        not defined, possibly an argument which was not supplied to
        a macro.

\item   Identifier expected following @UNDEF
\item   Identifier expected following @DEFINE
\item   Identifier expected following @MACRO

        See Section 2 (above)

\item   Unbalanced parenthesis in expression

        If unbalanced parenthesis are desired, use @) or @( to designate
        the unbalanced member.

\item   Illegal factor in expression

        The expression parser expected to see an ID, NOT, or \~\,.

\item   Expected @ELSE or @END
\item   Expected @END
\item   Expected THEN
\item   Expected END after IF-THEN

        See Section 2 (above), under IF-STATEMENT

\item   Expected , or )
\item   Expected Identifier for macro argument name

        See Section 2 (above) for macro instances.

\item   Too many arguments in macro

        You may not supply more arguments to a macro than those given
        in the definition, but you may supply less.

\item   Argument to macro not terminated

        EOF was encountered before finding , or ) in a macro instance.
        
\item   Macro definition not terminated with @ENDM

        End Of File was encountered in a macro definition.

\item   Internal Failure

        An internal sanity check failed.  Please try to replicate and
        then report any such problems.

\end{itemize}

%====================================================================

\section{Amiga CLI Return Codes}

\begin{description}

\item[5:  IOError,]
        A required file could not be opened.
        
\item[10: NoMemory,]
        Could not allocate storage for a dynamic list node.

\item[15: Errors,]
        Preprocessing is complete, but there were errors in the file.
        
\item[20: BadParams,]
        Unrecognizable command line parameters

\end{description}

%====================================================================

\section{Debugging Features}

M2P also supports four hidden command line switches, TOKENS, TRACERDP,
SYMTAB, and MACLIST.  TOKENS will report every token recogized by the 
lexical analyzer after it is recoginized.  TRACERDP will trace calls in the
recursive descent parser.  SYMTAB will report all accesses to the symbol
table.  MACLIST will report uses of the MacLists module functions.
They must be given as the first arguments and may be given in any order.

\begin{quote}\begin{verbatim}
> M2P TRACERDP TOKENS - TO -
\end{verbatim}\end{quote}

This will read from the standard input, and write to the standard output,
and report token information and recursive calls.

{\em Note:}  for performance, these features are optional at compile time.  
       The compiled version included does not support these options.
       To include any one if these features in a recompilation, simply
       include the flag DEBUG with the M2FLAGS in the makefile.

%====================================================================

\section{To Do}

The support for M2Amiga and BenchMark is not really there with the
exception of getting a few compiler directives right.  Important things
like interfacing to their CommandLine, debugging, I/O, and RunTime
libraries still needs to be resolved.  I don't have either of these
compilers, so really couldn not do much.

%====================================================================

\section{Implementation Notes}

The M2Sprint version is much better than the TDI version.  Due to 
the way TDI's Runtime library is set up, some features had to be left
out.  To be specific, the TDI version does not trap CtrlC's, or work
from the Workbench.  Also, it is larger and slower than the M2Sprint 
version.

%====================================================================

\section{Porting}

I have tried to avoid as much OS and compiler dependent stuff
as possible, but some things were hard to justify losing.  This
describes the major areas that may need looking at in order to port M2P.

\begin{itemize}
 \item The Args module needs to be able to get at the command line
       arguments and parse them.  It uses special parsing methods to
       separate (, --, and )'s from other arguments.  This way you can
       avoid hitting the space bar a few times.
       
       If your compiler provides the C-like argv and argc variables, then
       you could use the following code to feed the arguments to Args.
       
\begin{quote}\begin{verbatim}
FOR i:=0 TO <ModuleName>.argc-1 DO
   AddArgs(<ModuleName>.argv[i]^);
END; 
\end{verbatim}\end{quote}
       
       See the internal documentation in Args for more details.
       
 \item IO houses the file handling stuff.  There are three areas of
       implementation dependent code.
       
       \begin{itemize}
       
        \item The type definition of 'RealFileType'.  This is at the
              top of the Implementaton module.
              
        \item After all the interface procedures, there is an 
              implementation dependent section of code that houses
              all the dirty stuff, such as:  ReadInfo, Flush, OSOpen,
              OSCreate, OSAppend, OSClose, and OSGoodFile.
              
        \item The standard I/O channels are picked up in the 
              inialization code.
             
       \end{itemize}
              
       I plan to substitute the standard Modula-2 I/O library 
       for this library in a future release.

 \item The Env module uses the Amiga specific ENV: handler.  Please
       insert your own local method for getenv and setenv.

 \item You will have do some simple bootstrapping to move this code to
       a totally new platform.  This is because all the .mod files here
       are .mpp files and have to be ran through M2P before compilation.

       To bootstrap M2P, use the .mod files in the 'mods' subdirectory.  
       This are the results straight from M2P with the flags DEBUG and 
       AMIGA.  Make the changes needed to
       get M2P running on your target system, and be sure to write them
       down.  When you are done, incorporate those changes into the 
       original .mpp files and remake the system.  You will probably
       have to define some unique flag for your compiler and system.
       Define that flag in the M2FLAGS variable in the Makefile.
      
 \item Most of the .mpp files @INCLUDE the file "MACROS"  This file, as
       the name implies contains some macros.  All of these macros 
       involve the inclusion/exclusion of compiler directives since
       this varies from implementation to implementation.  Edit the
       macros as needed to add the proper directives for your system.
       If in doubt, no harm is done if no directives are included,
       just some runtime performance loss.  Some of the code in "MACROS"
       also serves to configure M2P dynamically to the system.

\end{itemize}

%====================================================================

\section{References and Thanks}

Much of the ideas behind M2P was inspired from the C-Preprocesser and 
also the Modula-2 preprocessor supplied with the YAML distribution 
(by Olsen \& Associates).

%====================================================================

\section{Disclaimer}

This program is placed in the public domain, and may be used or distributed
as you like.
If you make useful changes or bug fixes to M2P, 
I would like to know about them so they may be incorporated into future
versions.  M2P is provided on an as-is basis;  I assume no responsibility 
for GURUS, system crashes, bedwettings, or failed marriages due to the 
use of M2P.


\section{WhoAmI}

\begin{tabbing}
        Dennis Brueni\\
        900 Glade Road, \#9 \\
        Blacksburg, VA 24060 \\
        \\
        EM:  brueni@csgrad.cs.vt.edu\\
\end{tabbing}

If you by any chance feel guilty about getting something for nothing, 
then send a picture of Abe Lincoln to me at the above address, and 
we'll both feel better.  $\smile$

\end{document}
