As you may be aware, when a Basic program is stored in memory (or in a file), the keywords in it (PRINT, REPEAT, SIN etc.) are replaced by a token - a number unique to that particular keyword. This saves RAM, and greatly improves execution speed because each keyword is much more easily identified when the program is run. The penalty is that the keywords must be tokenised when the program is entered, and detokenised when it is listed.
On the earlier BBC micros, any utility that wished to convert a token to its corresponding keyword, or vice-versa, either had to have a complete table of keywords (lengthy), or had to illegally access the table in the Basic ROM (prone to incompatibility problems). Basic V on the Archimedes is much more civilised, and provides a routine that converts a token into its keyword. It is this technique which is used in the program listing utility elsewhere in this issue.
The routine in question, given the name TOKENADDR, is one of a number of routines whose addresses are made available when a machine code routine is called using the CALL statement. These will be the subject of a future article, but all that is necessary to use the TOKENADDR routine is to include a short machine code patch in your program. This patch can be found in the example program (explained later) between lines 80 and 150. You should include this in your own code, and execute it when the program is first run.
To use this routine, set A% to the value of the token (and B% to the second byte for composite tokens - see below) and execute CALLcode. This will call the TOKENADDR routine, and then store the address where the keyword can be found in the variable temp. This will be a memory address within the Basic module. The keyword is terminated by a byte with a value of &7F or greater, and therefore, the keyword can be printed using the following code:
ptr=!temp
WHILE ?ptr<&7F
VDU ?ptr:ptr+=1
ENDWHILE
The tokens for most keywords fall in the ranges &7F-&8C, &8E-&C5 or &C9-&FF. However, some of the keywords new to Basic V have two-byte tokens. These consist of a byte of &C6, &C7 or &C8, followed by a second byte identifying the actual token. The valid range for this second byte depends on the first byte:
First byte | Valid range of |
second byte | |
&C6 | &8E-&8F |
&C7 | &8E-&9B |
&C8 | &8E-&A4 |
There are a few points to note about token values:
1. The value &8D is not a valid token. Instead, it is a marker used to identify line references in GOTO, RESTORE etc. It is followed by three bytes containing a line number in coded form.
2. The keyword BY (used for relative co-ordinates) isn't tokenised
3. There are two tokens for ELSE. A value of &8C is used for an ELSE in an IF..THEN..ELSE statement (and ON..GOTO..ELSE etc.), and a value of &CC for an ELSE in an IF..THEN..ELSE..ENDIF structure.
4. TOP and LISTO do not have their own tokens as such. Instead, they use the token for TO followed by a 'P', and the token for LIST followed by an 'O' respectively. TWINO, on the other hand, does have a token separate from that for TWIN.
The TOKENADDR routine is fine for converting a token into its keyword, but what about converting a keyword into its token in the first place. Unfortunately, Basic doesn't make available the routine to do this. The only simple solution is to include a list of keywords and tokens in your program. Rather than doing this manually, you can use TOKENADDR with each valid token in turn to produce the entire list.
To allow you to experiment with the TOKENADDR routine, the following program will print out all the valid tokens with their corresponding keywords. Note that the composite token C8 A4 represents the keyword 'OVERLAY' which is new to the Basic 1.04 supplied with RISC OS. If this program is run on an older version of Basic, then the keyword printed for this token will be meaningless.
10 REM >TList
20 REM Program Basic Token Lister
30 REM Version A 1.0
40 REM Author David Spencer
50 REM RISC User April 1989
60 REM Program subject to copyright
70 :
80 DIM code 100
90 FOR pass=0 TO 2 STEP 2
100 P%=code:[OPT pass
110 STMFD R13!,{R14}:MOV R2,R14
120 ADR R12,temp:STR R1,[R12]
130 ADR R14,back:ADD PC,R2,#&4C
140 .back STR R1,temp:LDMFD R13!,{PC}
150 .temp EQUD 0:]NEXT
160 REPEAT READ al%,ah%,bl%,bh%
170 IF al%<>0 THEN
180 FOR A%=al% TO ah%
190 FOR B%=bl% TO bh%
200 CALL code
210 PRINT~A%;:IF B% THEN PRINT" ";~B%;
220 PRINTTAB(12);:ptr=!temp
230 WHILE ?ptr<&7F
240 VDU ?ptr:ptr+=1
250 ENDWHILE
260 PRINT:NEXT
270 NEXT
280 PRINT
290 ENDIF
300 UNTIL al%=0
310 END
320 :
330 DATA 127,140,0,0
340 DATA 142,197,0,0
350 DATA 201,255,0,0
360 DATA 198,198,142,143
370 DATA 199,199,142,155
380 DATA 200,200,142,163
390 DATA 0,0,0,0