Overview
This appendix describes the tokenised BBC BASIC file format used by the environment's BASIC conversion tools. It is separated from the general naming appendix so that runtime BASIC representation can be referenced on its own.
Tokenised file structure
Tokenised BBC BASIC file format
The riscos-basictokenise tool writes a tokenised BBC BASIC file as a sequence of line records followed by a fixed trailer. The editable source form should still normally be kept as textual ,fd1 files in Git, but it is useful to know the runtime format when detokenising, packaging, or inspecting generated BASIC programs.
Overall file structure
The file structure emitted by the tokeniser is:
| Offset | Size | Contents |
|---|---|---|
| 0 | variable | One or more tokenised line records. |
| end-1 | 2 | Trailer bytes &0D &FF, marking end of file. |
If the textual source does not include explicit line numbers, the tokeniser numbers lines automatically from 10 in steps of 10 before encoding them.
Line record structure
Each BASIC line is stored with a four-byte header followed by the tokenised text of the line:
| Offset | Size | Contents |
|---|---|---|
| 0 | 1 | Line introducer byte &0D. |
| 1 | 1 | High byte of the line number. |
| 2 | 1 | Low byte of the line number. |
| 3 | 1 | Total record length in bytes, including the four-byte header. |
| 4 | variable | Tokenised line content. |
Unlike the other multi-byte fields described in this book, which are little-endian, this line-number field is big-endian (high byte first). The single-byte total-record-length field bounds a tokenised line to 255 bytes in total, leaving at most 251 bytes for the tokenised content after the four-byte header. The two-byte line-number field can represent 0-65535, though conventional BBC BASIC interpreters restrict usable line numbers to a narrower range (typically 0-65279); this tokeniser itself performs no range check on the value supplied.
The tokeniser copies variable names, most punctuation, and quoted strings through unchanged. A line that begins with a * command is copied verbatim rather than tokenised.
Special encodings
Some items are encoded specially rather than by a simple keyword token:
| Bytes | Meaning |
|---|---|
| 0D FF | End-of-file trailer. |
| 8D xx xx xx | Encoded line-number constant, used after keywords such as GOTO, GOSUB, RESTORE, THEN, or ELSE when they are followed by a line number. |
| CC | Bare ELSE when it appears alone on a line. |
| CF | Statement-form PTR used as a left-value. |
| F4 + text | A REM token followed by the remainder of the line copied without further tokenisation. |
| THEN/ELSE + *command | If a * command follows THEN or ELSE, the command tail is copied verbatim. |
| C6 xx | Two-byte function-style token sequence. |
| C7 xx | Two-byte directive-style token sequence. |
| C8 xx | Two-byte statement-style token sequence. |
The three bytes following the &8D introducer encode the target line number (0-65535) using the same bit-shuffled transformation as the classic Acorn tokenisers:
- Split the line number into hi (bits 8-15) and lo (bits 0-7).
- Build a third value from the top two bits of each: top = ((hi >> 6) << 2) | ((lo >> 6) << 4) (bits 6-7 of hi become bits 2-3 of top; bits 6-7 of lo become bits 4-5).
- Mask hi and lo down to their low 6 bits each.
- Emit three bytes, in this order: top XOR &54, then lo XOR &40, then hi XOR &40 (the low byte of the line number is emitted before the high byte).
To decode, XOR each of the three bytes back with the same constant (&54, &40, &40 respectively) to recover top, lo, and hi, then reconstruct: hi |= ((top >> 2) & 3) << 6, lo |= ((top >> 4) & 3) << 6, and the line number is (hi << 8) | lo. This matches the algorithm originally described by Matt Godbolt's analysis of the BBC BASIC line-number format, which the tokeniser's own source comments cite directly.
Direct one-byte tokens
These tokens appear directly in the line stream without a leading prefix byte:
| Byte | Keyword or form |
|---|---|
| 7F | OTHERWISE |
| 80 | AND |
| 81 | DIV |
| 82 | EOR |
| 83 | MOD |
| 84 | OR |
| 85 | ERROR |
| 86 | LINE |
| 87 | OFF |
| 88 | STEP |
| 89 | SPC |
| 8A | TAB( |
| 8B | ELSE |
| 8C | THEN |
| 8E | APPEND or OPENIN |
| 8F | AUTO or PTR in expression form |
| 90 | CRUNCH or PAGE |
| 91 | DELETE or TIME |
| 92 | EDIT or LOMEM |
| 93 | HIMEM |
| 94 | ABS or LIST |
| 95 | ACS or LOAD |
| 96 | ADVAL |
| 97 | ASC |
| 98 | ASN |
| 99 | ATN or RENUMBER |
| 9A | BGET or SAVE |
| 9B | COS or TEXTLOAD |
| 9C | COUNT or TEXTSAVE |
| 9D | DEG |
| 9E | ERL or TWINO |
| 9F | ERR or INSTALL |
| A0 | EVAL |
| A1 | EXP |
| A2 | EXT |
| A3 | FALSE |
| A4 | FN |
| A5 | GET |
| A6 | INKEY |
| A7 | INSTR( |
| A8 | INT |
| A9 | LEN |
| AA | LN |
| AB | LOG |
| AC | NOT |
| AD | OPENUP |
| AE | OPENOUT |
| AF | PI |
| B0 | POINT( |
| B1 | POS |
| B2 | RAD |
| B3 | RND |
| B4 | SGN |
| B5 | SIN |
| B6 | SQR |
| B7 | TAN |
| B8 | TO |
| B9 | TRUE |
| BA | USR |
| BB | VAL |
| BC | VPOS |
| BD | CHR$ |
| BE | GET$ |
| BF | INKEY$ |
| C0 | LEFT$( |
| C1 | MID$( |
| C2 | RIGHT$( |
| C3 | STR$ |
| C4 | STRING$( |
| C5 | EOF |
| C9 | WHEN |
| CA | OF |
| CB | ENDCASE |
| CD | ENDIF |
| CE | ENDWHILE |
| D4 | SOUND |
| D5 | BPUT |
| D6 | CALL |
| D7 | CHAIN |
| D8 | CLEAR |
| D9 | CLOSE |
| DA | CLG |
| DB | CLS |
| DC | DATA |
| DD | DEF |
| DE | DIM |
| DF | DRAW |
| E0 | END |
| E1 | ENDPROC |
| E2 | ENVELOPE |
| E3 | FOR |
| E4 | GOSUB |
| E5 | GOTO |
| E6 | GCOL |
| E7 | IF |
| E8 | INPUT |
| E9 | LET |
| EA | LOCAL |
| EB | MODE |
| EC | MOVE |
| ED | NEXT |
| EE | ON |
| EF | VDU |
| F0 | PLOT |
| F1 | |
| F2 | PROC |
| F3 | READ |
| F4 | REM |
| F5 | REPEAT |
| F6 | REPORT |
| F7 | RESTORE |
| F8 | RETURN |
| F9 | RUN |
| FA | STOP |
| FB | COLOUR |
| FC | TRACE |
| FD | UNTIL |
| FE | WIDTH |
| FF | OSCLI |
Prefixed two-byte tokens
The tokeniser also emits these prefixed token sequences:
| Bytes | Keyword or form |
|---|---|
| C6 8E | SUM |
| C6 8F | BEAT |
| C7 93 | HELP |
| C7 96 | LVAR or WAIT |
| C7 97 | NEW |
| C7 98 | OLD or QUIT |
| C7 9D | TWIN |
| C8 8E | CASE |
| C8 8F | CIRCLE |
| C8 90 | FILL |
| C8 91 | ORIGIN |
| C8 92 | POINT |
| C8 93 | RECTANGLE |
| C8 94 | SWAP |
| C8 95 | WHILE |
| C8 97 | MOUSE |
| C8 99 | SYS |
| C8 9B | LIBRARY |
| C8 9C | TINT |
| C8 9D | ELLIPSE |
| C8 9E | BEATS |
| C8 9F | TEMPO |
| C8 A0 | VOICES |
| C8 A1 | VOICE |
| C8 A2 | STEREO |
| C8 A3 | OVERLAY |
Conversion workflow
Tokenising and detokenising a program
riscos-basictokenise and riscos-basicdetokenise convert between textual and tokenised BASIC:
riscos-basictokenise program.bas program,ffb
riscos-basicdetokenise -i program,ffb
For a program using only ordinary statements, the round trip is exact: the detokenised text matches the original line for line.
riscos-basicdetokenise does not decode the &8D line-number constant token described above: it prints the three raw transformed bytes as characters under an unlistable token label rather than recovering the original line number. Programs that use GOTO, GOSUB, or similar with a literal line-number target should be checked by eye against the source, not assumed to round-trip cleanly through this tool.