Package java.io

Previous

Java API

Index

Next

Class StreamTokenizer

public  class  java.io.StreamTokenizer
    extends  java.lang.Object  
{
        // Fields
    public double nval;	
    public String sval;	
    public int ttype;	

        // possible values for the ttype field
    public final static int TT_EOF;	
    public final static int TT_EOL;	
    public final static int TT_NUMBER;	
    public final static int TT_WORD;	

        // Constructors
    public StreamTokenizer(InputStream  I);	

        // Methods
    public void commentChar(int  ch);	
    public void eolIsSignificant(boolean  flag);	
    public int lineno();	
    public void lowerCaseMode(boolean  fl);	
    public int nextToken();	
    public void ordinaryChar(int  ch);	
    public void ordinaryChars(int  low, int  hi);	
    public void parseNumbers();	
    public void pushBack();	
    public void quoteChar(int  ch);	
    public void resetSyntax();	
    public void whitespaceChars(int  low, int  hi);	
    public void slashStarComments(boolean  flag);	
    public String toString();	
    public void whitespaceChars(int  low, int  hi);	
    public void wordChars(int  low, int  hi);	
}

The StreamTokenizer class takes an input stream and parses it into "tokens", allowing the tokens to be read one at a time. The parsing process is controlled by a table and a number of flags that can be set to various states. The stream tokenizer can recognize identifiers, numbers, quoted strings, and various comment styles.

Each byte read from the input stream is regarded as a character in the range '\u0000' through '\u00FF'. The character value is used to look up five possible attributes of the character: whitespace, alphabetic, numeric, string quote, and comment character. Each character can have zero or more of these attributes.

In addition an instance has four flags. These flags indicate:

• Whether line terminators are to be returned as tokens or treated as whitespace that merely separates tokens.
• Whether C-style comments are to be recognized and skipped.
• Whether C++-style comments are to be recognized and skipped.
• Whether the characters of identifiers are converted to lowercase.

A typical application first constructs an instance of this class, sets up the syntax tables, and then repeatedly loops calling the nextToken method in each iteration of the loop until it returns the value TT_EOF .

Fields

nval

public double nval 

If the current token is a number, this field contains the value of that number. The current token is a number when the value of the ttype field is TT_NUMBER .

sval

public String sval 

If the current token is a word token, this field contains a string giving the characters of the word token. When the current token is a quoted string token, this field contains the body of the string.

The current token is a word when when the value of the ttype field is TT_WORD . The current token is a quoted string token when the value of the ttype field is a quote character (see quoteChar).

ttype

public int ttype 

After a call to the nextToken method , this field contains the type of the token just read. For a single character token, its value is the single character, converted to an integer. For a quoted string token (see quoteChar), its value is the quote character. Otherwise, its value is one of the following:

• TT_WORD indicates that the token is a word.

• TT_NUMBER indicates that the token is a number
• TT_EOL indicates that the end of line has been read. The field can only have this value if the eolIsSignficant method has been called with the argument true.
• TT_EOF indicates that the end of the input stream has been reached.

TT_EOF

public final static int TT_EOF = -1 

A constant indicating that the end of the stream has been read.

TT_EOL

public final static int TT_EOL = '\n' 

A constant indicating that the end of line has been read.

TT_NUMBER

public final static int TT_NUMBER = -2 

A constant indicating that a number has been read.

TT_WORD

public final static int TT_WORD = -3 

A constant indicating that a token has been read.

Constructors

StreamTokenizer

public StreamTokenizer(InputStream  I) 

Creates a stream tokenizer that parses the specified input stream. The stream tokenizer is initialized to the following default state:

• All byte values 'A' through 'Z', 'a' through 'z', and '\u00A0' through '\u00FF' are considered to be alphabetic.

• All byte values '\u0000' through '\u0020' are considered to be whitespace.
• '/' is a comment character.
• Single quote '\'' and double quote '"' are string quote characters.
• Numbers are parsed
• End of lines are not treated as whitespace, not as separate tokens.
• C-style and C++-style comments are not recognized.

Parameter	Description
I	the input stream

Methods

commentChar

public void commentChar(int  ch) 

Specified that the character argument starts a single line comment. All characters from the comment character to the end of the line are ignored by this stream tokenizer.

Parameter	Description
ch	the character

eolIsSignificant

public void eolIsSignificant(boolean  flag) 

If the flag argument is true, this tokenizer treats end of lines as tokens; the nextToken method returns TT_EOL and also sets the ttype field to this value when an end of line is read.

A line is a sequence of characters ending with either a carriage return character ('\r') or a newline character ('\n'). In addition, a carriage return character followed immediately by a newline character is treated as a single end- of-line token.

If the flag is false, end of line characters are treated as whitespace and serve only to separate tokens.

Parameter	Description
flag	true indicates that end-of-line characters are separate tokens; false indicates that end-of-line characters are whitespace

lineno

public int lineno() 

Return Value:

Returns the current line number of this stream tokenizer

lowerCaseMode

public void lowerCaseMode(boolean  fl) 

If the flag argument is true, then the value in the sval field is lowercased whenever a word token is returned (the ttype field has the value TT_WORD ) by the nextToken method of this tokenizer.

If the flag argument is false, then the sval field is not modified.

Parameter	Description
fl	true indicates that all word tokens should be lowercased

nextToken

public int nextToken() 

throws IOException 

Parses the next token from the input stream of this tokenizer. The type of the next token is returned in the ttype field . Additional information about the token may be in the nval field or the sval field of this tokenizer.

Return Value:

Returns The value of the ttype field .

Throw:

IOException

If an I/O error occurs.

ordinaryChar

public void ordinaryChar(int  ch) 

Specifies that the character argument is "ordinary" in this tokenizer. It removes any special significance the character has as a comment character, word component, string delimiter, whitespace, or number character. When such a character is encountered by the parser, the parser treates it as a single-character token and sets ttype field to the character value.

Parameter	Description
ch	the character

ordinaryChars

public void ordinaryChars(int  low, int  hi) 

Specifies that all characters c in the range low £ c £ hi are "ordinary" in this tokenizer. See the ordinaryChar method for more information on a character being ordinary.

Parameter	Description
low	the low end of the range
hi	the high end of the range

parseNumbers

public void parseNumbers() 

Specifies that numbers should be parsed by this tokenizer. The syntax table of this tokenizer is modified so that each of the twelve characters

	0 1 2 3 4 5 6 7 8 9 . - 

has the "numeric" attribute.

When the parser encounters a word token that has the format of a double precision floating point number, it treates the token as a number rather than a word, by setting the the ttype field to the value TT_NUMBER and putting the numeric value of the token into the nval field .

pushBack

public void pushBack() 

Causes the next call to the nextToken method of this tokenizer to return the current value in the ttype field , and not to modify the value in the nval or sval field.

quoteChar

public void quoteChar(int  ch) 

Specifies that matching pairs of this character delimit string constants in this tokenizer.

When the nextToken method encounters a string constant, the ttype field field is set to the string delimiter and the sval field is set to the body of the string.

If a string quote character is encountered, then a string is recognized, consisting of all characters after (but not including) the string quote character, up to (but not including) the next occurrence of that same string quote character, or a line terminator, or end of file. The usual escape sequences such as "\n" and "\t" are recognized and converted to single characters as the string is parsed.

Parameter	Description
ch	the character

resetSyntax

public void resetSyntax() 

Resets this tokenizer's syntax table so that all characters are "ordinary." See the ordinaryChar method for more information on a character being ordinary.

slashSlashComments

public void slashSlashComments(boolean  flag) 

If the flag argument is true, this stream tokenizer recognizes C++ style comments. Any occurrence of two consecutive slash characters ('/') is treated as the beginning of a comment that extends to the end of the line.

If the flag argument is false, then C++ style comments are not treated specially.

Parameter	Description
flag	true indicates to recognize and ignore C++ style comments

slashStarComments

public void slashStarComments(boolean  flag) 

If the flag argument is true, this stream tokenizer recognizes C style comments. All text between successive occurrences of /* and */ are discarded.

If the flag argument is false, then C style comments are not treated specially.

Parameter	Description
flag	true indicates to recognize and ignore C style comments

toString

public String toString() 

Returns the string representation of the current stream token.

Return Value:

Returns a string representation of the token specified by the ttype , nval , and sval fields.

Overrides:

toString in class Object .

whitespaceChars

public void whitespaceChars(int  low, int  hi) 

Specifies that all characters c in the range low £ c £ hi are whitespace character. Whitespace characters serve only to separate tokens in the input stream.

Parameter	Description
low	the low end of the range
hi	the high end of the range

wordChars

public void wordChars(int  low, int  hi) 

Specifies that all characters c in the range low £ c £ hi are word constituents. A word token consists of a word constitutent followed by zero or more word constituents or number constituents.

Parameter	Description
low	the low end of the range
hi	the high end of the range