The token_info/1,2 functions may change in the R13B release.
This module contains functions for tokenizing characters into Erlang tokens.
category() = atom() column() = integer() > 0 line() = integer() location() = line() | {line(), column()} reserved_word_fun() -> fun(atom()) -> bool() symbol() = atom() | float() | integer() | string() token() = {category(), token_data()} | {category(), token_data(), symbol()} token_data() = line() | tuple()
string(String) -> Return
string(String, StartLocation) -> Return
string(String, StartLocation, Options) -> Return
Types:
String = string()
Return = {ok, Tokens, EndLocation} | Error
Tokens = [token()]
Error = {error, ErrorInfo, EndLocation}
StartLocation = EndLocation = location()
Options = Option | [Option]
Option = {reserved_word_fun,reserved_word_fun()}
| return_comments | return_white_spaces | return
Takes the list of characters String and tries to scan (tokenize) them. Returns {ok, Tokens, EndLocation}, where Tokens are the Erlang tokens from String. EndLocation is the first location after the last token.
{error, ErrorInfo, EndLocation} is returned if an error occurs. EndLocation is the first location after the erroneous token.
string(String) is equivalent to string(String, 1).
StartLocation indicates the initial location when scanning starts. If StartLocation is a line, token_data() as well as EndLocation and ErrorLocation will be lines. If StartLocation is a pair of a line and a column, token_data() takes the form of a tuple, and EndLocation and ErrorLocation will be pairs of a line and a column. The token data tuple contains information about the column and the line where the token begins, as well as the text of the token, all of which can be accessed by calling token_info/1,2.
A token is a tuple containing information about syntactic category, the token data, and the actual terminal symbol. For punctuation characters (e.g. ;, |) and reserved words, the category and the symbol coincide, and the token is represented by a two-tuple. Three-tuples have one of the following forms: {atom, Info, atom()}, {char, Info, integer()}, {comment, Info, string()}, {float, Info, float()}, {integer, Info, integer()}, {var, Info, atom()}, and {white_space, Info, string()}.
The valid options are:
tokens(Continuation, CharSpec, StartLocation) -> Return
tokens(Continuation, CharSpec, StartLocation, Options) -> Return
Types:
Continuation = [] | Continuation1
Return = {done, Result, LeftOverChars} | {more, Continuation1}
LeftOverChars = CharSpec
CharSpec = string() | eof
Continuation1 = tuple()
Result = {ok, Tokens, EndLocation} | {eof, EndLocation} | Error
Tokens = [token()]
Error = {error, ErrorInfo, EndLocation}
StartLocation = EndLocation = location()
Options = Option | [Option]
Option = {reserved_word_fun,reserved_word_fun()}
| return_comments | return_white_spaces | return
This is the re-entrant scanner which scans characters until a dot ('.' followed by a white space) or eof has been reached. It returns:
The CharSpec eof signals end of file. LeftoverChars will then take the value eof as well.
See string/3 for a description of the various options.
Types:
Atom = atom()
Returns true if Atom is an Erlang reserved word, otherwise false.
token_info(Token) -> InfoResult
Types:
Token = token()
InfoResult = [InfoTuple]
InfoTuple = {Item, Info}
Item = atom()
Info = term()
Returns a list containing information about the Erlang token Token. The order of the InfoTuples is not defined. Currently the following Items are returned: category, column, length, line, symbol, and text. See token_info/2 for information about specific InfoTuples.
Note that if token_info(Token, Item) returns undefined for some Item in the list above, the item is not included in InfoResult.
token_info(Token, ItemSpec) -> InfoResult
Types:
Token = token()
ItemSpec = Item | [Item]
InfoResult = InfoTuple | undefined | [InfoTuple]
InfoTuple = {Item, Info}
Item = atom()
Info = term()
Returns a list containing information about the Erlang token Token. If ItemSpec is a single Item, the returned value is the corresponding InfoTuple, or undefined if the Item has no value. If ItemSpec is a list of ItemSpec, the result is a list of InfoTuple. The InfoTuples will appear with the corresponding Items in the same order as the Items appeared in the list of Items. Items with no value are not included in the list of InfoTuple.
Currently the following InfoTuples with corresponding Items are valid:
The token_info/1,2 functions may change in the R13B release.
format_error(ErrorDescriptor) -> string()
Types:
ErrorDescriptor = errordesc()
Takes an ErrorDescriptor and returns a string which describes the error or warning. This function is usually called implicitly when processing an ErrorInfo structure (see below).
The ErrorInfo mentioned above is the standard ErrorInfo structure which is returned from all IO modules. It has the following format:
{ErrorLocation, Module, ErrorDescriptor}
A string which describes the error is obtained with the following call:
Module:format_error(ErrorDescriptor)
The continuation of the first call to the re-entrant input functions must be []. Refer to Armstrong, Virding and Williams, 'Concurrent Programming in Erlang', Chapter 13, for a complete description of how the re-entrant input scheme works.