Package antlr
Class TokenStreamRewriteEngine
- java.lang.Object
-
- antlr.TokenStreamRewriteEngine
-
- All Implemented Interfaces:
IASDebugStream
,TokenStream
public class TokenStreamRewriteEngine extends java.lang.Object implements TokenStream, IASDebugStream
This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser. This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1 You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2")); You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer. If you don't use named rewrite streams, a "default" stream is used. Terence Parr, parrt at antlr.org University of San Francisco February 2004
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static class
TokenStreamRewriteEngine.DeleteOp
(package private) static class
TokenStreamRewriteEngine.InsertBeforeOp
(package private) static class
TokenStreamRewriteEngine.ReplaceOp
I'm going to try replacing range from x..y with (y-x)+1 ReplaceOp instructions.(package private) static class
TokenStreamRewriteEngine.RewriteOperation
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DEFAULT_PROGRAM_NAME
protected BitSet
discardMask
Which (whitespace) token(s) to throw outprotected int
index
track index of tokensprotected java.util.Map
lastRewriteTokenIndexes
Map String (program name) -> Integer indexstatic int
MIN_TOKEN_INDEX
static int
PROGRAM_INIT_SIZE
protected java.util.Map
programs
You may have multiple, named streams of rewrite operations.protected TokenStream
stream
Who do we suck tokens from?protected java.util.List
tokens
Track the incoming list of tokens
-
Constructor Summary
Constructors Constructor Description TokenStreamRewriteEngine(TokenStream upstream)
TokenStreamRewriteEngine(TokenStream upstream, int initialSize)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
addToSortedRewriteList(TokenStreamRewriteEngine.RewriteOperation op)
If op.index > lastRewriteTokenIndexes, just add to the end.protected void
addToSortedRewriteList(java.lang.String programName, TokenStreamRewriteEngine.RewriteOperation op)
Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency).void
delete(int index)
void
delete(int from, int to)
void
delete(Token indexT)
void
delete(Token from, Token to)
void
delete(java.lang.String programName, int from, int to)
void
delete(java.lang.String programName, Token from, Token to)
void
deleteProgram()
void
deleteProgram(java.lang.String programName)
Reset the program so that no instructions existvoid
discard(int ttype)
java.lang.String
getEntireText()
Returns the entire text input to the lexer.int
getLastRewriteTokenIndex()
protected int
getLastRewriteTokenIndex(java.lang.String programName)
TokenOffsetInfo
getOffsetInfo(Token token)
Returns the offset information for the tokenprotected java.util.List
getProgram(java.lang.String name)
TokenWithIndex
getToken(int i)
int
getTokenStreamSize()
int
index()
void
insertAfter(int index, java.lang.String text)
void
insertAfter(Token t, java.lang.String text)
void
insertAfter(java.lang.String programName, int index, java.lang.String text)
void
insertAfter(java.lang.String programName, Token t, java.lang.String text)
void
insertBefore(int index, java.lang.String text)
void
insertBefore(Token t, java.lang.String text)
void
insertBefore(java.lang.String programName, int index, java.lang.String text)
void
insertBefore(java.lang.String programName, Token t, java.lang.String text)
Token
nextToken()
void
replace(int from, int to, java.lang.String text)
void
replace(int index, java.lang.String text)
void
replace(Token from, Token to, java.lang.String text)
void
replace(Token indexT, java.lang.String text)
void
replace(java.lang.String programName, int from, int to, java.lang.String text)
void
replace(java.lang.String programName, Token from, Token to, java.lang.String text)
void
rollback(int instructionIndex)
void
rollback(java.lang.String programName, int instructionIndex)
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.protected void
setLastRewriteTokenIndex(java.lang.String programName, int i)
int
size()
java.lang.String
toDebugString()
java.lang.String
toDebugString(int start, int end)
java.lang.String
toOriginalString()
java.lang.String
toOriginalString(int start, int end)
java.lang.String
toString()
java.lang.String
toString(int start, int end)
java.lang.String
toString(java.lang.String programName)
java.lang.String
toString(java.lang.String programName, int start, int end)
-
-
-
Field Detail
-
MIN_TOKEN_INDEX
public static final int MIN_TOKEN_INDEX
- See Also:
- Constant Field Values
-
DEFAULT_PROGRAM_NAME
public static final java.lang.String DEFAULT_PROGRAM_NAME
- See Also:
- Constant Field Values
-
PROGRAM_INIT_SIZE
public static final int PROGRAM_INIT_SIZE
- See Also:
- Constant Field Values
-
tokens
protected java.util.List tokens
Track the incoming list of tokens
-
programs
protected java.util.Map programs
You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)
-
lastRewriteTokenIndexes
protected java.util.Map lastRewriteTokenIndexes
Map String (program name) -> Integer index
-
index
protected int index
track index of tokens
-
stream
protected TokenStream stream
Who do we suck tokens from?
-
discardMask
protected BitSet discardMask
Which (whitespace) token(s) to throw out
-
-
Constructor Detail
-
TokenStreamRewriteEngine
public TokenStreamRewriteEngine(TokenStream upstream)
-
TokenStreamRewriteEngine
public TokenStreamRewriteEngine(TokenStream upstream, int initialSize)
-
-
Method Detail
-
nextToken
public Token nextToken() throws TokenStreamException
- Specified by:
nextToken
in interfaceTokenStream
- Throws:
TokenStreamException
-
rollback
public void rollback(int instructionIndex)
-
rollback
public void rollback(java.lang.String programName, int instructionIndex)
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!
-
deleteProgram
public void deleteProgram()
-
deleteProgram
public void deleteProgram(java.lang.String programName)
Reset the program so that no instructions exist
-
addToSortedRewriteList
protected void addToSortedRewriteList(TokenStreamRewriteEngine.RewriteOperation op)
If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
-
addToSortedRewriteList
protected void addToSortedRewriteList(java.lang.String programName, TokenStreamRewriteEngine.RewriteOperation op)
Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency). The list is ordered so that toString() can be done efficiently. When there are multiple instructions at the same index, the instructions must be ordered to ensure proper behavior. For example, a delete at index i must kill any replace operation at i. Insert-before operations must come before any replace / delete instructions. If there are multiple insert instructions for a single index, they are done in reverse insertion order so that "insert foo" then "insert bar" yields "foobar" in front rather than "barfoo". This is convenient because I can insert new InsertOp instructions at the index returned by the binary search. A ReplaceOp kills any previous replace op. Since delete is the same as replace with null text, i can check for ReplaceOp and cover DeleteOp at same time. :)
-
insertAfter
public void insertAfter(Token t, java.lang.String text)
-
insertAfter
public void insertAfter(int index, java.lang.String text)
-
insertAfter
public void insertAfter(java.lang.String programName, Token t, java.lang.String text)
-
insertAfter
public void insertAfter(java.lang.String programName, int index, java.lang.String text)
-
insertBefore
public void insertBefore(Token t, java.lang.String text)
-
insertBefore
public void insertBefore(int index, java.lang.String text)
-
insertBefore
public void insertBefore(java.lang.String programName, Token t, java.lang.String text)
-
insertBefore
public void insertBefore(java.lang.String programName, int index, java.lang.String text)
-
replace
public void replace(int index, java.lang.String text)
-
replace
public void replace(int from, int to, java.lang.String text)
-
replace
public void replace(Token indexT, java.lang.String text)
-
replace
public void replace(java.lang.String programName, int from, int to, java.lang.String text)
-
replace
public void replace(java.lang.String programName, Token from, Token to, java.lang.String text)
-
delete
public void delete(int index)
-
delete
public void delete(int from, int to)
-
delete
public void delete(Token indexT)
-
delete
public void delete(java.lang.String programName, int from, int to)
-
discard
public void discard(int ttype)
-
getToken
public TokenWithIndex getToken(int i)
-
getTokenStreamSize
public int getTokenStreamSize()
-
toOriginalString
public java.lang.String toOriginalString()
-
toOriginalString
public java.lang.String toOriginalString(int start, int end)
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
toString
public java.lang.String toString(java.lang.String programName)
-
toString
public java.lang.String toString(int start, int end)
-
toString
public java.lang.String toString(java.lang.String programName, int start, int end)
-
toDebugString
public java.lang.String toDebugString()
-
toDebugString
public java.lang.String toDebugString(int start, int end)
-
getLastRewriteTokenIndex
public int getLastRewriteTokenIndex()
-
getLastRewriteTokenIndex
protected int getLastRewriteTokenIndex(java.lang.String programName)
-
setLastRewriteTokenIndex
protected void setLastRewriteTokenIndex(java.lang.String programName, int i)
-
getProgram
protected java.util.List getProgram(java.lang.String name)
-
size
public int size()
-
index
public int index()
-
getEntireText
public java.lang.String getEntireText()
Description copied from interface:IASDebugStream
Returns the entire text input to the lexer.- Specified by:
getEntireText
in interfaceIASDebugStream
- Returns:
- The entire text or
null
, if error occured or System.in was used.
-
getOffsetInfo
public TokenOffsetInfo getOffsetInfo(Token token)
Description copied from interface:IASDebugStream
Returns the offset information for the token- Specified by:
getOffsetInfo
in interfaceIASDebugStream
- Parameters:
token
- the token whose information need to be retrieved- Returns:
- offset info, or
null
-
-