Package antlr

Class TokenStreamRewriteEngine

  • All Implemented Interfaces:
    IASDebugStream, TokenStream
    public class TokenStreamRewriteEngine
extends java.lang.Object
implements TokenStream, IASDebugStream
    This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser. This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1 You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2")); You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer. If you don't use named rewrite streams, a "default" stream is used. Terence Parr, parrt at antlr.org University of San Francisco February 2004

    • Field Detail

      • DEFAULT_PROGRAM_NAME

        public static final java.lang.String DEFAULT_PROGRAM_NAME
        See Also:
        Constant Field Values

      • tokens

        protected java.util.List tokens
        Track the incoming list of tokens

      • programs

        protected java.util.Map programs
        You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

      • lastRewriteTokenIndexes

        protected java.util.Map lastRewriteTokenIndexes
        Map String (program name) -> Integer index

      • index

        protected int index
        track index of tokens

      • stream

        protected TokenStream stream
        Who do we suck tokens from?

      • discardMask

        protected BitSet discardMask
        Which (whitespace) token(s) to throw out

    • Constructor Detail

      • TokenStreamRewriteEngine

        public TokenStreamRewriteEngine​(TokenStream upstream)

      • TokenStreamRewriteEngine

        public TokenStreamRewriteEngine​(TokenStream upstream,
                                int initialSize)

    • Method Detail

      • rollback

        public void rollback​(int instructionIndex)

      • rollback

        public void rollback​(java.lang.String programName,
                     int instructionIndex)
        Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

      • deleteProgram

        public void deleteProgram()

      • deleteProgram

        public void deleteProgram​(java.lang.String programName)
        Reset the program so that no instructions exist

      • addToSortedRewriteList

        protected void addToSortedRewriteList​(java.lang.String programName,
                                      TokenStreamRewriteEngine.RewriteOperation op)
        Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency). The list is ordered so that toString() can be done efficiently. When there are multiple instructions at the same index, the instructions must be ordered to ensure proper behavior. For example, a delete at index i must kill any replace operation at i. Insert-before operations must come before any replace / delete instructions. If there are multiple insert instructions for a single index, they are done in reverse insertion order so that "insert foo" then "insert bar" yields "foobar" in front rather than "barfoo". This is convenient because I can insert new InsertOp instructions at the index returned by the binary search. A ReplaceOp kills any previous replace op. Since delete is the same as replace with null text, i can check for ReplaceOp and cover DeleteOp at same time. :)

      • insertAfter

        public void insertAfter​(Token t,
                        java.lang.String text)

      • insertAfter

        public void insertAfter​(int index,
                        java.lang.String text)

      • insertAfter

        public void insertAfter​(java.lang.String programName,
                        Token t,
                        java.lang.String text)

      • insertAfter

        public void insertAfter​(java.lang.String programName,
                        int index,
                        java.lang.String text)

      • insertBefore

        public void insertBefore​(Token t,
                         java.lang.String text)

      • insertBefore

        public void insertBefore​(int index,
                         java.lang.String text)

      • insertBefore

        public void insertBefore​(java.lang.String programName,
                         Token t,
                         java.lang.String text)

      • insertBefore

        public void insertBefore​(java.lang.String programName,
                         int index,
                         java.lang.String text)

      • replace

        public void replace​(int index,
                    java.lang.String text)

      • replace

        public void replace​(int from,
                    int to,
                    java.lang.String text)

      • replace

        public void replace​(Token indexT,
                    java.lang.String text)

      • replace

        public void replace​(Token from,
                    Token to,
                    java.lang.String text)

      • replace

        public void replace​(java.lang.String programName,
                    int from,
                    int to,
                    java.lang.String text)

      • replace

        public void replace​(java.lang.String programName,
                    Token from,
                    Token to,
                    java.lang.String text)

      • delete

        public void delete​(int index)

      • delete

        public void delete​(int from,
                   int to)

      • delete

        public void delete​(Token indexT)

      • delete

        public void delete​(Token from,
                   Token to)

      • delete

        public void delete​(java.lang.String programName,
                   int from,
                   int to)

      • delete

        public void delete​(java.lang.String programName,
                   Token from,
                   Token to)

      • discard

        public void discard​(int ttype)

      • getTokenStreamSize

        public int getTokenStreamSize()

      • toOriginalString

        public java.lang.String toOriginalString()

      • toOriginalString

        public java.lang.String toOriginalString​(int start,
                                         int end)

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • toString

        public java.lang.String toString​(java.lang.String programName)

      • toString

        public java.lang.String toString​(int start,
                                 int end)

      • toString

        public java.lang.String toString​(java.lang.String programName,
                                 int start,
                                 int end)

      • toDebugString

        public java.lang.String toDebugString()

      • toDebugString

        public java.lang.String toDebugString​(int start,
                                      int end)

      • getLastRewriteTokenIndex

        public int getLastRewriteTokenIndex()

      • getLastRewriteTokenIndex

        protected int getLastRewriteTokenIndex​(java.lang.String programName)

      • setLastRewriteTokenIndex

        protected void setLastRewriteTokenIndex​(java.lang.String programName,
                                        int i)

      • getProgram

        protected java.util.List getProgram​(java.lang.String name)

      • size

        public int size()

      • index

        public int index()

      • getEntireText

        public java.lang.String getEntireText()
        Description copied from interface: IASDebugStream
        Returns the entire text input to the lexer.
        Specified by:
        getEntireText in interface IASDebugStream
        Returns:
        The entire text or null, if error occured or System.in was used.

      • getOffsetInfo

        public TokenOffsetInfo getOffsetInfo​(Token token)
        Description copied from interface: IASDebugStream
        Returns the offset information for the token
        Specified by:
        getOffsetInfo in interface IASDebugStream
        Parameters:
        token - the token whose information need to be retrieved
        Returns:
        offset info, or null