Class DefaultJFlexLexer

java.lang.Object
com.pnfsoftware.jeb.util.encoding.jflex.DefaultJFlexLexer
Direct Known Subclasses:
JavaLexer, PythonLexer

public abstract class DefaultJFlexLexer extends Object
This is a default, and abstract implementation of a Lexer using JFLex with some utility methods that Lexers can implement. See flex examples at https://github.com/nordfalk/jsyntaxpane.
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected int
     
    protected int
     
    protected long
     
  • Constructor Summary

    Constructors
    Constructor
    Description
     
  • Method Summary

    Modifier and Type
    Method
    Description
    void
    parse(Reader in, List<Token> tokens)
     
     
    protected Token
    Create and return a Token of given type.
    protected Token
    token(TokenType type, int pairValue)
    Create and return a Token of given type and pairValue.
    protected Token
    token(TokenType type, int tStart, int tLength, int newStart, int newLength)
    Helper method to create and return a new Token from of TokenType tokenStart and tokenLength will be modified to the newStart and newLength params
    protected Token
    token(TokenType type, long start, int length)
    Create and return a Token of given type from start with length offset is added to start
    abstract long
    Return the char number from beginning of input stream.
    abstract char
    yycharat(int pos)
    Returns the character at position pos from the matched text.
    abstract int
    Returns the length of the matched text region.
    abstract Token
    This is called to return the next Token from the Input Reader
    abstract void
    yyreset(Reader reader)
    This will be called to reset the the lexer.
    abstract String
    Returns the text matched by the current regular expression.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • tokenStart

      protected long tokenStart
    • tokenLength

      protected int tokenLength
    • offset

      protected int offset
  • Constructor Details

    • DefaultJFlexLexer

      public DefaultJFlexLexer()
  • Method Details

    • token

      protected Token token(TokenType type, int tStart, int tLength, int newStart, int newLength)
      Helper method to create and return a new Token from of TokenType tokenStart and tokenLength will be modified to the newStart and newLength params
    • token

      protected Token token(TokenType type, long start, int length)
      Create and return a Token of given type from start with length offset is added to start
    • token

      protected Token token(TokenType type)
      Create and return a Token of given type. start is obtained from yychar() and length from yylength() offset is added to start
    • token

      protected Token token(TokenType type, int pairValue)
      Create and return a Token of given type and pairValue. start is obtained from yychar() and length from yylength() offset is added to start
    • parse

      public List<Token> parse(String s) throws IOException
      Throws:
      IOException
    • parse

      public void parse(Reader in, List<Token> tokens) throws IOException
      Throws:
      IOException
    • yyreset

      public abstract void yyreset(Reader reader)
      This will be called to reset the the lexer. This is created automatically by JFlex.
    • yylex

      public abstract Token yylex() throws IOException
      This is called to return the next Token from the Input Reader
      Returns:
      next token, or null if no more tokens.
      Throws:
      IOException
    • yycharat

      public abstract char yycharat(int pos)
      Returns the character at position pos from the matched text. It is equivalent to yytext().charAt(pos), but faster
      Parameters:
      pos - the position of the character to fetch. A value from 0 to yylength()-1.
      Returns:
      the character at position pos
    • yylength

      public abstract int yylength()
      Returns the length of the matched text region. This method is automatically implemented by JFlex lexers
    • yytext

      public abstract String yytext()
      Returns the text matched by the current regular expression. This method is automatically implemented by JFlex lexers
    • yychar

      public abstract long yychar()
      Return the char number from beginning of input stream. This is NOT implemented by JFlex, so the code must be added to create this and return the private yychar field