public final class FTLexer extends FTIterator implements IndexToken
Constructor and Description |
---|
FTLexer()
Constructor, using the default full-text options.
|
FTLexer(FTOpt ftopt)
Default constructor.
|
Modifier and Type | Method and Description |
---|---|
FTLexer |
all()
If called, all tokens will be returned (including non-fulltext tokens).
|
int |
count()
Returns total number of tokens.
|
FTOpt |
ftOpt()
Returns the full-text options.
|
byte[] |
get()
Returns the original token.
|
boolean |
hasNext() |
int[][] |
info()
Gets full-text info for the specified token.
|
void |
init()
Initializes the iterator.
|
FTLexer |
init(byte[] txt)
Initializes the iterator.
|
static StringList |
languages()
Lists all languages for which tokenizers and stemmers are available.
|
int |
lserror(byte[] token)
Returns the Levenshtein error for the specified token.
|
void |
lserror(int ls)
Sets the Levenshtein error.
|
FTSpan |
next() |
byte[] |
nextToken()
Returns the next token.
|
FTLexer |
original()
If called, the original tokens will be returned (including non-fulltext tokens).
|
boolean |
paragraph()
Returns if the current token starts a new paragraph.
|
int |
pos(int word,
FTUnit unit)
Calculates a position value, dependent on the specified unit.
|
byte[] |
text()
Returns the text to be processed.
|
IndexType |
type()
Returns the index type.
|
remove
public FTLexer()
FTFilter
, and the map visualizations.public FTLexer(FTOpt ftopt)
ftopt
- full-text optionspublic FTLexer original()
public FTLexer all()
public void init()
public void lserror(int ls)
ls
- errorpublic int lserror(byte[] token)
token
- tokenpublic FTLexer init(byte[] txt)
FTIterator
init
in class FTIterator
txt
- textpublic byte[] nextToken()
FTIterator
Iterator.next()
to avoid the creation of new FTSpan
instances.nextToken
in class FTIterator
public int count()
public IndexType type()
IndexToken
type
in interface IndexToken
public byte[] get()
IndexToken
;
use next()
or nextToken()
if not using this interface.get
in interface IndexToken
public FTOpt ftOpt()
null
)public byte[] text()
public boolean paragraph()
public int pos(int word, FTUnit unit)
word
- word positionunit
- unitpublic int[][] info()
Tokenizer.info()
for more info.public static StringList languages()
Copyright © 2005–2015 BaseX Team. All rights reserved.