Class SimplePatternTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.pattern.SimplePatternTokenizer
-
- All Implemented Interfaces:
Closeable,AutoCloseable
public final class SimplePatternTokenizer extends Tokenizer
This tokenizer uses a LuceneRegExpor (expert usage) a pre-built determinizedAutomaton, to locate tokens. The regexp syntax is more limited thanPatternTokenizer, but the tokenization is quite a bit faster. The provided regex should match valid token characters (not token separator characters, likeString.split). The matching is greedy: the longest match at a given start point will be the next token. Empty string tokens are never produced.- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description SimplePatternTokenizer(String regexp)SeeRegExpfor the accepted syntax.SimplePatternTokenizer(AttributeFactory factory, String regexp, int determinizeWorkLimit)SeeRegExpfor the accepted syntax.SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)Runs a pre-built automaton.SimplePatternTokenizer(Automaton dfa)Runs a pre-built automaton.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidend()booleanincrementToken()voidreset()-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, setReader, setReaderTestPoint
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Constructor Detail
-
SimplePatternTokenizer
public SimplePatternTokenizer(String regexp)
SeeRegExpfor the accepted syntax.
-
SimplePatternTokenizer
public SimplePatternTokenizer(Automaton dfa)
Runs a pre-built automaton.
-
SimplePatternTokenizer
public SimplePatternTokenizer(AttributeFactory factory, String regexp, int determinizeWorkLimit)
SeeRegExpfor the accepted syntax.
-
SimplePatternTokenizer
public SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)
Runs a pre-built automaton.
-
-
Method Detail
-
incrementToken
public boolean incrementToken() throws IOException- Specified by:
incrementTokenin classTokenStream- Throws:
IOException
-
end
public void end() throws IOException- Overrides:
endin classTokenStream- Throws:
IOException
-
reset
public void reset() throws IOException- Overrides:
resetin classTokenizer- Throws:
IOException
-
-