Uses of Class
org.apache.lucene.analysis.Tokenizer
Packages that use Tokenizer
Package
Description
Text analysis.
Fast, general-purpose grammar-based tokenizer
StandardTokenizer implements the Word Break rules from the
Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.-
Uses of Tokenizer in org.apache.lucene.analysis
Methods in org.apache.lucene.analysis that return TokenizerModifier and TypeMethodDescriptionfinal TokenizerTokenizerFactory.create()Creates a TokenStream of the specified input using the default attribute factory.abstract TokenizerTokenizerFactory.create(AttributeFactory factory) Creates a TokenStream of the specified input using the given AttributeFactoryConstructors in org.apache.lucene.analysis with parameters of type TokenizerModifierConstructorDescriptionTokenStreamComponents(Tokenizer tokenizer) Creates a newAnalyzer.TokenStreamComponentsfrom a TokenizerTokenStreamComponents(Tokenizer tokenizer, TokenStream result) Creates a newAnalyzer.TokenStreamComponentsinstance -
Uses of Tokenizer in org.apache.lucene.analysis.standard
Subclasses of Tokenizer in org.apache.lucene.analysis.standardModifier and TypeClassDescriptionfinal classA grammar-based tokenizer constructed with JFlex.