Package org.apache.lucene.analysis.email
Class UAX29URLEmailTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.email.UAX29URLEmailTokenizer
-
- All Implemented Interfaces:
Closeable,AutoCloseable
public final class UAX29URLEmailTokenizer extends Tokenizer
This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description static intALPHANUMAlpha/numeric token typestatic intEMAILEmail token typestatic intEMOJIEmoji token type.static intHANGULHangul token typestatic intHIRAGANAHiragana token typestatic intIDEOGRAPHICIdeographic token typestatic intKATAKANAKatakana token typestatic intMAX_TOKEN_LENGTH_LIMITAbsolute maximum sized tokenstatic intNUMNumeric token typestatic intSOUTHEAST_ASIANSoutheast Asian token typestatic String[]TOKEN_TYPESString token types that correspond to token type int constantsstatic intURLURL token type-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description UAX29URLEmailTokenizer()Creates a new instance of the UAX29URLEmailTokenizer.UAX29URLEmailTokenizer(AttributeFactory factory)Creates a new UAX29URLEmailTokenizer with a givenAttributeFactory
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidclose()voidend()intgetMaxTokenLength()booleanincrementToken()voidreset()voidsetMaxTokenLength(int length)Set the max allowed token length.-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
correctOffset, setReader, setReaderTestPoint
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
ALPHANUM
public static final int ALPHANUM
Alpha/numeric token type- See Also:
- Constant Field Values
-
NUM
public static final int NUM
Numeric token type- See Also:
- Constant Field Values
-
SOUTHEAST_ASIAN
public static final int SOUTHEAST_ASIAN
Southeast Asian token type- See Also:
- Constant Field Values
-
IDEOGRAPHIC
public static final int IDEOGRAPHIC
Ideographic token type- See Also:
- Constant Field Values
-
HIRAGANA
public static final int HIRAGANA
Hiragana token type- See Also:
- Constant Field Values
-
KATAKANA
public static final int KATAKANA
Katakana token type- See Also:
- Constant Field Values
-
HANGUL
public static final int HANGUL
Hangul token type- See Also:
- Constant Field Values
-
URL
public static final int URL
URL token type- See Also:
- Constant Field Values
-
EMAIL
public static final int EMAIL
Email token type- See Also:
- Constant Field Values
-
EMOJI
public static final int EMOJI
Emoji token type.- See Also:
- Constant Field Values
-
TOKEN_TYPES
public static final String[] TOKEN_TYPES
String token types that correspond to token type int constants
-
MAX_TOKEN_LENGTH_LIMIT
public static final int MAX_TOKEN_LENGTH_LIMIT
Absolute maximum sized token- See Also:
- Constant Field Values
-
-
Constructor Detail
-
UAX29URLEmailTokenizer
public UAX29URLEmailTokenizer()
Creates a new instance of the UAX29URLEmailTokenizer. Attaches theinputto the newly created JFlex scanner.
-
UAX29URLEmailTokenizer
public UAX29URLEmailTokenizer(AttributeFactory factory)
Creates a new UAX29URLEmailTokenizer with a givenAttributeFactory
-
-
Method Detail
-
setMaxTokenLength
public void setMaxTokenLength(int length)
Set the max allowed token length. Tokens larger than this will be chopped up at this token length and emitted as multiple tokens. If you need to skip such large tokens, you could increase this max length, and then useLengthFilterto remove long tokens. The default isUAX29URLEmailAnalyzer.DEFAULT_MAX_TOKEN_LENGTH.- Throws:
IllegalArgumentException- if the given length is outside of the range [1, 1048576].
-
getMaxTokenLength
public int getMaxTokenLength()
- See Also:
setMaxTokenLength(int)
-
incrementToken
public final boolean incrementToken() throws IOException- Specified by:
incrementTokenin classTokenStream- Throws:
IOException
-
end
public final void end() throws IOException- Overrides:
endin classTokenStream- Throws:
IOException
-
close
public void close() throws IOException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Overrides:
closein classTokenizer- Throws:
IOException
-
reset
public void reset() throws IOException- Overrides:
resetin classTokenizer- Throws:
IOException
-
-