Package org.apache.lucene.analysis.ja
Class Token
- java.lang.Object
-
- org.apache.lucene.analysis.ja.Token
-
public class Token extends Object
Analyzed token with morphological data from its dictionary.
-
-
Constructor Summary
Constructors Constructor Description Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description StringgetBaseForm()StringgetInflectionForm()StringgetInflectionType()intgetLength()intgetOffset()StringgetPartOfSpeech()intgetPosition()Get index of this token in input textintgetPositionLength()Get the length (in tokens) of this token.StringgetPronunciation()StringgetReading()char[]getSurfaceForm()StringgetSurfaceFormString()JapaneseTokenizer.TypegetType()Returns the type of this tokenbooleanisKnown()Returns true if this token is known wordbooleanisUnknown()Returns true if this token is unknown wordbooleanisUser()Returns true if this token is defined in user dictionaryvoidsetPositionLength(int positionLength)Set the position length (in tokens) of this token.StringtoString()
-
-
-
Constructor Detail
-
Token
public Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary)
-
-
Method Detail
-
getSurfaceForm
public char[] getSurfaceForm()
- Returns:
- surfaceForm
-
getOffset
public int getOffset()
- Returns:
- offset into surfaceForm
-
getLength
public int getLength()
- Returns:
- length of surfaceForm
-
getSurfaceFormString
public String getSurfaceFormString()
- Returns:
- surfaceForm as a String
-
getReading
public String getReading()
- Returns:
- reading. null if token doesn't have reading.
-
getPronunciation
public String getPronunciation()
- Returns:
- pronunciation. null if token doesn't have pronunciation.
-
getPartOfSpeech
public String getPartOfSpeech()
- Returns:
- part of speech.
-
getInflectionType
public String getInflectionType()
- Returns:
- inflection type or null
-
getInflectionForm
public String getInflectionForm()
- Returns:
- inflection form or null
-
getBaseForm
public String getBaseForm()
- Returns:
- base form or null if token is not inflected
-
getType
public JapaneseTokenizer.Type getType()
Returns the type of this token- Returns:
- token type, not null
-
isKnown
public boolean isKnown()
Returns true if this token is known word- Returns:
- true if this token is in standard dictionary. false if not.
-
isUnknown
public boolean isUnknown()
Returns true if this token is unknown word- Returns:
- true if this token is unknown word. false if not.
-
isUser
public boolean isUser()
Returns true if this token is defined in user dictionary- Returns:
- true if this token is in user dictionary. false if not.
-
getPosition
public int getPosition()
Get index of this token in input text- Returns:
- position of token
-
setPositionLength
public void setPositionLength(int positionLength)
Set the position length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.
-
getPositionLength
public int getPositionLength()
Get the length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.- Returns:
- position length of token
-
-