UAX29URLEmailTokenizer (Lucene 3.6.2 API)
Keywords: org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer class

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.lucene.analysis.standard
Class UAX29URLEmailTokenizer

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.Tokenizer
                  org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer

All Implemented Interfaces:
  Closeable

    public final class UAX29URLEmailTokenizer

extends Tokenizer

This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.

Tokens produced are of the following types:

- <ALPHANUM>: A sequence of alphabetic and numeric characters
- <NUM>: A number
- <URL>: A URL
- <EMAIL>: An email address
- <SOUTHEAST_ASIAN>: A sequence of characters from South and Southeast Asian languages, including Thai, Lao, Myanmar, and Khmer
- <IDEOGRAPHIC>: A single CJKV ideographic character
- <HIRAGANA>: A single hiragana character

You must specify the required `Version` compatibility when creating UAX29URLEmailTokenizer:

- As of 3.4, Hiragana and Han characters are no longer wrongly split from their combining characters. If you use a previous version number, you get the exact broken behavior for backwards compatibility.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary
`static int` | `ALPHANUM`
`static int` | `EMAIL`
`static String` | `EMAIL_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `HANGUL`
`static String` | `HANGUL_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `HIRAGANA`
`static String` | `HIRAGANA_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `IDEOGRAPHIC`
`static String` | `IDEOGRAPHIC_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `KATAKANA`
`static String` | `KATAKANA_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `NUM`
`static String` | `NUMERIC_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static String` | `SOUTH_EAST_ASIAN_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static int` | `SOUTHEAST_ASIAN`
`static String[]` | `TOKEN_TYPES` String token types that correspond to token type int constants
`static int` | `URL`
`static String` | `URL_TYPE` Deprecated. use `TOKEN_TYPES` instead
`static String` | `WORD_TYPE` Deprecated. use `TOKEN_TYPES` instead

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`UAX29URLEmailTokenizer(AttributeSource.AttributeFactory factory, Reader input)` Deprecated. use `UAX29URLEmailTokenizer(Version, AttributeSource.AttributeFactory, Reader)` instead.
`UAX29URLEmailTokenizer(AttributeSource source, Reader input)` Deprecated. use `UAX29URLEmailTokenizer(Version, AttributeSource, Reader)` instead.
`UAX29URLEmailTokenizer(InputStream input)` Deprecated. use `UAX29URLEmailTokenizer(Version, Reader)` instead.
`UAX29URLEmailTokenizer(Reader input)` Deprecated. use `UAX29URLEmailTokenizer(Version, Reader)` instead.
`UAX29URLEmailTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader input)` Creates a new UAX29URLEmailTokenizer with a given `AttributeSource.AttributeFactory`
`UAX29URLEmailTokenizer(Version matchVersion, AttributeSource source, Reader input)` Creates a new UAX29URLEmailTokenizer with a given `AttributeSource`.
`UAX29URLEmailTokenizer(Version matchVersion, Reader input)` Creates a new instance of the UAX29URLEmailTokenizer.

Method Summary
`void` | `end()` This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API).
`int` | `getMaxTokenLength()`
`boolean` | `incrementToken()` Consumers (i.e., `IndexWriter`) use this method to advance the stream to the next token.
`void` | `reset(Reader reader)` Expert: Reset the tokenizer to a new reader.
`void` | `setMaxTokenLength(int length)` Set the max allowed token length.

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close, correctOffset`

Methods inherited from class org.apache.lucene.analysis.TokenStream
`reset`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### ALPHANUM

    public static final int ALPHANUM

See Also:
  Constant Field Values

### NUM

    public static final int NUM

See Also:
  Constant Field Values

### SOUTHEAST_ASIAN

    public static final int SOUTHEAST_ASIAN

See Also:
  Constant Field Values

### IDEOGRAPHIC

    public static final int IDEOGRAPHIC

See Also:
  Constant Field Values

### HIRAGANA

    public static final int HIRAGANA

See Also:
  Constant Field Values

### KATAKANA

    public static final int KATAKANA

See Also:
  Constant Field Values

### HANGUL

    public static final int HANGUL

See Also:
  Constant Field Values

### URL

    public static final int URL

See Also:
  Constant Field Values

### EMAIL

    public static final int EMAIL

See Also:
  Constant Field Values

### TOKEN_TYPES

    public static final String[] TOKEN_TYPES

  String token types that correspond to token type int constants

### WORD_TYPE

    @Deprecated
    public static final String WORD_TYPE

  Deprecated. use `TOKEN_TYPES` instead
  Alphanumeric sequences

### NUMERIC_TYPE

    @Deprecated
    public static final String NUMERIC_TYPE

  Deprecated. use `TOKEN_TYPES` instead
  Numbers

### URL_TYPE

    @Deprecated
    public static final String URL_TYPE

  Deprecated. use `TOKEN_TYPES` instead
  URLs with scheme: HTTP(S), FTP, or FILE; no-scheme URLs match HTTP syntax

### EMAIL_TYPE

    @Deprecated
    public static final String EMAIL_TYPE

  Deprecated. use `TOKEN_TYPES` instead
  E-mail addresses

### SOUTH_EAST_ASIAN_TYPE

    @Deprecated
    public static final String SOUTH_EAST_ASIAN_TYPE

  Deprecated. use `TOKEN_TYPES` instead
  Chars in class \p{Line_Break = Complex_Context} are from South East Asian scripts (Thai, Lao, Myanmar, Khmer, etc.). Sequences of these are kept together as as a single token rather than broken up, because the logic required to break them at word boundaries is too complex for UAX#29.

  See Unicode Line Breaking Algorithm: http://www.unicode.org/reports/tr14/#SA

### IDEOGRAPHIC_TYPE

    @Deprecated
    public static final String IDEOGRAPHIC_TYPE

  Deprecated. use `TOKEN_TYPES` instead

### HIRAGANA_TYPE

    @Deprecated
    public static final String HIRAGANA_TYPE

  Deprecated. use `TOKEN_TYPES` instead

### KATAKANA_TYPE

    @Deprecated
    public static final String KATAKANA_TYPE

  Deprecated. use `TOKEN_TYPES` instead

### HANGUL_TYPE

    @Deprecated
    public static final String HANGUL_TYPE

  Deprecated. use `TOKEN_TYPES` instead

Constructor Detail

### UAX29URLEmailTokenizer

    @Deprecated
    public UAX29URLEmailTokenizer(Reader input)

  Deprecated. use `UAX29URLEmailTokenizer(Version, Reader)` instead.

### UAX29URLEmailTokenizer

    @Deprecated
    public UAX29URLEmailTokenizer(InputStream input)

  Deprecated. use `UAX29URLEmailTokenizer(Version, Reader)` instead.

### UAX29URLEmailTokenizer

    @Deprecated
    public UAX29URLEmailTokenizer(AttributeSource source,
                                             Reader input)

  Deprecated. use `UAX29URLEmailTokenizer(Version, AttributeSource, Reader)` instead.

### UAX29URLEmailTokenizer

    @Deprecated
    public UAX29URLEmailTokenizer(AttributeSource.AttributeFactory factory,
                                             Reader input)

  Deprecated. use `UAX29URLEmailTokenizer(Version, AttributeSource.AttributeFactory, Reader)` instead.

### UAX29URLEmailTokenizer

    public UAX29URLEmailTokenizer(Version matchVersion,
                                  Reader input)

  Creates a new instance of the UAX29URLEmailTokenizer. Attaches the `input` to the newly created JFlex scanner.

  Parameters:
    `input` - The input reader

### UAX29URLEmailTokenizer

    public UAX29URLEmailTokenizer(Version matchVersion,
                                  AttributeSource source,
                                  Reader input)

  Creates a new UAX29URLEmailTokenizer with a given `AttributeSource`.

### UAX29URLEmailTokenizer

    public UAX29URLEmailTokenizer(Version matchVersion,
                                  AttributeSource.AttributeFactory factory,
                                  Reader input)

  Creates a new UAX29URLEmailTokenizer with a given `AttributeSource.AttributeFactory`

Method Detail

### setMaxTokenLength

    public void setMaxTokenLength(int length)

  Set the max allowed token length. Any token longer than this is skipped.

### getMaxTokenLength

    public int getMaxTokenLength()

  See Also:
    `setMaxTokenLength(int)`

### incrementToken

    public final boolean incrementToken()
                                 throws IOException

  Description copied from class: `TokenStream`
  Consumers (i.e., `IndexWriter`) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate `AttributeImpl`s with the attributes of the next token.

  The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use `AttributeSource.captureState()` to create a copy of the current attribute state.

  This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to `AttributeSource.addAttribute(Class)` and `AttributeSource.getAttribute(Class)`, references to all `AttributeImpl`s that this stream uses should be retrieved during instantiation.

  To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in `TokenStream.incrementToken()`.

  Specified by:
    `incrementToken` in class `TokenStream`
  Returns:
    false for end of stream; true otherwise
  Throws:
    `IOException`

### end

    public final void end()

  Description copied from class: `TokenStream`
  This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API). Streams implementing the old API should upgrade to use this feature.

  This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a `WhitespaceTokenizer` was used.

  Overrides:
    `end` in class `TokenStream`

### reset

    public void reset(Reader reader)
               throws IOException

  Description copied from class: `Tokenizer`
  Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

  Overrides:
    `reset` in class `Tokenizer`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD