EdgeNGramTokenizer (Lucene 3.6.2 API)
Keywords: org.apache.lucene.analysis.ngram.EdgeNGramTokenizer class

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.lucene.analysis.ngram
Class EdgeNGramTokenizer

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.Tokenizer
                  org.apache.lucene.analysis.ngram.EdgeNGramTokenizer

All Implemented Interfaces:
  Closeable

    public final class EdgeNGramTokenizer

extends Tokenizer

Tokenizes the input from an edge into n-grams of given size(s).

This `Tokenizer` create n-grams from the beginning edge or ending edge of a input token. MaxGram can't be larger than 1024 because of limitation.

Nested Class Summary
`static class` | `EdgeNGramTokenizer.Side` Specifies which side of the input the n-gram should be generated from

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary
`static int` | `DEFAULT_MAX_GRAM_SIZE`
`static int` | `DEFAULT_MIN_GRAM_SIZE`
`static EdgeNGramTokenizer.Side` | `DEFAULT_SIDE`

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`EdgeNGramTokenizer(AttributeSource.AttributeFactory factory, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
`EdgeNGramTokenizer(AttributeSource.AttributeFactory factory, Reader input, String sideLabel, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
`EdgeNGramTokenizer(AttributeSource source, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
`EdgeNGramTokenizer(AttributeSource source, Reader input, String sideLabel, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
`EdgeNGramTokenizer(Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
`EdgeNGramTokenizer(Reader input, String sideLabel, int minGram, int maxGram)` Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

Method Summary
`void` | `end()` This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API).
`boolean` | `incrementToken()` Returns the next token in the stream, or null at EOS.
`void` | `reset()` Resets this stream to the beginning.

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close, correctOffset, reset`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### DEFAULT_SIDE

    public static final EdgeNGramTokenizer.Side DEFAULT_SIDE

### DEFAULT_MAX_GRAM_SIZE

    public static final int DEFAULT_MAX_GRAM_SIZE

See Also:
  Constant Field Values

### DEFAULT_MIN_GRAM_SIZE

    public static final int DEFAULT_MIN_GRAM_SIZE

See Also:
  Constant Field Values

Constructor Detail

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(Reader input,
                              EdgeNGramTokenizer.Side side,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `input` - `Reader` holding the input to be tokenized
    `side` - the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(AttributeSource source,
                              Reader input,
                              EdgeNGramTokenizer.Side side,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `source` - `AttributeSource` to use
    `input` - `Reader` holding the input to be tokenized
    `side` - the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
                              Reader input,
                              EdgeNGramTokenizer.Side side,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `factory` - `AttributeSource.AttributeFactory` to use
    `input` - `Reader` holding the input to be tokenized
    `side` - the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(Reader input,
                              String sideLabel,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `input` - `Reader` holding the input to be tokenized
    `sideLabel` - the name of the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(AttributeSource source,
                              Reader input,
                              String sideLabel,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `source` - `AttributeSource` to use
    `input` - `Reader` holding the input to be tokenized
    `sideLabel` - the name of the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

### EdgeNGramTokenizer

    public EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
                              Reader input,
                              String sideLabel,
                              int minGram,
                              int maxGram)

  Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

  Parameters:
    `factory` - `AttributeSource.AttributeFactory` to use
    `input` - `Reader` holding the input to be tokenized
    `sideLabel` - the name of the `EdgeNGramTokenizer.Side` from which to chop off an n-gram
    `minGram` - the smallest n-gram to generate
    `maxGram` - the largest n-gram to generate

Method Detail

### incrementToken

    public boolean incrementToken()
                           throws IOException

  Returns the next token in the stream, or null at EOS.

  Specified by:
    `incrementToken` in class `TokenStream`
  Returns:
    false for end of stream; true otherwise
  Throws:
    `IOException`

### end

    public void end()

  Description copied from class: `TokenStream`
  This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API). Streams implementing the old API should upgrade to use this feature.

  This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a `WhitespaceTokenizer` was used.

  Overrides:
    `end` in class `TokenStream`

### reset

    public void reset()
               throws IOException

  Description copied from class: `TokenStream`
  Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. `TokenStream.reset()` is not needed for the standard indexing process. However, if the tokens of a `TokenStream` are intended to be consumed more than once, it is necessary to implement `TokenStream.reset()`. Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after `TokenStream.reset()`).

  Overrides:
    `reset` in class `TokenStream`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD