org.apache.lucene.util.encoding (Lucene 3.6.2 API)
Keywords: org.apache.lucene.util.encoding package

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES

## Package org.apache.lucene.util.encoding

Offers various encoders and decoders for integers, as well as the mechanisms to create new ones.

See:
Description

Class Summary
ChunksIntEncoder | An `IntEncoder` which encodes values in chunks.
DGapIntDecoder | An `IntDecoder` which wraps another `IntDecoder` and reverts the d-gap that was encoded by `DGapIntEncoder`.
DGapIntEncoder | An `IntEncoderFilter` which encodes the gap between the given values, rather than the values themselves.
EightFlagsIntDecoder | Decodes data which was encoded by `EightFlagsIntEncoder`.
EightFlagsIntEncoder | A `ChunksIntEncoder` which encodes data in chunks of 8.
FourFlagsIntDecoder | Decodes data which was encoded by `FourFlagsIntEncoder`.
FourFlagsIntEncoder | A `ChunksIntEncoder` which encodes values in chunks of 4.
IntDecoder | Decodes integers from a set `InputStream`.
IntEncoder | Encodes integers to a set `OutputStream`.
IntEncoderFilter | An abstract implementation of `IntEncoder` which is served as a filter on the values to encode.
NOnesIntDecoder | Decodes data which was encoded by `NOnesIntEncoder`.
NOnesIntEncoder | A variation of `FourFlagsIntEncoder` which translates the data as follows: Values ≥ 2 are trnalsated to `value+1` (2 ⇒ 3, 3 ⇒ 4 and so forth).
SimpleIntDecoder | A simple stream decoder which can decode values encoded with `SimpleIntEncoder`.
SimpleIntEncoder | A simple `IntEncoder`, writing an integer as 4 raw bytes.
SortingIntEncoder | An `IntEncoderFilter` which sorts the values to encode in ascending order before encoding them.
UniqueValuesIntEncoder | An `IntEncoderFilter` which ensures only unique values are encoded.
VInt8IntDecoder | An `IntDecoder` which can decode values encoded by `VInt8IntEncoder`.
VInt8IntEncoder | An `IntEncoder` which implements variable length encoding.

## Package org.apache.lucene.util.encoding Description

Offers various encoders and decoders for integers, as well as the mechanisms to create new ones. The super class for all encoders is `IntEncoder` and for most of the encoders there is a matching `IntDecoder` implementation (not all encoders need a decoder).

An encoder encodes the integers that are passed to `encode` into a set output stream (see `reInit`). One should always call `close` when all integers have been encoded, to ensure proper finish by the encoder. Some encoders buffer values in-memory and encode in batches in order to optimize the encoding, and not closing them may result in loss of information or corrupt stream.

A proper and typical usage of an encoder looks like this:

>     int[] data = <the values to encode>
>     IntEncoder encoder = new VInt8IntEncoder();
>     OutputStream out = new ByteArrayOutputStream();
>     encoder.reInit(out);
>     for (int val : data) {
>       encoder.encode(val);
>     }
>     encoder.close();

>     // Print the bytes in binary
>     byte[] bytes = out.toByteArray();
>     for (byte b : bytes) {
>       System.out.println(Integer.toBinaryString(b));
>     }

Each encoder also implements `createMatchingDecoder` which returns the matching decoder for this encoder. As mentioned above, not all encoders have a matching decoder (like some encoder filters which are explained next), however every encoder should return a decoder following a call to that method. To complete the example above, one can easily iterate over the decoded values like this:

>     IntDecoder d = e.createMatchingDecoder();
>     d.reInit(new ByteArrayInputStream(bytes));
>     long val;
>     while ((val = d.decode()) != IntDecoder.EOS) {
>       System.out.println(val);
>     }

Some encoders don't perform any encoding at all, or do not include an encoding logic. Those are called `IntEncoderFilter`s. A filter is an encoder which delegates the encoding task to a given encoder, however performs additional logic before the values are sent for encoding. An example is `DGapIntEncoder` which encodes the gaps between values rather than the values themselves. Another example is `SortingIntEncoder` which sorts all the values in ascending order before they are sent for encoding. This encoder aggregates the values in its `encode` implementation and decoding only happens upon calling `close`.

#### Extending IntEncoder

Extending `IntEncoder` is a very easy task. One only needs to implement `encode` and `createMatchingDecoder` as the base implementation takes care of re-initializing the output stream and closing it. The following example illustrates how can one write an encoder (and a matching decoder) which 'tags' the stream with type/ID of the encoder. Such tagging is important in scenarios where an application uses different encoders for different streams, and wants to manage some sort of mapping between an encoder ID to an IntEncoder/Decoder implementation, so a proper decoder will be initialized on the fly:

>     public class TaggingIntEncoder extends IntEncoderFilter {

>       public TaggingIntEncoder(IntEncoder encoder) {
>         super(encoder);
>       }

>       @Override
>       public void encode(int value) throws IOException {
>         encoder.encode(value);
>       }

>       @Override
>       public IntDecoder createMatchingDecoder() {
>         return new TaggingIntDecoder();
>       }

>       @Override
>       public void reInit(OutputStream out) {
>         super.reInit(os);
>         // Assumes the application has a static EncodersMap class which is able to
>         // return a unique ID for a given encoder.
>         int encoderID = EncodersMap.getID(encoder);
>         this.out.write(encoderID);
>       }

>       @Override
>       public String toString() {
>         return "Tagging (" + encoder.toString() + ")";
>       }

>     }

And the matching decoder:

>     public class TaggingIntDecoder extends IntDecoder {

>       // Will be initialized upon calling reInit.
>       private IntDecoder decoder;

>       @Override
>       public void reInit(InputStream in) {
>         super.reInit(in);

>         // Read the ID of the encoder that tagged this stream.
>         int encoderID = in.read();

>         // Assumes EncodersMap can return the proper IntEncoder given the ID.
>         decoder = EncodersMap.getEncoder(encoderID).createMatchingDecoder();
>       }

>       @Override
>       public long decode() throws IOException {
>         return decoder.decode();
>       }

>       @Override
>       public String toString() {
>         return "Tagging (" + decoder == null ? "none" : decoder.toString() + ")";
>       }

>     }

The example implements `TaggingIntEncoder` as a filter over another encoder. Even though it does not do any filtering on the actual values, it feels right to present it as a filter. Anyway, this is just an example code and one can choose to implement it however it makes sense to the application. For simplicity, error checking was omitted from the sample code.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES