InputFormat (Hadoop 1.2.1 API)
Date: 2013-07-22

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.hadoop.mapred
Interface InputFormat<K,V>

All Known Subinterfaces:
  ComposableInputFormat<K,V>

All Known Implementing Classes:
  AutoInputFormat, CombineFileInputFormat, CompositeInputFormat, DBInputFormat, DelegatingInputFormat, FileInputFormat, KeyValueTextInputFormat, LineDocInputFormat, MultiFileInputFormat, MultiFileWordCount.MyInputFormat, NLineInputFormat, Parser.Node, SequenceFileAsBinaryInputFormat, SequenceFileAsTextInputFormat, SequenceFileInputFilter, SequenceFileInputFormat, SleepJob.SleepInputFormat, StreamInputFormat, TeraInputFormat, TextInputFormat

    public interface InputFormat<K,V>

`InputFormat` describes the input-specification for a Map-Reduce job.

The Map-Reduce framework relies on the `InputFormat` of the job to:

1. Validate the input-specification of the job.
2. Split-up the input file(s) into logical `InputSplit`s, each of which is then assigned to an individual `Mapper`.
3. Provide the `RecordReader` implementation to be used to glean input records from the logical `InputSplit` for processing by the `Mapper`.

The default behavior of file-based `InputFormat`s, typically sub-classes of `FileInputFormat`, is to split the input into logical `InputSplit`s based on the total size, in bytes, of the input files. However, the `FileSystem` blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Clearly, logical splits based on input-size is insufficient for many applications since record boundaries are to respected. In such cases, the application has to also implement a `RecordReader` on whom lies the responsibilty to respect record-boundaries and present a record-oriented view of the logical `InputSplit` to the individual task.

See Also:
  `InputSplit`, `RecordReader`, `JobClient`, `FileInputFormat`

Method Summary
`RecordReader<K,V>` | `getRecordReader(InputSplit split, JobConf job, Reporter reporter)` Get the `RecordReader` for the given `InputSplit`.
`InputSplit[]` | `getSplits(JobConf job, int numSplits)` Logically split the set of input files for the job.

Method Detail

### getSplits

    InputSplit[] getSplits(JobConf job,
                           int numSplits)
                           throws IOException

  Logically split the set of input files for the job.

  Each `InputSplit` is then assigned to an individual `Mapper` for processing.

  Note: The split is a logical split of the inputs and the input files are not physically split into chunks. For e.g. a split could be <input-file-path, start, offset> tuple.

  Parameters:
    `job` - job configuration.
    `numSplits` - the desired number of splits, a hint.
  Returns:
    an array of `InputSplit`s for the job.
  Throws:
    `IOException`

### getRecordReader

    RecordReader<K,V> getRecordReader(InputSplit split,
                                      JobConf job,
                                      Reporter reporter)
                                      throws IOException

  Get the `RecordReader` for the given `InputSplit`.

  It is the responsibility of the `RecordReader` to respect record boundaries while processing the logical split to present a record-oriented view to the individual task.

  Parameters:
    `split` - the `InputSplit`
    `job` - the job that this split belongs to
  Returns:
    a `RecordReader`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2009 The Apache Software Foundation