TaskTracker (Hadoop 1.2.1 API)
Date: 2013-07-22

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.hadoop.mapred
Class TaskTracker

    java.lang.Object
      org.apache.hadoop.mapred.TaskTracker

All Implemented Interfaces:
  Runnable, VersionedProtocol, TaskTrackerMXBean, TaskUmbilicalProtocol

    public class TaskTracker

extends Object
implements TaskUmbilicalProtocol, Runnable, TaskTrackerMXBean

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.

Nested Class Summary
`static class` | `TaskTracker.MapOutputServlet` This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.

Field Summary
`static org.apache.commons.logging.Log` | `ClientTraceLog`
`static long` | `COUNTER_UPDATE_INTERVAL`
`static long` | `DEFAULT_DISK_HEALTH_CHECK_INTERVAL` How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
`static int` | `FILE_NOT_FOUND`
`static String` | `FOR_REDUCE_TASK` The reduce task number for which this map output is being transferred
`static String` | `FROM_MAP_TASK` The map task from which the map output data is being transferred
`static int` | `HEARTBEAT_INTERVAL_MIN`
`static org.apache.commons.logging.Log` | `LOG`
`static String` | `MAP_OUTPUT_LENGTH` The custom http header used for the map output length.
`static String` | `MR_CLIENTTRACE_FORMAT`
`static String` | `RAW_MAP_OUTPUT_LENGTH` The custom http header used for the "raw" map output length.
`static String` | `SUBDIR`
`static int` | `SUCCESS`
`static String` | `TT_KEYTAB_FILE`
`static String` | `TT_LOG_TMP_DIR`
`static String` | `TT_RESOURCE_CALCULATOR_PLUGIN`
`static String` | `TT_USER_NAME`
`static String` | `WORKDIR`

Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
`versionID`

Constructor Summary
`TaskTracker(JobConf conf)` Start with the local machine name, and the default JobTracker

Method Summary
`boolean` | `canCommit(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)` Child checking whether it can commit
`void` | `cleanupStorage()` Removes all contents of temporary storage.
`void` | `close()` Close down the TaskTracker and all its components.
`void` | `commitPending(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext)` Task is reporting that it is in commit_pending and it is waiting for the commit Response
`void` | `done(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)` The task is done.
`void` | `fatalError(TaskAttemptID taskId, String msg, org.apache.hadoop.mapred.JvmContext jvmContext)` A child task had a fatal error.
`void` | `fsError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext)` A child task had a local filesystem error.
`String` | `getConfigVersion()`
`String` | `getHostname()`
`int` | `getHttpPort()`
`static String` | `getJobCacheSubdir(String user)`
`org.apache.hadoop.mapred.InterTrackerProtocol` | `getJobClient()` The connection to the JobTracker, used by the TaskRunner for locating remote files.
`static String` | `getJobJarFile(String user, String jobid)`
`String` | `getJobTrackerUrl()`
`org.apache.hadoop.mapred.JvmManager` | `getJvmManagerInstance()`
`static String` | `getLocalJobDir(String user, String jobid)`
`static String` | `getLocalTaskDir(String user, String jobid, String taskid)`
`static String` | `getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)`
`MapTaskCompletionEventsUpdate` | `getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id, org.apache.hadoop.mapred.JvmContext jvmContext)` Called by a reduce task to get the map output locations for finished maps.
`static String` | `getPrivateDistributedCacheDir(String user)`
`long` | `getProtocolVersion(String protocol, long clientVersion)` Return protocol version corresponding to protocol interface.
`static String` | `getPublicDistributedCacheDir()`
`int` | `getRpcPort()`
`JvmTask` | `getTask(org.apache.hadoop.mapred.JvmContext context)` Called upon startup by the child process, to fetch Task data.
`TaskController` | `getTaskController()`
`org.apache.hadoop.mapred.TaskMemoryManagerThread` | `getTaskMemoryManager()`
`String` | `getTasksInfoJson()`
`org.apache.hadoop.mapred.TaskTrackerInstrumentation` | `getTaskTrackerInstrumentation()`
`InetSocketAddress` | `getTaskTrackerReportAddress()` Return the port at which the tasktracker bound to
`static String` | `getUserDir(String user)`
`String` | `getVersion()`
`boolean` | `isHealthy()`
`boolean` | `isIdle()` Is this task tracker idle?
`boolean` | `isIdleAndClean()` Is this task tracker idle and clean?
`boolean` | `isTaskMemoryManagerEnabled()` Is the TaskMemoryManager Enabled on this system?
`protected void` | `launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip, JobConf jobConf, org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)`
`static void` | `main(String[] argv)` Start the TaskTracker, point toward the indicated JobTracker
`void` | `mapOutputLost(TaskAttemptID taskid, String errorMsg)` A completed map task's output has been lost.
`boolean` | `ping(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)` Child checking to see if we're alive.
`void` | `reportDiagnosticInfo(TaskAttemptID taskid, String info, org.apache.hadoop.mapred.JvmContext jvmContext)` Called when the task dies before completion, and we want to report back diagnostic info
`void` | `reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range, org.apache.hadoop.mapred.JvmContext jvmContext)` Report the record range which is going to process next by the Task.
`void` | `run()` The server retry loop.
`void` | `shuffleError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext)` A reduce-task failed to shuffle the map-outputs.
`void` | `shutdown()`
`boolean` | `statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext)` Called periodically to report Task progress, from 0.0 to 1.0.
`void` | `updatePrivateDistributedCacheSizes(JobID jobId, long[] sizes)` The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### LOG

    public static final org.apache.commons.logging.Log LOG

### MR_CLIENTTRACE_FORMAT

    public static final String MR_CLIENTTRACE_FORMAT

See Also:
  Constant Field Values

### ClientTraceLog

    public static final org.apache.commons.logging.Log ClientTraceLog

### SUBDIR

    public static final String SUBDIR

See Also:
  Constant Field Values

### TT_LOG_TMP_DIR

    public static final String TT_LOG_TMP_DIR

See Also:
  Constant Field Values

### TT_RESOURCE_CALCULATOR_PLUGIN

    public static final String TT_RESOURCE_CALCULATOR_PLUGIN

See Also:
  Constant Field Values

### TT_USER_NAME

    public static final String TT_USER_NAME

See Also:
  Constant Field Values

### TT_KEYTAB_FILE

    public static final String TT_KEYTAB_FILE

See Also:
  Constant Field Values

### HEARTBEAT_INTERVAL_MIN

    public static final int HEARTBEAT_INTERVAL_MIN

See Also:
  Constant Field Values

### COUNTER_UPDATE_INTERVAL

    public static final long COUNTER_UPDATE_INTERVAL

See Also:
  Constant Field Values

### DEFAULT_DISK_HEALTH_CHECK_INTERVAL

    public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL

  How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

  See Also:
    Constant Field Values

### SUCCESS

    public static final int SUCCESS

See Also:
  Constant Field Values

### FILE_NOT_FOUND

    public static final int FILE_NOT_FOUND

See Also:
  Constant Field Values

### MAP_OUTPUT_LENGTH

    public static final String MAP_OUTPUT_LENGTH

  The custom http header used for the map output length.

  See Also:
    Constant Field Values

### RAW_MAP_OUTPUT_LENGTH

    public static final String RAW_MAP_OUTPUT_LENGTH

  The custom http header used for the "raw" map output length.

  See Also:
    Constant Field Values

### FROM_MAP_TASK

    public static final String FROM_MAP_TASK

  The map task from which the map output data is being transferred

  See Also:
    Constant Field Values

### FOR_REDUCE_TASK

    public static final String FOR_REDUCE_TASK

  The reduce task number for which this map output is being transferred

  See Also:
    Constant Field Values

### WORKDIR

    public static final String WORKDIR

See Also:
  Constant Field Values

Constructor Detail

### TaskTracker

    public TaskTracker(JobConf conf)
                throws IOException,
                       InterruptedException

  Start with the local machine name, and the default JobTracker

  Throws:
    `IOException`
    `InterruptedException`

Method Detail

### getTaskTrackerInstrumentation

    public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

### getTaskController

    public TaskController getTaskController()

### getUserDir

    public static String getUserDir(String user)

### getPrivateDistributedCacheDir

    public static String getPrivateDistributedCacheDir(String user)

### getPublicDistributedCacheDir

    public static String getPublicDistributedCacheDir()

### getJobCacheSubdir

    public static String getJobCacheSubdir(String user)

### getLocalJobDir

    public static String getLocalJobDir(String user,
                                        String jobid)

### getJobJarFile

    public static String getJobJarFile(String user,
                                       String jobid)

### getLocalTaskDir

    public static String getLocalTaskDir(String user,
                                         String jobid,
                                         String taskid)

### getLocalTaskDir

    public static String getLocalTaskDir(String user,
                                         String jobid,
                                         String taskid,
                                         boolean isCleanupAttempt)

### getProtocolVersion

    public long getProtocolVersion(String protocol,
                                   long clientVersion)
                            throws IOException

  Description copied from interface: `VersionedProtocol`
  Return protocol version corresponding to protocol interface.

  Specified by:
    `getProtocolVersion` in interface `VersionedProtocol`
  Parameters:
    `protocol` - The classname of the protocol interface
    `clientVersion` - The version of the protocol that the client speaks
  Returns:
    the version that the server will speak
  Throws:
    `IOException`

### cleanupStorage

    public void cleanupStorage()
                        throws IOException

  Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run.

  Throws:
    `IOException`

### launchTaskForJob

    protected void launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip,
                                    JobConf jobConf,
                                    org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
                             throws IOException

  Throws:
    `IOException`

### shutdown

    public void shutdown()
                  throws IOException,
                         InterruptedException

  Throws:
    `IOException`
    `InterruptedException`

### close

    public void close()
               throws IOException,
                      InterruptedException

  Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

  Throws:
    `InterruptedException`
    `IOException`

### getJobClient

    public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()

  The connection to the JobTracker, used by the TaskRunner for locating remote files.

### getTaskTrackerReportAddress

    public InetSocketAddress getTaskTrackerReportAddress()

  Return the port at which the tasktracker bound to

### getJvmManagerInstance

    public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

### run

    public void run()

  The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

  Specified by:
    `run` in interface `Runnable`

### getTask

    public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                    throws IOException

  Called upon startup by the child process, to fetch Task data.

  Specified by:
    `getTask` in interface `TaskUmbilicalProtocol`
  Parameters:
    `context` - the JvmContext of the JVM w.r.t the TaskTracker that launched it
  Returns:
    Task object
  Throws:
    `IOException`

### statusUpdate

    public boolean statusUpdate(TaskAttemptID taskid,
                                TaskStatus taskStatus,
                                org.apache.hadoop.mapred.JvmContext jvmContext)
                         throws IOException

  Called periodically to report Task progress, from 0.0 to 1.0.

  Specified by:
    `statusUpdate` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task-id of the child
    `taskStatus` - status of the child
    `jvmContext` - context the jvmContext running the task.
  Returns:
    True if the task is known
  Throws:
    `IOException`

### reportDiagnosticInfo

    public void reportDiagnosticInfo(TaskAttemptID taskid,
                                     String info,
                                     org.apache.hadoop.mapred.JvmContext jvmContext)
                              throws IOException

  Called when the task dies before completion, and we want to report back diagnostic info

  Specified by:
    `reportDiagnosticInfo` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - the id of the task involved
    `info` - the text to report
    `jvmContext` - context the jvmContext running the task.
  Throws:
    `IOException`

### reportNextRecordRange

    public void reportNextRecordRange(TaskAttemptID taskid,
                                      org.apache.hadoop.mapred.SortedRanges.Range range,
                                      org.apache.hadoop.mapred.JvmContext jvmContext)
                               throws IOException

  Description copied from interface: `TaskUmbilicalProtocol`
  Report the record range which is going to process next by the Task.

  Specified by:
    `reportNextRecordRange` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - the id of the task involved
    `range` - the range of record sequence nos
    `jvmContext` - context the jvmContext running the task.
  Throws:
    `IOException`

### ping

    public boolean ping(TaskAttemptID taskid,
                        org.apache.hadoop.mapred.JvmContext jvmContext)
                 throws IOException

  Child checking to see if we're alive. Normally does nothing.

  Specified by:
    `ping` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - the id of the task involved
    `jvmContext` - context the jvmContext running the task.
  Returns:
    True if the task is known
  Throws:
    `IOException`

### commitPending

    public void commitPending(TaskAttemptID taskid,
                              TaskStatus taskStatus,
                              org.apache.hadoop.mapred.JvmContext jvmContext)
                       throws IOException

  Task is reporting that it is in commit_pending and it is waiting for the commit Response

  Specified by:
    `commitPending` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task's id
    `taskStatus` - status of the child
    `jvmContext` - context the jvmContext running the task.
  Throws:
    `IOException`

### canCommit

    public boolean canCommit(TaskAttemptID taskid,
                             org.apache.hadoop.mapred.JvmContext jvmContext)
                      throws IOException

  Child checking whether it can commit

  Specified by:
    `canCommit` in interface `TaskUmbilicalProtocol`
    `jvmContext` - context the jvmContext running the task.
  Returns:
    true/false
  Throws:
    `IOException`

### done

    public void done(TaskAttemptID taskid,
                     org.apache.hadoop.mapred.JvmContext jvmContext)
              throws IOException

  The task is done.

  Specified by:
    `done` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task's id
    `jvmContext` - context the jvmContext running the task.
  Throws:
    `IOException`

### shuffleError

    public void shuffleError(TaskAttemptID taskId,
                             String message,
                             org.apache.hadoop.mapred.JvmContext jvmContext)
                      throws IOException

  A reduce-task failed to shuffle the map-outputs. Kill the task.

  Specified by:
    `shuffleError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### fsError

    public void fsError(TaskAttemptID taskId,
                        String message,
                        org.apache.hadoop.mapred.JvmContext jvmContext)
                 throws IOException

  A child task had a local filesystem error. Kill the task.

  Specified by:
    `fsError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### fatalError

    public void fatalError(TaskAttemptID taskId,
                           String msg,
                           org.apache.hadoop.mapred.JvmContext jvmContext)
                    throws IOException

  A child task had a fatal error. Kill the task.

  Specified by:
    `fatalError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### getMapCompletionEvents

    public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                                int fromEventId,
                                                                int maxLocs,
                                                                TaskAttemptID id,
                                                                org.apache.hadoop.mapred.JvmContext jvmContext)
                                                         throws IOException

  Description copied from interface: `TaskUmbilicalProtocol`
  Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

  Specified by:
    `getMapCompletionEvents` in interface `TaskUmbilicalProtocol`
  Parameters:
    `jobId` - the reducer job id
    `fromEventId` - the index starting from which the locations should be fetched
    `maxLocs` - the max number of locations to fetch
    `id` - The attempt id of the task that is trying to communicate
  Returns:
    A `MapTaskCompletionEventsUpdate`
  Throws:
    `IOException`

### mapOutputLost

    public void mapOutputLost(TaskAttemptID taskid,
                              String errorMsg)
                       throws IOException

  A completed map task's output has been lost.

  Throws:
    `IOException`

### isIdle

    public boolean isIdle()

  Is this task tracker idle?

  Returns:
    has this task tracker finished all its assigned tasks?

### isIdleAndClean

    public boolean isIdleAndClean()

  Is this task tracker idle and clean?

  Returns:
    has this task tracker finished and cleaned up all of its tasks?

### main

    public static void main(String[] argv)
                     throws Exception

  Start the TaskTracker, point toward the indicated JobTracker

  Throws:
    `Exception`

### isTaskMemoryManagerEnabled

    public boolean isTaskMemoryManagerEnabled()

  Is the TaskMemoryManager Enabled on this system?

  Returns:
    true if enabled, false otherwise.

### getTaskMemoryManager

    public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

### getHostname

    public String getHostname()

  Specified by:
    `getHostname` in interface `TaskTrackerMXBean`
  Returns:
    the hostname of the tasktracker

### getVersion

    public String getVersion()

  Specified by:
    `getVersion` in interface `TaskTrackerMXBean`
  Returns:
    the version of the code base

### getConfigVersion

    public String getConfigVersion()

  Specified by:
    `getConfigVersion` in interface `TaskTrackerMXBean`
  Returns:
    the config version (from a config properties)

### getJobTrackerUrl

    public String getJobTrackerUrl()

  Specified by:
    `getJobTrackerUrl` in interface `TaskTrackerMXBean`
  Returns:
    the URL of the jobtracker

### getRpcPort

    public int getRpcPort()

  Specified by:
    `getRpcPort` in interface `TaskTrackerMXBean`
  Returns:
    the RPC port of the tasktracker

### getHttpPort

    public int getHttpPort()

  Specified by:
    `getHttpPort` in interface `TaskTrackerMXBean`
  Returns:
    the HTTP port of the tasktracker

### isHealthy

    public boolean isHealthy()

  Specified by:
    `isHealthy` in interface `TaskTrackerMXBean`
  Returns:
    the health status of the tasktracker

### getTasksInfoJson

    public String getTasksInfoJson()

  Specified by:
    `getTasksInfoJson` in interface `TaskTrackerMXBean`
  Returns:
    a json formatted info about tasks of the tasktracker

### updatePrivateDistributedCacheSizes

    public void updatePrivateDistributedCacheSizes(JobID jobId,
                                                   long[] sizes)
                                            throws IOException

  Description copied from interface: `TaskUmbilicalProtocol`
  The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

  Specified by:
    `updatePrivateDistributedCacheSizes` in interface `TaskUmbilicalProtocol`
  Parameters:
    `jobId` - the job to update
    `sizes` - the array of sizes that were computed
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2009 The Apache Software Foundation