JobTracker (Hadoop 1.2.1 API)
Date: 2013-07-22

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.hadoop.mapred
Class JobTracker

    java.lang.Object
      org.apache.hadoop.mapred.JobTracker

All Implemented Interfaces:
  VersionedProtocol, AdminOperationsProtocol, JobTrackerMXBean, RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol

    public class JobTracker

extends Object
implements RefreshUserMappingsProtocol, RefreshAuthorizationPolicyProtocol, AdminOperationsProtocol, JobTrackerMXBean

JobTracker is the central location for submitting and tracking MR jobs in a network environment.

Nested Class Summary
`static class` | `JobTracker.IllegalStateException` A client tried to submit a job before the Job Tracker was ready.
`static class` | `JobTracker.SafeModeAction` JobTracker SafeMode
`static class` | `JobTracker.State`

Field Summary
`static long` | `COUNTER_UPDATE_INTERVAL`
`static long` | `DEFAULT_DISK_HEALTH_CHECK_INTERVAL` How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
`static long` | `DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT`
`static String` | `DELEGATION_KEY_UPDATE_INTERVAL_KEY`
`static long` | `DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT`
`static String` | `DELEGATION_TOKEN_MAX_LIFETIME_KEY`
`static long` | `DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT`
`static String` | `DELEGATION_TOKEN_RENEW_INTERVAL_KEY`
`static int` | `FILE_NOT_FOUND`
`static String` | `FOR_REDUCE_TASK` The reduce task number for which this map output is being transferred
`static String` | `FROM_MAP_TASK` The map task from which the map output data is being transferred
`static int` | `HEARTBEAT_INTERVAL_MIN`
`static String` | `JT_KEYTAB_FILE`
`static String` | `JT_USER_NAME`
`static org.apache.commons.logging.Log` | `LOG`
`static String` | `MAP_OUTPUT_LENGTH` The custom http header used for the map output length.
`static String` | `MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY`
`static String` | `MAPRED_CLUSTER_MAX_MAP_MEMORY_MB_PROPERTY`
`static String` | `MAPRED_CLUSTER_MAX_REDUCE_MEMORY_MB_PROPERTY`
`static String` | `MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY`
`static String` | `MAX_USER_JOBCONF_SIZE_KEY` the config key for max user jobconf size
`static String` | `RAW_MAP_OUTPUT_LENGTH` The custom http header used for the "raw" map output length.
`static int` | `SUCCESS`
`static int` | `TRACKERS_OK`
`static int` | `UNKNOWN_TASKTRACKER`
`static long` | `versionID` version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with `heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short)` version 4 changed TaskReport for HADOOP-549.
`static long` | `versionID`
`static String` | `WORKDIR`

Fields inherited from interface org.apache.hadoop.security.RefreshUserMappingsProtocol
`versionID`

Fields inherited from interface org.apache.hadoop.security.authorize.RefreshAuthorizationPolicyProtocol
`versionID`

Fields inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol
`versionID`

Method Summary
`Collection<TaskTrackerStatus>` | `activeTaskTrackers()` Get the active task tracker statuses in the cluster
`void` | `addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)` Registers a `JobInProgressListener` for updates from this `TaskTrackerManager`.
`Collection<TaskTrackerStatus>` | `blacklistedTaskTrackers()` Get the statuses of the blacklisted task trackers in the cluster.
`void` | `cancelDelegationToken(Token<DelegationTokenIdentifier> token)` Discard a current delegation token.
`Vector<JobInProgress>` | `completedJobs()`
`Vector<JobInProgress>` | `failedJobs()`
`void` | `failJob(JobInProgress job)` Fail a job and inform the listeners.
`static InetSocketAddress` | `getAddress(Configuration conf)`
`String` | `getAliveNodesInfoJson()`
`JobStatus[]` | `getAllJobs()` Get all the jobs submitted.
`String` | `getAssignedTracker(TaskAttemptID taskId)` Get tracker name for a given task id.
`String` | `getBlacklistedNodesInfoJson()`
`String` | `getBuildVersion()` Returns the VersionInfo build version of the JobTracker
`TaskReport[]` | `getCleanupTaskReports(JobID jobid)` Grab a bunch of info on the cleanup tasks that make up the job
`org.apache.hadoop.mapred.Clock` | `getClock()`
`ClusterMetrics` | `getClusterMetrics()`
`ClusterStatus` | `getClusterStatus()` Deprecated. use `getClusterStatus(boolean)`
`ClusterStatus` | `getClusterStatus(boolean detailed)` Get the current status of the cluster
`JobConf` | `getConf()` Returns a handle to the JobTracker's Configuration
`String` | `getConfigVersion()`
`Token<DelegationTokenIdentifier>` | `getDelegationToken(Text renewer)` Get a new delegation token.
`DelegationTokenSecretManager` | `getDelegationTokenSecretManager()`
`String` | `getFilesystemName()` Grab the local fs name
`String` | `getGraylistedNodesInfoJson()`
`String` | `getHostname()`
`int` | `getInfoPort()`
`JobInProgress` | `getJob(JobID jobid)` Obtain the job object identified by jobid
`Counters` | `getJobCounters(JobID jobid)` Grab the current job counters
`JobProfile` | `getJobProfile(JobID jobid)` Grab a handle to a job that is already known to the JobTracker.
`JobStatus[]` | `getJobsFromQueue(String queue)` Gets all the jobs submitted to the particular Queue
`JobStatus` | `getJobStatus(JobID jobid)` Grab a handle to a job that is already known to the JobTracker.
`String` | `getJobTrackerMachine()`
`static String` | `getLocalJobFilePath(JobID jobId)` Get the localized job file path on the job trackers local file system
`TaskReport[]` | `getMapTaskReports(JobID jobid)` Grab a bunch of info on the map tasks that make up the job
`JobID` | `getNewJobId()` Allocates a new JobId string.
`int` | `getNextHeartbeatInterval()` Calculates next heartbeat interval using cluster size.
`Node` | `getNode(String name)` Return the Node in the network topology that corresponds to the hostname
`Collection<Node>` | `getNodesAtMaxLevel()` Returns a collection of nodes at the max level
`int` | `getNumberOfUniqueHosts()`
`int` | `getNumResolvedTaskTrackers()`
`int` | `getNumTaskCacheLevels()`
`static Node` | `getParentNode(Node node, int level)`
`long` | `getProtocolVersion(String protocol, long clientVersion)` Return protocol version corresponding to protocol interface.
`QueueAclsInfo[]` | `getQueueAclsForCurrentUser()` Gets the Queue ACLs for current user
`AccessControlList` | `getQueueAdmins(String queueName)` Get the administrators of the given job-queue.
`JobQueueInfo` | `getQueueInfo(String queue)` Gets scheduling information associated with the particular Job queue
`String` | `getQueueInfoJson()`
`org.apache.hadoop.mapred.QueueManager` | `getQueueManager()` Return the `QueueManager` associated with the JobTracker.
`JobQueueInfo[]` | `getQueues()` Gets set of Job Queues associated with the Job Tracker
`String` | `getReasonsForBlacklisting(String host)`
`String` | `getReasonsForGraylisting(String host)`
`long` | `getRecoveryDuration()` How long the jobtracker took to recover from restart.
`TaskReport[]` | `getReduceTaskReports(JobID jobid)` Grab a bunch of info on the reduce tasks that make up the job
`List<JobInProgress>` | `getRunningJobs()` Version that is called from a timer thread, and therefore needs to be careful to synchronize.
`TaskReport[]` | `getSetupTaskReports(JobID jobid)` Grab a bunch of info on the setup tasks that make up the job
`String` | `getStagingAreaDir()` Get a hint from the JobTracker where job-specific files are to be placed.
`long` | `getStartTime()`
`String` | `getSummaryJson()`
`String` | `getSystemDir()` Grab the jobtracker system directory path where job-specific files are to be placed.
`TaskCompletionEvent[]` | `getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents)` Get task completion events for the jobid, starting from fromEventId.
`String[]` | `getTaskDiagnostics(TaskAttemptID taskId)` Get the diagnostics for a given task
`TaskTracker` | `getTaskTracker(String trackerID)`
`TaskTrackerStatus` | `getTaskTrackerStatus(String trackerID)`
`int` | `getThreadCount()`
`org.apache.hadoop.mapred.TaskInProgress` | `getTip(TaskID tipid)` Returns specified TaskInProgress, or null.
`int` | `getTotalSubmissions()`
`String` | `getTrackerIdentifier()` Get the unique identifier (ie.
`int` | `getTrackerPort()`
`String` | `getVersion()`
`String` | `getVIVersion()` Returns the VersionInfo version of the JobTracker
`Collection<TaskTrackerStatus>` | `graylistedTaskTrackers()` Get the statuses of the graylisted task trackers in the cluster.
`boolean` | `hasRecovered()` Whether the JT has recovered upon restart
`boolean` | `hasRestarted()` Whether the JT has restarted
`org.apache.hadoop.mapred.HeartbeatResponse` | `heartbeat(TaskTrackerStatus status, boolean restarted, boolean initialContact, boolean acceptNewTasks, short responseId)` The periodic heartbeat mechanism between the `TaskTracker` and the `JobTracker`.
`void` | `initJob(JobInProgress job)` Initialize the Job
`boolean` | `isBlacklisted(String trackerID)` Whether the tracker is blacklisted or not
`boolean` | `isGraylisted(String trackerID)` Whether the tracker is graylisted or not
`boolean` | `isInSafeMode()` Get safe mode.
`boolean` | `isNodeGroupAware()`
`JobStatus[]` | `jobsToComplete()` Get the jobs that are not completed and not failed
`void` | `killJob(JobID jobid)` Kill the indicated job
`boolean` | `killTask(TaskAttemptID taskid, boolean shouldFail)` Kill indicated task attempt.
`static void` | `main(String[] argv)` Start the JobTracker process.
`void` | `offerService()` Run forever
`void` | `refreshNodes()` Rereads the config to get hosts and exclude list file names.
`void` | `refreshQueues()` Refresh the queue acls in use currently.
`void` | `refreshServiceAcl()` Refresh the service-level authorization policy in-effect.
`void` | `refreshSuperUserGroupsConfiguration()` Refresh superuser proxy group list
`void` | `refreshUserToGroupsMappings()` Refresh user to group mappings.
`void` | `removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)` Unregisters a `JobInProgressListener` from this `TaskTrackerManager`.
`long` | `renewDelegationToken(Token<DelegationTokenIdentifier> token)` Renew a delegation token to extend its lifetime.
`void` | `reportTaskTrackerError(String taskTracker, String errorClass, String errorMessage)` Report a problem to the job tracker.
`Node` | `resolveAndAddToTopology(String name)`
`Vector<JobInProgress>` | `runningJobs()`
`void` | `setJobPriority(JobID jobid, String priority)` Set the priority of the specified job
`boolean` | `setSafeMode(JobTracker.SafeModeAction safeModeAction)` Set safe mode for the JobTracker.
`static JobTracker` | `startTracker(JobConf conf)` Start the JobTracker with given configuration.
`static JobTracker` | `startTracker(JobConf conf, String identifier)`
`static JobTracker` | `startTracker(JobConf conf, String identifier, boolean initialize)`
`void` | `stopTracker()`
`JobStatus` | `submitJob(JobID jobId, String jobSubmitDir, Credentials ts)` JobTracker.submitJob() kicks off a new job.
`List<List<String>>` | `taskTrackerNames()` Get the active, blacklisted, and graylisted task tracker names in the cluster.
`Collection<TaskTrackerStatus>` | `taskTrackers()` Get all the task trackers in the cluster

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### MAX_USER_JOBCONF_SIZE_KEY

    public static final String MAX_USER_JOBCONF_SIZE_KEY

  the config key for max user jobconf size

  See Also:
    Constant Field Values

### DELEGATION_KEY_UPDATE_INTERVAL_KEY

    public static final String DELEGATION_KEY_UPDATE_INTERVAL_KEY

See Also:
  Constant Field Values

### DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT

    public static final long DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT

See Also:
  Constant Field Values

### DELEGATION_TOKEN_RENEW_INTERVAL_KEY

    public static final String DELEGATION_TOKEN_RENEW_INTERVAL_KEY

See Also:
  Constant Field Values

### DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT

    public static final long DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT

See Also:
  Constant Field Values

### DELEGATION_TOKEN_MAX_LIFETIME_KEY

    public static final String DELEGATION_TOKEN_MAX_LIFETIME_KEY

See Also:
  Constant Field Values

### DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT

    public static final long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT

See Also:
  Constant Field Values

### LOG

    public static final org.apache.commons.logging.Log LOG

### JT_USER_NAME

    public static final String JT_USER_NAME

See Also:
  Constant Field Values

### JT_KEYTAB_FILE

    public static final String JT_KEYTAB_FILE

See Also:
  Constant Field Values

### MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY

    public static final String MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY

See Also:
  Constant Field Values

### MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY

    public static final String MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY

See Also:
  Constant Field Values

### MAPRED_CLUSTER_MAX_MAP_MEMORY_MB_PROPERTY

    public static final String MAPRED_CLUSTER_MAX_MAP_MEMORY_MB_PROPERTY

See Also:
  Constant Field Values

### MAPRED_CLUSTER_MAX_REDUCE_MEMORY_MB_PROPERTY

    public static final String MAPRED_CLUSTER_MAX_REDUCE_MEMORY_MB_PROPERTY

See Also:
  Constant Field Values

### HEARTBEAT_INTERVAL_MIN

    public static final int HEARTBEAT_INTERVAL_MIN

See Also:
  Constant Field Values

### COUNTER_UPDATE_INTERVAL

    public static final long COUNTER_UPDATE_INTERVAL

See Also:
  Constant Field Values

### DEFAULT_DISK_HEALTH_CHECK_INTERVAL

    public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL

  How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

  See Also:
    Constant Field Values

### SUCCESS

    public static final int SUCCESS

See Also:
  Constant Field Values

### FILE_NOT_FOUND

    public static final int FILE_NOT_FOUND

See Also:
  Constant Field Values

### MAP_OUTPUT_LENGTH

    public static final String MAP_OUTPUT_LENGTH

  The custom http header used for the map output length.

  See Also:
    Constant Field Values

### RAW_MAP_OUTPUT_LENGTH

    public static final String RAW_MAP_OUTPUT_LENGTH

  The custom http header used for the "raw" map output length.

  See Also:
    Constant Field Values

### FROM_MAP_TASK

    public static final String FROM_MAP_TASK

  The map task from which the map output data is being transferred

  See Also:
    Constant Field Values

### FOR_REDUCE_TASK

    public static final String FOR_REDUCE_TASK

  The reduce task number for which this map output is being transferred

  See Also:
    Constant Field Values

### WORKDIR

    public static final String WORKDIR

See Also:
  Constant Field Values

### versionID

    public static final long versionID

  version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with `heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short)` version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208 version 11 changes string to JobID in getTaskCompletionEvents(). version 12 changes the counters representation for HADOOP-1915 version 13 added call getBuildVersion() for HADOOP-236 Version 14: replaced getFilesystemName with getSystemDir for HADOOP-3135 Version 15: Changed format of Task and TaskStatus for HADOOP-153 Version 16: adds ResourceStatus to TaskTrackerStatus for HADOOP-3759 Version 17: Changed format of Task and TaskStatus for HADOOP-3150 Version 18: Changed status message due to changes in TaskStatus Version 19: Changed heartbeat to piggyback JobTracker restart information so that the TaskTracker can synchronize itself. Version 20: Changed status message due to changes in TaskStatus (HADOOP-4232) Version 21: Changed information reported in TaskTrackerStatus' ResourceStatus and the corresponding accessor methods (HADOOP-4035) Version 22: Replaced parameter 'initialContact' with 'restarted' in heartbeat method (HADOOP-4305) Version 23: Added parameter 'initialContact' again in heartbeat method (HADOOP-4869) Version 24: Changed format of Task and TaskStatus for HADOOP-4759 Version 25: JobIDs are passed in response to JobTracker restart Version 26: Added numRequiredSlots to TaskStatus for MAPREDUCE-516 Version 27: Adding node health status to TaskStatus for MAPREDUCE-211 Version 28: Adding user name to the serialized Task for use by TT. Version 29: Adding available memory and CPU usage information on TT to TaskTrackerStatus for MAPREDUCE-1218 Version 30: Adding disk failure to TaskTrackerStatus for MAPREDUCE-3015 Version 31: Adding version methods for HADOOP-8209

  See Also:
    Constant Field Values

### TRACKERS_OK

    public static final int TRACKERS_OK

See Also:
  Constant Field Values

### UNKNOWN_TASKTRACKER

    public static final int UNKNOWN_TASKTRACKER

See Also:
  Constant Field Values

### versionID

    public static final long versionID

See Also:
  Constant Field Values

Method Detail

### getClock

    public org.apache.hadoop.mapred.Clock getClock()

### startTracker

    public static JobTracker startTracker(JobConf conf)
                                   throws IOException,
                                          InterruptedException

  Start the JobTracker with given configuration. The conf will be modified to reflect the actual ports on which the JobTracker is up and running if the user passes the port as `zero`.

  Parameters:
    `conf` - configuration for the JobTracker.
  Throws:
    `IOException`
    `InterruptedException`

### startTracker

    public static JobTracker startTracker(JobConf conf,
                                          String identifier)
                                   throws IOException,
                                          InterruptedException

  Throws:
    `IOException`
    `InterruptedException`

### startTracker

    public static JobTracker startTracker(JobConf conf,
                                          String identifier,
                                          boolean initialize)
                                   throws IOException,
                                          InterruptedException

  Throws:
    `IOException`
    `InterruptedException`

### stopTracker

    public void stopTracker()
                     throws IOException

  Throws:
    `IOException`

### getProtocolVersion

    public long getProtocolVersion(String protocol,
                                   long clientVersion)
                            throws IOException

  Description copied from interface: `VersionedProtocol`
  Return protocol version corresponding to protocol interface.

  Specified by:
    `getProtocolVersion` in interface `VersionedProtocol`
  Parameters:
    `protocol` - The classname of the protocol interface
    `clientVersion` - The version of the protocol that the client speaks
  Returns:
    the version that the server will speak
  Throws:
    `IOException`

### getDelegationTokenSecretManager

    public DelegationTokenSecretManager getDelegationTokenSecretManager()

### hasRestarted

    public boolean hasRestarted()

  Whether the JT has restarted

### hasRecovered

    public boolean hasRecovered()

  Whether the JT has recovered upon restart

### getRecoveryDuration

    public long getRecoveryDuration()

  How long the jobtracker took to recover from restart.

### getAddress

    public static InetSocketAddress getAddress(Configuration conf)

### offerService

    public void offerService()
                      throws InterruptedException,
                             IOException

  Run forever

  Throws:
    `InterruptedException`
    `IOException`

### getTotalSubmissions

    public int getTotalSubmissions()

### getJobTrackerMachine

    public String getJobTrackerMachine()

### getTrackerIdentifier

    public String getTrackerIdentifier()

  Get the unique identifier (ie. timestamp) of this job tracker start.

  Returns:
    a string with a unique identifier

### getTrackerPort

    public int getTrackerPort()

### getInfoPort

    public int getInfoPort()

### getStartTime

    public long getStartTime()

### runningJobs

    public Vector<JobInProgress> runningJobs()

### getRunningJobs

    public List<JobInProgress> getRunningJobs()

  Version that is called from a timer thread, and therefore needs to be careful to synchronize.

### failedJobs

    public Vector<JobInProgress> failedJobs()

### completedJobs

    public Vector<JobInProgress> completedJobs()

### taskTrackers

    public Collection<TaskTrackerStatus> taskTrackers()

  Get all the task trackers in the cluster

  Returns:
    `Collection` of `TaskTrackerStatus`

### activeTaskTrackers

    public Collection<TaskTrackerStatus> activeTaskTrackers()

  Get the active task tracker statuses in the cluster

  Returns:
    `Collection` of active `TaskTrackerStatus`

### taskTrackerNames

    public List<List<String>> taskTrackerNames()

  Get the active, blacklisted, and graylisted task tracker names in the cluster. The first element in the returned list contains the list of active tracker names; the second element in the returned list contains the list of blacklisted tracker names; and the third contains the list of graylisted tracker names. Note that the blacklist is disjoint from the active list, but the graylist is not: initially, graylisted trackers are still active and therefore will appear in both lists. (Graylisted trackers can later be blacklisted, in which case they'll be removed from the active list and added to the blacklist, but they remain on the graylist in this case. Blacklisting comes about via the health-check script, while graylisting is heuristically based on the number of per-job blacklistings in a specified time interval.)

### blacklistedTaskTrackers

    public Collection<TaskTrackerStatus> blacklistedTaskTrackers()

  Get the statuses of the blacklisted task trackers in the cluster.

  Returns:
    `Collection` of blacklisted `TaskTrackerStatus`

### graylistedTaskTrackers

    public Collection<TaskTrackerStatus> graylistedTaskTrackers()

  Get the statuses of the graylisted task trackers in the cluster.

  Returns:
    `Collection` of graylisted `TaskTrackerStatus`

### isBlacklisted

    public boolean isBlacklisted(String trackerID)

  Whether the tracker is blacklisted or not

  Parameters:
    `trackerID` -
  Returns:
    true if blacklisted, false otherwise

### isGraylisted

    public boolean isGraylisted(String trackerID)

  Whether the tracker is graylisted or not

  Parameters:
    `trackerID` -
  Returns:
    true if graylisted, false otherwise

### getTaskTrackerStatus

    public TaskTrackerStatus getTaskTrackerStatus(String trackerID)

### getTaskTracker

    public TaskTracker getTaskTracker(String trackerID)

### resolveAndAddToTopology

    public Node resolveAndAddToTopology(String name)
                                 throws UnknownHostException

  Throws:
    `UnknownHostException`

### getNodesAtMaxLevel

    public Collection<Node> getNodesAtMaxLevel()

  Returns a collection of nodes at the max level

### getParentNode

    public static Node getParentNode(Node node,
                                     int level)

### getNode

    public Node getNode(String name)

  Return the Node in the network topology that corresponds to the hostname

### getNumTaskCacheLevels

    public int getNumTaskCacheLevels()

### getNumResolvedTaskTrackers

    public int getNumResolvedTaskTrackers()

### getNumberOfUniqueHosts

    public int getNumberOfUniqueHosts()

  Returns:
    The number of unique hosts running tasktrackers.

### isNodeGroupAware

    public boolean isNodeGroupAware()

### addJobInProgressListener

    public void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

  Registers a `JobInProgressListener` for updates from this `TaskTrackerManager`.

### removeJobInProgressListener

    public void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)

  Unregisters a `JobInProgressListener` from this `TaskTrackerManager`.

### getQueueManager

    public org.apache.hadoop.mapred.QueueManager getQueueManager()

  Return the `QueueManager` associated with the JobTracker.

  Returns:
    the `QueueManager`

### getVIVersion

    public String getVIVersion()
                        throws IOException

  Returns the VersionInfo version of the JobTracker

  Throws:
    `IOException`

### getBuildVersion

    public String getBuildVersion()
                           throws IOException

  Returns the VersionInfo build version of the JobTracker

  Throws:
    `IOException`

### heartbeat

    public org.apache.hadoop.mapred.HeartbeatResponse heartbeat(TaskTrackerStatus status,
                                                                boolean restarted,
                                                                boolean initialContact,
                                                                boolean acceptNewTasks,
                                                                short responseId)
                                                         throws IOException

  The periodic heartbeat mechanism between the `TaskTracker` and the `JobTracker`. The `JobTracker` processes the status information sent by the `TaskTracker` and responds with instructions to start/stop tasks or jobs, and also 'reset' instructions during contingencies.

  Parameters:
    `status` - the status update
    `restarted` - `true` if the process has just started or restarted, `false` otherwise
    `initialContact` - `true` if this is first interaction since 'refresh', `false` otherwise.
    `acceptNewTasks` - `true` if the `TaskTracker` is ready to accept new tasks to run.
    `responseId` - the last responseId successfully acted upon by the `TaskTracker`.
  Returns:
    a `HeartbeatResponse` with fresh instructions.
  Throws:
    `IOException`

### getNextHeartbeatInterval

    public int getNextHeartbeatInterval()

  Calculates next heartbeat interval using cluster size. Heartbeat interval is incremented by 1 second for every 100 nodes by default.

  Returns:
    next heartbeat interval.

### getFilesystemName

    public String getFilesystemName()
                             throws IOException

  Grab the local fs name

  Throws:
    `IOException`

### getConf

    public JobConf getConf()

  Returns a handle to the JobTracker's Configuration

### reportTaskTrackerError

    public void reportTaskTrackerError(String taskTracker,
                                       String errorClass,
                                       String errorMessage)
                                throws IOException

  Report a problem to the job tracker.

  Parameters:
    `taskTracker` - the name of the task tracker
    `errorClass` - the kind of error (eg. the class that was thrown)
    `errorMessage` - the human readable error message
  Throws:
    `IOException` - if there was a problem in communication or on the remote side

### getNewJobId

    public JobID getNewJobId()
                      throws IOException

  Allocates a new JobId string.

  Returns:
    a unique job name for submitting jobs.
  Throws:
    `IOException`

### submitJob

    public JobStatus submitJob(JobID jobId,
                               String jobSubmitDir,
                               Credentials ts)
                        throws IOException

  JobTracker.submitJob() kicks off a new job. Create a 'JobInProgress' object, which contains both JobProfile and JobStatus. Those two sub-objects are sometimes shipped outside of the JobTracker. But JobInProgress adds info that's useful for the JobTracker alone.

  Throws:
    `IOException`

### getStagingAreaDir

    public String getStagingAreaDir()
                             throws IOException

  Get a hint from the JobTracker where job-specific files are to be placed.

  Returns:
    the directory where job-specific files are to be placed.
  Throws:
    `IOException`
  See Also:
    `JobSubmissionProtocol.getStagingAreaDir()`

### getClusterStatus

    @Deprecated
    public ClusterStatus getClusterStatus()

  Deprecated. use `getClusterStatus(boolean)`

  Returns:
    a summary of the cluster's status.

### getClusterStatus

    public ClusterStatus getClusterStatus(boolean detailed)

  Get the current status of the cluster

  Parameters:
    `detailed` - if true then report tracker names and memory usage
  Returns:
    summary of the state of the cluster

### getClusterMetrics

    public ClusterMetrics getClusterMetrics()

### killJob

    public void killJob(JobID jobid)
                 throws IOException

  Kill the indicated job

  Throws:
    `IOException`
  See Also:
    `JobSubmissionProtocol.killJob(org.apache.hadoop.mapred.JobID)`

### cancelDelegationToken

    public void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
                               throws IOException,
                                      InterruptedException

  Discard a current delegation token.

  Parameters:
    `token` - the token to cancel
  Throws:
    `IOException`
    `InterruptedException`

### getDelegationToken

    public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                        throws IOException,
                                                               InterruptedException

  Get a new delegation token.

  Parameters:
    `renewer` - the user other than the creator (if any) that can renew the token
  Returns:
    the new delegation token
  Throws:
    `IOException`
    `InterruptedException`

### renewDelegationToken

    public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
                              throws IOException,
                                     InterruptedException

  Renew a delegation token to extend its lifetime.

  Parameters:
    `token` - the token to renew
  Returns:
    the new expiration time
  Throws:
    `IOException`
    `InterruptedException`

### initJob

    public void initJob(JobInProgress job)

  Initialize the Job

  Parameters:
    `job` - JobInProgress object

### failJob

    public void failJob(JobInProgress job)

  Fail a job and inform the listeners. Other components in the framework should use this to fail a job.

  Parameters:
    `job` - JobInProgress object

### setJobPriority

    public void setJobPriority(JobID jobid,
                               String priority)
                        throws IOException

  Set the priority of the specified job

  Parameters:
    `jobid` - ID of the job
    `priority` - Priority to be set for the job
  Throws:
    `IOException`

### getJobProfile

    public JobProfile getJobProfile(JobID jobid)
                             throws IOException

  Grab a handle to a job that is already known to the JobTracker.

  Returns:
    Profile of the job, or null if not found.
  Throws:
    `IOException`

### getJobStatus

    public JobStatus getJobStatus(JobID jobid)
                           throws IOException

  Grab a handle to a job that is already known to the JobTracker.

  Returns:
    Status of the job, or null if not found.
  Throws:
    `IOException`

### getJobCounters

    public Counters getJobCounters(JobID jobid)
                            throws IOException

  Grab the current job counters

  Throws:
    `IOException`

### getMapTaskReports

    public TaskReport[] getMapTaskReports(JobID jobid)
                                   throws IOException

  Grab a bunch of info on the map tasks that make up the job

  Throws:
    `IOException`

### getReduceTaskReports

    public TaskReport[] getReduceTaskReports(JobID jobid)
                                      throws IOException

  Grab a bunch of info on the reduce tasks that make up the job

  Throws:
    `IOException`

### getCleanupTaskReports

    public TaskReport[] getCleanupTaskReports(JobID jobid)
                                       throws IOException

  Grab a bunch of info on the cleanup tasks that make up the job

  Throws:
    `IOException`

### getSetupTaskReports

    public TaskReport[] getSetupTaskReports(JobID jobid)
                                     throws IOException

  Grab a bunch of info on the setup tasks that make up the job

  Throws:
    `IOException`

### getTaskCompletionEvents

    public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                                         int fromEventId,
                                                         int maxEvents)
                                                  throws IOException

  Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

  Parameters:
    `jobid` - job id
    `fromEventId` - event id to start from.
    `maxEvents` - the max number of events we want to look at
  Returns:
    array of task completion events.
  Throws:
    `IOException`

### getTaskDiagnostics

    public String[] getTaskDiagnostics(TaskAttemptID taskId)
                                throws IOException

  Get the diagnostics for a given task

  Parameters:
    `taskId` - the id of the task
  Returns:
    an array of the diagnostic messages
  Throws:
    `IOException`

### getTip

    public org.apache.hadoop.mapred.TaskInProgress getTip(TaskID tipid)

  Returns specified TaskInProgress, or null.

### killTask

    public boolean killTask(TaskAttemptID taskid,
                            boolean shouldFail)
                     throws IOException

  Kill indicated task attempt.

  Parameters:
    `taskid` - the id of the task to kill.
    `shouldFail` - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.
  Returns:
    true if the task was found and successfully marked to kill
  Throws:
    `IOException`
  See Also:
    `JobSubmissionProtocol.killTask(TaskAttemptID, boolean)`

### getAssignedTracker

    public String getAssignedTracker(TaskAttemptID taskId)

  Get tracker name for a given task id.

  Parameters:
    `taskId` - the name of the task
  Returns:
    The name of the task tracker

### jobsToComplete

    public JobStatus[] jobsToComplete()

  Get the jobs that are not completed and not failed

  Returns:
    array of JobStatus for the running/to-be-run jobs.

### getAllJobs

    public JobStatus[] getAllJobs()

  Get all the jobs submitted.

  Returns:
    array of JobStatus for the submitted jobs
  See Also:
    `JobSubmissionProtocol.getAllJobs()`

### getSystemDir

    public String getSystemDir()

  Grab the jobtracker system directory path where job-specific files are to be placed.

  Returns:
    the system directory where job-specific files are to be placed.
  See Also:
    `JobSubmissionProtocol.getSystemDir()`

### getQueueAdmins

    public AccessControlList getQueueAdmins(String queueName)
                                     throws IOException

  Get the administrators of the given job-queue. This method is for hadoop internal use only.

  Returns:
    Queue administrators ACL for the queue to which job is submitted to
  Throws:
    `IOException`
  See Also:
    `JobSubmissionProtocol.getQueueAdmins(String)`

### getJob

    public JobInProgress getJob(JobID jobid)

  Obtain the job object identified by jobid

  Returns:
    jobInProgress object

### refreshNodes

    public void refreshNodes()
                      throws IOException

  Rereads the config to get hosts and exclude list file names. Rereads the files to update the hosts and exclude lists.

  Specified by:
    `refreshNodes` in interface `AdminOperationsProtocol`
  Throws:
    `IOException`

### getLocalJobFilePath

    public static String getLocalJobFilePath(JobID jobId)

  Get the localized job file path on the job trackers local file system

  Parameters:
    `jobId` - id of the job
  Returns:
    the path of the job conf file on the local file system

### main

    public static void main(String[] argv)
                     throws IOException,
                            InterruptedException

  Start the JobTracker process. This is used only for debugging. As a rule, JobTracker should be run as part of the DFS Namenode process.

  Throws:
    `IOException`
    `InterruptedException`

### getQueues

    public JobQueueInfo[] getQueues()
                             throws IOException

  Gets set of Job Queues associated with the Job Tracker

  Returns:
    Array of the Job Queue Information Object
  Throws:
    `IOException`

### getQueueInfo

    public JobQueueInfo getQueueInfo(String queue)
                              throws IOException

  Gets scheduling information associated with the particular Job queue

  Parameters:
    `queue` - Queue Name
  Returns:
    Scheduling Information of the Queue
  Throws:
    `IOException`

### getJobsFromQueue

    public JobStatus[] getJobsFromQueue(String queue)
                                 throws IOException

  Gets all the jobs submitted to the particular Queue

  Parameters:
    `queue` - Queue name
  Returns:
    array of JobStatus for the submitted jobs
  Throws:
    `IOException`

### getQueueAclsForCurrentUser

    public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                               throws IOException

  Gets the Queue ACLs for current user

  Returns:
    array of QueueAclsInfo object for current user.
  Throws:
    `IOException`

### refreshServiceAcl

    public void refreshServiceAcl()
                           throws IOException

  Description copied from interface: `RefreshAuthorizationPolicyProtocol`
  Refresh the service-level authorization policy in-effect.

  Specified by:
    `refreshServiceAcl` in interface `RefreshAuthorizationPolicyProtocol`
  Throws:
    `IOException`

### refreshSuperUserGroupsConfiguration

    public void refreshSuperUserGroupsConfiguration()

  Description copied from interface: `RefreshUserMappingsProtocol`
  Refresh superuser proxy group list

  Specified by:
    `refreshSuperUserGroupsConfiguration` in interface `RefreshUserMappingsProtocol`

### refreshUserToGroupsMappings

    public void refreshUserToGroupsMappings()
                                     throws IOException

  Description copied from interface: `RefreshUserMappingsProtocol`
  Refresh user to group mappings.

  Specified by:
    `refreshUserToGroupsMappings` in interface `RefreshUserMappingsProtocol`
  Throws:
    `IOException`

### refreshQueues

    public void refreshQueues()
                       throws IOException

  Description copied from interface: `AdminOperationsProtocol`
  Refresh the queue acls in use currently. Refresh the queues used by the jobtracker and scheduler. Access control lists and queue states are refreshed.

  Specified by:
    `refreshQueues` in interface `AdminOperationsProtocol`
  Throws:
    `IOException`

### getReasonsForBlacklisting

    public String getReasonsForBlacklisting(String host)

### getReasonsForGraylisting

    public String getReasonsForGraylisting(String host)

### getHostname

    public String getHostname()

  Specified by:
    `getHostname` in interface `JobTrackerMXBean`
  Returns:
    hostname of the jobtracker

### getVersion

    public String getVersion()

  Specified by:
    `getVersion` in interface `JobTrackerMXBean`
  Returns:
    version of the code base

### getConfigVersion

    public String getConfigVersion()

  Specified by:
    `getConfigVersion` in interface `JobTrackerMXBean`
  Returns:
    the config version (from a config property)

### getThreadCount

    public int getThreadCount()

  Specified by:
    `getThreadCount` in interface `JobTrackerMXBean`
  Returns:
    number of threads of the jobtracker jvm

### getSummaryJson

    public String getSummaryJson()

  Specified by:
    `getSummaryJson` in interface `JobTrackerMXBean`
  Returns:
    the summary info in json

### getAliveNodesInfoJson

    public String getAliveNodesInfoJson()

  Specified by:
    `getAliveNodesInfoJson` in interface `JobTrackerMXBean`
  Returns:
    the alive nodes info in json

### getBlacklistedNodesInfoJson

    public String getBlacklistedNodesInfoJson()

  Specified by:
    `getBlacklistedNodesInfoJson` in interface `JobTrackerMXBean`
  Returns:
    the blacklisted nodes info in json

### getGraylistedNodesInfoJson

    public String getGraylistedNodesInfoJson()

  Specified by:
    `getGraylistedNodesInfoJson` in interface `JobTrackerMXBean`
  Returns:
    the graylisted nodes info in json

### getQueueInfoJson

    public String getQueueInfoJson()

  Specified by:
    `getQueueInfoJson` in interface `JobTrackerMXBean`
  Returns:
    the queue info json

### setSafeMode

    public boolean setSafeMode(JobTracker.SafeModeAction safeModeAction)
                        throws IOException

  Description copied from interface: `AdminOperationsProtocol`
  Set safe mode for the JobTracker.

  Specified by:
    `setSafeMode` in interface `AdminOperationsProtocol`
  Parameters:
    `safeModeAction` - safe mode action
  Returns:
    current safemode
  Throws:
    `IOException`

### isInSafeMode

    public boolean isInSafeMode()

  Get safe mode.

  Returns:

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2009 The Apache Software Foundation