The environmental variable for APPLICATIONWEBPROXY_BASE. Set in ApplicationMaster's environment only. This states that for all non-relative web URLs in the app masters web UI what base should they have.

APP_STATES: enum value to state map. Used for converting state for display.

The environment variable for APPSUBMITTIME. Set in AppMaster environment only

ATTEMPT_STATES: enum value to state map. Used for converting state for display.

The cache file into which container token is written

FINALAPPSTATES: enum value to state map. Used for converting state for display.

Default length (bytes) upto which to pre-fetch block metadata. (10 blocks of default size)

The temporary environmental variable for container log directory. This should be replaced by real container log directory on container launch.

The environment variable for MAXAPPATTEMPTS. Set in AppMaster environment only

NODE_STATES: enum value to state map. Used for converting state for display.

HDFSBlockReader reads one block of data from a datanode.

HDFSBlockWriter writes one block worth of data to a datanode.

The write method returns number of bytes accepted for writing into this block, which may be less than what was requested if block is full.

Packets are sent and acks received as and when data being written exceed packet size limit.

The close operation flushes remaining data in the block and waits for all pending acks.

HDFSClient

A client to the namenode in a HDFS cluster and holds a connection to it. It also stores the folder context for using relative paths in APIs that use the client.

HDFSException

Thrown by HDFS APIs.

HDFSFile

Holds a HDFSClient and path pointing to a file on HDFS

HDFSFileInfo

Stat structure for file/foledr in HDFS

HDFSFileReader

Provides Julia IO APIs for reading HDFS files. Communicates with namenode for file metadata (through HDFSClient) and to datanodes for file data (through HDFSBlockReader)

HDFSFileWriter

Provides Julia IO APIs for writing HDFS files.

Steps:

• get datanodes to write onto from namenode
• break bytes into packets
• write packets into first datanode's data channel (datanode will mirror packets to downstream datanodes)
• when complete, confirm namenode with a blockReceived

On close call NameNode.complete to:

• remove lease lease is a write lock for file modification. no leases are required for reading files. leases are managed by namenode lease is added on file create or append dfs client should start thread to renew leases periodically
• change file from under construction to complete

HDFSProtocol: Hadoop RPC client for namenode protocol.

HadoopDataChannel is the connection to a datanode. Also associated with it is a buffer to help in message preparation.

HadoopDataChannelPool is a connection pool that holds connections to datanodes. Connections are deemed stale after keepalivesecs.

HadoopRpcException is thrown on Rpc interaction errors either with namenode or datanode. Field status contains error code (enum) if received from the connected entity or just ERROR (1) to indicate failure. Though HadoopRpcException is used while communicating with both namenodes and datanodes, SUCCESS and ERROR are coded with the same values in both cases. Other specific error codes need knowledge of the context to interpret.

HadoopRpcProtocol binds a channel and controller with a service protocol implementation. Used by actual service implementations.

PipelinedPacket holds a packet of data while it is sent to datanodes and acknowledged. The data bytes are reset once acks are received successfully to reduce memory usage.

RequestPipeline holds entities while they are requested for from yarn resource manager. Application master thread extracts pending items and requests them from RM, whereupon they are moved on to the requested state.

WriterPipeline holds all packets of data for a block and provides methods to maintain their states. Pipeline failed status is set if any of the acks received is a failure. Field acked_bytes contains the count of bytes successfully sent till an error is encountered.

YarnAMNMProtocol: Hadoop RPC client for Yarn application master to node manager protocol.

YarnAMRMProtocol: Hadoop RPC client for Yarn application master to resource manager protocol.

YarnApp represents one instance of application running on the yarn cluster

YarnAppAttemptStatus wraps the protobuf type for ease of use

YarnAppMaster is a skeleton application master. It provides the generic scafolding methods which can be used to create specific application masters for different purposes.

When initializing a YarnAppMaster instance as a managed app master, the scheduler address is picked up from the environment variable JULIA_YARN_RESOURCEMANAGER_SCHEDULER_ADDRESS. Tokens set by Yarn in the file pointed to by HADOOP_TOKEN_FILE_LOCATION are also read in automatically.

When run as a managed app master, if a function is provided to be executed, then the application master is registered, the function is executed and then the application master is deregistered. This provides a convenient way to run simple Julia applications in a Yarn cluster. E.g.:

using Elly

YarnAppMaster() do
    ...
    # execute Julia code
end

YarnAppStatus wraps the protobuf type for ease of use

YarnClient holds a connection to the Yarn Resource Manager and provides APIs for application clients to interact with Yarn.

YarnClientProtocol: Hadoop RPC client for application client to Yarn resource manager protocol.

YarnContainers holds all containers related to the application. It also holds the allocation and release pipelines that are used by application master for requesting actions from resource manager. Also schedules callbacks as tasks when containers are allocated or terminated.

YarnException

Thrown by Yarn APIs.

The Yarn ClusterManager for Julia.

ugi: UserGroupInformation representing the user to represent (default is current logged in user) yarnhost: yarn host name rmport: yarn resource manager port schedport: yarn scheduler port launchtimeout: number of seconds to wait until all workers are launched (default is 60, provide larger value for more workers) keepconnected: if false, YarnManager will disconnect from the cluster once all workers are removed (default true, use false if you do not plan to add more workers after removing all existing workers)

YarnNode represents a node manager in the yarn cluster and its communication address, resource state and run state.

YarnNodes holds node information as visible to the application master. It also caches connection to node masters. Connection are reused if they are required before a set keepalivesecs time.

Applications that write infrequently/slowly must call renewlease periodically to prevent the namenode from assuming the client from having abandoned the file or some other client from recovering the lease.

Read one packet into inbuff starting from offset. If inbuff has insufficient space, returns the minimum additional space required in inbuff to read the packet as a negative number. Otherwise, returns the number of bytes available in inbuff after reading the packet.