General Hadoop FileSystem Shell Commands

The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS):

syntax:

bin/hadoop fs <args>

For Data Movement between HDFS and Unix

put

Usage: hdfs dfs -put <localsrc> ... <dst>

Copy single src, or multiple srcs from local file system to the destination Hadoop file system.

·         hdfs dfs -put localfile /user/hadoop/hadoopfile

·         hdfs dfs -put localfile1 localfile2 /user/hadoop/hadoopdir

·         hdfs dfs -put localfile hdfs://nn.example.com/hadoop/hadoopfile

get

Usage: hdfs dfs –get <src> <localdst>

Copy Hadoop files to the local file system.

copyFromLocal

Usage: hdfs dfs -copyFromLocal [-f] <localsrc> URI

Similar to put command, except that the source is restricted to a local file reference.

Options:

·         The -f option will overwrite the destination if it already exists.

copyToLocal

Usage: hdfs dfs -copyToLocal URI <localdst>

Similar to get command, except that the destination is restricted to a local file reference.

moveFromLocal

Usage: hdfs dfs -moveFromLocal <localsrc> <dst>

Similar to put command, except that the source localsrc is deleted after it's copied.

moveToLocal

Usage: hdfs dfs -moveToLocal [-crc] <src> <dst>

Similar to get  command, except that the source src is deleted after it's copied.

Other Commands:

cat

Usage: hadoop fs -cat <path[filename]>

See contents of a file.

mv

Usage: hdfs dfs -mv URI [URI ...] <dest>

Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.

Example:

·         hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2

·         hdfs dfs -mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs://nn.example.com/dir1

rm

Usage: hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]

Delete files specified as args.

Options:

·         The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does not exist.

·         The -R option deletes the directory and any content under it recursively.

·         The -r option is equivalent to -R.

·         The -skipTrash option will bypass trash, if enabled, and delete the specified file(s) immediately. This can be useful when it is necessary to delete files from an over-quota directory.

Example:

·         hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir

rmr

Usage: hdfs dfs -rmr [-skipTrash] URI [URI ...]

Recursive version of delete.

count

Usage: hdfs dfs -count [-q] [-h] <paths>

Count the number of directories, files and bytes under the paths that match the specified file pattern.

du

Usage: hdfs dfs -du [-s] [-h] URI [URI ...]

Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.

Options:

·         The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files.

·         The -h option will format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)

expunge

Usage: hdfs dfs -expunge

Empty the Trash.

ls

Usage: hdfs dfs -ls [-R] <args>

Options:

·         The -R option will return stat recursively through the directory structure.

For a file returns stat on the file with the following format:

permissions number_of_replicas userid groupid filesize modification_date modification_time filename

For a directory it returns list of its direct children as in Unix. A directory is listed as:

permissions userid groupid modification_date modification_time dirname

Example:

·         hdfs dfs -ls /user/hadoop/file1

lsr

Usage: hdfs dfs -lsr <args>

Recursive version of ls.

setrep

Usage: hdfs dfs -setrep [-R] [-w] <numReplicas> <path>

Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.

tail

Usage: hdfs dfs -tail [-f] URI

Displays last kilobyte of the file to stdout.

test

Usage: hdfs dfs -test -[ezd] URI

Options:

·         The -e option will check to see if the file exists, returning 0 if true.

·         The -z option will check to see if the file is zero length, returning 0 if true.

·         The -d option will check to see if the path is directory, returning 0 if true.

touchz

Usage: hdfs dfs -touchz URI [URI ...]

Create a file of zero length.

To Learn HDFS ACL find here.