Hadoop Commands Cheat Sheet

Nixon Data Hadoop Commands Cheat Sheet

Hadoop Commands Cheat Sheet

Apache Hadoop is a framework for distributed storage and processing of large datasets on commodity hardware. Here are some important Hadoop commands that you may find useful:

  1. hadoop fs -ls <path>: This command lists the files and directories in the specified path.
  2. hadoop fs -mkdir <path>: This command creates a new directory in the specified path.
  3. hadoop fs -rm <path>: This command deletes the specified file or directory.
  4. hadoop fs -put <local file> <hdfs file>: This command copies a file from the local file system to HDFS (Hadoop Distributed File System).
  5. hadoop fs -get <hdfs file> <local file>: This command copies a file from HDFS to the local file system.
  6. hadoop fs -mv <src> <dst>: This command moves a file or directory from the source path to the destination path.
  7. hadoop jar <jar file> <main class> <arguments>: This command runs a Java application packaged in a JAR file on a Hadoop cluster.
  8. yarn application -list: This command lists all the applications that are currently running or have completed on the YARN (Yet Another Resource Negotiator) cluster.
  9. yarn node -list: This command lists all the nodes (machines) that are currently registered with the YARN cluster.

These are just a few examples of Hadoop commands. There are many other commands available for managing and monitoring a Hadoop cluster.

  1. hdfs dfs -ls: This command is used to list the files and directories in HDFS. Example: hdfs dfs -ls /
  2. hdfs dfs -mkdir: This command is used to create a directory in HDFS. Example: hdfs dfs -mkdir /data
  3. hdfs dfs -put: This command is used to copy a file from the local file system to HDFS. Example: hdfs dfs -put /local/file.txt /data/file.txt
  4. hdfs dfs -get: This command is used to copy a file from HDFS to the local file system. Example: hdfs dfs -get /data/file.txt /local/file.txt
  5. hdfs dfs -cp: This command is used to copy a file or directory within HDFS. Example: hdfs dfs -cp /data/file.txt /data/copy.txt
  6. hdfs dfs -mv: This command is used to move a file or directory within HDFS. Example: hdfs dfs -mv /data/file.txt /data/newfile.txt
  7. hdfs dfs -rm: This command is used to delete a file from HDFS. Example: hdfs dfs -rm /data/file.txt
  8. hdfs dfs -rmdir: This command is used to delete an empty directory from HDFS. Example: hdfs dfs -rmdir /data/empty
  9. hdfs dfs -rmr: This command is used to recursively delete a directory and all its contents from HDFS. Example: hdfs dfs -rmr /data/dir
  10. hdfs dfs -du: This command is used to display the size of a file or directory in HDFS. Example: hdfs dfs -du /data/file.txt
  11. hdfs dfs -cat: This command is used to display the contents of a file in HDFS. Example: hdfs dfs -cat /data/file.txt
  12. hdfs dfs -tail: This command is used to display the last few lines of a file in HDFS. Example: hdfs dfs -tail /data/file.txt
  13. hdfs dfs -head: This command is used to display the first few lines of a file in HDFS. Example: hdfs dfs -head /data/file.txt
  14. hdfs dfs -count: This command is used to count the number of files and directories in HDFS. Example: hdfs dfs -count /data
  15. hdfs dfs -chmod: This command is used to change the permissions of a file or directory in HDFS. Example: hdfs dfs -chmod 755 /data/file.txt
  16. hdfs dfs -chown: This command is used to change the owner of a file or directory in HDFS. Example: hdfs dfs -chown user:group /data/file.txt
  17. hdfs dfs -getmerge: This command is used to merge multiple files in HDFS into a single file in the local file system. Example: hdfs dfs -getmerge /data/files /local/merged.txt
  18. hdfs dfs -df: This command is used to display the amount of disk space used and available in HDFS. Example: hdfs dfs -df /
  19. hdfs dfs -expunge: This command is used to empty the trash in HDFS. Example: hdfs dfs -expunge
  20. hdfs dfs -copyFromLocal: This command is used to copy a file from the local file system to HDFS, equivalent to the “-put” command. Example: hdfs dfs -copyFromLocal /local/file.txt /data/file.txt
  21. hdfs dfs -copyToLocal: This command is used to copy a file from HDFS to the local file system, equivalent to the “-get” command. Example: hdfs dfs -copyToLocal /data/file.txt /local/file.txt
  22. hdfs dfs -setrep: This command is used to set the replication factor of a file in HDFS. Example: hdfs dfs -setrep 3 /data/file.txt
  23. hdfs dfs -text: This command is used to display the contents of a file in HDFS in text format. Example: hdfs dfs -text /data/file.txt
  24. hdfs dfs -stat: This command is used to display the status of a file in HDFS. Example: hdfs dfs -stat /data/file.txt
  25. hdfs dfs -tail: This command is used to display the last few lines of a file in HDFS. Example: hdfs dfs -tail /data/file.txt
  26. hdfs dfs -touchz: This command is used to create an empty file in HDFS. Example: hdfs dfs -touchz /data/empty.txt
  27. hdfs dfs -usage: This command is used to display the usage information of a file or directory in HDFS. Example: hdfs dfs -usage /data/file.txt
  28. hdfs fsck: This command is used to check the health of the file system and detect any issues. Example: hdfs fsck /
  29. hdfs balancer: This command is used to balance the data across nodes in the cluster. Example: hdfs balancer
  30. hdfs dfsadmin: This command is used to perform administrative operations on HDFS. Example: hdfs dfsadmin -report
  31. hdfs oiv: This command is used to perform offline image viewer operations. Example: hdfs oiv -i /image -o /local/output
  32. hdfs version: This command is used to display the version of HDFS. Example: hdfs version