Hadoop distcp command how to copy files across clusters

This article shows you how to copy files across clusters with the Hadoop distcp command. The content is concise and easy to understand, and it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Hadoop provides Hadoop distcp commands for data replication and copy between different clusters of Hadoop.

The format is: hadoop distcp-pbc hdfs://namenode1/test hdfs://namenode2/test

Distcp copy, only Map, no Reduce.

Usage: distcp OPTIONS [source_path...]

OPTIONS

-append Reuse existing data in target files and append new

Data to them if possible

-async Should distcp execution be blocking

-atomic Commit all changes or none

-bandwidth Specify bandwidth per map in MB

-delete Delete from target, files missing in source

-diff Use snapshot diff report to identify the

Difference between source and target

-f List of files that need to be copied

-filelimit (Deprecated!) Limit number of files copied to