Exception in thread “main” org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testtxt. could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

Issue:

When you upload a file to HDFS, you may encounter following exception.

Exception:

Exception in thread “main” org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testtxt. could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

Reason:

If you are sure the capacity of data node is not exhausted and the communication between master node and data node is working well, then the issue may be caused by url resolving. Specifically, HDFS client cannot connect to datanode, although it can be connected with master node.

Solution:

1. add following configuration on the org.apache.hadoop.conf.Configuration instance:

  1. Configuration conf = new Configuration();
  2. conf.set(“dfs.client.use.datanode.hostname”“true”);

or add the following to hdfs-site.xml

  
    dfs.client.use.datanode.hostname
    true
  

2. confirm your client could ping datanode

E.g. datanode host name: quickstart.cloudera

If ping is failed, then add resolving mapping manually.

1) In Mac OS, use following command to edit hosts file.

[plain] view plain copy

  1. sudo vi /private/etc/hosts

2) add following mapping to the file.

[plain] view plain copy

  1. 127.0.0.1       quickstart.cloudera

Then, try your program again. The issue should be gone.

Leave a Reply

Your email address will not be published. Required fields are marked *