hadoop错误could only be replicated to > 0 nodes, instead of 1
万能的sf 求指教hadoop的这个错误 org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/hadoop/testfiles/testfiles/file1.txt could only be replicated to > 0 nodes, instead of 1 困扰我好久了 任何办法 包括权限检查、顺序启动、hdfs格式化等都试过了 一直不行 反复配置还是不行 不知道大家有没碰到过
桐之丶卧所
11 years, 1 month ago
Answers
从代码看,可能是楼主没有启动datanode,或者所有datanode都和namenode断连了。楼主可以去namenode的web页面看看。
chooseTarget方法是说从datanode中选择n个存储楼主的文件。
/** * Choose target datanodes according to the replication policy. * * @throws IOException * if the number of targets < minimum replication. * @see BlockPlacementPolicy#chooseTarget(String, int, DatanodeDescriptor, * List, boolean, HashMap, long) */ public DatanodeDescriptor[] chooseTarget(final String src, final int numOfReplicas, final DatanodeDescriptor client, final HashMap<Node, Node> excludedNodes, final long blocksize) throws IOException { // choose targets for the new block to be allocated. final DatanodeDescriptor targets[] = blockplacement.chooseTarget(src, numOfReplicas, client, new ArrayList<DatanodeDescriptor>(), false, excludedNodes, blocksize); if (targets.length < minReplication) { throw new IOException("File " + src + " could only be replicated to " + targets.length + " nodes instead of minReplication (=" + minReplication + "). There are " + getDatanodeManager().getNetworkTopology().getNumOfLeaves() + " datanode(s) running and " + (excludedNodes == null? "no": excludedNodes.size()) + " node(s) are excluded in this operation."); } return targets; }
Akashic
answered 11 years, 1 month ago
这 是因为重复format,导致后续创建 namespaceID 和 datanode上的ID不一致导致的。可以查看datanode节点上的日志有
2010-07-06 12:41:04,473 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /hadoop-datanode1/data: namenode namespaceID = 1557013239; datanode namespaceID = 603577766
这个问题我做了检查验证,确实存在这样的问题。可以手工修改
namenode
上的类似 /disk1/hdfs/name/current/VERSION 这 样的配置文件(根据配置可能有多处),将
namespaceID
修改为和datanode一致。
#Wed Jul 07 04:01:14 EDT 2010
namespaceID=755662237
cTime=0
storageType=NAME_NODE
layoutVersion=-18
另一种解决方法(最终采用)是完全修改hadoop配置文件,将配置的datanode节点的存储位置修改了,这样再
namenode -format
时可以完全重建整个HDFS。
赏你颗大白兔
answered 10 years, 2 months ago