hadoop错误could only be replicated to > 0 nodes, instead of 1


万能的sf 求指教hadoop的这个错误 org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/hadoop/testfiles/testfiles/file1.txt could only be replicated to > 0 nodes, instead of 1 困扰我好久了 任何办法 包括权限检查、顺序启动、hdfs格式化等都试过了 一直不行 反复配置还是不行 不知道大家有没碰到过

hadoop

桐之丶卧所 11 years, 1 month ago

从代码看,可能是楼主没有启动datanode,或者所有datanode都和namenode断连了。楼主可以去namenode的web页面看看。

chooseTarget方法是说从datanode中选择n个存储楼主的文件。

/**
   * Choose target datanodes according to the replication policy.
   * 
   * @throws IOException
   *           if the number of targets < minimum replication.
   * @see BlockPlacementPolicy#chooseTarget(String, int, DatanodeDescriptor,
   *      List, boolean, HashMap, long)
   */
  public DatanodeDescriptor[] chooseTarget(final String src,
      final int numOfReplicas, final DatanodeDescriptor client,
      final HashMap<Node, Node> excludedNodes,
      final long blocksize) throws IOException {
    // choose targets for the new block to be allocated.
    final DatanodeDescriptor targets[] = blockplacement.chooseTarget(src,
        numOfReplicas, client, new ArrayList<DatanodeDescriptor>(), false,
        excludedNodes, blocksize);
    if (targets.length < minReplication) {
      throw new IOException("File " + src + " could only be replicated to "
          + targets.length + " nodes instead of minReplication (="
          + minReplication + ").  There are "
          + getDatanodeManager().getNetworkTopology().getNumOfLeaves()
          + " datanode(s) running and "
          + (excludedNodes == null? "no": excludedNodes.size())
          + " node(s) are excluded in this operation.");
    }
    return targets;
  }
Akashic answered 11 years, 1 month ago

这 是因为重复format,导致后续创建 namespaceID 和 datanode上的ID不一致导致的。可以查看datanode节点上的日志有

2010-07-06 12:41:04,473 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /hadoop-datanode1/data: namenode namespaceID = 1557013239; datanode namespaceID = 603577766

这个问题我做了检查验证,确实存在这样的问题。可以手工修改

namenode

上的类似 /disk1/hdfs/name/current/VERSION 这 样的配置文件(根据配置可能有多处),将

namespaceID

修改为和datanode一致。

#Wed Jul 07 04:01:14 EDT 2010
namespaceID=755662237
cTime=0
storageType=NAME_NODE
layoutVersion=-18

另一种解决方法(最终采用)是完全修改hadoop配置文件,将配置的datanode节点的存储位置修改了,这样再

namenode -format

时可以完全重建整个HDFS。

赏你颗大白兔 answered 10 years, 2 months ago

Your Answer