Question

0 0

Spark+Hive的example运行问题

最近做实验需要收集Spark+Hive的一些指令trace，但在运行Spark自带的Scala版Hive样例代码时出问题。

Hive环境我已经配置了，并且试着执行，创建数据文件试着执行了create table和select等语句，测试成功了。然后我将hive-site.xml拷贝到spark/conf目录下，使用命令：


 ./bin/run-example org.apache.spark.examples.sql.hive.HiveFromSpark --driver-class-path $SPARK_HOME/lib/mysql-connector-java-5.1.34-bin.jar

执行后报错如下：


 14/11/27 09:59:12 ERROR metastore.RetryingRawStore: JDO datastore error. Retrying metastore command after 1000 ms (attempt 1 of 1)
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table src
    at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:958)
    at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:905)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:8999)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8313)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
    at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:186)
    at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:250)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:247)
    at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
    at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:37)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
    at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
    ... 23 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
    ... 28 more
Caused by: javax.jdo.JDOException: Exception thrown when executing query
NestedThrowables:
java.sql.SQLSyntaxErrorException: Table/View 'VERSION' does not exist.
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:5693)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:5675)
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5634)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:5622)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
    at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:403)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
    ... 33 more
Caused by: java.sql.SQLSyntaxErrorException: Table/View 'VERSION' does not exist.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
    at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:974)
    at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
    at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:195)
    at org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:626)
    at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
    at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
    at org.datanucleus.store.query.Query.execute(Query.java:1654)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)
    ... 51 more
Caused by: java.sql.SQLException: Table/View 'VERSION' does not exist.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
    ... 72 more
Caused by: ERROR 42X05: Table/View 'VERSION' does not exist.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromList.bindTables(Unknown Source)
    at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(Unknown Source)
    at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(Unknown Source)
    at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown Source)
    at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown Source)
    at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
    at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
    ... 66 more
14/11/27 09:59:14 INFO ql.Driver: Semantic Analysis Completed
14/11/27 09:59:14 INFO ql.Driver: </PERFLOG method=semanticAnalyze start=1417053550072 end=1417053554036 duration=3964>
14/11/27 09:59:14 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
14/11/27 09:59:14 INFO ql.Driver: </PERFLOG method=compile start=1417053550043 end=1417053554042 duration=3999>
14/11/27 09:59:14 INFO ql.Driver: <PERFLOG method=Driver.execute>
14/11/27 09:59:14 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/11/27 09:59:14 INFO ql.Driver: Starting command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING)
14/11/27 09:59:14 INFO ql.Driver: </PERFLOG method=TimeToSubmit start=1417053550043 end=1417053554045 duration=4002>
14/11/27 09:59:14 INFO ql.Driver: <PERFLOG method=runTasks>
14/11/27 09:59:14 INFO ql.Driver: <PERFLOG method=task.DDL.Stage-0>
14/11/27 09:59:14 INFO exec.DDLTask: Default to LazySimpleSerDe for table src
14/11/27 09:59:14 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
14/11/27 09:59:14 INFO metastore.ObjectStore: ObjectStore, initialize called
14/11/27 09:59:14 INFO metastore.ObjectStore: Initialized ObjectStore
14/11/27 09:59:14 ERROR metastore.RetryingRawStore: JDO datastore error. Retrying metastore command after 1000 ms (attempt 1 of 1)
14/11/27 09:59:15 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:602)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3661)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:252)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
    at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:186)
    at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:250)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:247)
    at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
    at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:37)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:596)
    ... 23 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
    ... 28 more
Caused by: javax.jdo.JDOException: Exception thrown when executing query
NestedThrowables:
java.sql.SQLSyntaxErrorException: Table/View 'VERSION' does not exist.
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:5693)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:5675)
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5634)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:5622)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
    at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:403)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
    ... 33 more
Caused by: java.sql.SQLSyntaxErrorException: Table/View 'VERSION' does not exist.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
    at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:974)
    at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
    at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:195)
    at org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:626)
    at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
    at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
    at org.datanucleus.store.query.Query.execute(Query.java:1654)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)
    ... 51 more
Caused by: java.sql.SQLException: Table/View 'VERSION' does not exist.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
    ... 72 more
Caused by: ERROR 42X05: Table/View 'VERSION' does not exist.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(Unknown Source)
    at org.apache.derby.impl.sql.compile.FromList.bindTables(Unknown Source)
    at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(Unknown Source)
    at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(Unknown Source)
    at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown Source)
    at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown Source)
    at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
    at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
    ... 66 more

14/11/27 09:59:15 INFO ql.Driver: </PERFLOG method=task.DDL.Stage-0 start=1417053554045 end=1417053555119 duration=1074>
14/11/27 09:59:15 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
14/11/27 09:59:15 INFO ql.Driver: </PERFLOG method=Driver.execute start=1417053554042 end=1417053555120 duration=1078>
14/11/27 09:59:15 INFO ql.Driver: <PERFLOG method=releaseLocks>
14/11/27 09:59:15 INFO ql.Driver: </PERFLOG method=releaseLocks start=1417053555120 end=1417053555120 duration=0>
14/11/27 09:59:15 INFO ql.Driver: <PERFLOG method=releaseLocks>
14/11/27 09:59:15 INFO ql.Driver: </PERFLOG method=releaseLocks start=1417053555121 end=1417053555121 duration=0>
14/11/27 09:59:15 ERROR hive.LocalHiveContext: 
======================
HIVE FAILURE OUTPUT
======================
set javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/root/spark-1.0.0-bin-hadoop2/metastore;create=true
set hive.metastore.warehouse.dir=/root/spark-1.0.0-bin-hadoop2/warehouse
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

======================
END HIVE FAILURE OUTPUT
======================

Exception in thread "main" org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:190)
    at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:250)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:247)
    at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
    at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:37)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我把输出的log都复制过来了，所以有点长 =。=

我在这里 http://find.searchhub.org/document/4337487133fe5d80#9a32608a1d5a709f 看到了一些解答但是还是很含糊，求大神解答！！

使用 spark-shell --driver-class-path ./lib/mysql-connector-java-5.1.34-bin.jar 启动交互命令行，将scala版example下的sql/hive/HiveFromSpark的代码一行行执行，能够正常执行，但是怎么直接run-example运行就报错呢？

spark hive

10 years, 2 months ago

工口D大魔王

share

工口D大魔王 10 years, 2 months ago

Answer 1

0

好坑，这里有好几个坑，一一说明：

首先，我给hive配的Mysql作为元数据服务器，因此需要对应的jar包，也就是在上面看到的--driver-class-path ./lib/mysql-connector-java-5.1.34-bin.jar 参数。一开始是没有的，这是第二次编辑的时候加上去的。
加了之后仍然报错，也就是上面的错误提示。spark给的默认代码中使用的是 val hiveContext = LocalHiveContext(sc) ，我这里配Hive配的是远程模式，不是本地模式，因此改为 val hiveContext = HiveContext(sc) ,当然也要对应的import HiveContext包了。
修改了源码还不够，需要将修改的源码重新打包编译，然后使用spark-submit 来提交任务执行。具体的方法可以搜网上的。

以上就是上面的问题的解决思路。

answered 10 years, 2 months ago

兄贵D旋律

share

兄贵D旋律 answered 10 years, 2 months ago

Spark+Hive的example运行问题

工口D大魔王

Answers

兄贵D旋律

Your Answer