Spark 通过HIVE ON HBASE表读取数据源，报错：unread block data
具体错误信息示例如下：
18/03/19 20:46:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
18/03/19 20:46:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
问题原因分析：该问题查了好长时间(开始以为是数据源有问题；后来各种查、各种试，发现是缺少jar包，因为是CDH环境，通过增加spark/jars下面的jar即可：
cd /opt/...../CDH/spark/jars/
ln -s /opt/...../CDH/jars/hive-hbase-handler-***.jar hive-hbase-handler***.jar
总结：引起该问题的原因有很多，最好由简入繁、从最简单直接的原因查起，逐一排除，提高排查问题效率。
相关的解决方案:
方法一：spark-submit --jars
根据spark官网，在提交任务的时候指定--jars，用逗号分开。这样做的缺点是每次都要指定jar包，如果jar包少的话可以这么做，但是如果多的话会很麻烦。可以封装一个sh脚本做目录/引用文件扫描也可以。
spark-submit --master yarn-client --jars ***.jar,***.jar(你的jar包，用逗号分隔) myjar.jar
方法二：extraClassPath 提交时在spark-default中设定参数，将所有需要的jar包考到一个文件里，然后在参数中指定该目录就可以了，较上一个方便很多：
spark.executor.extraClassPath=/extlib/*
spark.driver.extraClassPath=/extlib/*
# 修改为自己的目录
方法二：fat-jar
如果你还是觉得第二种麻烦，这种方法是将所有依赖的jar,包括你写的代码全部打包在一起（fat-jar），就是提交的时候比较慢，毕竟大啊。还有可能和系统jar冲突。自己注意解决把。
参考链接：
setting-spark-classpaths-on-ec2-spark-driver-extraclasspath-and-spark-executorSpark任务提交jar包依赖解决方案
参考阅读
您阅读本篇文章共花了：
发表评论