如何解决pyspark 从 hbase 读取数据
Hadoop: 3.2.1
hbase:2.3.4
火花:2.4.7
蟒蛇:3.7.6
Hbase 表:"tmp"
hbase(main):001:0> scan "tmp"
ROW COLUMN+CELL
1 column=cols:age,timestamp=2021-06-22T14:17:31.735,value=10
1 column=cols:name,timestamp=2021-06-22T14:17:23.037,value=tom
2 column=cols:age,timestamp=2021-06-22T14:17:40.157,value=11
2 column=cols:name,timestamp=2021-06-22T14:17:48.516,value=dim
火花壳:
pyspark \
--master 纱线 \
--deploy-mode 客户端 \
--num-executors 5 \
--executor-cores 1 \
--driver-memory 6g \
--executor-memory 1g \
--packages org.apache.hbase.connectors.spark:hbase-spark:1.0.0
火花代码:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = (spark.read.format("org.apache.hadoop.hbase.spark")
.option("hbase.table","tmp")
.option("hbase.columns.mapping","col1 STRING :key,col2 STRING cols:name,col3 STRING cols:age")
.load())
df.show()
我在 pyspark shell 中运行 pyspark 代码。 但我收到一个错误。
Traceback (most recent call last):
File "<stdin>",line 1,in <module>
File "/usr/lib/spark-current/python/pyspark/sql/readwriter.py",line 172,in load
return self._df(self._jreader.load())
File "/usr/lib/spark-current/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 1257,in __call__
File "/usr/lib/spark-current/python/pyspark/sql/utils.py",line 67,in deco
return f(*a,**kw)
File "/usr/lib/spark-current/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py",line 328,in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o169.load.
: java.lang.NullPointerException
at org.apache.hadoop.hbase.spark.HBaseRelation.<init>(DefaultSource.scala:138)
at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:69)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:365)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:242)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。