如何解决Sqoop 从 hive 表导出到 Teradata 表
我正在尝试将数据从 hive 加载到 teradata。 为此，我在 hive 中创建了一个外部表，将值插入其中，并运行了 sqoop 导出命令。
create external table Sample_load
(
name varchar(32),addr varchar(50),);
insert into table Sample_load select F_name,Address from data where age > 18;
sqoop export -Dhadoop.security.credential.provider.path=pwd_file_for_connection-
Dmapreduce.map.java.opts=" -Duser.timezone=GMT" --connect jdbc_url/database=tera_db,charset=utf8 -m
10 --username uname --password-alias pwd --table teradata_table --hcatalog-database db_name --
hcatalog-table sample_load ;
但我收到以下错误：
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
... 10 more
Caused by: com.teradata.connector.common.exception.ConnectorException: java.lang.classNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatSplit
at java.net.urlclassloader.findClass(urlclassloader.java:382)
at java.lang.classLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.classLoader.loadClass(ClassLoader.java:351)
at java.lang.class.forName0(Native Method)
at java.lang.class.forName(Class.java:264)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:276)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:281)
... 15 more
21/01/05 03:21:01 INFO mapreduce.Job: Task Id : attempt_1608639506866_65714_m_000000_1,Status : Failed
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
... 10 more
Caused by: com.teradata.connector.common.exception.ConnectorException: java.lang.classNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatSplit
at java.net.urlclassloader.findClass(urlclassloader.java:382)
at java.lang.classLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.classLoader.loadClass(ClassLoader.java:351)
at java.lang.class.forName0(Native Method)
at java.lang.class.forName(Class.java:264)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:276)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:281)
... 15 more
21/01/05 03:21:08 INFO mapreduce.Job: Task Id : attempt_1608639506866_65714_m_000000_2,Status : Failed
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
... 10 more
Caused by: com.teradata.connector.common.exception.ConnectorException: java.lang.classNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatSplit
at java.net.urlclassloader.findClass(urlclassloader.java:382)
at java.lang.classLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.classLoader.loadClass(ClassLoader.java:351)
at java.lang.class.forName0(Native Method)
at java.lang.class.forName(Class.java:264)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:276)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:378)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupinformation.doAs(UserGroupinformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
at com.teradata.connector.hcat.ConnectorCombineFileHCatSplit.<init>(ConnectorCombineFileHCatInputFormat.java:281)
... 15 more
21/01/05 03:21:16 INFO mapreduce.Job: map 100% reduce 0%
21/01/05 03:21:16 INFO mapreduce.Job: Job job_1608639506866_65714 Failed with state Failed due to: Task Failed task_1608639506866_65714_m_000000
Job Failed as tasks Failed. FailedMaps:1 FailedReduces:0 killedMaps:0 killedReduces: 0
21/01/05 03:21:16 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=76036
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=19009
Total vcore-milliseconds taken by all map tasks=19009
Total megabyte-milliseconds taken by all map tasks=77860864
21/01/05 03:21:16 INFO processor.teradataOutputProcessor: output postprocessor com.teradata.connector.teradata.processor.teradataBatchInsertProcessor starts at: 1609834876349
21/01/05 03:21:16 INFO processor.teradataBatchInsertProcessor: drop stage table "Sample_A01_032038570"
21/01/05 03:21:16 INFO processor.teradataOutputProcessor: output postprocessor com.teradata.connector.teradata.processor.teradataBatchInsertProcessor ends at: 1609834876349
21/01/05 03:21:16 INFO processor.teradataOutputProcessor: the total elapsed time of output postprocessor com.teradata.connector.teradata.processor.teradataBatchInsertProcessor is: 0s
21/01/05 03:21:16 INFO teradata.teradataSqoopExportHelper: teradata export job completed with exit code 1
21/01/05 03:21:16 ERROR tool.ExportTool: Error during export:
Import Job Failed
at org.apache.sqoop.teradata.teradataConnManager.exportTable(teradataConnManager.java:550)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:94)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:113)
at org.apache.sqoop.Sqoop.run(Sqoop.java:151)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:187)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:241)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:250)
at org.apache.sqoop.Sqoop.main(Sqoop.java:259)
21/01/05 03:21:16 INFO metastore.HivemetastoreClient: Closed a connection to metastore,current connections: 1
检查日志时，我找不到正在运行的地图进程，即它甚至在地图进程启动之前就失败了 请帮我找出问题所在。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
