如何解决运行Hadoop需要多少内存？
我正在为课程项目使用Hadoop MapReduce，现在我正在尝试在Windows Linux子系统上正确安装Hadoop 2.6.5，以便（现在）可以运行WordCount example。
现在我正在运行2个示例文本文件，每个文件包含1个句子和大约5个单词，但是当我去运行hadoop命令时
bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output
我收到以下错误
20/09/24 12:26:28 INFO mapreduce.Job: Job job_1600964711999_0001 Failed with state Failed due to:
Application application_1600964711999_0001 Failed 2 times due to AM Container for
appattempt_1600964711999_0001_000002 exited with exitCode: -103
For more detailed output,check application tracking page:http://LAPTOP-
MF1IL5JD.localdomain:8088/proxy/application_1600964711999_0001/Then,click on links to logs of each
attempt.
Diagnostics: Container [pid=20730,containerID=container_1600964711999_0001_02_000001] is running
beyond virtual memory limits. Current usage: 176.8 MB of 2 GB physical memory used; 566.9 GB of 4.2
GB virtual memory used. Killing container.
Dump of the process-tree for container_1600964711999_0001_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYstem_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 20730 20728 20730 20730 (bash) 0 4 127286341632 398 /bin/bash -c /usr/lib/jvm/java-11-openjdk-
amd64/bin/java -Dlog4j.configuration=container-log4j.properties -
Dyarn.app.container.log.dir=/home/joe/hadoop/hadoop-
2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001/stdout 2>/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001/stderr
|- 20738 20730 20730 20730 (java) 373 64 481410650112 44862 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
20/09/24 12:26:28 INFO mapreduce.Job: Counters: 0
现在我看到它说当我向Java提供大约4G内存时，它使用了超过TB的虚拟内存，但是我应该在配置文件中更改多少“虚拟内存”，因为我总共只有大约16G ，因此1T的内存不足。如果我当前在这么小的数据集上运行，为什么Hadoop需要这么多的内存？
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
