如何解决使用 SparkTrials 进行 hyperopt 并行化
我需要优化模型的超参数,并且希望并行化代码以使其更快。 运行此代码:
from hyperopt import hp,fmin,tpe,SparkTrials
def objective(space):
loss = space['a'] + space['b']
return loss
space = {'a': hp.quniform('a',1,10,1),'b': hp.quniform('b',1)}
spark_trials = SparkTrials()
best = fmin(objective,space,algo=tpe.suggest,max_evals=100,trials=spark_trials)
print(best)
我收到此错误:
JAVA_HOME is not set
Traceback (most recent call last):
File "hyperparameters.py",line 13,in <module>
spark_trials = SparkTrials()
File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/hyperopt/spark.py",line 84,in __init__
if spark_session is None
File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/sql/session.py",line 228,in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 384,in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 144,in __init__
SparkContext._ensure_initialized(self,gateway=gateway,conf=conf)
File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 331,in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/home/el/miniconda3/envs/Xpresso/lib/python3.6/site-packages/pyspark/java_gateway.py",line 108,in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。