如何解决使用QT5Reactor在PyQt5中创建可重启的刮板蜘蛛反应器
我的GUI有一个“更新数据库”按钮,每当用户按下该按钮时,我都想启动一个Scrapy蜘蛛,将抓取的数据存储到sqlite3数据库中。我将qt5reactor
的实现方式设置为this answer suggests,但是当我第二次按下更新按钮时,却遇到了ReactorNotRestartable
错误。我该如何解决?我尝试从CrawlerRunner
切换到CrawlerProcess
,但是它仍然会引发相同的错误(但是,也许我做错了)。我也不能使用this answer,因为q.get()
锁定了事件循环,因此在运行Spider时GUI冻结。我是多处理人员的新手,如果我遗漏了一些非常明显的内容,对不起。
在main.py
... # PyQt5 imports
import qt5reactor
from scrapy import crawler
from twisted.internet import reactor
from currency_scraper.currency_scraper.spiders.investor import InvestorSpider
class MyGUI(QMainWindow):
def __init__(self):
self.update_db_button.clicked.connect(self.on_clicked_update)
...
def on_clicked_update(self):
"""Gives command to run scraper and fetch data from the website"""
runner = crawler.CrawlerRunner(
{
"USER_AGENT": "currency scraper","SCRAPY_SETTINGS_MODULE": "currency_scraper.currency_scraper.settings","ITEM_PIPELInes": {
"currency_scraper.currency_scraper.pipelines.sqlite3Pipeline": 300,}
}
)
deferred = runner.crawl(InvestorSpider)
deferred.addBoth(lambda _: reactor.stop())
reactor.run() # has to be run here or the crawling doesn't start
update_notification()
... # other stuff
if __name__ == "__main__":
open_window()
qt5reactor.install()
reactor.run()
错误日志:
Traceback (most recent call last):
File "c:/Users/Familia/Documents/ProgramaþÒo/Python/Projetos/Currency_converter/main.py",line 330,in on_clicked_update
reactor.run()
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1282,in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1262,in startRunning
ReactorBase.startRunning(self)
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 765,in startRunning
raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。