微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

使用QT5Reactor在PyQt5中创建可重启的刮板蜘蛛反应器

如何解决使用QT5Reactor在PyQt5中创建可重启的刮板蜘蛛反应器

我的GUI有一个“更新数据库”按钮,每当用户按下该按钮时,我都想启动一个Scrapy蜘蛛,将抓取的数据存储到sqlite3数据库中。我将qt5reactor的实现方式设置为this answer suggests,但是当我第二次按下更新按钮时,却遇到了ReactorNotRestartable错误。我该如何解决?我尝试从CrawlerRunner切换到CrawlerProcess,但是它仍然会引发相同的错误(但是,也许我做错了)。我也不能使用this answer,因为q.get()锁定了事件循环,因此在运行Spider时GUI冻结。我是多处理人员的新手,如果我遗漏了一些非常明显的内容,对不起。

在main.py

... # PyQt5 imports
import qt5reactor
from scrapy import crawler
from twisted.internet import reactor
from currency_scraper.currency_scraper.spiders.investor import InvestorSpider

class MyGUI(QMainWindow):

    def __init__(self):
        self.update_db_button.clicked.connect(self.on_clicked_update)
        ...

    def on_clicked_update(self):
        """Gives command to run scraper and fetch data from the website"""
        runner = crawler.CrawlerRunner(
            {
                "USER_AGENT": "currency scraper","SCRAPY_SETTINGS_MODULE": "currency_scraper.currency_scraper.settings","ITEM_PIPELInes": {
                    "currency_scraper.currency_scraper.pipelines.sqlite3Pipeline": 300,}
            }
        )
        deferred = runner.crawl(InvestorSpider)
        deferred.addBoth(lambda _: reactor.stop())
        reactor.run() # has to be run here or the crawling doesn't start
        update_notification()

    ... # other stuff

if __name__ == "__main__":
   open_window()
   qt5reactor.install()
   reactor.run()

错误日志:

Traceback (most recent call last):
  File "c:/Users/Familia/Documents/ProgramaþÒo/Python/Projetos/Currency_converter/main.py",line 330,in on_clicked_update
    reactor.run()
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1282,in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1262,in startRunning
    ReactorBase.startRunning(self)
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 765,in startRunning    
    raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。