我尝试实例化多个蜘蛛.第一个工作正常,但第二个给我一个错误:ReactorNotRestartable.
Feeds = { 'nasa': { 'name': 'nasa','url': 'https://www.nasa.gov/RSS/dyn/breaking_news.RSS','start_urls': ['https://www.nasa.gov/RSS/dyn/breaking_news.RSS'] },'xkcd': { 'name': 'xkcd','url': 'http://xkcd.com/RSS.xml','start_urls': ['http://xkcd.com/RSS.xml'] } }
通过上面的项目,我尝试在循环中运行两个蜘蛛,如下所示:
from scrapy.crawler import CrawlerProcess from scrapy.spiders import XMLFeedSpider class MySpider(XMLFeedSpider): name = None def __init__(self,**kwargs): this_Feed = Feeds[self.name] self.start_urls = this_Feed.get('start_urls') self.iterator = 'iternodes' self.itertag = 'items' super(MySpider,self).__init__(**kwargs) def parse_node(self,response,node): pass def start_crawler(): process = CrawlerProcess({ 'USER_AGENT': CONfig['USER_AGENT'],'DOWNLOAD_HANDLERS': {'s3': None} # boto issues }) for Feed_name in Feeds.keys(): MySpider.name = Feed_name process.crawl(MySpider) process.start()
第二个循环的例外看起来像这样,蜘蛛打开了,但随后:
... 2015-11-22 00:00:00 [scrapy] INFO: Spider opened 2015-11-22 00:00:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min) 2015-11-22 00:00:00 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 2015-11-21 23:54:05 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 Traceback (most recent call last): File "env/bin/start_crawler",line 9,in <module> load_entry_point('Feed-crawler==0.0.1','console_scripts','start_crawler')() File "/Users/bling/py-Feeds-crawler/Feed_crawler/crawl.py",line 51,in start_crawler process.start() # the script will block here until the crawling is finished File "/Users/bling/py-Feeds-crawler/env/lib/python2.7/site-packages/scrapy/crawler.py",line 251,in start reactor.run(installSignalHandlers=False) # blocking call File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py",line 1193,in run self.startRunning(installSignalHandlers=installSignalHandlers) File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py",line 1173,in startRunning ReactorBase.startRunning(self) File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py",line 684,in startRunning raise error.ReactorNotRestartable() twisted.internet.error.ReactorNotRestartable
我是否必须使第一个MySpider无效或我做错了什么,需要改变它的工作原理.提前致谢.
解决方法
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。