<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,Arial,'Hiragino Sans GB','Microsoft YaHei',simsun;vertical-align:baseline;color:rgb(93,93,93);background-color:rgb(255,255,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只有MD5的超链接可以点,点进去看看,<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">我写这个的爬虫的目的是下载样本,只关心样本的下载地址,并不关心其他的信息。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">现在我们可以理清下思路:<pre style="margin-bottom:0px;padding-right:0px;padding-left:0px;border:0px;font-size:18px;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(93,255);"><pre class="source-code" style="margin-bottom:0px;padding:15px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:rgb(243,244,245);"><code style="margin:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">获取每一页的网页源码解析当前页的每一个<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(0,134,179);">md5对应的详细信息链接
在详细信息页面解析下载地址。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">提取规则<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">先从第一页开始爬起<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">scrapy支持xss选择器和css选择器,一般是用哪个比较方便就用哪个。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">对应链接的xpath选择器为 <code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//td/a/@href ,<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">下一页的链接的xpath选择器为 <code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//a[contains(text(),'Next')]/@href<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">登陆<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">之前写爬虫的时候,是可以直接通过post登陆的,而我开始写这篇文章的时候开始加上了google的ReCaptcha验证码。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只能换种方式使用selenium来登陆Malwr.得到登陆后的cookie后,我们就可以用这个cookie开始下载样本了。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">FilesPipeline是从file_urls中拿到下载地址然后开始下载文件,因此我们还需要Item.py定义一个file_urls.<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">解析下载地址,将解析到的下载链接列表放入item的file_urls,返回item。pipeline会file_urls拿到链接开始调用get_media_requests下载文件<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);"><p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。