Scrapy Splash 截图网页从 png 到 webp 不保存渲染图像

如何解决Scrapy Splash 截图网页从 png 到 webp 不保存渲染图像

Scrapy 启动屏幕截图网页并即时将 image.png 转换为 image.webp。图片占用大量内存，scrapy splash 返回 .png 文件。我抓取了很多网页，所以我需要将其转换为 image.webp 以减少磁盘消耗。

解决方法

使用 BytesIO() 将 png 图像转换为 webp

import scrapy
import json
import base64
from scrapy_splash import SplashRequest
from io import BytesIO
from PIL import Image

class WebScreenshots(scrapy.Spider):
    name = 'screenshot'

    def start_requests(self):
        links = ["your","review","links"]
        for url in links:
            yield SplashRequest(
                                    url=url,callback=self.parse,dont_filter=True,args={
                                            'html': 1,'png': 1,'wait': 20,'url':url,'render_all': 1
                                        },endpoint='render.json'
                                )


    def parse(self,response):
        """
           file_name = I just use the link as a file name,you can specify own file name.
           directory = the path where you will saved the converted image 
        """
        url = response.url
        file_name = "{}.webp".format(url)
        imgdata = base64.b64decode(response.data['png'])
        image = Image.open(BytesIO(imgdata))
        image.save(directory + file_name)

        print (file_name)
        print ('screenshot done...')