如何使用pyppeteer异步获取url一个浏览器多个标签

如何解决如何使用pyppeteer异步获取url一个浏览器多个标签

我想要我的脚本

1. 打开说 3 个标签

2. 异步获取一个 url（每个标签都相同）

3. 保存回复

4. 睡眠 4 秒

5. 使用正则表达式解析响应（我尝试过 BeautifulSoup 但它太慢了）并返回一个令牌

6. 在 3 个标签中循环多次

我的问题是 2。我有一个示例脚本，但它同步获取 url。我想让它异步。

from pyppeteer import launch

urls = ['https://www.example.com']


async def main():
    browser = await launch(headless=False)
    for url in urls:
        page1 = await browser.newPage()
        page2 = await browser.newPage()
        page3 = await browser.newPage()

        await page1.goto(url)
        await page2.goto(url)
        await page3.goto(url)

        title1= await page1.title()
        title2= await page2.title()
        title3= await page3.title()

        print(title1)
        print(title2)
        print(title3)

    #await browser.close()


asyncio.get_event_loop().run_until_complete(main())

另外，正如你所看到的，代码并不那么简洁。我如何让它异步？

此外，如果有帮助，我还有其他不适合我需要的 pyppeteer 脚本，以防万一转换这些脚本会更容易

import asyncio
from pyppeteer import launch

url = 'http://www.example.com'
browser = None

async def fetchUrl(url):
    # Define browser as a global variable to ensure that the browser window is only created once in the entire process
    global browser
    if browser is None:
        browser = await launch(headless=False)

    page = await browser.newPage()

    await page.goto(url)
    #await asyncio.wait([page.waitForNavigation()])
    #str = await page.content()
    #print(str)

 # Execute this function multiple times for testing
asyncio.get_event_loop().run_until_complete(fetchUrl(url))
asyncio.get_event_loop().run_until_complete(fetchUrl(url))

该脚本是异步的，但它一次执行一个事件循环，因此它与同步一样好。

# cat test.py
import asyncio
import time
from pyppeteer import launch

WEBSITE_LIST = [
    'http://envato.com','http://amazon.co.uk','http://example.com',]

start = time.time()

async def fetch(url):
    browser = await launch(headless=False,args=['--no-sandBox'])
    page = await browser.newPage()
    await page.goto(f'{url}',{'waitUntil': 'load'})
    print(f'{url}')
    await asyncio.sleep(1)
    await page.close()
    #await browser.close()

async def run():
    tasks = []

    for url in WEBSITE_LIST:
        task = asyncio.ensure_future(fetch(url))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)

print(f'It took {time.time()-start} seconds.')

该脚本是异步的，但它为每个 url 启动一个单独的浏览器，这最终会占用太多资源。

解决方法

这将在单独的标签中打开每个 URL：

import asyncio
import traceback

from pyppeteer import launch

URLS = [
    "http://envato.com","http://amazon.co.uk","http://example.com",]


async def fetch(browser,url):
    page = await browser.newPage()

    try:
        await page.goto(f"{url}",{"waitUntil": "load"})
    except Exception:
        traceback.print_exc()
    else:
        html = await page.content()
        return (url,html)
    finally:
        await page.close()


async def main():
    tasks = []
    browser = await launch(headless=True,args=["--no-sandbox"])

    for url in URLS:
        tasks.append(asyncio.create_task(fetch(browser,url)))

    for coro in asyncio.as_completed(tasks):
        url,html = await coro
        print(f"{url}: ({len(html)})")

    await browser.close()


if __name__ == "__main__":
    main = asyncio.run(main())