如何加速请求 Python

如何解决如何加速请求 Python

所以我有这段代码可以抓取 javascript 内容：

from requests_html import HTMLSession

#create the session
session = HTMLSession()

#define our URL
url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+displayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'

#use the session to get the data
r = session.get(url)

#Render the page,up the number on scrolldown to page down multiple times on a page
r.html.render(sleep=0,keep_page=True,scrolldown=0)

#take the rendered html and find the element that we are interested in
links = r.html.find('#href')

#loop through those elements extracting the text and link
for item in links:
    link = {
        'link': item.absolute_links
    }
print(link)

但是它需要 2-3 秒，这对我来说加载时间太长了。有没有办法加快速度？

解决方法

根本不需要抓取网站。查看源代码时，您可以看到 javascript 正在从输入 url 生成 Amazon url：

document.getElementById(
          "href"
        ).href = `https://www.amazon${tld}/dp/${asin}?tag=${tag}&linkCode=ogi&th=1&psc=1&smid=${smid}`;

这意味着您只需在 python 中复制此函数即可生成您的网址。您可以通过urllib.parse获取url参数的值，然后使用字符串格式生成新的url：

from urllib.parse import urlsplit,parse_qs

url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+DisplayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'
query = urlsplit(url).query
params = parse_qs(query)
amazon_url = f"https://www.amazon{params['tld'][0]}/dp/{params['asin'][0]}?tag={params['tag'][0]}&linkCode=ogi&th=1&psc=1&smid={params['smid'][0]}"

结果：

https://www.amazon.de/dp/B08L8LG4M3?tag=partalertde-21&linkCode=ogi&th=1&psc=1&smid=A3JWKAKR8XB7XF