尝试刮群仅用于练习为什么它返回的结果较少？

如何解决尝试刮群仅用于练习为什么它返回的结果较少？

我是 Python 抓取的新手，因此作为练习的一部分，我尝试了其他几个通常根本不返回数据的站点，但是当我检查 Groupon 时，我发现 urllib 只返回第一个8个结果，而浏览器页面有36个结果。

我正在使用 urllib 和 BS4。下面是代码

from urllib.request import Request,urlopen
from bs4 import BeautifulSoup
req = Request('https://www.groupon.com/browse/chicago?category=beauty-and-spas')
req.add_header('User-Agent','Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/85.0.4183.121 Safari/537.36')
try:
    with urlopen(req) as response:
        htmlcontent = response.read().decode('utf-8')
except:
    htmlcontent = None
soup = BeautifulSoup(htmlcontent,'lxml')
all_links = soup.find('div',{ 'id': 'pull-results' }).select('figure > div > a')

有人可以告诉我，我在代码中缺少什么才能提取所有数据吗？ 如果这不起作用或不应该起作用，那么我们是否将硒作为唯一的选择？

解决方法

其他 28 个链接是动态加载的，因此 urllib 不支持。但是，您可以通过向以下地址发送 GET 请求来抓取它们：

https://www.groupon.com/partial/browse/get-lazy-loaded-cards?category=beauty-and-spas&_csrf=P6rFPl1o-xDta8uOABKo_9LOiUajyK9bieMg

from bs4 import BeautifulSoup
from urllib.request import Request,urlopen


req = Request(
    "https://www.groupon.com/partial/browse/get-lazy-loaded-cards?category=beauty-and-spas&_csrf=P6rFPl1o-xDta8uOABKo_9LOiUajyK9bieMg"
)
req.add_header(
    "User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/85.0.4183.121 Safari/537.36",)

with urlopen(req) as response:
    htmlcontent = response.read().decode("utf-8")
    soup = BeautifulSoup(htmlcontent,"lxml")

    for tag in soup.find_all(class_=r'\"cui-content\"'):
        try:
            link = tag.find("a")["href"]
        except TypeError:
            continue
        print(link.replace('\\"',"").replace("\\n",""))

输出：

https://www.groupon.com/deals/yupo-health-2?deal_option=75076cab-9268-4c2d-9459-e79a559dfed6
https://www.groupon.com/deals/infinity-laser-spa-2-4?deal_option=4dc14f7d-29ac-45e1-a664-ea76ddc44718
https://www.groupon.com/deals/dr-laser-nyc-3?deal_option=232325a0-8b3f-4fa9-b42b-7dc668ff7474
...
...

,

尝试以下操作以获取遍历下一页的所有项目及其链接：

import requests
from bs4 import BeautifulSoup

base_link = 'https://www.groupon.com/browse/chicago?category=beauty-and-spas'
url = 'https://www.groupon.com/partial/browse/get-paginated-cards?'

params = {
    'category': 'beauty-and-spas','page': 1
}

with requests.Session() as s:
    s.headers['user-agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/87.0.4280.88 Safari/537.36'
    r = s.get(base_link)
    params['_csrf'] = r.cookies['_csrf']

    while True:
        print("current page----------->",params['page'])
        res = s.get(url,params=params)
        soup = BeautifulSoup(res.json()[0]['cardSlotHtml'],"lxml")
        if not soup.select_one("figure[data-pingdom-info='purchasable-deal']"):break
        for item in soup.select("figure[data-pingdom-info='purchasable-deal']"):
            item_title = item.select_one(".cui-udc-details .cui-udc-title").get_text(strip=True)
            item_link = item.select_one(".cui-content > a[href]").get("href")
            print(item_title,item_link)

        params['page']+=1