响应 401，晨星抓取

如何解决响应 401，晨星抓取

我想获取共同基金的投资组合。我可以在morningStar上看到它。(https://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F00000ZG4Z&tab=3)

在开发工具中，我找到了 API，但我只能得到响应 401 - 未授权。

代码如下：

import requests

url = 'https://www.us-api.morningstar.com/sal/sal-service/fund/portfolio/holding/v2/F00000ZG4Z/data'

params = {
'premiumNum': '10','freeNum': '10','languageId': 'en-GB','locale': 'en-GB','clientId': 'MDC_intl','benchmarkId': 'category','component': 'sal-components-mip-holdings','version': '3.40.1'
}

headers = {
'accept': '*/*','accept-encoding': 'gzip,deflate,br','accept-language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7','authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI3RUI4NDcyRi1FRjk4LTQ0ODgtQjJGQS0zQzMyQUQyQjdCNUYiLCJjdXN0b206ZGF0YV9yb2xlIjpbIlFTLk1hcmtldHMiLCJRUy5QdWxscXMiLCJTQUwuU2VydmljZSJdLCJjdXN0b206dWltX3JvbGVzIjoiIiwiY3VzdG9tOmNvbmZpZ19pZCI6IkVVUlRMX0VDIiwiaXNzIjoiVUlNIiwiZXhwIjoxNjEyNzkzNzgwLCJjdXN0b206cm9sZSI6WyJFQy5TZXJ2aWNlLkNvbmZpZ3VyYXRpb24iLCJFQy5TZXJ2aWNlLkhvc3RpbmciXSwiaWF0IjoxNjEyNzkwMTgwLCJjdXN0b206Y29tcGFueV9pZCI6IjUzNGJiMTM0LTMwNWYtNDhhYi04MmYxLWNjZTBlZDFlYTgzYSIsImN1c3RvbTppbnRlcm5hbF9jb21wYW55X2lkIjoiQ2xpZW50MCJ9.rVnAGGBsBxv3XMnBkAUJdo9tJmrsBGqjj2Wxtz0CLdz7MzErej8RBbVfDOEd1xdxmSd5YpAxFA_kuu41r8jyhCTAAvtBE6gWzizXMKxbZ7RQ9hxnWm3dVnsHspWKCwCCQl5B2WpBjDbTH-edBHmTLuUrq9Qc3_2r8Lx28-oFbxXtzdmlSssgtZqdz2dMXN-Fo8LYMrTPtqt0Kv_lwQMmH42_PwZd6BqtzMvpqN1rzCHRFN-y5xL4yZ_PGBviuhD841mXVNCWnv8WGYHOK5p0JEJ7pg17O2fvEhuMxrpwvmC4LRELsA-QRgCo15UpOp_AHcxg_r7HPxPA9TtHZUy6Rw','credentials': 'omit','origin': 'https://www.morningstar.co.uk','referer': 'https://www.morningstar.co.uk/','sec-ch-ua': '"Chromium";v="88","Google Chrome";v="88",";Not A Brand";v="99"','sec-ch-ua-mobile': '?0','sec-fetch-dest': 'empty','sec-fetch-mode': 'cors','sec-fetch-site': 'cross-site','user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.150 Safari/537.36','x-api-requestid': '6c20fb56-8055-aa34-1632-5f5a8dbaddde','x-sal-contenttype': 'e7FDDltrTy+tA2HnLovvGL0LFMwT+KkEptGju5wXVTU='
}

response = requests.get(url,params=params,headers=headers)

print(response)

你知道我怎样才能得到回复吗？

谢谢

解决方法

Mael，看来您确实需要更新 Beaerer 令牌。只是玩弄网站，它看起来来自 def close_popup(): try: <CLOSE POPUP> return True except: return False def main(): try: <YOUR CODE> except: if close_popup(): <YOUR CODE IF POPUP CLOSED> else: <YOUR CODE TO RUN IF NOT SUCCESSFULL> 中的 <script> 标记。

所以你可以做的是在那里做一个请求，然后使用 BeautifulSoup 提取令牌，然后将其输入到你的第二个请求标头中。

https://www.morningstar.fr/Common/funds/snapshot/PortfolioSAL.aspx

输出：

import requests
from bs4 import BeautifulSoup

# Get the Bearer Token
url = 'https://www.morningstar.fr/Common/funds/snapshot/PortfolioSAL.aspx'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.150 Safari/537.36'}
payload = {
'Site': 'fr','FC': 'F000010S65','IT': 'FO','LANG': 'fr-FR'}
response = requests.get(url,headers=headers,params=payload)
soup = BeautifulSoup(response.text,'html.parser')
script = soup.find('script',{'type':'text/javascript'})
bearerToken = str(script).split('tokenMaaS:')[-1].split('}')[0].replace('"','').strip()



# API request
url = 'https://www.us-api.morningstar.com/sal/sal-service/fund/portfolio/holding/v2/F00000ZG4Z/data'

params = {
'premiumNum': '10','freeNum': '10','languageId': 'en-GB','locale': 'en-GB','clientId': 'MDC_intl','benchmarkId': 'category','component': 'sal-components-mip-holdings','version': '3.40.1'
}

headers = {
'accept': '*/*','accept-encoding': 'gzip,deflate,br','accept-language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7','authorization': 'Bearer %s' %bearerToken,'credentials': 'omit','origin': 'https://www.morningstar.co.uk','referer': 'https://www.morningstar.co.uk/','sec-ch-ua': '"Chromium";v="88","Google Chrome";v="88",";Not A Brand";v="99"','sec-ch-ua-mobile': '?0','sec-fetch-dest': 'empty','sec-fetch-mode': 'cors','sec-fetch-site': 'cross-site','user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.150 Safari/537.36','x-api-requestid': '6c20fb56-8055-aa34-1632-5f5a8dbaddde','x-sal-contenttype': 'e7FDDltrTy+tA2HnLovvGL0LFMwT+KkEptGju5wXVTU='
}

response = requests.get(url,params=params,headers=headers).json()

print(response)

,

问题是您需要提供更新的不记名令牌，因为它已过期。您是否监控网络流量以了解如何在每次请求之前获取有效令牌？