微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

为什么 UrlFetchApp.fetch(url).getContentText() 返回不一致的结果?

如何解决为什么 UrlFetchApp.fetch(url).getContentText() 返回不一致的结果?

我正在遍历亚马逊产品列表,以获取列表中每个产品的负面评论。 但是,大约 90% 的情况下,UrlFetchApp.fetch("url").getContentText(); 将返回 HTML 的简短版本,其中不包含页面上的任何实际内容

现在我正在强制循环一次又一次地尝试,直到返回正确的(更长的)HTML,但这需要很长时间。我想知道是否有人知道任何替代方案或任何技巧以使其更加一致?

例如,以下是其中一个 URL:(网页的动态特性是否与不一致有关?)

https://www.amazon.com/product-reviews/B00J2DGTD8/ref=cm_cr_arp_d_viewopt_srt?ie=UTF8&filterByStar=one_star&reviewerType=all_reviews&pageNumber=1&sortBy=recent#reviews-filter-bar

这是一个最小的可重复示例:

function mre() {

  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = ss.getActiveSheet();
  var lastRow = sheet.getLastRow();

  var star = ["one","two","three"];

  for(var a = 0; a < 3; a++) { //star 

    htmlRaw = UrlFetchApp
      .fetch(`https://www.amazon.com/product-reviews/B00J2DGTD8/ref=cm_cr_arp_d_viewopt_srt?ie=UTF8&filterByStar=${star[a]}_star&reviewerType=all_reviews&pageNumber=1&sortBy=recent#reviews-filter-bar`).getContentText(); 

    html = htmlRaw.toString();  
  
    //Logger.log(htmlRaw);
    //Logger.log(html);

    //Making sure we have the correct HTML
    //The below if statement acts like a gate. It tries again and again until it receives the correct version of the HTML. Then it will "open the gate" to the else{} below
  
    if(html.substring(100000,100500) == ``) { //the longer (correct) version of the HTML has 100,000+ characters
      
      Logger.log(`if its empty,then try again: ${html.substring(100000,100500)}`);
      a--;
    }

    else {

      lastRow = sheet.getLastRow(); //so we actually move the down the rows
      sheet.getRange(lastRow+1,1).setValue(html.substring(100000,100500)); //html snippet

    } //else (correct html)
  } //for (a - stars)
} //function mre()

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。