带有“很多”行的文件的Promise错误

如何解决带有“很多”行的文件的Promise错误

我正在读取一个csv文件，每一行都包含一个网站网址。我具有读取文件的功能：

function readCSV(csv){

  var lines=csv.split("\n");

  var result = [];

  var headers=lines[0].split(",");

  // for every line of the file I call check_page function to check the policies (csp and xfo)
  Promise.all(
    lines.map(line => {
      var obj = {};
      var currentline=line.split(",");
      console.log("currentline: "+currentline[1])  
      return check_page("https://www."+currentline[1])
    })
  ).then(() => console.log('it worked')).catch(err => console.log(err));
}

此函数调用内部另一个函数，以通过http-request获取csp和xfo标头。

async function check_page(web_page){

    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(web_page)
    
    
      
    console.log("MAIN: "+page.mainFrame().url())
    /* I  send for every iframe an http request for retrieve the policies from http header */
    var XMLHttpRequest = require("xmlhttprequest").XMLHttpRequest;       
    var req = new XMLHttpRequest();
    console.log("FACCIO LA GET: "+page.mainFrame().url())
    req.open('GET',page.mainFrame().url(),false);
    req.send(null)
    var headers = req.getAllResponseHeaders().toLowerCase();       
    var arr = headers.trim().split(/[\r\n]+/);
        // Create a map of header names to values
    var headerMap = {};
    arr.forEach(function (line) {
      var parts = line.split(': ');
      var header = parts.shift();
      var value = parts.join(': ');
      headerMap[header] = value;
    });
        
         

    await browser.close();
  
}

如果我的行数很少，我的代码将起作用：但是如果我的文件中包含100行，则会出现此错误：

(node:1076) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:1076) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:1076) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:1076) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

我认为我必须以某种方式划分工作，但是我不知道如何。

解决方法

您要检查一堆URL，并且想要查看响应头。

完全不需要使用puppeteer，更不用说为每个URL启动完整的浏览器了。这是完全没有意义的，而且令人难以置信浪费。每个URL发送一个HTTP请求就足够了。

使用request-promise模块，这是一个非常简单的任务。

const request = require('request-promise');

function readCSV(csv) {                               // -> 'a,b,c\na,c'
    var lines = csv.split("\n");                      // -> ['a,c','a,c']
    var table = lines.map(line => line.split(","));   // -> [['a','b','c'],['a','c']]
    var requests = table.map(row => request({         // -> [request,request]
        method: 'GET',uri: "https://www." + row[1],resolveWithFullResponse: true
    }));

    return Promise.all(requests).then(responses => {  // -> [response,response]
        console.log('it worked');
        responses.forEach(response => {
            var hrds = response.headers;
            // hrds is an object. print it,extract info from it,whatever
            // don't forget to look at the other properties of `response`,as well
        });
    }).catch(err => console.log(err));
}

请考虑使用比readCSV更好的名称，因为读取CSV并不是该功能的作用。