requests_HTML 分页：无法存储列表

如何解决requests_HTML 分页：无法存储列表

我尝试使用 request_HTML 将各种页面存储为列表。虽然无法存储整个页面，但您能否帮助存储为页面链接列表...

我可以知道在哪里失踪吗？。我可以知道如何创建像scrapy这样的分页吗？

        from requests_html import HTMLSession

        def pagecheck(checkl):
            x=se.get(checkl)
            if x.html.find('div.isaiminida'):
                print (''.join(x.html.find('a.pagination_next',first=True).absolute_links))
                xplink= (''.join(x.html.find('a.pagination_next',first=True).absolute_links))
                if x.html.find('a.pagination_next'):
                    pages.append(xplink);return p(xplink)
            else:
                print ('Page doesnt exist')

        def p(xplink):
            x=se.get(xplink)
            if x.html.find('a.pagination_next'):
                pagecheck(x.url)
            else:
                print (z,'\t Pages scrape completed.....')
            return pages
            
            
        if __name__ == '__main__':

            #:: movies index
            xa = ['a','z','c']
            for z in xa:

                url = f'https://isaidubb.co/tamil-atoz-dubbed-movies/{z}'
                se = HTMLSession()
                x=se.get(url)
                pages =[];px=[]
                pages=pagecheck(x.url)

解决方法

在主函数中添加append和extend时，我可以列出/存储所有页面。

此代码适合使用列出所有页面进行抓取的人......

            from requests_html import HTMLSession

            '''
            request HTML; pagination implementation

            '''

            def pagecheck(checkl):
                x=se.get(checkl)
                if x.url not in px:
                    px.append(x.url)
            
                if x.html.find('div.isaiminida'):
                    xplink= (''.join(x.html.find('a.pagination_next',first=True).absolute_links))
                    if xplink not in px:
                            px.append(xplink)
                    if x.html.find('a.pagination_next'):
                    return p(xplink)
                else:
                    print (z,'\t Page doesnt exist')

            def p(xplink):
                x=se.get(xplink)
                if x.html.find('a.pagination_next'):
                    pagecheck(x.url)
                else:
                    print (z,'\t Pages scrape completed.....')
                return px
                
                
            if __name__ == '__main__':

                #:: movies index
                xa = ['a','z','c']
                pz = []
                for z in xa:

                    url = f'https://isaidubb.co/tamil-atoz-dubbed-movies/{z}'
                    se = HTMLSession()
                    x=se.get(url)
                    px=[];pagecheck(x.url)
                    if px not in pz:
                        pz.extend(px)

                print (pz)

输出将是： [运行] python -u "c:\Users\karthik\lala\FilesDump\movies\pageV1.py" 页面抓取完成..... z 页面不存在 c 页面抓取完成..... ['http://isaidub7.com/tamil-atoz-dubbed-movies/a'、'http://isaidub7.com/tamil-atoz-dubbed-movies/a/2'、'http://isaidub7. com/tamil-atoz-dubbed-movies/a/3','http://isaidub7.com/tamil-atoz-dubbed-movies/a/4','http://isaidub7.com/tamil-atoz-配音电影/z'、'http://isaidub7.com/tamil-atoz-dubbed-movies/c'、'http://isaidub7.com/tamil-atoz-dubbed-movies/c/2'、' http://isaidub7.com/tamil-atoz-dubbed-movies/c/3']

[完成]