CRAWING SPIDER
Goal -> Recursively list all links starting from a base URL.
1. Read page HTML.
2. Extract all links.
3. Repeat for each new link that is not already on the list.
#!/usr/bin/env python import re import requests from urllib.parse import urljoin target_url = "http://10.0.0.45/mutillidae/" target_links = [] def extract_links_from(url): response = requests.get(url) return re.findall('(?:href=")(.*?")', response.content.decode()) def crawl(url): href_links = extract_links_from(url) for link in href_links: link = urljoin(url, link) if "#" in link: link = link.split("#")[0] if target_url in link and link not in target_links: target_links.append(link) print(link) crawl(link) crawl(target_url)
The Python program runs perfectly.
http://10.0.0.45/mutillidae/favicon.ico" http://10.0.0.45/mutillidae/styles/global-styles.css" http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css" http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css" http://10.0.0.45/mutillidae/index.PHP?page=home.PHP" http://10.0.0.45/mutillidae/index.PHP?page=login.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=login.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=login.PHP" http://10.0.0.45/mutillidae/set-up-database.PHP" http://10.0.0.45/mutillidae/index.PHP?page=show-log.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=show-log.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=show-log.PHP" http://10.0.0.45/mutillidae/index.PHP?page=captured-data.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=captured-data.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=captured-data.PHP" http://10.0.0.45/mutillidae/index.PHP?page=credits.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=credits.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=credits.PHP" http://10.0.0.45/mutillidae/" http://10.0.0.45/mutillidae/index.PHP?page=user-info.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=user-info.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=user-info.PHP" http://10.0.0.45/mutillidae/index.PHP?page=register.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=register.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=register.PHP" http://10.0.0.45/mutillidae/index.PHP?page=view-someones-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=view-someones-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=view-someones-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?page=add-to-your-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=add-to-your-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=add-to-your-blog.PHP" http://10.0.0.45/mutillidae/index.PHP?page=site-footer-xss-discussion.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=site-footer-xss-discussion.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=site-footer-xss-discussion.PHP" http://10.0.0.45/mutillidae/index.PHP?page=html5-storage.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=html5-storage.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=html5-storage.PHP" http://10.0.0.45/mutillidae/index.PHP?page=capture-data.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=capture-data.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=capture-data.PHP" http://10.0.0.45/mutillidae/index.PHP?page=dns-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=dns-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=dns-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP" http://10.0.0.45/mutillidae/index.PHP?page=password-generator.PHP&username=anonymous" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=password-generator.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=password-generator.PHP" http://10.0.0.45/mutillidae/index.PHP?page=user-poll.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=user-poll.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=user-poll.PHP" http://10.0.0.45/mutillidae/index.PHP?page=set-background-color.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=set-background-color.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=set-background-color.PHP" http://10.0.0.45/mutillidae/index.PHP?page=pen-test-tool-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=pen-test-tool-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=pen-test-tool-lookup.PHP" http://10.0.0.45/mutillidae/index.PHP?page=text-file-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=text-file-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=text-file-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?page=browser-info.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=browser-info.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=browser-info.PHP" http://10.0.0.45/mutillidae/index.PHP?page=source-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=source-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=source-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?page=arbitrary-file-inclusion.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=arbitrary-file-inclusion.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=arbitrary-file-inclusion.PHP" http://10.0.0.45/mutillidae/index.PHP?page=secret-administrative-pages.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=secret-administrative-pages.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=secret-administrative-pages.PHP" http://10.0.0.45/mutillidae/index.PHP?page=framing.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=framing.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=framing.PHP" http://10.0.0.45/mutillidae/framer.html" http://10.0.0.45/mutillidae/index.PHP?page=change-log.htm" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=change-log.htm" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=change-log.htm" http://10.0.0.45/mutillidae/index.PHP?page=installation.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=installation.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=installation.PHP" http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf" http://10.0.0.45/mutillidae/index.PHP?page=documentation/vulnerabilities.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=documentation/vulnerabilities.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=documentation/vulnerabilities.PHP" http://10.0.0.45/mutillidae/index.PHP?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=home.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=home.PHP" http://10.0.0.45/mutillidae/ http://10.0.0.45/mutillidae/?page=add-to-your-blog.PHP" http://10.0.0.45/mutillidae/?page=view-someones-blog.PHP" http://10.0.0.45/mutillidae/?page=show-log.PHP" http://10.0.0.45/mutillidae/?page=text-file-viewer.PHP" http://10.0.0.45/mutillidae/?page=user-info.PHP" http://10.0.0.45/mutillidae/?page=login.PHP" http://10.0.0.45/mutillidae/?page=credits.PHP" http://10.0.0.45/mutillidae/?page=source-viewer.PHP" http://10.0.0.45/mutillidae/index.PHP?page=usage-instructions.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=usage-instructions.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=usage-instructions.PHP" http://10.0.0.45/mutillidae/index.PHP?page=PHP-errors.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=PHP-errors.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=PHP-errors.PHP" http://10.0.0.45/mutillidae/index.PHP?page=notes.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-hints&page=notes.PHP" http://10.0.0.45/mutillidae/index.PHP?do=toggle-security&page=notes.PHP"
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。