如何解决内容获取失败,协议状态为:异常16,lastModified = 0:Http代码= 403,url = https://www.modernfamilydental.net/
我正在对URL url=https://www.modernfamilydental.net/
进行parsechecker
o / p
提取失败,协议状态为:exception(16),lastModified=0: Http code=403,url=https://www.modernfamilydental.net/
我可以知道问题是什么以及如何解决。我尝试更改代理名称,但是没有用。请帮助我。
nutch-site.xml
<property>
<name>http.agent.name</name>
<value>crawlbot</value>
</property>
<property>
<name>plugin.includes</name>
<value>protocol-httpclient|urlfilter-regex|query-(basic|site|url|lang)|indexer-csv|nutch-extensionpoints|protocol-httpclient|urlfilter-regex|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)protocol-http|urlfilter-regex|parse-(html|tika|metatags|text|js|feed)|index-(basic|anchor|more|metadata)</value>
</property>
<property>
<name>db.ignore.external.links</name>
<value>true</value>
</property>
<property>
<name>db.ignore.external.links.mode</name>
<value>byDomain</value>
</property>
<property>
<name>fetcher.server.delay</name>
<value>2</value>
</property>
<property>
<name>fetcher.server.min.delay</name>
<value>0.5</value>
</property>
<property>
<name>fetcher.threads.fetch</name>
<value>400</value>
</property>
<property>
<name>fetcher.max.crawl.delay</name>
<value>10</value>
<description> If the Crawl-Delay in robots.txt is set to greater than this value (in seconds) then the fetcher will skip this page,generating an error report. If set to -1 the fetcher will never skip such pages and will wait the amount of time retrieved from robots.txt Crawl-Delay,however long that might be. </description>
</property>
解决方法
根据您的要求在评论中
如何为Nutch集成代理设置?
有很多免费(如https://www.sslproxies.org/)和付费(您可以在网上找到很多付费代理)代理服务器,可以轻松集成到Nutch。
Nutch(1.16)提供了许多与代理服务器集成相关的配置。
<property>
<name>http.proxy.host</name>
<value>ip-address</value>
<description>The proxy hostname. If empty,no proxy is used.</description>
</property>
<property>
<name>http.proxy.port</name>
<value>proxy port</value>
<description>The proxy port.</description>
</property>
<property>
<name>http.proxy.username</name>
<value>blahblah</value>
<description>Username for proxy. This will be used by
'protocol-httpclient',if the proxy server requests basic,digest
and/or NTLM authentication. To use this,'protocol-httpclient' must
be present in the value of 'plugin.includes' property.
NOTE: For NTLM authentication,do not prefix the username with the
domain,i.e. 'susam' is correct whereas 'DOMAIN\susam' is incorrect.
</description>
</property>
<property>
<name>http.proxy.password</name>
<value>blahblah</value>
<description>Password for proxy. This will be used by
'protocol-httpclient','protocol-httpclient' must
be present in the value of 'plugin.includes' property.
</description>
</property>
<property>
<name>http.proxy.realm</name>
<value></value>
<description>Authentication realm for proxy. Do not define a value
if realm is not required or authentication should take place for any
realm. NTLM does not use the notion of realms. Specify the domain name
of NTLM authentication as the value for this property. To use this,'protocol-httpclient' must be present in the value of
'plugin.includes' property.
</description>
</property>
<property>
<name>http.proxy.type</name>
<value>HTTP</value>
<description>
Proxy type: HTTP or SOCKS (cf. java.net.Proxy.Type).
Note: supported by protocol-okhttp.
</description>
</property>
<property>
<name>http.proxy.exception.list</name>
<value>nutch.org,abc.com</value>
<description>A comma separated list of hosts that don't use the proxy
(e.g. intranets). Example: www.apache.org</description>
</property>
如果您偶然发现 lib-http插件代码,它是所有http库(例如protocal-http,protocal-httpclient,protocal-okhttp等)的接口插件
org.apache.nutch.protocol.http.api.HttpBase
public void setConf(Configuration conf) {
this.conf = conf;
this.proxyHost = conf.get("http.proxy.host");
this.proxyPort = conf.getInt("http.proxy.port",8080);
this.proxyType = Proxy.Type.valueOf(conf.get("http.proxy.type","HTTP"));
this.proxyException = arrayToMap(conf.getStrings("http.proxy.exception.list"));
this.useProxy = (proxyHost != null && proxyHost.length() > 0);
this.timeout = conf.getInt("http.timeout",10000);
.........................................
.........................................
从上面的代码中可以看到Nutch在初始化 HTTPclient 对象时会使用这些配置。
在查看您的 plugin.includes conf后,您正在使用 protocol-httpclient 如果您在 configureClient 方法
中查看**org.apache.nutch.protocol.httpclient.Http**
的代码
This particular specific code will integrate proxy server to httpclient
// HTTP proxy server details
if (useProxy) {
hostConf.setProxy(proxyHost,proxyPort);
if (proxyUsername.length() > 0) {
AuthScope proxyAuthScope = getAuthScope(this.proxyHost,this.proxyPort,this.proxyRealm);
NTCredentials proxyCredentials = new NTCredentials(this.proxyUsername,this.proxyPassword,Http.agentHost,this.proxyRealm);
client.getState().setProxyCredentials(proxyAuthScope,proxyCredentials);
}
}
nutch正在设置代理对象,以便您通过httpclient发出的每个请求都将到达代理服务器。
我建议您将fetcher.server.min.delay增加到2秒,这样可以确保另一端不会被滥用。
出于测试目的,您可以使用this教程
,这是http.agent.version的问题,它们在更改后解决了问题,正在阻止代理版本。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。