HTTP Collector throwing exception "Host must be set to create a host URL"

Description

The following stack trace is being logged in collectd.log when attempting to use a HTTP Collector.

2011-02-17 18:24:38,361 ERROR [CollectdScheduler-50 Pool-fiber2] CollectableService: An undeclared throwable was caught during data collection for interface <ip-address>/Apache-Stats
org.opennms.netmgt.collectd.CollectionException: An undeclared throwable was caught during data collection for interface <ip-address>/Apache-Stats
at org.opennms.netmgt.collectd.CollectableService.doCollection(CollectableService.java:408)
at org.opennms.netmgt.collectd.CollectableService.run(CollectableService.java:314)
at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:295)
at org.opennms.core.concurrent.RunnableConsumerThreadPool$FiberThreadImpl.run(RunnableConsumerThreadPool.java:427)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.IllegalStateException: Host must be set to create a host URL
at org.apache.commons.httpclient.HostConfiguration.getHostURL(HostConfiguration.java:316)
at org.opennms.netmgt.collectd.HttpCollector.persistResponse(HttpCollector.java:466)
at org.opennms.netmgt.collectd.HttpCollector.doCollection(HttpCollector.java:253)
at org.opennms.netmgt.collectd.HttpCollector.access$000(HttpCollector.java:98)
at org.opennms.netmgt.collectd.HttpCollector$HttpCollectionSet.collect(HttpCollector.java:173)
at org.opennms.netmgt.collectd.HttpCollector.collect(HttpCollector.java:132)
at org.opennms.netmgt.collectd.CollectionSpecification.collect(CollectionSpecification.java:277)
at org.opennms.netmgt.collectd.CollectableService.doCollection(CollectableService.java:380)
... 4 more

I have replaced the IP address with <ip-address> in this bug for privacy. Below is an excerpt from http-datacollection-config.xml

<http-collection name="Apache-Stats">
<rrd step="300">
<rra>RRA:AVERAGE:0.5:1:8928</rra>
<rra>RRA:AVERAGE:0.5:12:8784</rra>
<rra>RRA:MIN:0.5:12:8784</rra>
<rra>RRA:MAX:0.5:12:8784</rra>
</rrd>
<uris>
<uri name="apache">
<url path="/server-status/?auto"
user-agent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"
matches="(?s).?Total\sAccesses:\s([0-9]+).?Total\skBytes:\s([0-9]).*?CPULoad:\s([0-9\.]).?Uptime:\s([0-9]+).?ReqPerSec:\s([0-9\.]).*?BytesPerSec:\s([0-9\.]).?BytesPerReq:\s([0-9\.]+).?BusyWorkers:\s([0-9]).*?IdleWorkers:\s([0-9]).*" response-range="100-399" >
</url>
<attributes>
<attrib alias="TotalAccesses" match-group="1" type="gauge32"/>
<attrib alias="TotalkBytes" match-group="2" type="gauge32"/>
<attrib alias="CPULoad" match-group="3" type="gauge32"/>
<attrib alias="Uptime" match-group="4" type="gauge32"/>
<attrib alias="ReqPerSec" match-group="5" type="gauge32"/>
<attrib alias="BytesPerSec" match-group="6" type="gauge32"/>
<attrib alias="BytesPerReq" match-group="7" type="gauge32"/>
<attrib alias="BusyWorkers" match-group="8" type="gauge32"/>
<attrib alias="IdleWorkers" match-group="9" type="gauge32"/>
</attributes>
</uri>
</uris>
</http-collection>

And below is the config from capsd-configuration.xml:

<protocol-plugin protocol="Apache-Stats" class-name="org.opennms.netmgt.capsd.plugins.HttpPlugin" scan="on" user-defined="false">
<property key="port" value="80" />
<property key="timeout" value="3000" />
<property key="retry" value="2" />
<property key="url" value="/server-status/?auto" />
</protocol-plugin>

And from collectd-configuration.xml:

Thanks
Paul

Environment

CentOS 5.5

Acceptance / Success Criteria

None

Linked issues

duplicates

NMS-4147

HTTP data collection causes IllegalStateException, "Host must be set to create a host URL"

Lucidchart Diagrams

Activity

Show:

Ron Roskens November 6, 2012 at 12:35 PM

I think this was fixed with commit 1f3d2e14 back on Tue Aug 3 2010.

HttpCollector.doCollection(collectionSet, collectionResource) method no longer throws a new HttpCollectorException with an HttpClient object attached to it. So when this thrown exception gets logged inside CollectableService.doCollection() it does not cause a new java.lang.IllegalStateException to be thrown.

Seth Leger April 5, 2012 at 2:24 PM

I enhanced the HttpCollectorTest in 1.11 so that it tries to collect from a system while using a malformed regex and it does not trigger this error. Instead, it throws a PatternSyntaxException which makes sense and should point users at regex problems in the config.

We still have occasional reports of this error occurring so I'm going to leave the issue open. We need to figure out how to reproduce it and add a test case for this to HttpCollectorTest.

Alejandro Galue September 29, 2011 at 7:48 AM

I tried the sample configuration on 1.9.92-SNAPSHOT and 1.8.15-SNAPSHOT, and it works for me on both.

Here are the details (I'm using a provisioned node with the same configuration on both instances of OpenNMS):

<detector class="org.opennms.netmgt.provision.detector.simple.HttpDetector" name="Apache-Stats">
   <parameter value="/server-status/?auto" key="url"/>
</detector>

<service name="Apache-Stats" interval="30000" user-defined="false" status="on">
   <parameter key="collection" value="Apache-Stats"/>
   <parameter key="thresholding-enabled" value="false"/>
</service>
...
<collector service="Apache-Stats" class-name="org.opennms.netmgt.collectd.HttpCollector"/>

<http-collection name="Apache-Stats">
   <rrd step="30">
     <rra>RRA:AVERAGE:0.5:1:8928</rra>
     <rra>RRA:AVERAGE:0.5:12:8784</rra>
     <rra>RRA:MIN:0.5:12:8784</rra>
     <rra>RRA:MAX:0.5:12:8784</rra>
   </rrd>
   <uris>
     <uri name="apache">
       <url path="/server-status/?auto" 
            user-agent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412" 
            matches="(?s).*?Total\sAccesses:\s([0-9]+).*?Total\skBytes:\s([0-9]+).*?CPULoad:\s([0-9\.]+).*?Uptime:\s([0-9]+).*?ReqPerSec:\s([0-9\.]+).*?BytesPerSec:\s([0-9\.]+).*?BytesPerReq:\s([0-9\.]+).*?BusyWorkers:\s([0-9]+).*?IdleWorkers:\s([0-9]+).*" response-range="100-399" >
       </url>
       <attributes>
         <attrib alias="TotalAccesses" match-group="1" type="gauge32"/>
         <attrib alias="TotalkBytes" match-group="2" type="gauge32"/>
         <attrib alias="CPULoad" match-group="3" type="gauge32"/>
         <attrib alias="Uptime" match-group="4" type="gauge32"/>
         <attrib alias="ReqPerSec" match-group="5" type="gauge32"/>
         <attrib alias="BytesPerSec" match-group="6" type="gauge32"/>
         <attrib alias="BytesPerReq" match-group="7" type="gauge32"/>
         <attrib alias="BusyWorkers" match-group="8" type="gauge32"/>
         <attrib alias="IdleWorkers" match-group="9" type="gauge32"/>
       </attributes>
     </uri>
   </uris>
</http-collection>

Here is part of the logs from 1.8.15-SNAPSHOT:

2011-09-29 07:11:22,551 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: doSubs: getHost substituted as "192.168.0.8"
2011-09-29 07:11:22,551 INFO  [CollectdScheduler-50 Pool-fiber0] HttpCollector: doCollection: collecting for client: org.apache.commons.httpclient.HttpClient@5e707dc9 using method: org.apache.commons.httpclient.methods.GetMethod@2b52478f
2011-09-29 07:11:22,574 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse:
2011-09-29 07:11:22,574 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: responseBody = Total Accesses: 242
2011-09-29 07:11:22,574 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: getmatches = (?s).*?Total\sAccesses:\s([0-9]+).*?Total\skBytes:\s([0-9]+).*?CPULoad:\s([0-9\.]+).*?Uptime:\s([0-9]+).*?ReqPerSec:\s([0-9\.]+).*?BytesPerSec:\s([0-9\.]+).*?BytesPerReq:\s([0-9\.]+).*?BusyWorkers:\s([0-9]+).*?IdleWorkers:\s([0-9]+).*
2011-09-29 07:11:22,574 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: flags = 0
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: found matching attributes: true
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: TotalAccesses:gauge32:242
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: TotalkBytes:gauge32:6
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: CPULoad:gauge32:0.137046
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: Uptime:gauge32:2532
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: ReqPerSec:gauge32:0.0955766
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: BytesPerSec:gauge32:2.42654
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: BytesPerReq:gauge32:25.3884
2011-09-29 07:11:22,575 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: BusyWorkers:gauge32:1
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector: processResponse: adding found numeric attribute: HttpAttribute: IdleWorkers:gauge32:7
2011-09-29 07:11:22,576 INFO  [CollectdScheduler-50 Pool-fiber0] OneToOnePersister: Persisting data for resource org.opennms.netmgt.collectd.HttpCollector$HttpCollectionResource@26d81381
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] AttributeGroup: Visiting Group AttrGroupType[name=apache, ifType=all] for org.opennms.netmgt.collectd.HttpCollector$HttpCollectionResource@26d81381
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] AttributeGroup: AttrGroupType[name=apache, ifType=all] for org.opennms.netmgt.collectd.HttpCollector$HttpCollectionResource@26d81381.shouldPersist = true
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: TotalkBytes:gauge32:6
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: BusyWorkers:gauge32:1
2011-09-29 07:11:22,576 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: Uptime:gauge32:2532
2011-09-29 07:11:22,577 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: IdleWorkers:gauge32:7
2011-09-29 07:11:22,577 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: CPULoad:gauge32:0.137046
2011-09-29 07:11:22,577 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: BytesPerReq:gauge32:25.3884
2011-09-29 07:11:22,577 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: BytesPerSec:gauge32:2.42654
2011-09-29 07:11:22,577 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: ReqPerSec:gauge32:0.0955766
2011-09-29 07:11:22,578 DEBUG [CollectdScheduler-50 Pool-fiber0] HttpCollector$HttpCollectionAttribute: Visiting attribute HttpAttribute: TotalAccesses:gauge32:242

The relevant part of the logs on 1.9.92-SNAPSHOT is very similar.

Is there any additional stuff you want me to try ?

Tarus Balog September 28, 2011 at 5:27 AM
Edited

Actually, the issue is that if the is a failure in the regex, the collector will throw the error:

2011-09-28 11:20:12,651 DEBUG [CollectdScheduler-50 Pool-fiber2] HttpCollector: processResponse: found matching attributes: false
2011-09-28 11:20:12,651 ERROR [CollectdScheduler-50 Pool-fiber2] CollectableService: An undeclared throwable was caught during data collection for interface 212.218.252.54/OpenNMS-Login
org.opennms.netmgt.collectd.CollectionException: An undeclared throwable was caught during data collection for interface 212.218.252.54/OpenNMS-Login
at org.opennms.netmgt.collectd.CollectableService.doCollection(CollectableService.java:414)
at org.opennms.netmgt.collectd.CollectableService.run(CollectableService.java:316)
at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:295)
at org.opennms.core.concurrent.RunnableConsumerThreadPool$FiberThreadImpl.run(RunnableConsumerThreadPool.java:427)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalStateException: Host must be set to create a host URL

Since the default error level is set to WARN, you'll only see the error for the URL and not that the regex failed. I think this should be corrected.

Tarus Balog September 28, 2011 at 5:15 AM

This is definitely broken in 1.8.14. Trying to run the server-status?auto example in class and it fails with the same host must be set log.

Details
Assignee
Unassigned
Reporter
Paul Vitty
Labels
RB
Components
Data Collection - HTTP
Affects versions
1.8.9
Priority
Minor

PagerDuty

Created February 17, 2011 at 1:35 PM

Updated September 21, 2021 at 6:24 PM

HTTP Collector throwing exception "Host must be set to create a host URL"

Description

Environment

Acceptance / Success Criteria

Linked issues

duplicates

Lucidchart Diagrams

Activity

Ron Roskens November 6, 2012 at 12:35 PM

Seth Leger April 5, 2012 at 2:24 PM

Alejandro Galue September 29, 2011 at 7:48 AM

Tarus Balog September 28, 2011 at 5:27 AM
Edited

Tarus Balog September 28, 2011 at 5:15 AM

Details
Assignee
Unassigned
Reporter
Paul Vitty
Labels
RB
Components
Data Collection - HTTP
Affects versions
1.8.9
Priority
Minor

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

PagerDuty

PagerDuty

Flag notifications

Something's gone wrong

Something's gone wrong

HTTP Collector throwing exception "Host must be set to create a host URL"

Description

Environment

Acceptance / Success Criteria

Linked issues

duplicates

Lucidchart Diagrams

Activity

Ron Roskens November 6, 2012 at 12:35 PM

Seth Leger April 5, 2012 at 2:24 PM

Alejandro Galue September 29, 2011 at 7:48 AM

Tarus Balog September 28, 2011 at 5:27 AMEdited

Tarus Balog September 28, 2011 at 5:15 AM

DetailsAssigneeUnassignedUnassignedReporterPaul VittyPaul VittyLabelsRBComponentsData Collection - HTTPAffects versions1.8.9PriorityMinor

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

PagerDutyPagerDuty Incident

PagerDuty

Flag notifications

Something's gone wrong

Something's gone wrong

Tarus Balog September 28, 2011 at 5:27 AM
Edited

Details
Assignee
Unassigned
Reporter
Paul Vitty
Labels
RB
Components
Data Collection - HTTP
Affects versions
1.8.9
Priority
Minor

PagerDuty