SnmpInterfacePoller fails when device returns NoSuchInstance
Description
2011-02-22 08:54:44,017 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmp.snmp4j.Snmp4JStrategy: get: OID: [.1.3.6.1.2.1.2.2.1.7.2, .1.3.6.1. 2.1.2.2.1.7.4, .1.3.6.1.2.1.2.2.1.7.9, .1.3.6.1.2.1.2.2.1.7.8, .1.3.6.1.2.1.2.2.1.7.6, .1.3.6.1.2.1.2.2.1.7.1, .1.3.6.1.2.1.2.2.1.7.3, .1.3.6.1.2.1.2.2.1.7.10, .1.3.6.1.2.1.2.2.1.7.7, .1.3. 6.1.2.1.2.2.1.7.5, .1.3.6.1.2.1.2.2.1.8.2, .1.3.6.1.2.1.2.2.1.8.4, .1.3.6.1.2.1.2.2.1.8.9, .1.3.6.1.2.1.2.2.1.8.8, .1.3.6.1.2.1.2.2.1.8.6, .1.3.6.1.2.1.2.2.1.8.1, .1.3.6.1.2.1.2.2.1.8.3, .1 .3.6.1.2.1.2.2.1.8.10, .1.3.6.1.2.1.2.2.1.8.7, .1.3.6.1.2.1.2.2.1.8.5] for Agent:AgentConfig[Address: /A.B.C.D, ProxyForAddress: null, Port: 161, Community: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, Timeout: 5000, Retries: 12, MaxVarsPerPdu: 3, MaxRepetitions: 2, Max request size: 65535, Version: v2c] 2011-02-22 08:54:44,028 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmp.snmp4j.Snmp4JStrategy: send: Snmp operation successful. Value: [noS uchInstance, 1, 1, 1, 1, 1, 1, 1, 1, 1, noSuchInstance, 1, 1, 1, 1, 1, 1, 1, 1, 1] 2011-02-22 08:54:44,028 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmpinterfacepoller.SnmpPollInterfaceMonitor: got 20 SnmpValues 2011-02-22 08:54:44,029 ERROR [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmpinterfacepoller.SnmpPollInterfaceMonitor: Number operator used on a non-number For input string: "noSuchInstance" 2011-02-22 08:54:44,029 INFO [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: doPoll: PollerMonitor return interface s number: 10 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 2 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down 2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing ..... 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 4 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down 2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing ..... 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 9 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down 2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing ..... 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 8 2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down 2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing.....
<snip>
As shown by the above logs, my Cisco FWSM is returning noSuchInstance, since the interface with ifindex 2 has been deleted (all the interfaces are vlan interfaces, so deleting them is very easy). As you can see, this causes the snmpinterface poller to fail, and all interfaces in the same snmpinterface poll group to fail, even though other interfaces have valid replies.
I have created a patch to return 0 (ie, down) in case of a noSuchInstance reply, which I have attached. I have done some limited testing and it seemed OK. I will be rebuilding my current 1.8.5 version with the patch and testing in production, and I will let you know how it goes.
Environment
Redhat 5
Acceptance / Success Criteria
None
Attachments
2
Lucidchart Diagrams
Activity
Show:
Christian King February 23, 2011 at 8:43 PM
Also, while researching this bug I upgraded to 1.8.9 and it still exhibits this behavior.
2011-02-22 08:54:44,017 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmp.snmp4j.Snmp4JStrategy: get: OID: [.1.3.6.1.2.1.2.2.1.7.2, .1.3.6.1.
2.1.2.2.1.7.4, .1.3.6.1.2.1.2.2.1.7.9, .1.3.6.1.2.1.2.2.1.7.8, .1.3.6.1.2.1.2.2.1.7.6, .1.3.6.1.2.1.2.2.1.7.1, .1.3.6.1.2.1.2.2.1.7.3, .1.3.6.1.2.1.2.2.1.7.10, .1.3.6.1.2.1.2.2.1.7.7, .1.3.
6.1.2.1.2.2.1.7.5, .1.3.6.1.2.1.2.2.1.8.2, .1.3.6.1.2.1.2.2.1.8.4, .1.3.6.1.2.1.2.2.1.8.9, .1.3.6.1.2.1.2.2.1.8.8, .1.3.6.1.2.1.2.2.1.8.6, .1.3.6.1.2.1.2.2.1.8.1, .1.3.6.1.2.1.2.2.1.8.3, .1
.3.6.1.2.1.2.2.1.8.10, .1.3.6.1.2.1.2.2.1.8.7, .1.3.6.1.2.1.2.2.1.8.5] for Agent:AgentConfig[Address: /A.B.C.D, ProxyForAddress: null, Port: 161, Community: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, Timeout: 5000, Retries: 12, MaxVarsPerPdu: 3, MaxRepetitions: 2, Max request size: 65535, Version: v2c]
2011-02-22 08:54:44,028 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmp.snmp4j.Snmp4JStrategy: send: Snmp operation successful. Value: [noS
uchInstance, 1, 1, 1, 1, 1, 1, 1, 1, 1, noSuchInstance, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2011-02-22 08:54:44,028 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmpinterfacepoller.SnmpPollInterfaceMonitor: got 20 SnmpValues
2011-02-22 08:54:44,029 ERROR [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.snmpinterfacepoller.SnmpPollInterfaceMonitor: Number operator used on a
non-number For input string: "noSuchInstance"
2011-02-22 08:54:44,029 INFO [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: doPoll: PollerMonitor return interface
s number: 10
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 2
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down
2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing
.....
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 4
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down
2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing
.....
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 9
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down
2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing
.....
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Working on interface with ifindex: 8
2011-02-22 08:54:44,029 DEBUG [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: Interface PollStatus is Down
2011-02-22 08:54:44,029 WARN [SnmpinterfacepollerScheduler-3000 Pool-fiber16] OpenNMS.SnmpPoller.org.opennms.netmgt.poller.pollables.PollableService: run: SNMP not available, doing nothing.....
<snip>
As shown by the above logs, my Cisco FWSM is returning noSuchInstance, since the interface with ifindex 2 has been deleted (all the interfaces are vlan interfaces, so deleting them is very easy). As you can see, this causes the snmpinterface poller to fail, and all interfaces in the same snmpinterface poll group to fail, even though other interfaces have valid replies.
I have created a patch to return 0 (ie, down) in case of a noSuchInstance reply, which I have attached. I have done some limited testing and it seemed OK. I will be rebuilding my current 1.8.5 version with the patch and testing in production, and I will let you know how it goes.