SNMP collection of ipNetToMedia table fails

Description

Hello,
I just re-installed a fresh opennms version.
I have two network that I setup for auto discovery.
The first network is ok, snmp is collected and graph are ok.
when the second network is discovered, each host is ok, snmp description of the host is ok but graphes stays empty with NaN.
I then start to have error in postgres :
2015-09-09 22:42:23 CEST [47265-1] opennms@opennms ERROR: operator does not exist: character varying = bytea at character 378
2015-09-09 22:42:23 CEST [47265-2] opennms@opennms HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
2015-09-09 22:42:23 CEST [47265-3] opennms@opennms STATEMENT: select ipnettomed0_.id as id9_, ipnettomed0_.createTime as createTime9_, ipnettomed0_.lastPollTime as lastPoll3_9_, ipnettomed0_.netAddress as netAddress9_, ipnettomed0_.physAddress as physAddr5_9_, ipnettomed0_.sourceIfIndex as sourceIf6_9_, ipnettomed0_.sourceNodeId as sourceNo7_9_ from ipNetToMedia ipnettomed0_ where ipnettomed0_.netAddress=$1 and ipnettomed0_.physAddress=$2
2015-09-09 22:42:23 CEST [47265-4] opennms@opennms ERROR: current transaction is aborted, commands ignored until end of transaction block
2015-09-09 22:42:23 CEST [47265-5] opennms@opennms STATEMENT: SELECT NULL AS TABLE_CAT, n.nspname AS TABLE_SCHEM, c.relname AS TABLE_NAME, CASE n.nspname ~ '^pg_' OR n.nspname = 'information_schema' WHEN true THEN CASE WHEN n.nspname = 'pg_catalog' OR n.nspname = 'information_schema' THEN CASE c.relkind WHEN 'r' THEN 'SYSTEM TABLE' WHEN 'v' THEN 'SYSTEM VIEW' WHEN 'i' THEN 'SYSTEM INDEX' ELSE NULL END WHEN n.nspname = 'pg_toast' THEN CASE c.relkind WHEN 'r' THEN 'SYSTEM TOAST TABLE' WHEN 'i' THEN 'SYSTEM TOAST INDEX' ELSE NULL END ELSE CASE c.relkind WHEN 'r' THEN 'TEMPORARY TABLE' WHEN 'i' THEN 'TEMPORARY INDEX' WHEN 'S' THEN 'TEMPORARY SEQUENCE' WHEN 'v' THEN 'TEMPORARY VIEW' ELSE NULL END END WHEN false THEN CASE c.relkind WHEN 'r' THEN 'TABLE' WHEN 'i' THEN 'INDEX' WHEN 'S' THEN 'SEQUENCE' WHEN 'v' THEN 'VIEW' WHEN 'c' THEN 'TYPE' WHEN 'f' THEN 'FOREIGN TABLE' WHEN 'm' THEN 'MATERIALIZED VIEW' ELSE NULL END ELSE NULL END AS TABLE_TYPE, d.description AS REMARKS FROM pg_catalog.pg_namespace n, pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_description d ON (c.oid = d.objoid AND d.objsubid = 0) LEFT JOIN pg_catalog.pg_class dc ON (d.classoid=dc.oid AND dc.relname='pg_class') LEFT JOIN pg_catalog.pg_namespace dn ON (dn.oid=dc.relnamespace AND dn.nspname='pg_catalog') WHERE c.relnamespace = n.oid AND c.relname LIKE 'PROBABLYNOT' AND (false OR ( c.relkind = 'r' AND n.nspname !~ '^pg_' AND n.nspname <> 'information_schema' ) ) ORDER BY TABLE_TYPE,TABLE_SCHEM,TABLE_NAME

in enlinkd.log I have :
2015-09-09 22:42:23,554 ERROR [DefaultUDPTransportMapping_0.0.0.0/0] o.h.u.JDBCExceptionReporter: ERROR: operator does not exist: character varying = bytea
2015-09-09 22:42:23,554 ERROR [DefaultUDPTransportMapping_0.0.0.0/0] o.h.u.JDBCExceptionReporter: ERROR: operator does not exist: character varying = bytea
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Position: 378
2015-09-09 22:42:23,554 WARN [DefaultUDPTransportMapping_0.0.0.0/0] o.o.n.s.TableTracker: Failed to process response
org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute query; SQL [select ipnettomed0_.id as id9_, ipnettomed0_.createTime as createTime9_, ipnettomed0_.lastPoll
Time as lastPoll3_9_, ipnettomed0_.netAddress as netAddress9_, ipnettomed0_.physAddress as physAddr5_9_, ipnettomed0_.sourceIfIndex as sourceIf6_9_, ipnettomed0_.sourceNodeId as sourceNo7_9
_ from ipNetToMedia ipnettomed0_ where ipnettomed0_.netAddress=? and ipnettomed0_.physAddress=?]; nested exception is org.hibernate.exception.SQLGrammarException: could not execute query
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:642) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:414) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:341) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.findUnique(AbstractDaoHibernate.java:202) ~[opennms-dao-16.0.3.jar:?]
at org.opennms.netmgt.dao.hibernate.IpNetToMediaDaoHibernate.getByNetAndPhysAddress(IpNetToMediaDaoHibernate.java:54) ~[opennms-dao-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl$9.query(EnhancedLinkdServiceImpl.java:765) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl$9.query(EnhancedLinkdServiceImpl.java:761) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.dao.support.UpsertTemplate.doUpsert(UpsertTemplate.java:157) ~[opennms-dao-16.0.3.jar:?]
at org.opennms.netmgt.dao.support.UpsertTemplate.access$000(UpsertTemplate.java:123) ~[opennms-dao-16.0.3.jar:?]
at org.opennms.netmgt.dao.support.UpsertTemplate$1.doInTransaction(UpsertTemplate.java:147) ~[opennms-dao-16.0.3.jar:?]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) ~[org.apache.servicemix.bundles.spring-tx-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.dao.support.UpsertTemplate.execute(UpsertTemplate.java:143) ~[opennms-dao-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl.saveIpNetToMedia(EnhancedLinkdServiceImpl.java:792) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl.store(EnhancedLinkdServiceImpl.java:756) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl$$FastClassBySpringCGLIB$$fe4d2451.invoke(<generated>) ~[org.apache.servicemix.bundles.spring-core-4.0.5.RELEASE_1.jar:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[org.apache.servicemix.bundles.spring-core-4.0.5.RELEASE_1.jar:?]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:640) ~[org.apache.servicemix.bundles.spring-aop-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.enlinkd.EnhancedLinkdServiceImpl$$EnhancerBySpringCGLIB$$5b2ed46e.store(<generated>) ~[org.apache.servicemix.bundles.spring-core-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.enlinkd.NodeDiscoveryIpNetToMedia$1.processIpNetToMediaRow(NodeDiscoveryIpNetToMedia.java:81) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.enlinkd.snmp.IpNetToMediaTableTracker.rowCompleted(IpNetToMediaTableTracker.java:217) ~[opennms-services-16.0.3.jar:?]
at org.opennms.netmgt.snmp.SnmpTableResult.handleCompleteRows(SnmpTableResult.java:99) ~[org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.SnmpTableResult.storeResult(SnmpTableResult.java:75) ~[org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.TableTracker.storeResult(TableTracker.java:106) ~[org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.CollectionTracker.storeResult(CollectionTracker.java:77) ~[org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.ColumnTracker$1.processResponse(ColumnTracker.java:100) ~[org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.TableTracker$CombinedColumnResponseProcessor.processResponse(TableTracker.java:170) [org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.AggregateTracker$ChildTrackerResponseProcessor.processResponse(AggregateTracker.java:165) [org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.SnmpWalker.processResponse(SnmpWalker.java:202) [org.opennms.core.snmp.api-16.0.3.jar:?]
at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker.access$700(Snmp4JWalker.java:48) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker$Snmp4JResponseListener.processResponse(Snmp4JWalker.java:151) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.opennms.netmgt.snmp.snmp4j.Snmp4JWalker$Snmp4JResponseListener.onResponse(Snmp4JWalker.java:180) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.Snmp.processPdu(Snmp.java:1203) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.MessageDispatcherImpl.fireProcessPdu(MessageDispatcherImpl.java:675) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:302) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:373) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:333) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.transport.AbstractTransportMapping.fireProcessMessage(AbstractTransportMapping.java:76) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at org.snmp4j.transport.DefaultUdpTransportMapping$ListenThread.run(DefaultUdpTransportMapping.java:414) [org.opennms.core.snmp.implementations.snmp4j-16.0.3.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: org.hibernate.exception.SQLGrammarException: could not execute query

Any help would be really appreciated !

Environment

Debian jessie postgresql 9.4 fresh install

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Antonio Russo October 17, 2015 at 5:52 AM

Duplicate already Fixed in Foundation

Seth Leger October 6, 2015 at 4:45 PM

This exception appears to be inside Enlinkd so I'm assigning to Antonio to investigate.

zfzefz September 11, 2015 at 4:35 PM

hello
a few more details about that issue, I have enabled full log on postgresql and discovered that in ipnettomedia table I have two bad records :
the physadress for two IP is '....................'
those two interfaces are infiniband device using IPoIB and are secondary interface on those servers.
Probably there should be a test when they get inserted to check that they are an hexadecimal value

Duplicate

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

PagerDuty

Created September 9, 2015 at 5:09 PM
Updated November 9, 2015 at 1:49 PM
Resolved October 17, 2015 at 5:52 AM