null value in column "eventlog" PSQLException

Description

Eventd is frequently logging this exception. Please be advised that some information has been redacted in the below snippet. The full log can be provided upon request or visit SUPPORT ticket https://opennms.atlassian.net/browse/SUPPORT-2033.

2023-08-04 19:25:37,973 WARN  [EventIpcManagerDefaultImpl-Thread-5-of-5] o.h.u.JDBCExceptionReporter: SQL Error: 0, SQLState: 23502
2023-08-04 19:25:37,973 ERROR [EventIpcManagerDefaultImpl-Thread-5-of-5] o.h.u.JDBCExceptionReporter: ERROR: null value in column "eventlog" of relation "events" violates not-null constraint
  Detail: Failing row contains (11195107, uei.opennms.org/nodes/nodeLostService, 14821, (redacted), (redacted), OpenNMS.Poller.DefaultPollContext, (10.x.y.z), null, 41, null, 2023-08-04 19:25:37.972+01, <p>A Juniper-PowerSupply outage was identified on interface 10.2..., null, Unknown, 5, null, null, null, source_type=(redacted);item_type=Router;item_name=acc115..., null, null, null, null, null, null, null, null, null, null, null, null, null, null, a48ef069-336a-4a10-ad76-334a518a9aa2).
2023-08-04 19:25:37,974 WARN  [EventIpcManagerDefaultImpl-Thread-5-of-5] o.o.n.e.DefaultEventHandlerImpl: Unknown exception processing event with processor org.opennms.netmgt.eventd.processor.HibernateEventWriter@6ec0c8b4; not processing with any later processors.
org.springframework.dao.DataIntegrityViolationException: could not insert: [org.opennms.netmgt.model.OnmsEvent]; SQL [insert into events (alarmId, systemId, eventAckTime, eventAckUser, eventAutoAction, eventCorrelation, eventCreateTime, eventDescr, eventDisplay, eventForward, eventHost, eventLog, eventLogGroup, eventLogMsg, eventMouseOverText, eventNotification, eventOperAction, eventOperActionMenuText, eventOperInstruct, eventPathOutage, eventSeverity, eventSnmp, eventSnmpHost, eventSource, eventSuppressedCount, eventTTicket, eventTTicketState, eventTime, eventUei, ifIndex, ipAddr, nodeId, serviceId, eventId) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [eventlog" of relation "events]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [org.opennms.netmgt.model.OnmsEvent]
	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:659) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
	at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:800) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
	at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:670) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
	at org.opennms.netmgt.eventd.processor.HibernateEventWriter.process(HibernateEventWriter.java:186) ~[org.opennms.features.events.daemon-2023.1.5.jar:?]
	at org.opennms.netmgt.eventd.processor.HibernateEventWriter.process(HibernateEventWriter.java:160) ~[org.opennms.features.events.daemon-2023.1.5.jar:?]
	at org.opennms.netmgt.eventd.DefaultEventHandlerImpl$EventHandlerRunnable.run(DefaultEventHandlerImpl.java:174) [org.opennms.features.events.daemon-2023.1.5.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124) [opennms-util-2023.1.5.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.hibernate.exception.ConstraintViolationException: could not insert: [org.opennms.netmgt.model.OnmsEvent]
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:96) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2455) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2875) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:662) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
	... 10 more
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "eventlog" of relation "events" violates not-null constraint
  Detail: Failing row contains (11195107, uei.opennms.org/nodes/nodeLostService, 14821, 2023-08-04 19:25:37.967+01, (redacted), OpenNMS.Poller.DefaultPollContext, (redacted), null, 41, null, 2023-08-04 19:25:37.972+01, <p>A Juniper-PowerSupply outage was identified on interface 10.2..., null, Unknown, 5, null, null, null, source_type=(redacted);item_type=Router;item_name=acc115..., null, null, null, null, null, null, null, null, null, null, null, null, null, null, a48ef069-336a-4a10-ad76-334a518a9aa2).
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[postgresql-42.5.4.jar:42.5.4]
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:152) ~[postgresql-42.5.4.jar:42.5.4]
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[HikariCP-5.0.1.jar:?]
	at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2435) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2875) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
	at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:662) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
	... 10 more

Environment

PostgreSQL 15.3

Acceptance / Success Criteria

None

Activity

fooker September 4, 2023 at 12:59 PM

Updated PR: https://github.com/OpenNMS/opennms/pull/6604

fooker August 24, 2023 at 2:01 PM

PR: https://github.com/OpenNMS/opennms/pull/6577

fooker August 24, 2023 at 1:51 PM

This seems to happen when the user defines an event which is not related to traps but sets the dest attribute of the logmsg element in this particular message to discardtraps (or any other value that has no assigned meaning to the normal event processing).

Looks like our docs mention the discardtraps value as a valid one but do not explicitly mention that this is only valid for events related to traps.

JianYet August 24, 2023 at 1:15 PM

@fooker Dump is provided from ticket https://opennms.atlassian.net/browse/SUPPORT-2033. Let me know if you can’t access it.

fooker August 24, 2023 at 11:22 AM

@JianYet I have a vague idea why this is happening. Can you provide a dump of etc/events so I can verify?

Fixed

Details
Assignee
fooker
Reporter
JianYet
Labels
support
HB Grooming Date
Aug 15, 2023
HB Backlog Status
Refined Backlog
Sprint
None
Fix versions
Meridian-2023.1.7
Affects versions
Meridian-2023.1.5
Priority
Medium

PagerDuty

Created August 4, 2023 at 7:14 PM

Updated September 6, 2023 at 9:38 AM

Resolved September 6, 2023 at 9:38 AM

null value in column "eventlog" PSQLException

Description

Environment

Acceptance / Success Criteria

Activity

fooker September 4, 2023 at 12:59 PM

fooker August 24, 2023 at 2:01 PM

fooker August 24, 2023 at 1:51 PM

JianYet August 24, 2023 at 1:15 PM

fooker August 24, 2023 at 11:22 AM

Details
Assignee
fooker
Reporter
JianYet
Labels
support
HB Grooming Date
Aug 15, 2023
HB Backlog Status
Refined Backlog
Sprint
None
Fix versions
Meridian-2023.1.7
Affects versions
Meridian-2023.1.5
Priority
Medium

Details

Assignee

Reporter

Labels

HB Grooming Date

HB Backlog Status

Sprint

Fix versions

Affects versions

Priority

PagerDuty

PagerDuty

Flag notifications

Something's gone wrong

Something's gone wrong

null value in column "eventlog" PSQLException

Description

Environment

Acceptance / Success Criteria

Activity

fooker September 4, 2023 at 12:59 PM

fooker August 24, 2023 at 2:01 PM

fooker August 24, 2023 at 1:51 PM

JianYet August 24, 2023 at 1:15 PM

fooker August 24, 2023 at 11:22 AM

DetailsAssigneefookerfookerReporterJianYetJianYetLabelssupportHB Grooming DateAug 15, 2023HB Backlog StatusRefined BacklogSprintNone+1Fix versionsMeridian-2023.1.7Affects versionsMeridian-2023.1.5PriorityMedium

Details

Assignee

Reporter

Labels

HB Grooming Date

HB Backlog Status

Sprint

Fix versions

Affects versions

Priority

PagerDutyPagerDuty Incident

PagerDuty

Flag notifications

Something's gone wrong

Something's gone wrong

Details
Assignee
fooker
Reporter
JianYet
Labels
support
HB Grooming Date
Aug 15, 2023
HB Backlog Status
Refined Backlog
Sprint
None
Fix versions
Meridian-2023.1.7
Affects versions
Meridian-2023.1.5
Priority
Medium

PagerDuty