Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

alarmd calls onEvent() multiple times events enriched by Drools

Description

Given a Drools rule that retracts an event, changes its logmessage.dest, and sends it back to eventd, if many events hit this rule in a very short time, alarmd will call onEvent() more than once for each resulting event that's published to the bus. This causes multiple reductions against the alarm for each event, and increments the alarm's counter.

This does not occur if there is a small delay between events; a delay of 1 second was tested. I do not know the threshold at which the problem happens.

A small Drools engine and eventconf is attached that reliably reproduce this problem for me.

To reproduce:

  1. Install the Event configuration file, and add it to your eventconf.xml

  2. Enable Drools in your installation

  3. Install the sample Drools engine in your drools-engine.d directory

  4. Restart OpenNMS

  5. Ensure that send-event.pl is in your PATH

  6. Update the included send_test_events.sh with the nodeId of a valid node in your OpenNMS installation.

  7. Execute the included send_test_events.sh script to send 15 events in short succession.

Results:

An alarm is created, and its Counter increments to some multiple of the events actually reduced. I've seen values between 27-33 for 15 total events. If you enable debug logging on eventd and alarmd, you'll see that each event is published once, but alarmd handles it more than once.

Expected result:

alarmd should only call onEvent() once for each event published to the bus, regardless of how quickly the events are published, and the counter should reflect the actual number of events reduced against that alarm.

Environment

RedHat Enterprise Linux Server release 6.6 x86_64

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Will Keaney September 7, 2017 at 9:36 PM

If, instead of

event.setUei("uei.opennms.org/tests/transformed_eventflood"); event.getLogmsg().setDest("logndisplay"); engine.sendEvent(event);

I first clone the event, and then modify and send the clone, the phantom event reductions cease to happen:

retract(event); final String serializedEvent = JaxbUtils.marshal(event); final Event clonedEvent = JaxbUtils.unmarshal(Event.class, serializedEvent); clonedEvent.setUei("uei.opennms.org/tests/transformed_eventflood"); clonedEvent.getLogmsg().setDest("logndisplay"); engine.sendEvent(clonedEvent);

Will Keaney September 7, 2017 at 8:48 PM

The number of false event reductions seems to be related to how quickly alarmd responds to inbound events - with alarmd debugging enabled, I get fewer false reductions.

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

PagerDuty

Created September 7, 2017 at 8:21 PM
Updated September 21, 2021 at 6:25 PM