Acknowledging alarms with a huge Description causes error when the Alarm-change-notifier plugin is running

Description

After installing and running the alarm-change-notifier plugin, we start receiving the below errors in the alarmd log

alarmd.log

and the below error in the postgresql.log

postgresql.log

This only happens when the alarm-change-notifier plugin is running.
Below error from the opennms gui

Environment

Centos 7, Postgresql 9.3, OpenNMS Horizon 18.0.3

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Seth Leger June 17, 2017 at 10:43 AM

Ditto what Craig said, it appears that the size limit for text types in PostgreSQL is 1GB so the JSON for the old and new alarm values must be at least 500MB. This seems... very excessive.

Craig Gallen June 14, 2017 at 1:06 PM
Edited

So my question is what are 'alarms with a huge Description' Are these supplied OpenNMS alarm definitions or new definitions from the user.
Could we have an example alarm definition which is causing this problem please.

This problem appears to be caused by limitations to the maximum size of a notification within postgresql. As such, there probably isn't much we can do other than advise that alarms should not have unreasonably large alarm descriptions (or any other text field) if the alarm change notifier is used.
The postgresql trigger definitions are defined in
/opennms/features/alarm-change-notifier/main-module/src/main/resources/OSGI-INF/blueprint/mainModuleBlueprint.xml
The trigger takes the old alarm row OLD, and the new alarm row NEW, converts them to json and then concatenates them into a single json message containing the old alarm values and the new alarm values:
temp_new = row_to_json(NEW);
temp_old = row_to_json(OLD);
IF (temp_new::text != temp_old::text) THEN notification=CONCAT('[', new_data::text,',', old_data::text, ']');
PERFORM pg_notify('opennms_alarm_changes',notification);

Clearly in this case, the notification field is too long for pg_notify which causes the error.
We could possibly add a test in the SQL function to reject alarms which generate too large a notification string but I don't know what the limits should be and whether this is actually a widely experienced case.

Seth Leger June 14, 2017 at 12:24 PM

How large is the alarm payload?

Details

Assignee

Reporter

Components

Affects versions

Priority

PagerDuty

Created February 14, 2017 at 10:25 PM
Updated July 26, 2023 at 2:14 PM