Lock contention when processing large volume of REST API requests

Description

The following stack trace was identified as a bottleneck when running performance tests:

This write lock is invoked in the path of validating user credentials and creates a source of contention for Jetty's worker threads: https://github.com/OpenNMS/opennms/blob/opennms-28.1.1-1/opennms-config/src/main/java/org/opennms/netmgt/config/UserManager.java#L1177

Overview of system threads:

Acceptance / Success Criteria

None

Attachments

2

Lucidchart Diagrams

Activity

Jesse White October 14, 2021 at 6:00 PM

Jesse White October 14, 2021 at 12:45 AM

Related doUpdate() code: https://github.com/OpenNMS/opennms/blob/opennms-28.1.1-1/opennms-config/src/main/java/org/opennms/netmgt/config/UserFactory.java#L191

We should A) avoid a write lock in this critical path unless we are in fact actually updating the file, and B) we should not have to read the last modified time on every call (and only do so after some reasonable delay).

We have a better version of this pattern already in this class: https://github.com/OpenNMS/opennms/blob/opennms-28.1.1-1/core/spring/src/main/java/org/opennms/core/spring/FileReloadContainer.java

Fixed

Details

Assignee

Reporter

Fix versions

Affects versions

Priority

PagerDuty

Created October 14, 2021 at 12:38 AM
Updated April 27, 2022 at 4:38 PM
Resolved October 18, 2021 at 10:04 PM