Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Issues

Select view

Select search mode

 
6 of 6

Abnormal persistent file permissions upon upgrade to Meridian2023

Description

Upon upgrade to Meridian 2023, OpenNMS apparently breaks the file permissions or umask. Files are either created/modified to abnormal permission mode (non 644 for files) across multiple directory paths under /opt/opennms/ such as logs/, data/, etc/, share/rrd/, etc. This issue is persistent across restarts. Fixing permission manually either by running fix-permissions or chmod binary may relieve the symptoms for a few minutes but it’s recurring after that. This seems like it’s more likely to occur when coming from an older version of kernel. Updating the kernel version to the latest can’t always fully fix the problem across different environment.

For example,

in logs/

in data/tmp/ some files don’t have permission at all

There are many symptoms that can relate to this issue.

  1. rrd files becomes inaccessible to user opennms so resource graphs are not rendered.

  2. jsp files becomes inaccessible to user opennms and it breaks the web UI. For example,

and throws errors

For more details and longer report, please refer to https://opennms.atlassian.net/browse/SUPPORT-1999?focusedCommentId=101141

It appears that opennms process Umask is abnormal as opposed to the standard 0002.

Environment

RHEL7.9 RHEL8.7 CentOS 7.9

Acceptance / Success Criteria

None

Attachments

2

Details

Assignee

Reporter

Labels

HB Grooming Date

HB Backlog Status

Sprint

Fix versions

Affects versions

Priority

PagerDuty

Created September 22, 2023 at 3:26 PM
Updated January 12, 2024 at 7:54 PM

Activity

Show:

JianYetDecember 5, 2023 at 2:55 PM
Edited

The issue occurred on older Ubuntu release 18.04 as well when running opennms as non-root user after upgrading from Horizon 28 to Horizon 32. The output in the attached file is after implementing the workaround running as root. There’s no selinux running there as far as I can tell.

 

JianYetOctober 26, 2023 at 3:16 PM
Edited

One of the instances have selinux in permissive mode. See https://opennms.atlassian.net/browse/SUPPORT-1999?focusedCommentId=100833. On the call, it seemed that this issue was isolated to opennms process only as the permission issue was not observed when creating the files manually. Consequently, the java process umask was also messed up.

Christian PapeOctober 5, 2023 at 1:21 PM

I mean something like SE Linux or any other Linux hardening mechanism, that may affect file creation, umask or permissions.

JianYetOctober 4, 2023 at 1:21 PM
Edited

None of us was able to reproduce it either in our lab locally. Perhaps it has to do with the kernel version. I’ve seen in some cases updating the kernel version helped but not 100%. For example, we have seen this issue in RHEL7.9 3.10.0-1160.99.1.el7.x86_64,3.10.0-1160.90.1.el7 RHEL8.8 4.18.0-477.10.1.el8_8. Good question though. What security policy are you thinking about? Let me know if you have some ideas how to retrieve this information.

Christian PapeOctober 4, 2023 at 5:59 AM

I wasn’t able to reproduce the behavior. Was anyone of support able to reproduce the problem? Did the reporting customers enabled any security policy in their RHEL installation?