OpenNMS won't start after upgrade: error hidden!

Description

Nothing more than some lines within manager.log:

2020-03-24 17:50:26,922 DEBUG [Main] o.o.n.v.Controller: Could not find OpenNMS JVM ("opennms_bootstrap.jar start") among JVMs ("/opt/opennms/lib/opennms_boots
trap.jar -u service:jmx:rmi:///jndi/rmi://127.0.0.1:1099/jmxrmi status")
2020-03-24 17:50:27,004 DEBUG [Main] o.o.n.v.Controller: No PID specified for OpenNMS JVM
2020-03-24 17:50:27,005 DEBUG [Main] o.o.n.v.Controller: Could not attach to JVM, falling back to JMX over RMI
2020-03-24 17:50:27,047 DEBUG [Main] o.e.j.u.log: Logging to org.apache.logging.slf4j.Log4jLogger@16cbc9c1 via org.eclipse.jetty.util.log.Slf4jLog
2020-03-24 17:50:27,052 INFO [Main] o.e.j.u.log: Logging initialized @1759ms to org.eclipse.jetty.util.log.Slf4jLog
2020-03-24 17:50:27,056 DEBUG [Main] jndi: InitialContextFactory.getInitialContext()
2020-03-24 17:50:27,066 DEBUG [Main] jndi: supportDeepBinding=false
2020-03-24 17:50:27,067 DEBUG [Main] jndi: Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@e2b8c21
{{2020-03-24 17:50:27,129 DEBUG [Main] o.o.n.v.StatusGetter: Could not fetch status: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException }}
{{[Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: }}
{{ java.net.ConnectException: Connection refused (Connection refused)]}}

================================================================================
From the upgrade log....

Updating : opennms-core-25.2.1-1.noarch 2/70
warning: /opt/opennms/etc/opennms.properties created as /opt/opennms/etc/opennms.properties.rpmnew
- making symlink for /opt/opennms/logs... done
- making symlink for /opt/opennms/share... done
- moving *.sql.rpmnew files (if any)... done
- checking for old update files... done
- cleaning up $OPENNMS_HOME/data... done*** Installation complete. You must still run the installer at
{{ *** $OPENNMS_HOME/bin/install -dis to be sure your database is up}}
{{ *** to date before you start OpenNMS. See the install guide at}}
{{ *** http://www.opennms.org/wiki/Installation:RPM and the}}
{{ *** release notes for details.}}

...

  1. $OPENNMS_HOME/bin/install -dis

Found upgrade task KscReportsMigrator
Found upgrade task JmxRrdMigratorOffline
Found upgrade task MonitoringLocationsMigratorOffline
Found upgrade task MagicUsersMigratorOffline
Found upgrade task DiscoveryConfigurationLocationMigratorOffline
Found upgrade task DiscoveryConfigurationMigratorOffline
Processing RequisitionsMigratorOffline: Remove non-ip-snmp-primary and non-ip-interfaces from requisitions: NMS-5630, NMS-5571
{{ Task RequisitionsMigratorOffline has been executed at Sun Nov 09 14:24:48 CET 2014}}
Processing SnmpInterfaceRrdMigratorOnline: Merge SNMP Interface directories (Online Version): NMS-6056
{{ Task SnmpInterfaceRrdMigratorOnline requires OpenNMS is running but it is stopped (ignoring)}}
Processing JmxRrdMigratorOffline: Fix the JRB names for the new JMX Collector: NMS-1539, NMS-3485, NMS-4592, NMS-4612, NMS-5247, NMS-5279, NMS-5824
{{ Task JmxRrdMigratorOffline has been executed at Sun Nov 09 14:24:50 CET 2014}}
Processing JettyConfigMigratorOffline: Adds jetty.xml if necessary: NMS-6629
{{ Task JettyConfigMigratorOffline has been executed at Sun Nov 09 14:24:50 CET 2014}}
Processing ServiceConfigMigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12: NMS-6970
{{ Task ServiceConfigMigratorOffline has been executed at Sun Nov 09 14:24:50 CET 2014}}
Processing KscReportsMigrator: Fixes the KSC Reports because of SNMP Interface directories changes
{{ Task KscReportsMigrator has been executed at Thu Aug 07 14:11:10 CEST 2014}}
Processing ServiceConfig1701MigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12: NMS-6970
{{ Task ServiceConfig1701MigratorOffline has been executed at Fri Feb 19 20:16:35 CET 2016}}
Processing DataCollectionConfigMigratorOffline: Modifies the packages names used in etc/datacollection/*.xml. See NMS-7612.
{{ Task DataCollectionConfigMigratorOffline has been executed at Sun Jun 07 16:35:35 CEST 2015}}
Processing EventParameterMigratorOffline: Moves event parameters from the 'eventparms' column to the 'event_parameters' table.
{{ Task EventParameterMigratorOffline has been executed at Sat Oct 21 13:29:55 CEST 2017}}
Processing MonitoringLocationsMigratorOffline: Moves monitoring locations from monitoring-locations.xml into the monitoringlocations tables in the database
{{ Task MonitoringLocationsMigratorOffline has been executed at Sun Jan 03 11:46:58 CET 2016}}
Processing DataCollectionConfigMigrator17Offline: Fixes the missing resource types on datacollection-config.xml. See NMS-7816.
{{ Task DataCollectionConfigMigrator17Offline has been executed at Sun Jan 03 11:46:59 CET 2016}}
Processing EOLServiceConfigMigratorOffline: Fixes service-configuration.xml if necessary when upgrading to 17.0.0: HZN-545
{{ Task EOLServiceConfigMigratorOffline has been executed at Sun Jan 03 11:47:00 CET 2016}}
Processing DiscoveryConfigurationMigratorOffline: Removes the 'threads' attribute from discovery-configuration.xml. See HZN-599.
{{ Task DiscoveryConfigurationMigratorOffline has been executed at Wed May 25 14:56:44 CEST 2016}}
Processing MagicUsersMigratorOffline: Moves security roles from magic-users.properties into the users.xml file: HZN-871
{{ Task MagicUsersMigratorOffline has been executed at Tue May 09 07:38:35 CEST 2017}}
Processing DiscoveryConfigurationLocationMigratorOffline: Changes the name for the default location from 'localhost' to 'Default'. See HZN-940.
{{ Task DiscoveryConfigurationLocationMigratorOffline has been executed at Thu Feb 23 19:59:07 CET 2017}}Upgrade completed successfully!

  1. service opennms start
    Starting opennms (via systemctl): [ OK ]

================================================================================

  1. yum list installed|fgrep opennms

iplike.x86_64 2.0.7-1.el7.centos @opennms-repo-stable-rhel7
jdk1.8.0_144.x86_64 2000:1.8.0_144-fcs @opennms-repo-stable-common
jicmp.x86_64 2.0.5-1.el7.centos @opennms-repo-stable-rhel7
jicmp6.x86_64 2.0.4-1.el7.centos @opennms-repo-stable-rhel7
opennms.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-core.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-minion.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-plugin-ticketer-otrs.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-remote-poller.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-repo-stable.noarch 1.0-109 installed
opennms-webapp-jetty.noarch 25.2.1-1 @opennms-repo-stable-common
opennms-webapp-remoting.noarch 25.2.1-1 @opennms-repo-stable-common{{postgresql12.x86_64 12.2-2PGDG.rhel7 @pgdg12 }}
{{postgresql12-libs.x86_64 12.2-2PGDG.rhel7 @pgdg12 }}
postgresql12-server.x86_64 12.2-2PGDG.rhel7 @pgdg12

 

Environment

CentOS Linux release 7.7.1908 (Core)

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Ronny Trommer March 25, 2020 at 10:18 AM

It seems like you made an update, the rpmnew files in your /opt/opennms directory prevent your OpenNMS from startup. Here are some hints related to upgrades https://opennms.discourse.group/t/guidance-to-migrate-configuration-files-on-update/271.

guenther.schreiner March 24, 2020 at 7:38 PM

Take a look:

[root@server3 lib]# /etc/init.d/opennms start
Starting opennms (via systemctl): {color:#00875a}[ OK ]{color}

 

And another look:

[root@server3 lib]# /etc/init.d/opennms -f -v start
{{Starting OpenNMS: }}
WARNING! You have files that end in .rpmnew in your
OPENNMS_HOME (/opt/opennms) directory.The format of the original files may have changed since
you modified them before installing a new version.
Please double-check that your configuration files are
up-to-date and delete any leftover .rpmnew files or
OpenNMS will not start.

{color:#de350b}                                  [FAILED]{color}

Fixed

Details

Assignee

Reporter

Components

Affects versions

Due date

Priority

PagerDuty

Created March 24, 2020 at 7:09 PM
Updated June 15, 2021 at 9:22 PM
Resolved June 15, 2021 at 1:22 PM