Using IPLIKE filtering throws exception on CentOS

Description

AoA;

While creating a Notification, OpenNMS resulted in error when I clicked either Validate rule results or Skip results validation. Without much effort, it was revealed that only rules containing "IPLIKE" operator cause this issue, e.g IPADDR IPLIKE ... or any other form of ip mask - even a proper ip address with IPLIKE - gives the error. The error is: The OpenNMS Web User Interface Has Experienced an Error.

[opennms@localhost ~]$ yum list | grep iplike
iplike.i386 2.0.2-1 installed
iplike.x86_64 2.0.2-1 installed

[opennms@localhost ~]$ cat /etc/redhat-release
CentOS release 5.5 (Final)

System Details are :

OpenNMS Version: 1.8.13
Java Version: 1.6.0_27 Sun Microsystems Inc.
Java Virtual Machine: 20.2-b06 Sun Microsystems Inc.
Operating System: Linux 2.6.18-194.el5 (amd64)
Servlet Container: jetty/6.1.24 (Servlet Spec 2.5)
User Agent: Mozilla/5.0 (X11; Linux i686; rv:6.0.2) Gecko/20100101 Firefox/6.0.2

Following is revealed under Exception Trace:

java.io.EOFException
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:276)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
at com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
at org.opennms.netmgt.config.NotificationManager.getServiceNames(NotificationManager.java:854)
at org.apache.jsp.admin.notification.noticeWizard.buildRule_jsp.buildServiceOptions(org.apache.jsp.admin.notification.noticeWizard.buildRule_jsp:29)
at org.apache.jsp.admin.notification.noticeWizard.buildRule_jsp._jspService(org.apache.jsp.admin.notification.noticeWizard.buildRule_jsp:230)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at org.extremecomponents.table.filter.AbstractExportFilter.doFilter(AbstractExportFilter.java:49)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.opennms.web.StoreRequestPropertiesFilter.doFilter(StoreRequestPropertiesFilter.java:80)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
at org.opennms.web.springframework.security.AuthFilterEnabler.doFilterHttp(AuthFilterEnabler.java:110)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.ui.SessionFixationProtectionFilter.doFilterHttp(SessionFixationProtectionFilter.java:67)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Hope the info provided helps resolve the issue

PS
This is my first such issue-report so I shall follow the post so as to provide any other details.

Environment

CentOS 5.5, OpenNMS 1.8.13

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Arshad Parvez September 20, 2011 at 11:07 PM

Yes, that is the issue. Am pasting the results below:

opennms=# select * from ipinterface where iplike(ipaddr, '...');
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>
!> \q

-----------
[root@localhost bin]# ./install -dis
==============================================================================
OpenNMS Installer
==============================================================================

Configures PostgreSQL tables, users, and other miscellaneous settings.

  • searching for jicmp:

  • trying to load /usr/lib64/libjicmp.so: OK

  • searching for jrrd:

  • trying to load /usr/lib64/libjrrd.so: NO

  • trying to load /usr/java/jdk1.6.0_27/jre/lib/amd64/server/libjrrd.so: NO

  • trying to load /usr/java/jdk1.6.0_27/jre/lib/amd64/libjrrd.so: NO

  • trying to load /usr/java/jdk1.6.0_27/jre/../lib/amd64/libjrrd.so: NO

  • trying to load /libjrrd.so: NO

  • trying to load /opt/opennms/lib/libjrrd.so: NO

  • trying to load /opt/opennms/lib/linux64/libjrrd.so: NO

  • trying to load /usr/java/packages/lib/amd64/libjrrd.so: NO

  • trying to load /usr/lib64/libjrrd.so: NO

  • trying to load /lib64/libjrrd.so: NO

  • trying to load /lib/libjrrd.so: NO

  • trying to load /usr/lib/libjrrd.so: NO

  • trying to load /usr/lib/jni/libjrrd.so: NO

  • trying to load /usr/lib/libjrrd.so: NO

  • trying to load /usr/local/lib/libjrrd.so: NO

  • trying to load /opt/NMSjicmp/lib/32/libjrrd.so: NO

  • trying to load /opt/NMSjicmp/lib/64/libjrrd.so: NO

  • Failed to load the optional jrrd library.

  • This error is not fatal, since jrrd is only required for optional features.

  • For more information, see http://www.opennms.org/index.php/jrrd

  • using SQL directory... /opt/opennms/etc

  • using create.sql... /opt/opennms/etc/create.sql

  • using 'postgres' as the PostgreSQL user for OpenNMS

  • using 'opennms' as the PostgreSQL database name for OpenNMS

  • checking if database "opennms" is unicode... ALREADY UNICODE

  • Checking for old import files in /opt/opennms/etc... DONE

  • Migrating/creating database:
    Sep 21, 2011 6:57:17 AM liquibase.database.template.JdbcTemplate comment
    INFO: Lock Database
    Sep 21, 2011 6:57:17 AM liquibase.lock.LockHandler acquireLock
    INFO: Successfully acquired change log lock
    Sep 21, 2011 6:57:18 AM liquibase.database.AbstractDatabase getRanChangeSetList
    INFO: Reading from databasechangelog
    Sep 21, 2011 6:57:26 AM liquibase.database.template.JdbcTemplate comment
    INFO: Release Database Lock
    Sep 21, 2011 6:57:26 AM liquibase.lock.LockHandler releaseLock
    INFO: Successfully released change log lock

  • checking if iplike is usable... NO

  • removing existing iplike definition (if any)... FAILED
    Exception in thread "main" org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:714)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:230)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:191)
    at org.opennms.netmgt.dao.db.InstallerDb.dropExistingIpLike(InstallerDb.java:534)
    at org.opennms.netmgt.dao.db.InstallerDb.updateIplike(InstallerDb.java:469)
    at org.opennms.install.Installer.install(Installer.java:278)
    at org.opennms.install.Installer.main(Installer.java:931)

Matt Brozowski September 20, 2011 at 3:43 PM
Edited

This is a symptom of the database connection being closed on the server. Please make sure your iplike works in your database by doing

I also recommend running /opt/opennms/bin/install -dis

this will verify the iplike function

Matt Brozowski September 20, 2011 at 3:43 PM

I think this is a system configuration issue since it happens only to this user

Arshad Parvez September 20, 2011 at 5:53 AM

Google has got me to a similar issue (I hardly understand how the issue was resolved, though)
http://markmail.org/message/ne3oqrdwvurbfaav
Please note that there is a typo in the filter statement in the post but its correct in the logs

Arshad Parvez September 20, 2011 at 1:33 AM

Could it be due to some incompatibilities that mostly happen on 64-bit platforms? I am using a 64-bit machine.

Configuration

Details

Assignee

Reporter

Components

Affects versions

Priority

PagerDuty

Created September 14, 2011 at 3:38 AM
Updated September 20, 2011 at 11:07 PM
Resolved September 20, 2011 at 3:43 PM