It seems that there is another open file bug. Since we updated our installation to 1.10.4 we notice more problems with open Files. We have a ulimit of 15000. In the poller.log I see thousends like this:
java.net.SocketException: Too many open files at java.net.Socket.createImpl(Socket.java:397) at java.net.Socket.connect(Socket.java:527) at org.opennms.netmgt.poller.monitors.HttpMonitor$HttpMonitorClient.connect(HttpMonitor.java:420) at org.opennms.netmgt.poller.monitors.HttpMonitor$HttpMonitorClient.access$000(HttpMonitor.java:298) at org.opennms.netmgt.poller.monitors.HttpMonitor.poll(HttpMonitor.java:148) at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.poll(LatencyStoringServiceMonitorAdaptor.java:104) at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:109) at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:178) at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:292) at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:305) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:263) at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:249) at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:227) at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:312) at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:205) at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:305) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) ....
and
2012-08-10 07:06:47,207 WARN [PollerScheduler-110 Pool-fiber61] BSFMonitor: BSFMonitor poll for service 'Check_Databases' failed with BSFException: exception from Groovy: org.opennms. netmgt.model.events.EventProxyException: Could not connect to event daemon /127.0.0.1:5817 to send event: Connection refused org.apache.bsf.BSFException: exception from Groovy: org.opennms.netmgt.model.events.EventProxyException: Could not connect to event daemon /127.0.0.1:5817 to send event: Connection refused
or
2012-08-09 20:08:48,417 ERROR [Thread-2853478] JavaMailNotificationStrategy: send: Error sending notification. org.opennms.javamail.JavaMailerException: Failed to construct mailer at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:147) at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:162) at org.opennms.netmgt.notifd.JavaMailNotificationStrategy.buildMessage(JavaMailNotificationStrategy.java:86) at org.opennms.netmgt.notifd.JavaMailNotificationStrategy.send(JavaMailNotificationStrategy.java:64) at org.opennms.netmgt.notifd.ClassExecutor.execute(ClassExecutor.java:63) at org.opennms.netmgt.notifd.NotificationTask.run(NotificationTask.java:261) Caused by: java.io.FileNotFoundException: /opt/opennms/etc/javamail-configuration.properties (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at org.opennms.javamail.JavaMailerConfig.getProperties(JavaMailerConfig.java:56) at org.opennms.javamail.JavaMailer.configureProperties(JavaMailer.java:176) at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:145) ... 5 more
Marking fixed in 1.10.5 since all of the file handle fixes were committed in that release.
Seth Leger August 14, 2012 at 11:05 AM
Most of this issue should have been addressed by my changes in OpenNMS 1.10.3. Please check your /opt/opennms/lib directory to ensure that you only have the latest 1.10.4 OpenNMS JARs on your system.
If you are still having problems, I recommend updating to the latest 1.10 snapshot version which has additional fixes for bug . If you want to wait for a stable release, we should be releasing 1.10.5 which will contain these changes by early September at the latest.
Daniel Hammon August 14, 2012 at 3:22 AM
Thanks for the hint. I removed everything out the foreign-sources and its running without any issues since I created this report. We had several http and https detectors running before. So it seems related to your issue.
It seems that there is another open file bug. Since we updated our installation to 1.10.4 we notice more problems with open Files. We have a ulimit of 15000.
In the poller.log I see thousends like this:
java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:397)
at java.net.Socket.connect(Socket.java:527)
at org.opennms.netmgt.poller.monitors.HttpMonitor$HttpMonitorClient.connect(HttpMonitor.java:420)
at org.opennms.netmgt.poller.monitors.HttpMonitor$HttpMonitorClient.access$000(HttpMonitor.java:298)
at org.opennms.netmgt.poller.monitors.HttpMonitor.poll(HttpMonitor.java:148)
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.poll(LatencyStoringServiceMonitorAdaptor.java:104)
at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:109)
at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:178)
at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:292)
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:263)
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:249)
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:227)
at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:312)
at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:205)
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
....
and
2012-08-10 07:06:47,207 WARN [PollerScheduler-110 Pool-fiber61] BSFMonitor: BSFMonitor poll for service 'Check_Databases' failed with BSFException: exception from Groovy: org.opennms.
netmgt.model.events.EventProxyException: Could not connect to event daemon /127.0.0.1:5817 to send event: Connection refused
org.apache.bsf.BSFException: exception from Groovy: org.opennms.netmgt.model.events.EventProxyException: Could not connect to event daemon /127.0.0.1:5817 to send event: Connection refused
or
2012-08-09 20:08:48,417 ERROR [Thread-2853478] JavaMailNotificationStrategy: send: Error sending notification.
org.opennms.javamail.JavaMailerException: Failed to construct mailer
at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:147)
at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:162)
at org.opennms.netmgt.notifd.JavaMailNotificationStrategy.buildMessage(JavaMailNotificationStrategy.java:86)
at org.opennms.netmgt.notifd.JavaMailNotificationStrategy.send(JavaMailNotificationStrategy.java:64)
at org.opennms.netmgt.notifd.ClassExecutor.execute(ClassExecutor.java:63)
at org.opennms.netmgt.notifd.NotificationTask.run(NotificationTask.java:261)
Caused by: java.io.FileNotFoundException: /opt/opennms/etc/javamail-configuration.properties (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at org.opennms.javamail.JavaMailerConfig.getProperties(JavaMailerConfig.java:56)
at org.opennms.javamail.JavaMailer.configureProperties(JavaMailer.java:176)
at org.opennms.javamail.JavaMailer.<init>(JavaMailer.java:145)
... 5 more