Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Too many open file - OpenNMS running over Debian into openvz Virtual Machine

Description

m experimenting the too many open file trouble with 1.8.11 on debian.

I set up the ulimit to:

  1. Maximum file Descriptors + 1to be setted by ulimit -n
    #MAXIMUM_FILE_DESCRIPTORS=81920
    MAXIMUM_FILE_DESCRIPTORS=512000

  1. maximum size of stack segment (in kbytes) to be setted by ulimit -s
    MAXIMUM_SIZE_STACK_SEGMENT=8192

But it seems that I reach the 512.000 open file in 4 days of work.

This is not a special installation....I have provisioned 2000 nodes.

I only poll ICMP every 5 minutes And only on SnmpPrimary Interface but on 600 nodes every 2 minutes.
Yeah I do response time graphs. .

Also I have SnmpPoller active on 13671 snmp interfaces.

And I also do Snmp Data Collection over 18008 interfaces.

So I suspect that there is some leak somewhere!

Here is some simple statistics I'm running...

mar 26 apr 2011, 10.29.09, CEST found open socket 1904 - found open pipe 38022
mar 26 apr 2011, 10.34.11, CEST found open socket 2056 - found open pipe 41142

You see that open socket and open pipe every 5 minutes increase...

On 5 minutes I guess should be stable....

Well, the most important thing: opennms is running

Here is the startup settings:

=== OpenNMS Complimentary Thread Dump ===
------------------------------------------------------------------------------ mar 26 apr 2011, 16.39.25, CEST
begin ulimit settings:
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 256842
nofiles 512000
vmemory(kbytes) unlimited
locks unlimited
end ulimit settings
Executing command: /usr/share/opennms/bin/runjava -r – -Djava.endorsed.dirs=/usr/share/opennms/lib/endorsed -Dopennms.home=/usr/share/opennms -Dcom.sun.management.jmxremote.port=18980 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmx8192m -Djava.net.preferIPv4Stack=true -Dorg.mortbay.jetty.Request.maxFormContentSize=300000 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:LargePageSizeInBytes=4m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote=true -jar /usr/share/opennms/lib/opennms_bootstrap.jar start

Environment

Linux opennms 2.6.32-4-pve #1 SMP Tue Mar 29 09:08:37 CEST 2011 x86_64 GNU/Linux root@opennms:/var/log/opennms/daemon# more /etc/debian_version 6.0.1 Virtual Machine openvz.

Acceptance / Success Criteria

None

Attachments

3
  • 27 Apr 2011, 10:43 AM
  • 27 Apr 2011, 04:35 AM
  • 26 Apr 2011, 02:02 PM

Lucidchart Diagrams

Activity

Show:

Antonio Russo July 7, 2011 at 5:21 AM

I fixed with an Upgrade to java 6. I guess that the java 6 garbage collection that close all the open files.

Alex Bennee July 7, 2011 at 4:23 AM

Was there actually a code changed committed for this or did the bug just "go away" when you increased memory?

Jason Lixfeld May 4, 2011 at 12:19 PM

Ditto on the Java upgrade. Seems to have fixed it. I can't update the heap space though; my machine only has 4GB of memory in it.

Antonio Russo April 30, 2011 at 9:05 AM

I can say now that the upgrade to java 6 fixes the trouble.
No patch applied just upgraded the JVM.

The java 6 required more java_heap_space I moved from 8GBybe of version java 5 to 12 Gbyte required by java 6.

Antonio Russo April 30, 2011 at 9:03 AM

What is changed is that I have to set java_heap_size to 12GByte.
Seems that java 6 requires more memory!

Fixed

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

PagerDuty

Created April 26, 2011 at 2:02 PM
Updated January 27, 2017 at 4:21 PM
Resolved April 30, 2011 at 9:05 AM