Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Default netsnmp group ns-dskPercentNode high threshold raises alerts for devices that are not real physical devices that always report 100% inode utilization

Description

Net-SNMP, at least with Ubuntu 10.10 servers (Net-SNMP 5.4.3~dfsg-2.2ubuntu1), returns 100% inode utilization for the Net-SNMP ns-dskPercentNode value for several virtual file systems (e.g. /proc, /sys, /dev). This causes the related threshold to raise false positive space warnings.

The similar check in the hrstorage threshold group has a filter that excludes these types of file systems:

e.g. <resource-filter field="hrStorageType">^\.1\.3\.6\.1\.2\.1\.25\.2\.1\.4$</resource-filter>

However, there is no similar filter in the netsnmp threshold group.

In looking through the MIB, there seems to be no similar Net-SNMP data to indicate if a device is a real, physical media device. The closest I could find was the physical device path, which starts with /dev/ for real devices. I added that as suggested change below:

<threshold
description="Trigger an alert when the percentage of inodes used on any disk with device name starting with '/dev/' (that is, an actual physical device) reaches or passes 90% for two measurement intervals"
type="high" ds-type="dskIndex" value="90.0" rearm="75.0"
trigger="2" ds-label="ns-dskPath"
filterOperator="or" ds-name="ns-dskPercentNode">
<resource-filter field="ns-dskDevice">^/dev/.+$</resource-filter>
</threshold>

The "relativeChange" threshold on the same value should have the same filter applied.

Here is the SNMP walk for more information:

  1. snmpwalk -v 2c -c (comstring) mdWWW01 .1.3.6.1.4.1.2021.9
    UCD-SNMP-MIB::dskIndex.1 = INTEGER: 1
    UCD-SNMP-MIB::dskIndex.2 = INTEGER: 2
    UCD-SNMP-MIB::dskIndex.3 = INTEGER: 3
    UCD-SNMP-MIB::dskIndex.4 = INTEGER: 4
    UCD-SNMP-MIB::dskIndex.5 = INTEGER: 5
    UCD-SNMP-MIB::dskIndex.6 = INTEGER: 6
    UCD-SNMP-MIB::dskIndex.7 = INTEGER: 7
    UCD-SNMP-MIB::dskIndex.8 = INTEGER: 8
    UCD-SNMP-MIB::dskIndex.9 = INTEGER: 9
    UCD-SNMP-MIB::dskIndex.10 = INTEGER: 10
    UCD-SNMP-MIB::dskIndex.11 = INTEGER: 11
    UCD-SNMP-MIB::dskIndex.12 = INTEGER: 12
    UCD-SNMP-MIB::dskPath.1 = STRING: /
    UCD-SNMP-MIB::dskPath.2 = STRING: /proc
    UCD-SNMP-MIB::dskPath.3 = STRING: /sys
    UCD-SNMP-MIB::dskPath.4 = STRING: /sys/fs/fuse/connections
    UCD-SNMP-MIB::dskPath.5 = STRING: /sys/kernel/debug
    UCD-SNMP-MIB::dskPath.6 = STRING: /sys/kernel/security
    UCD-SNMP-MIB::dskPath.7 = STRING: /dev
    UCD-SNMP-MIB::dskPath.8 = STRING: /dev/pts
    UCD-SNMP-MIB::dskPath.9 = STRING: /run
    UCD-SNMP-MIB::dskPath.10 = STRING: /run/lock
    UCD-SNMP-MIB::dskPath.11 = STRING: /run/shm
    UCD-SNMP-MIB::dskPath.12 = STRING: /boot
    UCD-SNMP-MIB::dskDevice.1 = STRING: /dev/mapper/mdWWW01-root
    UCD-SNMP-MIB::dskDevice.2 = STRING: proc
    UCD-SNMP-MIB::dskDevice.3 = STRING: sysfs
    UCD-SNMP-MIB::dskDevice.4 = STRING: fusectl
    UCD-SNMP-MIB::dskDevice.5 = STRING: none
    UCD-SNMP-MIB::dskDevice.6 = STRING: none
    UCD-SNMP-MIB::dskDevice.7 = STRING: udev
    UCD-SNMP-MIB::dskDevice.8 = STRING: devpts
    UCD-SNMP-MIB::dskDevice.9 = STRING: tmpfs
    UCD-SNMP-MIB::dskDevice.10 = STRING: none
    UCD-SNMP-MIB::dskDevice.11 = STRING: none
    UCD-SNMP-MIB::dskDevice.12 = STRING: /dev/vda1
    UCD-SNMP-MIB::dskMinimum.1 = INTEGER: 10000
    UCD-SNMP-MIB::dskMinimum.2 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.3 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.4 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.5 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.6 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.7 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.8 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.9 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.10 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.11 = INTEGER: -1
    UCD-SNMP-MIB::dskMinimum.12 = INTEGER: -1
    UCD-SNMP-MIB::dskMinPercent.1 = INTEGER: -1
    UCD-SNMP-MIB::dskMinPercent.2 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.3 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.4 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.5 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.6 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.7 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.8 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.9 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.10 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.11 = INTEGER: 10
    UCD-SNMP-MIB::dskMinPercent.12 = INTEGER: 10
    UCD-SNMP-MIB::dskTotal.1 = INTEGER: 9555048
    UCD-SNMP-MIB::dskTotal.2 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.3 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.4 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.5 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.6 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.7 = INTEGER: 247780
    UCD-SNMP-MIB::dskTotal.8 = INTEGER: 0
    UCD-SNMP-MIB::dskTotal.9 = INTEGER: 100824
    UCD-SNMP-MIB::dskTotal.10 = INTEGER: 5120
    UCD-SNMP-MIB::dskTotal.11 = INTEGER: 252056
    UCD-SNMP-MIB::dskTotal.12 = INTEGER: 233191
    UCD-SNMP-MIB::dskAvail.1 = INTEGER: 7375112
    UCD-SNMP-MIB::dskAvail.2 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.3 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.4 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.5 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.6 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.7 = INTEGER: 247768
    UCD-SNMP-MIB::dskAvail.8 = INTEGER: 0
    UCD-SNMP-MIB::dskAvail.9 = INTEGER: 100584
    UCD-SNMP-MIB::dskAvail.10 = INTEGER: 5120
    UCD-SNMP-MIB::dskAvail.11 = INTEGER: 252056
    UCD-SNMP-MIB::dskAvail.12 = INTEGER: 166979
    UCD-SNMP-MIB::dskUsed.1 = INTEGER: 1694560
    UCD-SNMP-MIB::dskUsed.2 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.3 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.4 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.5 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.6 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.7 = INTEGER: 12
    UCD-SNMP-MIB::dskUsed.8 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.9 = INTEGER: 240
    UCD-SNMP-MIB::dskUsed.10 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.11 = INTEGER: 0
    UCD-SNMP-MIB::dskUsed.12 = INTEGER: 53771
    UCD-SNMP-MIB::dskPercent.1 = INTEGER: 19
    UCD-SNMP-MIB::dskPercent.2 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.3 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.4 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.5 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.6 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.7 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.8 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.9 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.10 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.11 = INTEGER: 0
    UCD-SNMP-MIB::dskPercent.12 = INTEGER: 24
    UCD-SNMP-MIB::dskPercentNode.1 = INTEGER: 5
    UCD-SNMP-MIB::dskPercentNode.2 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.3 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.4 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.5 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.6 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.7 = INTEGER: 1
    UCD-SNMP-MIB::dskPercentNode.8 = INTEGER: 100
    UCD-SNMP-MIB::dskPercentNode.9 = INTEGER: 0
    UCD-SNMP-MIB::dskPercentNode.10 = INTEGER: 0
    UCD-SNMP-MIB::dskPercentNode.11 = INTEGER: 0
    UCD-SNMP-MIB::dskPercentNode.12 = INTEGER: 0
    UCD-SNMP-MIB::dskErrorFlag.1 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.2 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.3 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.4 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.5 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.6 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.7 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.8 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.9 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.10 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.11 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorFlag.12 = INTEGER: noError(0)
    UCD-SNMP-MIB::dskErrorMsg.1 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.2 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.3 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.4 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.5 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.6 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.7 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.8 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.9 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.10 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.11 = STRING:
    UCD-SNMP-MIB::dskErrorMsg.12 = STRING:

Regards,

Tim Miller Dyck

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Seth Leger August 7, 2012 at 4:04 PM

I've added the resource-filter to the ns-dskPercentNode threshold so that special filesystems are excluded from disk percentage thresholding. Marking as fixed.

commit 47994338c50b6b7b7b36f5d891fb056b64586ab3

Tim Miller Dyck May 8, 2012 at 8:25 AM

Hi, here is an update. This threshold tests as expected. When I changed "/proc" to "/prXXoc", I got a threshold alert for the /proc file system being full but not the others, and the collectd debug log output showed the regexp filtering out as desired.

Here are all four of the NetSNMP disk thresholds. For the percentage disk space one, the negative filter is not needed as the file systems return 0 percent full, so not an error.

For the relative thresholds (33% increase), it seemed useful to do a filter a different way to only trigger if the filesystem is between 20% and 99% full. This has two good effects: it automatically filters out the file systems above that return 0% or 100% and it does not trigger when the file system is mostly empty, and thus triggers frequently when very small changes in file system space occur.

Tested with OpenNMS 1.10.2-1 and NetSNMP 5.4.3 on Ubuntu 12.04 and Debian 6.0.

Regards,
Tim

Tim Miller Dyck May 2, 2012 at 7:30 PM

Hi Alejandro, yes, that is definitely better. The device names could well be different depending on what virtual file systems are loaded.

Here is a second try:

<threshold
description="Trigger an alert when the percentage of inodes used on any disk (excluding devices known to return 100% full false positive values) reaches or passes 90% for two measurement intervals"
type="high" ds-type="dskIndex" value="90.0" rearm="75.0"
trigger="2" ds-label="ns-dskPath"
filterOperator="or" ds-name="ns-dskPercentNode">
<resource-filter field="ns-dskPath">^(??!(/proc|/sys|/dev/pts).)+$</resource-filter>
</threshold>

Using dskDevice doesn't seem as good as dskPath as there are some devices with dskDevice of "none" that do report valid inode data (in the snmpwalk dump above).

The non-capturing group lookahead is from http://stackoverflow.com/questions/977251/regular-expressions-and-negating-a-whole-character-group

I'm not sure if the non-capturing part is really needed or not.

This is untested – I will test still today or tomorrow.

-Tim

Alejandro Galue May 2, 2012 at 6:48 PM

Hello,

The intention is really good, but the solution doesn't sound quite right.

I mean, I don't like the assumption that anything starting with "/dev" represents a real device. Maybe that is true for the particular version of Ubuntu you're using, but what about other versions of Ubuntu and/or other distributions of Linux ?

To avoid violate the principle of least surprise, I think it's better to use a more complicated regular expression in the resource-filter, like the negative lookahead to say "we don't want these values" (i.e., blacklist rather than whitelist); for example:

^(?!(devpts|proc|none))

Does that make sense ?

Fixed

Details

Assignee

Reporter

Labels

Original estimate

Time tracking

No time logged15m remaining

Components

Fix versions

Affects versions

Priority

PagerDuty

Created December 29, 2011 at 6:01 PM
Updated January 27, 2017 at 4:20 PM
Resolved August 7, 2012 at 4:04 PM