A minor change on ThresholdEvaluatorHighLow.java did the trick.

Now when a monitored value is rearmed before the threshold state reaches the trigger value, the counter will be reseted, and it will start again if the monitored value pass the configured threshold.

Gabriel Gwartney, gave me a better picture of why this problem reappear again.

Here is his analysis:

I'm currently evaluating OpenNMS 1.8.4 and I noticed that there is
still a bug in thresholding that is similar to bug 1582. When
specifying trigger value N in a threshold, the trigger count does not
reset when the value falls below the threshold unless it has occurred N
times and a threshold rearm event has been generated. For example, we
gather an SNMP integer value from our devices in the field which
represent the current status of certain attributes. We evaluate whether
certain bits are set or not. If a certain combination of set bits
occurs N times in a row, we want an event generated. However, there are
instances where these certain bits were set for a brief period of time
(and the thresholding daemon increased the trigger count) but then
became unset. The thresholding daemon does not reset the trigger count
at this point. According to bug 1582, this behavior was fixed in 1.7.6
(at least that's the target milestone) but appears to still be present.
Is there a certain threshold configuration now that will create an event
when the threshold has been breached N times in a row?
Here is an example:

High threshold has been rearmed:
2010-10-15 14:02:21,742 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEntity: evaluate: value= 0.0 against threshold:
{evaluator=high, dsName=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2),
dsType=tgbNtpIndex,
evaluators=[{ds=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2), value=2.0,
rearm=1.0, trigger=6}]}
2010-10-15 14:02:21,742 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEvaluatorHighLow$ThresholdEvaluatorStateHighLow: evaluate: high
threshold rearmed

First trigger of high threshold:
2010-10-15 14:12:26,046 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEntity: evaluate: value= 2.5 against threshold:
{evaluator=high, dsName=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2),
dsType=tgbNtpIndex,
evaluators=[{ds=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2), value=2.0,
rearm=1.0, trigger=6}]}
2010-10-15 14:12:26,046 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEvaluatorHighLow$ThresholdEvaluatorStateHighLow: evaluate: high
threshold exceeded, count=1

Next, the value has fallen below the rearm level:
2010-10-15 14:17:27,115 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEntity: evaluate: value= 0.0 against threshold:
{evaluator=high, dsName=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2),
dsType=tgbNtpIndex,
evaluators=[{ds=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2), value=2.0,
rearm=1.0, trigger=6}]}

Notice no reset of the trigger count.

Now, threshold has been breached once again.
2010-10-15 14:22:28,666 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEntity: evaluate: value= 2.5 against threshold:
{evaluator=high, dsName=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2),
dsType=tgbNtpIndex,
evaluators=[{ds=(tgbGpsSyncFlags%2)+((tgbGpsSyncFlags/2)%2), value=2.0,
rearm=1.0, trigger=6}]}
2010-10-15 14:22:28,666 DEBUG [CollectdScheduler-50 Pool-fiber0]
ThresholdEvaluatorHighLow$ThresholdEvaluatorStateHighLow: evaluate: high
threshold exceeded, count=2

You'll see the count still increments instead of starting at 1 again.
It will continue in this manner until count=6. Then it will fire off
the high threshold alarm. The next time the value falls below the
threshold, a rearm event is triggered and the count resets to 0. To me
these are false positives. If a value falls to the rearm level, the
count should be reset.

I hope this helps.

Fixed on trunk using new in-line thresholding on pollerd (not checked with deprecated threshd).

PETER: If you still have problems with this, please reopen this bug.

iMacAlejo:opennms agalue$ svn log -l 1 opennms-services/src/test/java/org/opennms/netmgt/threshd/ThresholdingVisitorTest.java
------------------------------------------------------------------------ r13555 | agalue | 2009-06-03 13:04:31 -0430 (Wed, 03 Jun 2009) | 3 lines

Updating ThresholdingVisitorTest.testLatencyThresholdingSet to validate that has been fixed.
New test added ThresholdingVisitorTest.testThresholsFilters, just to be sure that resource filters work fine.
Minor correction on ThresholdingSet.
------------------------------------------------------------------------

(In reply to comment #2)
> any chance to resolve this bug in ONMS 1.7.x? It's very annoying :-/
>
Hello Peter,

The current trunk version has in-line thresholds processing for pollerd. I will really appreciate if you can make tests against the current trunk.

In order to use the new thresholding processing engine, you must disable thresd (I mean, comment thresd entry on service-configuration.xml or remove any thresholder tag on thresd-configuration.xml).

And then only create a package on thresd that match your nodes and services; something like this:

<package name="myLatencyPackage">
<filter>IPADDR != '0.0.0.0'</filter>
<specific>10.10.10.10</specific>
<include-range begin="192.168.0.1" end="192.168.0.254"/>
<service name="ICMP" interval="300000" user-defined="false" status="on">
<parameter key="thresholding-group" value="icmp-snmp"/>
</service>
</package>

Then enable new in-line thresholding on pollerd-configuration.xml

<service name="ICMP" interval="300000" user-defined="false" status="on">
<parameter key="retry" value="2"/>
<parameter key="timeout" value="3000"/>
<parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
<parameter key="ds-name" value="icmp"/>
<parameter key="thresholding-enabled" value="true"/>
</service>

Now pollerd will process thresholds like collectd.

The only difference between this and old threshd is that RRD will never be touched again (a boost in performance) and the compared value will be exactly the same value collected (gauge, not a rate).

Any issues please let me know in order to fix the new implementation.

In the meanwhile, I will try to add some junit tests against the trunk version to see if I can reproduce the problem.

Alejandro.

(In reply to comment #1)
> moving to 1.6.1 target milestone; only things left pending for 1.6.0 are
> blocker-level bugs

Hi,

any chance to resolve this bug in ONMS 1.7.x? It's very annoying :-/

Petr Herzig