Flows API doesn't recognize the exporters
Description
Acceptance / Success Criteria
Attachments
Lucidchart Diagrams
Activity
John Blake May 17, 2018 at 9:56 PM
Verified fixed in latest 22 snaps. :ship:
Jesse White May 2, 2018 at 8:26 PM
PR related to fixing the timestamps for NF9 was merged.
Alejandro Galue May 1, 2018 at 4:16 PM
Am I missing something ?
I'm also seeing the second error on Grafana.
I'm using latest Helm compiled from its develop branch, and the latest elasticsearch-drift-plugin compiled from its master branch with Elasticsearch 6.2.4 (I've updated the drift plugin POM prior compiling it to reflect the ES version I'm using).
Alejandro Galue May 1, 2018 at 3:49 PM
Today, I decided to apply a more comprehensive flow configuration on the Cisco router, instead of using the default.
Here is what I have now:
The above is based on the recommendations from here: https://www.youtube.com/watch?v=TZUW5lqzZDc
In theory, yesterday's work was running for enough time so the "simple" config for Netflow 9 should work (as based on what I'm seeing on Elasticsearch, it looks very close to what I saw yesterday). This one is just more specific, specially with the timing and the cache.
The interesting thing that I didn't try yesterday, but I did today was that I found that I have to increase the range in order to get data:
The good news is that I'm getting data. I'll try with Helm soon!
If there is something specific you guys want me to try let me know.
Alejandro Galue April 30, 2018 at 8:11 PM
I've uploaded 2 files:
netflow-data.pcap
netflow-esdata.json (contain the newest 100 documents, all of them enhanced).
I still have the same behavior as before. Let me know if you need something else (I still have the data of the Elasticsearch available, as it was also running on my Mac).
This time, I've compiled the branch from the PR on my Mac, and used my GNS3 Lab. On that Lab, I have a Minion machine forwarding the Netflow9 Packets to the OpenNMS on my mac. I took the packet capture from the Minion machine (as this one is receiving the Netflow 9 packets from the virtual Cisco Router).
Using latest RPMs from features/drift and a lab with the full architecture of OpenNMS for using Drift in production, detailed here:
https://github.com/OpenNMS/opennms-drift-aws
I found that the flows are received by the Minion, forwarded to Kafka, extracted and processed by the Flow Adapters, and finally stored on Elasticsearch.
I can even see that the ES documents for flows are "enriched" (or enhanced) with the node_exporter, node_dst or node_str; but when I query the Flows API using /opennms/rest/flows/exporters, that always return an empty array.
The only workaround that I found is pre-populating the OpenNMS database with the nodes that will be "exporters" (meaning, I have to provision the routers/switches with flows disabled, synchronize the requisition, make sure that everything was populated, and then start sending flow data from the routers/switches to the Minion).
When I do the provisioning later, I waited enough time (even multiple days), restarted OpenNMS multiple times, and I was not able to see the exporters using the Flows API.
As I said, the crazy part is that the "flow enrichment" is "seeing" the nodes from the database. This doesn't happen all the time though.