Requisition File is Unmarshalled for each Node as it is Scanned
Description
We have a client with 41K nodes in a requisition. We noticed that it was taking 30 seconds to scan a node after the initial import. In looking at the logs we found that we'd see a log like:
2012-11-07 22:21:28,946 DEBUG [scanExecutor-7] ForeignSource: found unmarshaller for class org.opennms.netmgt.provision.persist.foreignsource.ForeignSource 2012-11-07 22:21:28,946 DEBUG [scanExecutor-7] ForeignSource: created unmarshaller for class org.opennms.netmgt.provision.persist.foreignsource.ForeignSource
and it would take ten seconds before the thread was ready to do more work. Seth checked the code and noticed that as each node was prepared for scanning, the code is going to the filesystem to unmarshall the requisition file. With large requisitions this can take time, and with multiple threads doing it as well, it could take even longer.
There does appear to be some caching code available, but no easy way to access it. This should be corrected before 1.10.7 because it will impact anyone with large requisition files.
Acceptance / Success Criteria
None
Lucidchart Diagrams
Activity
Show:
Matt Brozowski September 26, 2013 at 5:06 PM
shouldn't have closed this... i fixed a different issue
Seth Leger February 5, 2013 at 3:52 PM
Wasn't this fixed by adding the FileReloadContainer? See commit 55c454b00b23189141bd5c653d19ab2e42564cd3 for more details.
We have a client with 41K nodes in a requisition. We noticed that it was taking 30 seconds to scan a node after the initial import. In looking at the logs we found that we'd see a log like:
2012-11-07 22:21:28,946 DEBUG [scanExecutor-7] ForeignSource: found unmarshaller for class org.opennms.netmgt.provision.persist.foreignsource.ForeignSource
2012-11-07 22:21:28,946 DEBUG [scanExecutor-7] ForeignSource: created unmarshaller for class org.opennms.netmgt.provision.persist.foreignsource.ForeignSource
and it would take ten seconds before the thread was ready to do more work. Seth checked the code and noticed that as each node was prepared for scanning, the code is going to the filesystem to unmarshall the requisition file. With large requisitions this can take time, and with multiple threads doing it as well, it could take even longer.
There does appear to be some caching code available, but no easy way to access it. This should be corrected before 1.10.7 because it will impact anyone with large requisition files.