Import nodes from VMware vCenter 4 fails

Description

This issue was seen with vCenter Server Version 4.1.0 build-345043

In the first step it is necessary to enable legacy MD2 encryption in Java. The MD2 encryption algorithm for certificates is disabled in Java 7. To reproduce the issue it is first necessary to enable it with changing in

$JAVA_HOME/jre/lib/security/java.security
jdk.certpath.disabledAlgorithms="MD2"

jdk.certpath.disabledAlgorithms=

After configuring Java to accept older MD2 based certs.The requisition import ends after a few hours with the following exception:

Exception in provisiond.log

2013-11-20 15:05:00,037 INFO  [importExecutor-1] CoreImportActivities: Loading requisition from resource URL [vmware://{vcenter-ip}/{requisition-name}]
2013-11-20 15:05:00,040 DEBUG [importExecutor-1] FilesystemForeignSourceRepository: importing requisition from URL [vmware://{vcenter-ip}/{requisition-name}]
2013-11-20 19:50:16,025 INFO  [importExecutor-1] CoreImportActivities: The import has been aborted, skipping audit phase import.
2013-11-20 19:50:16,026 INFO  [importExecutor-1] CoreImportActivities: The import has been aborted, skipping scan phase import.
2013-11-20 19:50:16,027 INFO  [importExecutor-2] CoreImportActivities: The import has been aborted, skipping relate phase.
2013-11-20 19:50:16,029 ERROR [provisiond_Worker-1] Provisioner: Exception importing vmware://{vcenter-ip}/{requisition-name}
org.opennms.netmgt.provision.service.ModelImportException: Import failed for resource URL [vmware://{vcenter-ip}/{requisition-name}]
        at org.opennms.netmgt.provision.service.Provisioner.importModelFromResource(Provisioner.java:408)
        at org.opennms.netmgt.provision.service.Provisioner.doImport(Provisioner.java:488)
        at org.opennms.netmgt.provision.service.ImportJob.execute(ImportJob.java:57)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: org.opennms.core.xml.MarshallingResourceFailureException: Failed to perform IO while getting a configuration resource from spring: org.apache.commons.io.IOExceptionWithCause: Problem getting input stream: java.lang.RuntimeException: java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null; nested exception is org.apache.commons.io.IOExceptionWithCause: Problem getting input stream: java.lang.RuntimeException: java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null
        at org.opennms.core.xml.MarshallingExceptionTranslator.translate(MarshallingExceptionTranslator.java:55)
        at org.opennms.core.xml.JaxbUtils.unmarshal(JaxbUtils.java:154)
        at org.opennms.core.xml.JaxbUtils.unmarshal(JaxbUtils.java:147)
        at org.opennms.netmgt.provision.persist.AbstractForeignSourceRepository.importResourceRequisition(AbstractForeignSourceRepository.java:59)
        at org.opennms.netmgt.provision.persist.FusedForeignSourceRepository.importResourceRequisition(FusedForeignSourceRepository.java:96)
        at org.opennms.netmgt.provision.service.DefaultProvisionService.loadRequisition(DefaultProvisionService.java:880)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
        at com.sun.proxy.$Proxy79.loadRequisition(Unknown Source)
        at org.opennms.netmgt.provision.service.CoreImportActivities.loadSpecFile(CoreImportActivities.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.opennms.netmgt.provision.service.lifecycle.Phase$PhaseMethod.doInvoke(Phase.java:191)
        at org.opennms.netmgt.provision.service.lifecycle.Phase$PhaseMethod.access$100(Phase.java:157)
        at org.opennms.netmgt.provision.service.lifecycle.Phase$PhaseMethod$1.run(Phase.java:174)
        at org.opennms.core.tasks.SyncTask.run(SyncTask.java:89)
        at org.opennms.core.tasks.SyncTask$1.run(SyncTask.java:100)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.commons.io.IOExceptionWithCause: Problem getting input stream: java.lang.RuntimeException: java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null
        at org.opennms.netmgt.provision.service.vmware.VmwareRequisitionUrlConnection.getInputStream(VmwareRequisitionUrlConnection.java:860)
        at org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:124)
        at org.opennms.core.xml.JaxbUtils.unmarshal(JaxbUtils.java:152)
        ... 32 more
Caused by: java.lang.RuntimeException: java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null
        at com.vmware.vim25.mo.ManagedObject.retrieveObjectProperties(ManagedObject.java:158)
        at com.vmware.vim25.mo.ManagedObject.getCurrentProperty(ManagedObject.java:179)
        at com.vmware.vim25.mo.ManagedObject.getManagedObject(ManagedObject.java:369)
        at com.vmware.vim25.mo.ManagedEntity.getParent(ManagedEntity.java:109)
        at org.opennms.netmgt.provision.service.vmware.VmwareRequisitionUrlConnection.createRequisitionNode(VmwareRequisitionUrlConnection.java:354)
        at org.opennms.netmgt.provision.service.vmware.VmwareRequisitionUrlConnection.iterateHostSystems(VmwareRequisitionUrlConnection.java:641)
        at org.opennms.netmgt.provision.service.vmware.VmwareRequisitionUrlConnection.buildVMwareRequisition(VmwareRequisitionUrlConnection.java:555)
        at org.opennms.netmgt.provision.service.vmware.VmwareRequisitionUrlConnection.getInputStream(VmwareRequisitionUrlConnection.java:825)
        ... 34 more
Caused by: java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null
        at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:182)
        at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:124)
        at com.vmware.vim25.ws.VimStub.retrieveProperties(VimStub.java:77)
        at com.vmware.vim25.mo.PropertyCollector.retrieveProperties(PropertyCollector.java:107)
        at com.vmware.vim25.mo.ManagedObject.retrieveObjectProperties(ManagedObject.java:155)
        ... 41 more

Acceptance / Success Criteria

None

Attachments

22 Nov 2013, 09:46 AM
22 Nov 2013, 09:46 AM
21 Nov 2013, 08:51 PM
21 Nov 2013, 08:51 PM

Lucidchart Diagrams

Activity

Show:

Ronny Trommer March 1, 2018 at 10:22 AM

We have changed the VMware API cause VIJava is not supported anymore. Please verify with latest 21.0.4 and reopen if this issue still exist.

Ronny Trommer November 22, 2013 at 9:46 AM

Improved version which displays also Host System and Virtual Machines.

Ronny Trommer November 21, 2013 at 8:51 PM
Edited

We created a VMware connection to check if the communication between OpenNMS and vCenter can be established. It is just a simple connect test, query some vCenter information and disconnect. If we cannot establish this connection and can't get information we have an API - vCenter issue. So this is reduced to the minimum we need for the communication.

If you extract the two files with tar xzf name.tar.gz you will have the following folder structure:

Folder structure

.
├── vi-java-5.2
│   ├── config.properties
│   └── vmwareConDiag-vijava-52.jar
├── vi-java-5.2.tar.gz
├── vi-java-5.5.tar.gz
└── vi-java-5.5beta
   ├── config.properties
   └── vmwareConDiag-vijava-55beta.jar

You have to modify the config.properties file. It contains the host, username and password for the vCenter connection. It will just connect and gives you some information about the connection and the vCenter itself. Here an example output:

VMware API Type means what is the endpoint of the connection.
VMware API Version gives you the version and build number of your vCenter you connect to.
VMware Operating System gives you the operating system which is installed on the vCenter Server.

Example output

51:48.214 [main] INFO  vmwareConDiag.Starter - Using host:'192.168.31.200', user:'root', pass (SHA-256):'0d3e0981aa23680f72afcd22c3ebe1b396e7c8cde5fe2580b1b7b95fbc999209' for connection
51:48.227 [main] INFO  vmwareConDiag.Starter - Try to connect: '192.168.31.200'
51:49.183 [main] INFO  vmwareConDiag.Starter - Connect successfull
51:49.183 [main] INFO  vmwareConDiag.Starter - VMware API Type:         'VirtualCenter'
51:49.183 [main] INFO  vmwareConDiag.Starter - VMware API Version:      '5.1' build '1123961'
51:49.184 [main] INFO  vmwareConDiag.Starter - VMware operating system: 'linux-x64'

The code is publicly available on Github: https://github.com/indigo423/vmwareConDiag

Ronny Trommer November 21, 2013 at 12:26 PM
Edited

The error message "VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null" indicates we don't get any data from the vCenter. As we can see in the stack trace the web service client inside of the VI Java library isn't doesn't get any data ether. It seems we have to debug this issue with the VI-Java library alone. I think it should be helpful to have a vmware-com-test tool which just tries the communication part and has a more verbose debug output with logs for diagnoses. It could be also necessary to have tcpdump for the communication. It seems the TCP session is hanging which means. The connection is established correctly, there is no socket timeout or something, the java exception needs a few hours to appear.

Ronny Trommer November 21, 2013 at 11:17 AM
Edited

I found a similar hint to an issue here running with VMware vCenter Server 4.1.0 build-258902

http://sourceforge.net/p/vijava/discussion/826592/thread/ad2078f3

Cannot Reproduce

Details

Assignee

Ronny Trommer

Reporter

Ronny Trommer

Priority

Major

PagerDuty

Created November 21, 2013 at 10:35 AM

Updated March 1, 2018 at 10:22 AM

Resolved March 1, 2018 at 10:22 AM

Configure

Import nodes from VMware vCenter 4 fails

Description

Acceptance / Success Criteria

Attachments

Lucidchart Diagrams

Activity

Ronny Trommer March 1, 2018 at 10:22 AM

Ronny Trommer November 22, 2013 at 9:46 AM

Ronny Trommer November 21, 2013 at 8:51 PMEdited

Ronny Trommer November 21, 2013 at 12:26 PMEdited

Ronny Trommer November 21, 2013 at 11:17 AMEdited

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

PagerDuty

Ronny Trommer November 21, 2013 at 8:51 PM
Edited

Ronny Trommer November 21, 2013 at 12:26 PM
Edited

Ronny Trommer November 21, 2013 at 11:17 AM
Edited