vVol Troubleshooting – What to do with a Empty VP URL

I have been wanting to put together a kind of a series of vVol (I still have a hard time typing vVol instead of VVol) troubleshooting, tips or some things I’ve been seeing. I’ve gotten a couple of those already, but I should be getting more out and maybe actually help out someone.

One of the more common issues I have seen when getting started with vVols is when an ESXi host is unable to authenticate with the VASA Providers. Even though the vCenter Server has registered the Storage Providers successfully. There can be a few reasons for this happening, so I wanted to dig into some of those reasons, how to troubleshoot them and hopefully fix them.

The Setup and Issue

Let’s dive into what we are wanting to look at. In this example I am just getting vVols setup with the vCenter. In this vCenter I have just registered the storage providers and everything there is healthy. The protocol endpoint is connected to all of the hosts in the ESXi Cluster and I just created the vVol Datastore. Now here is when I see the issue, I get a report that the vVol Datastore “inaccessible” for 1 or 2 of the hosts in the ESXi Cluster. Well, that’s strange, it’s fine on all of the other hosts in the cluster, why is it broken with this host?

The first place I generally will be looking at is the vvold log on the ESXi host that the vVol Datastore reports as inaccessible.

The Empty VP URL Issue

2019-08-12T22:59:18.124Z info vvold[2298793] [Originator@6876 sub=Default] VendorProviderMgr::AddOrRef VP (sn1-m20-c12-25-ct0) Uid () URL (https://10.21.88.113:8084/version.xml)
 2019-08-12T22:59:18.124Z info vvold[2298793] [Originator@6876 sub=Default] DlgSoapCtxPool: Created new DlgSoapCtxPool ([14DlgSoapCtxPool:0x000000c29694e690]) for VP (sn1-m20-c12-25-ct0) session (000000c29694e520)
 2019-08-12T22:59:18.124Z info vvold[2298793] [Originator@6876 sub=Default] VendorProviderMgr:UpdateFirewallRules: Firewall operation add on rule id:VVOLD_8084 success

 2019-08-12T22:59:18.124Z info vvold[2298793] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://10.21.88.113:8084/version.xml
 2019-08-12T22:59:18.128Z warning vvold[2298793] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:
 --> PeerThumbprint: 0B:2F:E1:52:F3:85:EB:CD:BE:B4:FF:E3:B2:64:1B:FB:F5:BA:55:F4
 --> ExpectedThumbprint:
 --> ExpectedPeerName: 10.21.88.113
 --> The remote host certificate has these problems:
 -->
 --> * unable to get local issuer certificate, using default
 2019-08-12T22:59:18.128Z info vvold[2298793] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
 2019-08-12T22:59:18.128Z warning vvold[2298793] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (sn1-m20-c12-25-ct0)!
 2019-08-12T22:59:18.128Z info vvold[2298793] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://10.21.88.113:8084/version.xml
 2019-08-12T22:59:18.128Z error vvold[2298793] [Originator@6876 sub=Default] Initialize: Unable to init session to VP sn1-m20-c12-25-ct0 state: 0

I see that when the host is attempting to authenticate with the VP (VASA Provider) it is unable to locate the VP’s issuer certificate, and because that issuer certificate is missing the VP URL for Initialization is missing the host is unable to authenticate with the VASA Provider.

A look at a Good Authentication

One of my favorite troubleshooting workflows is to compare the problem with a healthy/normal working example. So, next I want to take a look at the vvold log for the host that everything is working for just fine. I’ll see that the request to GetEndPoint is made and succeeds right away.

2019-08-12T22:59:18.086Z info vvold[2571765] [Originator@6876 sub=Default] VendorProviderMgr::AddOrRef VP (sn1-m20-c12-25-ct0) Uid () URL (https://10.21.88.113:8084/version.xml)
 2019-08-12T22:59:18.086Z info vvold[2571765] [Originator@6876 sub=Default] DlgSoapCtxPool: Created new DlgSoapCtxPool ([14DlgSoapCtxPool:0x0000002177da0d40]) for VP (sn1-m20-c12-25-ct0) session (0000002177da0bd0)
 2019-08-12T22:59:18.087Z info vvold[2571765] [Originator@6876 sub=Default] VendorProviderMgr:UpdateFirewallRules: Firewall operation add on rule id:VVOLD_8084 success
 2019-08-12T22:59:18.087Z info vvold[2571765] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://10.21.88.113:8084/version.xml
 2019-08-12T22:59:18.098Z info vvold[2571765] [Originator@6876 sub=Default] VasaSession::Parse xml: id=4, loc=/vasa, myVer=4, ver=0
 2019-08-12T22:59:18.098Z info vvold[2571765] [Originator@6876 sub=Default] VasaSession::ParseVersionXml: Effective VP endpoint is https://10.21.88.113:8084/vasa (version 4)
 2019-08-12T22:59:18.098Z info vvold[2571765] [Originator@6876 sub=Default] VasaSession::DoSetContext: Setting GUID: 'D2CCA827-514D-E611-0000-78654321006B'  VC UID: '1143a5e7-f7de-43