VM Inaccessible - or is it?

I originally posted this on LinkedIn. I'm still figuring out syntax highlighting on Blogger so please bear with the basic formatting for the moment.

This morning I got a strange IM from one of my colleagues. A virtual machine he was working on had become inaccessible in vSphere, but seemed fine otherwise. I decided to have a look around. Hostnames have been changed and sequences have been shortened to protect my institution.

The Summary page for the server, which we'll call dummy-vm-3, indicated that the host it was registered on was vm-host-13. However, the last migration in the Tasks logs before the machine disappeared, indicated a vMotion from vm-host-7 to vm-host-6. Odd.

From the ESXi shell, we can tell which VMs are running, whether they are listed in the client or not. I sshed into "vm-host-13" and ran the following:
~ # vim-cmd vmsvc/getallvms
Skipping invalid VM '1394'
Vmid Name  File         Guest OS   Version Annotation
1 dummy-vm-1 [datastore-1] dummy-vm-1/dummy-vm-1.vmx winLonghornGuest vmx-08
1290 dummy-vm-2 [datastore-2] dummy-vm-2/dummy-vm-2.vmx rhel6_64Guest  vmx-10

~ # esxcli vm process list
dummy-vm-1
   World ID: 15566863
   Process ID: 0
   VMX Cartel ID: 15566860
   UUID: 42 3b c7 ff 3f b2 c2 92-6f 3e 21 24 75 17 c5 d6
   Display Name: dummy-vm-1
   Config File: /vmfs/volumes//dummy-vm-1/dummy-vm-1.vmx

dummy-vm-2
   World ID: 38217841
   Process ID: 0
   VMX Cartel ID: 38217840
   UUID: 42 3b 17 c0 1a b9 31 a7-f6 70 ad ab 1f 07 51 35
   Display Name: dummy-vm-2
   Config File: /vmfs/volumes//dummy-vm-2/dummy-vm-2.vmx

Interestingly, dummy-vm-3 was not running, but the hypervisor helpfully informs us that an invalid (i.e. vmx file cannot be read) VM with id 1394 is being hidden from the getallvms output. Could it still be on vm-host-6? I ran the following on vm-host-6:
~ # esxcli vm process list

dummy-vm-3
   World ID: 38659305
   Process ID: 0
   VMX Cartel ID: 38656655
   UUID: 42 3b 41 fa 78 c5 48 61-44 bc 42 e6 68 46 f2 4d
   Display Name: dummy-vm-3
   Config File: /vmfs/volumes//dummy-vm-3/dummy-vm-3.vmx

dummy-vm-4
   World ID: 34269567
   Process ID: 0
   VMX Cartel ID: 34269564
   UUID: 42 3b 4f c0 38 dc 40 1b-e9 5a 85 40 ac b0 b4 40
   Display Name: dummy-vm-4
   Config File: /vmfs/volumes//dummy-vm-4/dummy-vm-4.vmx

Bingo. The registration had somehow moved to vm-host-13, but not the VM process itself. We would later find out that this was a fumbled power-on DRS placement after the VM was shut down to remove an extra disk.

At this point, all we needed to do was to unregister dummy-vm-3 from vm-host-13 and register it on vm-host-6. Back on vm-host-13:
~ # vim-cmd vmsvc/unregister 1394
The command doesn't return anything, but the "Unknown" VM shown when connecting directly to vm-host-3 disappears. Let's register it on vm-host-6 and hope for the best:
~ # vim-cmd /solo/register /vmfs/volumes//dummy-vm-3/dummy-vm-3.vmx
1724
After that, the VM re-appeared in the vSphere client and we could access the console again. It was immediately picked up by DRS and thrown to another host, this time successfully.

References:
https://kb.vmware.com/s/article/1005051
http://www.yellow-bricks.com/2011/11/16/esxi-commandline-work/
http://www.everythingvm.com/content/fixing-invalid-virtual-machines-vmware

Comments

Post a Comment

Popular posts from this blog

Secure Boot failure on ESXi 7.0 U3 - Solved

Image
At the beginning of April, an ice storm hit Quebec and caused power outages lasting several days, particularly in areas with older trees where utility lines are not underground. When I started my ESXi host after 5 days offline, I ran into an error I've never seen before: It seemed like all of the VIBs were now failing UEFI Secure Boot validation. My first instinct was that due to an unclean shutdown, data on the boot volume was corrupted, and a reinstall would be required. The voice of my VMware TAM was in the back of my mind: I'm running non- HCL hosts, with NVMe SSDs not on the VSAN HCL , I should be thankful this works at all. And he's right. And I am.  However, I'm happy to report this had a quick fix: while goofing around in the BIOS to see how I could bypass Secure Boot and load ESXi, I noticed that my system clock had rolled back to 2017! As it turns out, I had been foiled by a $1 CR2032 battery  meant to keep time while the board is not connected to power. By s...
Read more

ADFS RelayState (IdP-initiated sign-on deep links)

The university, as part of its digital strategy initiative, is pushing harder than ever for SSO across its major services. Part of this means standardizing on an IdP infrastructure to tie everything into, and for a lot of reasons (future post) we chose ADFS. Recently a service provider asked us if, since we are working this way with our Shibboleth IdP now, we can provide a deep link on the IdP domain that will redirect to the SP after the authentication flow. I figured it was possible, but it was the first time we had to think about this with ADFS. Turns out it's quite easy, at least on ADFS v5 (Server 2019). I'm probably explaining this wrong, but in SAML parlance the flow where the user connects to the IdP first and then gets punted to an SP is called RelayState, and typically it is triggered by appending a query string to the IdP-initiated sign-on page. First, enable IdP-initiated sign-on if you haven't already: Set-AdfsProperties -EnableIdpInitiatedSignonPage $tr...
Read more

MikroTik CRS309-1G-8S+IN in 2023

Image
In late 2019, ServeTheHome  reviewed  the MikroTik  CRS309-1G-8S+IN . TL;DR  it's an 8x SFP+, 1x 1G RJ45 managed switch that retails around $350 CAD. When these came out, nothing else on the market came close on cost per port, even unmanaged. STH called it an "absolutely great buy". In 2021 I picked the CRS309 as the 10G backbone for my new home network and vSphere lab. Out of the box, running RouterOS 6, it did everything I needed at the time: I wanted a 10G switch to play with VSAN, upgrade my NAS to 10G, and a few ports for future projects. I set it up and basically forgot about it. The CRS309 was a slam dunk then and I'm still happy with my decision. Is the CRS309 still a good buy in 2023?   There are now at least two other affordable fanless ~8 port 10G managed switches: TP-Link TL-SX3008F : 8 SFP+, works with or without their Omada controller. $299 CAD Ubiquiti  USW-Aggregation : 8 SFP+, requires a UniFi controller. $359 CAD They'll do VLA...
Read more