Month: December 2019

VMWare Workstation – DUP! packet issue resolved…sort of

I was getting VERY frustrated with some networking issues with my virtual guests in VMW 15.5 (and prior), on Windows 10.  See below:

dup-packets.png

If you look, you’ll see that for every ping request I’m sending to my gateway (or ANY other IP address outside the Windows host), I’m getting FOUR RESPONSES. This also manifests itself in *very* slow downloads for packages or updates I’m installing on the VM’s.  And, it’s just wrong so it needed fixing.

Note that the standard Google answer to this issue is to stop and delete the Routing and Remote Access Service.  The first time this happened, this solved the problem! There were a ton of other ‘solutions’ out there but none really understood the problem. Windows was creating some sort of packet amplification. (When I have time I’m going to reinstall pcap and dig into this).

But then….months later….

It came back.  I hadn’t re-enabled routing and remote access. I hadn’t made any networking changes inside the host or on my network.   I HAD done some other stuff, such as enabling Windows Services for Linux and installing Ubuntu for bash scripting purposes.  You know…messing around. Some of this could’ve re-written the bindings and orders of networks/protocols/services etc., but if so, it wasn’t reflected anywhere in the basic or advanced network settings. VERY frustrating!

I deleted a TON of stuff I’d installed that I no longer needed (which had to be done anyway, but I was saving that for New Years’). I re-installed the VMware bridge protocol. I repaired VMware Workstation. I REMOVED and re-installed VMware Workstation.

**Here’s what finally RE-solved the problem:

  • I RE-ENABLED RRAS (!)
  • I went into the properties of “Incoming Connections” in Network Adapter Settings and UNCHECKED IPv4, leaving IPv6 checked. (I’m not sure if this matters, try it without this step first).
  • I RE-DISABLED RRAS (!)

And…here’s the result.

non-dup-packets.png

I can only surmise that the act of STOPPING RRAS does a config of the network stack where it doesn’t amplify packets. And, you can’t stop a service unless it’s already started, right?

Makes complete sense.

NOT.

But, all’s well that ends.

VMWare Workstation 15 REST API – Control power state for multiple machines via Python

Or..How to easily power up/suspend your entire K8s cluster at once in VMWare Workstation 15

In VMWare Workstation 15, VMWare introduced the REST API, which allows all sorts of automation.  I was playing around with it, and wrote a quick Python script to fire up (or suspend) a bunch of machines that I listed in an array up top in the initialization section. In this case, I want to control the state of a 4-node Kubernetes cluster, as it was just annoying me to click on the play/suspend 4 times (I have other associated virtual machines as well, which only added to the annoyance.)

Your REST API exe (vmrest.exe) MUST be running if you’re going to try this. If you haven’t set that up yet, stop here and follow these instructions.  You’ll notice that Vmrest.exe normally runs as an interactive user mode application, but I’ve now set up the executable to run as a service on my Windows 10 machine using NSSM, I’ll have a separate blog entry to show how that’s done.

Some notes on the script:

  • Script Variables – ip/host:port (you need the port, as vmrest.exe gives you an ephemeral port number to hit), machine list, and authCode
  • Regarding the authCode.  WITH vmrest.exe running, go to “https://ip_of_vmw:port” to get the REST API explorer page (shown below). Click “authorization” up top, and you’ll get to log in. Use the credentials you used to set up the VMW Rest API via these instructions. 

Screen Shot 2019-12-24 at 3.05.04 PM.png

Then do a “Try it out!” on any GET method that doesn’t require variables and your Auth Code will appear in the Curl section in the “Authorization” header. Grab that code, you’ll use it going forward.

curl.png

Here’s the script, with relatively obvious documentation. Since more than likely your SSL for the vmrest.exe API server will use a self-signed, untrusted certificate, you’re probably going to need to ignore any SSL errors that will occur. That’s what the “InsecureRequestWarning” stuff is all about, we disable the warnings.  My understanding is that the disabled state is reset with every request made, so we need to re-disable it before every REST call.

I’ve posted this code on GitHub HERE.

#!/usr/bin/env python3
import requests
import urllib3
import sys
from urllib3.exceptions import InsecureRequestWarning

'''Variable Initiation'''

ip_addr = 'your-ip-or-hostname:Port' #change ip:port to what VMW REST API is showing
machine_list = ['k8s-master','k8s-worker1','k8s-worker2','k8s-worker3']
authCode = 'yourAuthCode'

'''Section to handle the script arg'''

acceptable_actions = ['on', 'off', 'shutdown', 'suspend', 'pause', 'unpause']

try:

    sys.argv[1]

except NameError:

        action = "on"

else:

    if sys.argv[1] in acceptable_actions:

        action = sys.argv[1]

    else:

        print("ERROR: Action must be: on, off, shutdown, suspend, pause, or unpause")

        exit()


'''Section to get the list of all VM's '''

urllib3.disable_warnings(category=InsecureRequestWarning)

resp = requests.get(url='https://' + ip_addr + '/api/vms', headers={'Accept': 'application/vnd.vmware.vmw.rest-v1+json', 'Authorization': 'Basic ' + authCode}, verify=False)

if resp.status_code != 200:

    #something fell down

    print("Status Code " + resp.status_code + ": Something bad happened")

result_json = resp.json()


'''Go through entire list and if the VM is in the machine_list, if so,
 act! '''

for todo_item in resp.json():

    current_id = todo_item['id']

    current_path = todo_item['path']

    for machine in machine_list:

        if current_path.find(machine) > -1:

        print(machine + ': ' + current_id)

        urllib3.disable_warnings(category=InsecureRequestWarning)

        current_url = 'https://' + ip_addr + '/api/vms/' + current_id + '/power'

        resp = requests.put(current_url, data=action, headers={'Content-Type': 'application/vnd.vmware.vmw.rest-v1+json', 'Accept': 'application/vnd.vmware.vmw.rest-v1+json', 'Authorization': 'Basic ' + authCode}, verify=False)

        print(resp.text)

        '''Better exception handling should be written here of course.
**12/27/19 NOTE!** – I’ve noticed what I believe to be a bug in VMW 15.5 where if you control power state via the REST API, you lose the ability to control the VM via the built-in VMWare console in the app.  The VMs behave fine (assuming everything else is working), but for some reason the VMW app doesn’t attach the console process correctly.  If you want to follow this issue I’ve submitted to the community here.