Docker + Windows “Error starting userland proxy”

Docker Start Error

I ran into a new docker issue today.  Basically, I restarted my PC, and when I tried to bring up a container with a Postgres instance I use for testing, I received this confusing error:

Error response from daemon: driver failed programming external connectivity on endpoint postgres (15b348b1f5bf8d2bfd17c1c41b340d1c66f63ace7cab39ea69aeca3f69ed7442): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:5432:tcp:172.17.0.2:5432: input/output error
Error: failed to start containers: postgres

What Does it Mean?

It turns out this is a big headache which is still unresolved, and which has one of the longer Git Issue threads I’ve ever seen right here.

Here’s a summary of it:

  • Windows 10 has a “Fast Start Up Mode”, and Docker doesn’t play well with it (or vice versa).
  • So, after a restart, you may find that you see this issue.
  • Theoretically, restating the Docker Daemon fixes this (which is a little annoying but fine).  You should be able to do that in Services.
  • This personally didn’t help me the first try.  So, I went and disabled Fast Start mode (which is also annoying) by:
    • Go to start and type “Power and Sleep”, click it when it pops up.
    • Click “Additional power settings” on the right.
    • Click “Choose what the power buttons do”.
    • Click “Change settings that are currently unavailable” and log in if you can’t already toggle the “Turn on fast startup (recommended) checkbox.
    • Turn off that checkbox.

Note that once you reboot you have to wait a bit for docker to come up (it can take a few minutes).  For example, the first 4 or 5 times I ran “docker version”, the daemon showed as down even though I could see the service running.  But a minute later it was up and working fine.

Installing Minikube on Windows 10

Installing Minikube on Windows 10 was not as straightforward as I had expected.  I tried it at home and gave up originally (instead using https://labs.play-with-k8s.com/).

I wanted to get it set up properly for a new job though, so I worked through the issues.  Please note in case your issues are different than mine, these links are extremely enlightening:

Much of the online documentation I found was outdated.  The actual Git Hub repository was fine though; so I recommend using the instructions there : https://github.com/kubernetes/minikube.  Mine are based on those and the previously noted links.

Before We Start – If You’re Already in a Bad State

If you’ve already started installing Minikube and you failed, or if you have been hacking at this with my instructions and are stuck, you may end up needing to delete the C:\users\user-name{.minikube and .kube} directories to clean things up.  I had to do this as I messed up the virtual switch/etc set-up originally.  Don’t do this unless you do get stuck though.

You can’t do this folder clean-up if the Hyper-V minikube VM is still running (you can see that in the Hyper-V manager).  If you try to delete them while its running, it will partially delete files and then you’re in a state where you can’t even stop the VM on purpose!  To fix that, you need to kill the processes related to the VM and set it to not automatically start up in the Hyper-V manager. Then you can remove the VM and try again.

Some detailed information on how to kill a hung Hyper-V VM is here: http://woshub.com/how-to-stop-a-hung-virtual-machine-on-hyper-v-2016/. This was painful to fix.

The Minikube start command will hang if you’re in these bad states, so you really do need to delete those directories to get it set up cleanly.

Warning: Shut Down Issues

I found after I had this all working that (1) my PC was having trouble restarting – it would never stop, and (2) if I came and tried to do “minikube stop”, it would hang forever.  It turns out the current version of minikube has issues noted here: https://github.com/kubernetes/minikube/issues/2914.  So, to shut it down you need to do “minikube ssh” and then “sudo poweroff”.  Even trying to manage it from Hyper-V will not work properly.  They recommend downgrading to v.27; the current version I saw the issue with is .30.  I haven’t tried the downgrade yet.  So, these instructions will get it working with v.30; but you will have this shut-down issue potentially.

For now, I’m personally just going to stay with v.30, but I told it to NOT automatically start the minikube VM when my PC starts up in Hyper-V.  If I use this too often and they don’t fix this issue, I may downgrade at a later date though.

Installation Steps

Here is the full set of steps I had to use to get Minikube correctly running on my Windows 10 desktop:

  • Enable the “Hyper-V” windows feature (I also enabled containers, and WIndows Hypervisor Platform, they probably aren’t needed though).
  • Go into Hyper-V manager (search your start menu) and:
    • Click Virtual Switch Manager
    • Create a new virtual switch of type “External”.
    • Name it “External”.
    • Set it to use the “External network” and pick the NIC you commonly use.
    • Press OK.
  • Get an administrative command prompt open.
  • Install https://chocolatey.org/ if you don’t already use it; it’s like yum or apt-get in Linux.
  • choco install minikube
  • choco install kubernetes-cli
  • minikube start –vm-driver “hyperv” –hyperv-virtual-switch “External” –v=7
    • Note that we’re telling it to use Hyper-V and the switch we created.
    • We’re also setting a very verbose log level.

If you have issues after this, you may want to clean up and try again with the information I provided before the steps.

Validation

Let’s try and launch something to make sure it’s working (with instructions copied from the Git Hub link).

  • Launch the “hello-minikube” sample image as a pod.
    • kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
  • Expose it as a service.
    • kubectl expose deployment hello-minikube --type=NodePort
  • Verify the pod is up and running (wait if it is still starting).
    • kubectl get pod
  • Ask Kubernetes for the service URL:
    • minikube service hello-minikube --url
  • Hit that URL in your browser or CURL (you should get back a bunch of text with CLIENT VALUES at the top of it.
  • Hopefully all that is working.  So, let’s remove the service and pod we deployed:
    • kubectl delete service hello-minikube
    • kubectl delete deployment hello-minikube
  • Now you have Minikube running, and it’s a clean copy.

I hope that helps you get started!

User Interface

Refer to this blog for the quick user-interface setup instructions: https://coding-stream-of-consciousness.com/2018/11/08/minikube-set-up-kubernetes-ui/.

Doing Upgrades

You will want to update minikube and the kubernetes CLI after a while.  You can see their versions in the status command and then you can update them easily with chocolatey.

  • minikube status
  • choco upgrade kubernetes-cli
  • choco upgrade minikube
  • minikube update-context