Kubernetes Ingress Service 60 Second Timeout

Kubernetes has multiple levels of timeouts for calls, including at the load balancer, inbound to the ingress itself, and at the individual ingress resources.

Assuming you have the first two configured and are still hitting a timeout on your  app, this is the annotation you need to add to your service’s ingress resource to boost its timeout:

nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

This will change it up to 1 hour from 1 minute.

Use wget with self-signed certificate and output to stdout

Not much of an article here… but if you want to use WGET to validate a spring boot app with a self-signed certificate, for example, and you don’t want to bother outputting the results to file, you can do this:

wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate

The output would be right inline and would avoid the certificate validation:

# wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate
{“status”:”UP”} #

You could achieve this with curl as well I’m sure – but it isn’t installed on alpine it seems and I didn’t want to install extra tools.

 

Kubernetes View NGINX Ingress Config File

When you are using the (nginx) ingress resource in kubernetes, you may want to view the actual configuration it is running, even though you are fairly abstracted from it. It can be very useful for debugging complex issues.

You can run these two lines to get the config file into a local nginx.conf file for inspection when you need to:

POD=$(kubectl get pods -n kube-system | grep ingress | awk '{print $1}')
kubectl exec -it -n kube-system ${POD} cat /etc/nginx/nginx.conf > nginx.conf

Minikube Start Failure; Streaming server stopped, cannot assign requested address.

My Problem

I was attempting to downgrade my minikube kubernetes version to match an EKS cluster I had running in AWS.

This should have been fairly simple:

sudo minikube delete
minikube start --vm-driver=none --kubernetes-version 1.14.9

Unfortunately, it failed! Minikube would pause on staring kubernetes for about 4 minutes, and then fail. The kubelet was not coming up for some reason.  The output was huge, but this caught my eye:

Streaming server stopped unexpectedly: listen tcp … bind: cannot assign requested address

I spent about 2 hours going back and forth and even tried rebooting my laptop and starting a current/new version cluster again (which was already working), all to no avail.

The Solution

Eventually, I saw a post which suggested I had networking problems, and from that point I worked out that my /etc/hosts file was messed up.  This line was commented out from when I was toying around with some DNS name faking on another project:

#127.0.0.1 localhost

So, localhost wasn’t defined and weird things were happening (obviously). Un-commenting this and starting minikube worked after that.

I’m sure this error can manifest from other networking issues as well; hopefully this saves you some time or points you in the right direction at least.

Minikube helm list – error forwarding port, socat not found.

I was trying to downgrade my minikube cluster to have it match my cloud EKS cluster version wise.  I had various issues in this process, but got everything working after a while… except helm.

So, I could use kubectl and list pods, I had done a helm init to install tiller (or enabled the minikube tiller addon), and everything was working from that angle.

Doing a “helm list” was giving an interesting error though:

$ helm list                                           
E0115 15:11:56.503124   32232 portforward.go:391] an error occurred forwarding 33511 -> 44134: error forwarding port 44134 to pod 9d0d87a8b6d37fc96b7947d1b21c273c3e9dd63207253570f0694ee2db91c545, uid : unable to
 do port forwarding: socat not found.

After a while, I found this github entry: https://github.com/helm/helm/issues/1371 which says to install socat (which would seem obvious from the error message; but I don’t like to be hasty).

So, I ran:

sudo apt-get install socat -y

and then helm started working like a charm =). Hope that helps someone else too!