Kubernetes Ingress Service 60 Second Timeout

Kubernetes has multiple levels of timeouts for calls, including at the load balancer, inbound to the ingress itself, and at the individual ingress resources.

Assuming you have the first two configured and are still hitting a timeout on your  app, this is the annotation you need to add to your service’s ingress resource to boost its timeout:

nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

This will change it up to 1 hour from 1 minute.

Bash xargs – Use Anywhere in Your Command

If you want to use the xargs command to build a new command and place the standard input from your stream wherever you like (or in multiple places), here is an example:

Let’s say we have a hosts file:

111.222.111.222 host1.mydomain.com
222.111.222.111 host2.mydomain.com

We can take out the first column with awk (the IP column), and we can pipe it to xargs and use -I. Then we can use the % sign to both ssh to the IP and to display a message with the IP written from the host.

This is obviously very flexible and useful for pretty much any command you’re building from earlier inputs :).

cat dynamic_hosts | awk '{print $1}' | xargs -I % ssh % "echo hi from %"

Use wget with self-signed certificate and output to stdout

Not much of an article here… but if you want to use WGET to validate a spring boot app with a self-signed certificate, for example, and you don’t want to bother outputting the results to file, you can do this:

wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate

The output would be right inline and would avoid the certificate validation:

# wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate
{“status”:”UP”} #

You could achieve this with curl as well I’m sure – but it isn’t installed on alpine it seems and I didn’t want to install extra tools.

 

Kubernetes View NGINX Ingress Config File

When you are using the (nginx) ingress resource in kubernetes, you may want to view the actual configuration it is running, even though you are fairly abstracted from it. It can be very useful for debugging complex issues.

You can run these two lines to get the config file into a local nginx.conf file for inspection when you need to:

POD=$(kubectl get pods -n kube-system | grep ingress | awk '{print $1}')
kubectl exec -it -n kube-system ${POD} cat /etc/nginx/nginx.conf > nginx.conf

Generate a Self-Signed p12 (pfx) certificate File (Automated / Scripted)

Here is a brief script to generate a self-signed p12 (pfx) certificate file with no interactive input.

You can parameterize the three inputs and use them to run this from a CI pipeline/etc as required.

#! /bin/bash

# Set some variables for ease/re-use (can be parameterized).
NAME_PREFIX="mycertname"
PASSPHRASE="some-password"
DNS_NAME="*.some.domain.net"

# Generate cacert.pem and cakey.pem.  The private key is encrypted.
openssl req -newkey rsa:2048 -x509 -keyout ${NAME_PREFIX}-cakey.pem -out ${NAME_PREFIX}-cacert.pem -days 3650 -subj "/C=US/ST=New York/L=New York/CN=${DNS_NAME}" -passout pass:${PASSPHRASE}

# Get a decrypted copy of the private key.
openssl rsa -in ${NAME_PREFIX}-cakey.pem -out ${NAME_PREFIX}-cakey-decrypted.pem -passin pass:${PASSPHRASE}

# Generate the p12 file from the private key and certificate.
openssl pkcs12 -export -out ${NAME_PREFIX}.p12 -inkey ${NAME_PREFIX}-cakey-decrypted.pem -in ${NAME_PREFIX}-cacert.pem -passout pass:${PASSPHRASE}

# Remove the decrypted private key.
rm ${NAME_PREFIX}-cakey-decrypted.pem

Minikube Start Failure; Streaming server stopped, cannot assign requested address.

My Problem

I was attempting to downgrade my minikube kubernetes version to match an EKS cluster I had running in AWS.

This should have been fairly simple:

sudo minikube delete
minikube start --vm-driver=none --kubernetes-version 1.14.9

Unfortunately, it failed! Minikube would pause on staring kubernetes for about 4 minutes, and then fail. The kubelet was not coming up for some reason.  The output was huge, but this caught my eye:

Streaming server stopped unexpectedly: listen tcp … bind: cannot assign requested address

I spent about 2 hours going back and forth and even tried rebooting my laptop and starting a current/new version cluster again (which was already working), all to no avail.

The Solution

Eventually, I saw a post which suggested I had networking problems, and from that point I worked out that my /etc/hosts file was messed up.  This line was commented out from when I was toying around with some DNS name faking on another project:

#127.0.0.1 localhost

So, localhost wasn’t defined and weird things were happening (obviously). Un-commenting this and starting minikube worked after that.

I’m sure this error can manifest from other networking issues as well; hopefully this saves you some time or points you in the right direction at least.

Minikube helm list – error forwarding port, socat not found.

I was trying to downgrade my minikube cluster to have it match my cloud EKS cluster version wise.  I had various issues in this process, but got everything working after a while… except helm.

So, I could use kubectl and list pods, I had done a helm init to install tiller (or enabled the minikube tiller addon), and everything was working from that angle.

Doing a “helm list” was giving an interesting error though:

$ helm list                                           
E0115 15:11:56.503124   32232 portforward.go:391] an error occurred forwarding 33511 -> 44134: error forwarding port 44134 to pod 9d0d87a8b6d37fc96b7947d1b21c273c3e9dd63207253570f0694ee2db91c545, uid : unable to
 do port forwarding: socat not found.

After a while, I found this github entry: https://github.com/helm/helm/issues/1371 which says to install socat (which would seem obvious from the error message; but I don’t like to be hasty).

So, I ran:

sudo apt-get install socat -y

and then helm started working like a charm =). Hope that helps someone else too!