Kubernetes Ingress Service 60 Second Timeout

Kubernetes has multiple levels of timeouts for calls, including at the load balancer, inbound to the ingress itself, and at the individual ingress resources.

Assuming you have the first two configured and are still hitting a timeout on your  app, this is the annotation you need to add to your service’s ingress resource to boost its timeout:

nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

This will change it up to 1 hour from 1 minute.

Use wget with self-signed certificate and output to stdout

Not much of an article here… but if you want to use WGET to validate a spring boot app with a self-signed certificate, for example, and you don’t want to bother outputting the results to file, you can do this:

wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate

The output would be right inline and would avoid the certificate validation:

# wget -q -O – https://172.20.234.165/actuator/health –no-check-certificate
{“status”:”UP”} #

You could achieve this with curl as well I’m sure – but it isn’t installed on alpine it seems and I didn’t want to install extra tools.

 

Kubernetes View NGINX Ingress Config File

When you are using the (nginx) ingress resource in kubernetes, you may want to view the actual configuration it is running, even though you are fairly abstracted from it. It can be very useful for debugging complex issues.

You can run these two lines to get the config file into a local nginx.conf file for inspection when you need to:

POD=$(kubectl get pods -n kube-system | grep ingress | awk '{print $1}')
kubectl exec -it -n kube-system ${POD} cat /etc/nginx/nginx.conf > nginx.conf

Minikube Start Failure; Streaming server stopped, cannot assign requested address.

My Problem

I was attempting to downgrade my minikube kubernetes version to match an EKS cluster I had running in AWS.

This should have been fairly simple:

sudo minikube delete
minikube start --vm-driver=none --kubernetes-version 1.14.9

Unfortunately, it failed! Minikube would pause on staring kubernetes for about 4 minutes, and then fail. The kubelet was not coming up for some reason.  The output was huge, but this caught my eye:

Streaming server stopped unexpectedly: listen tcp … bind: cannot assign requested address

I spent about 2 hours going back and forth and even tried rebooting my laptop and starting a current/new version cluster again (which was already working), all to no avail.

The Solution

Eventually, I saw a post which suggested I had networking problems, and from that point I worked out that my /etc/hosts file was messed up.  This line was commented out from when I was toying around with some DNS name faking on another project:

#127.0.0.1 localhost

So, localhost wasn’t defined and weird things were happening (obviously). Un-commenting this and starting minikube worked after that.

I’m sure this error can manifest from other networking issues as well; hopefully this saves you some time or points you in the right direction at least.

Minikube helm list – error forwarding port, socat not found.

I was trying to downgrade my minikube cluster to have it match my cloud EKS cluster version wise.  I had various issues in this process, but got everything working after a while… except helm.

So, I could use kubectl and list pods, I had done a helm init to install tiller (or enabled the minikube tiller addon), and everything was working from that angle.

Doing a “helm list” was giving an interesting error though:

$ helm list                                           
E0115 15:11:56.503124   32232 portforward.go:391] an error occurred forwarding 33511 -> 44134: error forwarding port 44134 to pod 9d0d87a8b6d37fc96b7947d1b21c273c3e9dd63207253570f0694ee2db91c545, uid : unable to
 do port forwarding: socat not found.

After a while, I found this github entry: https://github.com/helm/helm/issues/1371 which says to install socat (which would seem obvious from the error message; but I don’t like to be hasty).

So, I ran:

sudo apt-get install socat -y

and then helm started working like a charm =). Hope that helps someone else too!

Lookup Tiller Version in Kubernetes Using Kubectl

This is just a simple command to help you in finding the tiller version running in kubernetes.  I made it when trying to make sure my laptop’s helm install matched the cluster tiller install:

TILLER_POD=`kubectl get pods -n kube-system | grep tiller | awk '{print $1}'`
kubectl exec -n kube-system $TILLER_POD -- /tiller -version

Debugging Spring Boot Multi Stage Docker Build – JAR Extract Not Working

Containerized Spring Boot – Multi Stage Build

I was following this good tutorial on deploying spring-boot apps to kubernetes:

https://spring.io/guides/gs/spring-boot-kubernetes/

Their docker file looks like this:

FROM openjdk:8-jdk-alpine AS builder
WORKDIR target/dependency
ARG APPJAR=target/*.jar
COPY ${APPJAR} app.jar
RUN jar -xf ./app.jar

FROM openjdk:8-jre-alpine
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY --from=builder ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --from=builder ${DEPENDENCY}/META-INF /app/META-INF
COPY --from=builder ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.example.demo.DemoApplication"]

So, it is using a multi-sage docker build.  The reasoning for this seems to be described well in this other article here: https://spring.io/blog/2018/11/08/spring-boot-in-a-container.  Basically:

A Spring Boot fat jar naturally has “layers” because of the way that the jar itself is packaged. If we unpack it first it will already be divided into external and internal dependencies. To do this in one step in the docker build, we need to unpack the jar first. For example (sticking with Maven, but the Gradle version is pretty similar):

There are now 3 layers, with all the application resources in the later 2 layers. If the application dependencies don’t change, then the first layer (from BOOT-INF/lib) will not change, so the build will be faster, and so will the startup of the container at runtime as long as the base layers are already cached.

Build Failing

In my case, I was getting an error like this:

Step 9/12 : COPY --from=0 ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY failed: stat /var/lib/docker/overlay2/72b76f414c0b527f00df9b17f1e87afb4109fa6448a1466655aeaa5e9e358e27/merged/target/dependency/BOOT-INF/lib: no such file or directory

This was a little confusing.  At first I assumed, that the overlay file system path didn’t exist.  When I tried to go there (to the 72… folder), it in fact was not present.  This was a red herring though.

It is failing to copy BOOT-INF/ content.  So, on a hunch, I assumed the 72… folder path was ephemeral.  If that is true, then it did exist at some time, and the path did not have BOOT-INF content in it.

So, the next step was to debug the intermediate container for the multi-stage build.  In my docker file, the first like is:

FROM openjdk:8-jdk-alpine AS builder

So, the name of the intermediate container is “builder”.  We can stop the build at that container to interrogate it by doing this:

docker build . --target=builder
Sending build context to Docker daemon   17.2MB
Step 1/5 : FROM openjdk:8-jdk-alpine AS builder
 ---> a3562aa0b991
Step 2/5 : WORKDIR target/dependency
 ---> Using cache
 ---> 1dbeb7fad4c0
Step 3/5 : ARG APPJAR=target/*.jar
 ---> Using cache
 ---> 67ba4a4a1863
Step 4/5 : COPY ${APPJAR} app.jar
 ---> Using cache
 ---> d9da25b3bc23
Step 5/5 : RUN jar -xf ./app.jar
 ---> Using cache
 ---> b513f7190731
Successfully built b513f7190731

And then interactively running in the built image (b513f7190731):

docker run -it b513f7190731

In my case, I could clearly see that basicalyl no files existed in the /target/dependency folder.  There was only a JAR that was never extracted by the “jar -xf ./app.jar” command.

Debugging the JAR Extraction

I tried to manually run the jar extraction command myself, and it gave no error and did absolutely nothing.  After that, I went to make sure the JAR was valid.  When I looked at its size, it was fine.  When I looked at it with “vi”, I could see that the header was a bash script.

This is because my spring-boot project had the spring-boot-maven plugin in its POM with executable = true; this embeds a bash script in the front to allow it to run as a Linux init.d service.

So, my first move was to remove the plugin altogether.  This did not help as, after that, the JAR did extract but it had no BOOT-INF folder.  Spring boot was not building the fat-jar with dependencies as I removed the plugin.  So, I had to put the plugin back in like this:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
</plugin>

With executable = true removed.  This builds the fat JAR without the bash file embedded, which allows the “jar -xf ./app.jar” command to properly extract the contents including the BOOT-INF dependencies.

This was not a problem in the tutorial because they’re relying on generated projects from Initializr rather than existing projects you made yourself with possibly different use cases.

Summary

I hope this helped you learn about putting spring-boot in docker, and that it shed some light on how to debug a multi-stage docker build!  I knew a fair bit about docker and spring-boot going into this, but combining them revealed some process differences I had taken for granted over the years.  So, it was a good experience.