We’ve been hitting docker rate limiting pretty hard lately in our EKS clusters. Here are some interesting things we learned:
- The anonymous request rate limit for DockerHub is 100 requests per IP address per hour.
- If you are in a private IP space and have internet gateways, you are probably being rate limited on the IPs of the gateways.
- So, if you have 600 servers going through 6 gateways, you have 600 requests, not 60,000 (obviously this is a massive difference).
- In kubernetes, you should specify an image tag (which is not mandatory) and pull-if-not-present in order to ensure you pull images less frequently.
If you need to observe your servers and how they are acting with the rate limit, you can refer here -> https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/.
For anonymous requests, basically just run:
TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token) curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep ratelimit
And you will get output like this, showing the rate limit (100) and how many you have left (100 for me as I haven’t pulled recently).
RateLimit-Limit: 100;w=21600 RateLimit-Remaining: 100;w=21600