Docker Fail

One of my favorite truisms in software development is the following:

“When two things aren’t the same, then they’re different.”

I don’t care how hard you work to make development and production environments the same, I don’t care how portable you think your programming language is across operating systems, and I especially don’t care about claims made on a product’s marketing material. Case in point:

[Docker] makes for efficient, lightweight, self-contained systems and guarantees that software will always run the same, regardless of where it’s deployed.

No doubt the Docker team has worked incredibly hard to make the above statement true, but nothing in life is guaranteed with 100% certainty. Last week I discovered that the cron utility will not execute any crontab or script that has more than one hard link pointed at it (don’t ask me why not, it just doesn’t). And due to the way Docker’s overlay filesystem works, the operating system can report a regular file as having multiple links. At least it does when the container is running under Kubernetes. But not always! When running with docker-compose on my Mac the crontab only reported one hard link, and thus cron worked great.

Took me nearly a day of fiddling to determine why the container worked great locally but failed in Kubernetes. Argh.