Spent my morning debugging a data extraction pipeline that’s been failing to properly update a dashboard, despite it completing without error (lovely). Digging through the output of the pipeline software got me thinking about logging. There’s a real art to getting it right: too much is impossible to parse (and potentially leaks sensitive information), but too little is useless. Sometimes the difference between finding the source of the problem and not is having logs in a Goldilocks Zone.

In today’s case, the log looked something like this:

- Calling the API
- The API call succeeded

Better than nothing, at least it would have let me know if the API call was failing altogether. But this would have been so much more helpful:

- Calling the API with parameters X=a and Y=b
- The API call succeeded and returned N items

I updated the data pipeline to provide the above output, and will see how it runs tonight.

Details matter, y’all! I wish I knew a way to teach the above to software engineers, but I fear there’s no real way to learn except through the experience of having to do this sort of debugging. It’s not fun work, but it’s oh so valuable.