The causes of, and solutions to, all of life’s problems.

It isn’t often that an intermittent network connect is a benefit, but in this case a connection hiccup actually tipped me off to a useful workaround.

When you’re an engineering manager, you’re “important”, which means you have to go to a lot of meetings. And because you’re so very “important”, you can’t be troubled to close your laptop when walking across the office to said meetings, because you might miss someone’s giphy on Slack. Pretty sure I looked like an idiot, but that’s the price you pay for being in charge. Or something.

Anyhow, I’d been fighting an npm issue all morning (natch), where a particular module (bcrypt) was core dumping on my Mac. Not cool, bcrypt, not cool. Couldn’t figure out what was going on, but as is typical, “have you tried erasing your node_modules folder and re-running npm install?” Actually I had, but I was getting desperate, so thought I’d give it one more go. While simultaneously picking up my laptop to head to a meeting (keeping it open as I walked, because “important things” happening on it).

I arrived at the meeting (no idea what it was about, also pretty typical), and when the npm install had finally finished, I tried the program again, and lo and behold, it worked! I think at this moment I audibly exclaimed my excitement, despite the outburst not fitting the context of the meeting, that’s how happy I was. But I was also a bit befuddled. What had changed?

So I pored over the logs, both from the install that didn’t work, and the one that had (God bless anyone that ever has to review an npm log, it’s a special kind of hell).  Check out extracts from the install that failed to run, and the one that worked. Do you see the difference?

Please go look. I’ll wait.

Figure it out?

Did you notice that the binary of bcrypt failed to download in the second log, and npm fell back to compiling from source? That was the secret! Something must’ve been wrong with the prebuilt version for Mac. Now, I never solved what caused the crash in that build, but it was easy enough to work around it with npm --build-from-source.

But the real serendipity was the likely cause of the download failure. The only explanation I can think of is that our office’s crummy WiFi happened to flake out briefly as I was carrying my open laptop across the hall, just at the moment when the bcrypt binary was being downloaded, causing it to fail. But the network was back by the time the source tarball was downloaded, and the reset of the process finished normally.

Even as I write, it sounds preposterous. What are the odds? Maybe it was something else, I don’t have any proof. But you’ll never convince me.

This blog is turning into a bit of a “debugging adventure of the day”, which wasn’t the original intent, but whatever. I’ll get around to meatier posts someday.

This morning’s adventure was debugging a pretty serious issue with a web application where selections would not save. Initially it only manifest itself on Windows machines, so we thought Microsoft was to blame. Oddly, though, Chrome failed to work but Edge was fine. Weird.

Wanting to check a few other variables, we grabbed another co-worker’s Windows laptop, and it worked fine in Chrome. WTF!?! Nearly identical hardware, same OS, same browser, and same browser version, but one machine worked, and the other didn’t. Cleared caches, rebooted, etc. Same results.

But did you catch the word “nearly” there? Turns out there was a difference between the two laptops. One had a touchscreen, and the other didn’t. Fast-forward through an hour of slogging through a bunch of hacked together JavaScript code to support drag-and-drop on a variety of devices, and sure enough, the “non-mobile yet touchscreen enabled” case had its own special snowflake handler that wasn’t working correctly in Chrome.

Why you ask? Well, in some browsers, touch events are delayed by 300ms (this is so things like double-tap and dragging can be detected properly). This causes the mouse click event to go first, and when clicking on a button that submits the form, the form gets processed before the touch event can fire that needs to run and configure the inputs to the form submission. If you touch the button instead of clicking with the trackpad (a condition that only makes sense on a touch-enabled laptop vs a tablet), it works okay.

That means this situation only manifest itself on a touchscreen-enabled non-mobile device, in Chrome, when the user clicks the submit button with the trackpad vs touching the button. Can’t hardly blame QA for not finding that one.

Software development is not for the faint of heart.

• The SQL DELIMITER  statement is particular to the mysql client tool, so you can’t use a standard dump file to bootstrap initialize a new database instance using mysqld –bootstrap. Eight hours later I figured out a workaround, namely exporting stored procedures directly from the mysql.proc  table.
• A single stray character can entirely break a critical piece of system functionality; finding and removing it required a late night from multiple team members.

And that was just yesterday.

Yesterday the team spent most of the day debugging an urgent production issue. The root cause: a seemingly innocuous change made to a custom storage library eight months ago. Lesson learned: changes may have unintended side-effects.

And today I’m back to decompiling Java bytecode from a production server. The original source is long gone. Good times.