Don’t run your tests with production data. Just don’t. Otherwise, your customers will end up with delightful emails like this:
In jobs past I was pretty cavalier with production data. I was never part of a major incident, but that was more due to luck than strong security controls and separation of concerns. Really thankful that my current employer takes this stuff as seriously as it deserves.
Yesterday I was notified via email that some of my personal information had been compromised by the ParkMobile app. Besides the obvious question of why it took them two months to contact users proactively (vs a post on their website), I found this paragraph a blend of confusing and concerning:
Encrypted passwords were accessed, but not the encryption keys required to read them. We protect user passwords by encrypting them with advanced hashing and salting technologies.
So many things wrong with the above. First, passwords should not typically be encrypted, and certainly not in a way that would allow a company to read them. The better approach is to hash them, which ParkMobile mentions later, but encryption and hashing are not the same thing. It’s hard to trust an organization that doesn’t know the difference.
Additionally, the adjective “advanced” gives me pause. It’s not communicating anything of value. Why not instead tell me the exact algorithm used, such as bcrypt, Argon2, or PBKDF2? Most likely the person writing the response doesn’t really understand the technologies used to protect passwords. And claiming something is advanced when one has no ability to know if such a claim is true screams Dunning-Kruger effect.
Ironically in an attempt to express competence, ParkMobile has done the opposite.
Details matter.
I’ve been enamored recently with building little Python packages that provide useful capabilities for my colleagues. Things like a client for our internal ticketing system, a convenience wrapper to interact with our LDAP servers, etc. And I talk about them non-stop with anyone who will listen, because I truly believe they are helpful for automating mundane tasks across the organization.
But a pesky thing happens when actual people try to use your code. Bugs pop up. Installation errors occur. Weird behaviors manifest. Whatever can go wrong, will go wrong, and if you’re lucky, you’re going to hear about it from your users. If you’re unlucky, no one will bother to contact you or use your code again.
The gap between a toy script you write for yourself and a full-fledged tool usable by others is vast. Numerous compatibility and usability considerations must be considered: differing hardware, operating systems, language versions, user familiarity, probably even more things I’m not thinking of.
Case in point. A few weeks ago I added @functools.cache to a couple packages in key places where expensive API calls were being made. It’s a rad little annotation, but was new to Python 3.9. Didn’t take long before someone hit an error message that was anything but clear to them. Quickly I patched the code up by replacing with @functools.lru_cache which does basically the same thing and has been around for a while longer. But did I bother to test on the older version? No I did not (and shame on me, because tox and pyenv make doing so easy). Once again my user was thwarted, because while Python 3.8 allowed the use of this annotation without a parameter, Python 3.6 and 3.7 do not (and he was running 3.6.9). Once again, I had to quickly patch, but this time I made sure my tox setup tested across versions.
The worst part of the above is that I could have better guarded against an esoteric error message by specifying precise compatibility in setup.py (in this case with a python_requires='>=3.6' instead of python_requires='>=3'). If I’d have done that, my user would have still gotten an error, but it would have happened at install time, and been much clearer as to the cause. Two extra characters in an install script, but hugely improved customer experience.
Like I said, details matter.
During interviews, a common question directed at me is what my day-to-day looks like. While it feels like a cop-out, the answer is that I do whatever my customers need me to do. Some days that might mean writing Python, Node.js, or Java. Other days it might mean terraform, CDK, or (ugh) CloudFormation. Still other days it might not mean code at all, but leading daily stand-ups, drawing architecture diagrams, writing documentation, giving training, or planning long-term goals. And that’s not to mention all my managerial tasks like 1-on-1s, goal setting, performance reviews, an mentoring, since I view my team as customers of a sort.
We talk at AWS of being comfortable with ambiguity. It’s a posture that doesn’t suit everyone, but pretty much any technical job of worthwhile complexity requires the ability to deal with unknowns. There are no hard and fast rules. No absolutes. System design, software development, team leadership, they are all just as much art as science, requiring both good intuitions and good data to get right.
Don’t believe me? Then consider how Stack Overflow tells it.
We talk all the time about customer obsession, it’s the first amongst equals of our Leadership Principles. The general idea is that you always imagine the customer is in the room with you when making decisions, and consider their preferences when implementing a feature.
As email got more prevalent (and with it, junk emails), one feature that came along in the spirit of customer obsession was one-click unsubscribe. Me being a person who strives for inbox zero, stopping email subscriptions easily was a welcomed capability. But who knew it could be done even better. Behold, zero-click unsubscribe:
How cool is that? Recognizing that I haven’t interacted with any of their emails or visited their site, they automatically unsubscribed me from their lists. That’s customer obsession in a nutshell.
The really surprising thing about this capability is that it’s surely dirt simple to implement. I bet I could pseudo-code it off the top of my head in one minute. Ready, go!
all_emails = database.get_mailing_list()
for email_address in all_emails:
last_contact = database.get_last_contact(email_address)
if datetime.today() - last_contact > ONE_YEAR:
database.remove(email_address)
See how easy that was? Any marketing email system worth its salt already knows when you click on a link in any of their mailings, so the data is there. Just takes a company thinking more about their customers than the size of their distro.
I’ve written before about the importance of writing documentation. Like anything else it’s a skill that takes both training and practice. The latter only takes time, but there’s not a lot of material out there on the former. Therefore it made me happy to discover Write The Docs, a global community of people who care about documentation.
The recommendations in their Beginner’s Guide To Writing Documentation I found quite helpful. And loved this quote, which is only tangentially related to docs:
Fear is what happens when you’re doing something important. If you are doing work that isn’t scary, it isn’t improving you or the world.
The article argues that writing documentation for yourself and others is one way to alleviate fear, and I agree. So go forth and write!
Today I came across this statement from Alfred North Whitehead, and instantly loved it as an extension of my previous post on abstractions.
“Civilization advances by extending the number of important operations which we can perform without thinking about them.”
That to me is the essence of abstractions. Not that one needn’t ever be required to dig down into the implementation details, but that the layer on top of those details enables them to be ignored to an increasing degree.
Incidentally, this is the second Alfred North Whitehead reference I’ve come across recently, the first being a mention of his book Science and the Modern World in one of my favorite podcasts. Something tells me I need to dive deeper.
I doubt we’ll ever learn the root cause of a particularly nasty security vulnerability recently revealed in MacOS High Sierra. But it’s fun to theorize. I’d wager it went down about like this:
root password, because it could cause usability issues for non-technical people buying Macsroot as an option, or at least not to allow it if no password had been setroot passwordroot account in various places (see #4)Good times, Apple. I feel your pain.
I’ve had the pleasure of working with a large variety of technologies over the course of my career. Yesterday I was working on an interface to an old government database, without the aid of documentation, natch. After a few hours I was able to extract data from the system, but I was unable to decode it. Google is a great tool (I couldn’t get 15 minutes into my day without it), but if you don’t know what to search for you can’t find anything. Thankfully some careful guesses led me to the Wikipedia entry for EBCDIC, a character encoding developed in the mid-60s.
“Extended Binary Coded Decimal Interchange Code (EBCDIC) is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. EBCDIC descended from the code used with punched cards and the corresponding six bit binary-coded decimal code used with most of IBM’s computer peripherals of the late 1950s and early 1960s.”
It’s a fun thing in this industry to say you’ve worked with 50-year-old technology, but only once you’ve figured it out. Surprisingly there’s some modern tooling to interface with this particular encoding (yay Python!), so once I knew what I was dealing with it was straightforward enough. Maybe even easier than what it used to be, if this anecdote is to be believed:
EBCDIC: An alleged character set used on IBM dinosaurs. It exists in at least six mutually incompatible versions, all featuring such delights as non-contiguous letter sequences and the absence of several ASCII punctuation characters fairly important for modern computer languages (exactly which characters are absent varies according to which version of EBCDIC you’re looking at). IBM adapted EBCDIC from punched card code in the early 1960s and promulgated it as a customer-control tactic, spurning the already established ASCII standard. Today, IBM claims to be an open-systems company, but IBM’s own description of the EBCDIC variants and how to convert between them is still internally classified top-secret, burn-before-reading. Hackers blanch at the very name of EBCDIC and consider it a manifestation of purest evil.
There are really only two tasks a company must perform:
Each employee, no matter her job title or role, must contribute in some way to one or both of those two goals. Any other task is only a means to one of these ends; everyone involved does well to remember that.
And yes, I realize strictly speaking products and services are two different things, but for the sake of pith can we consider the latter a type of the former? Thanks.