Knobs And Buttons

Knobs And Buttons

By | | Comments 0 Comment

There’s a lot to consider when using AI. There are a number of input variables, not the least of which is a free text prompt that can be shaped and rephrased an incredibly large (but not infinite) number of ways for similar requests. Getting to an “optimal” result is probably impossible, and event getting to a local maximum is tough. A ton of testing is required, and that’s neither quick nor cheap.

In order to target variables with high effect, it’s helpful to have heuristics and other rules-of-thumb. To that end, I found this article from Anthropic instructive.

When a result misses the mark, ask, “did Claude not know enough or did it not try hard enough?”

If the former, use a bigger model. If the latter, increase the effort setting. Simple enough!

Another quick win is to consider how much planning a model can do on its own. Per Ken Huang in Claude Fable 5 (Part 1): What Changed, and How to Stop Prompting It Like Opus:

For three model generations we compensated for planning weakness by doing the planning ourselves: numbered step lists, “think step by step,” long behavior checklists, skills files that read like flight manuals. Fable 5 plans better than those crutches, and the crutches now get in the way.

In short: let bigger models do more of the work for you. In my recent experience, Fable 5 especially (and even Sonnet 5 at times) don’t need nearly the step-by-step prompting to get the job done, and over-specifying both wastes your time and potentially gives worse results.

Shiny Happy People

Shiny Happy People

By | | Comments 0 Comment

I just got back this week from a vacation that I’ve been dreaming to avail myself of for a decade. It was delightful, and once again served as a reminder that to be effective at my job requires periodic distance from it (though admittedly I did check in a couple times, because sometimes I need a break from my break).

Before heading out I’d wrapped a multi-week project that was a collaborative effort across several organizations. That kind of achievement feels good in and of itself, but I was also reminded that working with great people can be quite a bit of fun. And that fun can power great results. Happiness in work matters!

The project in question was related to verifiable credentials, of which I’ve spoken before. That post has some solid background links on the topic; here’s another series from NIST that’s also instructive:

Or if you have even less time, these one-page introductions to three fundamental concepts, from my friends at Freewallet:

Happy reading!

After My Own Heart

After My Own Heart

By | | Comments 0 Comment

Went back in my draft backlog and found this gem from 2020: Unit Testing is Overrated.

In the age of AI-generated code, it feels even more applicable. When a model writes unit tests (especially when it does it in view of the code it’s testing) they’re at risk of being overfitted to the functions under test. They may indeed prove software executes as it’s written, but that has little to do with proving the software meets requirements (for example, Kiro had created hundreds of perfectly passing tests for this project).

The key takeaways from the article are all worth sticking in your coding agent instructions, because without explicit directives, LLMs are probably biased to do the opposite of these recommendations given the weight of training data pushing so-called “best practices.”

In particular, I can see value in using a separate (perhaps even adversarial) agent/model to write the tests. It’ll be less biased on the context used to write the code, and it can be instructed to “aim at the highest level of integration while maintaining reasonable speed and cost.”

Maintenance Matters

Maintenance Matters

By | | Comments 0 Comment

In case I haven’t said it explicitly enough, I’ll do so again here.

I wrote these definitions a couple years ago in an effort to bring some formality to what would be covered under a maintenance contract. They served well enough to pass along here.

Major Issue: Problems that impede core functionality, such as:

  • application fails to load
  • users cannot log in
  • workflows are inaccessible or fail

Minor Issue: Problems that do not impede core functionality, such as:

  • slow, but functional, responses to user interactions
  • workflows succeed but are sub-optimal
  • typos and visual glitches

Feature Enhancement: Updates that meet all of the following criteria:

  • do not fundamentally alter an existing capability
  • do not create a new user workflow
  • do not require new integrations with or data from external systems

New Feature: Updates that meet one or more of the following criteria:

  • add a capability that did not previously exist in the system
  • create a new user workflow
  • require integration with an external system

Environments

  • Dev: where software engineers do their daily development work
  • QA: where vendor work is tested before going to customer for acceptance
  • UAT: where customer performs final test and acceptance of features
  • Production: where users access the system
All’s Well That Ends Well

All’s Well That Ends Well

By | | Comments 0 Comment

I reposted the broader announcement on LinkedIn, but thought it’d also be worth capturing here, that the final proposals for the Technical Executive Leadership Initiative class I took earlier this year have been published. I was part of a great collaborative team; while I had some contributions across the board, my focus was primarily the mockup website and accompanying demonstration.

Digitizing the JobsNOW! Intake Process for Medi-Cal Clients in San Francisco

If you’re a technologist working in the public sector and want to get better at working with government, I highly recommend this program. You’ll also get a nifty public profile out of the deal (as if I needed more to add to the collection).

Judson Neer

I’m now reminded that it’s been a few years since I’ve had a professional headshot taken, something I should probably rectify.

Not All Rainbows and Unicorns

Not All Rainbows and Unicorns

By | | Comments 0 Comment

As much as I love being a generalist and believe it’s the better end of the breadth/depth spectrum in the current tech environment, it doesn’t always feel like a great way to operate, for several reasons:

First, it’s often difficult for a generalist to describe their job, not just to folks like friends and family who are unfamiliar with their domain, but also to managers in 1-on-1s, performance reviews, and promotion documents (I’ve had firsthand experience with the latter). A term related to generalist, factotum, can even have negative connotations in certain settings, despite a “glue person” being an essential role within pretty much any organization. Specialists, on the other hand, are easier to understand, as their work is often simply “what it says on the tin” (i.e. their job title).

Second, while a generalist has proximate knowledge of many things, when they spend time with legitimate experts in a topic (which is often, because they’re so curious), they are quick to realize how much they don’t know. And since this happens so much, the experience of “I don’t know as much as that other person” compounds into a malaise of “eh, I don’t know much about anything.” Inferiority complex ensues.

When interests and aptitudes are varied, it’s also tough to determine where to focus, paradox of choice and all that. Similarly, while there’s value in doing extra at times, it’s easy to get distracted and fail to deliver the most important thing, losing the proper balance between work work and non-work work. This leads to a feeling that one is “mildly disappointing everyone all the time.” When I have that feeling, burnout lurks not far behind.

I don’t have a tidy conclusion to give here, it’s just stuff I’ve been sitting with.

Follow The Crowd

Follow The Crowd

By | | Comments 0 Comment

“The more you tighten your grip, Tarkin, the more star systems will slip through your fingers.”

An addendum to yesterday’s post on tools: while there’s a lot of value in allowing individual engineers to customize their development setups, that doesn’t extend to organizations. It’s a rare company that’s such a special snowflake that it should eschew the broader ecosystem and build its own homegrown tooling (even one as big as Amazon, with which I have firsthand experience). And it’s doubly bad if it mandates use of said tooling from on high.

It’s much easier to motivate folks to care for the tools of their trade when they can select them, including being able to bring what they’ve used in prior roles, and then know that any further customization they do currently will be applicable to their next role and can be taken with them when they leave.

Thinking that adoption and embrace of custom solutions will produce loyalty will do quite the opposite. The cost of supporting diverse individuality is worth paying.

Great Power, Great Responsibility

Great Power, Great Responsibility

By | | Comments 0 Comment

On my weekend long run I was listening to this interview with Stewart Brand, amongst many other things the author of Maintenance: Of Everything, a book I read last month and highly recommend, since maintenance matters.

The whole episode is worth a listen, but what jumped out at me most was a discussion about the right to repair. The thought I had was not about the traditional sense of a consumer right relative to a manufacturer (though that’s important too), but for an employee relative to the equipment provided by their employer.

When a person is given the capability, and indeed the responsibility, to care for the tools of their trade, it sets individual ownership of outcomes as the standard. Excuses for not using the very best fade away. Denying that responsibility leads to blameshifting, “not my problem”-ism, and an inevitable regression to median (read: middling) results, and that’s if you’re lucky.

Being good at maintaining and tools is also a decent heuristic for whether a person is good at creating what those tools are designed to produce (not foolproof, but probably better than this one). “Tell me about 3 ways you’ve customized your development laptop and how they’ve made you better” is an interview question rich in signal.

We haven’t even talked about the “actual work” having the right access to tools empowers. It’s hard to dive deep when access to all but the kiddie pool is roped off. It’s even harder to experiment. Discouraging both behaviors is a cultural failure with tremendous cost. Not just to a company’s ability to innovate, but in their engineers’ ability to learn. So much of the breadth of skill I’ve picked up over my 25 years has come from solving ancillary problems related to tooling. To offload that burden would have been pedagogically disastrous. Perhaps yak shaving shouldn’t have the negative connotation it usually gets?

I realize there are situations where giving full administrative power at scale isn’t feasible. The blast radius of a laptop is quite a bit wider than a toaster, and compliance regimes have guardrails for hard-earned reasons. Nevertheless, I believe the risk is worth the reward.

Truth Behind The Truth

Truth Behind The Truth

By | | Comments 0 Comment

I’m now six months into a new full-time gig, just about the time when it feels like getting settled. As I was going back through old drafts to get published, this one felt particularly applicable in this moment.

Every organization has two org charts: the official one, which is usually traceable via something like Outlook Org Explorer or internal tooling like the one at Amazon whose name I won’t say (but that I enjoyed immensely, and even wrote software to interact with it programmatically).

(Credit to Manu Cornet for this extremely true diagram)

The other is less obvious, rarely written down, and can only be discovered through concerted social effort. It’s also the more important of the two, because it’s through the multi-layered collaborative connections woven throughout a company where the actual work gets done.

It’s incumbent on leaders to make some efforts to bring these two org charts into alignment (a Platonic ideal of sorts), but it’s ultimately not possible, because human collaboration is complex. Complaining about the fact of two hierarchies and sets of inter-relationships doesn’t help either (believe me, I’ve tried). The only path forward, even when it’s frustrating, is to accept the reality and operate within it.

In short: know how your org works.

The Truth Sets You Free

The Truth Sets You Free

By | | Comments 0 Comment

Editor’s Note: I’m going to be going through some old drafts and either get them published or delete them. This beauty is from a few years back when I was trying to reap at least some benefit from visiting my least favorite city, Las Vegas. Thank goodness I can now get Marriott Bonvoy points at any MGM hotel instead.

This is an impressively bad error message I ran into while trying to create an account on the MGM Resorts website. The input screen even had instructions on password format, so the framework was there to do the right thing, but they didn’t mention length, nor did they validate it in the UI.

Of course I had to dig into Developer Tools to see what was going on. A useful error message was right there. C’mon MGM, do better.

Also, a 20 character limit for passwords? That’s dumb.