Tag: Are Right A Lot

Memento Mori

Memento Mori

By | | Comments 0 Comment

It’s 2019, I’m in Seattle, and my new Amazon boss and I are sitting down for lunch. I hadn’t gone through an onboarding process in some time, and never one at a company that large (my prior employment tenures were 11 and 5 years, respectively). His advice was sound: “it’s going to take a year before you’re comfortable, and that’s okay.”

Onboarding is hard, no matter how long you think it should take (but please don’t call it drinking from a fire hose). Can you even imagine what it would be like to have to completely re-onboard on the regular? Sounds pretty awful, as the film Memento makes abundantly clear.

This is exactly the experience of an LLM. Every. Single. Time.

Setting aside if there’s an “experience” there in any meaningful sense (though if there is, we should pay attention to Steve Yegge), the above has a huge implication for coaxing the best results from AI. Managing context is about much more than raw quantity of and access to data. Those are reasonably solved problems with massive context windows and tool use, respectively. Now it’s about the quality, accuracy, and timeliness of the information that an LLM “pays attention” to.

If your connected SharePoint’s a mess, full of documents long stale (including all those important_v2.3_final_FINAL.docx gems), if you have a whole bunch of dead code in your git repository, and if your agentic tool has written memories from wandering conversations full of evolving ideas that now contradict themselves (as dialog is wont to do if my own experience is to be believed), an LLM is going to get confused. No different than a new hire who has to wade through similar flotsam with no institutional knowledge of what actually matters.

The thing about an onboarding human being is that eventually, through social cues and sheer determination, they will develop their own mapping layer that separates the wheat from the chaff. LLMs have no such advantage.

The machines have not made thoughtful data curation obsolete. Quite the opposite. The good news is that the machines can help with organization also, including leaving notes for their future selves (like Leonard Shelby’s tattoos). You just need to ask. Do that regularly, add a healthy dose of your own efforts, and the dividends will compound.

Sharing The Stage

Sharing The Stage

By | | Comments 0 Comment

My friend and former colleague Mark is a prolific writer. If you’re a front-end developer you definitely should follow him, but even if not, I recommend reading the story of how he came to adopt AI. He brings thoughtfulness and care to his work that’s uncommon.

The world of software engineering needs more content like this: honest, intensely personal, and decidedly human. Only Mark could have written these essays, and that’s what gives them value.

What story can only you tell?

Moo!

Moo!

By | | Comments 0 Comment

“Plan to throw one away; you will anyhow” – Fred Brooks

It’s easier said than done, but don’t get precious about your work. I’m regularly guilty of this, thinking too much like a craftsperson enamored by their creation and not like a servant of the customer and their desired outcomes. So what if the first thing I build gets thrown away? That’s often what it takes to get an idea of what actually needs to be made. The learning is well worth the price of a discarded draft.

Bring a willingness to heavily revise or even restart to your own self-reflection and embrace being told to do so by others who understand the domain better than you (this can be especially hard but it’s essential). With some practice, you may even come to enjoy the experience.

We’ve understood for a while that infrastructure should be treated like cattle, not pets. It’s now true of software in the age of coding agents, where products can be built and rebuilt with relative ease.

Knobs And Buttons

Knobs And Buttons

By | | Comments 0 Comment

There’s a lot to consider when using AI. There are a number of input variables, not the least of which is a free text prompt that can be shaped and rephrased an incredibly large (but not infinite) number of ways for similar requests. Getting to an “optimal” result is probably impossible, and even getting to a local maximum is tough. A ton of testing is required, and that’s neither quick nor cheap.

In order to target variables with high effect, it’s helpful to have heuristics and other rules-of-thumb. To that end, I found this article from Anthropic instructive.

When a result misses the mark, ask, “did Claude not know enough or did it not try hard enough?”

If the former, use a bigger model. If the latter, increase the effort setting. Simple enough!

Another quick win is to consider how much planning a model can do on its own. Per Ken Huang in Claude Fable 5 (Part 1): What Changed, and How to Stop Prompting It Like Opus:

For three model generations we compensated for planning weakness by doing the planning ourselves: numbered step lists, “think step by step,” long behavior checklists, skills files that read like flight manuals. Fable 5 plans better than those crutches, and the crutches now get in the way.

In short: let bigger models do more of the work for you. In my recent experience, Fable 5 especially (and even Sonnet 5 at times) doesn’t need nearly the step-by-step prompting to get the job done, and over-specifying both wastes your time and potentially gives worse results.

After My Own Heart

After My Own Heart

By | | Comments 0 Comment

Went back in my draft backlog and found this gem from 2020: Unit Testing is Overrated.

In the age of AI-generated code, it feels even more applicable. When a model writes unit tests (especially when it does it in view of the code it’s testing) they’re at risk of being overfitted to the functions under test. They may indeed prove software executes as it’s written, but that has little to do with proving the software meets requirements (for example, Kiro had created hundreds of perfectly passing tests for this project).

The key takeaways from the article are all worth sticking in your coding agent instructions, because without explicit directives, LLMs are probably biased to do the opposite of these recommendations given the weight of training data pushing so-called “best practices.”

In particular, I can see value in using a separate (perhaps even adversarial) agent/model to write the tests. It’ll be less biased on the context used to write the code, and it can be instructed to “aim at the highest level of integration while maintaining reasonable speed and cost.”

Not All Rainbows and Unicorns

Not All Rainbows and Unicorns

By | | Comments 0 Comment

As much as I love being a generalist and believe it’s the better end of the breadth/depth spectrum in the current tech environment, it doesn’t always feel like a great way to operate, for several reasons:

First, it’s often difficult for a generalist to describe their job, not just to folks like friends and family who are unfamiliar with their domain, but also to managers in 1-on-1s, performance reviews, and promotion documents (I’ve had firsthand experience with the latter). A term related to generalist, factotum, can even have negative connotations in certain settings, despite a “glue person” being an essential role within pretty much any organization. Specialists, on the other hand, are easier to understand, as their work is often simply “what it says on the tin” (i.e. their job title).

Second, while a generalist has proximate knowledge of many things, when they spend time with legitimate experts in a topic (which is often, because they’re so curious), they are quick to realize how much they don’t know. And since this happens so much, the experience of “I don’t know as much as that other person” compounds into a malaise of “eh, I don’t know much about anything.” Inferiority complex ensues.

When interests and aptitudes are varied, it’s also tough to determine where to focus, paradox of choice and all that. Similarly, while there’s value in doing extra at times, it’s easy to get distracted and fail to deliver the most important thing, losing the proper balance between work work and non-work work. This leads to a feeling that one is “mildly disappointing everyone all the time.” When I have that feeling, burnout lurks not far behind.

I don’t have a tidy conclusion to give here, it’s just stuff I’ve been sitting with.

On The Other Hand

On The Other Hand

By | | Comments 0 Comment

Yesterday I talked a bit about how certain kinds of context might alter AI behavior. However, this research argues that maybe context isn’t that important after all, at least with certain kinds of tasks: Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

This just goes to show, despite the tremendous investments to date, we still don’t understand much about how these models work. That’s equal parts fascinating and terrifying.

Know What You Know

Know What You Know

By | | Comments 0 Comment

For the past couple years I’ve doing work in the ecosystem of verifiable digital credentials, a space that perhaps is finally gaining some national traction given the introduction of the MATCH Act by Congressman Burgess Owens.

What are verifiable credentials? So glad you asked? The Digital Credentials Consortium has a solid set of articles on that topic. Here are several of my favorites:

My own work has been varied. For example, I’ve participated in a handful of standards working groups. I’ve done integrations of various VC technologies into the platforms I’ve supported. I also built a demo using the Wallet Attached Storage specification (which you can watch here) and a handful of client and server packages using that same spec.

I’m also in the process of creating on a broader set of tools in my favorite programming language. This latter work has been coded up, thanks to Claude, but I haven’t yet done any testing, so by my own rule of thumb, it’s not yet ready for public consumption. But perhaps soon!

Imagining Dragons

Imagining Dragons

By | | Comments 0 Comment

Editor’s Note: I wrote the first draft of this post back in December, before I’d truly discovered Claude Code. Not sure it’d play out this same way now, several months later. I really ought to get back to it and find out.

I used Amazon Kiro to build a thing that I hope to publish eventually. But in the meantime, I’ll share an anecdote from my experience with it.

The spec-driven development model makes a lot of sense to me. In a few minutes with Kiro, I thought I had a solid description of what I wanted to build. Kicked off the tasks, let things cook for a while, and after a bit, I was told things were ready to test.

Not quite sure where to begin, I asked for a full end-to-end walkthrough in the README. The model wrote a great one with detailed, step-by-step command line instructions. I was excited to try it out. Opened up my terminal, Ctrl-C Ctrl-V-ed the first command, and… error: option not supported.

Tried another one, same thing. Weird.

Did a bit more investigation and came to a shocking realization: Kiro had hallucinated the entire walkthrough.

At first I was upset, but in truth, it was okay! Because I just told Kiro to read the README in detail, and turn the walkthrough into reality by building all the stuff it had invented, and retroactively put it in the spec.

Legitimate approach? Perhaps. But next time, maybe I’ll have it build the experience first, and then the code? Work backwards from the customer, anyone?