The Hidden Coupling Between an AI Feature and the Team That Understands It

This one's speculative — a thought experiment about design principles, not a report on something I've built.

The system runs fine until a specific person is unavailable

An AI system can be technically robust — well-tested, well-monitored, running smoothly in production — and still have a serious fragility that never shows up on a dashboard: almost nobody besides the one or two people who built it actually understand why its prompts are worded the way they are, why a specific threshold was set where it was, or what tradeoff a particular design choice was resolving.

Prompts and thresholds do not explain themselves

Unlike a piece of traditional code, a prompt or a similarity threshold carries no visible trace of the reasoning that produced it — reading the prompt tells you what it says, not why it says that specific thing instead of an equally plausible alternative, and that missing context is exactly what a successor needs and exactly what tends to live only in one person's memory.

Write down the why, not just the what

The fix isn't more code comments — it's a deliberate practice of documenting the reasoning behind non-obvious prompt wording, threshold choices, and design tradeoffs at the time they're made, while the reasoning is still fresh, rather than trusting it to survive in someone's head indefinitely or trying to reconstruct it later from the artifact alone.

Treat this as a real risk, not a nice-to-have

A system whose correct behavior depends on tacit knowledge held by one or two specific people is carrying real organizational risk, in exactly the same category as a critical process nobody but one retiring employee knows how to run — it's just less visible, because the system itself keeps working right up until the day someone needs to change it and discovers nobody left can explain why it was built the way it was.

I'm Jesse Myers — Marine veteran, 32 years in enterprise IT, now building production AI systems. This site is where I write about what I've actually built, and occasionally about ideas I haven't built yet but think are worth taking seriously.