The Same Prompt Will Produce Different Output Next Year: Designing for That

This one's speculative — a thought experiment about design principles, not a report on something I've built.

Reproducibility was never actually promised

Traditional software has an implicit promise: run the same code against the same input, get the same output, indefinitely. AI-generated content never had that promise in the first place, even though it's easy to build systems as if it did — the same prompt against a future model version, or even the same version with a routine backend update, can produce meaningfully different output with no warning that anything changed.

This matters most for content you plan to regenerate

A one-time piece of generated content that gets stored and never regenerated is fine — it is what it is, permanently. The problem shows up specifically for anything designed to be regenerated later from the same prompt: a template, a recurring report, a piece of content meant to refresh on a schedule, where an assumption of consistency across regenerations quietly stops holding the moment the underlying model changes.

Version and snapshot anything that needs to stay stable

If a specific piece of generated output needs to remain exactly as it was — a legal document, an approved piece of customer-facing copy — the system needs to store the actual generated result as a stable artifact, not just the prompt that could theoretically regenerate it later. The prompt is a recipe. The recipe producing a different dish next year isn't a bug in the recipe; it's just what recipes do when the underlying ingredients change without telling you.

Decide explicitly which category a piece of content is in

Every piece of AI-generated content in a system falls into one of two categories — meant to be regenerated fresh each time, or meant to be a fixed, stored artifact — and the design has to make that distinction explicit and deliberate, rather than defaulting to "generate it fresh" everywhere and discovering the difference only once something that should have been stable wasn't.

I'm Jesse Myers — Marine veteran, 32 years in enterprise IT, now building production AI systems. This site is where I write about what I've actually built, and occasionally about ideas I haven't built yet but think are worth taking seriously.