So this happened to me recently and I honestly hadn't thought much about it before.
We had an agent that had been running fine for a few weeks.
Then we changed one part of the prompt.
Nothing that seemed particularly risky, tested it quickly and pushed the change.
A few hours later, some of the answers were just... wrong.
Turned out the change was affecting how it was using one of its tools. Took a little while to figure that out.
What really annoyed me was that I couldn't easily get back to the old version.
I'd overwritten the prompt. There wasn't a proper history for it, so I ended up going through old messages and trying to work out what we'd had before.
The actual fix was pretty quick once we found the problem. Getting there was the painful bit.
And it got me thinking about how weird this is compared with normal software.
If I change code, there's a commit.
I can see exactly what changed and revert it if I need to.
But with agents, I've noticed I tend to think of the prompt, tool settings, memory/config etc. as configuration rather than something that needs the same treatment.
Maybe that's the wrong way to look at it.
If changing any of those things can completely change how an agent behaves, shouldn't the whole thing have a version history?
I've been looking into this a bit since.
GitAgent was one thing I came across. The idea of keeping the agent definition in a version-controlled setup makes a lot of sense once you start thinking about agents as actual applications rather than just prompts. Read it about it through Lyzr.
I also came across Portkey while looking into the broader agent tooling space. It's more focused on the gateway and observability side, which is another piece of the puzzle when you're trying to understand what happened during a run.
Still not sure what the "right" setup is, especially for smaller projects.
For people actually running agents in production, what do you guys do?
Do you version the prompt? The tools? The whole agent?
Or honestly just keep backups and hope for the best? 😅