Architecting in the Age of AI: StopArguing, Start Testing
Every system starts in the same room. Someone draws boxes on a whiteboard, someone else crosses out an arrow and draws another, and the design goes to a review to be argued over. Eventually one view wins, often the one held by the person with the most experience: the one who has built three systems like this and can name the last project where the alternative failed.
That is a reasonable way to decide. Experience is the best predictor we have had. Good architects run a spike when they can afford one, but most decisions never get one. So most of a design is built on a prediction, and nobody finds out whether it works until months later.
A prediction also comes with a way out. When the system struggles a year later, there is always an explanation that spares the design: the requirements changed, the team didn’t build it as drawn, nobody could have foreseen that load. The architect is rarely wrong, because the architecture was never in a position to be.
What changed
Testing an idea used to mean building it, which took weeks, so reasoning was all anyone could afford and the best-argued design won. That cost has collapsed. An AI agent can now build a working version of a design, stress it with generated data and report back within an hour. A spike that once needed a week’s justification now costs less than the meeting about it, so every decision that can be tested should be.
Meanwhile, more and more code is written by AI agents, and an agent doesn’t push back on a weak design the way a team does. It builds what it is given and fills the gaps with its own guesses, quickly and everywhere, so a flaw in the architecture ends up copied into all of the code. Testing is getting cheaper just as untested decisions are getting more expensive.
Working a real problem this way
I recently designed the core of a context layer: one place that holds what a company knows about itself, its projects, people, teams and rules, and how they all relate. People and AI agents ask it questions before they act. It had to be right, fast, and never show anyone something they weren’t cleared to see.
I knew how I would have designed it. Instead, an AI agent researched the common patterns and where they break, I drafted an architecture from that and my own experience, and then I tested it one decision at a time: fifteen experiments in two days.

The biggest decision was where to put the language model. The obvious design, and the one I would have taken to review, has a model read the records and write the answer. The alternative uses a model only to work out what is being asked, then fills in fixed sentences from ordinary queries. I expected the model-written answers to be accurate enough to justify being slower.
The agent built both, gave the model-written one a check that rejected any answer naming something it couldn’t find, and ran them against a deliberately messy generated company. The narrow version was right 98% of the time against 88% (on 958 and 150 questions), and answered in 0.28 seconds instead of 1.3. It could still misread a question, but it couldn’t invent a fact: no model writes its sentences.
To make sure I wasn’t grading my own homework, another model wrote 644 fresh questions that I kept sealed until the end. The narrow version still scored 98%, and an independent model that checked 400 of its answers found none that contradicted the records.
The model-written design is kept only as a fallback. The most useful thing the AI did was show me where not to use AI. In a review, that would have sounded like a lack of ambition.
Most of my smaller decisions held. The ones that failed were among the biggest, and nearly half of what I would have built was gone. A few of the others:
My instinct | What the test showed | Outcome |
The database will be the bottleneck | It handled 594 reads a second. The real limit was the model quota, under nine questions a second | Effort moved |
Hand the model the whole document in one go | All three models ran out of room on a 59,000-character standard. Reading it in chunks, then a “what did you miss?” pass and a pass per section, found all 41 rules | Replaced |
Force the model’s answers into a strict format | Formatting errors vanished, but it started missing rules: in 2 of 4 runs it found only 88–90% of them | Not used |
A better model will catch bad instructions | Hidden instructions were ignored in 6 of 6 runs, but a fake “approved by the CISO” update was recorded as a rule 3 of 3 times. Only tracking who supplied the document stopped it | Fixed outside the model |
Use the textbook guarded entry point for writes | A known database trap, a temporary table posing as a real one, let an agent retire a rule it had no right to touch | Fixed before it shipped |
Enforce access in the database | Probes against 146 hidden projects returned nothing, not even a difference in timing | Kept |
What was cut wasn’t argued away. It was measured away. And I couldn’t have told you in advance which instincts would fail: the ones that did felt exactly as solid as the ones that held. My confidence felt like evidence. It wasn’t.
What testing can’t settle
This has limits, though fewer than I assumed. An hour of testing settles how fast, accurate and safe a design is. With a little more effort it gets close on things we used to leave to judgment: have the agent make the same change to two designs and see which one fights back, or run the load test and price the bill. What it can’t settle is how a design fits the people who will run it, or which future you are building for. And however messy you make generated data, real data is messier.
That is where the architect’s job now sits: deciding which questions are worth asking, designing the conditions that would break the design, knowing when the evidence is strong enough to commit, and making the trade-offs no number can decide.
So next time a review stalls on an argument, don’t settle it in the room. Ask what result would end it, and go and get it before the next meeting.
The room doesn’t go away. Someone will still draw boxes on a whiteboard, and experience will still decide what gets drawn. It just won’t decide what stays. The boxes leave the room as bets and come back as results, and the architect has to be willing to be wrong early, and on the record. “I believe this will work” used to be the best anyone could offer. It is now the weakest claim in the room.



