How to Build Software with AI | Ep 8: The Smoke Alarm You Actually Press

Transcript

The full word-for-word transcript of this episode. Plain text version

Marcus: Imagine looking up at your living room ceiling and seeing a brand new smoke alarm.

Devon: Right, okay.

Marcus: It looks perfectly fitted. The little plastic casing is pristine, you know, the white disc is mounted flush exactly where it should be on the ceiling.

Devon: Yeah, and you've got the little red light blinking away.

Marcus: Exactly. You sit on your sofa feeling entirely secure, but what you cannot see from down there is that someone has quietly removed the actual sensor inside.

Devon: Oh, wow. That is a deeply unsettling image.

Marcus: It really is, and that is exactly what a green tick in software development represents when you do not actually know what ran.

Devon: A completely hollow assurance.

Marcus: Precisely. It is an alarm that looks fitted but is completely incapable of firing.

Devon: I mean, it captures the dark comedy of the massive failure at the centre of our source material for this deep dive perfectly.

Marcus: It's the exact same scenario. We are looking at a team who lived that exact reality.

Devon: For months, wasn't it?

Marcus: Yes, for months. They were greeted by these serene, comforting green ticks from their automated systems.

Devon: So every time they change a line of code, the system gives them a thumbs up.

Marcus: Yeah, and they patted themselves on the back feeling like absolute engineering titans.

Devon: Right, until someone finally stopped to look at the wiring.

Marcus: Exactly. Someone eventually asked, well, what is actually running in the background here? And they uncovered a structural failure so deep it's almost funny, in a horrifying sort of way.

Devon: The automated clean room, the system that was supposed to be meticulously checking all their code, it had no database installed.

Marcus: None at all.

Devon: And because there was no database to connect to, every single security test,

Marcus: Including the critical wrong key tests, yeah.

Devon: Right, the tests that verify whether a random person on the internet can just break into the system, all of those tests had politely and silently stepped aside for months.

Marcus: The automated system basically saw there was no database, decided it could not run those tests, and simply skipped them.

Devon: But it didn't throw an error, did it? It didn't sound an alarm.

Marcus: No, not at all. It just handed the team a lovely green tick for the handful of trivial, meaningless checks that it actually could run.

Devon: So they were trusting an alarm that was literally never wired into the mains.

Marcus: Spot on. And that specific disaster is what we are dismantling in this deep dive.

Devon: Which is so crucial for anyone building things right now.

Marcus: Exactly. Our mission is to arm you, whether you are a complete beginner building your own software with AI, or part of a small team that just never quite got round to setting a formal quality policy.

Devon: We want to give you the knowledge to pierce through that false security.

Marcus: We're going to fundamentally change your perspective. We need to turn the question from, is it green? Into, what exactly ran?

Devon: Right, because to understand how an entire security suite can just silently step aside, we first need to strip away the jargon. We need to look at what an AI-assisted test actually is.

Marcus: Yes, yes.

Devon: And I think that's a vital starting point because the word testing, well, it has a terrible branding problem.

Marcus: It really does.

Devon: It sounds clinical. You know, it sounds like this highly specialised ritual requiring decades of engineering experience sitting in a dark room writing thousands of lines of dense validation logic.

Marcus: Well, the common story is that testing is incredibly complex, but the reality is much simpler. A test is nothing more than a promise written down.

Devon: Just a promise.

Marcus: That is all it is. In this new workflow where AI does the heavy lifting, you, the beginner, act as the director.

Devon: Okay.

Marcus: You don't write the complex validation code. You write the promise in plain English. For example, your promise might just be, erm, prove a saved task survives a system restart.

Devon: And then the AI agent acts as the builder.

Marcus: Exactly. The agent writes all the underlying plumbing to make that check happen.

Devon: So you're basically taking the intention out of your head and making it concrete. It reminds me of the pre-flight checklist analogy from the sources.

Marcus: That's a great comparison.

Devon: Because pilots don't run through checklists before take-off because they're forgetful or bad at their jobs.

Marcus: Right.

Devon: They do it because human memory isn't a safety system. And neither is confidence. A checklist externalises the checking so it happens perfectly in the exact same sequence every single time.

Marcus: And tests do exactly this for your software. They take the checking out of a fallible human head and put it somewhere absolutely reliable.

Devon: I completely agree with the checklist parallel, but I do think we need to address the glaring issue with this mechanism.

Marcus: Go on.

Devon: If you're acting as the director, right, and you're just tossing plain English prompts to an AI agent, and that same agent is writing the software code and writing the test code to check it, aren't we just letting the agent mark its own homework?

Marcus: Exactly.

Devon: That feels like a total recipe for disaster. If the AI is both the student and the examiner, who is actually holding the standard?

Marcus: It's the most common objection I hear, and it makes complete sense on the surface, but it fundamentally misunderstands the timeline of how these systems are built.

Devon: How so?

Marcus: The independence is preserved because the promise, the exam, is set before the code change even exists.

Devon: Oh, I see.

Marcus: You tell the agent what the system must prove, and the agent writes the test to check for that exact outcome. The exam is written before the student sits it.

Devon: So the standard is anchored rigidly to your plain English instruction.

Marcus: Yes, rather than the agent's interpretation of its own code after the fact. That keeps the marking fiercely independent. The test exists as a fixed gate that the new code has to pass through.

Devon: Right, that makes sense. Now, if writing tests is as simple as writing an English promise, the natural assumption is that AI makes testing a breeze.

Marcus: Well, yeah. Having an AI write your code surely means there is less testing to worry about because the machine doesn't make silly human typos.

Devon: The common story is that it means less testing, but that assumption hides a massive trap.

Marcus: Wait, I really would assume the machine's capability reduces the need for testing. If an AI writes perfect syntax, why would tests matter more?

Devon: Because of a concept called regression.

Marcus: Okay, let's unpack that.

Devon: For anyone new to the term, a regression is simply something in your software that used to work perfectly, which then gets broken by a later change.

Marcus: Because software is deeply interconnected.

Devon: Exactly. It isn't a stack of independent bricks. It's like a house where all the plumbing and electrical wiring are tangled behind the walls. I'll give you an embarrassing personal example. I once asked an AI agent to fix a really minor typo on a user profile screen.

Marcus: Just a typo.

Devon: Just a typo. It fixed the typo brilliantly, but somehow, in the process, it completely unlinked the database connection for the main billing system three rooms away.

Marcus: Oh, brilliant. So you touch a light switch in the kitchen and the shower upstairs stops working.

Devon: That is exactly it.

Marcus: And this is where the cultural and human reality really collides with the technology. AI agents build software at a superhuman speed, and they do it with supreme confidence.

Devon: They really do.

Marcus: You know, a human programmer types slowly. They think about the architecture. They might hesitate and review the dependencies before altering a core file.

Devon: But an AI agent doesn't hesitate.

Marcus: No, it doesn't. It will rewrite 50 lines of complex interconnected logic in a fraction of a second, which means agents cause regressions vastly faster than humans ever could.

Devon: They move so fast that you physically cannot manually check every corner of the house after every single change. You would spend your entire life clicking buttons to make sure nothing broke.

Marcus: And that introduces the real trap. The trap isn't a lack of tests, it's comfort.

Devon: Yes, comfort.

Marcus: A team gets lulled into a false sense of security by the automated systems. You see the agent working quickly, you see the system giving you a green tick, and you just assume the house is perfectly intact.

Devon: You become comfortable.

Marcus: And comfort, when you are building at superhuman speed, is incredibly dangerous.

Devon: Yeah.

Marcus: You go and pour yourself a cup of tea while your entire user database is just left wide open.

Devon: Exactly. So to maintain this blinding speed without shattering the software into a thousand pieces, developers rely on an automated robot to check all those plain English promises we talked about earlier.

Marcus: Right, this is the robot.

Devon: But this robot, this automated checker, is exactly where the trap springs shut. The technical term for this robot is continuous integration, or CI, but let's just keep it concrete and call it the tireless robot.

Marcus: Let's look very closely at how this robot operates because its literal-mindedness is the core issue here.

Devon: Yes. Here's what it does. On every single change you or your AI agent makes, this robot takes a fresh copy of your code, it places it on a completely clean machine, it provisions a blank server, downloads all the dependencies, and rebuilds your entire software from absolute scratch.

Marcus: Right.

Devon: Then it presses every single alarm button, meaning every test you have written, and reports back with either a green tick or a red cross.

Marcus: And there are sacred rules for this, aren't there?

Devon: Two sacred rules when you employ this robot. Rule one, nothing merges into your main project on a red cross. The red cross is an absolute gate.

Marcus: Right. And rule two?

Devon: Rule two, the main copy of your code stays green, always. This completely eliminates the age-old developer excuse of, well, it works on my computer.

Marcus: Because if it only works on your computer, the robot's clean machine will throw a red cross and the code just doesn't merge.

Devon: Exactly.

Marcus: I mean, that sounds incredibly robust on paper.

Devon: It does.

Marcus: But let's bring this right back to the horror story from our opening hook. The robot does exactly what it's told to do, but it has absolutely no common sense.

Devon: None.

Marcus: It takes the fresh copy, it builds the machine, it checks the tests. But skipped tests don't turn anything red.

Devon: No, they don't.

Marcus: If the robot goes to press the smoke alarm, notices there is no database to run the security check against, it simply notes in some buried log file that it couldn't run the test and moves on. It politely steps aside.

Devon: And crucially, it still gives the green tick at the end.

Marcus: Yes. The tests quietly just don't happen. The green tick stays perfectly green.

Devon: Which means the only thing that tick is actually telling you is that the handful of trivial checks that could run kept passing. Everything else was bypassed. Which brings us to the really critical question here.

Marcus: Hmm.

Devon: Who actually pays for this?

Marcus: Right.

Devon: When AI agents are building software at this blinding speed and the alarms look fitted but never fire, who suffers?

Marcus: It's the beginner.

Devon: It is.

Marcus: It's the solo founder or the small team director who ultimately pays the price of that false security. They ship broken or entirely unsecured software while genuinely believing they are protected by industry standard systems.

Devon: And this tension between the incredible speed of the AI and the silent failure of the robot leads directly to a massive clash over whether this entire automated ecosystem is actually safe for someone who doesn't code.

Marcus: I'll be completely honest with you. For a solo beginner just trying to ship something, I find this system quietly terrifying.

Devon: Terrifying?

Marcus: Yes, terrifying. We are taking people who don't have a deep background in software architecture and handing them a profoundly false sense of safety.

Devon: I see where you're coming from, but,

Marcus: A comforting green tick that means far less than they think it means is a massive danger. It masks structural rot. If the system tells a non-coder that everything is fine when actually all the security tests were skipped, I genuinely think that's worse than having no tests at all.

Devon: Really? Worse than no tests?

Marcus: Yeah, because with no tests, you at least know you are vulnerable. You tread carefully. With a fake green tick, you're utterly blind and running at full speed.

Devon: I understand why it looks terrifying from the outside, I really do. But I have to disagree completely that it is a danger.

Marcus: Really? Empowering to be lied to by a green tick? I'm not sure how masking structural rot empowers anyone.

Devon: It's empowering because of how AI agent tools respond to failure, provided you establish the right workflow. The beginner only has to write the promises in English.

Marcus: Right.

Devon: The AI does the heavy lifting of writing the code and the plumbing. But more importantly, major agent tools thrive on a test-first workflow.

Marcus: Okay, go on.

Devon: A failing test, a red cross, is actually the clearest possible brief an AI agent can receive. It leaves zero ambiguity.

Marcus: But how does that actually play out in practice? If the agent is just chasing a red cross, doesn't it just get stuck in a loop?

Devon: Not at all. Take tools like Claude Code or Codex. If you're using Claude Code, it will run your test suite, read the failures, and automatically iterate to fix them.

Marcus: Automatically?

Devon: Yes. You can literally add a single line to a configuration file called CLAUDE.md that says, no work is finished until every test passes.

Marcus: Wow, just that one line.

Devon: That one plain English sentence makes the agent treat a failing test as an unfinished job. It will tirelessly read the error, adjust the code, rerun the robot, and repeat until the red cross turns green.

Marcus: So it does the grunt work for you.

Devon: Exactly. Codex operates exactly the same way inside its sandbox, treating a red cross as an incomplete brief. The system is designed to tirelessly chase the green tick for you.

Marcus: Okay, I will concede that the mechanism of an AI tirelessly chasing a fix is brilliant. It takes all the manual drudgery out of debugging.

Devon: It really does.

Marcus: But, and this is a big but, that entire cycle relies on the red cross actually appearing. If the tests skip because the database is missing, the AI sees a green tick too and it stops working.

Devon: Ah, yes.

Marcus: So it assumes the job is done perfectly. So we actually converge on a shared truth here.

Devon: Which is?

Marcus: The system is incredibly empowering, but only if the beginner learns the exact habit required to pierce through the false safety of that green tick.

Devon: Yes. If they blindly trust the interface, it's a terrifying liability.

Marcus: But if they know how to interrogate it, it is the ultimate leverage.

Devon: That is the perfect convergence. The danger isn't the machine, the danger is the assumption.

Marcus: Exactly.

Devon: The danger is looking at the plastic casing of the smoke alarm and just assuming the battery is inside.

Marcus: So how do we practically protect against this?

Devon: Well, not with complex code or years of engineering study. We protect against it with two simple rules and one vital question.

Marcus: Let's lay out the concrete takeaways then, because this is about establishing a learnable habit, not preparing for a disaster. Anti-hype, actionable steps.

Devon: First, the director's question. You must completely banish the phrase, is it green?

Marcus: Mm-hmm.

Devon: From your vocabulary. You never ask your agent that.

Marcus: Never.

Devon: Instead, you must ask, what exactly ran?

Marcus: Right. That simple semantic shift changes the entire dynamic of how you interact with the system.

Devon: It changes everything. You ask the agent, what exactly ran? How many tests were executed? How many were skipped? Which specific tests were skipped and why?

Marcus: And you just type that into the prompt.

Devon: In plain English, yeah. And the agent will comb through the logs and tell you if the security suite quietly stepped aside because a database was missing.

Marcus: Oh, brilliant.

Devon: Second, you document two rules for your project permanently in that configuration file you mentioned earlier. Line one, no work is finished until every test passes.

Marcus: Right. And line two?

Devon: Line two, skip tests must be reported loudly, never silently.

Marcus: That is crucial.

Devon: By explicitly stating that, you force the AI to flag a skipped test as a critical issue that demands your attention, rather than just hiding it behind a reassuring green tick. Which brings us to the final piece of protection, the guard rail.

Marcus: Right, the guard rail.

Devon: We mentioned earlier that tests check things by performing actions. Tests do destructive things on purpose.

Marcus: They're essentially crash test dummies.

Devon: Spot on. They create rubbish records, they try wrong passwords, they delete folders, they fill contact forms with complete nonsense, erm, like trying to register a user named drop table just to see if the system will accidentally delete its own database.

Marcus: Just aggressively trying to break things.

Devon: Exactly. That is how they verify the software handles bad inputs correctly. Because of this, they must never ever run against your real cabinet, your actual live user database.

Marcus: Because if a crash test dummy drives a car into a wall in a test facility, it's a success.

Devon: Yes.

Marcus: But if it drives your actual family car into a wall, it's a catastrophe.

Devon: That's exactly the right framing. So a vital practice is having the test suite check its own address before it does anything destructive.

Marcus: Okay, how does that work?

Devon: You instruct the AI to write a strict rule that the test suite must flatly refuse to run if it discovers it is pointed at the real live data.

Marcus: So it checks where it is first.

Devon: Yes. It must strictly only run in the disposable practice room that the tireless robot builds. That one guard rail ensures that your aggressive destructive tests never accidentally wipe out your actual business data while trying to prove the system works.

Marcus: If you combine that guard rail with the director's question, you strip away the false comfort and replace it with actual oversight.

Devon: You really do.

Marcus: Which leads me to a final thought I want to leave you to mull over. We're operating in an era where AI can build out whole, incredibly complex software systems in seconds. It is a level of leverage that is historically unprecedented for individuals and small teams.

Devon: Absolutely.

Marcus: But as we marvel at this incredible speed, we have to realise where our actual weakness lies. Our greatest vulnerability isn't the machine making a mistake. The machine will make mistakes. That is a given.

Devon: Of course it will.

Marcus: Our greatest vulnerability is our own human eagerness to trust a reassuring green tick without looking at the wiring. We want the comfort. We want to believe the house is built perfectly so we can move on to the next task and pour that cup of tea.

Devon: We really do.

Marcus: But before you let an automated builder construct your foundation at light speed, you have to ask yourself, have you actually audited your own smoke alarms?