Run an AAR on Your Last AI Pilot: A Veteran's Playbook
By Mykel Stanley, StrategixAI
Every AI pilot teaches you something. Most mid-market companies never collect the lesson.
In the Marines, after every mission, we ran an After Action Review. Not a meeting where the senior officer talked while everyone took notes. A structured conversation where the team picked the operation apart, named what worked, named what failed, and walked out with a sharper plan for the next one. The format is older than I am, and it is the single most underused tool in business.
If you ran an AI pilot in the last 12 months, you have a debrief sitting on the table whether you held one or not. The question is whether you bothered to extract it.
What an After Action Review Actually Is
An After Action Review answers four questions, in order.
What was supposed to happen. What actually happened. What worked, and why. What did not work, and why.
That is it. No blame, no defense, no slide deck. Twenty to ninety minutes with the people who did the work and the leader who owned the outcome. Done well, you leave with three to five concrete adjustments and a short list of issues that need to escalate above the room.
Compare that to the typical mid-market AI project review. A vendor demo recap. A slide on the savings the system was supposed to deliver. A polite conversation about next phases. Nobody writes down what the operations team actually learned. The next pilot makes the same mistakes.
Why AI Pilots Need This More Than Most Projects
AI deployments are uniquely good at hiding their own failures. The model runs, the dashboard updates, the outputs look reasonable. If nobody is asking what the system is doing on the edge cases, you can be six months in before someone notices the invoices are getting coded to the wrong cost center about ten percent of the time.
A clean AAR pulls those numbers out of the work and into the open.
I ran one last quarter with a logistics client who had deployed an AI document classifier nine months earlier. Their team said it was working. The AAR surfaced three things they had not seen. The model was routing one specific carrier's paperwork to a human queue at four times the rate of the others, and nobody owned that queue. Their supervisors had built shadow spreadsheets to catch the misroutes, which meant they were spending more time on the workflow than before. And the original use case had quietly drifted, because the team kept feeding the system new document types nobody had trained for.
None of that surfaced in a status meeting. It surfaced because we sat the team down and walked the four questions.
How to Run One on Your Next AI Pilot
Schedule it within two weeks of the pilot ending or hitting a major checkpoint. The longer you wait, the more the memory blurs and the polite version of the story sets in.
Put the right people in the room. The operators who ran the workflow. The supervisor who owned the outcome. One person from IT or the vendor side who can answer technical questions without taking over the room. The senior leader who funded the work attends, but listens more than they talk.
Write the original intent on the board first. What the project was supposed to accomplish, in plain language, before anybody talks about results. Half the discoveries in an AAR come from realizing the team was solving a different problem than the leadership thought they were funding.
Walk the timeline. Then ask what worked, then what did not. Push for specifics. Not "the training was rushed," but "we had two hours of training, the team needed about six, and we saw the gap on the third week when accuracy dropped."
End with a written list of adjustments and owners. Anything that needs to escalate above the room gets named, with a date.
The Literacy Layer Behind a Useful AAR
You cannot run a useful After Action Review on a system the team does not understand.
If the operators cannot describe what the model is doing, where it is likely to fail, and what they should be checking, the conversation collapses into vibes. That is why we keep coming back to literacy. A team that has been through real AI literacy training can debrief a pilot in detail. A team that has not will tell you it worked because it did not break.
This is the same thread that runs through every veteran-led method we use. Plan it like a mission. Train as you will operate. Debrief every iteration. Then improve.
Where to Start
If your last AI pilot is closed out and you never ran a proper review, run one this month. Use the four questions. Write down what you learn. The next pilot will be cheaper and faster because of it.
If you are still inside an active project, set the AAR date now, on the calendar, with the people who need to be there. The discipline is in the scheduling, not in the meeting.
At StrategixAI, we build the AI Literacy Pipeline that makes those debriefs sharp. Veteran-led, built for mid-market operations, with the discipline most AI vendors leave out.
If you have a pilot that stalled and you want a structured way to find out why, book a consultation and we will help your team run one.