- The four dimensions to audit, and the evidence that proves each one rather than asserting it
- The questions to ask instead of "is it accurate" and "is it secure"
- Five failure patterns that show up in nearly every AI system built at speed
- How to convert findings into a sequenced build plan with effort and owners attached
- Why the team that built the system cannot be the team that grades it
An audit that ends in a ranked list of worries is a cost. An audit that ends in a build plan is the cheapest engineering decision you will make this quarter.
That distinction is missing from most best-practice lists, which are written as inspection checklists and stop at the moment of discovery. Inspection is the easy half. Sequencing the fixes, pricing them, and assigning them is the half that changes what happens to your system.
What makes an AI audit worth the money?
An AI audit is a structured review of the AI systems a company already runs, testing accuracy, security, cost, and traceability against evidence rather than intent, and ending in a prioritized plan of what to fix and what to build.
The test is whether an engineer could start on Monday with what you handed them. If a finding reads “improve model evaluation coverage,” it will sit untouched for two quarters. If it reads “build an eval set of 200 real support tickets, wire it into CI, block deploys under 92 percent, roughly four days,” it gets done.
Everything below is written to produce the second kind of finding.
What should an AI audit inspect?
Four dimensions, each with a form of evidence that settles it.
Accuracy and evaluation. Good looks like an evaluation set drawn from real traffic, versioned in the repo, run automatically on every change to a prompt, model, or retrieval index, with a threshold that blocks deploys. The question that settles it: show me the last ten evaluation runs and what changed between them. Generative systems fail in ways classical software does not, which is why NIST published a Generative AI Profile of its risk framework, NIST-AI-600-1, on July 26, 2024, to name the risks unique to these systems.
Security and access. Good looks like credentials scoped to the narrowest set of data and actions the system’s job requires, untrusted text treated as data and never as instructions, and every tool the system can call written down next to what it could damage. This surface is still moving fast: OWASP maintains a Top 10 for LLM applications and keeps reissuing it, with the current edition published on August 4, 2026. For agentic systems the access review matters more than the model review, because the agent holds keys and acts without waiting for anyone.
Cost. Good looks like token spend attributed per feature, per customer, and per workflow, with a monthly number a named person owns. The mechanics are unglamorous and the money is real. On Claude, cache reads cost a tenth of the base input token price, while a five-minute cache write costs 1.25 times base. Work that can wait runs through the Batch API at half price, and most batches finish inside an hour. A team that never separated interactive traffic from background traffic is paying full retail for both.
Governance and traceability. Good looks like per-interaction logs carrying model and prompt versions, a documented human override path, and an owner who is a person rather than a committee. Regulation has gotten specific here. EU AI Act Article 12 requires high-risk systems to technically allow automatic recording of events over the system’s lifetime, including what data was checked and who verified the result. Article 14 requires that whoever oversees the system can interpret its output, disregard or reverse it, and interrupt it through a stop button or equivalent. The Act became generally applicable on August 2, 2026. In the US, the NIST AI Risk Management Framework, released January 26, 2023, organizes the same work into four functions: govern, map, measure, and manage. It is voluntary, and enterprise procurement questionnaires are where voluntary standards get enforced anyway. If your concern is mostly this dimension, the deeper version is AI governance auditing.
Which failure patterns show up most often?
The same five, in systems built by good engineers under deadline pressure.
Evaluation happened once. Quality was measured before launch, in a spreadsheet, by someone who has since changed teams. Nothing has measured it since, so nobody knows which of the last forty prompt changes made it worse.
One credential does everything. The service account that powers the AI feature has read access to the whole database because that was faster in week one. The permission model of the app and the permission model of the AI diverged the moment the feature shipped.
Retrieval sees more than the user should. The index was built from a document store without carrying the store’s access rules, so the model can surface a paragraph the person asking would never have been allowed to open.
Nobody owns the bill. Spend is a single line item on a vendor invoice with no attribution. When it doubles, the investigation takes a week and ends in a guess.
The logs cannot answer a customer. A customer disputes what the system told them. You have request counts and latency graphs, and no way to reconstruct the conversation, the prompt version, or the model that served it.
None of these are exotic. They are the predictable residue of shipping fast, and they are the reason most AI projects stall after launch rather than during it.
How do you turn findings into a build plan?
For each finding, write four things: the failure it prevents, the fix in one sentence, the effort in days, and the person who owns it. A finding without effort attached cannot be scheduled, and a finding without an owner is a note.
Then sequence by exposure rather than by severity label. Anything that can leak data or spend money without a ceiling goes first, because those failures compound while you deliberate. Next comes anything a customer’s security review will ask for, since that work has a revenue date attached. Then quality and evaluation, which is where the durable gains live. Cost work goes last unless the number is already alarming, because caching and batching are easier to get right once traffic patterns are understood.
Sequencing this way usually produces a plan with two or three fixes in the first two weeks and the rest laid out over a quarter. That plan is the deliverable. The findings were just how you got there.
This is why DevHawk’s audit is built the way it is. Praveen Ghanta, our founder, runs every audit personally: two weeks reviewing the business, the team, and the stack, ending in a written, vendor-neutral playbook of what to fix and what to build. It costs $8,000 flat, and you never have to hire us to act on it.
Should your own team run the audit?
They should do the work, and they should not grade it. Every engineer who builds a system also builds a mental model of where it is fine, and that model is exactly what an audit is supposed to test. The person who wrote the retrieval layer is the least likely person to discover that its access rules diverged from the app’s.
Internal review still gets you most of the way. Have one team document the evidence and another team try to break the conclusions. What internal review rarely produces is the sequencing, because prioritizing across security, cost, and quality means trading off work that different people own.
If you have AI in production and cannot answer the four questions above with evidence, that is the whole signal you need. Start with the evaluation runs and the credential scope. Those two will tell you how the rest of the audit is going to go, and if you would rather someone else run it, our audit is the front door.
Frequently asked questions
What are the most important AI audit best practices?
Test evidence rather than intent, cover all four dimensions (accuracy and evaluation, security and access, cost, governance and traceability), and end with a sequenced build plan instead of a findings deck. The single highest-value practice is attaching effort estimates and owners to every finding, because that is what converts a report into scheduled work.
How often should AI systems be audited?
Treat the full audit as an annual or major-change event, and make the pieces continuous in between. Evaluation runs belong in CI on every change. Cost attribution should be reviewed monthly. Access scope should be reviewed whenever the system gains a new tool or data source, which in agentic systems is often.
What should an AI audit deliverable include?
A prioritized list of fixes, each with the failure it prevents, a one-sentence fix, an effort estimate, and an owner, plus the evidence behind each finding. If the deliverable is a risk matrix with recommended “workstreams,” you have bought a description of your problem rather than a plan for it.
Can we run an AI audit internally?
Partly. Your team can gather the evidence faster than an outsider can, but they cannot independently assess assumptions they made while building. Separate the documenting from the judging, and expect the prioritization across security, cost, and quality to be the hardest part to do internally.
How much does an AI audit cost?
Ours is $8,000 flat for a two-week review, run personally by our founder, ending in a written vendor-neutral playbook. Prices elsewhere vary widely with scope, so compare deliverables rather than day rates: ask whether you finish with a sequenced build plan or a ranked list of concerns.
- European Commission, AI Act Service Desk. "Article 12: Record-keeping." Accessed August 2026. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-12
- European Commission, AI Act Service Desk. "Article 14: Human oversight." Accessed August 2026. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
- European Commission. "AI Act | Shaping Europe's digital future." Accessed August 2026. https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
- NIST. "AI Risk Management Framework." AI RMF 1.0 released January 26, 2023; Generative AI Profile (NIST-AI-600-1) released July 26, 2024. https://www.nist.gov/itl/ai-risk-management-framework
- OWASP. "OWASP Top 10 for Large Language Model Applications." Current edition, OWASP GenAI LLM Top 10 2026, published August 4, 2026. https://owasp.org/www-project-top-10-for-large-language-model-applications/
- Anthropic. "Prompt caching." Cache read tokens priced at 0.1x base input tokens; five-minute cache writes at 1.25x. Accessed August 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Anthropic. "Batch processing." Message Batches API reduces costs by 50 percent, with most batches finishing in less than one hour. Accessed August 2026. https://platform.claude.com/docs/en/build-with-claude/batch-processing