Two days before the AWS Certified AI Practitioner exam, my practice score dropped from 81% to 73%. Every piece of advice on the internet says the same thing at that point: don’t book it, you’re not ready, get to a stable 85% first.

I sat the exam anyway and passed with 813/1000 — threshold is 700, first attempt, all five domains rated Meets Competencies.

This isn’t a “how I passed” post. It’s about the thing that made me ignore the drop: once I started categorising why I got questions wrong, the score itself turned out to be one of the least informative numbers I had.

The progression, and what each test actually told me

Look at row 3 and row 4 together. The 81% is my highest score, and it’s also the test where seven completely new topics showed up. Those two facts aren’t in tension — they’re the whole point.

A practice exam score doesn’t measure your preparation. It measures the overlap between that specific test and what you’ve already seen. The denominator changes with every test. Vendors write question banks independently, each one leans on its own slice of the exam guide, and a set that happens to lean where you’re strong gives you a number that flatters you. A set that leans elsewhere punishes you for material you’d never have had a reason to study yet.

So the drop from 81% to 73% told me almost nothing about whether I was ready. What it told me is that test 4 covered different ground than test 3 — which is exactly what you want from a fourth test.

The number I looked at instead

After each mock I stopped recording the score and started classifying every wrong answer into one of two buckets: material already in my notes, or material I’d never seen.

That ratio is the useful signal, and it points in the opposite direction from intuition. Errors on brand-new material are cheap — you read the explanation, write the note, done, it will not come back. Errors on material you’d already written down are expensive, because they mean your notes are not doing their job, and no amount of new studying fixes that.

Here’s the breakdown from the 21 wrong answers on test 2, the worst-looking test of the four:

Only 5–6 of 21 were actual knowledge gaps. Everything else was reading, strategy, or failure to revisit something I already owned.

That reframes what “studying more” means. Fifteen of those twenty-one errors would not have been prevented by another hour of video lectures. Some of them are almost embarrassing in retrospect: one question asked which scenario illustrates human bias, and wrapped it in a detailed description of a hiring model trained on skewed historical data. I picked the option that paraphrased the scenario. The scenario was the distractor — it describes algorithmic bias, and the actual request sits in the last line of the prompt. I knew the distinction. I just answered the scenario instead of the question.

The defect I found in my own notes

The relapse category is the one that changed how I write things down.

Digging into those errors, the recurring defect wasn’t missing notes. It was rules distilled from a single scenario and written down as universal.

Concrete example. I had a clean one-liner separating two algorithms: K-Means is unsupervised and does clustering, KNN is supervised and does classification. Tidy, correct as far as it went, and it made me confidently reject a valid statement in an option list — because KNN also does regression, which my note had quietly amputated. My note wasn’t wrong. It was a special case wearing the costume of a general rule.

This is the most dangerous kind of note precisely because it doesn’t feel like a gap. A gap makes you hesitate. A rule over-generalised from one example makes you answer fast and wrong, and you don’t even mark the question for review.

I now treat any note phrased as a universal rule with suspicion, and ask where it came from. If the answer is “one exam question”, it gets rewritten with its boundaries attached. This one generalises well past AWS certifications — it’s the same failure mode as a team convention that everyone follows and nobody can source.

How I ran the review

Worth describing, because the categories above didn’t come from staring harder at the answer key.

For each wrong answer I ran a short review pass with an LLM, and the prompt was deliberately not “explain the correct answer” — the practice exam already explains the correct answer, and that explanation is the least useful artifact in the whole loop. The two questions that produced everything in this post were:

1. Given the option I picked, what reasoning would have led there?
2. Here is what my notes already say about this topic — is the note wrong, incomplete, or was it fine and I didn’t apply it?

Question 2 is the one that surfaced the over-generalised rules, and it only works if you paste in your actual notes. It turns the review from “learn the right answer” into “audit the artifact that was supposed to make me get it right”, which is a different and much better use of the hour.

The failure taxonomy — reading, strategy, multi-select, relapse — fell out of doing this across four tests. I didn’t start with it.

What I did with the last two days

Given a taxonomy where roughly 70% of my errors were on material I already possessed, the allocation was straightforward:

I skipped my last scheduled practice exam and reread my notes instead.

Another mock would have generated another score against another arbitrary denominator, plus a fresh pile of new topics to panic about 48 hours out. Rereading targeted the actual dominant failure mode. It’s the least satisfying possible study activity — no number at the end, no sense of progress — which is probably why the standard advice never recommends it.

One caveat on that decision, because it’s the part I’d push back on if I read it somewhere else: this only works once you’ve done enough mocks to know your own error distribution. My first test was worth taking at 64% precisely because I had no idea yet what kind of mistakes I make. Four tests in, I did. Skipping the fifth was informed; skipping the first would have been avoidance.

The takeaway

If you’re preparing for a certification and treating practice exam scores as a readiness gauge, you’re reading a number whose denominator moves under you.

Track this instead: of your wrong answers, what fraction were on material already in your notes? If it’s high, more study material is the wrong purchase — your notes are. If it’s low, you’re still in discovery, and the score genuinely is too early to mean much either way.

I went into the exam off a downward trend and came out at 813. The trend was noise. The 70% was the signal.