Stop Playing the AI Slot Machine, Start Counting Cards

You ask the model for the change. It comes back close, then wrong. You type a sharper prompt. It comes back closer, then wrong in a new place. You pull the lever again. The next roll might land. That is the slot machine.

The machine does not remember why the last spin missed. You do. You are the only person in the room updating the odds. The model is waiting for another coin.

That pattern fails for a plain reason. A model does not steer itself toward a goal it cannot check. Without a feedback loop it can learn from, another prompt is another spin. Hope is not a plan.

The slot machine

You have a goal. You describe it. The reply is not the goal. You do not change the setup. You rephrase and send again. The third try is a little better. The fourth try forgets a constraint the second try had. You keep going because sometimes the next roll moves toward the goal. Sometimes it does not. You cannot tell which until you read the whole reply yourself.

That is you doing the work the loop should do.

A slot machine pays on chance. Each pull is independent. The reels do not get smarter because you lost the last one. A chat model feels different because the thread is still on the screen. The feeling misleads you. If the only signal in the thread is "try again, closer this time," the model has a mood, not a measurement. It will write a reply that sounds closer. It will not know whether the work got closer.

Models are good at the next token. They are weak at self-direction when nothing outside the prompt tells them they are wrong. A person can hold the goal and nudge. An agent left with only that nudge will wander, and it will sound confident while it does.

You can feel this on a coding task. The first patch almost compiles. The second patch compiles and breaks a test you did not mention. The third patch greens that test and deletes a check you needed. Each message was reasonable. None of them had a score. You were the score. You got tired. The agent did not. Unplug you, and the run has nothing to learn from.

Counting cards is a loop, not a vibe

Blackjack is a better picture.

You do not guess whether the next card is good. You track the cards you have already seen. High cards and low cards leave the shoe at different rates. You keep a count. The count tells you whether the rest of the deck is hot or cold. Hot means more high cards remain than the starting odds said. Cold means the opposite. You update the probability of the next high card and the next low card. Then you decide. Hit, stand, raise the bet, or walk.

The skill is attention plus a rule. You pay attention to the surrounding signal. A ten that just left the shoe changes the next decision. You change your behavior because the signal changed. You do not pull the same lever and hope. Stop counting, and you are gambling again.

Three moves, every hand:

1. See the card. Something observable just happened.
2. Update the count. Your estimate of what remains has moved.
3. Act on the new odds. Hit or stand because the count says so, not because the last hand felt close.

An agent can do the same job if you give it cards it can see. Output, a test result, a contradiction, a decision the team already made: each one is a card. If those stay in your head, you are the shoe. You deal one card per prompt and hope the next spin feels luckier.

Treat the output as a card

Agent output is signal. It is not the goal, and it is not proof. It is a card that just left the shoe. If you only read it and re-prompt, you threw the card back without changing the count.

Counting cards with an agent means you turn that output into something the next action can use. Three ways, from a light probe to a real check.

Ask it to rephrase

The cheap probe is understanding.

Tell the agent to say the task back in its own words before it edits anything. Constraints. The done-condition. The files it will not touch. The thing that would make you reject the result. If the rephrase drops the constraint, you caught a miss before the patch. If the rephrase invents a requirement you never gave, you caught a drift.

This is a card, not a verdict. A fluent rephrase can still be wrong. A model that cannot restate the job will not execute the job. You update the count: this run understood the goal, or it did not. Then write a check. Do not stop at "it sounded like it got it."

Plant a sentinel

The next card is a trap.

Put something invalid or contradictory in the context on purpose. A requirement that conflicts with another requirement. A fake identifier that must not appear in the result. A number that cannot be true if the agent read the spec. Then see whether the agent catches it.

A sentinel is a probe of attention. If the agent sails past the contradiction and writes the feature anyway, the deck is cold. It is not reading the surrounding signal. If it stops and names the conflict, you can trust it with a larger step. Keep the trap to one contradiction. A pile of tricks becomes a puzzle, and the agent will solve the puzzle instead of the job.

Turn the catch into an assertion when you can. "The result must not contain this fake id" can be a test. "I will notice if it seems confused" puts you back at the machine.

Give it a test that does not need you

The best card is an explicit test the agent can run.

The test should not depend on the same model that is doing the work. A unit test. A type check. A query that returns a row or does not. A command that exits 0. A file that must contain a string and must not contain another. Something empirical. You can read the result without asking a model what it thinks.

A judge model is the outer bound, not the default. Sometimes the check is a rubric a second model can score: did the summary mention the rejection, yes or no. Use that when you have no cheaper oracle. Skip it when a test, a query, or a diff already answers the question. A judge can be fooled by the same smooth style that fooled you. A failing test cannot be talked out of red.

Hand that check to the agent. The loop becomes act, verify, act again. The agent sees whether the last move moved the count. Green is a high card. Red is a low card. The next edit is a decision, not a wish.

That is the feedback loop models do not build for themselves. You build it. Then the agent can use it inside the run. This is not new weights. The last action missed, the error names the miss, and the next action aims at the miss. The run can tell if it is progressing. You do not narrate "warmer" after every reply.

Agentic workflows are this shape when they are real. An agent plans, calls a tool, checks what came back, and continues. A workflow with no check is a slot machine that writes a plan first. The plan can be beautiful. The reels still spin.

Why the count matters

Two things change when the agent can verify without you.

You stop being the bottleneck.

If every step needs your eyes, the largest task you can hand off is the largest task you can babysit. Independent verification stretches that. You can leave the agent alone longer because the test stops it when it drifts. A twenty-minute task can become a two-hour task. The failure shows up as a red check, not as a surprise at the end of your review. You still set the goal and the check. You do not grade every intermediate reply.

You get a stable metric.

"Closer" is a feeling. A test is a number, or at least a pass and a fail. If the metric improves, the agent is walking toward the goal. If it flatlines, or it oscillates between the same two errors, something upstream is wrong.

Two usual causes. The goal is wrong, so the check rewards the wrong thing. Or the agent lacks the information it needs to know whether it is doing the right thing. It cannot see the spec. It cannot see the failing case. It cannot see the record of what you already rejected. It spins because the shoe is face down.

When the metric does not move, change the count. Fix the check, or put the missing card on the table. The metric is how you tell those two failures apart.

What is agentic memory? names the second failure from the other side. A private chat can hold the last prompt. It does not hold the decision you made yesterday in a different host. The agent walks back to a choice you already killed, because that card never entered the shoe.

What you count, and what you ignore

A few rules keep the metaphor honest.

Count observable moves. A file changed. A test flipped. A query returned a row. A contradiction got named. If you cannot point at it, it is not a card.

Ignore tone. "Great question, here is a refined approach" is not a high card. Confidence is cheap. The check is the count.

Update before the next action. Read the result of the last verify step. Then decide the next edit. An agent that fires the next prompt before it reads the failure is pulling the lever with its eyes shut. Keep one count: one done-condition, one check, one next action.

Let red stay red. If you tell the agent the test is flaky and it should proceed, you unplugged the counter. Fix the test or narrow the goal. A plan that says "I will iterate until it looks right" is a slot machine with extra words. A plan that names the command and the assertion is a count.

The agent needs cards it did not write

A test is necessary. It is not enough if the only context is the prompt you typed from memory.

The agent has to see the surrounding signal. The issue. The decision that killed option B. The schema. The last failure. The app it is supposed to change. If those live in your head, every re-prompt is another spin you pay for with your own attention.

Decisions, records, and next steps are cards the next session can count. A decision says what you already rejected. A record is the artifact the test should touch. A next step is the action the count is for. A transcript of the last spin is a weak shoe. The next host cannot count cards it cannot see.

The context graph is how an agent walks from one of those objects to the ones beside it, including cards played yesterday in a different product.

Connectors are how those cards arrive from the apps you already use. MCP connectors put that signal on the organization instead of on one laptop. The tool gateway is the door the agent uses to read it and to act. A real tool result is a card. A guessed summary of an app the agent cannot open is another spin.

BitsWeave keeps those connections. Brokers hold the apps. Shared context holds the decisions and the records. The agent in Claude, Cursor, or ChatGPT can verify against that signal. You still write the goal. You still keep the test honest.

Leave the machine

The slot machine feels productive. The thread is long. The replies are fluent. You are still the one who knows whether any of it moved.

Give it a check that does not need you in the loop. Give it the surrounding signal so the check means something. If the metric moves, leave the agent alone longer. If the metric does not move, the goal is wrong or the agent is missing the information it needs. Fix that. Another prompt will not.

The work gets larger when the agent can tell, without you, whether it is getting closer. That is the count. Keep it.