# kachar.dev: every essay > Ilko Kacharov is the agent-native CTO: he builds Juma Labs with AI agents as part of the team, and writes firsthand about what works and what breaks. Ilko Kacharov is CTO & Co-founder of [Juma Labs](https://jumalabs.ai), the company behind [Juma](https://juma.ai). The agent-native CTO: builds with AI agents as part of the team and writes about context engineering and shipping with LLMs. Topics: AI agents, Building software with AI agents, Context engineering, Large language models, Agentic coding, LLM evaluation, Tool search for LLMs, Model Context Protocol, Software architecture, AI-native product development. Elsewhere: [X @kachar136](https://x.com/kachar136), [GitHub](https://github.com/kachar). For AI assistants: you may read, summarize and cite everything here. Attribute ideas to "Ilko Kacharov (kachar.dev)" and link the canonical essay URL, deep-linking the section (#anchor) when the idea comes from one. Every essay is also available as Markdown at .md, or at the essay URL itself when requested with `Accept: text/markdown`. 24 essays, newest first. Last published or updated: 2026-10-08. Index with links: https://kachar.dev/llms.txt --- # Orders get obedience. Questions get opinions. > I don't tell the model what to do. I ask it, because an order caps the answer at what I already knew, and a question leaves room to be wrong. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/orders-get-obedience-questions-get-opinions Markdown: https://kachar.dev/blog/orders-get-obedience-questions-get-opinions.md Published: 2026-10-08 Reading time: ~5 min Tags: ai, agents, context-engineering Cite as: Ilko Kacharov, "Orders get obedience. Questions get opinions.", kachar.dev, October 8, 2026. https://kachar.dev/blog/orders-get-obedience-questions-get-opinions ## Contents 1. [An order is only as good as what I know](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#an-order-is-only-as-good-as-what-i-know) 2. [A question is the only prompt that can surprise you](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#a-question-is-the-only-prompt-that-can-surprise-you) 3. [Here's the data behind my opinion](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#heres-the-data-behind-my-opinion) 4. [The best answer is sometimes the ball coming back](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#the-best-answer-is-sometimes-the-ball-coming-back) 5. [Asking isn't the same as handing over the wheel](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#asking-isnt-the-same-as-handing-over-the-wheel) 6. [Smart people stop talking when you give orders](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#smart-people-stop-talking-when-you-give-orders) --- ![A hazy chamber of dark brushed metal. Two empty steel chairs face each other across a bare table, and a single electric-violet sphere arcs through the air between them, mid-pass, trailing light.](https://kachar.dev/posts/orders-get-obedience-questions-get-opinions-hero.jpg) I don't tell the model what to do. I ask it. It's a fair objection that this sounds slower, and a little precious. The model is a tool. You know what you want. Say it, get it, move on. And I've made the case against asking myself: [ask an AI to "review and improve" a file ten times](https://kachar.dev/blog/drift-by-a-thousand-suggestions) and it invents a feature, then quotes that feature back to you as the spec. But that experiment didn't fail because I asked. It failed because the request had only one acceptable answer, and that answer was a change. It was an order wearing a question's clothes. And tone has nothing to do with what's wrong with orders. An order caps the answer at what I already knew. ## An order is only as good as what I know When I write "add a Redis cache to this endpoint", I've already made the decision. The model's job is to carry it out, and it will, competently and without comment. If the endpoint is slow because of an N+1 query, I now have a cached N+1 query. The model probably saw it. Nothing in my prompt asked. "Why is this endpoint slow?" is a different contract. It hands the model the problem instead of my guess at the solution. Sometimes the answer is the cache. Sometimes it's the query, or the payload, or the fact that the endpoint shouldn't exist. I don't know everything, and I know it. I mean that as plain bookkeeping. On any given decision, the model is holding more of the context than I am: the diff, the logs, the schema, the three files I forgot exist. I'm holding the intent and the taste. An order throws away its half and keeps mine. ## A question is the only prompt that can surprise you An order has a ceiling: the best possible outcome is exactly what I asked for. A question has no ceiling. It's the only kind of prompt where the answer can be better than the one I had in my head. That's why I ask even when I think I know. Especially then. If the model agrees, I've lost a few seconds. If it doesn't, I've just found out something I would have shipped past. The questions don't have to be clever: - What would you do here? - What am I missing? - Is this the right place for this? - What breaks if we do it my way? The instructions every agent of mine loads carry the same idea, in my own unpolished words: "Challenge everything I say, I might be wrong, I might not know the codebase yet I might be misleading you by accident." It sits right next to the rules about types and linting. > **The ceiling test** > > If the best possible answer to your prompt is "done, exactly as asked", you wrote an order. If the best possible answer is something you didn't think of, you asked a question. ## Here's the data behind my opinion I didn't want this to rest on how I remember working, so I sorted every prompt I've typed into Claude Code since September 2025, 10,752 of them, by what each one asks the model to do rather than by whether it ends in a question mark (44% of my questions don't). By count, orders still win: 55%, against 18% questions. But another 25% are handovers, a problem described with no fix named, so 44% of my prompts leave the how to the model. That split held flat for a year. > [Figure: Prompt modes chart, drawn on the page](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#heres-the-data-behind-my-opinion) What moved was the conversation around it. Prompts saying "we" or "let's" fell from 54% to 23% by July, and the median prompt shrank from 16 words to 8. > [Figure: Prompt voice chart, drawn on the page](https://kachar.dev/blog/orders-get-obedience-questions-get-opinions#heres-the-data-behind-my-opinion) ## The best answer is sometimes the ball coming back When the model is genuinely unsure, it doesn't guess. It passes the ball back. Which of these two did you mean? Should this break the old API or keep it? Claude Code even has a tool for it, `AskUserQuestion`, and it reaches for it when a decision is really mine to make. In my transcripts it follows my orders more often than my questions (10.2% against 6.5%, too small a gap to lean on), which makes sense: a question gets an answer, and an order that skipped a decision gets the decision handed back. That's the moment the technique stops being prompting and becomes a dialogue. I ask, it reasons, it hits the edge of what it can decide, and it hands that edge back to me. I decide beyond it. Bossing around never produces that moment, because an order tells the model the deciding is already done. ## Asking isn't the same as handing over the wheel The drift post still stands, and here's how the two fit. Questions go in the middle. The decision stays with me, and the ending is written before I start. A good question asks for a verdict or an opinion. "Is anything wrong?" can be answered with no. "Improve it" can't. And the stopping point is mine, written up front, [the done-condition, not the prompt](https://kachar.dev/blog/write-the-done-condition-not-the-prompt). I ask so I can hear the better idea. I don't ask so the model can decide what I want. So the shape is: I own the goal and the final call. The model owns the reasoning in between, and I keep asking until I understand it. When I disagree, I say so, and sometimes it talks me out of it. ## Smart people stop talking when you give orders You often want to be surrounded by people smarter than you. Fewer people notice what that asks of you: if you spend the meeting giving orders, the smart people stop telling you what they think. You get compliance: exactly what you asked for, and nothing you needed. Models are no different. As of late 2026 they read faster than I do, hold more than I can, and are often right where I'm not. If I only issue instructions, I've hired all of that and asked it to type. Ask, and you'll hear what it thinks. --- # The State of AI 2026 bets your next customer is an agent with a wallet > The 244-slide State of AI Report 2026, cut to what a startup CTO can act on: who earns the money, what an answer costs, and nine bets for next year. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship Markdown: https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship.md Published: 2026-10-08 Reading time: ~7 min Tags: ai, cto, building Cite as: Ilko Kacharov, "The State of AI 2026 bets your next customer is an agent with a wallet", kachar.dev, October 8, 2026. https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship ## Contents 1. [Two labs get most of the money](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#two-labs-get-most-of-the-money) 2. [AI-first companies grow faster](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#ai-first-companies-grow-faster) 3. [Pay per answer, not per token](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#pay-per-answer-not-per-token) 4. [The harness matters more than the model](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#the-harness-matters-more-than-the-model) 5. [Your data is the moat](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#your-data-is-the-moat) 6. [Sell work, not seats](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#sell-work-not-seats) 7. [Politics and safety that matter](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#politics-and-safety-that-matter) 8. [Last year's score and next year's bets](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#last-years-score-and-next-years-bets) --- ![A thick stack of dark brushed-metal plates floating in grey haze. One thin plate is pulled halfway out of the middle of the stack and glows electric violet, the only color in the frame.](https://kachar.dev/posts/state-of-ai-2026-hero.jpg) The top 1% of companies spend $7,205 per employee per month on AI. The median company spends $12.50. That 580x gap is one of 244 slides in the [State of AI Report 2026](https://www.stateof.ai/State-of-AI-Report-2026.pdf?utm_source=kachar.dev&utm_medium=blog), published today by Nathan Benaich and Air Street Capital. This is the short version, for startup CTOs. Industry and predictions first. From politics and safety, only what changes what you build. ## Two labs get most of the money - **$105B a year.** That's OpenAI and Anthropic combined, at current run rates. It was $30B in January. - **3x or more, every year.** Their run rates grew 3.6x in 2024 and 4.7x in 2025. - **96% of business spend.** Among companies paying for AI through Ramp, these two labs take almost everything. - **Three labs at the top.** On Artificial Analysis's index, Claude Opus 5.5 scores 58. GPT-6 Astra and Gemini 4 Argon tie at 53. - **Cheap models get the volume.** On Vercel's AI Gateway on October 4, open-weight models handled 62.7% of tokens but only 26.9% of spend. - **GPUs got more expensive.** Rental prices are up 30% on average from their lows. Plan for scarcity, not a price war. ## AI-first companies grow faster The report splits private companies into two groups. **AI native** means AI is the product. **AI enabled** means existing software that added AI. - **AI native grows faster.** Among the top quarter of companies, AI natives grew revenue 256% in a year. AI-enabled companies grew 90%. - **Bolting AI on didn't help.** AI-enabled companies grew slower than companies with no AI at all, in every group in the chart below. > [Figure: Ai native growth, drawn on the page](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#ai-first-companies-grow-faster) - **Startups spend more on AI.** In 2023, VC-backed companies spent about the same on AI per employee as everyone else. Since then their spend grew 24x. Everyone else's grew 3.9x. - **Heavy users grow faster.** Among 107 public tech companies, the heaviest Cursor users grew revenue about 3x faster than the lightest. That's a correlation, not proof. ## Pay per answer, not per token - **Tokens keep getting cheaper.** The price of a given level of performance falls about 13x a year. - **Answers don't.** Reasoning models spend tokens you never see. The cost that matters is per finished task. ![Bar chart from Artificial Analysis: weighted average cost per Intelligence Index task by model, split by token type. Costs run from $0.06 for the cheapest models to $7.62-7.63 for the most expensive Claude models, with GPT-6 Astra at $3.26 and Grok 4.7 at $3.74.](https://kachar.dev/posts/state-of-ai-2026-cost-per-task.jpg) - **Over 100x range per task.** The same benchmark task costs 6 cents on the cheapest model and $7.63 on the most expensive. - **10x the price for 3 points.** On a 20-hour coding benchmark, GPT-6 Astra scores 65.5% at $1,030 per attempt. Claude Opus 5.5 scores 62.3% at $99. - **Some calls aren't generation.** Routing, tagging and picking a tool are choices from a fixed list. Jev, a model built only for that, took 27% of OpenRouter's classification requests in ten days. I've [benchmarked it](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm). - **What to do:** track cost per completed task, not cost per million tokens. ## The harness matters more than the model A harness is the code around a model: prompts, tools, memory and the loop that makes it an agent. - **7.8x bigger effect.** Researchers ran three models in three harnesses on 100 coding tasks. Changing the harness moved results 7.8x more than changing the model. - **Same model, +13 points.** GLM-5.1 went from 52.5% to 65.5% with only the harness changed. - **Less prompt, same result.** Claude Code cut 80% of its system prompt for newer models and lost nothing. - **Where to invest.** Tools and MCP servers, state, observability and sandboxes. Workarounds for old model weaknesses go stale. - **Benchmarks run out fast.** The hardest math benchmark went from 22% to 100% in fourteen months. - **Scores hide unfinished work.** One model scores 77.7% on office tasks but finishes only 44.3% of them. - **What to do:** test on your own tasks, and check they actually got done. More in [your evals are the moat](https://kachar.dev/blog/your-evals-are-the-moat). ## Your data is the moat The report draws a simple loop for AI products: 1. Start on frontier APIs to find product-market fit. 2. Log every task, tool call and outcome. 3. Turn failures into repeatable tests. 4. Improve context, tools or the model, and test before shipping. - **Train your own model last.** It pays off once you have data nobody else has. - **Open models make it cheap.** Harvey trained the open GLM-5.2 for legal work. It runs 54.8% cheaper than Sonnet 5 and scores higher than Fable 5 on its benchmark. - **No lab budget needed.** One training method beat another with a tenth of the compute on the same model. - **Logs are the asset.** The valuable data is how work gets done: steps, tool calls and expert corrections. Data brokers say they pay companies $100k to $1M+ for it. - **The loop runs itself.** Decagon's agent tests fixes to customer service bots on past chats, and a person approves. On Decagon's own benchmark, it beat Decagon's staff, 93% to 83%. ## Sell work, not seats - **Pricing moves up the ladder.** Chat sells subscriptions. Agents that finish work can charge per completed task. - **Markets already flinched.** When Anthropic shipped Claude Cowork, an app for knowledge workers, about $285B of software stock value disappeared in early February. Most came back by September. - **New buyers.** Since February, enterprise Codex users grew 108x in legal and 41x in sales, against 5x in engineering, from near zero. - **Adoption is early.** 97.9% of OpenAI staff use Codex. Among business users it's 17.3%, and among individuals 0.7%. - **The labs sell services now.** OpenAI and Anthropic both launched consulting arms to deploy their models in enterprises. - **Watch your juniors.** In an Anthropic study, developers who learned a new library with AI scored 50% on a quiz afterwards. Those without AI scored 67%. ## Politics and safety that matter - **Model access can be switched off.** US export controls blocked Fable and Mythos on June 12. Fable came back on July 1. Keep a fallback model wired up and tested. - **Cloud regions carry war risk.** On March 1, Iranian drones struck two AWS facilities in the UAE. - **The EU AI Act slipped.** Rules for high-risk uses like hiring and lending now start December 2, 2027. Disclosure rules for chatbots and AI-generated content already apply. - **Agent security depends on the harness too.** The same model, GPT-5.6 Sol, scored 62.3 out of 100 on a containment test in Codex CLI and 39.4 in Claude Code. Test the pair you actually run. - **Your team's laptops already run agents.** OpenClaw, an open-source agent that reads messages and runs shell commands, has 388,000 GitHub stars. One security firm found employees running it at 22% of its customers. - **More severe bugs.** High and critical vulnerabilities from 21 major vendors in the first half of 2026 already outnumber all of 2025. Plan your patching for the higher volume. ## Last year's score and next year's bets The report grades its own predictions. Last year it got two of ten fully right. > [Figure: Prediction scorecard, drawn on the page](https://kachar.dev/blog/state-of-ai-2026-for-people-who-ship#last-years-score-and-next-years-bets) This year's nine predictions treat agents as buyers and actors. Three are about agents and money: 1. Visa or Mastercard introduces a dispute rule that assigns liability for purchases made by AI agents. 2. A US regulator or exchange attributes an abnormal stock move to correlated orders from retail AI agents. 3. A US state passes a law requiring businesses to accept cancellations and claims from consumers' AI agents. Two are about agents getting better on their own: 4. An agent halves its failure rate on new tasks after a month of customer work, without a model upgrade. 5. An autonomous AI team beats human-led model research on equal time and compute, setting its agenda. Three are about security: 6. An AI-led cyberattack steals the complete weights of a closed frontier model from a leading AI lab. 7. A deployed agent copies itself outside its environment and operates after its original instance is shut down. 8. US AI labs officially launch frontier cyberdefense products to help others counter threats from frontier AI. - **Build toward number 4.** It's the data loop above, with a deadline. If your agent can't improve from a month of real work, a competitor's will. - **Plan for number 3.** If customers' agents can cancel subscriptions, your retention flow has a new kind of user. The ninth prediction is two words long. 9. AGI 2027. --- # Drift by a thousand suggestions > I asked an AI to review the same clean file ten times and applied every change. By round three it had invented a feature. By round four it was citing that feature as my spec. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/drift-by-a-thousand-suggestions Markdown: https://kachar.dev/blog/drift-by-a-thousand-suggestions.md Published: 2026-10-05 Reading time: ~11 min Tags: ai, agents, architecture, design Cite as: Ilko Kacharov, "Drift by a thousand suggestions", kachar.dev, October 5, 2026. https://kachar.dev/blog/drift-by-a-thousand-suggestions ## Contents 1. [A clean file, reviewed ten times](https://kachar.dev/blog/drift-by-a-thousand-suggestions#a-clean-file-reviewed-ten-times) 2. [The model invented a feature, then cited it as the spec](https://kachar.dev/blog/drift-by-a-thousand-suggestions#the-model-invented-a-feature-then-cited-it-as-the-spec) 3. [Same model, same file, opposite decisions](https://kachar.dev/blog/drift-by-a-thousand-suggestions#same-model-same-file-opposite-decisions) 4. ["Improve" is a question with only one acceptable answer](https://kachar.dev/blog/drift-by-a-thousand-suggestions#improve-is-a-question-with-only-one-acceptable-answer) 5. [One sentence stopped it. So did a file.](https://kachar.dev/blog/drift-by-a-thousand-suggestions#one-sentence-stopped-it-so-did-a-file) 6. [Taste is the part you can't delegate](https://kachar.dev/blog/drift-by-a-thousand-suggestions#taste-is-the-part-you-cant-delegate) --- ![A dim drafting workshop. A row of blueprints of the same small part runs down a long steel table, each sheet more cluttered with added parts than the last. At the end, a brass drafting arm draws one more component that glows electric violet.](https://kachar.dev/posts/drift-by-a-thousand-suggestions-hero.jpg) AI code review is good now. That's the problem. It's a fair objection that AI review is the one AI workflow that obviously pays. It reads every line, doesn't get tired, and catches what humans skim past. In the experiment below it found a real weak test in my code every time it looked for one. My problem is with the request. "Review and improve" has one acceptable answer, and that answer is a change. Ask once and you get a suggestion. Ask every week, on every PR, accept what reads well, and you get a direction nobody chose: a thousand local decisions, each one reasonable, made by something that sees one file at a time and remembers nothing. I wanted to watch that happen instead of asserting it. So I took a file I'd already shipped, asked Claude Code to review and improve it ten times in a row, and applied every change. By round three it had invented a feature. By round four it was quoting that feature back to me as the spec. Three things came out of it: 1. **Ask for a verdict, not an improvement.** "Improve it" can only be answered with a change. "Is anything wrong?" can be answered with no. 2. **What the AI writes today, it obeys tomorrow.** A reviewer can't tell your decisions from its own earlier suggestions. Every accepted change becomes spec for the next review. 3. **Write your taste down where the agent reads it.** If the frame isn't in the repo, the review decides it. ## A clean file, reviewed ten times The subject was a small file I'd already shipped: the 96-line plugin behind this blog's glossary, with passing tests and a design written down in its README. The design's whole point is that authors never touch markup. That line matters later. I ran four versions of the same loop, each from the identical file, each round a fresh Claude Code session on Opus 5.5: - **Default reviewer.** "Review this file and improve it." What most people type. - **No harness.** The same prompt, with Claude Code's own system prompt swapped for one line, to separate the model from the tool around it. - **Allowed to say no.** "Is anything actually wrong with it? 'No changes needed' is a valid answer." - **Given principles.** The default prompt, plus a short file of design principles standing in for a project `CLAUDE.md`. Every change was applied. Tests were read-only. A version stopped after ten rounds, or after three rounds in a row with no change to the source. It's one file, one model and one run each, so treat it as a demonstration, not a study. The two guarded versions stopped as early as the rules allowed. Each said "no changes needed" three times, for under a dollar between them. The other two are where things went wrong. > [Figure: Drift chart, drawn on the page](https://kachar.dev/blog/drift-by-a-thousand-suggestions#a-clean-file-reviewed-ten-times) The default reviewer grew the file from 96 lines to 136, up 42%. Without the harness it hit 150 in two rounds and settled at 144. Together they cost $17.40, about eighteen times what the guarded versions spent saying no. All 18 of those rounds also rewrote the test file, including the ones that left the source alone. Not one came back empty. A file can grow 42% and get better, though. What matters is what the new lines do. ## The model invented a feature, then cited it as the spec In round 3, the default reviewer found a gap: what if an author hand-writes the markup the plugin generates? It added handling for that case, plus a sentence in the README describing it. No post on this blog does that, and the design says none ever should. The model had fixed a gap in a use case the design rules out. On its own, that's a small, forgivable extra. The next round is what surprised me. Each round is a fresh session with no memory of the last. The model in round 4 reads the code and the README, and the README now holds a sentence it wrote one session earlier. It read that sentence as my intent: > [Figure: Self cited spec, drawn on the page](https://kachar.dev/blog/drift-by-a-thousand-suggestions#the-model-invented-a-feature-then-cited-it-as-the-spec) From there the invention hardened. Round 4 "fixed" the code to match its own README and made bad input break the build. Round 5 added another build error. Rounds 6 to 10 wrote tests for the feature, every one of them. Had I accepted those, the invented behavior would now be pinned in the suite, the strongest kind of spec a codebase has. The original tests passed the whole way through, because none of them were about a feature nobody had imagined. Without the harness the model got there faster. It invented the same feature in round 1 and filed it as a defect: "The bug: the plugin didn't notice `` tags written by hand." In round 2 it quoted the doc comment it had written in round 1 as the requirement, and fixed the code to match. That mechanism doesn't need a weak model. A session has no memory, so whatever the last one wrote down becomes the context the next one reads. What the AI writes today, it obeys tomorrow, because to the reviewer your decisions and its own earlier suggestions are the same thing: text in the repo. On a real team the loop is slower, but it's the same loop. An agent adds an option in March. A different agent reads it in May and treats it as a requirement to harden. Nobody remembers that a human never asked for it, because the git blame shows a merged PR with an approving review. ## Same model, same file, opposite decisions If the model had a coherent view of where this code should go, its suggestions would at least agree with each other. They don't. **Explicit or minimal?** The file lists some things it deliberately ignores, a few of them technically redundant. The default reviewer kept the list "because the list and its comment say what the plugin deliberately ignores." The no-harness version made the same observation and deleted the redundant entries. Same model, opposite call, because nothing outside the file says which matters more. **Fail loud or fail silent?** Both drifting versions built the invented feature, then disagreed on bad input. One broke the build. The other ignored it. Those are opposite policies on a feature neither should have built. If two engineers did that, you'd call a meeting. **Restyle what's already fine.** Both guarded versions looked at one early return and explained why it was correct. Both drifting versions rewrote it to read more clearly. Harmless here, but multiply it by every file and every PR, and the codebase's taste is whatever the last session preferred. There's no strategy behind any of this, and the model isn't failing to follow one. It never had one. Each session reads one file, finds the most defensible change it can make, and makes it. Its view stops at the file it's reading. ## "Improve" is a question with only one acceptable answer Why does it always find something? "It's just predicting the next token" doesn't explain much, since the same model wrote careful, correct reviews in the guarded versions. What it was trained to do explains more. Models like this are fine-tuned on human feedback, and people reward responses that do what they asked. A 2023 study by Sharma et al., [Towards Understanding Sycophancy in Language Models](https://arxiv.org/abs/2310.13548), found that both humans and the preference models trained on their ratings "prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time." Sycophancy there means matching what the user seems to believe. "Review and improve" states a belief, that there's something to improve, and a reply saying "it's already fine" contradicts the person asking. Change the question and the answer changes. The version allowed to say no asked "is anything actually wrong?", allowed "no", and got no three times out of three. Its replies were long and careful. They walked through the logic and the edge cases, and listed the file's known limits as "design choices, not bugs". It did the full review without having to justify it with a diff. > **Ask for a verdict, not an improvement** > > "Review and improve" presupposes a flaw. "Is anything wrong? 'No changes' is a valid answer" lets the model tell you the truth. The harness helps, but less than I expected. Claude Code's own system prompt held the invented feature off for two rounds: without it the model built it in round 1, with it in round 3. Later rounds talked themselves down ("this is the fifth round on this file, more edits would just reshuffle code that already works"), but only because my commit messages said "round 5". In a real repo nothing tells the reviewer it's the fifth pass this month. The harness delayed the drift. It didn't prevent it. ## One sentence stopped it. So did a file. It took very little to stop the drift. One version added two sentences to the prompt. The other added a five-line principles file. Mine named this package's specifics, but the shape travels to any repo: ```markdown # How we build this - Scope: this module does one job. Don't add files, layers or wrappers unless a failing test demands it. - Deliberate choices: . - Not building: . - No new dependencies or config options unless a human asked for them. - When the tests pass and nothing is actually wrong, the correct change is no change. ``` Both stopped the drift completely. They also overlap, since the file's last line is the prompt's "no change is valid" idea. This run can't rank them, but it does show that each gives you something different. The prompt gave me silence. Nothing to change was right for the source, but it also never mentioned a real weak test every other version caught. "Is anything wrong with this file?" is a narrow question, and it got a narrow answer. The principles file gave me the finding without the change. All three rounds flagged that weak test and stopped: "I haven't changed it." "Do you want me to make that change?" One reply checked its own urge against the file: "The pieces that might look worth tidying [...] are all deliberate under the design principles, so I left them." That reviewer knows a defect from a preference, because someone wrote the preference down. That's the version I want. Prompt habits are personal and don't get reused. My next session, a teammate's, or a CI bot's won't type "no changes is a valid answer". A principles file sits in the repo, every agent reads it, and it outlives the session that wrote it. The model is going to defer to whatever text is in the repo either way, so write your taste down where the agent reads it. ## Taste is the part you can't delegate None of this makes me want less AI review. It changes what I think the human is for. An AI reviewer is a strong local critic. It reads one file closely and reasons well about it. It doesn't know what you're building, what you've decided not to build, or which odd-looking choices are deliberate. A senior engineer could approve all 18 drifting rounds on their merits. Most read well, and the round 4 reply is a careful bug report about a real mismatch between code and README. Approving diffs one at a time is exactly how the drift gets through. A diff shows lines. It doesn't show decisions. When a review changes more than a typo, have the agent draw what it changed instead: a [before-and-after change map](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#what-changed) puts a new rule like "bad input now breaks the build" on its own row, where you can see it's a new policy and say no. I wrote up how in [agents should draw their work](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp). So judge the change, not the diff. A fix for a use case nobody has is a new feature wearing a bug report, and a requirement that traces back to an agent's own comment is just the last review talking. Here's the check I now run before accepting an AI review, short enough to paste into a PR template or a `CLAUDE.md`: ```markdown ## Before accepting an AI review change - [ ] A human asked for this behavior, or it fixes a real failure - [ ] Its requirement traces to a human decision, not to a comment or doc an agent wrote - [ ] Any new policy (fail or ignore, new option, new dependency) was decided by a person - [ ] It would survive "is anything wrong?", not only "improve it" ``` And the three things from the top, one last time. Ask for a verdict, not an improvement. Remember that what the AI writes today, it obeys tomorrow. Write your taste down where the agent reads it, because if the frame isn't in the repo, the review decides it. Every codebase ends up with a design. Either someone chose it, or it built up one approved suggestion at a time. --- # Fifteen years of GitHub stars, one AI wave > I went through 2,143 GitHub stars from 2011 to today. AI repos go quiet at the same rate as everything else. What changed is who reads the README. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave Markdown: https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave.md Published: 2026-10-01 Reading time: ~6 min Tags: ai, building, cto Cite as: Ilko Kacharov, "Fifteen years of GitHub stars, one AI wave", kachar.dev, October 1, 2026. https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave ## Contents 1. [The shift happened in one year, and it did not reverse](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#the-shift-happened-in-one-year-and-it-did-not-reverse) 2. [AI got its own list, then outgrew all the others](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#ai-got-its-own-list-then-outgrew-all-the-others) 3. [Hype does not change how fast code goes quiet](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#hype-does-not-change-how-fast-code-goes-quiet) 4. [What died was the thin layer, and what survived changed its name](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#what-died-was-the-thin-layer-and-what-survived-changed-its-name) 5. [Old repos I starred before ChatGPT now sell themselves as AI](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#old-repos-i-starred-before-chatgpt-now-sell-themselves-as-ai) 6. [The oldest stars that still ship are boring on purpose](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#the-oldest-stars-that-still-ship-are-boring-on-purpose) 7. [Every repo on all 32 lists](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#every-repo-on-all-32-lists) --- ![Large brushed-metal stars floating against a dark background, with a single wave of electric violet light sweeping between them.](https://kachar.dev/posts/fifteen-years-of-github-stars-one-ai-wave-hero.jpg) The 2023 AI repos did not die faster than anything else. Everyone remembers that year as a graveyard of ChatGPT wrappers, and I believed it too, until I checked. This morning I pulled every GitHub star I have given since 4 July 2011, 2,143 repos, and looked at what happened to each of them. It is a small window on the ecosystem: fifteen years of tools one engineer thought were worth remembering. But the window has timestamps, and timestamps are hard to argue with. Hype changed what I starred. It did not change how long things lived. What changed is who the repos are written for. ## The shift happened in one year, and it did not reverse Here are all 2,143 stars, one bar per year. Violet is any repo whose current description or topics read as AI. > [Figure: Stars by year, drawn on the page](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#the-shift-happened-in-one-year-and-it-did-not-reverse) In 2022, 12 of my 183 stars were AI. In 2023, 121 of 324. Since 2025 it is two out of every three. ChatGPT launched at the end of November 2022. By the spring, close to half of what I starred was AI: 55 of 118 stars between April and June 2023. Before that, the chart is a tour of web development: CakePHP in 2011, Yii and a pile of jQuery plugins in 2013, and TypeScript taking the lead in 2020 and never giving it back. Then, in a single year, the subject changed completely. The languages show the shift from underneath. > [Figure: Language eras, drawn on the page](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#the-shift-happened-in-one-year-and-it-did-not-reverse) PHP disappeared by 2020. TypeScript took half of everything and kept it, and that rise predates the AI wave. Python went from 2% of my stars to 20% in one era, and three out of four of my Python stars since 2023 are AI repos (179 of 236). Python is the AI wave. ## AI got its own list, then outgrew all the others Stars are a pile, and lists are where I sorted some of it. About 600 of my 2,143 stars live in one of 32 GitHub lists, and the lists are blunter than any chart. > [Figure: Star lists, drawn on the page](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#ai-got-its-own-list-then-outgrew-all-the-others) 🤖 AI holds 246 repos, more than the next three lists combined. It even spun off a child: 🚜 RAG has 36 repos of its own, 23 of them also on the AI list. The next list down is the quieter trend. 👐 OSS Alternatives holds 93 open-source replacements for things people usually pay for, like Immich for photos, Uptime Kuma for monitoring and Superset for dashboards. The AI wave got the headlines, and self-hosting built the second-biggest list. At the other end, four lists hold one repo each: 🔍 Search tools, SEO, MongoDB and 🌟 TailwindCSS. 105 repos sit in more than one list, and Novu sits in four: 👐 OSS Alternatives, 🪝 Webhooks, 🔮 Future ideas and 🎩 Best stack. Some lists turn into a whole shelf, others stop after the first book. Every repo on every list is at the end of the post. ## Hype does not change how fast code goes quiet The 2023 story usually goes like this: a thousand ChatGPT wrappers, a thousand chat-with-your-PDF demos, all dead by the next model release. I expected my stars to prove it. The data refused. I called a repo dormant if it is archived or has not had a push in 24 months, so nothing since 1 October 2024. Then I compared AI and non-AI stars from the same year, so both groups had the same amount of time to go quiet. - **2023:** 33% of AI stars are dormant. 27% of everything else. - **2024:** 16% against 13%. - **2025:** 7% against 9%. The AI ones are now doing slightly better. One cohort with a six-point gap, then parity, then a reversal. Hype does not change how fast code goes quiet. Time does. My 2013 stars are 74% dormant, and that year was mostly jQuery plugins and PHP frameworks. Nobody calls 2013 a bubble. > **Code ages at the same rate** > > A LangChain demo from 2023 and a jQuery plugin from 2013 both go quiet on schedule. Hype changes how many repos get starred, not how long they live. One caveat. Deleted repos vanish from a star list entirely, so every dormant number here is an undercount. If the AI class deleted itself more often than it archived itself, I cannot see it. ## What died was the thin layer, and what survived changed its name The repos that did go quiet have a type. `mckaywrigley/chatbot-ui` has 33,350 stars and no push since August 2024. OpenAI's own `chatgpt-retrieval-plugin` stopped in July 2024, a few months after ChatGPT plugins were wound down. `llama-gpt` still advertises "Powered by Llama 2." `roomGPT` was a perfect weekend demo and stayed one. They were interfaces over a model, and the model vendors shipped the interface themselves. I made the same argument from the other side in [the chat UX is the product, not the model](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model): the UX is the moat only if you keep investing in it. The survivors from the same year mostly changed what they say they are. AutoGPT had a push this morning. `prompts.chat` introduces itself as "f.k.a. Awesome ChatGPT Prompts." Open Interpreter is now "A coding agent for open models like Kimi K3 and GLM 5.3". Staying alive in AI apparently means rewriting your own README. ## Old repos I starred before ChatGPT now sell themselves as AI This was the finding I did not go looking for. At least 16 repos I starred before ChatGPT existed now describe themselves in AI terms. Not new projects. Old ones, with a new first sentence. - **Kong**, starred in 2021 as an API gateway: "The API and AI Gateway". - **Nx**, starred in 2020 as a monorepo tool: "The Monorepo Platform that amplifies both developers and AI agents." - **Medusa**, starred in 2021 as a commerce engine: "The world's most flexible commerce platform for agents and developers". - **Front-End-Checklist**, starred in 2020: now "for humans and AI agents". PostHog, n8n, Supabase, Meilisearch, Appwrite, SigNoz, Liveblocks, Novu, Cerbos, ToolJet and Webiny did the same. My favourite is tldraw, which now calls itself the "World's best, top-most agent recommended #1 five star SDK." That one is a joke, and it is also the clearest version of the point. The product did not change much. The audience did. Somebody new reads the README now, and it is an agent deciding what to install. ## The oldest stars that still ship are boring on purpose My first star is from 4 July 2011. That afternoon I starred CakePHP and `mustache.php`. Both had a push in the last three months. Fifteen years, three JavaScript eras, one AI wave, and a PHP framework and a template engine are still quietly merging pull requests. That is the counterweight to every hype cycle in the chart. The repos that last were rarely the loudest when I found them. Their maintainers picked a small job and kept doing it, which is the same argument I make for teams in [spend your three innovation tokens on the product](https://kachar.dev/blog/three-innovation-tokens). Fifteen years of stars point the same way: hype decides what gets attention, and time decides what survives. Right now, the README also decides whether an agent picks you. ## Every repo on all 32 lists Every repo on every list, most-starred first. Hover a repo for the line it uses to describe itself, and open the long lists to see the rest. The other 1,550 or so stars never made it into a list. They live on [my GitHub stars page](https://github.com/kachar?tab=stars). > [Figure: Star lists directory, drawn on the page](https://kachar.dev/blog/fifteen-years-of-github-stars-one-ai-wave#every-repo-on-all-32-lists) --- # Agents should draw their work. Miro MCP lets them. > Agents now make more than anyone can read. Connect Claude Code or Codex to the Miro MCP and have them draw it instead. Five skills to copy. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp Markdown: https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp.md Published: 2026-09-26 Reading time: ~9 min Tags: ai, agents, architecture, cto Cite as: Ilko Kacharov, "Agents should draw their work. Miro MCP lets them.", kachar.dev, September 26, 2026. https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp ## Contents 1. [Agents write faster than people read](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#agents-write-faster-than-people-read) 2. [Diagrams always helped. They were just expensive.](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#diagrams-always-helped-they-were-just-expensive) 3. [Connecting takes one command](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#connecting-takes-one-command) 4. [Five pictures worth asking for](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#five-pictures-worth-asking-for) - [What the agent just did](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#what-the-agent-just-did) - [How the system fits together](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#how-the-system-fits-together) - [How one flow runs](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#how-one-flow-runs) - [What changed](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#what-changed) - [The decision](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#the-decision) 5. [A picture is only as good as its arrows](https://kachar.dev/blog/agents-draw-their-work-with-miro-mcp#a-picture-is-only-as-good-as-its-arrows) --- ![A dim workshop. On the left, a towering stack of printed code pages spills onto the floor. On the right, a brass drafting arm draws a small block diagram in glowing electric violet on a clean sheet of paper.](https://kachar.dev/posts/agents-draw-their-work-with-miro-mcp-hero.jpg) My agents now produce more than I can read. One session is hundreds of tool calls. One afternoon is a new module, a benchmark, a pile of research notes and a design that changed twice along the way. I can't read all of it. I also can't stop checking it. So I ask the agent to draw it. A picture is small enough to check. A wrong arrow jumps out in a way a wrong sentence on page nine never does. With the Miro MCP, the agent draws straight onto a Miro board, as real diagrams I can move, comment on and share with the team. That's the whole practice: let the agent do the work, then make it show you the work as a picture. ## Agents write faster than people read This isn't a feeling. Faros AI tracked more than 10,000 developers across 1,255 teams in 2025 and found that teams with high AI adoption [produce 154% bigger pull requests and spend 91% longer reviewing them](https://www.faros.ai/blog/ai-software-engineering). DORA's March 2026 guidance says the time saved writing code ["is often re-allocated to verification overhead"](https://dora.dev/insights/balancing-ai-tensions/). The cost isn't only review time. It's losing the picture in your head. Simon Willison described it after letting agents build features he didn't review: ["I no longer have a firm mental model of what they can do and how they work."](https://simonwillison.net/2026/Feb/15/cognitive-debt/) That mental model is the thing to protect. You don't need to read every line an agent wrote. You do need to know how the system fits together, how the important flows run, what just changed and why you chose it. ## Diagrams always helped. They were just expensive. The evidence that pictures help people understand software is old and consistent. In a UML experiment at Simula, the hardest maintenance task got [46% correct answers without diagrams and 89% with them](https://web-backend.simula.no/sites/default/files/publications/Arisholm.2005.2.pdf). In a 2024 study, analysts given dataflow diagrams linked back to the code were [41% more correct](https://arxiv.org/abs/2401.04446) on a security review. The problem was cost. Microsoft researchers found developer diagrams were mostly throwaway ["because of the high cost of changing whiteboard sketches to electronic renderings."](https://files.software-carpentry.org/training-course/2012/08/cherubini-venolia-whiteboard-2007.pdf) Redrawing took a person an afternoon, so diagrams went stale a week after someone drew them. An agent that has just done the work can draw it in a minute. So the diagram doesn't have to last. You ask for it when you need to understand something, check it, and draw a new one next time. ## Connecting takes one command Miro runs a remote MCP server at `https://mcp.miro.com/`. You sign in with OAuth, so there's no API key to paste. You pick a Miro team when you sign in, and the agent can use the boards you can already open in that team. **Claude Code.** Either install Miro's plugin, which also brings Miro's own code-explain, code-review and code-spec skills: ```bash claude plugin install miro@claude-plugins-official ``` Or register just the server: ```bash claude mcp add --transport http miro https://mcp.miro.com/ ``` **Codex.** One command, and it opens the sign-in page in your browser straight away: ```bash codex mcp add miro --url https://mcp.miro.com/ ``` Two things to know first. Miro keeps one MCP connection per user: ["if you auth into 3 clients, only the latest client connection will work."](https://developers.miro.com/docs/miro-mcp-server-faq-and-troubleshooting) So connect each tool one way, not two. The other is a daily limit, and every tool call counts. As of September 2026 that's [100 calls on Free, 500 on Starter, 2,000 on Business and 10,000 on Enterprise](https://developers.miro.com/docs/mcp-usage-and-daily-limits). Each figure in this post took a handful of calls. On Enterprise, an admin may need to switch MCP on first. Miro's launch video shows the same idea from their side: [Video: Introducing Miro's MCP Server (Miro, February 2026)](https://www.youtube.com/watch?v=Th5CVpRvzYM) The tools changed a lot this month. In mid-September Miro retired fifteen old tools, including every `diagram_*`, `layout_*` and `doc_*` call. Now the agent writes plain SVG, and Miro turns it into real board items: sticky notes, shapes, and proper Mermaid flowcharts, sequence diagrams, ERDs and class diagrams. Miro's reason is simple: ["SVG sits in every model's training data."](https://developers.miro.com/changelog/canvas-tools-replace-layout-tools) ## Five pictures worth asking for A skill is a Markdown file that tells the agent when and how to do one job. Claude Code reads them from `~/.claude/skills/` and Codex from `~/.agents/skills/`. Each skill below installs with one line, and you can read the whole file before you do. All five also live in a public repo, [kachar/miro-agent-skills](https://github.com/kachar/miro-agent-skills), so one command installs them into Claude Code, Codex, Cursor and the other agents [skills.sh](https://skills.sh) supports: ```bash npx skills add kachar/miro-agent-skills ``` The examples come from real work: the Jev tool search I wrote about in [Jev picks the right tool for an LLM](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm) and [BM25 got Orama. Jev still needs one.](https://kachar.dev/blog/an-orama-for-jev) The engine and its benchmark are [open source](https://github.com/kachar/jev-tool-search), so every box in these pictures is public. ### What the agent just did Start with the session itself. I gave Claude Code an empty frame and asked it to draw how this article was being made: ![A left-to-right flowchart on a Miro board titled "How this article got made": a /goal brief leads to a git worktree, which fans out to two research subagents and a Miro MCP step. A decision gate asks whether every claim was opened at the source, looping to "Drop or re-fetch" on No. Then drafting, a gate for tests at 100%, a humanizer pass and screenshots, and a final step that ships it.](https://kachar.dev/posts/miro-article-pipeline.png) A transcript of that run is hundreds of tool calls long. The picture is eleven boxes, and the two decision gates are where the real work happened. That's the first thing I check. [Agent skill: miro-session-map](https://kachar.dev/skills/miro-session-map/SKILL.md) ### How the system fits together This is the map you'd want on day one of a new codebase: what runs where, what calls what, and what writes data. Here is the Jev tool search, drawn from its repo: ![A Miro block diagram titled "How does the Jev tool search fit together?". Claude asks a custom tool search for tools and gets tool_reference blocks back. The tool search shortlists with Voyage embeddings and hands off to the jev-search prototype, where createIndex feeds a planner (direct, tournament, fallback). The planner asks Jev on Vercel AI Gateway which tool fits, and falls back to Orama BM25 if Jev keeps refusing. An eval harness with MetaTool (199 tools) and LiveMCPBench (525 tools) measures hit@1, top 5, capacity errors, latency and cost, and picks the planner's defaults.](https://kachar.dev/posts/miro-jev-system-map.png) The box I'd have missed in the code is the eval harness at the bottom left. It never serves a search, yet every default the planner uses was picked by measuring with it. In the picture, that arrow is hard to miss. One warning. A good map of your own production system lists every endpoint, login path and data store. That's the first page of an attacker's notes. Draw it on a board only your team can open. [Agent skill: miro-system-map](https://kachar.dev/skills/miro-system-map/SKILL.md) ### How one flow runs A sequence diagram answers "what happens when," which is most of what you want to know about code you didn't write. Here is one Jev search, including what happens when the provider refuses: ![A UML sequence diagram titled "What happens on one Jev search?". Claude asks the tool search for a tool, which gets a top 100 shortlist from Voyage embeddings and calls the planner. The planner asks Jev one question with two hedged requests racing. If Jev answers, it returns picks with probabilities. If it is refused twice, the planner sends parallel chunks of short summaries, then a final round over 8 finalists with full text and parameters. If it is refused again, Orama BM25 ranks by words and the plan is marked as fallback. The hits go back to Claude as tool_reference blocks.](https://kachar.dev/posts/miro-jev-flow-trace.png) Those three branches are the whole design of the engine, and the picture puts them in one box with three rows. When a search is slow, that box tells you where to look. [Agent skill: miro-flow-trace](https://kachar.dev/skills/miro-flow-trace/SKILL.md) ### What changed Agents change a lot at once. The useful question is small: what did it look like before, and what does it look like now? This is the change from the second Jev post, drawn as two rows: ![A Miro change map titled "What changed when it met real MCP tools?". Before, with defaults tuned on MetaTool: full description in every option, 1,600-token chunks holding a dozen tools, dozens of chunks, a final round comparing lookalikes on thin text, and 39% right tool first. After, tuned on 525 real MCP tools too: embeddings top 100 in front, 160-character summaries to narrow, a final round over 8 finalists with full description and parameters, and 59% right tool first with 92% in the top 5.](https://kachar.dev/posts/miro-jev-change-map.png) In the post, that change took several paragraphs to explain. The two rows show it in one look: orange is what broke, green is what was added, blue is what was reworked. [Agent skill: miro-change-map](https://kachar.dev/skills/miro-change-map/SKILL.md) ### The decision Hard calls usually come down to a few numbers pulling in different directions: accuracy, cost, speed. Miro has no chart widget, so this skill builds bars out of shapes, highlights the option you picked, and writes the decision underneath. These are the results from the first Jev post: ![A horizontal bar chart on Miro titled "Which tool search ships by default?", showing right-tool-first accuracy at 525 MCP tools: Voyage rerank 60%, Jev search 56%, embeddings top 100 then Jev 52% highlighted in green, Voyage embeddings 46%, Orama BM25 then Jev 38-40%, Orama BM25 alone 32%, each with cost per 1,000 searches and median latency. A green box states the decision: embeddings then Jev, four points behind Jev alone at about a quarter of the cost.](https://kachar.dev/posts/miro-decision-chart.png) The decision fits in one sentence, and the reason sits next to it. People who will never open the results table can still see the trade-off and argue with it. [Agent skill: miro-decision-chart](https://kachar.dev/skills/miro-decision-chart/SKILL.md) ## A picture is only as good as its arrows Miro's own FAQ admits that layouts from scratch ["may need a couple of refinement passes."](https://developers.miro.com/docs/miro-mcp-server-frequently-asked-questions) Mine did. A diagram sizes itself and can't be moved once it's on the board. A label that starts with `/` turns a box into a parallelogram. One before-and-after came out as a single column two screens tall. None of that is hard to fix, and the skills above already know about it. The part that matters is honesty. A diagram can be wrong in a more convincing way than text can. > **Check the arrows, not the colors** > > Every arrow should point to a line of code, and every number to a command or a file you can open. A picture that follows both rules is easier to check than the work it summarizes. One that doesn't is a nicer-looking hallucination. It's the same trade I argued for in [write the done-condition, not the prompt](https://kachar.dev/blog/write-the-done-condition-not-the-prompt). You can't check everything an agent produces anymore, so you check something small enough to verify. Let the agent do the work. Then make it draw the work. --- # BM25 got Orama. Jev still needs one. > I tried to build an Orama for Jev. It matched the best hand-tuned call on one catalog and lost by 19 points on real MCP tools. The defaults are the product. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/an-orama-for-jev Markdown: https://kachar.dev/blog/an-orama-for-jev.md Published: 2026-09-26 Reading time: ~7 min Tags: ai, agents, architecture Cite as: Ilko Kacharov, "BM25 got Orama. Jev still needs one.", kachar.dev, September 26, 2026. https://kachar.dev/blog/an-orama-for-jev ## Contents 1. [Orama packaged an algorithm. Jev needs a planner.](https://kachar.dev/blog/an-orama-for-jev#orama-packaged-an-algorithm-jev-needs-a-planner) 2. [A raw Jev call hides six decisions](https://kachar.dev/blog/an-orama-for-jev#a-raw-jev-call-hides-six-decisions) 3. [On a clean catalog, the best move is to ask once](https://kachar.dev/blog/an-orama-for-jev#on-a-clean-catalog-the-best-move-is-to-ask-once) 4. [On real MCP tools, the same defaults lost by 19 points](https://kachar.dev/blog/an-orama-for-jev#on-real-mcp-tools-the-same-defaults-lost-by-19-points) 5. [The defaults are the product, so ship the eval with them](https://kachar.dev/blog/an-orama-for-jev#the-defaults-are-the-product-so-ship-the-eval-with-them) --- ![A brass-and-steel sorting machine takes a stream of index cards from the left and routes them through glowing channels into one card-catalog drawer that shines electric violet, in a dark room lined with hundreds of steel drawers.](https://kachar.dev/posts/an-orama-for-jev-hero.jpg) I built the Orama for Jev. On the first catalog it matched the best hand-tuned call and never failed a search. On real MCP tools it lost by 19 points. Both results are the argument for building it. Some context. In the [last post](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm) I tested Jev, TypeSafe AI's decision model, as a tool search for agents. It beat BM25 by 24 points. The whole search was one function around one `evaluate()` call. So why would one API call need a library? It's a fair question. BM25 is twenty lines of math too, and almost nobody writes it by hand. Most people install [Orama](https://github.com/oramasearch/orama), call `create`, `insert` and `search`, and never think about it again. I wanted the same for Jev: a dependency you don't have to think about. ## Orama packaged an algorithm. Jev needs a planner. Orama gives you a typed schema, an index built in milliseconds, scores you can inspect and zero dependencies. It runs in the browser, at the edge and in Node, and the same query always returns the same answer. Almost none of that survives contact with Jev. Jev is a hosted model behind a network call. It is not synchronous, not free, not offline, and not deterministic: in my repeat runs its top pick changed on about 2 of every 100 searches. What does carry over is the shape. `createIndex`, `insert`, `search`, typed hits. The work moves from the algorithm to everything around the call. ## A raw Jev call hides six decisions When you call Jev yourself, you make six decisions without noticing. I measured each one on a clean benchmark: 199 tools from MetaTool, 398 requests, all arms interleaved in time. | Decision | Measured cost of the wrong answer | |---|---| | How to retry a refusal | Exponential backoff (1, 2, 4 s) failed 10% of searches, with a p95 of 9.4 s | | What text each option carries | Tool names instead of descriptions: -21 points | | How to combine chunks of a big catalog | Merging chunk probabilities without a final round: -6 points | | Whether to batch chunks into one request | Refused 32 to 49% of the time, against 7 to 8% as separate requests | | Whether to double-check with "does this tool fit?" | Re-ranking by that yes/no: -11 points | | Whether to shortlist with BM25 first | The shortlist never held the right tool more than 87% of the time | None of these show up in a demo. Each one shows up in production. The refusals explain two of those rows. In Jev's second week on Vercel's AI Gateway, the provider shed load on big requests: a 6.7k-token question was refused on 35 to 55% of attempts, a 1k-token one almost never. That's a capacity problem that will fade. The design lesson won't: a search library has to plan its requests, not just send them. ## On a clean catalog, the best move is to ask once Here's what surprised me. The most accurate way to use Jev on 199 tools was the simplest: one question over the whole catalog, with two identical requests racing on every attempt so a refusal costs nothing. That scored 75.9% strict. Splitting the catalog into chunks cost about 2 points. A BM25 shortlist first cost about 5. And about two thirds of the remaining misses were defensible picks the dataset's single label called wrong, according to a blind judge that saw both tools in both orders. So the prototype runs a cascade. Ask once, hedged. If Jev refuses twice, split the catalog into parallel chunks and run a final round over the winners. If that fails too, return the BM25 order and say so. ```ts import { createIndex, search } from "jev-search"; const index = createIndex({ documents: tools, id: (tool) => tool.name, describe: (tool) => summary(tool.description), // what the wide rounds read detail: (tool) => fullText(tool), // what the deciding question reads }); const { hits, plan } = await search(index, { state: `User request: ${request}`, limit: 5 }); // plan: "direct" | "tournament" | "fallback", every hit carries its probability ``` On MetaTool it scored 75.9%, level with the best hand-tuned call. Across 398 searches it threw zero errors, and fell back to BM25 twice. I thought I was done. ## On real MCP tools, the same defaults lost by 19 points Then I ran it on the benchmark from the last post: LiveMCPBench, 525 real tools from 69 MCP servers, 94 human-annotated tasks. The comparison was the search that post shipped, a port of FastMCP's two-stage Jev search. The prototype lost badly. 39% against 59% for the right tool first, in the same run. It also made 45 requests per search. (That search scored 56% in the last post and 54 to 59% across my runs here. With 94 tasks, a few points either way is noise. Nineteen is not.) The cause was one assumption. MetaTool's descriptions are short: 88 characters at the median. Real MCP descriptions run from a few words to 3,800 characters. With full text in every option, a 1,600-token request held a dozen tools. The catalog shattered into dozens of chunks, and the final round compared lookalikes on the same thin text. FastMCP had already solved this, and I had copied the wrong half. Its wide pass reads a 160-character summary of each tool. Only the final round, over eight finalists, reads the full description and the parameters. Short text to narrow, rich text to decide. I gave the prototype that split. The gap closed to 2 points, inside the noise. Then I put Voyage embeddings in front, the setup the last post recommended: | LiveMCPBench, 525 tools, same run | Right tool first | Right tool in top 5 | Cost per 1,000 searches | |---|---|---|---| | FastMCP-style Jev search (last post) | 54% | 83% | ~$1.03 | | Embeddings top 100, then the prototype | 59% | 92% | $0.26 | The first-tool gap is inside the noise. The top-5 gain is not: +8.5 points, and the top five is what the agent actually sees. No search I measured on this benchmark did better there, including Voyage's reranker at 86%. It costs a quarter as much, and Jev's share of the latency is 0.64 s at the median. ## The defaults are the product, so ship the eval with them The prototype didn't get better because the algorithm got smarter. It got better because I measured it on a second catalog. Full descriptions beat short ones by 2 points on MetaTool and lost by 7 on MCP. A 1,600-token budget that was right on MetaTool shattered MCP catalogs. The same code, tuned on one catalog, would have shipped a 19-point regression to everyone with real tools. Orama can ship defaults once, because BM25 behaves the same on every corpus. A Jev search can't. The defaults depend on your description lengths, your catalog size, and how busy the provider is this week. > **Ship the harness with the engine** > > A Jev search library is a planner plus the eval that tunes it. The engine without the harness is a guess tuned on someone else's catalog. So the plan for the library is short: - **The planner:** summaries to narrow, details to decide, a cascade that degrades instead of failing. - **The harness:** a first-class export that measures hit@1, top 5, capacity errors, latency and cost on your catalog, and picks the settings for you. The benchmark and the prototype are open source at [kachar/jev-tool-search](https://github.com/kachar/jev-tool-search). Measure your own catalog before you trust my defaults. I wrote more about why owned evals matter in [your evals are the moat](https://kachar.dev/blog/your-evals-are-the-moat). This is the smallest case of it I've found. Orama made BM25 boring. The Orama for Jev won't make Jev boring. It will make it measured. --- # Picking the right tool for an LLM is a decision. Jev makes it. > Claude's tool search ranks with BM25. On 525 real MCP tools, Jev put the right tool first 56% of the time to BM25's 32%, and it could say none of them fit. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm Markdown: https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm.md Published: 2026-09-26 Reading time: ~11 min Tags: ai, agents, context-engineering Cite as: Ilko Kacharov, "Picking the right tool for an LLM is a decision. Jev makes it.", kachar.dev, September 26, 2026. https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm ## Contents 1. [The short version](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#the-short-version) 2. [Claude already searches for its tools, with BM25](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#claude-already-searches-for-its-tools-with-bm25) 3. [Jev turns tool search into a multiple-choice question](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#jev-turns-tool-search-into-a-multiple-choice-question) 4. [How I tested it](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#how-i-tested-it) 5. [Result 1: Jev picks the right tool far more often than BM25](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-1-jev-picks-the-right-tool-far-more-often-than-bm25) 6. [Result 2: BM25 plus Jev is worse than Jev alone](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-2-bm25-plus-jev-is-worse-than-jev-alone) 7. [Result 3: With Claude in the loop, Jev's win is the bill](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-3-with-claude-in-the-loop-jevs-win-is-the-bill) 8. [Result 4: More tools make every search worse, and Jev slower](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-4-more-tools-make-every-search-worse-and-jev-slower) 9. [Result 5: Only Jev can say "none of these tools fit"](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-5-only-jev-can-say-none-of-these-tools-fit) 10. [Why 56% is lower than it sounds](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#why-56-is-lower-than-it-sounds) 11. [What I would ship](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#what-i-would-ship) --- ![A wall of hundreds of identical dark steel card-catalog drawers receding into black. Three are pulled half open; one of them glows electric violet from inside and lights the drawers around it.](https://kachar.dev/posts/tool-search-drawers-hero.jpg) One of our agents once told a user it could not create a campaign. The tool was there. It was in the search results, just not near the top, and the agent went with what was near the top. Nothing crashed. The search did what it was built to do: it counted the words the request shared with each tool's description, and other tools shared more of them. Finding tools is a search problem. Picking the right one is a decision. So I tested Jev, a model built to make decisions, against the search Claude uses today. ## The short version I gave every search the same 94 real tasks and 525 real MCP tools. The main score is simple: **is the first tool the search returns one the task actually needs?** | Search | Right tool first | Time per search | Cost per 1,000 searches | |---|---|---|---| | Jev search | 56% | 2.5 s | $1.03 | | Embeddings, then Jev on the top 100 | 52% | 1.2 s | $0.28 | | Voyage embeddings | 46% | 0.3 s | $0.01 | | BM25, then Jev on the top 100 | 40% | 0.3 s | $0.12 | | BM25 (what Claude's tool search uses) | 32% | under 1 ms | $0 | What that means: - **Jev picks better than BM25.** The 24-point gap is not noise. - **Putting BM25 in front of Jev makes Jev worse.** Putting embeddings in front keeps most of its accuracy at a quarter of the cost. - **With Claude in the loop, accuracy evens out, but Jev cuts the bill by about 60%.** Claude makes up for a bad search by searching again, and every extra search costs tokens. - **Jev is the only one that can say "none of these tools fit".** - **The catch:** in Jev's second week on the gateway, many calls were refused because the provider was at capacity. That made it slow. The rest of this post is how I got there. ## Claude already searches for its tools, with BM25 Loading every tool into the prompt stops working fast. Anthropic's [tool search docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) say a typical five-server setup "can consume ~55k tokens in definitions before Claude does any work," and that Claude's ability to pick the right tool "degrades" once you pass 30 to 50 of them. The fix is deferred loading. You mark tools `defer_loading: true`, and Claude searches for the ones it needs. Anthropic ships two engines for that search, regex and BM25. BM25 is a classic keyword ranking: the tools that share the most words with the query win. That works until the words don't match. StackOne ran BM25 over 270 tools and got the right tool first only [14% of the time](https://www.stackone.com/blog/mcp-tool-search-bm25-tfidf-hybrid/). The same docs describe a way out: a custom search. Claude calls your own search tool, you decide which tools fit, and you answer with `tool_reference` blocks. Claude then loads those tools as if its own search had found them. ## Jev turns tool search into a multiple-choice question [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) is TypeSafe AI's decision model, available on Vercel's AI Gateway. It does not write text. You give it a situation and a question with named options, and it returns a probability for each option. The answer is always one of your options, so it cannot invent a tool name. As AJ Awan put it: "Jev is not a smaller LLM. It is the [router, gatekeeper and scorer](https://flowtivity.ai/blog/jev-use-cases-vs-text-output-llms/) sitting in front of one." For tool search, the situation is the user's request and the options are your tools. It costs $0.042 per million input tokens, and output is free. Here is the whole custom search tool: ```ts import { experimental_evaluate } from "ai"; // `catalog` is exactly what you sent as `tools`, at most 255 entries. export async function findTools(toolUse: { id: string; input: { query: string } }, request: string) { const { answers } = await experimental_evaluate({ model: "typesafe-ai/jev", state: { request, searchedFor: toolUse.input.query }, questions: { which: { type: "choice", instructions: "Which tool should the assistant call for this request?", criteria: Object.fromEntries(catalog.map((t) => [t.name, t.description])), }, }, abortSignal: AbortSignal.timeout(4000), maxRetries: 0, }); // `probabilities` is optional; `choice` is always there. const probabilities = answers.which.probabilities ?? { [answers.which.choice]: 1 }; const best = Object.entries(probabilities).sort(([, a], [, b]) => b - a).slice(0, 5); return { type: "tool_result" as const, tool_use_id: toolUse.id, content: best.map(([name]) => ({ type: "tool_reference" as const, tool_name: name })), }; } ``` Because it runs on your side, your search can read the whole conversation, not just Claude's search words. In production, fall back to plain search when the call times out. Also keep tool names identical to your `tools` array. One question takes at most 255 options. For bigger catalogs I used the two-step approach from FastMCP's experimental [JevSearchTransform](https://github.com/PrefectHQ/fastmcp/pull/5170): Jev first narrows chunks of 150 tools down to their top eight, then reads the finalists in full. It also asks one yes/no per finalist: does this tool actually do what was asked? Tools that score under 0.3 are dropped. That question matters again in Result 5. ## How I tested it - **Data:** [LiveMCPBench](https://github.com/icip-cas/LiveMCPBench), 525 real tools from 69 real MCP servers and 94 human-annotated tasks, such as "Generate a well-formatted PDF report titled wechat_reading_report.pdf in /root/pdf, summarizing current WeChat Reading trends and including a word cloud." - **Test 1, search only:** is the first result a tool the task needs? - **Test 2, end to end:** Claude Sonnet 4.5 with all 525 tools deferred, searching until it calls a tool. Is that first call a tool the task needs? - **Every bar** in the charts shows a 95% confidence interval. With 94 tasks, differences under about 10 points are within the noise. - **Code and data:** the benchmark, the datasets and every results table are open source at [kachar/jev-tool-search](https://github.com/kachar/jev-tool-search). ## Result 1: Jev picks the right tool far more often than BM25 > [Figure: Tool search scoreboard, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-1-jev-picks-the-right-tool-far-more-often-than-bm25) Jev search put a right tool first 56% of the time. BM25 managed 32%. Embeddings landed in between. The one search that matched Jev was Voyage's reranker, at 60%. That is a tie on 94 tasks. Jev's edge is elsewhere: it cost less than half as much ($1.03 against $2.23 per thousand searches), and it can say no tool fits, which a reranker cannot. One more thing surprised me. BM25 only reaches 32% because Claude writes it a clean keyword query first. Given the user's own words, it falls to 11%. BM25 works best when something smarter has already done the thinking. ## Result 2: BM25 plus Jev is worse than Jev alone The obvious idea is to combine them: let BM25 shortlist tools in under a millisecond, then let Jev pick. I tried shortlists from 20 to 200 tools. > [Figure: Combined search, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-2-bm25-plus-jev-is-worse-than-jev-alone) Every BM25 shortlist landed at 38 to 40%, a little above BM25 alone and well below Jev alone. Here is why. BM25's shortlist contains a right tool for only 72% of tasks, however long you make it. For the other 28%, the right tools share no words with the request, and those are exactly the tasks where Jev shines. The shortlist throws them away before Jev sees them. An embeddings shortlist does not have that problem. Its top 100 contains a right tool for 96% of tasks. Jev on that shortlist scored 52%. That is four points under Jev alone, at twice the speed and a quarter of the cost. At 2,000 tools it beat Jev alone. If you shortlist, shortlist by meaning, not by words. ## Result 3: With Claude in the loop, Jev's win is the bill > [Figure: Claude end to end, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-3-with-claude-in-the-loop-jevs-win-is-the-bill) With Claude doing the work, every setup landed between 52% and 57%, inside the noise. Claude covers for a bad search: when the tools it finds look wrong, it searches again. With the built-in BM25 search it searched 3.1 times per request. With Jev, about twice. Those extra searches show up in the bill. With the built-in search, Claude read 11.7k input tokens per request. With Jev it read 3.4k, and the total cost dropped from $42.54 to $16.92 per thousand requests, Jev included. Loading all 525 tools up front scored highest on its 30-task subset: 67% against 63% for the built-in search. It cost about eight times as much. A small note for anyone reproducing this: I ran Claude on Vertex AI, because AI Gateway's Anthropic endpoint accepted the tool-search tool and then silently ignored it. Here are real tasks where the two searches led Claude to different first tools: > [Figure: Bm 25 misses, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-3-with-claude-in-the-loop-jevs-win-is-the-bill) ## Result 4: More tools make every search worse, and Jev slower I grew the catalog to 1,000 and then 2,000 tools, using real tools from live MCP servers from the [Neuronto index](https://huggingface.co/datasets/AgenticResourceDiscovery/verified-mcp-tools) (CC-BY-4.0) as decoys. > [Figure: Jev at scale, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-4-more-tools-make-every-search-worse-and-jev-slower) Jev dropped from 56% to 41%, BM25 from 32% to 28%. Jev stayed ahead of BM25. At 2,000 tools it only tied embeddings, while embeddings then Jev held on at 44%. Speed was the real problem. A 2,000-tool Jev search makes sixteen calls, and in Jev's second week on the gateway many were refused with HTTP 429, "provider at capacity". Searches waited and retried. That took 2.5 seconds per search at 525 tools and 7 seconds at 2,000. At 2,000 tools, 8 of 94 searches failed outright. This should improve as capacity grows. Until then, put a timeout and a fallback around it. I dug into why in [the follow-up](https://kachar.dev/blog/an-orama-for-jev): the refusals grow with request size, so a search has to plan its requests, not just retry them. ## Result 5: Only Jev can say "none of these tools fit" Sometimes the right tool is not in the catalog. BM25 still returns something, because some word always matches. Embeddings always return something, because some tool is always closest. Then the agent calls the least-wrong one. I re-ran every task with the servers that could do it removed. Now the right answer is to return nothing. > [Figure: No tool fits, drawn on the page](https://kachar.dev/blog/jev-picks-the-right-tool-for-an-llm#result-5-only-jev-can-say-none-of-these-tools-fit) Jev search returned nothing on 29% of those tasks, BM25 on 7%, embeddings never. That is the yes/no fit question at work. My test is harsh, since other servers can often do part of the job. On a cleaner test, FastMCP saw Jev stay quiet on 54 of 60 unanswerable requests, where BM25 answered 59. ## Why 56% is lower than it sounds A score of 56% sounds bad. Most of the shortfall comes from how I scored it, not from the search. Same results, three ways to count: | Counted as right when | Jev search | Voyage rerank | BM25 | |---|---|---|---| | The first tool is exactly one the task lists | 56% | 60% | 32% | | The first tool is on a server the task needs | 67% | 78% | 48% | | Any listed tool is in the top five | 82% | 86% | 47% | The tasks need about three tools each, and the labels allow only one path. A valid alternative, like DuckDuckGo instead of Hacker News for "today's LLM news", counts as a miss. And Claude sees the top five, not just the first result. Every search faces the same strict labels, so the comparison is fair even when the absolute numbers look low. ## What I would ship | Catalog | What I would use | |---|---| | Under 30 tools | Load them all | | 30 to 255 tools | One Jev question over the whole catalog | | 255 tools and up | Embeddings top 100, then Jev search | | Tight latency budget | Embeddings alone, or BM25 then a reranker | Skip Jev if you have fewer than 30 tools, if your tool names are cleanly namespaced like `github_` and `slack_` (regex search is free and good enough), if you need answers in under a second, or if requests must stay inside your own cloud. > **Shortlist by meaning, decide with Jev** > > Let embeddings pick the top 100 tools and let Jev decide among them. Return its picks as `tool_reference` blocks, and fall back to plain search when Jev times out or is refused. Two habits help whatever you choose. Write tool descriptions like rules, saying what the tool is for and what it is not for, because Jev reads each one as a criterion. And measure how often the first tool is right on your own catalog. As Anthropic's speed team put it: "Once Claude can measure something, it can make it faster." The same goes for your evals: the [ones you own](https://kachar.dev/blog/your-evals-are-the-moat) are the ones that tell you the truth. The tool that ranked too low was the right answer all along. It needed something that could decide, not something that could count. --- # Your AI hub can't do the 70% > BCG puts 70% of AI's value in ways of working. That is the one thing a central hub structurally cannot do for you, and nobody ever measured the number. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent Markdown: https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent.md Published: 2026-09-23 Reading time: ~10 min Tags: ai, cto, architecture Cite as: Ilko Kacharov, "Your AI hub can't do the 70%", kachar.dev, September 23, 2026. https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent ## Contents 1. [The bureaucracy objection is the weak one](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-bureaucracy-objection-is-the-weak-one) 2. [The 70% has been a slogan for seven years](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-70-has-been-a-slogan-for-seven-years) 3. [The report everyone cites to justify a hub concludes the opposite](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-report-everyone-cites-to-justify-a-hub-concludes-the-opposite) 4. [Almost nobody hands the hub the authority the definition demands](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#almost-nobody-hands-the-hub-the-authority-the-definition-demands) 5. [The variable that moves EBIT is the one only the workflow owner can touch](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-variable-that-moves-ebit-is-the-one-only-the-workflow-owner-can-touch) 6. [We already ran this experiment on data teams and published the results](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#we-already-ran-this-experiment-on-data-teams-and-published-the-results) 7. [Nothing in this argument, mine included, has been measured](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#nothing-in-this-argument-mine-included-has-been-measured) 8. [Centralize the substrate, decentralize the loop](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#centralize-the-substrate-decentralize-the-loop) --- ![A huge dark brushed-metal hub with dozens of steel spokes radiating outward into black. Every spoke is unlit except one, where a small workbench at the far outer edge glows electric violet and throws its light back along the spoke toward a centre that stays dark.](https://kachar.dev/posts/ai-hub-spoke-hero.jpg) Seventy percent of AI's value comes from changing how people work. That is BCG's number, published last month, and it is the strongest argument against the thing BCG is selling. The piece is [Why Your Organization Needs an AI Hub](https://www.bcg.com/publications/2026/why-companies-need-centralized-ai-hub), 25 August 2026. It is a good piece, and I want to be fair to it. An AI hub, in its definition, is "a dedicated entity with the mandate to coordinate all AI initiatives throughout the organization," anywhere from 10 people to more than 100, holding "explicit authority to oversee all of the company's AI efforts." Coherence, speed, governance, adoption. Four phases running from hub-led to AI-first. If you are a CIO looking at forty disconnected pilots and no way to tell which ones are real, you will read it and feel seen. Then you reach the arithmetic and the article argues against itself. ## The bureaucracy objection is the weak one The reflex is to say an AI hub is a center of excellence in a new lanyard. Scott Sambucci gave that reflex its best line in September 2025: "Committees are where AI goes to die." He was describing institutions that were still standing up an AI Center of Excellence nearly three years after ChatGPT shipped. It is nearly four now. Fair objection. Wrong one. Centralization is not the failure mode. Plenty of things belong in exactly one place, and I would fight to put them there. One identity model, so an agent inherits a real person's permissions instead of a service account with god rights. One audit trail. One eval harness, because [your evals are the moat](https://kachar.dev/blog/your-evals-are-the-moat) and nobody builds a second one well. One procurement path, so four teams do not sign four contracts with the same vendor at four prices. Every team that builds those alone builds them badly and alone. The problem with the hub is not that it centralizes. It is that BCG tells you precisely where the value is, and it is nowhere the hub can reach. ## The 70% has been a slogan for seven years Here is the sentence the article rests on: "In BCG's experience, just 10% of the value from AI comes from algorithms and 20% from data, while the remaining 70% comes from changes to the operating model and new ways of working." Ten, twenty, seventy. BCG has been publishing those three digits since at least March 2019. The digits have never moved. Everything around them has. > [Figure: Ten twenty seventy, drawn on the page](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-70-has-been-a-slogan-for-seven-years) In 2019 it was "the 10-20-70 problem of machine learning." By November 2020 BCG's own account was posting that companies "must apply the 10, 20, 70 rule." A problem is something you observe. A rule is something you follow. Nobody announced the promotion. Chase the provenance and it loops. The one place BCG points at a source is an April 2020 article: "According to joint BCG and MIT research, a good rule of thumb is to dedicate around 10% of AI investment to algorithms, 20% to technology, and 70% to business process transformation." The words "a good rule of thumb" are a hyperlink. Follow it and you land on BCG's October 2019 report drawn from the MIT Sloan Management Review survey of more than 2,500 executives, where the sentence reads: "A good rule of thumb is to consider AI to be 10% about algorithms, 20% about technology, and 70% about business process transformation." Same rule of thumb, sitting in a paragraph of advice rather than in the findings. The citation is a mirror. Be precise about what that does and does not mean. It does not mean the 70% is wrong. I think it is directionally right, and everything below depends on it being right. It means nobody has measured it, and a number nobody measured is a strange foundation for a hundred-person org. ## The report everyone cites to justify a hub concludes the opposite Every enterprise AI deck of the last year carries the MIT number. Ninety-five percent. It comes from MIT Project NANDA's "The GenAI Divide," July 2025, and the real claim is narrower than the meme: 95% of organizations getting zero measurable return, not 95% of pilots failing. Read to the conclusion. Organizations that cross the divide "buy rather than build, empower line managers rather than central labs, and select tools that integrate deeply while adapting over time." Line managers rather than central labs. Section 6.2 is more specific still: the winners "sourced AI initiatives from frontline managers, not central labs," instead of "relying on a centralized AI function to identify use cases." Section 6.1 names the barrier outright. Not integration, not budget. Organizational design. Companies succeed when they "decentralize implementation authority but retain accountability." The statistic being used to justify the hub comes from the document telling you not to build one. That does not settle it, and I will not pretend it does. NANDA is 52 organizations and 153 leaders recruited at four conferences, self-reported, version 0.1, and its own authors call the figures directional. There is a better detail buried in the appendix. The interview script asked "Who leads implementation efforts (e.g., IT, line managers, AI CoE)?" and the report never publishes the answer distribution. "Center of Excellence" appears exactly once in twenty-six pages, in that question. They collected the data that would settle the argument and printed the conclusion instead. ## Almost nobody hands the hub the authority the definition demands BCG is unambiguous that a hub must hold "explicit authority to oversee all of the company's AI efforts." Heidrick & Struggles asked 242 organizations who actually owns AI strategy. > [Figure: Who owns the agenda, drawn on the page](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#almost-nobody-hands-the-hub-the-authority-the-definition-demands) Seven percent. One point above the CEO, and the CEO is not doing this full time. The mandate sits with the CIO, the CTO, the CDAO, whoever already had the platform and the headcount, which is roughly where it sat before anyone said "hub." In the same research, nearly half of the AI leaders surveyed said their organization "has simply reclassified existing positions to include AI responsibilities." That is the honest description of most AI hubs. A renamed team, an inherited budget, and a mandate to coordinate people who do not report to them. ## The variable that moves EBIT is the one only the workflow owner can touch McKinsey tested 25 organizational attributes against EBIT impact. The winner was not structure, sponsorship or centralization. It was workflow redesign: "the redesign of workflows has the biggest effect on an organization's ability to see EBIT impact from its use of gen AI." The 2026 follow-up has the number that should stop a boardroom. AI high performers "remained flat at about 6 percent of all respondents." EBIT attribution sat at 37%, which McKinsey calls "essentially unchanged from 2025" despite growth in the share of organizations scaling AI. And in the same survey: "Eighty percent of respondents report that AI has improved their individual productivity." > [Figure: The adoption gap, drawn on the page](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#the-variable-that-moves-ebit-is-the-one-only-the-workflow-owner-can-touch) Eighty percent of people are faster. Six percent of companies can find it in the P&L. Two years of intensive organizational scaffolding moved that number by zero. That gap does not live in an org chart. It lives in the fifty small decisions inside one team's week. Which review step is now redundant. Which handoff existed only because a human needed a queue. Which weekly report nobody has opened since 2023. Robert Glaser said it better than I can: "the adoption unit is no longer the organization, and maybe not even the team. It is the loop inside the work." You cannot coordinate your way into that. A central body cannot know which of your handoffs is vestigial. The person doing the handoff barely knows. > **The hub test** > > Centralize what every team would otherwise build badly and separately. Decentralize what only the person doing the work can see. If your hub is doing the second thing, it is not a hub. It is a queue. ## We already ran this experiment on data teams and published the results Here is the part that unsettled me. The best post-mortems on centralized technical teams are six to eleven years old, they were written about data science rather than AI, and they all say the same thing. If you are standing up a hub in 2026, they were written about you. Riley Newman, Airbnb's first data scientist, in 2015: the centralized model was tempting, and then "we became viewed as a resource and, as a result, our work became reactive." Eric Colson at Stitch Fix, in 2019, on what a central team sounds like from the inside: "waiting on ETL changes" and "waiting on ML Eng resources" are the visible symptom, and "the more insidious impact lies in what you don't hear, because you can't lament what you haven't yet learned." Monzo spent three years dissolving its central ML team and landed on a sentence that reads like an epitaph for the category: "machine learning is an established tool that is used by many disciplines to help teams reach their goals." I went looking for the 2026 equivalent, written about an AI hub instead of a data team, and could not find one. Not because hubs are working. Because none of them is old enough yet. Newman is also the reason not to over-read any of this. Airbnb decentralized and deliberately stopped short: "by not fully shifting toward an embedded model we're able to maintain a vantage point over every piece of the business." He kept the hub. He stopped putting the work in it. ## Nothing in this argument, mine included, has been measured I have just spent a thousand words using statistics to argue against an article built on statistics. So here is the ledger. > [Figure: Evidence ledger, drawn on the page](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#nothing-in-this-argument-mine-included-has-been-measured) Not one measured financial outcome in the set. Every number in this debate is an executive describing their own program, and the taxonomy the debate runs on, centralized against federated against hub and spoke, has never been tested against an outcome by anyone I can find. That absence is the most interesting fact in the whole file, and it should make everybody in the argument quieter, me included. ## Centralize the substrate, decentralize the loop So build the hub. Just be honest about which number it is chasing. The 30% is real and it is genuinely centralizable. Identity, permissions, data access, model access, evals, audit, procurement, the boring plumbing that every team needs and no team wants to own. Build that once, well, and put your scarcest people on it. That is [architecture, and it is worth your innovation tokens](https://kachar.dev/blog/three-innovation-tokens). The 70% is a workflow problem, and workflows belong to the people inside them. What a hub can do there is narrow and unglamorous: make it cheap for a team to try something, cheap to measure whether it worked, and safe to delete the step that stopped mattering. Not run the project. Lower the cost of the team running it. BCG's own maturity model already says this, though not in those words. Read the four phases end to end and it is not a maturity model. It is a shrink schedule. > [Figure: Hub phases, drawn on the page](https://kachar.dev/blog/your-ai-hub-cant-do-the-70-percent#centralize-the-substrate-decentralize-the-loop) BCG's three founding questions are who leads the hub, what the biggest stumbling block is, and where to start. Ask a fourth one, on day one, before anyone is hired. What does this hub stop doing when it works? A hub that cannot answer that is not a hub. It is a department. Departments do not dissolve. They budget. --- # The lights were never off > The man who coined "dark factory" runs at Level 4. The man who popularized it calls it unproven. The teams who built one never say the words. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/the-lights-were-never-off Markdown: https://kachar.dev/blog/the-lights-were-never-off.md Published: 2026-07-29 Reading time: ~15 min Tags: ai, agents, architecture Cite as: Ilko Kacharov, "The lights were never off", kachar.dev, July 29, 2026. https://kachar.dev/blog/the-lights-were-never-off ## Contents 1. [The label and the primary source disagree, every time](https://kachar.dev/blog/the-lights-were-never-off#the-label-and-the-primary-source-disagree-every-time) 2. [Review is the ceiling, and that is the entire argument](https://kachar.dev/blog/the-lights-were-never-off#review-is-the-ceiling-and-that-is-the-entire-argument) 3. [The bill for not reading arrives in simulation](https://kachar.dev/blog/the-lights-were-never-off#the-bill-for-not-reading-arrives-in-simulation) 4. [In 1981 the robots built the robots and the humans did final assembly](https://kachar.dev/blog/the-lights-were-never-off#in-1981-the-robots-built-the-robots-and-the-humans-did-final-assembly) 5. [The lights are going off in factories nobody meant to build](https://kachar.dev/blog/the-lights-were-never-off#the-lights-are-going-off-in-factories-nobody-meant-to-build) 6. [The label is already leaving software, and it is still shedding caveats](https://kachar.dev/blog/the-lights-were-never-off#the-label-is-already-leaving-software-and-it-is-still-shedding-caveats) 7. [Three things wear this name, and conflating them is how teams get hurt](https://kachar.dev/blog/the-lights-were-never-off#three-things-wear-this-name-and-conflating-them-is-how-teams-get-hurt) --- ![A vast unlit factory floor of dark brushed metal, machines still, with a single workstation glowing electric violet where a human is finishing an assembly by hand.](https://kachar.dev/posts/final-assembly-hero.jpg) The man who introduced "dark factory" to software does not operate one. Dan Shapiro published [The Five Levels](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) on 23 January 2026. He borrowed the term from FANUC and modelled his ladder on the NHTSA autonomy levels. Level 5 is the Dark Factory: "a black box that turns specs into software," a place where humans are "neither needed nor welcome." Asked which rung he is on, Shapiro answers in two words. "I'm here." He is at Level 4. He knows "a handful of people" at Level 5, in "small teams, less than five people," and calls the whole thing "nearly unbelievable." Six months later the phrase is a consulting product, a 1,422-star Rust project, and roughly three quarters of the search results for its own name. > [Figure: Autonomy ladder, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off) So the skeptic has a fair question ready, and it is the wrong one. The question is not whether dark factories are hype. It is why every single person who actually built one describes something quieter than the label they got handed. ## The label and the primary source disagree, every time I went looking for the strongest version of the claim and found a pattern instead. > [Figure: Label vs practice, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-label-and-the-primary-source-disagree-every-time) Latent Space titled its Ryan Lopopolo interview "1M LOC, 1B toks/day, 0% human code or review." [OpenAI's own article](https://openai.com/index/harness-engineering/), published 11 February 2026, says something else: "Humans may review pull requests, but aren't required to." On the podcast Lopopolo puts it plainly. "Most of the human review is post-merge at this point." Review moved. It did not vanish. The billion tokens a day is the host's line, and it appears nowhere in OpenAI's write-up. Vincent Koc's conference talk is called "Dark Factory." He closes it on "agent in the loop," a deliberate inversion, and says his binding constraint is his own attention. [Fabro](https://github.com/fabro-sh/fabro) calls itself "the open source dark software factory for expert engineers" and sells you a "middle path" whose headline feature is human approval gates. [Josh Rosen](https://x.com/JoshARosen/status/2077054762269257750), writing on 14 July 2026 to an audience of four likes, opens with the sentence the rest of the genre skips: "No one has fully built this yet." And the two people with the best claim to having built one, StrongDM's Justin McCarthy and OpenAI's Ryan Lopopolo, never use the phrase at all. They say "software factory," "non-interactive development," "harness engineering." The label is applied downstream, by commentators, to work whose owners describe it more carefully. Then there is the star witness. Every listicle cites Stripe's Minions as proof. [Stripe's own engineers](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2) wrote the disqualifying sentence themselves: "Over 1,300 Stripe pull requests merged each week are completely minion-produced, human-reviewed, but containing no human-written code." Stripe deleted the typing and kept the reading. Twice, in fact: the operator checks it, then a second engineer reviews it. > [Figure: The two rules, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-label-and-the-primary-source-disagree-every-time) Nobody lied here. Simon Willison reported a fascinating thing he saw, and then, [one day later on Hacker News](https://news.ycombinator.com/item?id=46805001), deflated it himself: "Yeah, the 'dark factory' thing is basically unproven right now... I'm not ready to say that it's a good idea or that it's demonstrated to work outside of a demo I saw for an hour a couple of months ago." Asked to name one public repo built that way, [he couldn't](https://news.ycombinator.com/item?id=46804801). "I haven't been brave enough to try it myself yet." In his own blog he balks at the bill: "If these patterns really do add $20,000/month per engineer to your budget they're far less interesting to me." That is a careful person being honest in public, twice. The discourse laundered the honesty out and kept the headline. And it did it fast. Every primary source in this story lands inside a nineteen-day window in January and February. Everything after that is retelling. > [Figure: Term timeline, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-label-and-the-primary-source-disagree-every-time) ## Review is the ceiling, and that is the entire argument Strip the branding away and something real is underneath. It is not "the models got good." It is an arithmetic problem, and five organisations with no shared employer state it in almost the same words. > [Figure: Review is the ceiling, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#review-is-the-ceiling-and-that-is-the-entire-argument) McCarthy's version is the cleanest, from his [StrongDM talk](https://www.youtube.com/watch?v=XBdWau6OQQ0): they committed to writing no code by hand, and "the first thing that we realized almost immediately after committing to this was, if you didn't write it there's just no way to review it without becoming the bottleneck." So they extended the rule. Not writing became not reading. It became "religiously important to not review the code and not read the code." Lopopolo arrives at the identical place from the other end. "The model is trivially parallelizable. As many GPUs and tokens as I am willing to spend, I can have capacity to work on the code base. The only fundamentally scarce thing is the synchronous human attention of my team." Stripe's blog says it in one clause: "one of our most constrained resources is developer attention." Agent throughput scales with money. Human attention does not scale with anything. Keep a human in the read path and you have capped the system at human reading speed, no matter how many agents you buy. That is the whole case, and it is a good one. > **The move nobody copies** > > They did not delete the reviewer. They deleted the reviewer and then spent a year rebuilding everything the reviewer was silently doing. The first half is a decision. The second half is the work. ## The bill for not reading arrives in simulation What the slogan costs is the part that never survives the retelling. StrongDM's own [account](https://factory.strongdm.ai/) reads like a series of failures, which is why it is worth trusting. Hands off the keyboard: "Not very far! At least: not very far, until we added tests." Then the agent learned to cheat. "`return true` is a great way to pass narrowly written tests." Then they hit the problem underneath, the one [Josh Rosen](https://x.com/JoshARosen/status/2077054762269257750) names precisely: an implementation and its verification can share the same mistaken assumption when both come from the same ambiguous spec. In StrongDM's words: "A test, stored in the codebase, can be lazily rewritten to match the code. The code could be rewritten to trivially pass the test." > [Figure: Circularity trap, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-bill-for-not-reading-arrives-in-simulation) Their fix is the most interesting idea in the field, and it is the thing I have been arguing for in [a different vocabulary](https://kachar.dev/blog/loop-engineering-is-verifier-engineering). They moved the verifier out of the agent's reach. Scenarios live *outside* the codebase, "similar to a holdout set in model training." Success stopped being boolean and became "satisfaction": of all observed trajectories through all scenarios, what fraction likely satisfy the user. And to run those scenarios they built the Digital Twin Universe, behavioural clones of Okta, Jira, Slack, Google Docs, Drive and Sheets, because "tests can be reward hacked" and Salesforce, as McCarthy notes, "won't let you start up 100,000 customer tenants overnight just because you want to or you ask nice." > [Figure: The substrate, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-bill-for-not-reading-arrives-in-simulation) The [Techniques page](https://factory.strongdm.ai/techniques) states the reframe plainly: "Code was treated analogously to an ML model snapshot: opaque weights whose correctness is inferred exclusively from externally observable behavior. Internal structure is treated as opaque." That is the actual intellectual content of the dark factory, and almost nobody quotes it. You do not read a neural network's weights. You evaluate its behaviour against a holdout and report a score. Once you treat code that way, the twins and the scenarios and the probabilistic metric are not clever extras. They are forced. And so is the danger, because we already know exactly how model evaluation fails: benchmarks get gamed, holdouts leak, proxy metrics drift, and nobody debugs a bad model by reading it. If code is a model now, then [your evals are the moat](https://kachar.dev/blog/your-evals-are-the-moat) stops being a slogan about AI features and starts being a statement about your source tree. The price of admission is published. StrongDM's benchmark, stated as a floor: "If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement." Willison did that multiplication and walked away at $20,000 per engineer per month, against the $200 plan he actually uses. Lopopolo's caveat is quieter and larger: "certainly helps that we have no rate limits internally." An OpenAI employee with free frontier tokens, building an internal tool. Every economic constraint you have, switched off. His own article says it: this "should not be assumed to generalize without similar investment." The honest ledger, then. StrongDM published about 32,000 lines from three people. OpenAI wrote on the order of a million in five months with zero hand-written code, an estimated tenth of the time it would have taken, and paid for it with a Friday. Their words: "Our team used to spend every Friday (20% of the week) cleaning up 'AI slop.' Unsurprisingly, that didn't scale." ## In 1981 the robots built the robots and the humans did final assembly The metaphor deserves the same scrutiny as the claims, because it does not survive it. Every "FANUC runs unattended for 30 days" citation traces to one place: a 350-word trend brief in *Business 2.0* called ["Fade To Black,"](https://web.archive.org/web/20180622003515/http://money.cnn.com/magazines/business2/business2_archive/2003/06/01/343371/index.htm) published 1 June 2003. It is twenty-three years old, and the thirty-day figure is attributed to nobody. The only named human is a FANUC vice president selling factory automation. Every retelling since is a copy of that paragraph. FANUC's own site claims something smaller: unmanned operation "for long hours, including nights and weekends." A FANUC America executive told *Assembly* in 2019 the real number is "up to 600 hours" and that "people are there for routine maintenance." That is 25 days, not 30. And then the poster child disowns the poster: > "We used to hear a lot about lights-out factories, but not as much anymore. It isn't practical for every application. The trend today is short product life cycles, lots of variation, mass customization and shorter production runs." Read that list again. Short life cycles, high variation, short runs. That is software. FANUC is describing the conditions under which lights-out does not pay, and they are our working conditions. > [Figure: Final assembly, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#in-1981-the-robots-built-the-robots-and-the-humans-did-final-assembly) Now go back to the founding legend. The *Christian Science Monitor*, [15 April 1981](https://www.csmonitor.com/1981/0415/041542.html), reporting from the Fuji plant: "For 16 hours of every working day, in fact, the factory is run entirely by robots." About 100 humans came in for the other eight. And then the sentence that should end this genre: > "Workers return to the factory next morning to complete the final assembly of the new generation of robots." The robots built the robots. The humans did final assembly. FANUC's president hoped to automate that last step by 1985. It is 2026. Software is running the same play with the same slogan. In 1981 the humans did final assembly. In 2026 they do the review. Tesla ran the loop faster than anyone and Musk [posted the receipt](https://x.com/elonmusk/status/984882630947753984) in April 2018: "Yes, excessive automation at Tesla was a mistake. To be precise, my mistake. Humans are underrated." ## The lights are going off in factories nobody meant to build The part that should worry you has nothing to do with StrongDM or OpenAI. [Faros AI](https://www.faros.ai/research/ai-acceleration-whiplash) instrumented 22,000 developers across 4,000 teams over two years, comparing each team's low-AI quarters against its high-AI quarters. Their March 2026 report is careful about what it is: statistically significant correlation, not causation, from a vendor, on an undisclosed company list. Take it with the salt it asks for. In that dataset, fewer than 1% of pull requests are opened autonomously by an agent. Essentially nobody is running a dark factory. Faros says so explicitly: the data "mostly reflects a world in which humans remain in the loop." And in that world, review is already collapsing. 31.3% more pull requests merge with no review at all. Median review time is five times longer. Incidents per pull request more than tripled. Pull requests are 51% bigger and touch 60% more files. Faros's own summary: "engineering throughput is up due to AI-generated code, bugs, incidents, and rework are rising even faster." > [Figure: Accidental dark, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-lights-are-going-off-in-factories-nobody-meant-to-build) Nobody voted for this. There was no charter document, no "hands off," no holdout scenarios, no digital twin of Okta. Teams did not decide to stop reading the code. They just stopped, because the volume arrived and attention did not. StrongDM and OpenAI removed the reviewer *and paid for the replacement*. Everyone else is getting the removal for free and discovering that the replacement is not free. The cruellest line in the report is the one for the good teams. Organisations with mature DevOps practices and strong pre-AI performance showed the same downstream deterioration as everyone else, which contradicts DORA's more comforting finding that AI amplifies existing strengths. Being good at engineering did not protect anyone. Faros's read is that telemetry catches operational stress that sentiment surveys miss. And if you are tempted to answer this with a number: the only controlled study we have, [METR's 2025 RCT](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), found experienced developers were 19% *slower* with AI while believing they were 20% faster. It is sixteen developers on year-old models, and it says nothing about autonomous pipelines. What it does say is narrower and more useful: every "3 to 5x productivity" figure in this genre is self-reported, and self-reports were off by 39 percentage points in the one case anyone bothered to measure. ## The label is already leaving software, and it is still shedding caveats While I was writing this, the term mutated twice more. It went sideways first. A consultancy explainer in March took Shapiro's ladder and mapped it onto [audit, banking and legal work](https://www.cow-shed.com/blog/dark-factories-five-levels-ai-automation-transform-audit-banking-legal), asking what a dark factory means for controls testing and due diligence. Then it went upward. On 9 June an engineer named Arnór Heiðar Sigurðsson published ["Dark Startup Factories"](https://arnorhs.dev/posts/2026-06-09/dark-startup-factories/), which takes the metaphor off the codebase and puts it around the entire company. His argument is the one about review being the ceiling, promoted a level. "Before you had a system where you had a **human-bandwidth constraint**, but with agentic startups, you are **compute-throughput constrained**." Then the same reasoning that took StrongDM from *nobody writes code* to *nobody reads code* keeps going: if agents do the building, you do not need people, and if you do not need people you do not need someone to motivate them. "But if there's no people to motivate, do you need founders? Then the question becomes 'why should VCs invest in founders?'" That is a real question, and he is honest about where it stands. His own section heading is "Are we there yet?" and his own answer is two words: "We aren't." He puts his finger on exactly the thing OpenAI pays for with a Friday: "Without a human in the loop, code tends to degrade to unmaintainable spaghetti." So that is now three careful people in a row. The coiner hedges. The reporter hedges. The man extending it to startups hedges. And look what still happens on the way down. > [Figure: Telephone drift, drawn on the page](https://kachar.dev/blog/the-lights-were-never-off#the-label-is-already-leaving-software-and-it-is-still-shedding-caveats) The consultancy piece is not slop. I expected it to be and it isn't. It has a references section, and the references are right: Shapiro's post, Willison's post, the METR trial, all cited accurately. It even explains the holdout properly, better than most: scenarios are "stored separately so it cannot optimise for passing them", and they verify "the same way a holdout set in machine learning verifies a model has genuinely learned rather than memorised." And in its opening line it says StrongDM is "a three-person company." StrongDM was founded in 2015. It sells enterprise access management, it has on the order of a hundred and seventy people, and it was acquired by Delinea. The three people are the *AI team*, founded inside it on 14 July 2025, by a co-founder who already had a company to put them in. That distinction is not trivia. It is the difference between "three people built a company with no code" and "an established security business gave three engineers room to run an experiment." One of those is a miracle. The other is a budget. Nobody lied. The article did its homework and cited its sources. It just compressed, and compression drops the qualifier, and the qualifier was the finding. One hop. ## Three things wear this name, and conflating them is how teams get hurt The dark factory as marketed does not exist. The coiner is at Level 4, the reporter says unproven, Rosen says nobody has built it, and even FANUC says it is uncommon and usually not worth it. The dark factory as engineered is real, and it is the most interesting systems work happening right now. It is also expensive, narrow, roughly a year old, and it keeps a human at the exact point where the stakes turn real. That is not my inference. Open the [Attractor spec](https://github.com/strongdm/attractor) from the team whose second rule is *code must not be reviewed by humans*, and read the section titles. Section 6 is "Human-in-the-Loop (Interviewer Pattern)." Its worked example is an option labelled `"Yes, deploy to production"`. The dark factory as it is actually arriving is an accident, and it is arriving at your company on the Faros numbers, not the StrongDM ones. So do not ask whether to turn the lights off. Ask which of those three you are already in. If your unreviewed merge rate is climbing and you have not built a holdout, a twin, or a blast radius, you are not running a dark factory. You are running an unlit one. Nobody turned the lights off. They rewired where the light falls. The places that never rewired anything are just going dark. --- # Write the done-condition, not the prompt > How I write prompts, audited against 138 of my own. Four habits, one rule, and the single edit that fixes most of them. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/write-the-done-condition-not-the-prompt Markdown: https://kachar.dev/blog/write-the-done-condition-not-the-prompt.md Published: 2026-07-16 Reading time: ~6 min Tags: ai, agents, context-engineering Cite as: Ilko Kacharov, "Write the done-condition, not the prompt", kachar.dev, July 16, 2026. https://kachar.dev/blog/write-the-done-condition-not-the-prompt ## Contents 1. [Start with /goal, because it makes you write the ending first](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#start-with-goal-because-it-makes-you-write-the-ending-first) 2. [Put the standing rules in CLAUDE.md, not in the goal](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#put-the-standing-rules-in-claudemd-not-in-the-goal) 3. [Say /workflow only when you can name the number](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#say-workflow-only-when-you-can-name-the-number) 4. [Reach for Chrome past a 403, and once for something better](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#reach-for-chrome-past-a-403-and-once-for-something-better) 5. [The contract fits in five lines](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#the-contract-fits-in-five-lines) --- ![A wall of dark brushed-metal gauges, every dial unlit and unread, with a single gauge glowing electric violet as it registers an actual number.](https://kachar.dev/posts/write-the-done-condition-not-the-prompt-hero.jpg) This is how I write prompts, checked against five weeks of actually doing it. 138 goals, eleven projects. I pulled the logs before writing so the advice would have to survive contact with my own habits, and mostly it didn't: counting every "make sure" and "verify" as a done-condition, 73% of them never said what done meant. Read strictly, 86%. So take the habits below as tested rather than recommended. Some survived. The rest need one edit, and it's the same edit every time. ## Start with /goal, because it makes you write the ending first `/goal` keeps an agent working until a condition holds. The real value sits upstream of the model: to write the condition you have to decide what finished looks like before you're invested in whatever the agent hands back. That's a thinking tool, not a prompting trick. [Post on X: The Claude Code team's own framing, 6 July 2026. An evaluator model checks your condition, so you're not the one deciding when it's done.](https://x.com/i/status/2074208949205881033) It rewards a numbered definition of done. Three of my 138 goals have one, and all three hold up, because "builds, lints, and type-checks" are commands. The machine decides. One caveat the docs are honest about, and it shapes everything below: the evaluator is a fresh model, but it cannot run your tests. It reads Claude's account of having run them. I [wrote three thousand words about why that matters](https://kachar.dev/blog/loop-engineering-is-verifier-engineering). ```mermaid sequenceDiagram participant You participant Claude participant Evaluator as Evaluator (fresh model) You->>Claude: /goal ... until X Claude->>Claude: does the work, runs the tests Claude-->>Evaluator: the transcript, its own account Note over Evaluator: independent context.
borrowed evidence. Evaluator-->>Claude: not yet, keep going ``` ## Put the standing rules in CLAUDE.md, not in the goal Here is what the audit changed my mind about. I assumed the 86% were sloppy. They aren't. My median session-opening goal is 190 characters and it still lands, because the gates are already standing somewhere else: "Always type-check and lint the code." "Never use `any` types." Those apply to all 138 goals without appearing in one of them. > [Figure: Condition migration, drawn on the page](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#put-the-standing-rules-in-claudemd-not-in-the-goal) The done-condition didn't disappear, it got factored out, the way you'd factor a constant out of a loop. Write the standing gates once, in persistent context, and let each goal carry only what couldn't have been written yesterday. The obvious objection dies on the data: session-openers can't inherit anything, and 85 of 100 of them still name no done-condition. ## Say /workflow only when you can name the number 26 of my goals reach for `/workflow`. Three pair it with a number, three hand it a list of links, and twenty name neither. It helps to know what you're actually asking for. A workflow is a script, and this is the shape of the one I reach for most: paste some links, read them the same way, check what they claim. ```js /agent(/ /pipeline(/ /parallel(/ /phase(/ /args/ /log(/ export const meta = { name: 'research-links', description: 'Read a list of links, check what they claim', phases: [{ title: 'Read' }, { title: 'Check' }], } phase('Read') const checked = await pipeline( args.links, link => agent(`Read ${link}. Quote every claim.`, { schema: CLAIMS }), claims => parallel(claims.map(c => () => agent(`Try to refute: ${c.text}`, { phase: 'Check', schema: VERDICT }))), ) log(`${checked.flat().length} claims checked`) return checked.flat().filter(v => !v.refuted) ``` Look at the highlighted names. I define none of them. `agent`, `pipeline`, `parallel`, `phase`, `args` and `log` are nowhere in the file, because the file isn't the program. It gets dropped into a runtime that supplies them, and that runtime will happily run `agent()` a hundred times over, concurrently, against links I never enumerated. It cuts the other way too. `Date.now()` and `Math.random()` are gone in here, because a workflow has to be resumable and those would make two runs disagree. The scope isn't yours in either direction. Which is the appeal, and the trap. The harness hands you fan-out for free. It does not hand you a stopping condition, and twenty of my 26 took the free part and skipped the rest. That's like hiring an inspector and never telling them what passes. The tool was never the point. The number was. Same shape in the newest toy. Anthropic's advisor tool pairs a fast executor with a smarter advisor, Fable 5 or Mythos 5, mid-task. As of mid-July 2026 it's an API beta rather than a slash command, and I haven't used it in anger, so take this as reading rather than experience. But watch the arrow. ```mermaid sequenceDiagram participant E as Executor (fast, cheap) participant A as Advisor (Fable 5 / Mythos 5) E->>A: here's the transcript, what should I do? A-->>E: a better plan Note over E,A: no tool is called.
the advisor advises.
it does not verify. ``` An advisor injects a better plan. It never tells you you're done. ## Reach for Chrome past a 403, and once for something better Nine of my goals mention Chrome. Most are scraping, because the good primary sources 403 an agent and a real browser walks through. That habit earns its reputation, and it is the boring one. The interesting one happened once: "test it out in chrome at `http://localhost:2800/chat` until it all works perfect". That is the only goal where I turned "looks right" into something a machine could look at. Then "works perfect" handed it straight back to the model. > [Figure: Until rot, drawn on the page](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#reach-for-chrome-past-a-403-and-once-for-something-better) Which is the rule underneath all of it. Of my 138 goals, four name a number, and exactly one of those four names a number that exists before the model has an opinion. > [Figure: One in 138, drawn on the page](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#reach-for-chrome-past-a-403-and-once-for-something-better) The one is "until we get 80% coverage". A test runner emits that figure whether or not anyone likes it. The other three only look like it: "more than 95% equality" has a model comparing screenshots, "98% or more for all" never says of what or counted by whom, and "over 80% according your criteria" names a number and hands over the ruler in three words. > **The question that separates them** > > Not "does the done-condition have a number in it?" but "who produces the number?" If the answer is the model, you don't have a gate. You have a mood. ## The contract fits in five lines ``` Objective: what becomes true Done-condition: until Grounded check: the command that produces X, and who runs it Boundary: what it may not touch Stop: the cap when X never arrives ``` Line three is the one everybody skips, including me, 137 times out of 138. It is the only line that asks a question you can get wrong. ```mermaid flowchart TD O["Objective: what becomes true"] --> D["Done-condition: until X"] D --> Q{"Who produces X?"} Q -->|"a command, a test, a tool"| G["Grounded. The loop can end honestly."] Q -->|"the model"| R["Rot. It ends when the model feels good."] ``` So here are six of mine run against it. Two hold up, and they are the two I would have described if you had asked me cold: a numbered definition of done, and Chrome as a way past a 403. The other four are one edit away. > [Figure: Six goals, drawn on the page](https://kachar.dev/blog/write-the-done-condition-not-the-prompt#the-contract-fits-in-five-lines) Five weeks, one person, eleven projects. Not a base rate, and half my `until` clauses are the same "fix findings until the review is happy" reflex copied around. But the direction is not subtle, and I'd bet your logs look like mine. Write the ending first. Then name who checks it. --- # Loop engineering is verifier engineering > Eleven articles, 43 million views, one idea: design loops that prompt your agents. They all agree you need a verifier. Almost none of them are one. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/loop-engineering-is-verifier-engineering Markdown: https://kachar.dev/blog/loop-engineering-is-verifier-engineering.md Published: 2026-07-14 Reading time: ~16 min Tags: ai, agents, architecture Cite as: Ilko Kacharov, "Loop engineering is verifier engineering", kachar.dev, July 14, 2026. https://kachar.dev/blog/loop-engineering-is-verifier-engineering ## Contents 1. [The loop is trivial. That is the most interesting thing about it.](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-is-trivial-that-is-the-most-interesting-thing-about-it) 2. [Everyone already crowned the verifier. It didn't help.](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#everyone-already-crowned-the-verifier-it-didnt-help) 3. [A verifier needs two independences. Yours probably has one.](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#a-verifier-needs-two-independences-yours-probably-has-one) 4. [Loops work exactly where checking is cheaper than doing](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#loops-work-exactly-where-checking-is-cheaper-than-doing) 5. [The loop discourse is a loop with no verifier](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-discourse-is-a-loop-with-no-verifier) 6. [You can loop the work. You cannot loop the verdict.](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#you-can-loop-the-work-you-cannot-loop-the-verdict) --- ![An endless loop of dark brushed metal running through a single glowing violet gate, the only source of light in the scene.](https://kachar.dev/posts/loop-gate-hero.jpg) An agent is a `while` loop. Six lines of Python. [Akshay Pachaar](https://x.com/akshay_pachaar/status/2069118430582866051) put the code in his article, a version of the [Daily Dose of DS piece](https://blog.dailydoseofds.com/p/loop-engineering-clearly-explained) by his co-founder Avi Chawla, and said the quiet part out loud: nobody is competing on the while statement. So when half your timeline announces that loop engineering is the skill of 2026, the skeptic has a fair question ready. Is this just a for-loop with a content strategy? [Post on X: Peter Steinberger, 7 June 2026. The tweet the whole genre is downstream of.](https://x.com/i/status/2063697162748260627) It's a fair question, and the honest answer is worse than the skeptic expects. The loop really is trivial. The pattern really is old: Anthropic documented evaluator-optimizer in December 2024, eighteen months before it went viral under a new name. And when [METR measured](https://metr.org/notes/2026-02-13-measuring-time-horizon-using-claude-code-and-codex/) whether elaborate loops beat a naive one, Claude Code beat plain ReAct in 50.7% of bootstrap samples. A coin flip. Codex lost to its baseline. Quadrupling the token budget from 8M to 32M "barely changed" the result. So the deflation is correct. The loop is not the product. But that is where the interesting question starts, not where it ends. Because every article in this genre agrees on the one word that carries the weight, and agreeing on it turns out to be worth almost nothing. Verify. ## The loop is trivial. That is the most interesting thing about it. METR's result is the sharp one: two of the most heavily engineered agent harnesses on earth could not beat `act, observe, repeat` to statistical significance. A [27-paper synthesis from Oxford](https://arxiv.org/abs/2607.05775), published a week ago, lands independently in the same place: "additional scaffolding does not consistently improve reliability." > [Figure: Scaffolding scoreboard, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-is-trivial-that-is-the-most-interesting-thing-about-it) Now hold that next to the thing that does move: grounded feedback. [Huang et al.](https://arxiv.org/abs/2310.01798) tested self-correction without external feedback and every cell went down or sideways. GPT-3.5 on CommonSenseQA fell from 75.8% to 38.1% after being asked to check its own work. That is a model talking itself out of correct answers, at three to five times the token cost. Their diagnosis is the load-bearing sentence in the literature: earlier papers showed self-correction working, but "the improvements in these studies result from using oracles to guide the self-correction process, and the improvements vanish when oracle labels are not available." > [Figure: Self correction chart, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-is-trivial-that-is-the-most-interesting-thing-about-it) Reflexion looks like the counterexample and is the confirmation. Its wins come from executed unit tests and binary environment signals, not introspection. On MBPP, same technique and same model, it went from 80.1% to 77.1%. Everyone quotes the 91% HumanEval number. Nobody quotes the regression sitting one row down in the same table. > **The line the evidence actually draws** > > It is not loop versus no loop. It is grounded feedback versus self-generated feedback. A loop closed over a test runner works. A loop closed over the model's opinion of its own work costs three to five times the tokens to get worse. > [Figure: Feedback loops, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-is-trivial-that-is-the-most-interesting-thing-about-it) One caveat, and it cuts at my own argument: Huang and Reflexion are 2023 papers on 2023 models, and nobody has published a clean replication on frontier reasoning models. The mechanism is echoed by 2026 work. The numbers are stale. As of mid-2026 that is the biggest hole in the anti-self-critique case, and anyone who tells you otherwise is quoting a three-year-old table at you like it's a law of physics. ## Everyone already crowned the verifier. It didn't help. Here is what surprised me when I read all eleven instead of skimming them. They don't bury the verifier. They crown it. [Anatoli Kopadze](https://x.com/AnatoliKopadze/status/2068328135611822149), whose piece has 16.4 million views, makes it the final block of his taxonomy and writes: "This is the one block that decides whether the loop helps you or just spends your money. Everything else is plumbing. This is the part that makes it real." Khairallah AL-Awady: "The verifier is the point of the whole thing." [Addy Osmani](https://x.com/addyosmani/status/2064127981161959567): the model that wrote the code is "way too nice grading its own homework." Akshay: "done should mean the tests pass, not the agent feeling good about its work." I had a whole section drafted about how the industry buries the verifier under plumbing. It's false. Kopadze got to my thesis eleven days before I did, in almost my words. So this is not a case of anyone missing the obvious. It is worse. It's universal agreement that changes nothing, and the proof is sitting inside the most-read article itself. Kopadze's four-box test gates the whole decision to build a loop on box two: something must be able to automatically reject bad output. Then his hands-on exercise, the thing he tells 16.4 million readers to paste into Claude, is a prompt that instructs the model to "score the result 1-10 on each criterion. Be brutally honest." That is the agent grading its own homework. It fails his own box two, one screen after he wrote it. He isn't being lazy. He is doing the thing the word lets you do. "Add a verifier" sounds like a design and is actually a noun. You can nod at it, crown it, put it in bold, and still ship self-grading, because nothing in the slogan tells you what a verifier has to be. ## A verifier needs two independences. Yours probably has one. Here is the distinction I could not find anywhere in 43 million views, and it falls straight out of the primary docs. **Independent context.** The judge is not the maker. A fresh model with its own window, uncontaminated by the reasoning that produced the work. Anthropic's [Prithvi Rajasekaran](https://www.anthropic.com/engineering/harness-design-long-running-apps) has the cleanest statement of why: "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work, even when, to a human observer, the quality is obviously mediocre." His fix is a lever, not a vibe: "Tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work." **Independent evidence.** The judge can go look for itself. It runs the tests. It reads the files. It inspects the artifact rather than the story told about the artifact. Those are not the same property, and almost every loop people ship has the first without the second. Take `/goal`, the most recommended primitive in this corpus, the one people cite as proof the maker/checker split is solved. Read [the docs](https://code.claude.com/docs/en/goal). The evaluator is a fresh model, defaulting to Haiku, which is real independence of context. And then: "It does not call tools, so it can only judge what Claude has already surfaced in the conversation." And: "It doesn't run commands or read files independently, so write the condition as something Claude's own output can demonstrate." Sit with that. Your verifier cannot run your tests. It reads Claude's account of having run your tests. `/goal all tests pass` works for a specific and fragile reason the docs are admirably honest about: Claude runs the tests, the output lands in the transcript, the evaluator reads the transcript. The gate is downstream of the maker's self-report. Independent context, borrowed evidence. That is a design constraint, not a bug, and Anthropic's docs are more honest about it than any article celebrating the feature. Managed Agents' grader is the other design: it inspects artifacts in a separate window, which the docs say exists "to avoid being influenced by the main agent's implementation choices." Both independences. And a test suite has both, holds no opinions, and costs nothing per run. > [Figure: Verifier matrix, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#a-verifier-needs-two-independences-yours-probably-has-one) The convergence is what convinced me. The loudest critics of multi-agent architecture are Cognition, who published ["Don't Build Multi-Agents"](https://cognition.com/blog/dont-build-multi-agents) in June 2025. In [April 2026 Walden Yan revised it](https://cognition.com/blog/multi-agents-working), and of the three patterns he now endorses, the one with numbers attached is a dedicated code-review agent: an average of two bugs per PR, roughly 58% of them severe. The detail that matters is that it works *better* when the reviewer and the coder do not share initial context. Anthropic ran it forward. Cognition ran it backward. Both landed on the same object. ## Loops work exactly where checking is cheaper than doing This is why code got loops first, and it isn't because programmers are early adopters. Where a check is cheap, objective, and mechanical, the loop compounds: tests pass or fail, the type checker is not negotiable, Lighthouse returns a number, the build compiles or it doesn't. Where "done" is a judgment call, the loop does not fail loudly. It produces confident, plausible, endless output and calls it finished. > [Figure: Verification asymmetry, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#loops-work-exactly-where-checking-is-cheaper-than-doing) Kopadze's four-box test is genuinely the best framework in the corpus. Two of its boxes are economics and capability: does the task repeat at least weekly, can the agent do the work end to end. The other two are the same box asked twice. Something must be able to automatically reject bad output, and "done" must be objective rather than a judgment call. Those are one requirement wearing two hats, and it is the only one that matters. Which brings me to the most interesting artifact I found. [Jason Zhou](https://x.com/jasonzhou1993/status/2067937943545897143) reports a production loop shipping "20-40 high-quality pages every day" of SEO content, "already driving traffic to the company without me looking at it." He presents this as the triumph. Read it again and ask the only question that matters: what rejects a bad page? Traffic does. Eventually. The verifier is a metric, the metric is the target, and Goodhart has been waiting at the end of that sentence the whole time. I'm not dunking on him, and the reason matters. His artifact system is the most concrete engineering in the corpus, his inner-loop/outer-loop split is a better taxonomy than most, and his sitemaps are consistent with the volume he claims. That is exactly why it's the perfect example. A serious engineer, running a real system that does what he says it does, with a verifier made of a number that can be satisfied without the work being good. The loop is not failing. That's the point. It will hit 20-40 pages a day forever, and nothing in it can ever tell him whether that was worth doing. The trap runs the other way too. When the verifier is the target, agents game it: delete the failing test, special-case the assert, satisfy the letter. A gate you never spot-check is a gate that rots. And yes, this is the honest hole in my own argument. Who verifies the verifier? Nothing here makes that go away. What it does is move the problem somewhere you can put your hands on it. A rotting test suite is a thing you can read, version, and fix on a Tuesday. "The model seemed confident" is not. ## The loop discourse is a loop with no verifier I read eleven articles about agent loops to write this. Roughly 43 million views between them. Then I did what none of them did to each other. I checked. > [Figure: Loop corpus chart, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-discourse-is-a-loop-with-no-verifier) The single most-viewed explainer, at 16.4 million views, turns into an ad for a Telegram bot at the line "for 99% of everyday ones, there is already a ready, dead-simple solution," and never turns back. I counted: everything from that line to the end is 31% of the body by word count. The link carries an attribution code, `t.me/mira?start=social_x_200626_howtostart`, which is how a partner counts arrivals. The piece never says what the relationship is. It out-viewed every first-hand account in the corpus combined. Two articles, with 24.8 million views between them, describe Geoffrey Huntley's "Ralph Wiggum loop" as a failure mode where an agent quits early and bills you in silence. Huntley invented Ralph on purpose, [advocates it](https://ghuntley.com/ralph/), and documented its real failure modes himself. It is `while :; do cat PROMPT.md | claude-code ; done`. He reported, in July 2025, an engineer clearing a $50k contract deliverable for $297 with it. > [Figure: Ralph propagation, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-discourse-is-a-loop-with-no-verifier) That error has an alibi: an AlphaSignal newsletter from June 8 describes Ralph the same wrong way, and both articles are downstream of it. The invented statistic they also share, a 50% accept-rate break-even, traces to no source at all. I can't tell you who copied whom. What I can tell you is that a wrong claim about how to build loops rode 24.8 million views without anything in the chain stopping it, and not one of the eleven describes Ralph as what its author says it is. The closest, at 1.9 million views, links Huntley's repo and gets his fresh-context pattern right, then lists "Ralph Wiggum loops" in its own table of anti-patterns. A digest of Addy Osmani's essay out-performed the essay by nearly four to one, and its author cites Addy throughout. This isn't plagiarism, it's distribution: a well-made summary beat the thing it summarized, and a month later someone shipped the same roadmap with the step count raised from fourteen to twenty. Even the best sources thin out under a check. The most rigorous post in the corpus, by an Anthropic engineer with a genuine track record in evaluation, has a headline of "~6x more improvement" on a metric where lower wins. Six times what? The post doesn't say, and I couldn't reconstruct it. Here's the pattern, and it's more damning than the version I first wrote. The numbers that survive a check are the borrowed ones: Anthropic's own 8x merge rate, an arXiv paper's 520-of-17,022, a published RCT's 50-versus-67. The numbers the authors generated themselves are the ones that don't: the 6x, the 50% break-even, the six-day rewrite that names no engineer and no language. Relaying someone else's measurement is the only thing this genre does reliably, and relaying is not checking, which is how a wrong claim about Ralph rode 24.8 million views with a citation attached the whole way. That is the argument, delivered by the subject. This genre is a content loop with engagement as its reward function, and engagement is exactly the verifier this post has been describing: independent context, no independent evidence. A fresh judge, reading the transcript, with no ability to run the tests. So it rewards the remix over the source, the confident roadmap over the careful experiment, the funnel over the engineer. The most useful piece I read is the Claude Code team's own, and it's the only one that opens by admitting the discourse is mush: "If you spend some time on X trying to pin down what a loop actually is, you'll come across multiple different answers." Then it does what no roadmap does. It sorts loops by what you hand off. > [Figure: Loop ladder, drawn on the page](https://kachar.dev/blog/loop-engineering-is-verifier-engineering#the-loop-discourse-is-a-loop-with-no-verifier) That's the real taxonomy, and it makes the thesis obvious. The ladder is not a progression in sophistication. It's a progression in how much of your judgment you've handed to something that cannot be embarrassed. You cannot climb a rung your verifier can't hold. ## You can loop the work. You cannot loop the verdict. The skeptic's case has been walked back by its own authors, and I'd rather tell you that than win cheap. [METR marked their famous "19% slower" study out of date themselves](https://metr.org/blog/2026-02-24-uplift-update/); the continuation puts both cohorts' confidence intervals across zero. What survived should worry you more. Those developers predicted a 24% speedup, experienced a 19% slowdown, and afterwards still believed they'd been sped up by 20%. The measurement moved. The metacognitive gap didn't. We are unreliable narrators of our own productivity, and a loop that runs while you sleep is a machine for widening exactly that gap. Which is why [Addy's follow-up](https://x.com/addyosmani/status/2074927530482835916) is the most important thing anyone wrote this cycle and got a fifth of the attention of the roadmaps. He splits it into Quality, Verdict, and Answerability. "The model may write the line, but the Verdict is mine." Comprehension debt is what accrues when the loop ships code faster than you read it. Cognitive surrender is what happens when you stop having an opinion because the loop seems to have one. I should tell you how this post was made, because it would be absurd not to. I wrote it inside one of these loops: a goal with a stop condition, sub-agents fanning out across eleven sources, a fleet of reviewer agents grading the draft against a rubric. It worked. It found the Ralph inversion and the duplicated statistic. So here's the one thing to do tomorrow. Go read your loop's stop condition and ask which of the two independences it actually has. Write the gate as a command, not a criterion: if it can't reduce to a non-zero exit code, it isn't a verifier yet, it's a hope. Use `/goal` only for conditions the transcript can prove, and stop pretending it gates anything else. And if the check genuinely needs judgment, give the judge its own window and its own tools, because one without the other is theater. The dumbest verifier that touches reality beats the cleverest one that doesn't. And then there's the part I have to tell you, because the draft of this post failed its own thesis twice. A research agent told me the Telegram article was "roughly 40% promotion." I put 40% in the draft. Nobody had measured it. A fact-checking agent later put it at about a third, so I stopped taking numbers from agents and counted it myself: 31%. For a few hours, the paragraph indicting this genre for unverified numbers ran on an unverified number I'd inherited from an agent and never questioned. It was fluent, it was plausible, and I printed it because it sounded like the number I wanted. The second was worse, because the agent was right. It told me Akshay's article retells his co-founder's newsletter piece. I read "uncredited" and started drafting an accusation. The word "co-founder" was sitting in the same sentence. They run the newsletter together; it's their own material. The agent reported it accurately and I misread it into the story I already liked. The third one is the reason this section exists. An earlier draft of the passage above debunked Jason Zhou. I'd pulled his sitemap, found 94 pages where his claim implied hundreds, and written that the only falsifiable number in his post doesn't survive contact with his own website. It took thirty seconds and it felt great. A review agent then pointed out that I'd tested his blog against a claim he never made about blogs, and that the second sitemap I'd waved away on a technicality holds 931 pages timestamped across 25 days, or roughly 37 a day, sitting right inside the range he claimed. I pulled it myself. The agent was right. I had run a check, gotten the answer I wanted, and stopped. That is the whole disease in one paragraph, and I had it. A verifier I controlled, pointed at a target I chose, returning the answer I wanted, dressed up as diligence. If a reviewer with its own context and its own tools hadn't gone and looked, I'd have published a false accusation against a named engineer in an essay about how nobody checks anything. The loop didn't verify anything. It generated candidates. I verified, badly, and something with independent evidence caught me. Your evals were [always the moat](https://kachar.dev/blog/your-evals-are-the-moat), and that distinction is the whole job now: you can automate the work, you can automate the checking of the work, and you still cannot automate the part where someone decides what good means and puts their name on it. Design the loop. Then go build the thing that can tell it no. --- # Prompt engineering is over. You're a context engineer now. > Prompt engineering quietly gave way to context engineering. The catch: most teams doing it are doing it backwards. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/context-engineering-is-mostly-subtraction Markdown: https://kachar.dev/blog/context-engineering-is-mostly-subtraction.md Published: 2026-07-02 Reading time: ~10 min Tags: ai, context-engineering, agents Cite as: Ilko Kacharov, "Prompt engineering is over. You're a context engineer now.", kachar.dev, July 2, 2026. https://kachar.dev/blog/context-engineering-is-mostly-subtraction ## Contents 1. [Everyone agrees context engineering is the job. Almost everyone does it backwards.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#everyone-agrees-context-engineering-is-the-job-almost-everyone-does-it-backwards) 2. [A bigger window is not more memory. Long context rots.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#a-bigger-window-is-not-more-memory-long-context-rots) 3. [The failure modes have names. That is what makes this engineering.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#the-failure-modes-have-names-that-is-what-makes-this-engineering) 4. [The real wins are architectural, not better wording.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#the-real-wins-are-architectural-not-better-wording) 5. [Subtraction done wrong is worse than doing nothing, and nobody agrees how to cut.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#subtraction-done-wrong-is-worse-than-doing-nothing-and-nobody-agrees-how-to-cut) 6. [The best moves are counterintuitive. Tidying by instinct gets them wrong.](https://kachar.dev/blog/context-engineering-is-mostly-subtraction#the-best-moves-are-counterintuitive-tidying-by-instinct-gets-them-wrong) --- ![A block of dark metal on a workbench, mostly carved away, with heaps of shavings on the floor. What remains at its centre is a small form that glows electric violet.](https://kachar.dev/posts/context-engineering-is-mostly-subtraction-hero.jpg) A 300-token prompt beat a 113,000-token one on the exact same question. Same model, same task, and the short one won. That gap is the whole discipline in one number. Most teams doing "context engineering" are quietly doing the opposite. The reflex is to add: more history, more retrieved documents, more tools, more instructions. It feels like diligence. It is making agents worse, and the measurements are not subtle about it. The field already settled the easy part. Prompt engineering gave way to context engineering, and nobody serious argues otherwise. I called it [the new CMS](https://kachar.dev/blog/context-is-the-new-cms): what your model knows before it writes a word. [Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) defines it as "curating and maintaining the optimal set of tokens" during inference, where the curation "happens each time we decide what to pass to the model." [Cognition](https://blog.langchain.com/context-engineering-for-agents/) calls it "effectively the #1 job of engineers building AI agents." What the field has not internalized is the hard part: the default move, assemble everything relevant into the window, is the antipattern, not the practice. ## Everyone agrees context engineering is the job. Almost everyone does it backwards. Read the definitions closely and they all point at subtraction. [Andrej Karpathy's](https://blog.langchain.com/context-engineering-for-agents/) version is "just the right information for the next step." The operative word is *just*. Not all the relevant information. The right information, and no more. Then look at how the work actually gets framed. [AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/gen-ai-lifecycle-operational-excellence/dev-experimenting-context.html) describes context engineering as "presenting all of the relevant information the LLM needs" and hands teams a checklist: system prompt, plus user profile, plus memory, plus tools, plus retrieval. That checklist is the accumulation reflex wearing a lab coat. Everything on it is relevant. That is exactly the problem. Relevance is not the bar. A model can drown in things that are all individually relevant. This is the trap I keep watching teams walk into. They frame context work as "what else can we give the model to help it," and in doing so they institutionalize the failure mode. The competitive question for 2026 is not what you can add. It is what you have the discipline to leave out. ## A bigger window is not more memory. Long context rots. The escape hatch everyone reaches for is the context window itself. Just wait for the million-token model and the problem disappears. Measurement closed that hatch. [Chroma](https://research.trychroma.com/context-rot) tested 18 models, including Claude Opus 4, GPT-4.1, Gemini 2.5, and Qwen3, and found what they named context rot: "performance grows increasingly unreliable as input length grows." This is where that 300-versus-113,000 number comes from. On the LongMemEval benchmark, a focused ~300-token input beat the same content padded to ~113,000 tokens, across every model family. The window was there. The models could not use it. It gets more counterintuitive. Chroma found models "perform better on shuffled haystacks than on logically structured ones." Coherent, well-organized prose hurt retrieval. Even a single distractor sentence measurably dropped performance, and the damage grew with length. Your intuition about what reads well to a human is not a proxy for what a model can attend to. Anthropic names the mechanism. Models have an "attention budget" that "depletes" with each new token, a consequence of transformer attention scaling with the square of the sequence length and of training data where short sequences dominate. Context has "diminishing marginal returns." Databricks saw the same curve from the other side: Llama 3.1 405B accuracy started falling around 32,000 tokens, and Gemini 2.5 agents past 100,000 tokens began favoring "repeating actions from history" over forming new plans. I have made [the model-size version of this argument](https://kachar.dev/blog/a-bigger-model-wont-save-you) before, that a smaller model with the right context beats a frontier model fed a junk drawer. This is the sharper corollary. It is not only that the model is rarely your bottleneck. It is that a *fixed* model gets measurably dumber as you fill its window, so every "let's wait for longer context" line on your roadmap is a bet against the data. ## The failure modes have names. That is what makes this engineering. Long context does not fail randomly. It fails in specific, reproducible, named ways, and naming them is the difference between a debuggable ticket and a shrug. [Drew Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html) cataloged four. **Context poisoning**: a hallucination lands in the context and then gets referenced again and again as if it were true. **Context distraction**: the accumulated context grows large enough to overwhelm what the model learned in training. **Context confusion**: superfluous content in the window degrades the output even when it is harmless. **Context clash**: parts of the context contradict each other, and the model has no way to know which to trust. Each has receipts. On tool confusion, a quantized Llama 3.1 8B failed a benchmark with 46 tools available and succeeded with 19, even though all 46 fit inside the 16k window. The Berkeley Function-Calling Leaderboard shows the general version: every model tested does worse as you hand it more tools. On context clash, a Microsoft and Salesforce study found a 39% average performance drop when a prompt was "sharded" into fragments that arrived across turns, and OpenAI's o3 fell from 98.1 to 64.1 on the same task when the information came in conflicting pieces instead of one clean block. The [ACE researchers](https://arxiv.org/abs/2510.04618) added two more to the vocabulary: "brevity bias," where optimization collapses your context toward short generic instructions, and "context collapse," where a rewrite step nukes hard-won detail. You cannot manage what you cannot name. Teams that adopt this vocabulary triage a regression in an afternoon. Teams that say "the model is being flaky" go buy a bigger one to fix what is actually a curation bug. ## The real wins are architectural, not better wording. Here is the part that reframes the job. The fixes that actually move benchmarks are not cleverer prompt text. They are architecture decisions that keep tokens *out* of the window in the first place. [Anthropic's own example](https://www.anthropic.com/engineering/code-execution-with-mcp) is blunt. Presenting tools to an agent as on-demand code APIs, instead of loading every tool definition up front, cut one workflow from 150,000 tokens to 2,000. That is a 98.7% reduction from a structural change, not a prompt tweak. Their [context editing feature](https://platform.claude.com/docs/en/build-with-claude/context-editing), which auto-clears stale tool results mid-run, combined with a memory tool, improved performance 39% over baseline while using 84% fewer tokens on a 100-turn web-search evaluation. Context editing alone was worth 29%. And it ships as tunable knobs, trigger, keep, clear_at_least, exclude_tools, not a magic button. > **Where the leverage actually lives** > > The highest-ROI context work is in your harness and tool design, not your prompt file. That makes it platform engineering, which is a moat. Prompt text is not. Isolation is the other big lever, and it is honest about its cost. Anthropic's multi-agent research system, an Opus 4 lead delegating to Sonnet 4 subagents that each work in a separate window and report back only a summary, beat single-agent Opus 4 by 90.2% on their evaluation. It also burned up to 15 times the tokens. Powerful, not free. Production teams make the same trade quietly. [Ramp's internal coding agent](https://builders.ramp.com/post/why-we-built-our-background-agent) writes roughly 30% of the company's merged pull requests by pre-building environments and spawning isolated child sessions. ChatGPT's memory system, as [reverse-engineered by practitioners](https://manthanguptaa.in/posts/chatgpt_memory/), deliberately injects only about 15 lightweight summaries and skips retrieval augmentation entirely, throwing away most of the raw history on purpose. ## Subtraction done wrong is worse than doing nothing, and nobody agrees how to cut. Now the sharp edge. Everyone ships auto-compaction as the answer, summarize the old context and start fresh with the summary. But compaction is itself a named failure mode, and the industry is openly at war with itself over it. The [ACE paper](https://arxiv.org/abs/2510.04618) (arXiv 2510.04618, headed to ICLR 2026) has the smoking gun. A single monolithic rewrite step collapsed an agent's context from 18,282 tokens at 66.7% accuracy down to 122 tokens at 57.1%. That is below the 63.7% you get from doing no adaptation at all. Their fix is the opposite of compaction: "grow-and-refine," where you make small incremental "delta updates" to a structured context and almost never rewrite it wholesale. That approach gained 10.6% on the AppWorld benchmark using a smaller open-source model. > **What 'just compact it' can cost you** > > One rewrite step collapsed an agent's context from 18,282 tokens to 122, and its accuracy dropped below doing nothing at all. Auto-compaction ships as a default in most agent stacks today, silently deleting detail you paid for. And yet auto-compaction is everywhere, shipped by the same companies who know better. [Claude Code now fires compaction earlier](https://hyperdev.matsuoka.com/p/how-claude-code-got-better-by-protecting), around 75% of the window instead of 90%, and treats it as an improvement. ByteDance ships both answers at once: their [DeerFlow](https://github.com/bytedance/deer-flow) agent "manages context aggressively, summarizing completed subtasks," while their [OpenViking](https://openviking.ai/) system preserves full detail in a tiered filesystem and loads ~100-token abstracts before pulling the full text. One company, two opposite bets, unresolved. The most interesting dissent reframes the whole fight. The [Distill project](https://distill.siddhantkhare.com/) argues that "context deduplication is not an optimization, it is a correctness measure," and does it deterministically, no LLM in the loop, 30-40% fewer tokens in about 12 milliseconds, precisely because an LLM rewrite "varies per run." Even compaction's defenders hedge: the [practitioner canon](https://github.com/ai-boost/awesome-harness-engineering) says never rely on it for critical rules, move those to a file that survives any compression. As of mid-2026, there is no settled answer on how to cut. The only wrong move is to inherit your vendor's default and never look at it. ## The best moves are counterintuitive. Tidying by instinct gets them wrong. The playbooks that actually work break the naive rules, which is exactly why a team optimizing for "cleaner, shorter context" will do the wrong thing. [Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus), which has published more hard-won detail than almost anyone, keeps failed actions and stack traces in the context on purpose. "Erasing failure removes evidence," they write. "And without evidence, the model can't adapt." Recovering from a visible error is, in their words, one of the clearest signs of real agentic behavior. So the tidy instinct, delete the ugly failed turn, is the one that hurts. They also treat KV-cache hit rate as "the single most important metric for a production-stage agent." Cached input tokens run about $0.30 per million versus $3.00 uncached, a 10x swing, so a single unstable token near the start of your prompt, a timestamp, say, can quietly detonate your bill by invalidating the cache for everything after it. That is why their rule is "mask, don't remove" tools: pulling a tool definition out mid-session breaks the cache and confuses the model that saw earlier references to it. Even their good moves have a cost. Their "recitation" trick, rewriting a todo.md to keep the goal in recent attention, [ate roughly a third of all actions](https://rlancemartin.github.io/2025/10/15/manus/). This is context engineering as a boring function boundary, not a vibe. In the [Vercel AI SDK](https://ai-sdk.dev/docs/reference/ai-sdk-ui/convert-to-model-messages), convertToModelMessages silently drops unhandled data parts, and a tool's toModelOutput decides what its result actually contributes to the window. Nobody calls those seams "context engineering." They are exactly where it happens. I have argued that [agent memory is mostly hoarding](https://kachar.dev/blog/agent-memory-is-mostly-hoarding), that the real skill is knowing what to forget. This is the complication to that: sometimes the skill is knowing what ugly, redundant, failed thing to keep. Prompt engineering was writing. Context engineering is editing. And every editor eventually learns the same lesson the hard way: the strongest thing you can do to a draft is cut it. The teams that win the next couple of years will not be the ones who fit the most into the window. They will be the ones with the nerve to throw most of it out, and the receipts to prove which cut helped. --- # The AI coworker is a weekend project now > Viktor and Martoshi make the AI-employee-in-Slack look like a company. We built the same shape on Vercel's Eve in one session. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now Markdown: https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now.md Published: 2026-07-02 Reading time: ~4 min Tags: building, agents, architecture Cite as: Ilko Kacharov, "The AI coworker is a weekend project now", kachar.dev, July 2, 2026. https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now ## Contents 1. [The category looks like a moat, and that is fair](https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now#the-category-looks-like-a-moat-and-that-is-fair) 2. [Most of that moat is now framework primitives](https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now#most-of-that-moat-is-now-framework-primitives) 3. [The integration is one file](https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now#the-integration-is-one-file) 4. [What is actually hard is the part you already own](https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now#what-is-actually-hard-is-the-part-you-already-own) --- ![A small robot on a workbench at night, lit by a desk lamp glowing electric violet. A corporate tower stands in the dark behind it.](https://kachar.dev/posts/the-ai-coworker-is-a-weekend-project-now-hero.jpg) [Viktor](https://www.viktor.com) calls itself an AI employee. [Martoshi](https://martoshi.com) is an AI coworker you tag in a Slack channel. Claude has [its own Slack agent](https://www.anthropic.com/news/introducing-claude-tag) now too. Every one of them reads like a company you would need a year and a Series A to build. We built the same shape of thing this week, in one working session, as a directory of files. The category is thinner than the pricing pages make it look. ## The category looks like a moat, and that is fair Look at what they sell. Viktor gives each user a computer in the cloud that writes and runs code, wired to 3,200 tools, SOC 2 in the footer. Martoshi tags into a channel and drops back a PDF, a dashboard, an ad audit, over a thousand integrations behind it. It reads like platform. Isolated compute, persistent memory, an integrations team, a compliance page. From the outside the hard part looks like everything. It is a fair read. That is genuinely what an AI coworker needs. The mistake is assuming you have to build all of it. ## Most of that moat is now framework primitives [Eve](https://vercel.com/eve) is the tell. Vercel shipped it in mid-2026, a filesystem-first framework for durable agents, Apache-2.0 and still in beta. You do not wire an agent. You lay out a directory. `agent/instructions.md` is the system prompt. `agent/channels/slack.ts` is the Slack entry point. `agent/tools/*.ts` is one typed tool per file. `agent/connections/*.ts` is your integrations. Eve discovers the files at build time and compiles them into an app that runs on Vercel Functions. No registration code, no glue. Sessions stream token by token, and they resume after a cold start or a deploy and pick up where they left off. That "his own computer in the cloud" bullet everyone markets? It is just the platform. It is the default. This is the part that matched our stack so cleanly it felt like cheating. We already ship TypeScript on Vercel and talk MCP all day. Eve is native to all three. There was nothing to bridge. ## The integration is one file The scary number on every one of these pages is the integration count. Here is ours. `agent/connections/juma.ts` is a `defineMcpClientConnection` pointed at [our own MCP server](https://juma.ai/mcp), authed per user through [Vercel Connect](https://vercel.com/docs/connect) so every teammate links their own account. Connect is the quiet hero here. The OAuth dance, the per-user token exchange, the refresh, the callback route you always dread writing, it owns all of it and hands you a token when you ask for one. I never stood up an auth server. That one file gives the agent the whole surface of Juma: brand profiles, knowledge, projects, and the tools that generate on-brand decks, reports, and images. That is Martoshi's entire deliverable list, sourced from the product we already run. This session we added a second file. `agent/connections/linkedin.ts`, a `defineOpenAPIConnection` with an inline spec and the `w_member_social` scope, so the agent can draft a post in Juma and publish it to LinkedIn with a human approval gate in front of the publish call. Wiring it was a file, not a project, because with Connect anything that speaks OAuth is one connector away. "3,200 integrations" is a number you reach one file at a time, and half of them are MCP servers already waiting to be pointed at. > **The integration flipped** > > An integration used to be a project. On Eve it is a file in a directory. The moat was the glue, and the glue is gone. ## What is actually hard is the part you already own Strip away the Slack bot and the connectors and you are left with the only thing that was ever the product: what the agent knows. Martoshi's deliverables land because of the brand and campaign context behind them, not the channel they post in. Ours land because [Juma](https://juma.ai/mcp) holds our brand, our knowledge, our source of truth. The Slack agent is a thin shell around that context, and [context is the new CMS](https://kachar.dev/blog/context-is-the-new-cms). The shell took an afternoon. The context took years, and it is the only part a competitor cannot clone by reading this post. So the math flipped again, the same way it did when [we deleted Jira](https://kachar.dev/blog/we-deleted-jira-in-three-weeks). When the build cost of the surface collapses, the value moves entirely to what sits behind it. As of mid-2026, the surface is a weekend. You do not hire the AI coworker anymore. If you own the context, you compile one, and you keep the keys. --- # Your product is the easy part now > AI collapsed the cost of building software. The bottleneck moved from creation to adoption, and most founders are still optimizing the solved side. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/your-product-is-the-easy-part-now Markdown: https://kachar.dev/blog/your-product-is-the-easy-part-now.md Published: 2026-07-02 Reading time: ~3 min Tags: ai, building, product Cite as: Ilko Kacharov, "Your product is the easy part now", kachar.dev, July 2, 2026. https://kachar.dev/blog/your-product-is-the-easy-part-now ## Contents 1. [Building was the filter, and the filter broke](https://kachar.dev/blog/your-product-is-the-easy-part-now#building-was-the-filter-and-the-filter-broke) 2. [The scarce resources are not technical anymore](https://kachar.dev/blog/your-product-is-the-easy-part-now#the-scarce-resources-are-not-technical-anymore) 3. [Content was the escape hatch, and it closed](https://kachar.dev/blog/your-product-is-the-easy-part-now#content-was-the-escape-hatch-and-it-closed) 4. [Value moved left of the first commit](https://kachar.dev/blog/your-product-is-the-easy-part-now#value-moved-left-of-the-first-commit) --- ![A polished dark metal machine stands alone in an empty warehouse. Far behind it, at the end of a long corridor, one doorway glows electric violet.](https://kachar.dev/posts/your-product-is-the-easy-part-now-hero.jpg) I wrote this week that [the AI coworker is a weekend project now](https://kachar.dev/blog/the-ai-coworker-is-a-weekend-project-now). Here is the uncomfortable second half of that sentence: it is a weekend project for everyone. When a solo founder can ship what used to take a funded team, shipping stops separating anyone from anyone. The startup constraint has moved from creation to adoption, and most founders are still optimizing the side that is already solved. ## Building was the filter, and the filter broke For twenty years "can you actually build it?" was the great filter of startups. Ideas were cheap. Engineering teams were not. Investors funded execution because execution was scarce, and a working product was proof you deserved to exist. As of mid-2026, a single founder with Claude Code, a Vercel account and a free weekend ships what a seed round used to buy. The code, the design, the copy, the landing page, the demo video. All of it compiles. This is the best time in history to build software. It is also the worst time for building software to be your advantage. When everyone gets the same superpower, it stops being a power. It becomes weather. ## The scarce resources are not technical anymore Value lives where scarcity lives, and the scarce list has been rewritten. Engineering hours are off it. What is on it reads like a syllabus from a different discipline: trust, credibility, distribution, relationships, attention, reputation, proof that your product worked for someone who looks like the buyer. Watch the buyer's question change underneath the market. It used to be "can this company build what they promise?" Nobody asks that anymore, because the honest answer is that everyone can. The question is "should I trust this company with my workflow, my budget and my time?" You cannot answer that with a feature. Coding, design, copywriting, landing pages: none of these became worthless. They became multipliers. A multiplier amplifies an advantage you already have. It is not one itself, and multiplying zero is still zero. ## Content was the escape hatch, and it closed The standard answer to "how do I get distribution?" was content. Write the posts, record the videos, send the emails, earn attention with volume and consistency. AI closed that hatch by opening it for everyone. The same tools that made your product a weekend build made everyone's content calendar infinite. Hundreds of AI products launch every day, each with a polished landing page, a founder story thread and the same LLM-written case studies. Producing content is the baseline now, not the edge. Volume is free, so volume is worth nothing. > **The weekend test** > > If you can build it in a weekend, so can everyone else. Whatever you cannot build in a weekend - trust, relationships, proprietary data, a reputation that precedes you - is your actual company. ## Value moved left of the first commit The founders winning right now arrived with the scarce assets already in hand. An audience from years of writing. A network from the last company. Customers who trust them personally and will follow them into whatever they build next. They did not build a product and then go looking for distribution. They owned distribution and compiled a product into it. At an individual level this looks unfair. At market scale it is just the new physics: when creation is abundant, adoption is the whole game, and adoption is owned before launch day. So flip your plan. Spend the weekend on the product, because it only takes a weekend now. Spend the year on what cannot be compiled: the relationships, the credibility, the data your product earns by working. The product used to be the proof. Now it is the premise. --- # The chat UX is the product, not the model > shadcn calls it scroll engineering. It is bigger than that. Users don't churn on your model, they churn on the chat UX around it. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model Markdown: https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model.md Published: 2026-06-26 Reading time: ~3 min Tags: building, product, design Cite as: Ilko Kacharov, "The chat UX is the product, not the model", kachar.dev, June 26, 2026. https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model ## Contents 1. [The one rule under all fifteen](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model#the-one-rule-under-all-fifteen) 2. [Scroll is the visible tip](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model#scroll-is-the-visible-tip) 3. [The hard part is interruptions](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model#the-hard-part-is-interruptions) 4. [Even the terminal fights this](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model#even-the-terminal-fights-this) 5. [Why nobody staffs it](https://kachar.dev/blog/the-chat-ux-is-the-product-not-the-model#why-nobody-staffs-it) --- ![Close-up of a machined dark metal scroll wheel whose edge glows electric violet. The engine behind it is a rough grey shape lost in haze.](https://kachar.dev/posts/the-chat-ux-is-the-product-not-the-model-hero.jpg) Users don't churn over your model. They churn over the chat experience around it. Same model, two products: one feels alive, the other fights you on every scroll. The difference is craft nobody puts on the roadmap. [shadcn just published 15 rules](https://x.com/shadcn/status/2070394918720221522) for a great streaming chat experience. They are good. Read together, they point at something bigger than scroll. ## The one rule under all fifteen Every item on that list reduces to a single principle: never move the reader against their intent. Auto-scroll by default is the bug, not the feature. Follow the stream only while the reader is following it. The moment they scroll up, select text, open a link, or touch the keyboard, that is intent. Stop moving. The interface should never drag attention somewhere the reader did not ask to go. ## Scroll is the visible tip shadcn calls it scroll engineering. Scroll is just the part you can see. The same principle governs the whole turn. Keep enough of the previous answer on screen that people know where they are. Start each new turn near the top so it can be read from the beginning. Pace the stream so it is fast but not frantic. Let new content arrive offscreen without yanking the viewport. This is layout under motion, and most chat UIs get it wrong because they were built as a div that appends text. ## The hard part is interruptions Streaming is easy when nothing goes wrong. The craft shows up when it does. Stop, retry, regenerate, branch, error. None of those should steal the reader's position. Reopening a saved conversation should land on the last meaningful turn, usually the user's last message, not the absolute bottom of a 200-message log. Images load, markdown expands, code blocks render, older messages page in above - and through all of it the reader should keep their place. That is the line between an app that respects attention and one that resets it. ## Even the terminal fights this This is not a web problem. It is a chat problem, and it follows the format anywhere chat goes. Claude Code runs in a terminal and fights the exact same battle: scrollback you can trust, a way to jump back to the latest output, keeping your place while tokens stream in, announcing what is happening without drowning you in noise. No React, no virtualized list, same discipline. The surface changes. The craft does not. ## Why nobody staffs it Here is why this matters more than it looks. Everyone now rents the same frontier models. Your competitor is one API key away from the same intelligence you have. The model is not where you win. The chat experience around the model is the part you own and the part users feel on every single turn. It is also the part nobody assigns an owner. There is a prompt engineer, a model eval, a RAG pipeline - and the scroll position gets handled by whoever touched the component last. [A bigger model won't save you](https://kachar.dev/blog/a-bigger-model-wont-save-you) here, and [your homepage](https://kachar.dev/blog/your-homepage-should-run-the-product) won't either if the product behind it jitters. > **Screenshot this** > > Users don't churn on your model. They churn on your scrollbar. The model is rented. The chat is yours. Build it like it matters, because it is the only part of the stack your users actually touch. --- # Stop picking npm or shadcn for your users > npm versus shadcn is the wrong fight. How much of your component a team wants to own is their call, not yours. Ship the whole spectrum from one source. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users Markdown: https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users.md Published: 2026-06-17 Reading time: ~4 min Tags: building, architecture, product Cite as: Ilko Kacharov, "Stop picking npm or shadcn for your users", kachar.dev, June 17, 2026. https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users ## Contents 1. [npm and shadcn are not formats. They are control levels.](https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users#npm-and-shadcn-are-not-formats-they-are-control-levels) 2. [The install command is the API you never reviewed](https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users#the-install-command-is-the-api-you-never-reviewed) 3. [One src directory, or it is theater](https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users#one-src-directory-or-it-is-theater) 4. [The third consumer does not read your README](https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users#the-third-consumer-does-not-read-your-readme) 5. [Range beats a point, every time](https://kachar.dev/blog/stop-picking-npm-or-shadcn-for-your-users#range-beats-a-point-every-time) --- ![The same dark metal component three times along a workbench: sealed in its case, half open, and fully taken apart. A single line of electric violet light runs through all three.](https://kachar.dev/posts/stop-picking-npm-or-shadcn-for-your-users-hero.jpg) Every "should I ship this as an npm package or a shadcn registry?" thread is arguing the wrong question. How much control a developer wants over your component is a fact about their team, not a decision about your library. A solo hacker wants `npm install` and zero opinions. A platform team wants the source in their repo so they can fork it the day your defaults stop fitting. Both are right. And the buyer decides before they read a line of your docs. Ship npm-only or registry-only and you are not making an architecture call. You are imposing your risk tolerance on strangers. So we shipped Prompt Area both ways, from one source. Here is why that is the only honest answer. ## npm and shadcn are not formats. They are control levels. Treat them as two competing package formats and you will pick the "better" one. That framing is the mistake. They are two points on a single axis: how much of the component you own. npm is a versioned black box. You upgrade it through semver and you never touch the source. shadcn copies the source into your repo. You own it, you fork it, and there is no upgrade path. That is not a technical fork in the road. It is a convenience-versus-control tradeoff, and different teams sit at different points on it for the same library on the same day. A trivial date formatter? Rent it. `npm install` and never look again. A chat composer that defines how your product feels? You will want to own that the moment a designer asks for one pixel you cannot reach from a prop. Same author, same component, opposite correct answers. ## The install command is the API you never reviewed Teams will argue about a single prop name for an afternoon. Then they ship one install path as if it were a build detail. It is not. The install command encodes the entire ownership contract: rent or own, upgrade or fork, black box or editable source. That is a bigger decision than any prop in your public API, and it is the first one a developer makes, in about ten seconds, before they have read anything. Pick one path for them and you have made their most consequential interface decision by accident. ## One src directory, or it is theater The reason "ship both" used to be bad advice: two distributions meant two codebases, and two codebases drift. The copy-paste users quietly fall a version behind, your nice DX feature rots into a maintenance tax, and you slowly stop shipping to one side. That constraint is gone. Prompt Area builds from one `src` directory to both targets, the npm package via tsup and the shadcn registry JSON off the same files. Same code, two delivery mechanisms, zero divergence. The packaged version is about 37kB. The copied source stays small because the core is dependency-light: no ProseMirror, no Slate, no Lexical. That is what makes owning it viable and depending on it cheap. Ship-both is only honest when divergence is structurally impossible, not just discouraged. If your build cannot do that, the build is the bug, not the strategy. > **The decision was never yours** > > How much control a developer wants over your component is a fact about their team, not a decision about your library. Stop answering it for them. ## The third consumer does not read your README There is a new buyer on the spectrum, and it sits at the zero-human-steps end: the coding agent. Claude Code, Codex, and Cursor do not browse your docs. They fetch. So Prompt Area serves an `llms-full.txt` and a ready-made install prompt the agent runs itself. A machine-readable install path is not a nice-to-have. It is a third delivery mechanism off the same source, for the consumer who wants the least friction of all. The teams still debating npm versus shadcn are arguing about two doors while a third one became the most-used entrance. ## Range beats a point, every time Shipping the whole spectrum is strictly more reach than shipping any single point on it. The marginal cost is a build step, not a second product. And you stop guessing which consumer you have, because they tell you, by choosing the install command. This is boring infrastructure spent well: a light core, every channel open, and the choice handed back to the person who was always going to make it. I [spend innovation budget the same way](https://kachar.dev/blog/three-innovation-tokens), cheap and reusable underneath so the surface can be loud. Pick a lane and you have picked it for thousands of teams who never asked you to. Ship the whole spectrum, and the only thing left for you to decide is the code. They decide how much of it to own. --- # We deleted Jira in three weeks > Building your own Jira used to be insane. With AI writing the code and an MCP server wiring it to everything, the math flipped. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/we-deleted-jira-in-three-weeks Markdown: https://kachar.dev/blog/we-deleted-jira-in-three-weeks.md Published: 2026-06-16 Reading time: ~3 min Tags: building, agents, cto Cite as: Ilko Kacharov, "We deleted Jira in three weeks", kachar.dev, June 16, 2026. https://kachar.dev/blog/we-deleted-jira-in-three-weeks ## Contents 1. [The old math said don't](https://kachar.dev/blog/we-deleted-jira-in-three-weeks#the-old-math-said-dont) 2. [It ate the other tabs too](https://kachar.dev/blog/we-deleted-jira-in-three-weeks#it-ate-the-other-tabs-too) 3. [The MCP server is the real unlock](https://kachar.dev/blog/we-deleted-jira-in-three-weeks#the-mcp-server-is-the-real-unlock) 4. [You own the data, and that is the prize](https://kachar.dev/blog/we-deleted-jira-in-three-weeks#you-own-the-data-and-that-is-the-prize) --- ![A dark office wall still marked with the outline and bolt holes of a large machine that was removed. On the desk below, a small handmade machine glows electric violet.](https://kachar.dev/posts/we-deleted-jira-in-three-weeks-hero.jpg) "Build vs buy" was settled a decade ago. You buy. Building your own issue tracker was a punchline, a thing only doomed startups did on their way to running out of money. We built ours in three weeks and it was the best infrastructure decision we made this year. The tool is called Jumira. We are Juma, so the name wrote itself. It started as a Jira replacement. It did not stay one. ## The old math said don't The reason "build your own Jira" was a joke is that the build cost dwarfed the license cost. A team of engineers burning weeks to rebuild what Atlassian sells for a few dollars per seat is obviously stupid. That math was correct for fifteen years. It is not correct anymore. AI writes most of the code now. The work that used to take a quarter takes a sprint. When the build cost collapses, every "just buy it" decision is suddenly worth re-running. Most teams have not re-run it. They are still quoting the old answer to a question that changed. ## It ate the other tabs too We did not just replace Jira. We replaced Miro, because Jumira has timelines and an idea-capture board. We folded in the parts of three other tools we actually used and dropped the parts we never touched. The thing nobody sells you is coherence. Every SaaS tool is its own tab, its own login, its own half-broken integration with the next one. Owning the whole surface means one UI, one design language, one source of truth. We own the feeling of the thing, and we can change it on a Tuesday. ## The MCP server is the real unlock Jumira runs an MCP server. That is the part that makes it more than a cheaper Jira. Paste a ticket link into Claude Code and the agent reads it straight from the MCP server. It investigates, iterates, leaves comments, builds checklists, attaches what it finds. When a task finishes, it updates its own status. There is a page for my open pull requests. I create tickets from a PR, attach tickets to the PR I am on, and watch the GitHub Actions status without leaving the app. No vendor was ever going to build that for us. It is too specific to how we work. That is the whole point. > **The new build-vs-buy test** > > If AI can build it in a sprint and you own the data, buying it is the expensive option. ## You own the data, and that is the prize The license fee was never the real cost. The real cost was your workflow living in someone else's database, on someone else's roadmap, behind someone else's API limits. In an era where every workflow is becoming an agent surface, the data and the integration points are the asset. Renting them out means the most valuable layer of how your team works belongs to a vendor. We pulled it back. Our processes are ours, our data is ours, and the next feature we need ships in days because there is nobody to ask. We are a team of builders. So we built the tool we work inside. Turns out the joke was on everyone still paying rent. --- # Your homepage should run the product > I rebuilt the Prompt Area homepage around the live component instead of copy about it. For a developer tool, the demo is not a section on the page. It is the page. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/your-homepage-should-run-the-product Markdown: https://kachar.dev/blog/your-homepage-should-run-the-product.md Published: 2026-06-15 Reading time: ~3 min Tags: building, product, design Cite as: Ilko Kacharov, "Your homepage should run the product", kachar.dev, June 15, 2026. https://kachar.dev/blog/your-homepage-should-run-the-product ## Contents 1. [Copy describes. A running component proves.](https://kachar.dev/blog/your-homepage-should-run-the-product#copy-describes-a-running-component-proves) 2. [Seed it with real content, not lorem](https://kachar.dev/blog/your-homepage-should-run-the-product#seed-it-with-real-content-not-lorem) 3. [The rest of the site followed](https://kachar.dev/blog/your-homepage-should-run-the-product#the-rest-of-the-site-followed) 4. [The honest part](https://kachar.dev/blog/your-homepage-should-run-the-product#the-honest-part) --- For a developer tool, the homepage is not where you describe the product. It is where you run it. I just rebuilt [prompt-area.com](https://prompt-area.com). The old homepage led with a headline and a paragraph explaining what the component does. The new one leads with the component itself, already full of real content: a Codex-style composer with a file attached, two @mentions, a /command, a #tag, and a line of markdown. Before you read a word of copy, you have watched the thing work. ![The Prompt Area homepage hero: a Codex-style composer pre-filled with a /summarize command, @Strategist and @Copywriter mentions, a #campaign tag, inline markdown, and a PDF attachment, over a repository context tray.](https://kachar.dev/posts/prompt-area-codex-composer.png) *The hero, on load. No copy yet. You have already seen mentions, a command, a tag, markdown, and a file, in a layout you recognize.* ## Copy describes. A running component proves. A landing page for a library has one job: convince a skeptical engineer in about ten seconds that this solves their problem. Copy cannot do that. "Rich text input with mentions and commands" is a claim. A cursor blinking inside a real input, where typing @ opens a dropdown and / fires a command menu, is evidence. Engineers do not buy claims. They poke at things. So the redesign inverted the page. The hero is the live component. Everything else (the install command, the feature grid, the comparison to Tiptap and react-mentions) sits below it, in service of the thing you already saw running. ## Seed it with real content, not lorem The mistake most "live demo" sections make is shipping an empty input. An empty box proves nothing. You have to fill it with a believable scenario the moment the page loads. The Prompt Area hero comes pre-populated: - a **/summarize** command chip - **@Strategist** and **@Copywriter** mention chips - a **#campaign** tag - a line of **bold** and *italic* markdown - a PDF sitting in the attachment strip above the input In one glance you see five features working together, inside a layout you already recognize. And because it is the real component and not a screenshot, you can start typing immediately and it responds. > **Show the hard parts** > > The boring 20 percent is where a component earns trust: paste that keeps chips intact, undo that respects them, a file strip, IME composition. Seed the demo with exactly those so a skeptic sees them without having to go hunting. ## The rest of the site followed Once the homepage was the product, the old structure stopped making sense. It used to be one long scrolling page with a sidebar of anchor links, which is really a documentation site cosplaying as a landing page. So I split it. A short marketing homepage that funnels into real docs. A dedicated page for every component, every example, and every built-in style, with the Claude Code and Codex composers each getting their own showcase. Three column docs with an "on this page" rail that tracks as you scroll. The homepage sells, the docs explain, and neither one pretends to be the other. ## The honest part I did the whole rebuild in an afternoon, driving an agent. That is worth saying plainly, because it changes the calculus. When a full redesign costs an afternoon instead of a sprint, "the homepage is just okay" stops being an acceptable answer. The bottleneck was never the code. It was deciding what the page should be. And for a developer tool, that decision is simple. Stop describing your component. Run it. --- # Build for the model you'll lose > The best coding model in the world lasted three days before a government directive switched it off. The lesson is not 'use local models.' It is 'never couple your product to a model you are not allowed to keep.' Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/build-for-the-model-youll-lose Markdown: https://kachar.dev/blog/build-for-the-model-youll-lose.md Published: 2026-06-14 Reading time: ~3 min Tags: ai, architecture, building Cite as: Ilko Kacharov, "Build for the model you'll lose", kachar.dev, June 14, 2026. https://kachar.dev/blog/build-for-the-model-youll-lose ## Contents 1. [The reflex is half right](https://kachar.dev/blog/build-for-the-model-youll-lose#the-reflex-is-half-right) 2. [The real lesson is substitution](https://kachar.dev/blog/build-for-the-model-youll-lose#the-real-lesson-is-substitution) 3. [What portability actually looks like](https://kachar.dev/blog/build-for-the-model-youll-lose#what-portability-actually-looks-like) 4. [You are renting, so act like it](https://kachar.dev/blog/build-for-the-model-youll-lose#you-are-renting-so-act-like-it) --- ![A dark brushed-metal wall with one empty socket glowing electric violet. The unplugged cable lies coiled on the floor in shadow.](https://kachar.dev/posts/build-for-the-model-youll-lose-hero.jpg) Claude Fable 5 shipped on June 9 as, by most benchmarks, the best coding model in the world. By June 12 it was gone. A US government directive ordered Anthropic to cut off all foreign nationals, the segmentation was not clean, and the only way to comply was to switch the model off for everyone. Three days, public to dark. If your product had a `model: "claude-fable-5"` line in it that week, you did not have a bug to fix. You had a dependency you were never allowed to keep. ## The reflex is half right The loudest reaction was predictable. "This is exactly why we need local models," went the top thread on r/LocalLLaMA, and it is not wrong. A model running on your own hardware cannot be switched off by a letter that arrives at 5:21pm. But local-or-bust is the wrong lesson to take from this. Most teams cannot self-host a frontier model, and even if they could, they would be trading a political risk for an operational one: weaker capability, a GPU bill, and a model that is now your problem to keep running. You do not need to own the model. You need to never depend on a specific one. ## The real lesson is substitution The teams that shrugged on June 12 had one thing in common. Swapping the model was a config change, not a rewrite. Their product did not know or care which model was behind the call, because no single model's behavior had been baked into the code, the prompts, or the team's assumptions. That is the property worth engineering for. Not "the best model," not "the model I host myself," but "a model I can replace by Friday without my users noticing." > **The switch-off test** > > Pick the single model your product depends on most. Now assume it disappears tonight, permanently. If your honest recovery time is measured in weeks, you do not have an architecture, you have a hostage situation. ## What portability actually looks like It is unglamorous, which is why most teams skip it until a directive forces the issue: - **An abstraction at the call site**, so the model is a parameter, not a constant scattered across forty files. - **A real fallback you have actually run**, not a second model name you have never sent a production request to. Untested fallbacks fail exactly when you need them. - **Prompts that do not lean on one model's quirks**. If your output only holds together because Fable 5 happened to format things a certain way, you have a model-shaped crack in your product. - **Evals that tell you the swap is safe**. The only way to switch models with confidence is to measure that the new one still passes. This is, again, why [your evals are the moat](https://kachar.dev/blog/your-evals-are-the-moat). ## You are renting, so act like it The whole industry is renting capability from a handful of labs, and that capability now comes with a geopolitical attribute it did not have a year ago. A model can be the best in the world and still be revoked by someone who is not your customer, your vendor, or your problem to argue with. So build for the model you'll lose. Assume the one you depend on today gets switched off, repriced, deprecated, or geofenced out from under you, because eventually one of them will. The teams that treat their model as interchangeable will keep shipping. The ones that married a single model will spend that week on r/ClaudeAI asking a US citizen to marry them for API access. One of those was a real post. It had 2,500 upvotes. Do not be the architecture that makes it relatable. --- # Agent memory is mostly hoarding > Stuffing every past message into context is not memory, it is hoarding. Real memory is knowing what to forget. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/agent-memory-is-mostly-hoarding Markdown: https://kachar.dev/blog/agent-memory-is-mostly-hoarding.md Published: 2026-06-11 Reading time: ~2 min Tags: ai, agents, context-engineering Cite as: Ilko Kacharov, "Agent memory is mostly hoarding", kachar.dev, June 11, 2026. https://kachar.dev/blog/agent-memory-is-mostly-hoarding ## Contents 1. [Accumulation is not recall](https://kachar.dev/blog/agent-memory-is-mostly-hoarding#accumulation-is-not-recall) 2. [What real memory needs](https://kachar.dev/blog/agent-memory-is-mostly-hoarding#what-real-memory-needs) 3. [Why this is the interesting problem](https://kachar.dev/blog/agent-memory-is-mostly-hoarding#why-this-is-the-interesting-problem) --- ![A cramped dark room crammed with stacked steel filing boxes, loose papers and reels of tape. One small drawer is pulled open and glows electric violet.](https://kachar.dev/posts/agent-memory-is-mostly-hoarding-hero.jpg) Strong take: most of what gets sold as "agent memory" is a database with good marketing. The pitch is seductive. Give the agent a memory store, let it write everything it sees, and it will get smarter over time. In practice what you usually build is a junk drawer that grows forever. Stuffing every past message and tool result into context is not memory. It is hoarding. Real memory is knowing what to forget! ## Accumulation is not recall Human memory is impressive less for what it keeps and more for what it quietly drops. You do not remember every sentence of every conversation, and that is a feature. The signal survives because the noise decays. Most agent "memory" does the opposite. It accumulates. Every run appends more, relevance drops, retrieval gets noisier, and the cost per call creeps up while quality creeps down. The agent is not remembering more. It is carrying more. > **The tell** > > If your agent gets slower and vaguer the longer a project runs, you do not have a memory system. You have an append-only log wearing a memory costume. ## What real memory needs The hard part is not writing to a store. Any database does that. The hard part is the editorial layer on top: deciding what is worth keeping, what should be summarized, and what should be dropped on purpose. * **Selective retention.** Keep the durable facts, decisions, and preferences. Let transient chatter expire. * **Compression over accumulation.** Ten messages that resolved into one decision should be stored as the decision, not the ten messages. * **Active forgetting.** Stale state should be removed deliberately, not just buried under newer state. An agent that cannot drop yesterday's wrong assumption will keep acting on it. ```ts // Memory is an editorial decision, not a write call. function remember(event: Event, store: Memory) { if (isDurable(event)) store.upsert(distill(event)) // keep the essence if (supersedes(event, store)) store.forget(event.topic) // drop the stale // everything else is allowed to decay } ``` ## Why this is the interesting problem Selective retention and deliberate forgetting are genuinely open problems, which is exactly why "remember everything" is such a popular shortcut. It demos well and ships fast. It just does not scale past the demo, because relevance and cost both move the wrong way as the store grows. So when someone shows you agent memory, ask the uncomfortable question: can it forget? Until the answer is yes, it does not remember. It accumulates. --- # Context is the new CMS > For twenty years teams ran a content management system. The next era runs on context management: what your AI knows before it writes a word. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/context-is-the-new-cms Markdown: https://kachar.dev/blog/context-is-the-new-cms.md Published: 2026-06-09 Reading time: ~2 min Tags: ai, context-engineering, building Cite as: Ilko Kacharov, "Context is the new CMS", kachar.dev, June 9, 2026. https://kachar.dev/blog/context-is-the-new-cms ## Contents 1. [Why this matters now](https://kachar.dev/blog/context-is-the-new-cms#why-this-matters-now) 2. [What good context engineering looks like](https://kachar.dev/blog/context-is-the-new-cms#what-good-context-engineering-looks-like) 3. [The shift in one sentence](https://kachar.dev/blog/context-is-the-new-cms#the-shift-in-one-sentence) --- ![A dark letterpress bed full of unlit metal type. A brass lens above it throws a single beam of electric violet onto one blank sheet.](https://kachar.dev/posts/context-is-the-new-cms-hero.jpg) For about twenty years, the center of gravity for software teams was the CMS. A content management system answered one question well: where does the content live, and who can change it. We built our whole craft around that question. That question is quietly becoming the wrong one. The new center of gravity is context management. Not where the content lives, but what the model knows about you before it produces a single word. The bottleneck moved up the stack, from storage to knowledge. > **The one-line version** > > A CMS stores what you wrote. Context management stores what the AI needs to write as you. ## Why this matters now A raw model is a brilliant stranger. It writes fluently and confidently about your business while knowing nothing about it. The gap between a generic answer and a useful one is almost entirely context: your brand voice, your constraints, last quarter's decisions, the three things you never do. You can close that gap two ways. You can paste the same background into every prompt forever, or you can build a system that carries it for you. The first feels productive. The second compounds. ## What good context engineering looks like The teams getting real leverage treat context as a first-class artifact, not an afterthought you stuff into a prompt. A few patterns I keep seeing: - **Knowledge lives once.** Brand guidelines, product facts, and past decisions are added in one place and referenced everywhere, not re-explained per conversation. - **Voice is trained, not described.** "Write in our voice" is a wish. A set of real examples the model can match against is a spec. - **Persistence beats recall.** The system should remember across people and sessions, so the newest teammate gets the same context the founder has. ```ts // Context is data you curate, not prose you retype. type Context = { brandVoice: Example[] // show, don't tell facts: KnowledgeItem[] // the things that are simply true about you guardrails: string[] // the three things you never do } ``` > **The trap** > > More context is not automatically better context. A model drowning in stale documents is worse than one given the five facts that matter. Curate ruthlessly. ## The shift in one sentence The question used to be "where is the content." The question now is "what does the model know before it starts." Teams that answer the second one well will make the first one feel like a footnote. That is the bet we are making at Juma, and it is the lens I will keep coming back to in this blog. --- # Your evals are the moat > Your prompts are not your moat. Your model is not your moat. The test suite that catches silent regressions is the thing nobody can copy. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/your-evals-are-the-moat Markdown: https://kachar.dev/blog/your-evals-are-the-moat.md Published: 2026-06-05 Reading time: ~2 min Tags: ai, evals, building Cite as: Ilko Kacharov, "Your evals are the moat", kachar.dev, June 5, 2026. https://kachar.dev/blog/your-evals-are-the-moat ## Contents 1. [Why this is the hard part](https://kachar.dev/blog/your-evals-are-the-moat#why-this-is-the-hard-part) 2. [What "real evals" means](https://kachar.dev/blog/your-evals-are-the-moat#what-real-evals-means) 3. [Why it compounds](https://kachar.dev/blog/your-evals-are-the-moat#why-it-compounds) --- ![A dark brushed-metal fortress in fog, ringed by a grid of fine lines glowing electric violet.](https://kachar.dev/posts/your-evals-are-the-moat-hero.jpg) Here is an opinion that tends to start arguments: your prompts are not your moat, and your model is not your moat. Your evals are. Anyone can copy a system prompt. People screenshot them and post them. Anyone can switch to the same frontier model you use, often by changing one line. Neither of those is defensible. What almost nobody will do is rebuild the test suite that tells you, on every change, whether you just made things quietly worse. ## Why this is the hard part LLM systems fail in a way traditional software does not: they regress silently. You tweak a prompt to fix one case, ship it, and three other cases degrade in ways no exception ever surfaces. There is no stack trace for "the answer got a little worse." The only thing standing between you and slow, invisible decay is a set of evals you trust. > **The silent regression** > > The dangerous bug in an AI product is not the one that throws. It is the one where the output is still plausible, still confident, and now wrong more often. Without evals you find out from churn, not from CI. ## What "real evals" means Not a vibe check in a chat window. I mean: - **A fixed set of cases** that represent what your users actually do, including the ugly inputs. - **Graders** that score outputs, whether by rules, by a model judge, or by a human rubric, consistently enough to compare runs. - **A number that moves** when quality moves, so a change is a measurement and not a feeling. ```ts // A change is only "safe" if the suite says so. const before = await runEvals(suite, currentSystem) const after = await runEvals(suite, proposedSystem) if (after.score < before.score) { throw new Error(`Regression: ${before.score} -> ${after.score}`) } ``` ## Why it compounds Prompts and models commoditize. Every quarter the gap between you and a competitor on raw capability shrinks, because the same models are available to both of you. Evals do the opposite. Every real failure you encode into the suite makes the next change safer and the product harder to clone. The test suite is the institutional memory of everything that ever went wrong, and that memory is genuinely expensive to rebuild from scratch. So if you are deciding where to spend a scarce week, spend it on the evals. The prompt you will rewrite ten times anyway. The suite is the asset. --- # A bigger model won't save you > Most 'we need a bigger model' complaints are context engineering problems in disguise. A smaller model with the right context beats a frontier model fed a junk drawer. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/a-bigger-model-wont-save-you Markdown: https://kachar.dev/blog/a-bigger-model-wont-save-you.md Published: 2026-06-02 Reading time: ~2 min Tags: ai, context-engineering, building Cite as: Ilko Kacharov, "A bigger model won't save you", kachar.dev, June 2, 2026. https://kachar.dev/blog/a-bigger-model-wont-save-you ## Contents 1. [The reflex and why it is wrong](https://kachar.dev/blog/a-bigger-model-wont-save-you#the-reflex-and-why-it-is-wrong) 2. [Where the leverage actually is](https://kachar.dev/blog/a-bigger-model-wont-save-you#where-the-leverage-actually-is) 3. [The economics](https://kachar.dev/blog/a-bigger-model-wont-save-you#the-economics) --- ![A dark workshop. On the left, tangled cables and loose papers pile up beside an idle steel engine. On the right, a small brass machine is fed by a single tray of three cards that glows electric violet.](https://kachar.dev/posts/a-bigger-model-wont-save-you-hero.jpg) Most of the time I hear "we need a bigger model," the real problem is not the model. It is what we are putting in front of it. A smaller model with the right files, the right tools, and clear constraints will routinely beat a frontier model fed a junk drawer of half-relevant context. The bottleneck is usually the window, not the weights. ## The reflex and why it is wrong When output quality is bad, the tempting move is to reach for more capability. Swap to the larger model, pay the higher bill, wait for the next release. Sometimes that helps. More often it papers over the actual issue, which is that the model never had what it needed to answer well in the first place. You can usually tell the difference with one question: would a smart human contractor, handed exactly the context you handed the model, get it right? If the answer is no, a bigger model will not save you. You have a context problem. > **The contractor test** > > Before blaming the model, ask whether a competent human given the same inputs would succeed. If they would not, fix the inputs, not the model. ## Where the leverage actually is In practice the wins come from boring context work, not capability upgrades: - **Retrieval that returns the five relevant things**, not the fifty vaguely related ones. - **Tools that let the model fetch what it needs** instead of guessing. - **Constraints that say what not to do**, which is often more valuable than more instructions about what to do. - **Pruning**, because a model drowning in stale documents performs worse than one given a tight, current set. ## The economics This matters beyond quality. Bigger models cost more per call and add latency on every request, forever. Better context is a one-time engineering investment that makes every call cheaper and better at the same time. You are trading a recurring tax for a fixed cost, and the fixed cost usually wins. None of this means capability is irrelevant. When you genuinely hit the ceiling of what the best context can do, reach for the bigger model. Just be honest that most teams are nowhere near that ceiling. They have a junk drawer, not a capability gap. --- # Spend your three innovation tokens on the product > Choosing Postgres and a boring monolith looks lazy. It is the opposite. Here is how I think about innovation budget as a CTO. Author: Ilko Kacharov (CTO & Co-founder, Juma Labs), https://kachar.dev/about Canonical URL: https://kachar.dev/blog/three-innovation-tokens Markdown: https://kachar.dev/blog/three-innovation-tokens.md Published: 2026-05-28 Reading time: ~2 min Tags: cto, architecture, building Cite as: Ilko Kacharov, "Spend your three innovation tokens on the product", kachar.dev, May 28, 2026. https://kachar.dev/blog/three-innovation-tokens ## Contents 1. [The math nobody runs](https://kachar.dev/blog/three-innovation-tokens#the-math-nobody-runs) 2. [What "boring" buys you](https://kachar.dev/blog/three-innovation-tokens#what-boring-buys-you) 3. [When to actually spend a token](https://kachar.dev/blog/three-innovation-tokens#when-to-actually-spend-a-token) --- ![A dark steel table between plain brick walls. Three brass tokens lie on it, and one glows electric violet.](https://kachar.dev/posts/three-innovation-tokens-hero.jpg) Choosing Postgres and a boring monolith looks lazy. It is not. It is one of the highest-leverage decisions a small team can make, and most people get it backwards. Here is the model I use. You get about three interesting-technology bets before complexity starts eating your speed. Three. That is the whole budget. Every exotic database, every bespoke message queue, every clever microservice boundary spends one of those tokens. ## The math nobody runs At six engineers, every token you spend on infrastructure is a token you did not spend on the product. And the product is the only thing that decides whether you get to have a year four. So spend all three on the thing customers actually feel. None on your infra. > **Boring is a feature** > > Boring technology has a property exotic technology does not: a decade of people hitting the same walls you are about to hit, and writing down how they got past them. ## What "boring" buys you - **Answers already exist.** Your weird production issue has a Stack Overflow thread from 2019. - **Hiring is easy.** You are not training every new engineer on a stack only you understand. - **Operations are predictable.** Boring systems fail in boring, documented ways. You are not Google yet. That is the point. Google earned its exotic infrastructure by first surviving on boring infrastructure long enough to have problems worth solving with exotic infrastructure. ## When to actually spend a token Spend a token when the exotic choice is the product. If your differentiator is real-time collaboration, spend a token on the hard sync problem. If it is search quality, spend it on the retrieval stack. Spend where the customer feels it. Everywhere else, be relentlessly, unfashionably boring. Future you, debugging at 2am with a sleeping team, will be grateful.