Qikly MCP Server

qikly

The problem: Your AI writes both the code and its tests. How do you know the tests are really valid?

One logged run. The test agent builds the test suite: it was told, from the spec's acceptance criteria, that a rate may not exceed 100. The coding agent never saw the acceptance criteria and discovered that rate limit from a failing test: it then added or rate > 100 by itself, so the 9 of 9 is code eventually passing the test suite after applying fixes in an automated repair loop.

Why use qikly?

A student who writes the exam paper, writes the answer key and then sits the exam will pass. That is what happens when one model is given the acceptance criteria and asked to produce both the implementation and the suite that checks it. Everything goes green, and the green can mean nothing. qikly takes the answer key away from the student: the coding agent never sees the acceptance criteria.

Start here

Before you start, three things:

1. Install qikly: pip install --upgrade qikly.

2. Set one API key, for steps 2 and 3: Google Gemini (free tier, no card), Anthropic or OpenAI. In the terminal you will run qikly from:

   export GEMINI_API_KEY="..."          # macOS and Linux
   $env:GEMINI_API_KEY = "..."          # Windows PowerShell
   

   Getting a key, and the other two providers. Step 1 needs no key; without one, step 2 leaves every answer for you to write.

3. Check your tests pass on their own, from your project's root folder: python -m pytest. qikly runs them the same way, so a missing package or the wrong Python version stops step 1 too.

Then run these from that root folder. Each takes one module or a whole folder, and the same path in every step:

qikly --score my_module.py       # 1. what your tests miss         free, no key
qikly --scaffold my_module.py    # 2. the drafted spec, to check   a fraction of a cent
qikly --tasks my_module.py       # 3. the run                      a cent or so

1. --score scores the tests you have. It plants faults in a copy of your code, one at a time, and names the ones your tests did not notice, each as a question about behaviour nothing checks. It finds your tests itself: the test_*.py and *_test.py files that import your code, searched from the project root. To name them instead, add --score-tests tests/. No model is called and nothing of yours is changed.

2. --scaffold drafts the spec, inputs_private/config/tasks/MY_MODULE_VERIFY.yaml, with everything it can read from your code filled in, and a data file of the values your tests already call it with. With a key set (Google Gemini, Anthropic or OpenAI), a spec agent then drafts the parts only an owner could write: a line on what the module is for, and an answer to each of step 1's questions as an acceptance criterion. It reads your README and docstrings, never your code, so it says what the code should do rather than what it does, and a test written from it can disagree with the code:

   #? What should apply_discount(100, 100) do?
   #? Answer below drafted by qikly from your README or docstrings: "...so a rate
   #? of 100 or more is rejected with ValueError". Keep it, or change it.
   - "apply_discount(100, 100) raises ValueError"
   

   Each answer says where it came from: a quote from your documentation, checked word for word, or CHECK where the agent judged it from the module's purpose because nothing written decides it. Run the draft as it is, or change any answer you disagree with, with or without quotes. What the documentation cannot decide is left as a TODO: answer it with an option's number where qikly found values by running your code (1: <value> for one that asks for a value, keep for what the code does now), or a sentence that names a value. A TODO you leave is skipped. Without a key, every answer is yours to write, and the requirements line is marked TODO: MODIFY THIS:. On a folder it writes one task file for each module step 1 asked about. Lines starting #? are notes for you, and neither agent reads them. This file is the one source of truth: change it and run step 3 again. qikly --validate my_module.py checks it, free, and shows each short answer as the criterion it becomes.

3. --tasks runs it, which needs an API key (Google Gemini, Anthropic or OpenAI). A test agent writes tests from your criteria before any code is shown to it, and a separate coding agent, which never sees them, repairs a copy of your code until they pass. Your files are never edited. It ends by pointing at qikly --take-tests my_module.py, which copies the new tests into your own tests folder, under qikly_generated/, rewritten to import your module rather than qikly's copy, and runs them on your code. A test that fails there is one to compare with its criterion: either your code or the test is wrong, and the report says which failures were written from a criterion and which are the test writer's own reading, to check.

Try it on a worked example first: docs/examples/ has a one-module project and a two-module folder, each with its task files as they look once answered, to compare yours against.

The task is named after the module, MY_MODULE_VERIFY, where _VERIFY marks a task that tests code you already have. You never need to type it: every step takes the path you gave --score, and a task written with --task-id is found by its path too. qikly --scaffold my_module.py --fresh writes MY_MODULE.yaml instead, a task in which qikly writes a new implementation of the same interface.

Other ways in

Good to know

• --score on its own is the one to try in a meeting. Point it at a module or a package, with the tests you already have. It plants one fault at a time in a copy of your code, runs those tests against each one, and names the faults nothing noticed. Both flags are needed: one says what to break, the other says what should notice.
• It tells you what your tests would notice changing. It cannot tell you whether the code was right to begin with, because it works by breaking code that is there, so anything the code never did is invisible to it. No model is called, nothing of yours is modified, and no code leaves your machine.
• It plants every fault it can, up to 60 in one module or 150 in a folder, and a sample when your tests are slow. --score-mutants 48 sets the number, and --score-seed 1 draws a different sample.
• --demo works in a throwaway demo/throwaway_<timestamp>/ folder it expects you to delete. It is for watching, not for building in. --example and --scaffold create a real project in the directory you are standing in, and those are the two to build from.
• --upgrade rather than a bare install, because the Skill ships inside the package: pip install qikly on a machine that already has an older one prints "Requirement already satisfied", changes nothing, and --install-skill then writes that older Skill.
• The quick start walks the same steps, then each command on its own.
• test.qikly.com is the two minute version of this page, and the one to send to somebody else.

New: an agent Skill. qikly --install-skill teaches Claude Code, Gemini CLI, Codex, Cursor or GitHub Copilot how to drive qikly. What the Skill contains.

Who it is for: a developer or team pointing an AI coding agent at a self-contained Python module that transforms data, for example an ETL step, a merge, a calculation or a validation routine, who does not want to trust a green suite when the same agent wrote both the code and the tests. It suits one module at a time in small to mid-sized repositories: when a test fails, only the files that failure names are loaded, so runs stay small and quick. It fits most naturally where verification already has to be independent, such as automotive, medical devices, fintech and defence: ADAS_HEADWAY, a bundled example, checks following distance from forward-radar samples. See What it is for.

What it is for

Built for self-contained Python modules that transform data, not for a large existing repository: ETL, merges, calculations, validation. That is the layer where a wrong answer looks like a right answer, and where a test written from the rule is the only thing that catches it.

Where it does not fit today: an existing large repository. PATCH prompts load only the files a FIX names, and while a large file is now excerpted rather than loaded whole, there is no cross-file index. See Where it fits today.

Use it from your coding agent

qikly runs as an MCP server, so an agent in any MCP host can start a run and read the result without you leaving the conversation. It is tested in VS Code and Claude Code so far. With uv installed there is nothing else to install. In Claude Code:

claude mcp add qikly -- uvx --from "qikly[mcp]" qikly-mcp

In VS Code, the Install qikly MCP badge at the top of this page writes the same command for you.

No qikly tool returns your acceptance criteria, on success or on failure. Your agent sees which tests failed and the pytest output, never the rule it broke.

docs/mcp.md has the four tools, other hosts, installing with pip instead, what to do when the server does not start, and how to keep the generated tests out of your agent's reach.

Or as a Skill, which teaches rather than calls

qikly --install-skill

That copies a Skill into .claude/skills/qikly in the directory you are standing in. A Skill is a folder your agent reads: it tells the agent when qikly is the right move, which line of a spec is a decision the coder needs and which is a consequence to withhold, what a stalled repair loop means, and which commands cost nothing. No server, no configuration.

It loads, and is then used correctly, in Claude Code, Gemini CLI, Codex and Cursor: first time in each, in a project with a rival testing skill installed beside it and a request that never mentioned qikly. The agent had a competing option and no hint, which is what makes that worth reporting. --install-skill agents, cursor or gemini write where those tools document their own folders, and copilot writes the instruction file GitHub Copilot reads instead, which is the one path here nobody has watched load yet.

The two are not alternatives. The MCP server is code, so it can enforce: no tool returns your criteria, and a test fails the build if that ever changes. A Skill is instructions, so it can only inform, and it says so itself. The withholding is enforced by the tool whether or not the Skill is installed. docs/skill.md has what is in it, how to check it loads, and the difference stated in full.

---

The idea

qikly takes the answer key away from the student. It generates a test suite from the acceptance criteria, then writes an implementation and repairs it against that suite until every test passes or a retry budget runs out, recording every failure, every piece of reasoning and every diff.

The part that makes the result mean something: the coding agent never sees acceptance_criteria. It gets the specification with that section stripped out, the same vague brief a developer works from, while test generation gets it in full. When a test fails, the agent sees pytest's output for that test and never the acceptance criteria. Without that asymmetry both sides read the same spec identically and every test passes first try, which proves nothing.

Purple is what the coding agent can see. Teal is what the standard is written from. They never touch. A run that never converges is still worth having: it exits non-zero, names the blocking tests, and keeps the same complete record. The purple arrows are the repair loop, and that is where almost all of a run happens: a failing suite sends the agent the failure text and nothing else, it produces a FIX and a PATCH, and the suite runs again, until the stage passes or the retry budget runs out. It never sees the criteria themselves, so it cannot write code shaped to a bar it was handed. tests/test_withholding.py fails the build if any call site lets one through.

A run works through three stages, integration then system then unit:

1. Integration and system tests are generated first, from the spec alone, before any code exists. They cannot see an implementation because there is not one yet.
2. The coding agent writes the implementation, from the spec minus the criteria.
3. pytest runs. On failure the model produces a FIX (failure summary, root cause, plan, and the files it intends to touch) and then a PATCH (a unified diff of only those files), applied all or nothing. Repeat until the stage passes or the budget is spent.
4. Unit tests are generated last, once real code exists for them to name. This is the only stage allowed to see the implementation.
5. Clearing a stage re-runs the earlier ones, so a later fix cannot silently break something that already passed.

Every arrow back into FIX carries the pytest error text and nothing else, which means the failing tests and not the specification they came from. Unit tests come last because they are the only ones that need to name real functions, which makes them the only stage allowed to read the implementation. Re-running the earlier stages after each success is what stops a later repair quietly breaking something that already passed.

Test generation sees the requirements, the input and output contract, and every acceptance criterion in full. It writes integration, system and unit tests against the standard.

The coding agent sees the same specification with the criteria section removed, plus the text of whatever test just failed. The same vague brief a developer usually works from.

Two ways to get a test suite, and what each can prove

Both are useful and they answer different questions. qikly is not purely one or the other: integration and system tests are written from the criteria before any code exists, the unit stage is written last from the code that just passed them, and --refine-criteria reads a converged implementation to propose criteria the first draft missed. Each of those reads the code on purpose, and none of them can question it.

What makes this different

The tests come from the standard, not from the code. This is the one that matters most. Every other AI test generator in this space writes its tests from an implementation that already exists, so it can only describe what the code already does. That makes an excellent regression harness, and it cannot tell you the code is wrong. qikly writes the integration and system suites from the acceptance criteria before any implementation exists, so the standard cannot have been shaped by the thing it judges.

The withholding is a mechanism you can watch. Not a prompt asking a model to ignore a section, and not a convention someone has to remember. One command prints what each side is given and the difference between them, offline and free:

qikly --explain <MY_TASK>     # e.g. qikly --explain MERGE_SALES

Eleven criteria go to test generation. Twelve lines are removed before the coding agent sees the same file. tests/test_withholding.py fails the build if any call site ever lets one through, including one added next year by someone who has never read this. It is a property of the code, and it takes thirty seconds to check.

What you get is an executable suite you keep. The output is pytest files and JUnit XML. Read them, run them, put them in CI, and when one fails in six months it fails for a reason you can inspect and argue with. A suite is a durable asset in a way a model's verdict is not: a verdict cannot be re-run against tomorrow's commit.

It helps you write the standard, not just check against it. --init and --scaffold turn existing code into a task, --criteria-from lifts criteria out of a ticket you already wrote, --generate-criteria drafts a first bar from requirements alone, and --check-criteria looks for two statements anywhere in the specification that no implementation could satisfy at once, including two acceptance criteria that disagree with each other.

Every run is reproducible, and the whole trail is kept. A run records the provider, the model, the settings and the version that produced it, next to every failing test, every FIX with its stated root cause, and every PATCH as a diff. You can read back exactly why a line of code exists: which assertion forced it, what the model concluded, and what it changed. The record survives a run that never converges, which is when you most want it.

"Why not just use two different models?"

It is the first thing most people ask, and it does help a little. It does not reach the underlying issue, because both models still read the same criteria and so both still write to them: the code is still built to satisfy the standard it is about to be judged by, and changing who types it does not change what they were shown. It is also a habit rather than a mechanism, and nothing checks the two stayed different.

The two compose nicely, incidentally, since qikly picks a provider and model per agent role. You can withhold and use two models.

"Why not just add a reviewer agent?"

The newer version of the same question, and the one worth answering carefully, because independent verification steps are now shipping in mainstream coding agents: a second agent, often from a different model family, reviews what the first one produced.

It helps, and it does not reach this. A reviewer given the same specification has read the same acceptance criteria, and resolves the same ambiguity the same way. It will catch a mistake that is visible from that context: an inconsistency, a requirement plainly skipped, an obvious bug. It cannot catch the case this tool is built for: a line that could be read two ways, read once, with both the code and the standard written from that single reading. Nobody is wrong, so nothing looks wrong.

The problem was never that nothing was checking. It is that everything checking had already seen the answer key. Withholding is what makes the check structural rather than one more opinion drawn from the same context, and it is enforced by a test rather than by an arrangement someone has to remember to keep.

There is a second difference, and it outlasts the run: a reviewer emits a verdict, and this emits a pytest suite you still have in six months.

"Doesn't a failing test give the criteria away?"

It gives away one case, and that is by design. When a test fails, the coding agent sees pytest's output for it: the test name, its source down to the failing line, its docstring if it has one, and the assertion error. In the case study, that a tax rate of 150 was accepted when it should not have been. Since a test's docstring often paraphrases the rule it was written from, assume the criterion behind a failing test is effectively visible once it fails.

Of the tests it has not failed, it sees the names and nothing else. Pytest lists every test it collected, so the agent knows a check called test_tax_exempt_zero_rate exists and is green; it does not see that test's body, docstring or assertions.

What the agent never receives is the specification itself. The acceptance criteria are stripped from the task before it is prompted, and no criteria text reaches a FIX or PATCH prompt.

Three things travel through a failing test, and they are not equally costly to close:

The markers stay in the suite on disk either way, so the coverage report and anyone reading the tests still see which criterion each one enforces.

That does not undo the separation, because independence is a property of how the suite was written, not of how much feedback the code's author receives afterwards. The integration and system suites are generated from the criteria before any implementation exists, and nothing the coding agent learns later can reshape a test that is already written. A repair that games the one visible failure still has the rest of the suite in its way, and earlier stages run again each time a later one clears.

It is the position a developer is in when CI goes red: they see what broke, not the test plan. The agent in the case study wrote tax_rate > 100 only because a test told it 150 was wrong. Had it been handed the criteria, it would have written the bound first time, and the green would have proved nothing.

Design rationale, and the harder problem of where acceptance_criteria comes from in the first place: docs/design_1_case_study.md, the first of three parts.

---

How to use the tools in this project

Four ways in, and the table under the quick start says which command each one needs:

1. Verify code you did not write. Supply an implementation through seed: and the suite is written from your acceptance criteria by an agent that never reads that code. A suite generated from the same context as the code is a model agreeing with itself. A module or a whole package: point seed.implementation at a directory and it is copied in under its own name, so from mypkg.money import to_cents resolves as it does in your own tree and every module in it can be repaired.

   The limit is worth saying plainly: qikly cannot know what the author of supplied code saw. Withholding is a property of a run qikly performed, not of a file you hand it. If the same person or model wrote that code with the criteria open, this gives you an independent suite, not an independent author. What it does give, always, is a suite that was not derived from the code, which is the half a code-derived generator cannot give you at all.

2. Start from a spec. No code yet: get a first implementation and the suite that justifies it, in outputs/, never in your source tree.

3. Bring your own tests. Seed any stage and the loop becomes a repair procedure rather than a generator.

4. Run a catalog unattended. Non-zero exit on any non-convergence, so a scheduler or CI job can run many specs and keep the reports.

The free commands, in detail

Three commands that make no model call, need no API key, and cost nothing. --explain is in Other ways in; this is what it and its two neighbours actually do.

qikly --explain MERGE_SALES   # what each side is shown, and the difference
qikly --validate              # check your task files: YAML, criteria, fixtures
qikly --explain MERGE_SALES --json
qikly --explain MERGE_SALES --html   # the same, as a page to share

--explain is the one worth running first. It prints the acceptance criteria that test generation receives, then the same task file as the coding agent receives it, then the diff: on MERGE_SALES, all eleven acceptance criteria are removed, along with the acceptance_criteria: key they hang off. It builds those strings through the same function a real run uses, so it shows the mechanism rather than a description of it.

Add --html and it also writes qikly_explain_MERGE_SALES.html: both views side by side with every withheld line highlighted, in one file that loads nothing from anywhere. Attach it to a pull request, put it in a slide, or screenshot it for a post.

--validate reads your task files and nothing else: that they parse, that acceptance_criteria is a list rather than one long string, that fixture paths resolve, and that criteria name values instead of adjectives. It is also available as a pre-commit hook, qikly-validate, deliberately the free check rather than the paid one.

What --demo looks like

The last row of Other ways in, in full. It needs a provider key:

export GEMINI_API_KEY=...     # PowerShell: $env:GEMINI_API_KEY = "..."
qikly --demo

Python 3.10+ and GNU patch; on macOS run brew install gpatch first. It runs a bundled task end to end in a throwaway demo/throwaway_<timestamp>/ folder, in about thirty seconds, writes nothing outside it, and expects you to delete the folder afterwards rather than build there.

That is a real run on gemini-3.5-flash-lite, 38 seconds, not sped up.

Tried it? Tell us what happened, whether it worked, stalled or never got past install.

Your own data, in full

Starting from nothing rather than from code you have: make the two directories, drop your fixture data in, write the task file, run it. Nothing is written into the package and nothing is written into your source tree. The walkthrough with the task file explained field by field is in docs/QUICK_START_ON_YOUR_OWN_DATA.md.

qikly --init                    # creates inputs_private/ and a starter task
qikly --tasks MY_TASK

Want to watch it work before writing anything? qikly --example adds a worked example that runs as it stands: a module, the two task files a scaffold of it produces, sample data, and the requirements and acceptance_criteria already written, each at the path a real task uses. It is the one command here that puts a file in your project root, my_metrics.py, and you can delete the lot when you are done. Which starter is which, --init against --example, is set out in the quick start below.

docs/QUICK_START_ON_YOUR_OWN_DATA.md has the rest: the task file field by field, getting the split between requirements and acceptance_criteria right (decisions in one, their consequences in the other, and the gap between them is the whole mechanism), lifting criteria out of a ticket you already wrote, seeding your own implementation or test suites, where each file is read from, and proposing the fixture rows a criterion needs before any test can reach it.

Two kinds of test data, from two different sources

Worth knowing before you prepare your fixture data, because it decides what your data has to contain:

The test-writing agent never sees your fixture data. It is handed the task file and, for unit tests, the code, and nothing else, so when it writes

invalid_prices = ['0', '0.00', '-5.00', 'abc', '']

those five values came from reading a criterion, not from a row in your files.

Your data still decides what gets proven. The system stage is the only one that runs the real pipeline over the real files. So a criterion no row reaches is checked in miniature by an invented row, and never checked on the path your code will actually take. The run goes green and one rule was never exercised end to end. That is why the bundled sample data carries awkward rows on purpose, and why there is a command for finding the gap:

qikly --propose-fixtures --tasks MY_TASK

It reports which criteria nothing in your data reaches and proposes a row for each, next to the criterion it was proposed for. It writes a file for you to read and copy from, and never edits your fixtures: a row is only right or wrong relative to the criterion it was proposed for, and fixtures that grow in whatever direction a model finds interesting stop resembling the data you actually process.

Running

python run.py                            # every task found
python run.py --tasks ETL_ADDRESS        # one
python run.py --tasks ETL_ADDRESS,ETL_EMAIL
python run.py --demo --tasks MERGE_STOCK # isolated, any task

Each task runs in its own process, concurrently, with console output prefixed [task_id]. Exit code is non-zero if any task did not fully converge.

Expect some runs to stall, by design. Roughly 8 runs in 10 finish with the code passing every integration and system test. Roughly 6 in 10 pass everything including unit tests. Nearly the whole gap between those two figures is the unit stage.

Those are round numbers because they were measured three times: a 427-run sweep, a 140-run sweep sixteen days later on the same tasks and settings, and a 400-run sweep after correcting the benchmark itself, when eight of the ten tasks turned out to be carrying acceptance criteria that no input row could trigger. All three landed inside each other's intervals. All three used gemini-3.5-flash-lite, a small cheap model chosen to make repeated sweeps affordable, so treat them as a floor. Three sweeps agreeing is worth more than any one of them's decimal places, so the decimal places are not quoted.

A run that exhausts its budget exits non-zero, names the tests that blocked it, and keeps the full record. It never reports success on code its own tests reject.

Thirteen example tasks ship with the tool across five domains, listed in docs/design_3_mechanism.md.

Measuring rather than producing. One run is an artifact, not a rate: the same task with the same seed converges on some runs and not others. To claim how often anything converges, repeat the sweep and read the interval:

python -m qikly.orchestrator.run_all --repeat 10

That writes an aggregate report with confidence intervals and groups the non-converging runs by what they got stuck on. Details in docs/design_3_mechanism.md.

When a run does not converge

A stall is a normal outcome, not a broken tool: the run exits non-zero, names the blocking tests, keeps the whole record, and ships nothing.

The first thing to try is a stronger model, which moves convergence more than any setting in this file and costs one environment variable. After that, in triage order: read the timeline report, check for a collection error (nothing ran at all), rule out a forgotten setting with --trends, look for the same patch repeating (a criterion fighting the model's priors), and run --check-criteria, --validate and propose_fixtures.

Each of those, with the signature to look for and the fix: docs/TROUBLESHOOTING.md.

Questions people ask before they start, including what runs before you commit an API key and whether anything leaves your machine: docs/FAQ.md.

Pointing it at code you already have, including which of your modules it may change and what a run says when the defect turns out to be in one it may not: docs/EXISTING_CODE_AND_HELPERS.md.

Output

Everything is namespaced by task_id so concurrent runs never collide:

Documentation truncated — see the full README on GitHub.