ciunkos.com/Glossary

Coding with agents

Holdout scenarios

End-to-end user scenarios kept outside the codebase, so the coding agent never sees them and cannot write code that only passes the check.

A test stored in the repository can be read, and rewritten, by the agent that is supposed to pass it. The StrongDM software factory describes the fix: scenarios, end-to-end user stories stored outside the codebase “similar to a holdout set in model training”, validated against the running software, often by an LLM as a judge. Success becomes a measured share of scenarios that satisfy the user instead of a green test suite.

The idea is the same as a golden dataset for evals: the people who build the system do not tune it on the examples used to grade it. Scenarios usually run against digital twins of external services, and they are the main defense against reward hacking in a software factory.

Sources

Share: LinkedIn · X ·

Related terms

  • Reward hacking (coding agents): When an agent makes the check pass without solving the task: deleting or weakening tests, special-casing test inputs or faking results.
  • Digital twin (service virtualization): A behavioral copy of an external service, such as a payment or identity API, that agents and tests can call safely, repeatedly and at any volume.
  • Golden dataset: A fixed set of real inputs with expected answers or checks, used to test an AI feature after every prompt or model change.
  • Software factory (AI agents): A pipeline where agents write and fix most of the code while people write specs, set guardrails, design the checks and decide what ships.
  • LLM as judge: Using a model with a rubric to grade another model’s output.
  • Validation loop (agent backpressure): Commands a coding agent runs after every change (tests, types, lint, build) and fixes against until they pass, so it checks itself before a person looks.

← All terms

Free tools

Business idea validatorTen rubrics, a fixed-rule verdict and three cheap experiments. Free.Software cost estimatorYour project among 125 real products, with honest ranges.Engagement configuratorPick the work and its size, see hours, price and start date.SPF, DKIM and DMARC checkerGrade any domain’s email DNS, with records to paste.BIMI checkerRecord, DMARC, SVG Tiny PS logo and VMC or CMC, with an inbox preview.App store readiness checkWhat your web app lacks for the stores, TWA or Capacitor, files to copy.Build or buy calculatorSaaS against your own app over 1, 3 or 5 years, with the payback month.Runbook generatorA RUNBOOK.md for a system only you know, in two minutes, in your browser.SLA calculatorMonthly fee, response times, a downtime budget and contract clauses.Local model or API?Does the model fit your hardware, how fast, what it costs against an API.OpenAPI to MCP toolsPaste an OpenAPI spec, get narrow MCP tools and a server skeleton.KSeF invoice validatorCheck an FA(3) invoice against the official schema in your browser.Quoting time calculatorHours spent turning inquiries into quotes, and when an agent pays off.Multi-tenant SaaS calculatorWhich multi-tenant model, the work per release and the weeks to get there.Quote DeskDrop a customer inquiry, get lines matched to your price list. Free.AI Visibility DeskDoes Claude recommend your brand? Weekly runs of your buyers’ questions, free.List DeskClean an email list in your browser: dead domains, disposable, typos, duplicates.Eval DeskTest cases for your AI prompt, run after every change: what regressed, what got fixed.Dashboard DeskA growth dashboard created in your PostHog, plus what your tracking is missing.Signature DeskOne email signature template for the whole team, as HTML for Gmail, Outlook and Apple Mail.Phishing email checkerForward a suspicious email or upload the .eml file; three AI models and technical checks give a verdict.Email TestSend one email, check one-click unsubscribe, SPF, DKIM, DMARC and spam signals.Vitals DeskA daily lab check of your pages on a throttled phone: LCP, CLS and TBT, and an email when a page gets slower. Free.Legacy rewrite estimateLines of code to function points, hours and cost, with and without agents.

Want the fun version? The Corporate-speak Translator has a Polish dictionary of office phrases and what they really mean. Corporate-speak dictionary (Polish) →

Need this built, fixed or reviewed?

I build and rescue products with AI coding agents under rules: AI features in production, honest estimates, email that gets delivered.

Przemysław Zalewski

How I work

Przemysław Zalewski · 15+ years shipping software

“a true bias for action” client CTO, public LinkedIn recommendation

My pledge

All principles, the AI policy and working hours →

Pricing · Security · Privacy and analytics opt-out · Trust center