Offline and online evals

Offline evals test an AI feature on a fixed dataset before release; online evals score real production traffic after it.

Offline evals run on a golden dataset with known answers, so they are repeatable and can block a release. Online evals sample live conversations and score them without a reference answer, using a judge model, user feedback or business signals such as escalations to a human. Offline catches regressions; online catches what the dataset never contained: new kinds of questions, new documents, a model that changed silently.

Share: LinkedIn · X ·

Related terms

Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
Golden dataset: A fixed set of real inputs with expected answers or checks, used to test an AI feature after every prompt or model change.
LLM observability: Tracing every model call with inputs, outputs, latency, tokens, cost and the user outcome.
Prompt regression: An answer that used to be right goes wrong after a change to the prompt or the model.

← All terms

Free tools

Business idea validatorTen rubrics, a fixed-rule verdict and three cheap experiments. Free.Software cost estimatorYour project among 125 real products, with honest ranges.Engagement configuratorPick the work and its size, see hours, price and start date.SPF, DKIM and DMARC checkerGrade any domain’s email DNS, with records to paste.BIMI checkerRecord, DMARC, SVG Tiny PS logo and VMC or CMC, with an inbox preview.App store readiness checkWhat your web app lacks for the stores, TWA or Capacitor, files to copy.Build or buy calculatorSaaS against your own app over 1, 3 or 5 years, with the payback month.Runbook generatorA RUNBOOK.md for a system only you know, in two minutes, in your browser.SLA calculatorMonthly fee, response times, a downtime budget and contract clauses.Local model or API?Does the model fit your hardware, how fast, what it costs against an API.OpenAPI to MCP toolsPaste an OpenAPI spec, get narrow MCP tools and a server skeleton.KSeF invoice validatorCheck an FA(3) invoice against the official schema in your browser.Quoting time calculatorHours spent turning inquiries into quotes, and when an agent pays off.Multi-tenant SaaS calculatorWhich multi-tenant model, the work per release and the weeks to get there.Quote DeskDrop a customer inquiry, get lines matched to your price list. Free.AI Visibility DeskDoes Claude recommend your brand? Weekly runs of your buyers’ questions, free.List DeskClean an email list in your browser: dead domains, disposable, typos, duplicates.Eval DeskTest cases for your AI prompt, run after every change: what regressed, what got fixed.Dashboard DeskA growth dashboard created in your PostHog, plus what your tracking is missing.Signature DeskOne email signature template for the whole team, as HTML for Gmail, Outlook and Apple Mail.Phishing email checkerForward a suspicious email or upload the .eml file; three AI models and technical checks give a verdict.Email TestSend one email, check one-click unsubscribe, SPF, DKIM, DMARC and spam signals.Vitals DeskA daily lab check of your pages on a throttled phone: LCP, CLS and TBT, and an email when a page gets slower. Free.Legacy rewrite estimateLines of code to function points, hours and cost, with and without agents.

Want the fun version? The Corporate-speak Translator has a Polish dictionary of office phrases and what they really mean. Corporate-speak dictionary (Polish) →

Need this built, fixed or reviewed?

I build and rescue products with AI coding agents under rules: AI features in production, honest estimates, email that gets delivered.

Book a free 30-min call See price and start date

How I work

Przemysław Zalewski · 15+ years shipping software

“a true bias for action” client CTO, public LinkedIn recommendation

Confidential by default. NDA is standard. I never show your project, name or screenshots without written consent, which is why this site names no clients without permission.
Your code, your accounts. Code and data stay in your repository and accounts. AI tools run with training on your data turned off, and secrets never reach a model.
A handover you can use. Every project ends with the same checklist: a README that gets it running, one architecture diagram, written decisions, CI, a runbook and a recorded walkthrough. It is done when someone outside the project can run it from the docs.
Boring technology where it counts. I pick a stack you can hire for. Anything experimental needs your yes and an exit plan.
Approach matched to your risk appetite. Prototype, product or regulated: the level decides how much autonomy the agents get and how much review and validation each change needs.
Everything checkable. Every number on my pages links to its source. My tools are free, and the idea validator skill is open source (MIT).

My pledge

Reply within 24 hours. I reply within 24 hours on working days, personally, not a bot. Weekend messages: by Monday noon.
Late means free. If I am late, the first hour of work is free. You do not have to ask.
Scope in writing within 24 hours. A free 30-minute call with no company deck, then the scope and the price in writing within 24 hours. You decide.
The person who does the work. You talk to the person who does the work. No sales layer, no handoff.
Follow-up only with news. I follow up only when I have something new for you. Never a "did you see my offer".
Honest "not a fit". If I am not the right person, I say so on the call and point you to someone who is.

All principles, the AI policy and working hours →

Pricing · Security · Privacy and analytics opt-out · Trust center