Comparison, checked October 2026
Prompt tests without a config file
Eval Desk is a free AI prompt testing tool by Przemysław Zalewski, an independent engineer. This page compares it with promptfoo, the open-source CLI for LLM evals.
promptfoo is the stronger tool if you write code and want evals in CI. Eval Desk is for the person who changes the prompt by hand: paste it, write cases in a form, press run, see what regressed.
At a glance
| Feature | Eval Desk | promptfoo |
|---|---|---|
| Setup | Sign in by email link, paste the system prompt | Install the CLI (npx promptfoo@latest init), write a YAML config |
| Where it runs | Hosted on ciunkos.com, EU storage | On your machine or your CI; web viewer with promptfoo view |
| Models | Claude Sonnet 5.5 and Claude Haiku 5.5 | Many providers: OpenAI, Anthropic, Google, AWS Bedrock, Ollama, custom APIs |
| Checks | Contains, does not contain, valid JSON, max length, plain-language criteria graded by a second model | Many assertion types, incl. contains, is-json, javascript, llm-rubric; any can be negated with not- |
| Repeated runs | 1 or 3 per case; cases that pass some runs and fail others are flagged as unstable | --repeat <n> runs each test n times |
| Between prompt versions | Lists cases that regressed or got fixed since the last run, next to the prompt diff | Side-by-side output matrix of prompts and models in the web viewer |
| CI | None | Documented for GitHub Actions, GitLab CI, Jenkins, CircleCI and others |
| Code | None | YAML config; JavaScript or Python for custom checks |
| Price | Free: 1 assistant, 25 cases, 200 model calls a month | Community edition free (MIT license); Enterprise and On-Premise priced on request |
Where promptfoo is the better choice
- You want evals in CI. promptfoo runs from the command line and documents CI setups for GitHub Actions, GitLab CI, Jenkins and more, so a failing prompt can block a merge. Eval Desk has no CI hook.
- You test more than Claude. promptfoo talks to OpenAI, Anthropic, Google, AWS Bedrock, local models through Ollama and custom HTTP APIs. Eval Desk runs Claude Sonnet 5.5 and Haiku 5.5 only.
- You want it open source and local. The README says promptfoo "remains open source and MIT licensed" and that evals can run fully on your machine. Eval Desk is a hosted app.
- You need many cases, custom checks or red teaming. promptfoo has JavaScript and Python assertions, CSV and file-based test sets, and a red-teaming mode. Eval Desk stops at 25 cases and five check types.
Ownership: the promptfoo README says "Promptfoo is now part of OpenAI" and that it "remains open source and MIT licensed".
Where Eval Desk is easier
- No code, nothing to install. A form for the prompt and a form per case. No YAML, no API key, no Node.js.
- Hosted history. Every prompt version and run is saved, so "what changed since Tuesday?" is one click, not a folder of result files.
- Regressions next to the diff. After each run you see which cases broke and which got fixed since the previous run, with the prompt diff that caused it.
- Unstable cases flagged. Run each case 3 times and the cases that pass some runs and fail others are marked unstable instead of showing up as a regression or a fix.
- Criteria in plain language. Write "does not promise a delivery date" and a second model grades it (LLM-as-judge), with the reason shown for every grade.
Limits of Eval Desk
- Claude models only: Sonnet 5.5 and Haiku 5.5. The grader is always Sonnet 5.5.
- No CI hook and no API: runs start from the browser.
- 25 test cases and 200 model calls a month on the free plan. A case is one call, plus one when it has criteria for the grader, so a full run of 25 cases with criteria uses 50 calls.
- One assistant (one system prompt) per account, no tracing of production traffic, no tool calls or RAG pipelines.
- Results are kept for 180 days.
Moving your cases to promptfoo later
Eval Desk has no export today. The cases are short, so moving them is copying: open each case in the form (or keep the spreadsheet you pasted them from) and write one CSV row per case. promptfoo reads test cases from CSV: column headers become prompt variables and __expected1, __expected2 hold the checks.
| In Eval Desk | In a promptfoo CSV |
|---|---|
| User message | a variable column, e.g. message |
| Must contain | contains: 14 days |
| Must not contain | not-contains: guarantee |
| Valid JSON | is-json |
| Max characters | javascript: output.length <= 600 |
| Criteria for the grader | llm-rubric: does not promise a delivery date |
message,__expected1,__expected2
"Can I return shoes after 20 days?","contains: 14 days","llm-rubric: does not promise an exception to the return window"Your system prompt goes into the promptfoo config as the prompt, with {{message}} where the user message belongs. Check the promptfoo docs for the current syntax before you rely on it.
Need evals in CI for a product in production?
When a browser tab is no longer enough, I set up evals in CI that block a bad prompt before deploy, with tracing and cost caps, in promptfoo or your own stack. More on the AI developer page.
I reply within 2 working days. Price and start date →
Questions
Is Eval Desk a full replacement for promptfoo?
No. It covers the first step: a regression test set for one Claude-based assistant, run by hand after prompt changes. promptfoo covers CI, many providers, custom assertions and red teaming.
Is promptfoo free?
Its pricing page lists a Community edition as "Free Forever", with Enterprise and On-Premise plans priced on request. The code is MIT licensed.
Can I export my test cases from Eval Desk?
Not yet. You can copy each case from the form; the table above shows where each field goes in a promptfoo CSV.
Why run each case 3 times?
The same prompt can pass a case in one run and fail it in the next. With 3 runs, Eval Desk marks such cases as unstable, so noise is not mistaken for a regression or a fix. promptfoo can repeat tests with --repeat.
Does Eval Desk work with GPT or Gemini?
No. It runs Claude Sonnet 5.5 and Claude Haiku 5.5. For other providers promptfoo is the better fit.
Sources, checked October 2026
- promptfoo docs: introduction
- promptfoo on GitHub (README: license, OpenAI)
- promptfoo pricing
- promptfoo docs: CI/CD
- promptfoo docs: providers
- promptfoo docs: test cases from CSV
- promptfoo docs: assertions
- promptfoo docs: command line (--repeat, view)
- promptfoo docs: prompts and variables
Facts about promptfoo come from its own site and repository on the date above. Products change; if something here is out of date, the linked pages win.
