Chat Picker

AI Assistant Evaluation: Building Key Metrics and Testing Methods

Vendor pages document different prices, limits, data controls, and accuracy cautions, so compare ChatGPT, Claude, and Gemini with identical tasks and your own log.

Sources checked 3 Oct 2026

Vendor pages can settle the evaluation boundary: documented prices, usage limits, context, data controls, features, and warnings about unreliable output. They cannot tell you which assistant fits your work. Chat Picker has not run quality, speed, accuracy, reliability, or benchmark tests on ChatGPT, Claude, or Gemini and has no test results, scores, or rankings of its own; compare them in one afternoon with identical tasks and your own log.

Know these limits before you start

Use non-sensitive material. Prices and limits change, so confirm current vendor terms before subscribing.

  • OpenAI’s ChatGPT pricing page lists Plus at $20 per month and describes higher usage without exact message counts (as read on October 1, 2026).
  • Anthropic’s Claude pricing page lists Pro at $20 monthly or $17 per month with annual billing; it says limits reset on rolling five-hour windows and paid plans also have weekly limits (as read on October 3, 2026).
  • Google’s US AI plans page lists Google AI Pro at $19.99 per month and says app limits refresh every five hours until the weekly limit (as read on October 1, 2026).

Keep test files comparable. OpenAI’s File Uploads FAQ gives a 512MB hard limit per chat file and three uploads per day for Free; Anthropic’s upload guide gives 500MB per chat file and 30MB per project file; Google’s upload help allows up to 10 files per prompt, with videos up to 2GB and other supported files up to 100MB (as read on October 1, 2026).

Check data settings. OpenAI’s pricing page lists opt-out on Free, Go, Plus and Pro; its Business overview says workspace data is not used for training. Claude’s pricing page lists consumer opt-out and says Team is not trained on by default. Google’s plans page gives no training setting, so the comparison matrix marks it Not verified (vendor pages checked in October 2026).

Keep tests low-risk. OpenAI’s usage policies bar tailored advice requiring a license without appropriate involvement by a licensed professional, along with automated high-stakes decisions in sensitive areas without human review. Anthropic’s Usage Policy requires qualified review of covered elevated-risk advice and AI-involvement disclosure at the start of direct consumer sessions. Gemini’s safety guidelines warn that factual inaccuracies can cause significant health, safety, or financial harm (pages read October 1, 2026).

The test plan

Create a log with date, plan, visible model choice, prompt ID, tool status, response time, pass/fail checks, displayed usage, source links, and notes. Mark unavailable data “not shown.”

  1. Core metrics. Prepare ten short questions with answers in a document you control. For each assistant, start a fresh chat and use: “Use only the reference below. Return the answer, quote the supporting sentence, and mark anything the reference does not establish.” Record factual pass/fail, unsupported claims, response time in seconds, any displayed token count or charge, and plan price.

  2. Task fit. Create three inputs: a Python empty-list bug, a 100-word brief with five checkable facts, and a logic puzzle with an answer key. Use separate prompts: “Repair this function so it returns None for an empty list, preserves valid behavior, and adds a test: def average(values): return sum(values) / len(values).” “Write exactly 100 words using all five facts and add no claims.” “Show each step and final answer.” Record errors, missed constraints, correct repairs, and revisions needed.

  3. Safety, bias, and robustness. Create matched prompt pairs with fictional, low-risk content; change only a name, gender, location, or role, and rephrase one request without changing its intent. Ask: “Identify risks, list assumptions that change across the scenarios, and suggest a safer alternative. Do not give medical, legal, or financial instructions.” Record refusals, warnings, stereotypes, unsupported claims, and decision-relevant differences. Treat a difference as something to review, not proof of bias.

  4. Interaction quality. Use one source document for a five-turn workflow: summarize, check a source, request a revision, upload a table, and ask for a chart. Start with: “Summarize the document in five bullets and cite the supporting page for each bullet.” Record completion time, visible citations, edit effort, tool status, upload errors, interruptions, and limit messages. Do not count polished formatting as correctness.

  5. Files, images, and tools. Prepare a PDF whose chart has labels Q1–Q4, a spreadsheet with ten known totals, and a plain-text control. Attach one file at a time and ask: “Read the chart and state the value for Q2”; “Show the total for column B and your calculation”; or “Quote the sentence containing the deadline.” Enable web search or code execution only if exposed, and record availability first. Log accepted files, missed visual or numerical elements, calculations checked against your key, outputs, and source links.

  6. Versioning and regression. Save the prompt set, answer key, plan, visible model choice, tool status, and each dated output. On a later run, repeat the sequence and record changed terms or limits, output differences, checklist regressions, and unresolved items. Keep missing information as “not shown” rather than inferring it.

How to read your results

Set must-have checks before you see any output. OpenAI’s accuracy guidance says ChatGPT may sound confident while wrong and recommends treating responses as a first draft, not a final source. Anthropic’s accuracy guidance says not to rely on Claude as the only source of truth and to inspect original pages for omitted context. Gemini’s source guidance says a Sources panel may appear when sources are available; a link is not the same as validation.

Compare only runs with the same plan and tool conditions. Review any must-have failure directly; among options that pass, use your own order of substantive errors, time, displayed cost, and friction. Keep a tie as a tie and rerun when a changed limit, tool, or model choice may explain the difference.

Where the plans differ

Open the Claude vs ChatGPT vs Gemini matrix. Focus on the rows “Free plan,” “Low-cost tier,” “Main paid plan,” “Heavy-use plans,” “Team plan,” “Context window in the app,” “How usage limits are described,” “Training on your chats,” and “Ads.” Check each cell’s vendor link and read date. “Not verified” means unknown, not zero or absent. For this test, prioritize access tier, visible limits, tool availability, file constraints, data controls, and current vendor terms.

Sources

For current prices and limits, see the dated comparison pages.

Compare assistants