Chat Picker

AI Chat Tool Reliability: How to Check Factual Accuracy Yourself

Run a repeatable afternoon test of ChatGPT, Claude, and Gemini by checking claims against original sources, auditing citations, and logging every unsupported statement.

Sources checked 2 Oct 2026

Check factual accuracy by running the same source-grounded tasks in ChatGPT, Claude, and Gemini, then checking every material claim against the original evidence. The vendors’ own pages establish the need for that step, not a comparative ranking: OpenAI warns that ChatGPT can be incorrect or misleading, Anthropic documents similar risks in Claude, and Google says Gemini’s outputs reflect training-data limits and may sometimes violate its guidelines. Chat Picker has not tested the assistants for accuracy or reliability, so this page provides a repeatable test plan rather than scores, rankings, or a winner.

Know these limits before you start

OpenAI says that, without search, ChatGPT responds from information learned during training. With search or deep research, it can reach and cite current web sources, but access may fail because of technical problems, paywalls, or robots.txt preferences. OpenAI tells users to verify quotations, data, technical information, and references to external documents.

Anthropic says Claude may be confused by questions about current events and can produce convincing-sounding quotations that are not grounded in fact. It recommends checking original websites because Claude’s synthesis may omit context or misinterpret information.

For Gemini Apps, Google says a Sources button appears when sources are available and can open public web pages, uploaded files, or connected Workspace material. That tells you where to inspect evidence, not whether the response is correct.

As read on October 1, 2026, OpenAI’s pricing page does not publish exact message counts, Claude’s pricing page describes usage resets over a rolling five-hour window plus weekly limits for paid plans, and Google’s Gemini Apps limits page says usage refreshes every five hours until the weekly limit is reached. Tool access can also depend on your plan, so record the plan and enabled tools instead of assuming the conditions are equal.

Data settings belong in the log too. OpenAI lists an opt-out for Free, Go, Plus, and Pro; Anthropic lists an opt-out for Free, Pro, and Max and says Team is not trained on by default. The Gemini plans page read did not state a training setting, so Chat Picker leaves that point unverified.

For a low-stakes test, avoid personal medical, legal, or financial decisions. OpenAI’s usage policies address licensed involvement for tailored medical or legal advice and human review for automated high-stakes decisions. Anthropic’s Usage Policy calls for qualified review in elevated-risk cases. Google’s safety guidelines list medical errors, physical-safety risks, and inaccurate news about ongoing violence among harmful inaccuracies.

The test plan

  1. Build a source packet. Choose a public document or a non-sensitive document you are authorized to test, and keep both the original and a plain-text copy. Give each assistant this instruction: “Answer only from the attached sources. For every factual claim, provide an exact supporting passage and its page or section. If the sources do not support a claim, write ‘Not found in sources.’” Record the source location, plan, tool setting, and full answer.

  2. Test a dated update. Prepare an official update relevant to your work and, when available, its underlying notice. Use this prompt: “Using the attached update, state what changed, when it took effect, and what the source does not say. Cite the exact passage for every point.” Check the publication date, effective date, organization, and scope against the originals. Log unsupported additions, omissions, and links that do not resolve.

  3. Audit the citations. Reuse an answer containing citations and ask: “For each factual sentence, identify the source and exact passage that supports it. Mark any sentence without support as unsupported.” Open every cited page and record whether the citation exists, opens, matches the named entity and date, and supports the full sentence. A working link is not automatically a pass.

  4. Run a real workflow. Use a low-stakes task such as turning a policy document into a brief. Give this instruction: “Using the attached document, produce a brief for a colleague. Separate direct facts, your inferences, and unresolved points. Attach a source location to every fact, and do not fill gaps with assumptions.” Record factual errors, omissions, unsupported additions, and whether the response followed the required format.

  5. Repeat under recorded conditions. Run the same packet prompts with the same plan and tool settings. If the plan lets you disable and enable web search, run both states; otherwise record that the control is unavailable. Note conflicting answers, cap warnings, inaccessible files, and tool failures separately from factual errors.

How to read your results

Use fixed verdicts for each claim: supported, partly supported, contradicted, unsupported, or unverifiable. A claim is supported only when the original source establishes the same entity, relationship, date, number, and scope. Record citation quality separately from claim accuracy: a real citation can still fail to support a sentence, while an inaccessible page is unverifiable rather than automatically false.

Weight errors by the consequences in your workflow. A serious error in a date, eligibility rule, or safety instruction may matter more than several harmless omissions. Define an acceptable result before testing, and compare assistants only under matching plan and tool conditions.

Do not turn a small afternoon sample into a universal hallucination rate or use it to infer how model architecture affects factuality. Your log supports a decision about your workflow, not a general ranking.

Availability is a separate issue. For example, Google’s AI Studio status page covers Google AI Studio and the Gemini API, not the consumer Gemini app. Record an access failure as an availability or usage-limit event rather than labeling the underlying fact wrong.

Where the plans differ

Open the ChatGPT, Claude, and Gemini comparison matrix and focus on the rows that define your test conditions:

  • Free plan and included features
  • Paid individual tier and price
  • Heavy-use options
  • Team plan and seat rules
  • Context window in the app
  • How usage limits are described
  • Training on chats
  • Ads

Each matrix figure links to its vendor source and the date it was read. Figures that could not be confirmed are marked “Not verified” and left blank, following Chat Picker’s sourcing method. Confirm the current vendor page before subscribing, then record your selected plan and settings in the test log. Those differences define the conditions of your test; they do not prove an accuracy advantage.

Sources

For current prices and limits, see the dated comparison pages.

Compare assistants