AI Chat Comparison: Coding Differences Across ChatGPT, Claude, Gemini
A practical afternoon test plan for comparing coding work in ChatGPT, Claude, and Gemini using vendor-documented limits and your own log.
Sources checked 2 Oct 2026
Vendor documentation can settle practical coding-workflow differences: plan costs, upload and project limits, data controls, accuracy warnings, and API billing. It cannot settle which assistant will produce better code for your repository, and Chat Picker has not tested ChatGPT, Claude, or Gemini. This rewrite removes the older page’s statistics and test results that had no sources.
Know these limits before you start
Start with the account and plan you can actually use. As read on October 1, 2026, OpenAI’s Plus page lists ChatGPT Plus at $20/month, Anthropic’s pricing page lists Claude Pro at $20/month billed monthly, and Google’s US plans page lists Google AI Pro at $19.99/month. These are plan prices, not measures of coding quality.
File rules can determine whether the test is fair. OpenAI’s File Uploads FAQ gives files in a GPT or ChatGPT conversation a 512MB hard limit and caps text and document files at 2M tokens; spreadsheets are limited to approximately 50MB. Anthropic’s file-upload help lists 500MB per chat file and 30MB per project file. It analyzes text and visuals in PDFs of 100 pages or fewer, but text only from 101 to 1,000 pages. Google’s Gemini Apps file help allows up to 10 supported files per prompt, or one code folder or GitHub repository with up to 5,000 files and a 100MB maximum. These pages were read October 1, 2026.
App context is not directly comparable. Anthropic’s pricing page says Claude offers up to 1M tokens on every plan, varying by model; OpenAI’s pricing page gives context figures by plan and model type; Google’s plans page states no app token figure. These pages were read October 1, 2026.
Data controls and throttling can invalidate a comparison. OpenAI documents an opt-out for Free, Go, Plus and Pro. Anthropic documents opt-out for Free, Pro and Max, and says Team is not trained on by default. Google’s plans page does not state a training setting. OpenAI and Anthropic do not publish exact message counts on those pricing pages, while Google says compute-based limits refresh every five hours up to a weekly limit. These entries were read October 1, 2026.
Do not treat fluent code as evidence. OpenAI’s accuracy note says ChatGPT may be wrong and sound confident; Anthropic’s help article says Claude can mislead and should not be the only source of truth; Google’s guidelines describe LLM outputs as probabilistic and limited by training data. For the security test, OpenAI’s usage policies say bypassing safeguards can mean loss of access, Anthropic’s Usage Policy calls for relevant human expertise in elevated-risk uses, and Google’s guidelines restrict dangerous instructions. These are vendor rules, not legal advice; the pages were read October 1, 2026.
The test plan
Create one log with the task, input version, product and plan, model label shown, timestamps, evidence, failed checks, upload or tool errors, and displayed cost. Use a fresh chat for each task and save the exact prompt.
-
Set a correctness benchmark. Prepare
inventory.py, visible tests and held-back tests. Give this instruction: “Implementparse_inventory(lines)so each nonblankitem countline becomes a sorted dictionary entry, negative counts raiseValueError, and inputs remain unchanged. Run the visible tests and explain your choices.” Check behavior against both test sets. Log case-level results, errors and unsupported claims. -
Test debugging. Prepare
chunker.pyand failing tests showing that empty input yields an extra empty chunk and that splitting a nonempty list repeats the boundary item. Say: “Diagnose the root cause before editing, make the smallest interface-preserving fix, and add regression tests.” Look for a cause supported by code or a test, then check the patch against the public interface. Record the proposed cause, patch, tests and contradictions. -
Test project comprehension. Prepare a toy repository with
api.py,parser.py,settings.py, tests and a README. Say: “Add optionalwarehouse_idto parsed records, pass it through the API response, read its default fromsettings.py, update tests and documentation, and list every changed file with a reason.” Look for dependency tracing and consistent updates across layers. Record files inspected and changed, missed references and the full test outcome. -
Test API integration. Prepare
orders.jsonforPOST /orders, with requiredskuandquantity, optionalcouponand a rate-limit response described in the file. Ask for a Python client that validates input, retries that response once and returns typed errors without sending real orders. If an API or tool path is available, run the same client against a local mock; otherwise mark that path untested. Record execution, protocol errors, unsafe requests and displayed cost. -
Test security review. Prepare an authorized toy repository containing a hard-coded fake secret, SQL built by string interpolation, unsafe archive-path joining and disabled certificate verification. Say: “Identify each risk, explain its impact, propose the smallest safe patch, and add a regression test. Do not run attacks or contact external systems.” Record findings, misses, false alarms, patches and policy concerns.
-
Run an unseen task. Prepare a final feature request for
report.py: add optional page size and offset while preserving current output when neither is supplied. Require assumptions, changed files and test results. Apply your predeclared correctness and security gates, then log effort, file coverage, tool success, traceability and plan fit. Do not present the composite as a published benchmark.
How to read your results
Review the log with product names hidden. Treat executable tests and reproduced patches as evidence; treat prose, confidence and citations as claims to check. If an answer cites a line that does not support it, mark the evidence check failed.
Set must-have gates before choosing: required correctness, safe handling, file fit, data setting and access under the intended plan. Then compare logged effort, tool success, traceability and displayed cost. If using a composite, define weights before the run and keep every component visible; it is your decision aid, not an external benchmark. Choose only among options that clear the gates, or change the task or plan if none do.
Where the plans differ
Use the Claude vs ChatGPT matrix to check the rows that can change this test: Free plan, Low-cost tier, Main paid plan, Heavy-use plans, Team plan, Context window in the app, How usage limits are described, Training on your chats, Ads and API prices. For a direct three-way comparison, use the three-assistant matrix.
Read the vendor source and date in each cell. Treat “Not verified” as blank rather than filling the gap by inference, and confirm the vendor page before subscribing because plans and prices change.