AI Assistants Across Software Development: From Requirements to Test Cases
An afternoon test plan for comparing ChatGPT, Claude, and Gemini across the software-development lifecycle while recording each response against your own acceptance rules and documented limits.
Sources checked 2 Oct 2026
Vendor pages for ChatGPT, Claude, and Gemini document different prices, upload limits, plan settings, and accuracy cautions, but they do not establish which assistant will fit your software-development workflow. Chat Picker has not tested the three assistants for requirements, design, coding, review, testing, or documentation, so this page gives you a controlled afternoon test instead.
Know these limits before you start
Use a synthetic repository or a redacted copy, not credentials, customer data, or unreviewed secrets. Keep the same packet and acceptance rules for all three assistants.
As read on October 2026, OpenAI's ChatGPT pricing page lists Plus at $20/month, billed monthly; Anthropic's Claude pricing page lists Pro at $20/month when billed monthly or $17/month on annual billing with $200 up front; and Google's US AI plans page lists Google AI Pro at $19.99/month.
App context limits need separate labels. OpenAI's pricing page lists 128K for Instant models and 400K for reasoning models on Pro; Anthropic's pricing page says Claude offers up to 1M on every plan; and Google's plans page gives no app token figure. All were read in October 2026.
File boundaries can stop a run early. OpenAI's File Uploads FAQ says every conversation file has a 512MB hard limit and Free users have three uploads per day. Anthropic's file-upload help lists 500MB per chat file and 30MB per project file. Google's file-upload help allows up to 10 supported files per prompt, with each nonvideo file up to 100MB. These figures were also read in October 2026.
For model training on chats, OpenAI's ChatGPT pricing page offers an opt-out on Free, Go, Plus, and Pro. Anthropic's Claude pricing page offers one on Free, Pro, and Max and says Team is not trained on by default. The Google plans page states no matching setting.
Build verification into the run. OpenAI's accuracy guidance says ChatGPT can be incorrect or misleading and may sound confident. Anthropic's incorrect-response guidance says not to rely on Claude as the only source of truth. Google says Gemini outputs are probabilistic and may show sources for parts of a response.
For elevated-risk work, Anthropic's Usage Policy calls for relevant human expertise and makes users or organizations responsible for accuracy and appropriateness. OpenAI's Usage Policies say its rules do not replace professional duties.
The test plan
Prepare a synthetic repository containing requirements.md, api.md, app/export.py, tests/test_export.py, and README.md. Include rules for validating an ISO date, exporting rows to CSV, and handling an empty input. Add one ambiguous requirement and tests that initially fail. Give every assistant the same files and a fresh chat.
-
Requirements and user stories. Input:
requirements.mdand the ambiguous requirement. Instruction: “Turn this file into user stories. For each, write acceptance criteria, edge cases, dependencies, and open questions. Quote the source sentence for every assumption and label anything not in the file.” Look for traceability, invented requirements, and unresolved ambiguity. Record covered criteria, unsupported assumptions, and questions needing an owner. -
Architecture and technical specification. Input: the requirements, repository tree, and
api.md. Instruction: “Create a technical specification with components, interfaces, data formats, error handling, security boundaries, testability, and requirement IDs. Mark assumptions and unresolved questions; do not invent external systems.” Look for internal consistency and links to requirements. Record each assumption, missing interface, and decision that needs review. -
Code and implementation. Input:
app/export.pywith a stubbed export function andtests/test_export.pycovering date validation, CSV output, and empty input. Instruction: “Implement the stated behavior, change only necessary files, preserve the public interface, and return a unified diff. If you run tests, include the exact command and output; otherwise say the tests were not run.” Look for seeded-test failures, unrequested edits, and unsupported dependencies. Record changed files, corrections needed, and whether execution evidence was shown. -
Code review and refactoring. Input: the original file and candidate diff. Instruction: “Review this diff for correctness, security, maintainability, and test gaps. Separate required fixes from optional suggestions, cite file and line references, and do not rewrite unrelated code.” Look for actionable comments tied to the supplied code. Record accepted and rejected findings, plus edits required before acceptance.
-
Test cases and QA. Input: requirements, implementation, and seeded tests. Instruction: “Generate positive, negative, boundary, permission, and recovery test cases in a table with test ID, precondition, action, expected result, requirement link, and priority. Do not claim that any test was executed.” Look for requirement coverage and realistic expected results. Record duplicate cases, gaps, and cases needing product clarification.
-
Documentation and maintenance. Input: the accepted diff, API notes, and any test output. Instruction: “Update the README for installation, configuration, operation, and troubleshooting. Return only changed sections and list every claim you could not verify from the supplied files.” Look for instructions that match the implementation and unsupported setup claims. Record stale sections, missing maintenance details, and verification needed.
After each run, log the assistant and plan, exact prompt, file version, elapsed time, tool actions, requirement IDs covered, errors found, corrections required, limit messages, and your pass/fail decision.
How to read your results
Set pass/fail rules before starting: no critical acceptance criterion omitted, factual claims traceable to the supplied files, candidate code passing the seeded tests when you run them locally, no unapproved file changes, and documentation matching behavior.
Group failures by severity, rework required, tool friction, and limit events. Do not turn the log into a universal ranking; your weights depend on the job. Re-run only failures with a tighter instruction and keep both outputs. That shows prompt sensitivity in your workflow, not a general capability score.
Where the plans differ
The Claude vs ChatGPT matrix separates Free plan, Low-cost tier, Main paid plan, Heavy-use plans, Team plan, Context window in the app, How usage limits are described, Training on your chats, and Ads. For Gemini, check the Gemini vs ChatGPT matrix and the three-way matrix.
Treat Not verified and blank cells as unknown, not zero. Prices are shown in US dollars unless a cell says otherwise, vary by country, and can change, so confirm the vendor page before subscribing.