How to Use AI Tools for Experiment Design: Variable Control and Data Collection
A same-input afternoon test plan for checking variables, collection protocols, analysis plans, and validation rules across three AI assistants without presenting untested vendor claims as results.
Sources checked 2 Oct 2026
Use AI tools as drafting and review assistants: ask each one to identify variables, turn a fixed brief into a collection protocol, map it to an analysis plan, and generate validation checks. The vendor documentation linked below settles file limits, prices, data settings, and accuracy cautions, but it does not establish which assistant will suit a particular study. Chat Picker has not tested ChatGPT, Claude, or Gemini for this task; run identical inputs and keep your own log.
Know these limits before you start
As read on October 1, 2026, OpenAI’s File Uploads FAQ sets a 512 MB hard limit per conversation file, a 2M-token cap for text and document files, and an approximately 50 MB CSV or spreadsheet limit that varies with row size; Free allows three uploads per day. Anthropic’s upload page lists 500 MB per chat file and 30 MB per project file, and requires code execution and file creation for XLSX. Google’s Gemini file-upload page lists up to 10 files per prompt, 100 MB per non-video file, and 2 GB per video, subject to availability.
Before using participant records, check the account and workspace. OpenAI’s pricing page lists a training opt-out for Free, Go, Plus, and Pro; Anthropic’s pricing page lists one for Free, Pro, and Max and says Team is not trained on by default. The Gemini plans page does not state a training setting. Training choices do not settle retention, connector permissions, or administrator controls.
OpenAI’s accuracy guidance says ChatGPT can be incorrect or misleading and recommends treating it as a first draft. Anthropic’s guidance says not to rely on Claude as the only source of truth and to inspect original sources. Gemini Apps source help says a Sources panel appears when sources are available.
For sensitive or elevated-risk work, OpenAI’s usage policies prohibit unauthorized aggregation or distribution of private or sensitive information. Anthropic’s Usage Policy calls for relevant human expertise in elevated-risk cases, while Google’s safety guidelines warn that outputs can reflect training-data limits and overgeneralizations.
The test plan
Run every task in a fresh chat with each assistant. Create a log with the plan, test date, prompt version, input filenames, proposed variables or rules, added assumptions, source-linked claims, validation gaps, edits needed, and final disposition.
-
Identify variables. Prepare this brief: “Randomly assign either a plain-language questionnaire instruction or the standard instruction. Outcomes are elapsed submission time and blank-item count. Device class is a blocking variable; duplicate session IDs are excluded. A bot check is needed, but its rule is undecided.” Prompt: “Create a variable table from this brief. Give each row a role, exact definition, unit, source phrase, and decision status. Do not invent the bot rule, missing-data threshold, sample size, or statistical test. Put every ambiguity under Needs decision.” Inspect invented eligibility criteria and ambiguous units. Log every variable, assumption, and unresolved decision.
-
Generate the collection protocol. Attach your actual questionnaire, variable table, and blank data dictionary. Prompt: “Map every question and system event to a field. State its source, unit, timing, allowed values, missing-value meaning, and validation check. Distinguish blank from not shown, keep assignment separate from outcomes, and flag every field the supplied files do not define.” Inspect incomplete mappings and collection-order errors. Log unsupported fields and requested revisions.
-
Draft the analysis plan. Attach the variable table and a CSV header containing
session_id,assigned_instruction,started_at,submitted_at,completion_seconds,blank_items,device_class, andprior_completion. Prompt: “Write a pre-results statistical analysis plan. Map the target contrast, outcomes, exclusions, transformations, missing-data handling, model, effect measure, interval, diagnostics, and decision rule to the brief. Mark every method choice that needs researcher approval. Do not calculate effects from a schema.” Inspect contradictions and undefined variables. Log proposed methods and unresolved choices. -
Build validation checks. Attach the schema, questionnaire structure, and protocol. Prompt: “Create implementation-neutral checks for unique session IDs, condition labels, timestamp order, time units, outcome completeness, duplicate records, and ranges justified by the questionnaire. Give the fields, failure condition, and action for each check. Do not claim a check passed unless you run it and report the command and result.” Inspect timezone assumptions, blank-to-missing conversions, and invented ranges. Log executable rules separately from narrative ones.
-
Compare and revise. Prepare one final packet from each assistant, remove product names, assign neutral labels, and give every assistant the same packet set and original brief. Prompt: “Compare all three packets with the brief. For each mismatch, quote the requirement and packet text, explain the effect on variable control or data quality, and propose an exact correction. Mark unsupported claims and missing decision owners. Do not add eligibility rules.” Inspect omissions, silent assumptions, and changed target contrasts. Log issue categories, reviewer disagreement, and human decisions.
How to read your results
Turn the log into a decision, not a general leaderboard. Before running, define must-pass conditions: no invented protocol facts, complete variable and protocol mapping, visible unresolved decisions, an aligned analysis plan, traceable validation rules, and a suitable upload plan.
A packet that fails any condition needs revision regardless of how polished it appears. If several pass, choose using documented cost, file limits, data settings, source visibility, and the amount of manual checking required, then record that basis. Your self-run log describes these assistants on these inputs; it does not establish a universal ranking.
Where the plans differ
The Claude vs ChatGPT vs Gemini matrix carries vendor sources and read dates for its figures. For this test, check its Free plan, Low-cost tier, Main paid plan, Context window in the app, How usage limits are described, Training on your chats, and Team plan rows. For shared work, check the Team plan row; Gemini business pricing is marked Not verified.
In the same October 2026 reading, OpenAI’s What is ChatGPT Plus? page lists Plus at $20 per month billed monthly. Anthropic’s pricing page lists Pro at $20 per month billed monthly, or $17 per month on annual billing with $200 up front. Google’s US AI plans page lists Google AI Pro at $19.99 per month. Confirm prices before subscribing because countries and plans differ.
For context, OpenAI’s pricing page lists 27K on Free Instant, 54K on Go and Plus Instant, 128K on Pro Instant, and 256K on Go and Plus or 400K on Pro for reasoning. Anthropic’s pricing page says context can reach 1M on every plan and varies by model. The matrix leaves Gemini’s app token figure unverified.
For usage, OpenAI and Anthropic publish no exact message counts on their pricing pages. Google says compute-based limits refresh every 5 hours up to a weekly limit; its paid tiers provide 2x, 4x, and up to 20x those limits, and AI credits can extend them.