2026 Open-Source AI Chat Model Comparison: Llama, Mistral, and ChatGLM
An afternoon test plan for comparing Llama, Mistral, and ChatGLM against documented context limits, access terms, and hardware guidance.
Sources checked 3 Oct 2026
For Llama, Mistral, and ChatGLM, the maintainers’ own pages document model formats, context lengths, named capabilities, access conditions, and some hardware guidance; they do not settle accuracy, speed, or suitability for your workload. Chat Picker has not tested these models, so this comparison is a hands-on afternoon test plan, not a ranking. The projects covered are those read for this page, not the whole field.
An older version of this page contained statistics and test results without sources; they have been removed.
Know these limits before you start
Start with the documented constraints rather than model labels.
As read on October 1, 2026, the Llama models README lists Scout-17B-16E with a 10M-token context and Maverick-17B-128E with 1M. Those are listed context lengths, not evidence that your runner can load them. The page requires license acceptance; after approval, a signed URL expires after 24 hours and a certain download allowance. It says full bf16 Llama 4 inference needs at least four GPUs; Llama-4-Scout-17B-16E-Instruct in FP8 needs two GPUs with 80 GB of memory, while Int4 needs one 80 GB GPU. The weights are licensed for researchers and commercial entities.
The Mistral Docs call Mistral Large 3 an open-weight multimodal model, Small 4 a combined instruct, reasoning, and coding model, and Medium 3.5 a model optimized for agentic and coding uses. These are capability descriptions, not comparative results. The Mistral pricing page, as read on October 1, 2026, lists Free with limited messages and web searches, Pro at $14.99 per month excluding taxes, and Team at $24.99 per user per month excluding taxes. Do not treat the Pro or Team figures as self-hosting quotes; the material read gives no hardware floor.
The ChatGLM3 README, as read on October 1, 2026, calls ChatGLM3-6B open-source, lists an 8k sequence length, and describes multi-turn dialogue, function calling, code interpretation, and agent tasks. Separate variants list 32k and 128k sequence lengths. It says the weights are open for academic research and free commercial use is allowed after registration through a questionnaire. Its sample default FP16 load needs about 13 GB of VRAM. CPU inference is supported without a GPU, requires about 32 GB of memory, and is slower. The page says only local model loading is supported on macOS.
For serving, the Ollama FAQ, as read on October 1, 2026, says local operation does not see prompts or data, while cloud processing does not store, log, or train on their content. Basic account and limited usage metadata excludes prompt and response content. Ollama’s default context is 4,096 tokens and can be overridden with OLLAMA_CONTEXT_LENGTH. The llama.cpp README documents binary, Docker, and source installation plus several quantized formats. Neither page supplies a cross-project quality comparison.
The test plan
Before running, write down your must-have languages, permitted use, and resource ceiling. Use a new session for each task and record the model identifier, quantization, runner version, prompt, and date.
-
Reasoning accuracy. Prepare four questions with one source packet: an arithmetic item, an ambiguous question, a time-sensitive question, and a rule-based question. Instruction: “Answer each question, show the calculation, mark missing information, and cite the exact supporting passage. If the packet does not settle it, say so.” Record agreement with the answer key, unsupported claims, stated uncertainty, and elapsed time.
-
Coding. Create a small repository with one seeded bug and unit tests. Instruction: “Inspect the attached repository, reproduce the failing test, make the smallest fix, run the full suite, and summarize changed files without changing the tests.” Record the test outcome, changed files, explanation errors, time, and peak memory. If you run HumanEval-X or SWE-Bench, keep those results separate and record the benchmark version, selected tasks, and scoring script.
-
Multilingual support. Prepare release notes containing version strings, dates, and code snippets. Instruction: “Translate the attached release note into English, Simplified Chinese, Spanish, and Japanese; preserve every version string, number, and code snippet.” Record omissions, altered technical terms, broken formatting, and consistency across languages.
-
Context and memory. Make a long synthetic report containing a change, its cause in another section, and a later status update. Instruction: “Read the attached report, explain what changed and why, quote the relevant headings, and flag missing details.” Record the configured context, accepted input length, truncation, correct retrieval, and consistency after a follow-up question. Increase length only within your measured resource ceiling.
-
Speed and deployment cost. Use one fixed report-summarization and arithmetic task. Run it three times after a cold start and three times while warm. Record first-response and total time, peak RAM or VRAM, errors, and tokens per second only if the runner exposes it. Calculate an energy or hosting estimate only from a documented rate, and label it as an estimate.
-
License and commercial use. Write a one-paragraph deployment description covering users, data, research or commercial use, and distribution. Instruction: “Read the attached license, list only permissions and restrictions it states, and mark unanswered points for human review.” Record registration duties, commercial conditions, redistribution and attribution terms, model version, and file hash. This is a document check, not legal advice.
How to read your results
Start with gates, not averages. If the documented terms do not clearly permit your intended use, send that question to qualified review. If measured memory exceeds your ceiling or a must-have test fails, rule out that option for this workload.
Compare the remaining options using critical errors, coding outcomes, language failures, truncation, median and slowest run time, memory use, and setup effort. Set weights before testing if you need a personal rubric. A same-afternoon log can guide your own workload; it cannot support a general ranking.
Where the plans differ
The ChatGPT vs Claude vs Gemini matrix concerns hosted assistants rather than these model families. Its relevant rows are Free plan, low-cost tier, main paid plan, heavy-use plans, team plan, context window, usage limits, training on your chats, and ads. These rows separate subscription billing, usage caps, app context, chat-data settings, and advertising statements. Treat “Not verified” as unknown, not zero.
For a self-hosted choice, replace subscription comparisons with documented hardware requirements, measured memory use, storage, infrastructure, power, and maintenance. Keep hosted-plan prices separate from the resources needed to deploy and operate an open-weight model.