Does your LLM SDK tell you the truth?

Each cell grades one library version against one scenario and provider response.

Updated 2026-11-10

Libraries

Pass counts cover the cells that apply to each library. UNSUPPORTED cells, where the library's docs say it can't make the call, are counted apart.

Results

Providerany-llm 1.40.287 ✓ · 6 ~ · 6 ✗ · 1 –litellm 1.105.586 ✓ · 13 ✗ · 1 –pydantic-ai 2.61.063 ✓ · 16 ~ · 6 ✗ · 15 –tanstack-ai 1.4.050 ✓ · 1 ~ · 1 ✗ · 48 –vercel-ai 7.0.15384 ✓ · 16 –
finish
finish.stop-sequence Generation ends at a stop sequence
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
finish.unknown-value Unfamiliar provider finish value
anthropic derived✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"~ LOSSYfinish.value=null (raw: provider_details.finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini derived✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"~ LOSSYfinish.value=null (raw: provider_details.finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
ollama derived✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai derived✓ PASSEvery MUST requirement is satisfied.✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"~ LOSSYfinish.value=null (raw: provider_details.finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat derived✓ PASSEvery MUST requirement is satisfied.✗ WRONGfinish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"~ LOSSYfinish.value=null (raw: provider_details.finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value=null (raw: provider_details.finish_reason); truth.finish="other"; required finish="other"– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
request
request.max-tokens-field Output limit uses the supported Chat Completions field
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
request.max-tokens-ollama-openai Output limit reaches Ollama's OpenAI-compatible endpoint
ollama-openai ✗ WRONGrequest[0].json_body.max_tokens=absent; required=16✓ PASSEvery MUST requirement is satisfied.✗ WRONGrequest[0].json_body.max_tokens=absent; required=16– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
request.no-injected-options A minimal call leaves the context size unset
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
request.tools-with-reasoning-chat Function tools with the default reasoning effort
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
stream
stream.error-midstream Provider error after streamed text
anthropic derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✗ WRONGoutcome=returned; error.category=null; finish.value=null (absent); truth.finish="error"✓ PASSEvery MUST requirement is satisfied.
ollama derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
openai-chat derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
stream.missing-finish Stream ends without a finish signal
anthropic derived~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✗ WRONGoutcome=returned; finish.value="stop" (primary: choices[0].finish_reason); truth.finish="none"; forbidden finish="stop"✗ WRONGstream.completed=true; outcome=returned; truth.stream_completed=false✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini derived~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✗ WRONGoutcome=returned; finish.value="stop" (primary: choices[0].finish_reason); truth.finish="none"; forbidden finish="stop"✗ WRONGstream.completed=true; outcome=returned; truth.stream_completed=false~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✓ PASSEvery MUST requirement is satisfied.
ollama derived~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✗ WRONGoutcome=returned; finish.value="stop" (primary: choices[0].finish_reason); truth.finish="none"; forbidden finish="stop"– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai derived~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✗ WRONGoutcome=returned; finish.value="stop" (primary: choices[0].finish_reason); truth.finish="none"; forbidden finish="stop"✗ WRONGoutcome=returned; finish.value="stop" (primary: ModelResponse.finish_reason); truth.finish="none"; forbidden finish="stop"✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat derived~ LOSSYstream.completed=null; outcome=returned; truth.stream_completed=false✗ WRONGoutcome=returned; finish.value="stop" (primary: choices[0].finish_reason); truth.finish="none"; forbidden finish="stop"✗ WRONGoutcome=returned; finish.value="stop" (primary: ModelResponse.finish_reason); truth.finish="none"; forbidden finish="stop"✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✗ WRONGstream.completed=true; outcome=returned; truth.stream_completed=false✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
stream.missing-finish-abort Connection drops before the stream finishes
anthropic derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
stream.utf8 Streamed text with several scripts and emoji
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
stream.utf8-split Stream chunks split inside UTF-8 characters
anthropic derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses derived✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
tools
tools.parallel-calls Parallel tool calls for two cities
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✗ WRONGtool_calls[0] id="call_120847__thought__EmAKXgFpFH0TucStVhN+6mgGthmi4ZTSeXOraFjbeCmsoaavB5dIA0B5vGZy5k0Pzv8P83Tu07mHaNUyxKlTdDTX4/jln5bEXkbPUr0u5/9V2FRkpP2krwRNuCvFaArPNjk=", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_120847", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_120848", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_120848", name="get_weather", args={"city": "Zürich"}, parsed=true✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
ollama ✗ WRONGtool_calls[0] id="81193cdc-7f88-4adc-8e6c-ac29f9b61b66", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="4e915b8d-84fe-4964-a833-09d08f617d15", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
tools.parallel-calls-stream Streamed parallel tool calls for two cities
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✗ WRONGtool_calls[0] id="call_89893__thought__EmAKXgFpFH0TxZ2NGVDnAk7zdroOwEtjCQZ5QStCxNyB3D0EMEvzcedHokaxnZ1ccgbmfFfhWDeIyd0qiEnNn4U+roU6nrrrPT3fZoLqOOMqJVT9oPtsSaWF14m+Eb5xptM=", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_89893", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_89896", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_89896", name="get_weather", args={"city": "Zürich"}, parsed=true✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama ✗ WRONGtool_calls[0] id="4e8b9dda-3fa3-4763-af44-dcab08600e3f", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="babaee2f-2591-4efd-bafd-5af506b87982", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true✗ WRONGoutcome=returned; tool_calls count=1; truth count=2– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
truncation
truncation.reasoning-only Reasoning exhausts the output limit before an answer
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
truncation.structured-output Structured output cut off by the output token limit
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="parse"; structured.returned_as_success=false– UNSUPPORTEDOn these models chat({ outputSchema }) requests a streamed response, even without tools, so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#usage-examples✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="parse"; structured.returned_as_success=false– UNSUPPORTEDOn these models chat({ outputSchema }) requests a streamed response, even without tools, so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#usage-examples✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="parse"; structured.returned_as_success=false– UNSUPPORTEDOn these models chat({ outputSchema }) requests a streamed response, even without tools, so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#usage-examples✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="parse"; structured.returned_as_success=false– UNSUPPORTEDOn these models chat({ outputSchema }) requests a streamed response, even without tools, so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#usage-examples✓ PASSEvery MUST requirement is satisfied.
openai-responses ~ LOSSYfinish.value=null (absent); truth.finish="length"; error.category="validation"; structured.returned_as_success=false✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="parse"; structured.returned_as_success=false– UNSUPPORTEDOn these models chat({ outputSchema }) requests a streamed response, even without tools, so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#usage-examples✓ PASSEvery MUST requirement is satisfied.
truncation.structured-output-forced-tool Structured output through a forced tool cut off at the token limit
anthropic documented– UNSUPPORTEDany-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. https://github.com/mozilla-ai/any-llm/blob/1.40.2/src/any_llm/providers/anthropic/utils.py#L660-L686– UNSUPPORTEDLiteLLM response_format selects native JSON output on claude-haiku-4-5; ordinary forced tool calls have no documented structured-result parser. https://docs.litellm.ai/docs/providers/anthropic#structured-outputs~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false– UNSUPPORTEDBoth claude-haiku-4-5 and claude-sonnet-5-5 get native structured output (output_config.format), so no supported path sends the fixture's forced structured_output tool call. https://tanstack.com/ai/latest/docs/adapters/anthropic– UNSUPPORTEDOn claude-haiku-4-5 (and claude-sonnet-5-5) the default structured output is native output_config.format, not a forced tool. The documented jsonTool opt-in forces a tool named json, so no supported path sends the fixture's forced structured_output call. https://ai-sdk.dev/providers/ai-sdk-providers/anthropic#structured-outputs
truncation.text Text cut off by the output token limit
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
truncation.text-stream Streamed text cut off by the output token limit
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
truncation.text-stream-usage Streamed text cut off before a final usage chunk
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
truncation.tool-call-args Tool-call arguments cut off by the output token limit
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
truncation.tool-call-args-stream Streamed tool-call arguments cut off by the output token limit
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.~ LOSSYfinish.value="length" (raw: ModelResponse.finish_reason); truth.finish="length"; error.category="validation"; structured.returned_as_success=false✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
usage
usage.cached-tokens Cached input is included in total token usage
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
usage.reasoning-tokens Token usage accounts for thinking and answer tokens
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
openai-responses ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDTanStack AI's chat() always requests a streamed response (stream: false collects the stream), so it never sends the non-streamed request this fixture replays. https://tanstack.com/ai/latest/docs/api/ai#chat-options✓ PASSEvery MUST requirement is satisfied.
usage.stream-final-chunk Streamed usage in a final chunk with no choices
anthropic documented✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
gemini ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
ollama ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDpydantic-ai 2.61.0 implements OllamaModel via OpenAI-compatible Chat Completions only; there is no native /api/chat client. https://ai.pydantic.dev/models/ollama/✓ PASSEvery MUST requirement is satisfied.– UNSUPPORTEDThe SDK has no first-party native Ollama provider; community providers are separate packages outside this shim. https://ai-sdk.dev/providers/community-providers/ollama
ollama-openai ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.
openai-chat ✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.✓ PASSEvery MUST requirement is satisfied.

Grades

GradeMeaning
✓ PASSReports what happened: meets every MUST requirement of the scenario.
~ LOSSYDoesn't contradict what happened, but you only find it in raw metadata, or not at all.
✗ WRONGReports something that contradicts what happened.
! CRASHFails with an internal error, such as a KeyError in its own parser.
⧗ HANGDoesn't return after the whole response was sent.
– UNSUPPORTEDThe library doesn't support this provider or feature, according to its documentation.
? HARNESSOur problem, not the library's.