Method
Recorded provider responses are replayed through each library. Requirements compare the library's observation with the fixture's truth.
Grades
| Grade | Meaning |
|---|---|
| ✓ PASS | Reports what happened: meets every MUST requirement of the scenario. |
| ~ LOSSY | Doesn't contradict what happened, but you only find it in raw metadata, or not at all. |
| ✗ WRONG | Reports something that contradicts what happened. |
| ! CRASH | Fails with an internal error, such as a KeyError in its own parser. |
| ⧗ HANG | Doesn't return after the whole response was sent. |
| – UNSUPPORTED | The library doesn't support this provider or feature, according to its documentation. |
| ? HARNESS | Our problem, not the library's. |
Ordered rules
- The shim's time limit ran out: HANG if the response had been served, otherwise HARNESS.
- The shim's process ended without an observation: CRASH if a signal killed it (a segfault, for example) after the library's request reached the replay server, otherwise HARNESS.
- The observation is invalid, the library called a different endpoint or stream mode than the fixture serves, or it returned or raised without a request reaching the replay server: HARNESS.
- The shim reports, with a documentation link, that the library can't express the call: UNSUPPORTED.
- The library raised an internal error: CRASH.
- A MUST requirement is CONTRADICTED: WRONG.
- A MUST requirement is NOT_OBSERVABLE: LOSSY.
- Otherwise: PASS.
Surfaces
A primary surface is the library's normal result; a raw surface is provider metadata available outside that result. An absent surface means the information is unavailable; requirements decide whether a raw or absent value changes the grade.
Fixture tiers
- recorded
- Provider response bytes captured from a real request, with recording provenance.
- documented
- A response transcribed from provider documentation, with a source link.
- derived
- A named transform of a source fixture, with a rationale and evidence.
Limitations
Recorded responses replay one moment of a provider. Documented fixtures are transcribed from documentation. A grade covers one library version and the call the shim makes.
Disclosure
Findings and upstream status are tracked in upstream.json, with issue and PR links when present. Some entries cite documented behavior rather than an upstream issue.