JSON Schema validation compatibility

Texaryn’s schema adapters run the official JSON Schema Test Suite, vendored at a pinned revision. The figures below are read from the same file the test suite asserts against, so a number here cannot drift from the measured behaviour without a test failing.

This page is about validation, which Texaryn delegates to a validator library. What Texaryn itself guarantees is the form projection, described in JSON Schema support. The two are deliberately separate: const, not, contains and unevaluatedProperties affect whether data is valid without affecting which control is rendered.

Results

Suite revision f6fd52a0a95472e079cbfc6ef7f089702b80e045.

@texaryn/schema-json

Dialect	Mandatory passed	Classified failures	Optional passed, default configuration
Draft 7	917 / 929	External schema resolution (12)	661 / 898
2019-09	1244 / 1261	External schema resolution (17)	563 / 1019
2020-12	1278 / 1301	External schema resolution (23)	566 / 1023

@texaryn/schema-json-hyperjump

Dialect	Mandatory passed	Classified failures	Optional passed, default configuration
Draft 7	898 / 929	External schema resolution (23), Upstream validator (8)	474 / 898
2019-09	1221 / 1261	External schema resolution (36), Upstream validator (4)	542 / 1019
2020-12	1248 / 1301	External schema resolution (49), Upstream validator (4)	544 / 1023

Every number is per adapter. There is deliberately no combined figure, because a “Texaryn compliance” percentage would describe no package anyone can install. @texaryn/schema-json is the default and the one published; @texaryn/schema-json-hyperjump is a private reference adapter kept to check that SchemaEvaluationPort is not shaped around a single validator.

How to read the columns

The suite splits its cases into a mandatory set and an optional/ directory. Those are reported separately and never added together.

• Mandatory passed counts cases every implementation is expected to pass.
• Classified failures are mandatory cases that fail for a stated, recorded reason. Nothing is skipped: the remote-reference tests run, fail, and are classified rather than being excluded from the denominator.
• Optional passed, default configuration counts the optional/ cases, which cover behaviour the specification leaves to the implementation. A lower number here is not a defect, and it is not directly comparable between adapters: the suite expects an implementation running optional/format to turn format assertion on where it supports it, and these runs use each adapter’s shipped defaults instead. The count is recorded per failing test id, so a change is identified rather than merely detected.

What the failures are

Every mandatory failure is recorded individually in tests/conformance/json-schema-suite/baseline.json with a reason drawn from a closed set, validated when the file is read rather than only in the type system, and with the observed outcome (wrong answer, or the error class of a throw). There is no wildcard: a reason names one exact upstream test, so “skip all unevaluatedProperties tests” is not expressible. Recording the outcome is what stops a deviation covering a test that later fails for a new reason.

A schema may reference its dialect’s own metaschema to assert that it is itself a valid schema. Both adapters resolve those without any configuration, so that is not among the failures below; the published metaschemas are loaded on demand and never fetched over the network.

External schema resolution accounts for every failure of both adapters except the class below. The suite expects documents to be retrieved from http://localhost:1234, and neither adapter factory exposes a resolver hook.

Both fail closed, which is the part worth knowing, and they do it at different points. @texaryn/schema-json compiles, then reports the unresolved reference as a $ref validation error, so the instance is judged invalid whatever the referenced schema would have said; constraints beside the reference are still applied. @texaryn/schema-json-hyperjump refuses to compile at all. Neither silently drops the constraint, so an unresolvable reference cannot let data through unchecked.

Upstream validator deviations are departures from the suite’s expectations observed through the reference adapter. They are recorded that way on purpose: the measurement runs through Texaryn’s own code, so it does not by itself establish the validator library as the cause. Each carries an issue link, so the claim travels with its evidence rather than resting on a sentence in a JSON file, and each is to be reproduced against the validator directly before being reported upstream. One is documented upstream policy rather than a defect, which its note says.

Format assertion differs by dialect

format is annotation-only in 2019-09 and 2020-12 by default, and assertion is permitted in Draft 7. @texaryn/schema-json asserts the schema’s own format keywords in Draft 7, including through the Draft 7 fallback a schema whose $schema is missing or unrecognized, and never in 2019-09 or 2020-12. Declaring the format-assertion vocabulary does not turn it on in the later dialects, and the adapter has no option that does.

That is why Draft 7’s optional count sits above the other two. Those cases deliberately measure an implementation with format assertion switched on, so a dialect that correctly leaves it off scores lower there. It is the one place in this report where a lower number is the specification-correct one.

One exception: a schema that uses $ref to reach a published metaschema, to check that a document is itself a valid schema, has that metaschema’s format keywords asserted in every dialect, because json-schema-library compiles those documents with its own default, which asserts. A malformed pattern regex therefore fails with keyword format under Draft 7, 2019-09 and 2020-12 alike, which in the later two is stricter than their annotation default.

Strict schema linting would be a separate capability rather than a change to validation semantics.

The mandatory suite cannot see any of this, which is worth knowing about the suite rather than about the adapter. Only 2020-12’s format.json carries the “invalid string is only an annotation by default” cases; the Draft 7 and 2019-09 files contain no string cases at all. Correcting 2019-09 moved its optional count by 219 and left every mandatory result in every dialect untouched, so the behaviour is pinned by a direct test in @texaryn/schema-json rather than by anything here.

Regenerating the baseline

The suite gates drift, not the pass rate. A mandatory test that regresses fails; one that starts failing a different way fails; a recorded deviation that starts passing fails, so the declaration has to go; an unclassified failure fails from the first run; a moved pass count fails; and so does a change to the set of failing optional tests, even one that leaves the count identical. Maintainers re-record it with:

pnpm json-schema-suite:baseline

Reviewing that diff is the point of the exercise. Regenerating a baseline to turn a suite green, rather than to record an intended change, defeats it.