JSON Schema validation compatibility
Texaryn’s schema adapters run the official JSON Schema Test Suite, vendored at a pinned revision. The figures below are read from the same file the test suite asserts against, so a number here cannot drift from the measured behaviour without a test failing.
This page is about validation, which Texaryn delegates to a validator
library. What Texaryn itself guarantees is the form projection, described in
JSON Schema support. The two are deliberately
separate: const, not, contains and unevaluatedProperties affect whether
data is valid without affecting which control is rendered.
Results
Section titled “Results”Suite revision f6fd52a0a95472e079cbfc6ef7f089702b80e045.
@texaryn/schema-json
| Dialect | Mandatory passed | Classified failures | Optional passed, default configuration |
|---|---|---|---|
| Draft 4 | 618 / 618 | — | 280 / 391 |
| Draft 7 | 929 / 929 | — | 660 / 898 |
| 2019-09 | 1260 / 1261 | Suite known issue (1) | 562 / 1019 |
| 2020-12 | 1300 / 1301 | Suite known issue (1) | 566 / 1023 |
@texaryn/schema-json-hyperjump
| Dialect | Mandatory passed | Classified failures | Optional passed, default configuration |
|---|---|---|---|
| Draft 4 | 618 / 618 | — | 217 / 391 |
| Draft 7 | 929 / 929 | — | 474 / 898 |
| 2019-09 | 1256 / 1261 | Custom vocabulary support (5) | 542 / 1019 |
| 2020-12 | 1296 / 1301 | Custom vocabulary support (5) | 544 / 1023 |
Every number is per adapter. There is deliberately no combined figure, because
a “Texaryn compliance” percentage would describe no package anyone can
install. @texaryn/schema-json is the default and the one published;
@texaryn/schema-json-hyperjump is a private reference adapter kept to check
that SchemaEvaluationPort is not shaped around a single validator.
How to read the columns
Section titled “How to read the columns”The suite splits its cases into a mandatory set and an optional/ directory.
Those are reported separately and never added together.
- Mandatory passed counts cases every implementation is expected to pass.
- Classified failures are mandatory cases that fail for a stated, recorded reason. Nothing is skipped. Remote-reference tests resolve from the vendored suite fixtures and run in the normal denominator.
- Optional passed, default configuration counts the
optional/cases, which cover behaviour the specification leaves to the implementation. A lower number here is not a defect, and it is not directly comparable between adapters: the suite expects an implementation runningoptional/formatto turn format assertion on where it supports it, and these runs use each adapter’s shipped defaults instead. The count is recorded per failing test id, so a change is identified rather than merely detected.
What the failures are
Section titled “What the failures are”Every mandatory failure is recorded individually in
tests/conformance/json-schema-suite/baseline.json with a reason drawn from a
closed set, validated when the file is read rather than only in the type
system, and with the observed outcome (wrong answer, or the error class of a
throw). There is no wildcard: a reason names one exact upstream test, so “skip
all unevaluatedProperties tests” is not expressible. Recording the outcome is
what stops a deviation covering a test that later fails for a new reason.
The suite runner supplies the adapters’ public resource resolver with
http://localhost:1234 fixtures loaded from disk. The run is reproducible
offline. The primary adapter also loads each supported dialect’s official
metaschema locally.
The remaining mandatory failures are all in vocabulary.json for 2019-09 and
2020-12. The exact case identifiers and outcomes are recorded in the baseline.
The primary adapter has two failures: it applies its built-in validation
keywords when a custom metaschema omits the validation vocabulary. The suite’s
expected-valid result for the invalid minimum value is disputed in upstream
issue #574:
an implementation may offer familiar validation keywords even when that
custom metaschema does not declare the validation vocabulary. These two cases
are recorded as known suite issues, not as validator defects.
The private Hyperjump adapter has ten compilation failures. It rejects the custom dialect before reaching the instance assertions, including four cases for an unknown optional vocabulary and six cases using a custom metaschema that omits the validation vocabulary. The optional vocabulary cases and the applicator and valid-number assertions expose a comparison-adapter limitation. The invalid-number cases have the same disputed expected-valid result noted in issue #574. Neither adapter claims support for arbitrary custom vocabularies. The public adapter’s built-in keyword behavior must not be read as support for user-defined vocabularies or as proof that unknown required vocabularies are rejected.
The conformance count is per adapter and per dialect. For Draft 4, both adapters pass all 618 mandatory cases at the pinned suite revision, with no case skipped. This measures validation only. It does not establish tuple-item form projection, general custom vocabulary support, or cross-dialect external references. External resources must use the same dialect as the root schema.
Upstream validator deviations are departures from the suite’s expectations observed through the reference adapter. They are recorded that way on purpose: the measurement runs through Texaryn’s own code, so it does not by itself establish the validator library as the cause. Each carries an issue link, and each should be reproduced against the validator directly before being reported upstream.
Format assertion differs by dialect
Section titled “Format assertion differs by dialect”format is annotation-only in 2019-09 and 2020-12 by default, and assertion is
permitted in Draft 4 and Draft 7. @texaryn/schema-json asserts the schema’s
own format keywords in Draft 4 and Draft 7, including through the Draft 7
fallback for a schema whose $schema is missing or unrecognized, and never in
2019-09 or 2020-12.
Declaring the format-assertion vocabulary does not turn it on in the later
dialects, and the adapter has no option that does.
The optional totals are not directly comparable across drafts because the directories contain different test sets. The optional format cases measure each adapter with its shipped defaults, so a dialect that correctly leaves format annotation-only can pass fewer of those assertions.
One exception: a schema that uses $ref to reach a published metaschema, to
check that a document is itself a valid schema, has that metaschema’s format
keywords asserted in every dialect, because json-schema-library compiles those
documents with its own default, which asserts. A malformed pattern regex
therefore fails with keyword format under Draft 4, Draft 7, 2019-09 and
2020-12 alike, which in the later two is stricter than their annotation default.
Strict schema linting would be a separate capability rather than a change to validation semantics.
The mandatory suite cannot see every default-format distinction. The
2019-09 and 2020-12 format.json files carry cases that verify annotation-only
behavior. The adapter behavior is also pinned by direct tests in
@texaryn/schema-json.
Regenerating the baseline
Section titled “Regenerating the baseline”The suite gates drift, not the pass rate. A mandatory test that regresses fails; one that starts failing a different way fails; a recorded deviation that starts passing fails, so the declaration has to go; an unclassified failure fails from the first run; a moved pass count fails; and so does a change to the set of failing optional tests, even one that leaves the count identical. Maintainers re-record it with:
pnpm json-schema-suite:baselineReviewing that diff is the point of the exercise. Regenerating a baseline to turn a suite green, rather than to record an intended change, defeats it.