Download the generated PDF manuscript

NUIF authored-interface interchange research manuscript

Architecture, executable profiles, resource packaging, reconstruction research and open risks

Refpath contributors
Version working-2026-08-31 · 2026-08-31

Manuscript status: working technical manuscript; not peer reviewed. This generated document does not change the status of any included specification module.

Abstract

This manuscript evaluates a layered authored-interface interchange model through prior-art synthesis, bounded adapter profiles and executable conformance experiments. It also separates proposed resource packaging, source-backed capture and screenshot reconstruction from the narrower implemented alpha evidence. It records the architectural hypothesis, implemented evidence, unresolved risks and governance conditions without claiming specification stability, reconstruction accuracy or universal format coverage.

Reproducibility

The body is compiled from 13 canonical whitepaper modules at source digest sha256:d554926f107ca96300cd2ac76757a15e9c9cbdc7ced95047e6d1f380ebc3712c. Editorial changes belong in those source modules.

NUIF foundation

NUIF investigates a portable, vendor-neutral draft specification for authored user-interface documents. The candidate model is intended to preserve meaning across editors and implementation targets rather than treating a rendered bitmap, a vendor scene graph, or source-language AST as the universal truth.

Thesis

A useful portable interface specification must coordinate several representations instead of collapsing them into one:

1. semantic/document containment;
2. component and instance identity;
3. authored layout and responsive constraints;
4. resolved geometry at explicit evaluation contexts;
5. geometry, paint, typography, and assets;
6. design-token references and themes;
7. interaction/state and data-binding graphs;
8. source/tool provenance and correspondence;
9. extension payloads that can survive unknown intermediaries;
10. deterministic operations, diff, patch, and reconciliation.

Portable resources add a second identity boundary: editable semantic assets retain stable IDs, while exact image/font bytes use immutable content digests. Package paths and source URLs are locators/provenance, not identity.

NUIF therefore treats portability as a synchronization problem as much as a serialization problem.

Architectural hypothesis

The working model is a small canonical core plus coordinated graphs and extension dialects. The containment tree answers ownership and order. Typed relationship graphs express constraints, components, tokens, interactions, provenance, dependencies, and other relationships that do not belong in a tree.

The reference implementation will preserve both authored and resolved state. Resolved state is always scoped to an evaluation context and is never allowed to silently replace authored intent.

Fidelity model

Every adapter and transformation must classify material mappings:

• lossless — semantics are preserved exactly;
• representable — equivalent target semantics exist, even if encoded differently;
• approximated — a declared approximation is produced;
• preserved_unrenderable — data survives as an extension but the target cannot render/edit it;
• unsupported — data cannot currently be represented or preserved safely.

Silent loss is a conformance failure.

Explicit non-goals

NUIF does not promise to infer the unique original source program from pixels, reproduce arbitrary JavaScript execution, make every platform text renderer bit-identical, or force every target to support every capability. The draft specification should make such boundaries inspectable and machine-readable.

Screenshot reconstruction is therefore an optional inference client, not a new canonical truth. It may propose a validated editable hypothesis and calibrated alternatives, but screenshot-only evidence cannot be classified as lossless authored source.

Reference implementation role

The Rust implementation and editor are executable research instruments and conformance references. They do not define semantics by accident; normative behavior belongs in spec/ and must be testable independently.

Canonical module.

NUIF architecture thesis

NUIF is a specification-first authored-interface model. Its center is neither a vendor editor nor a source framework.

Core thesis

A portable interface document must retain intent, structure, relationships and evaluated results simultaneously. A single flattened scene tree cannot preserve enough information for loss-minimizing round trips across editors and runtime frameworks.

The recommended architecture is a layered hybrid:

Document containment tree
        │ stable IDs
        ├── component / instance graph
        ├── token / theme graph
        ├── layout constraint graph
        ├── interaction / state graph
        ├── provenance / correspondence graph
        └── asset dependency graph

Authored model ──evaluate/lower──► resolved model ──► render scene
      ▲                               │
      └──────── reconcile / lift ◄────┘

Borrowed foundations

• MLIR: dialects, explicit lowering, partial legality and multiple abstraction levels.
• OpenUSD: non-destructive composition, references, layers and variants.
• glTF: small core, extension registry, used/required capabilities.
• DTCG: token interchange.
• SVG/Unicode/OpenType: geometry and text foundations.
• Retentive/symmetric lenses: synchronization with preserved source regions.

New work required

NUIF must define the missing combination: authored UI semantics + resolved state + cross-tool provenance + structural loss accounting + source patch synchronization.

Canonical layers

1. Document layer — identity, containment, semantics, accessibility.
2. Component layer — definitions, instances, slots, parameters, variants and overrides.
3. Layout layer — authored sizing/layout intent independent of resolved geometry.
4. Visual layer — geometry, paint, text and effects.
5. Behavior layer — interactions, states, animation and data bindings.
6. Resolved layer — computed layout, shaped text, flattened paint/effect plans for a declared evaluation context.
7. Provenance layer — source/destination correspondence and fidelity diagnostics.
8. Resource layer — stable semantic assets bound to content-addressed bytes, package/resolver locators and derivation records.

No lower layer is permitted to silently erase a higher-level authored construct. Lowerings that cannot represent a construct must emit fidelity records.

Stable identity

Identity is semantic and independent of path, order and display name. Moving an entity does not change its ID. Content hashes identify immutable resources and canonical snapshots, not editable semantic entities.

Compiler and reconstruction ports

Deterministic source adapters and probabilistic screenshot reconstruction meet at the operation boundary:

retained source + resolved host observations ─┐
                                              ├─> typed operations -> core
pixels + OCR/CV/model hypotheses ─────────────┘                    -> render/evaluate

Source-backed and screenshot-only inputs retain distinct evidence classes. A model/provider is replaceable and cannot redefine the operation grammar, validator, layout semantics, resource identity or fidelity ceilings.

Falsifiability

The architecture fails if the v0 experiment cannot preserve a non-trivial responsive component through editor→HTML→NUIF→editor while retaining component identity, token bindings, layout intent, an opaque foreign extension and a minimal source patch after an edit.

The resource/reconstruction extension fails if independent package writers cannot reproduce the proposed bytes, if browser capture cannot be pinned without secret leakage, if visual objectives reward flat screenshot copies, or if adaptation fails to beat the untuned tool-assisted baseline on a frozen holdout.

Canonical module.

Layout and rendering research synthesis

Layout is not geometry

NUIF separates authored constraints from resolved boxes. Fixed x/y/width/height are valid authored values for freeform content, but they are not the universal layout representation.

The initial layout vocabulary contains families rather than one universal algorithm:

• freeform — transforms/anchors and explicit geometry.
• stack — one-dimensional flow with intrinsic sizing, distribution, alignment and gaps.
• flex — web-compatible flexible layout semantics.
• grid — bounded explicit fixed/fr tracks, spans and deterministic no-implicit-track placement in profile 0; broader CSS Grid features remain capability-reported adapter input.
• constraint — relational linear constraints for editor/native-layout cases.
• custom — extension/dialect-defined evaluator with declared fallback/resolved geometry.

Common sizing primitives are normalized across families: fixed, intrinsic-min, intrinsic-max, fit-content, fill/available, percentage, min/max clamps, aspect ratio and content measurement.

Taffy is the recommended first evaluator for CSS-compatible block/flex/grid behavior because it implements web algorithms in Rust. SwiftUI’s proposal-response model and Cassowary-style constraints demonstrate why the canonical schema must remain a superset rather than serializing Taffy’s Style directly.

Evaluation context

Resolved layout is keyed by an explicit context including viewport/container size, pixel ratio, locale, writing direction, font set, token/theme selection and feature/dialect capabilities. Multiple resolved snapshots may coexist as caches or conformance fixtures.

Rendering semantics

The draft specification defines the visual meaning of paths, fills, strokes, transforms, clipping, masks, gradients, compositing, images, text and supported effects. It does not specify GPU command buffers or a renderer implementation.

The reference renderer uses a backend trait. Vello/wgpu is the leading interactive experiment, but conformance requires deterministic raster comparisons and must permit CPU reference rendering where GPU differences would make tests unstable.

Text

Canonical text remains Unicode text + style runs + semantic annotations + font references. Shaping produces resolved glyph IDs, clusters, advances and offsets using pinned font data and a declared Unicode/shaping version. A glyph cache never replaces semantic text.

Portability reports must distinguish font substitution, missing glyphs, line-break differences and rasterization differences from document-model loss.

Canonical module.

Protocol, portability and synchronization

NUIF treats portability as an ongoing synchronization problem.

Operations

The protocol operates on stable entities and semantic properties. Operations include create/delete/move, set/unset property, list/set relation edits, component/instance overrides, token bindings, extension edits and transactions. Editor gestures lower to these operations.

A drag inside a stack should usually become a reorder or layout-property edit; a drag in freeform space may become a transform edit. GUI coordinates are input data, not the protocol abstraction.

Patch model

A patch is a deterministic ordered set of operations with base snapshot identity, optional preconditions, transaction metadata and provenance. Patches can be replayed headlessly.

Three-way merge uses stable identity first and structural matching only when identity is absent. Conflicts are typed: property, delete/edit, ordering, relationship, extension and semantic-lowering conflicts.

Correspondence

Adapters maintain correspondence records between NUIF entities/properties and foreign constructs such as DOM nodes, CSS declarations, Svelte component props or design-tool node IDs. Correspondence is separable from the canonical design so source-specific metadata can be detached when unnecessary.

Fidelity classes

Every adapter/evaluator may report:

• lossless — semantics preserved and reconstructable.
• representable — equivalent semantics represented through different constructs.
• approximated — visible/behavioral approximation with known semantic loss.
• preserved_unrenderable — data retained opaquely but not understood/rendered.
• unsupported — data could not be safely preserved.

Silent degradation is a conformance failure.

Canonical module.

Serialization, collaboration and governance

Logical model before encoding

NUIF defines one logical model with multiple conforming encodings.

Text form

A canonical, reviewable representation is implemented for examples, fixtures, diffs and Git workflows. nuif-text-0 fixes number formatting, UTF-8 key order, layout and strict decode/canonicalize behavior; later text profiles may evolve only through explicit versioning.

Binary form

Deterministic CBOR is the profile-0 binary form because the NUIF profile closes the choices left by RFC 8949 without coupling the logical model to generated code. The executable codec gate finds it near 41% of canonical-text size at 4,096 entities on an Apple M5 Pro run, while its typed decode path is slower than text. That result supports CBOR as a compact canonical form, not as a universal latency winner.

A candidate is timed only after complete-model round trip, canonical fixpoint and unknown-data preservation through a neighboring edit. Protobuf does not specify canonical binary output. FlatBuffers deliberately permits different byte layouts and old readers ignore new fields, so a rebuilding editor needs a separate retention strategy. Cap’n Proto specifies a schema-agnostic canonical form and is the preferred next experiment, but it still needs a complete NUIF mapping, bounded old-reader edit trial and two agreeing canonical writers. Compiled zero-copy runtime caches remain separate, explicitly noncanonical profiles rather than replacements for authoring interchange.

The experimental package form separates manifest/document records from content-addressed resources. RFC 0010 selects a candidate deterministic ZIP profile with fixed mimetype, canonical manifest/document records and SHA-256-addressed blobs. Bare encodings use explicit .nuif.json and .nuif.cbor names. Exact ZIP header fixtures, two independent local writers and bounded image/font segments now exist. Cross-platform and externally authored writer evidence remains required before package-profile acceptance.

Semantic document, resource and package hashes have different scopes. Stable asset identity is not content addressing. Unknown extension payloads remain explicit typed bytes/values and must not depend on accidental codec unknown- field behavior.

Collaboration

Canonical documents do not require CRDT tombstones, clocks or replica metadata. A collaboration profile maps NUIF operations to an append-only/change structure and can use Automerge, Yjs or another convergent transport. Checkpoints serialize back to canonical NUIF.

This keeps offline files simple and permits multiple collaboration engines.

The executable register profile uses causal multi-value registers. The separate existing-tree profile replays uniquely ordered moves, rejects cycles, models deletion as profile trash and orders siblings through stable RGA-style origins. Semantic move and deletion conflicts remain visible even when the profile can choose a deterministic checkpoint. Automerge is presently tested as an operation-set transport, not claimed as an implementation of the tree algorithm.

Governance

Early development occurs in refpath/nuif, but the architecture assumes eventual neutral stewardship. A plausible progression is:

1. OSS research/reference implementation under Refpath.
2. public RFC process + implementer registry.
3. independent community/working group once two independent implementations exist.
4. investigate W3C Community Group for UI/document semantics and/or Khronos-style governance if renderer/asset vendors become primary stakeholders.

Specification text, schemas and conformance tests need clear royalty-free contribution/IP terms before claiming standards-track stability.

Canonical module.

Prior art and competitive map

No surveyed system currently combines the whole NUIF thesis. Several solve important subsets.

Directly borrow

Stable standards concepts: SVG geometry, DTCG token values, Unicode/OpenType text foundations, CSS-compatible algorithms for matching profiles, glTF-style capability declarations, OpenUSD-style composition principles, MLIR-style dialect/lowering discipline.

Adapt

Retentive lenses → property/source correspondence; CRDTs → collaboration profile; WebRender/Vello/Skia → renderer boundary and conformance strategy; Tree-sitter → source-preserving adapter infrastructure.

Invent/prove

The proposed integration combines authored and resolved UI state, stable cross-tool semantic identity, opaque extension retention, fidelity accounting, bidirectional semantic patches and a native open editor whose internal state is the draft model itself.

Canonical module.

Implementation language and runtime choice

Decision: Rust reference core

Rust is the strongest default for the reference implementation because the project simultaneously requires untrusted binary parsing, graph/document transforms, geometry, text shaping, native/WASM embedding, GPU access, fuzzing and stable C-compatible boundaries.

Alternatives

• C++ has the deepest graphics ecosystem and mature Skia/Yoga integration, but expands memory-safety risk in parsers/plugins and makes a browser/WASM-safe reference core less attractive.
• Zig offers excellent systems control and C interoperability but has a smaller mature graphics/text/schema ecosystem and less API stability for a standards reference implementation.
• Go is strong for services/tooling but weaker for low-level rendering/WASM/native GUI integration and deterministic allocation-sensitive engines.
• TypeScript is ideal at web/editor adapter boundaries but unsuitable as the only renderer/codec/reference-core implementation.

Stack

• Rust: document model, operations, layout abstraction, codec, renderer scene, conformance, WASM bindings.
• Taffy: initial CSS-compatible evaluator behind NUIF types.
• Vello/wgpu: interactive renderer experiment behind a NUIF renderer trait.
• HarfBuzz-compatible shaping: text experiment with pinned font inputs.
• Masonry + AccessKit: reference editor shell (ADR 0006, accepted; toolchain 1.98.0, MSRV 1.96); Svelte 5 + TypeScript for the later browser demonstration over the WASM bindings.
• Tree-sitter/language-native parsers: source adapters where concrete syntax retention is required.

Adapters MAY be written in the ecosystem-native language; conformance is against behavior/protocol, not implementation language.

Canonical module.

Risk register and impossibility boundaries

Fundamental boundaries

1. Rendered output is underdetermined. Pixels/boxes cannot uniquely reveal whether layout came from flex, grid, constraints, absolute positioning or runtime code. Imported foreign content must mark inferred intent.
2. Arbitrary program behavior is not serializable as UI structure. NUIF does not promise to recover arbitrary JavaScript/Swift/Dart application logic.
3. Text is environment-sensitive. Font files, shaping versions, fallback and rasterization can differ. Exact conformance requires pinned inputs; portability must classify substitution separately.
4. Platform-native controls differ. Semantic equivalence may be possible while exact visuals/behavior are platform-specific.
5. Effects/shaders can exceed a portable core. Extensions may be preserved without being renderable.
6. Standard complexity can kill adoption. IFC/STEP demonstrate the cost of excessive semantic scope.
7. A single reference implementation can accidentally become the spec. Independent implementation is a standards gate.
8. Resource bytes carry legal and security constraints. Exact font/image preservation does not imply permission to redistribute or safe decoding.
9. Visual metrics are gameable. A flat screenshot can look exact while discarding editability, semantics, accessibility and responsive behavior.
10. Model confidence can be misleading. Raw probability is not calibrated correctness and cannot upgrade inferred evidence into source truth.
11. Capture can leak secrets. Browser/network/accessibility observations may expose credentials, personal data or proprietary content unless collection, export, retention and training are separately bounded.

Containment strategies

• explicit fidelity reports and inference confidence;
• authored + resolved snapshots instead of pretending either alone is canonical truth;
• opaque extension preservation;
• deterministic capability/evaluation contexts;
• small core + profiles;
• reference implementation backed by normative conformance fixtures;
• fuzzing/resource budgets for all untrusted inputs;
• early independent implementation and adapter experiments.
• stable asset IDs separated from resource digests, locators and provenance;
• verify resource size/digest before decoding and never fetch implicitly;
• typed operation output for models, atomic validation and finite correction loops;
• structural/text/resource/edit-task metrics alongside visual diagnostics;
• calibrated decision-level confidence, alternatives and abstention;
• private captures default to local processing, no retention and no training.

Current implementation risks

• The macOS graphics fork is a maintenance boundary. The editor pins refpath/xilem commit 1b96eb8; its parent eabfe0a moves the active renderer from wgpu 28 and metal-rs to wgpu 29 and the objc2 Metal bindings. The fork changes public API call sites and does not patch the Objective-C blocks ABI. Each fork update requires the editor tests, reverse dependency trace, and macOS Metal window smoke test recorded in nuif:research:macos-metal-block-future-incompatibility. A separate refpath/metal-rs move-to-block2 branch exists only for review; NUIF does not depend on the deprecated binding or that experimental patch.
• Release signing is credential-bound. The editor packaging gate builds, archives and smoke-tests an unsigned host package. Platform signing and notarisation require release credentials and remain separate from source conformance.
• Portable resources are only narrowly implemented. The deterministic package, RGBA8 PNG and static single-face TrueType subsets have executable gates, while CPU profile 0 remains unchanged. RFC 0010 cannot be accepted until broad media/font matrices, the configured Linux/Windows/macOS jobs produce passing hosted evidence, external reproduction, calibrated aggregate budgets and interoperability review pass.
• Capture and reconstruction accuracy remains unestablished. RFC 0011 and specification 14 now have bounded fixed-input contracts, a local pinned browser fixture and a typed synthetic evaluation report. They establish interfaces, refusal rules and metric consistency, not portable browser-capture or screenshot-reconstruction accuracy. No current release makes those accuracy claims.

Thesis falsifiers

The project should rethink its architecture if ordinary source round trips require broad regeneration, if unknown extension preservation cannot survive routine edits, if the layout vocabulary becomes a vendor-property dump, or if a second implementation cannot reproduce v0 behavior from specification + fixtures alone. The resource/reconstruction path should additionally narrow if package bytes cannot reproduce across writers, correction loops improve pixels by deleting semantics, confidence cannot support useful risk/coverage, or tuned models do not beat the untuned tool-assisted baseline.

Canonical module.

Governance and standardization strategy

NUIF starts under Refpath because research and implementation need a concrete home, but the target is neutral stewardship.

Repository governance now

• Public RFCs for semantic changes.
• ADRs for reference-implementation choices.
• Research evidence is distinct from normative requirements.
• Extension registry changes require examples and conformance fixtures.
• No vendor adapter can redefine core semantics.

Standardization path

A W3C Community Group is a plausible early venue for document/component semantics and coordination with Open UI/DTCG, but NUIF should not enter formal standards work before an executable v0 and at least one external implementer exist. Khronos/ASWF-style governance offers useful precedent for graphics/rendering and extension registries. A neutral foundation can become appropriate once multiple vendors/projects depend on the format.

IP/licensing goals

• reference code: permissive dual MIT/Apache-2.0;
• specification/schema/conformance text: permissive terms compatible with standards adoption;
• contributions: explicit patent/IP policy before standards-track claims;
• trademarks/conformance branding: separate from implementation copyright.

The project must not call itself an industry standard merely because the repository is public. Conformance, multiple implementations and neutral governance are prerequisites.

Canonical module.

Naming and project identity

NUIF is the current working name and repository slug, not yet a cleared standards trademark.

A public reconnaissance found prior acronym use in an old Nexus User Input Framework and in a 2024 computer-vision paper, plus unrelated uses. None currently appears to occupy the same open UI-authoring interchange category, but collision risk is non-zero.

The architecture therefore separates human branding from protocol identity. Stable schema namespaces, extension IDs and version identifiers must not depend on a product trademark remaining unchanged.

Before v0.1 branding is promoted outside the research project:

1. perform repository/package/domain/trademark clearance in key jurisdictions;
2. decide whether NUIF is an acronym or simply a proper project name;
3. reserve crate/package/extension namespaces;
4. define conformance-mark governance separately from the open specification;
5. make any rename before third-party persisted documents become common.

Canonical module.

Research coverage and continuous completeness

A research repository cannot truthfully claim to contain every paper that will ever be relevant. NUIF instead defines operational completeness: every planned architectural front must have an explicit status, evidence links, unresolved questions and an experiment/decision path.

research/coverage.yaml is the machine-readable coverage contract. It maps the founding research plan to research IDs, specification modules, RFCs/ADRs, code seams and experiments. The project can therefore identify gaps structurally rather than relying on prose search to infer that a topic was forgotten.

Current state

All founding fronts are represented. The resource, browser-capture, reconstruction-evaluation, adaptation/distillation and AI artifact-governance fronts are now explicit rather than hidden inside “serialization” or “inference.” Decisions that can safely be made from mature prior art are marked covered. Questions whose answer would be premature without an implementation are marked experiment-required. Areas whose evidence base will continuously evolve—prior art, adapters, reconstruction and data/model governance—remain ongoing by design.

This distinction is important: marking an open research problem as finished would be less rigorous than preserving it as a first-class graph node.

Additional boundaries from the final sweep

• WAI-ARIA and accessibility API mappings support a semantic-role/state layer distinct from platform-specific accessibility trees. The bounded nuif-web-accessibility-0 lowering now proves computed role/name/state agreement for one eleven-node fixture across pinned Chromium, Firefox and WebKit while retaining native-platform and behavior non-claims.
• KHR_interactivity provides contemporary precedent for portable, capability-aware behavior graphs rather than arbitrary scripts embedded in visual nodes. The smaller nuif-behavior-state-machine-0 sidecar now has exact Rust/Node traces for ordered guards, state, effects and explicit required/optional capability handling without claiming a final semantic schema. Its first wire experiment is one inert canonical-CBOR, content-addressed package resource: a Rust gate validates document binding and hostile cases while an independent Python ZIP reader checks exact container bytes. The attachment remains outside the canonical Document and never grants execution authority. A separate one-way web lowering maps the bounded effects through native activation, hidden and an ARIA status region; five events agree across pinned Chromium, Firefox and WebKit under one exact CSP-hash-authorized runtime without extending that evidence to native UI or screen-reader speech.
• ReverseORC and related layout-inference work show that multiple viewport observations materially improve recovery of responsive intent.
• Screenshot-to-code research continues to show that visual reconstruction is not equivalent to recovering authored layout or behavior.
• Merkle/content addressing is appropriate for immutable assets and snapshots but not for editable semantic identity.
• EPUB OCF and OCI descriptors support a narrow manifest-driven package with size/digest verification; NUIF now has an in-repository independent-writer fixture. A three-OS CI matrix now exercises the package, image and font gates, while successful hosted evidence and external reproduction remain open.
• OpenType and Fontations evidence now supports one executable static TrueType package baseline with a pinned HarfBuzz metadata oracle. The retired ttf-parser decision remains documented; broad font formats, portability outcomes and shaping/raster integration remain experiment-required. Warmed parser and packaged-validation allocation ceilings cover every accepted fixture.
• Browser source capture and screenshot reconstruction are different evidence lanes and cannot share a blanket lossless claim.
• Current screenshot-to-code work supports OCR/region/hierarchical and render- correction experiments, not a claim that authored UI recovery is solved.
• LoRA, quantized adaptation and distillation are conditional experiment techniques; evaluation, rights-cleared traces and artifact governance precede training.

The continuous-research process should periodically re-run topic searches, append or supersede research records, and update research/coverage.yaml only when new evidence or experiments change the status of a front.

Canonical module.

Cross-industry patterns: evidence, adoption and rejection

This document synthesizes 52 research records added on 2026-08-29 from visual-effects interchange, game-engine asset systems, programming-language research on bidirectional transformation and layout verification, distributed-systems testing, and 2D-rendering conformance practice. Each pattern below is classified as borrowed (adopted as is), adapted (adopted with a stated change) or rejected (ruled out with the reason). Record identifiers (nuif:research:*) carry the locators; this document does not repeat them.

Method

The 52 records synthesized here were reviewed from primary sources (specifications, source code at a named commit and papers with DOI). reviewed does not mean every material claim has completed locator-level verification; research/AUDIT.md defines the stricter verified state. Claims resting on one record inherit its evidence status as well as its confidence. Source conflicts remain explicit until an RFC or experiment resolves them.

Patterns

Document model and composition

Identity and ordering

Unknown data preservation and schema evolution

Operations, undo and merge

Canonical encoding

Layout

Rendering determinism

Text

Testing methodology

Editor automation

Ruled out

The following were examined and excluded from the architecture; the reason is recorded so the question is not reopened without new evidence.

• A self-describing binary struct layout (Blender DNA): solves version drift for one implementation but does not preserve data it cannot re-save and does not compose with a schema-versioned interchange model.
• Memento undo as the canonical history: does not commute, cannot be merged, and bloats logs; inverse operations are required by spec/06.
• Integer child indices in Move and Insert as the only order representation: non-commutative under concurrency; a list identifier is required for the collaboration profile and harmless for the canonical form.
• GPU rendering as a normative oracle: implementation-defined by the WebGPU and WGSL specifications.
• Perceptual UI screenshot tests as the primary editor test: platform-dependent text rendering makes them a shell-wiring check only.
• Heuristic tree matching for NUIF-native merges: unnecessary with stable identity and a source of spurious moves.
• Whole-project regeneration as the synchronization model: contradicted by the lens and delta-lens laws that NUIF’s patch model must satisfy (lenses-foster-boomerang, bidirectional-evaluation-direct-manipulation).

Consequences for the specification

The records imply the following changes. Items 1–3 were decided by follow-up research on 2026-08-29 and are recorded as accepted RFCs; items 4–6 remain proposals.

1. Sibling order is a canonical array without keys; Insert and Move use anchors (Start, After(id)); the collaboration profile maps anchors onto a Fugue-family list CRDT (RFC 0006).
2. nuif-cbor-0 follows deterministic preferred serialization while preserving integer/real data-model identity; real zero is positive floating zero, text and CBOR key orders are distinct, strict decoders reject non-canonical input, text hashes through CBOR and strings remain verbatim (RFCs 0005 and 0008).
3. Entities of unknown kind load as Unknown with typed core fields and an opaque payload; ignorant implementations preserve bytes, knowing ones may re-encode; validation severities follow the glTF pattern (RFC 0007).
4. Every serialized record kind carries a schema version; migrations are registered pure functions; newer-than-known versions load as Unknown for entities (RFC 0007) and are diagnosed for other records.
5. Validation, import and export reports follow one schema with stable codes, severities and pointers.
6. Layout conformance declares tolerance tiers per case and classifies every divergence from a browser reference as schema loss, evaluator defect or implementation-defined behavior.

Open questions raised by this synthesis

Recorded in research/questions.yaml: cbor-float-zero (decided, RFC 0005), sibling-order-identifier (decided, RFC 0006), unknown-kind-preservation (decided, RFC 0007), editor-toolchain-msrv (ADR 0006), layout-assertion-vocabulary, render-tolerance-tiers.

Canonical module.

Resources, capture and model-neutral reconstruction

NUIF’s next research front is not “add an AI converter.” It is a coordinated resource, capture and reconstruction architecture with explicit truth boundaries. Images and fonts must survive as verified resources. Source-backed browser imports must retain authored/resolved evidence. Screenshot-only imports must remain honest probabilistic hypotheses. Every path converges through one core operation, validation, rendering and fidelity contract.

Decision summary

The recommended direction is:

1. define stable assets separately from immutable byte resources;
2. make .nuif a deterministic portable package after cross-writer proof;
3. grow the executable narrow PNG and static TrueType resource baselines only through named profiles and measured hostile-input budgets;
4. add a pinned browser-capture adapter separate from static source sync;
5. build screenshot reconstruction as a replaceable observation/proposal loop;
6. freeze a structural and visual evaluation suite before training;
7. consider adaptation or distillation only after the untuned loop exposes a repeatable learnable error distribution.

RFC 0010 and RFC 0011 remain proposed contracts. Their bounded package, narrow PNG/static-font and capture/reconstruction experiments are implementation evidence only for the named subsets; they are not published conformance or standards claims.

One core, two import lanes

Source-backed lane
HTML/CSS + browser execution + resource responses
      -> retained source + resolved observations
      -> deterministic adapter/lowering + explicit inference where needed
                              |
                              v
                    typed NUIF operations
                              |
                              v
                     core validation/apply
                              |
                              v
                      resource-aware NUIF
                              ^
                              |
Screenshot-only lane          |
pixels + context              |
      -> OCR/CV/grounding observations
      -> hierarchy/layout/resource hypotheses
      -> typed operations -> render/diff/correct

The lanes differ in evidence, not in their mutation authority. Both use the same typed operations. Neither provider can write core structs directly. A browser observation may support an exact resolved value under one pinned context, but it does not automatically reveal authored intent. A screenshot cannot establish source equivalence regardless of visual score.

Portable resource model

Four identities must remain distinct:

An asset points to a content descriptor containing media type, SHA-256 and byte length. A package path or external URL only locates candidate bytes; size and digest are checked before decoding. External resolution is opt-in. Opening a document never triggers network access.

Resource roles clarify hash and retention behavior:

• source: original encoded bytes;
• authoring: exact bytes needed to evaluate/edit semantics;
• derived: crop, trace, selected frame, conversion or generated result with input digests and transformation identity;
• cache: decoded pixels, GPU textures or acceleration state that can be deleted without changing the semantic document.

This avoids a common failure: storing only a decoded bitmap and calling the source image preserved, or hashing an editable asset by its current bytes and therefore breaking every reference after replacement.

Candidate .nuif package

The proposed first package is a deliberately small deterministic ZIP profile:

mimetype
manifest.cbor
document.cbor
blobs/sha256/<digest>

mimetype is first and stored. Manifest and document are deterministic CBOR. Every embedded blob is addressed by exact SHA-256 bytes. The first profile uses stored members only, fixed metadata and sorted ASCII paths to make independent writer byte equality attainable and to avoid compression-version variability.

There are three hashes:

• semantic document hash over canonical document.cbor;
• resource digest over each exact blob;
• package hash over the exact ZIP artifact.

Cache or report changes may change the package hash while leaving the semantic document hash untouched. Bare canonical forms remain .nuif.json and .nuif.cbor. Historical alpha .nuif raw files need read-only detection during migration; new .nuif output becomes the package only after RFC acceptance.

The package reader rejects duplicate or unsafe paths, symlinks, directories, encryption, split archives, unsupported compression, inconsistent headers, undeclared/missing blobs and digest mismatch. It does not extract to a filesystem. Exact byte fixtures, member/resource limits and two independent writers are acceptance gates.

Package-to-session handoff uses shared immutable buffers. The release gate passes an 8 MiB resource through package, handle map and session with the same allocation pointer while keeping handoff allocator traffic and retained bookkeeping below 1 MiB. This prevents a host from paying one full resource copy merely to enter the core.

Images

The original encoded image is authoritative. A semantic image asset records its resource digest and intrinsic interpretation; each image paint records fit, crop, transform, sampling, opacity and color conversion. Derived decoded pixels and GPU textures are caches.

PNG is the correct first format because its current W3C specification covers lossless encoded pixels, alpha and explicit colour metadata. The executable nuif-png-rgba8-0 baseline chooses an intentionally smaller contract: non-interlaced RGBA8, no ancillary metadata or one valid sRGB chunk, encoded samples interpreted as sRGB, straight decoded alpha, identity encoded orientation, declared fit/crop/sampling/opacity and bounded integer CPU composition. png 0.18.1 and zune-png 0.5.2 must emit identical RGBA bytes for the accepted fixtures; encoded resources remain digest-identical through package edits.

This avoids pretending that decoder agreement on simple images settles PNG Third Edition. The compatible nuif-png-basic-rgba8-1 profile now admits the lossless-to-RGBA8 subset: 1/2/4/8-bit greyscale and indexed images, RGB8, greyscale-alpha8, RGBA8 and valid palette/colour-key transparency. It preserves encoded bytes and requires exact normalized RGBA agreement between both decoders. It is separately named so profile zero never changes meaning.

Image-paint affine semantics are orthogonal to decoder choice. The executable matrix [a c tx; b d ty; 0 0 1] maps crop-local source coordinates forward into the fitted rectangle. The CPU reference inverse-maps destination pixel centers, clips to the entity, and rejects singular or numerically unbounded matrices. Flip, rotation and translation fixtures make composition order observable; live host trials are still required for vendor interoperability.

Decoded pixels are interned once per digest/profile in the renderer-independent scene and commands carry compact deterministic handles. A 64 MiB total is preflighted before each new inflation. The release gate retains one 1 MiB surface for 1,024 image instances under 8 MiB allocated and 4 MiB retained, and rejects a 64 MiB plus 16 byte declared total before the second decode.

Any broader profile still has to pin:

• accepted chunks and metadata conflicts;
• color-space precedence and output space;
• Exif orientation;
• conversion and premultiplication points for every accepted colour signal;
• sampling and compositing;
• encoded, pixel, decoded-byte, chunk and metadata limits;
• independent decoder and malformed-input fixtures.

16-bit/interlaced PNG, CICP/ICC/gamma/chromaticity, Exif, animation, perspective/tiling and host-specific affine equivalence are not claimed. A Linux/Windows/macOS CI matrix runs the profile, but the cross-platform claim remains withheld until its hosted artifacts pass. JPEG, WebP, AVIF, video and SVG follow as separate profiles. Freezing a frame or tracing a screenshot crop is a derived approximation, not recovery of the original asset. Generative upscaling/inpainting requires an explicit user policy and cannot silently become canonical source evidence.

Fonts

Exact typography depends on exact font bytes, face/collection index, variation axes, features, coverage, shaping inputs and renderer parameters. Packaging also depends on redistribution policy.

The proposed policy states are portable, private_authoring, linked, substituted and unavailable. OpenType fsType is preserved as machine- readable evidence, including restricted/preview/editable/no-subsetting/bitmap flags, but is not treated as a complete legal license decision.

The executable nuif-opentype-static-single-0 baseline accepts only one canonically packed, checksummed TrueType-outline sfnt face at index zero. Skrifa 0.46.2 supplies package-facing metadata after NUIF validates the directory, ranges, packing and checksums and directly checks required sfnt and OS/2 fields. A committed hb-info 14.4.0 capture independently checks Ahem metrics, family, tables and Unicode coverage. Exact bytes, family names, coverage, fsType, license expression and explicit embedding review must agree. Package encode/decode and caller-resolved linked bytes run the same validation. Four static TrueType fixtures are accepted, while six package trials distinguish portable, private-authoring, linked, substituted and unavailable outcomes. Each accepted inspection and packaged-font validation is also measured after warmup against a 4 MiB allocator-traffic and 2 MiB retained reference ceiling; these are implementation regressions, not format semantics. Six additional trials retain requested identity separately from a stable font asset, render with an available declared replacement as approximated, and emit no text command with item-level unsupported fidelity when replacement bytes or the font are unavailable.

This is intentionally not general OpenType support. TTC, CFF/CFF2, variable, color, bitmap, SVG and WOFF/WOFF2 sources, historic ambiguous permission combinations, subsetting, cluster-level fallback, arbitrary packaged-font shaping and cross-platform raster behavior remain separate fixtures and profiles. The configured three-OS parser/package matrix does not establish cross-platform raster behavior. Parser acceptance and fsType do not grant redistribution rights.

Browser capture can identify platform fonts used for a node and capture downloaded web-font response bodies. It generally cannot retrieve arbitrary local font bytes. A family/PostScript name is therefore never exact resource identity. Missing bytes produce a link, substitution or unavailable fidelity record instead of a false portable-font claim.

Source-backed browser capture

Static Tree-sitter source synchronization and live browser capture solve different problems. The former preserves source spans for a bounded authored subset. The latter runs a pinned browser to observe actual cascade, layout, fonts, resource responses, accessibility and pixels. They should correlate through provenance, not become one oversized adapter.

The first browser-capture profile records browser/protocol build, OS, viewport, DPR, page scale, locale, timezone, color/reduced-motion preferences, font environment, scroll/pseudo state, navigation identity, readiness/network policy and animation freeze. It collects:

• original HTML/CSS and stylesheet text where accessible;
• DOM snapshots including available frame/template/shadow content;
• boxes, inline text boxes, paint order and a bounded style set;
• downloaded image/font/style response bodies and hashes;
• platform-font usage and font readiness;
• accessibility tree;
• reference screenshots with exact parameters.

Canvas, WebGL, video and worklets are bounded observation surfaces; a frame can be preserved without pretending its generating program was reconstructed. Cookies, authorization headers, credentials, storage and secret form values are not exported. Scripts remain inert.

Multiple viewports and states are more valuable than one oversized capture: they constrain layout hypotheses and permit held-out responsive evaluation.

The automated nuif-cdp-live-0 segment now implements that boundary for one loopback fixture and exact Chrome for Testing 152.0.7977.64. It starts fresh temporary profiles, retains a structured runtime context, waits for the exact navigation loader and declared freeze/readiness point, captures bounded DOM/layout/background/font/accessibility/resource/PNG evidence, and replaces opaque browser node IDs with deterministic preorder identities. Four runs at 360, 768, held-out 900 and repeated 360 px retain exactly the expected five response bodies and repeat the narrow capture bytes. Five exercised query/cookie/storage/authorization/header canaries are absent from serializable capture, observations, proposals and package bytes. The two fitted viewports beat the one-viewport freeform baseline on the held-out fixture.

The separate bounded nuif-layout-inference-0 artifact ranks row stack, column stack, Grid, linear constraint and fixed freeform candidates using only the 360/768 px observations. It then evaluates the untouched 900 px holdout, where the selected constraint records 0.0626 normalized error versus 0.2918 for freeform. All alternatives and exact geometry observation identities are retained; confidence is raw and uncalibrated, and the result remains inferred. The trial tests a mechanism on one fixture, not general accuracy or recovery of original authored intent.

This is a falsifiable local baseline, not the entire profile described above. Cross-browser/OS reproduction, opaque frames and response bodies, full matched-style/source correlation, canvas/video frame capture, authenticated sites and licensed real-page evaluation remain open. WebDriver BiDi is the standards-track transport to revisit as its implemented evidence surface grows; Playwright is the higher-level candidate when NUIF owns a real multi-engine matrix and can make one tool the browser-version authority.

Screenshot reconstruction

A screenshot supplies visible samples but not the unique scene graph or layout program. The recommended pipeline is:

screenshots + contexts
  -> OCR and baselines
  -> deterministic regions, colors, edges, repetitions and asset candidates
  -> optional replaceable UI grounding
  -> typed observation graph with confidence and evidence regions
  -> replaceable reasoner proposes hierarchy/layout and NUIF operations
  -> core validates and applies atomically
  -> deterministic layout/render
  -> text/structure/geometry/resource/visual differences
  -> bounded corrective operations

The model emits typed operations, not an unconstrained full document or code to execute. Invalid and stale transactions fail without partial state. High- resolution full views, overlapping tiles and semantic crops share explicit coordinate transforms; duplicate or conflicting observations remain visible.

The result contains a valid document or no-result, accepted operation log, observations, derived resources, item fidelity, alternatives/abstentions, evaluation report and exact pipeline artifact identities.

Evaluation before training

The benchmark has separate synthetic-exact, licensed real screenshot and source-backed suites. Synthetic NUIF rendering provides exact entities, properties, operations and resources. Real images need human-reviewed visible targets and must preserve ambiguity. Source-backed cases evaluate retained bytes and observations unavailable to the screenshot-only route.

Required metrics include:

• valid operation/document rate;
• OCR region recall, character/word error and baselines;
• element precision/recall and hierarchy error;
• property/geometry accuracy;
• held-out viewport behavior;
• exact resource digest only when bytes exist;
• provenance/fidelity honesty;
• accessibility evidence where justified;
• raw pixels, FLIP, SSIM and pinned LPIPS diagnostics;
• calibrated confidence, abstention and risk/coverage;
• latency, peak RAM/VRAM, iterations and cost.

No pixel score is sufficient. A page-sized screenshot can be visually perfect and semantically useless. Structural/text/resource/edit-task metrics prevent that reward shortcut. Dataset splits group by origin, template, component, font, resource and generator; near duplicates cannot cross splits.

The executable nuif-reconstruction-corpus-manifest-0 turns that rule into a bounded audit. It pins the data snapshot, dataset card, evaluator and every input/target by digest; records public/restricted/withheld disclosure and evaluation/calibration/adaptation/redistribution permission independently; and rejects exact artifact or declared family reuse across adaptation, calibration, validation and test. Private/authenticated records also require explicit authorization and a withdrawal-policy artifact. This is declaration integrity, not automated legal review or duplicate discovery; real records and their group assignments still need independent human/tool review.

The ablation ladder is deterministic OCR/CV, one-shot reasoner, observation- assisted reasoner, hierarchical crops, multi-viewport ranking, correction loop, then any tuned or distilled student. Every addition uses the same frozen holdout and budget.

Adaptation and distillation

Training is justified only after the untuned loop and error taxonomy are reproducible, rights-cleared traces exist, and the remaining errors appear learnable. Training examples contain input hashes, observation versions, proposals, diagnostics, accepted operations, intermediate renders/differences and final package/fidelity reports. Positive sequence targets are validated accepted transitions, not raw model transcripts.

Compare prompt/schema/tool improvements and retrieval before fine-tuning. If adaptation remains justified, compare ordinary supervised tuning, low-rank adaptation and quantized low-rank adaptation under equal data and evaluation. Quantized adaptation is a memory technique, not an accuracy claim.

Sequence-level distillation may train a smaller student from the best evaluated teacher pipeline. The teacher is a measured system of tools plus a model, not a provider name. Distillation transfers errors too, so render validation and held-out evaluation remain mandatory.

Models, processors, adapters and datasets are separately versioned optional artifacts with digests, model cards, dataset datasheets, license lineage and training manifests. They never redefine nuif-core or travel as ordinary document resources.

The executable provider boundary uses a deliberately small canonical wrapper, not a new AI bill-of-materials vocabulary. It binds NUIF capabilities, execution modes and observation/proposal profiles to exact implementation, model, processor, adapter, quantization, prompt and tool artifacts. Observation bundles carry the complete manifest registry, so a digest cannot dangle and a proposal cannot substitute an unpublished provider before mutation. Released or learned providers point to content-addressed SPDX 3.0.1 or CycloneDX 1.7 inventory; learned providers also point to a model card. This complements runtime packaging such as MLflow or ONNX external data without making either a required NUIF dependency.

Private/authenticated captures default to local processing, no retention and no training. Remote transfer, telemetry, retention and training are independent consent/policy decisions.

Maturity boundary

The current 0.1.0-alpha.3 label belongs to the developer editor application. It provides no evidence that the broad image/font resource, portable browser capture or screenshot reconstruction accuracy profiles are complete. A deterministic package, narrow PNG/static-font segments, fixed provider-input contracts and one pinned local live-browser segment are implemented, but their deliberately narrow evidence does not promote the broader profiles.

Promotion requires the package/resource cross-writer fixtures, pinned capture reproduction, baseline/closed-loop/calibration harness, leak-resistant licensed evaluation data, independent result reproduction and at least one real edit workflow that benefits from the inferred semantics. Until then the work is research and proposed specification text, not a standard or production reconstruction promise.

Primary research records

• nuif:research:resource-packaging-and-source-capture-synthesis
• nuif:research:model-agnostic-screenshot-reconstruction-and-training
• nuif:research:provider-artifact-manifests-and-ai-boms
• nuif:research:epub-ocf-package-container
• nuif:research:oci-resource-descriptors
• nuif:research:opentype-font-embedding-and-portability
• nuif:research:ttf-parser
• nuif:research:fontations
• nuif:research:chromium-source-backed-ui-capture
• nuif:research:live-chromium-cdp-capture
• nuif:research:design2code-real-world-benchmark
• nuif:research:pix2struct-screenshot-parsing-pretraining
• nuif:research:screenai-ui-annotation
• nuif:research:confidence-calibration-and-selective-prediction
• nuif:research:lora-low-rank-adaptation
• nuif:research:qlora-quantized-adaptation
• nuif:research:sequence-level-knowledge-distillation

Canonical module.