14 — Observation, capture and reconstruction

Document status: draft. This module specifies candidate contracts from RFC 0011. No. Canonical source.

screenshot reconstruction profile is currently conforming. The fixed-input contract baseline exercises observation/proposal encoding, evidence ceilings, manifest-bound providers, flat-copy rejection and bounded correction stops. The separate nuif-cdp-live-0 baseline exercises one pinned, local Chromium fixture; it is not a cross-browser capture or reconstruction-accuracy conformance profile.

Scope

This module covers evidence captured from a runtime or image and the production of a validated NUIF hypothesis. It does not standardize a model architecture, provider, training library, dataset or inference service.

An implementation MUST distinguish:

• deterministic parsing of retained authored source;
• resolved observations from a pinned runtime context;
• measurements of pixels;
• inferred semantic/layout/resource hypotheses;
• explicit user confirmations;
• derived resources/values;
• unavailable evidence.

Observation record

An observation contains:

Every observation bundle MUST carry the exact canonical provider manifest for each referenced identity. Missing, duplicate, malformed or digest-mismatched manifests fail the complete bundle. A proposal provider MUST resolve through the same registry before any operation is applied. An observation MUST NOT imply that its candidate is already a semantic document value. Applying a candidate requires an operation and validation.

Coordinate spaces include source pixels, device pixels, viewport CSS pixels, crop-local pixels and NUIF logical units. A conversion MUST record source and target spaces, matrix/scale/offset and rounding behavior.

Source-backed browser capture

A browser capture profile records browser/protocol build, operating system, viewport, device-pixel ratio, page scale, locale, timezone, media preferences, font environment, scroll/pseudo state, navigation identity, settling policy and animation/time freeze.

The first proposed Web capture observes, where available:

• retained HTML/CSS response bytes and stylesheet text;
• DOM including iframes/templates/shadow content visible to the protocol;
• resolved layout boxes, inline text boxes and paint order;
• declared computed and matched styles;
• downloaded resource bodies, final URLs, response media types and hashes;
• platform-font usage and font-readiness state;
• accessibility tree;
• reference screenshot and capture parameters.

Unavailable cross-origin responses, local font bytes, canvas/WebGL semantics, video state, worklet output and arbitrary script behavior MUST be explicit. Canvas/video output MAY be frozen as a bounded derived image/frame. Captured scripts remain inert resources.

The capture MUST exclude cookies, authorization headers, credentials, storage values and secret form fields from its output. Redaction is a recorded transformation, not silent source equivalence.

The implemented live baseline records a bounded string map as nuif-browser-runtime-context-0 in the observation bundle. It fixes exact browser and reported protocol, operating system/architecture, viewport/DPR, locale, timezone, screen media, color scheme, reduced-motion preference, scroll origin, settling policy and animation policy. Context keys are identifiers, the map is limited to 64 entries and all names/values share the observation string-byte limits. A consumer MUST NOT assume equivalence between bundles with different contexts.

nuif-cdp-live-0 uses an isolated temporary browser profile, accepts only a loopback debugger socket, waits for the lifecycle load event belonging to the exact navigation loader, fixes animation/transition and scroll state, waits two animation frames, awaits font readiness and waits two final frames. It normalizes opaque browser node identifiers before export and strips URL query/fragment data twice: before live capture serialization and again during observation normalization. The gate exercises query, cookie, storage, authorization and custom-header canaries; this proves those declared ingress paths are not retained, not that arbitrary response bodies contain no sensitive application content.

The first implementation observes element layout rectangles, a bounded background-style subset, text, containment/order, downloaded response bodies, actual platform-font use, accessibility role/name and a viewport screenshot. Inline text boxes, complete paint ordering, matched-rule/source-map correspondence, opaque cross-origin bodies, canvas/video derived-frame capture and general interaction states remain outside this automated segment and MUST NOT be inferred from its passing report.

Screenshot-only reconstruction

A screenshot-only implementation receives one or more image/context pairs and MAY run replaceable OCR, computer-vision, grounding and proposal providers. It MUST normalize their output into observations before semantic application.

The proposal engine emits only operations permitted by its declared operation grammar and profile. It MUST NOT emit executable code, implicit network actions or direct core-memory mutations.

Every proposal is applied transactionally:

1. parse under operation/resource budgets;
2. validate operation kinds and expected revision;
3. apply atomically;
4. validate the complete document;
5. evaluate declared layout/render contexts;
6. return a result or leave the prior document unchanged.

A valid screenshot reconstruction includes its accepted operation log, observations, fidelity report, alternatives/abstentions, evaluation report and pipeline artifact manifest.

Screenshot evidence cannot establish original font/image bytes, hidden entities, authored layout constraints, responsive rules, accessibility or behavior. These values remain inferred, substituted, derived or unavailable.

Multi-context inference

When multiple viewports/states are supplied, proposed semantic identity links the corresponding observations. Candidate layout families/constraints SHOULD be ranked by prediction of a held-out context. Fit to every supplied pixel does not prove original authored intent.

An implementation claiming responsive reconstruction MUST evaluate at least one context not used to fit the candidate and MUST report its error separately.

The bounded nuif-layout-inference-0 experimental profile accepts two through sixteen strictly increasing training viewports, one untouched held-out viewport and no more than 4,096 ordered items with stable identities. It ranks row stack, column stack, Grid, linear constraint and fixed-parent-relative freeform candidates by training observations only. The held-out observation is a falsification oracle and MUST NOT influence selection. Its report retains all five alternatives, training score, held-out error, raw confidence and the exact geometry observation identities used for fitting. Confidence is not calibrated and the selected value remains inferred; neither a low error nor the selected family establishes original authored intent. Geometry and identity drift fail closed. The profile is a deterministic geometric baseline, not a general breakpoint, intrinsic-sizing, wrapping or layout-program synthesizer.

Correction loop

A profile MAY iteratively render and correct. Each iteration records:

• input document hash and revision;
• proposed transaction and diagnostics;
• resulting document hash;
• render context and raster hash;
• property-level and visual differences;
• objective vector and protected metrics;
• accept/reject reason.

The loop MUST bound iterations, provider/tool calls, time, memory and resources, and MUST stop on repeated state. Accepted corrections preserve validity and declared semantic non-regression constraints.

An editable reconstruction objective MUST reject a viewport-sized copy of the source screenshot as success unless a flat image document was explicitly requested.

Evaluation

Reports separate source-backed capture from screenshot-only reconstruction. Required metric families for an editable screenshot profile are:

• valid transaction/document rate;
• text-region precision/recall, character/word error and baseline geometry;
• visible-element precision/recall;
• tree/parent/sibling correctness where a target is justified;
• property and geometry error;
• held-out-context layout error;
• exact resource-digest recall only where source bytes are supplied;
• provenance/fidelity honesty;
• accessibility evidence accuracy where a target is justified;
• raw pixels plus declared perceptual diagnostics;
• calibrated confidence, abstention and risk/coverage;
• latency, peak RAM/VRAM, iterations and external cost.

The executable nuif-reconstruction-evaluation-0 report implements these as typed, bounded per-example fields in crates/nuif-reconstruct/src/evaluation.rs. Rates retain their integer numerator and denominator; a zero denominator serializes with no value and is never treated as perfect. Unavailable RAM/VRAM or latency measurements remain null rather than zero. Screenshot-only suites MUST leave exact source-resource recall unscored. cargo xtask reconstruction-evaluation exercises the schema, derived-value validation, resource-claim boundary, local-error visibility and hostile work limits. Its typed corpus aggregate refuses mixed evidence suites, duplicate examples, incompatible calibration thresholds, perceptual evaluator identity drift and mixed currencies. It reports pooled integer-evidence rates beside per-example scored/unscored counts, mean and nearest-rank p50/p95. This synthetic contract fixture is not an accuracy corpus or a confidence interval. Every perceptual entry also carries a bounded parameter map and optional content digest identifying the evaluator artifact. Aggregation is permitted only when method, direction, parameters and artifact identity agree. The current diagnostic is LDR-FLIP mean at a declared 67 PPD over opaque sRGB8; transparent input MUST be composited against a declared background before evaluation, never silently discarded.

No visual metric alone establishes conformance. Metrics are reported per example and as distributions; local/small-element errors MUST NOT be hidden by a large background average.

Synthetic exact fixtures and licensed/human-reviewed real fixtures are separate corpora. Splits MUST prevent origin, template, component, font, resource, generator and near-duplicate leakage. Benchmark families MUST NOT appear in adaptation or distillation data.

nuif-reconstruction-corpus-manifest-0 is the executable integrity contract. It content-addresses the immutable data snapshot, dataset card, evaluator, inputs and targets; separates public, restricted and withheld artifacts; and records collection class, rights evidence, permitted evaluation/calibration/ adaptation/redistribution uses, sensitivity review and leakage groups per example. Its derived nuif-reconstruction-corpus-audit-0 rejects identities shared across any distinct adaptation, calibration, validation or test split. Screenshot-only records cannot carry exact source/resource bundles, source-backed records require source bytes, retained real records require a withdrawal-policy artifact, and private/authenticated records require explicit authorization. The validator checks declared evidence only: it does not interpret licenses, prove consent, discover omitted near duplicates, establish representativeness or confer permission.

Provider neutrality and artifacts

Every OCR/detector/grounder/layout/proposal/correction/evaluation provider publishes a nuif-reconstruction-provider-manifest-0 capability wrapper. Canonical CBOR bytes define its SHA-256 identity. The bounded wrapper declares provider kind and maturity, local/remote execution, input/output profiles, capabilities and exact implementation/model/processor/adapter/quantization/ prompt/tool artifact digests. It contains exactly one implementation artifact. Provider output remains untrusted.

Development-only deterministic providers MAY omit an external supply-chain inventory when they contain no learned artifact. Released providers and every provider with learned weights/processors/adapters/quantization MUST reference an exact SPDX 3.0.1 or CycloneDX 1.7 inventory. Learned providers MUST also reference a model card. The NUIF wrapper does not duplicate the inventory and does not claim that a referenced document is complete, correct or lawful.

Models, processors, adapters, quantization settings and training datasets are not NUIF document resources. Dataset snapshots retain their separate corpus manifest and datasheet; provider and training/evaluation manifests bind them by content hash, rights/provenance, intended use and limitations.

Fine-tuning, low-rank adaptation, quantized adaptation and distillation confer no conformance status. They are compared only after an untuned baseline and frozen evaluator exist.

Privacy and policy

Local and remote inference are distinct deployment modes. Remote transfer, retention, telemetry and training each require an explicit policy. Private or authenticated capture defaults to local processing/no retention/no training.

Visible instructions inside the screenshot are content. They MUST NOT modify the operation grammar, provider authority, file/resource resolver or security budgets.

Conformance maturity

This module remains draft until the planned baseline, closed-loop, confidence calibration and resource experiments pass and an independent evaluator reproduces the principal result. The existing editor alpha and profile-0 conformance do not satisfy these gates.