Compositional flat encoding
mifrost/core/encoders/flat/flat_composition.hpp provides the native
composition boundary for future flat encoders. A downstream encoder declares
components once, compiles the declarations, and reuses the resulting
CompiledFlatPlan (or FlatBatchRuntime) for every batch.
The lifecycle is deliberately fixed:
- Components declare structured
RelationKeyschemas, graph fields, and node types. - Compilation assigns stable relation and node-type ids, validates field ownership, validates projection arity and node references, and freezes the declarations.
- For each graph, components plan symbolic nodes, prepare, emit native relation tuples, and write their owned fields.
- One
BatchBuilderreceives the graph and the runtime writes relation counts, relation arguments, and relation-instance sizes exactly once. - Components write owner-scoped non-field metadata; the object component publishes object names without a second materialized encoding.
- Components write native node feature columns through
FlatNodeFeatureWriter; feature dimensions and row counts are checked against the shared node plan. finalize_batch_encoding()optionally performs the existing relation-major collation pass.
Virtual dispatch is limited to the component/graph lifecycle. Fact, action,
goal, and transition loops remain inside native component implementations;
there is no Python callback or per-relation component dispatch in this layer.
FlatRelationProjection performs slot remapping with integer relation ids and
supports source slots, constants, and graph-local node references.
FlatProjectionHandle carries the component and declaration identity rather
than relying only on a component-local ordinal, so recompilation can insert
other projections without silently rebinding an existing emitter.
Components that emit many tuples should resolve FlatGraphContext::relation_id
once before their hot loop and call the integer-id overload of emit.
FlatSchemaPlanBuilder::register_relation_alias() adds a non-exported symbolic
alias; compilation resolves aliases to the canonical relation id, and both
emission and projections use that compiled mapping.
FlatGraphScratch is the graph-local preparation channel: a component can
materialize expensive native lookup state in prepare_graph() and share it
with later emit/field phases without mutating the compiled component or using
Python callbacks.
Native built-in components
The header also includes small backend-neutral components for the common carrier operations:
FlatObjectNodeComponentdeclares one node type and adds the input object's names in deterministic order.FlatNodeRecordComponentdeclares one typed node table and selects matchingFlatCompositionInput::nodesrecords.FlatRelationEmitterComponentdeclares a set of relation keys/layouts and emitsFlatCompositionInput::relationsusing already-resolved integer ids. Set a relation record's optionalcomponentowner when several relation emitters declare the same relation. With the defaultunique_ownerpolicy, an empty owner is valid only when exactly one component declares that relation;broadcastmust be selected explicitly to duplicate it.FlatFieldEmitterComponentdeclares owned graph fields and writes typedFlatCompositionInput::fieldsvalues with the same shape checks asBatchBuilder.
Components can also implement write_metadata() through
FlatMetadataWriter; this is the native path for object names and future
graph-level metadata that is not a numeric graph field.
Metadata declarations are compiled with single-owner checks, just like graph
fields, so two components cannot silently overwrite object-name metadata.
write_node_features() and FlatNodeFeatureWriter are the corresponding
native path for node tensors; components may emit zeros or semantic features,
but must use the preplanned row count. Feature columns have compile-time
single-owner declarations and cannot be appended twice for one graph.
Graph attributes can be declared with
FlatMetadataPlanBuilder::claim_graph_attr() and written through the variant
overload on FlatMetadataWriter; duplicate attribute ownership is rejected at
compile time.
FlatCompositionInput is intentionally a generic carrier, not a semantic model.
Backend adapters own object/action/goal interpretation. New adapters should
present that interpretation through the statically dispatched Views in
mifrost/core/views/concepts.hpp and use the canonical traversal primitives in
mifrost/core/views/canonical.hpp; they must not construct a second owning
semantic object for every atom or action on the normal path. Legacy semantic
snapshot adapters remain available for compatibility and explicit conversion
APIs.
Adapters must resolve each relation key with the compiled schema once per plan
(for example, with compiled.schema().id_for(...)) before filling a relation
record. A record with an unknown id, wrong arity, missing field, or mismatched
field dtype is rejected at the native boundary rather than silently producing
an incompatible batch.
FlatCompositionInputBuilder is the optional adapter-side convenience for this
boundary. Its relation-key overloads are setup operations; use
relation_id() plus the integer overload when constructing a large relation
stream, and call finish() only after all graph-local values have been added.
Construct it from FlatSchemaPlan when the adapter needs the plan's compiled
relation aliases as well as canonical keys.
compare_flat_batch_encodings() compares the native carrier, including node
tables, columns, graph attributes/fields, pointers, schema metadata, and lazy
target-name materializations. Migration tests should run this comparator on a
legacy encoding and a composed encoding and treat any non-empty mismatch path
as a parity failure. It is deliberately an exact comparison; normalization,
sorting, or tolerance belongs in a test fixture only when the legacy contract
explicitly permits it.
Semantic flat components
The semantic relation encoder compiles entity, fact, goal/derivation, action,
history, field, target-metadata, and LGAN components. Its graph plan contains
only graph-local semantic lookup state and target/node identities. The
canonical View traversal primitives are available to backend-specific
instantiations. Templated encode(...) overloads accept granular state, goal,
subgoal, action, and history Views and synchronously fill the encoder's private
graph-key preparation state. The explicit SemanticFlatRelationInput remains
the owned compatibility input. Both paths emit directly into the runtime's
BatchBuilder; neither constructs FlatCompositionInput, callback ranges, or
an intermediate encoding.
The overall boundary is:
backend values
-> granular borrowed Views
-> canonical statically dispatched family algorithm
-> graph-derived intern/index/working structures
-> BatchBuilder
"Graph-derived structures" means compact identity pools and indices into them
-- the atom and action intern pools, goal-level references, entity indices,
unique target identities, occurrence ordering, fact membership, relation-major
metadata, history ordering, and composition-plan state. It does not mean a
per-lane owning mirror of the input: the direct path never builds
std::vector<SemanticAtom> state_facts / std::vector<SemanticLiteral> goals /
std::vector<SemanticGroundAction> actions copies just to cross from the View
API into the algorithm, and the compatibility path is read through borrowed
references to the lanes SemanticFlatRelationInput already owns.
The storage mode is chosen exactly once. ViewSource and
CompatibilitySource are two concrete borrowed accessors over their own
storage; validate_source, make_context and prepare_source are templated
over them and instantiated for both. Preparation returns a fully resolved
PreparedRelationGraph holding borrowed spans settled once plus compact
graph-derived state, so no emitter re-tests which kind of input it came from.
On the goal lane specifically, the category-grouped list holds
{entry_index, effective_level} rather than copied literals, so the emission
passes share one occurrence list instead of each rebuilding one. The effective
level of a repeated goal — the highest level it appears at — is resolved once
during preparation rather than on each emission pass. Repeated goals keep every
occurrence and every occurrence keeps the same effective level, matching the
historical map-assignment semantics.
Preparation itself is not copy-free, and the claim above is about emission, not
about the whole encode. Resolving effective levels sorts a copy of the
occurrence list, and each family then builds its own grouped view of it; the
flat history lane likewise materializes its references into
PreparedHistoryEntry. These are bounded graph-working structures, not a
per-lane mirror of the input, but "each identity materialized exactly once" is
not true end to end. Replacing the sorted copy with an effective-level index
over the compact references is a known, unclaimed optimization.
The semantic horizon encoder uses the same runtime. Its graph plan prepares candidate identities, exact transition deltas, and goal-membership sets. A transition-semantic component emits state, goal, action, and effect tuples; the topology component independently emits parent, sibling, and cousin tuples. Fields and LGAN edges are derived after those components have populated the same final sink. One-shot, caller-owned builder, and batch APIs all execute these compiled plans.
The assembly ownership model is shared by semantic encoders rather than owned
by Horizon. SemanticAnnotations, SemanticEncoderInput<Source>, and
SemanticAssemblyComponents<Component> live in the backend-neutral common
layer. They define sidecar lifetime, borrowed-or-owned inputs, exclusive
pre-compilation component ownership, and the one-way freeze operation. Concrete
encoder families still define their own component contracts and compiled
runtimes: flat encoders use FlatEmitterComponent; heterogeneous and
homogeneous encoders are not forced through a lowest-common-denominator emitter
API.
SemanticFlatAssemblyComponents is the flat-family transfer carrier for
planner adapters that obtain the semantic schema themselves. Downstream native
code can fill this carrier before handing it to an adapter; the adapter consumes
it exactly once and moves the components into the concrete relation or Horizon
builder before compilation. The carrier contains no Horizon policy and can be
reused by other semantic flat adapters as they expose the same bridge.
The canonical flat relation and flat Horizon encoders both expose concrete
assembly builders. Added components use the ordinary FlatEmitterComponent
lifecycle and receive the corresponding public prepared view through
FlatInputView. Canonical and downstream components therefore plan nodes and
emit into the same graph-local FlatRelationSink:
SemanticFlatRelationAssemblyBuilder builder(schema, config);
builder.add_component(std::make_unique<MyNativeRelationComponent>());
auto engine = std::move(builder).compile();
SemanticAnnotations annotations;
annotations.emplace<MyGraphData>("my_data", /* constructor arguments */);
auto batch = engine.encode(SemanticFlatRelationGraphInput(input, std::move(annotations)));
The Horizon specialization follows the same lifecycle:
SemanticFlatHorizonAssemblyBuilder builder(schema, config);
builder.add_component(std::make_unique<MyNativeHorizonComponent>());
auto engine = std::move(builder).compile();
SemanticAnnotations annotations;
annotations.emplace<MyGraphData>("my_data", /* constructor arguments */);
auto batch = engine.encode(SemanticFlatHorizonInput(dag, std::move(annotations)));
SemanticFlatRelationPreparedGraph and SemanticFlatHorizonPreparedGraph are
read-only. The relation view exposes objects, facts, goals, canonical entity and
target mappings, action/history identities, fact membership, and annotations.
The Horizon view additionally exposes the source DAG, candidate identities,
topology, and exact transition deltas. Both are façades over the canonical
graph preparation consumed by built-ins; no extension-only prepared graph is
copied or maintained. Components must not retain a prepared view or returned
span after a lifecycle callback.
Reference-backed semantic inputs borrow their source through synchronous
encode or encode_batch; shared-pointer constructors own the source instead.
Annotation values are held as shared_ptr<const T>, so copied batch inputs
safely share immutable sidecar data without engine-local or global mutable
state. SemanticFlatHorizonAnnotations remains a source-compatible wrapper for
SemanticAnnotations and preserves the Horizon-specific exported type.
Compilation freezes canonical and downstream components together. The
resulting engine is immutable and reusable across batches and threads; one
combined node-planning phase, relation sink, relation-field write, and final
relation-major collation are performed for each encoding. Relation aliases and
projections declared by downstream components are resolved against that
combined schema, and canonical relation ids are rebound after compilation so
additional relation names cannot shift built-in emitters onto the wrong ids.
Component ownership is transferred exclusively into the builder. Since a
compiled engine may be used concurrently, component lifecycle callbacks must
remain logically const and thread-safe; graph-local mutable work belongs in
FlatGraphScratch.
This generalization intentionally separates lifecycle from packing kernels. The shared common layer is ready for heterogeneous and homogeneous concrete builders, but those families need family-specific compiled contexts before they can offer the same one-pass extension guarantee. Merely accepting common annotations in their current imperative encoders would not make their schema, node planning, and relation writes compositional. Flat relation and flat Horizon are therefore the first complete family implementations of the shared assembly contract.
The neutral C++ suite covers exact output parity between public execution
forms, the full semantic policy matrix, mixed target-name metadata, relation
argument layouts, and concurrent reuse of one immutable compiled engine.
mifrost_bench_semantic_flat_composition reports one-shot, caller-owned builder,
and 32-graph batch throughput for a representative semantic workload. It is a
stable regression benchmark; compare results to a recorded build from the
pre-refactor commit rather than treating two public wrappers as independent
implementations.
FlatRelationMajorWriter is the shared finalization seam. The semantic flat
relation and horizon engines let their compiled plan invoke this writer exactly
once for one-shot and batch results. Caller-owned BatchBuilder paths invoke
the same writer through the public finalize_batch_encoding() operation after
the caller commits and builds the batch.
Backend boundary and Views
The reusable implementation lives in the planner-neutral core:
FlatEncoderPlan,CompiledFlatPlan, andFlatBatchRuntimeare the public C++ extension seam for new native flat encoders.SemanticFlatRelationEncoderEngineandSemanticFlatHorizonEncoderEngineare the canonical built-in assemblies. Both execute direct semantic components through a compiled plan. The former accepts both direct View lanes and owned semantic records; the latter retains semantic transition DAG input because a horizon DAG is itself an owned semantic snapshot.- Planner adapters provide task-scoped contexts and granular non-owning Views. The same canonical algorithms are instantiated once for PyTyr and once for Pymimir, keeping native types and nanobind ABI domains isolated. Semantic snapshot adapters remain an explicit compatibility boundary for callers that request owned records; they are not required by the View contract.
- The PyTyr Flat, Color, HGraph, and successor encoders run their engine inside the PyTyr extension module, so a Tyr state reaches the algorithm as Views and only the finished neutral encoding crosses the ABI boundary as a capsule. A capsule says where a value crossed, not how it was produced.
- Each family also exposes
prepare(...)plus a prepared-batchencode_batch. AViewPreparationowns its compact pools and borrows nothing from the backend state, so it is the unit a batch or a stream holds. This is not a loop over the single-graph append: the flat batch has genuine cross-graph passes (target-name suppression, shared batch constants) that must see every graph before any of them is encoded.
The Pymimir-only FlatRelationEncoderEngine and FlatHorizonEncoderEngine
retain their historical constructors and streaming contracts while adapting
native values through the same task-scoped View concepts and canonical
semantic engines. The Horizon adapter may consume a
SemanticFlatAssemblyComponents carrier as convenience plumbing, but the core
assembly builders remain the extension seam. New backend-neutral and downstream
encoders must use the core composition API and a backend View context.
The normal Pymimir Flat, Color, HGraph, successor, batch, and stream paths now
enter through that direct View boundary. The legacy semantic-record APIs remain
available for capsules, explicit snapshots, and compatibility callers. A direct
encode may still create private graph-local preparation state before relation
emission; this is distinct from materializing a public
SemanticFlatRelationInput and never stores a planner-native object in the
neutral engine.
View lifetime and streaming
Direct backend streams resolve each appended step immediately and retain only
planner-neutral state -- a native batch encoding, or a ViewPreparation when
the flush needs every graph at once. A caller must keep the planning task,
problem, and View context alive through the append call; a stream must not
retain a lazy range after its source state has been destroyed. Deferred flush
tests should exercise both mutable removal and replacement so a stale View
cannot be observed.
A prepared handle is borrowed by the flush, never consumed, so flushing the same stream twice is well defined and returns the same encoding.
Generic composition capabilities
Mifrost exposes only generic FlatCompositionCapability bits. A downstream
adapter may define its own named assemblies and validate them with
flat_composition_capabilities_satisfied() or
flat_composition_missing_capabilities(). Mifrost does not publish
downstream mode names or parity claims; exact mode parity belongs in the
downstream repository.
Compatibility and performance rules
- Compile a plan once per encoder configuration and cache it.
- Share a
std::shared_ptr<const CompiledFlatPlan>with oneFlatBatchRuntimeper worker; components must keep compiled state immutable (graph-local state belongs in the graph context), and do not append already-materializedBatchEncodingvalues to compose components. - Keep relation ordering, tuple arity, node identity, and graph-field ownership deterministic. Backend adapters should translate to semantic inputs once; they must not rebuild relation carriers or append materialized encodings.
- Set
FlatCompositionConfig::relation_args_node_typewhen the composed plan uses a node table other thanentity; relation argument fields then receive the correct graph offset source.