Compiler Memory Architecture

This document is the production memory contract for the 0x0 compiler. The goal

is simple: compiling the compiler must be bounded, repeatable, and measured.

The bootstrap seed is retained for origin proof only; it is not a memory model

for normal compiler production.

Ownership Model

Compiler work is divided into phases. Each phase owns its transient data and may

retain only the compact phase output documented here:

• source loading owns raw file text and import path resolution state;
• lexing owns token tables and compact spans;
• parsing owns AST nodes and parser diagnostics;
• validation owns symbol, function, local, type, effect, and capability tables;
• lowering owns backend-specific function bodies and relocation inputs;
• layout owns addresses, offsets, string pools, and section sizes;
• emission owns byte buffers or chunked writers;
• packaging owns release manifests, hashes, and measured artifacts.

When a phase completes, the next phase must not keep the previous phase's raw

scratch data alive unless the retained field is named in the receiving phase's

input. The important rule is that source text, token lists, full ASTs, lowered

code, final code, and serialized output are not allowed to accumulate as one

large live graph.

Current Production Path

The normal compiler succession path is:

build/release/v0.1.0/bin/zero-elf-compiler compiler/main.0x0 build/memory/zero-next
build/memory/zero-next compiler/main.0x0 build/memory/stage2
build/memory/stage2 compiler/main.0x0 build/memory/stage3
cmp build/memory/stage2 build/memory/stage3

make memory-check measures peak RSS for that chain and fails if any process

exceeds its budget. The gate also runs a symbol and string heavy OISA compile

through the released OISA compiler so front-end retention regressions are caught

without entering the seed path.

The direct ELF runtime now allocates list/cons values from an mmap-backed arena

instead of issuing one mapping per node. The arena checks mmap results before

publishing the new cursor/end pair; a failed mapping exits the generated process

with status 1 rather than returning a corrupt runtime value. The normal

production source excludes the old C/object compatibility backend, so the

released compiler no longer emits optional compatibility code while compiling

the next compiler. The C compatibility runtime in compiler/compat-main.0x0

interns short repeated strings. Large ELF outputs are written by the host

wrapper from the generated hex stream, while the roadmap requires native byte

buffers before object/archive/linker production becomes the normal path.

The released zero-link and zero-ar compatibility tools also follow the

chunked writer rule for their current production slices. zero-link computes

the executable size up front, then writes the ELF header, startup stub, each

relocated object text block, and ABI note directly to the output stream.

zero-ar computes deterministic member offsets from sizes, then streams the

archive header, symbol index, and member payloads from the input object files

without retaining all object bytes or constructing the whole archive as one byte

string. During archive index construction it scans only ELF headers, section

headers, string tables, and symbol tables. zero-link reads archive symbol

indexes first and loads only archive members selected to satisfy unresolved

symbols.

compile-file now loads the root file through load-file-forms-with-root.

That path parses the root source once, keeps those root forms only for the

module header, and passes the combined imported-plus-root forms to validation

and emission. This prevents the public OISA compiler entry from retaining a

second root source parse beside the import-expanded program.

Token and AST records store compact integer kind IDs instead of repeating kind

text in every node. Source line and column are packed into one integer span

field. The tok-kind, node-kind, tok-line, tok-col, node-line, and

node-col accessors expand those compact fields for existing compiler phases,

while tok-kind? and node-kind? compare IDs directly.

The compiler exposes memory-summary-file, a deterministic structural summary

for a source file after import loading. Version 2 reports root forms,

import-expanded forms, imports, functions, AST nodes, total symbol nodes, and

unique symbol payloads. This is not a replacement for RSS measurement; it is

the compiler-owned internal summary used to catch accidental growth in retained

phase outputs before invoking the heavier release chain.

Budgets

The enforced release budgets live in perf/memory-budgets.txt:

release.zero_next.max_rss_kib=262144
release.stage2.max_rss_kib=262144
release.stage3.max_rss_kib=262144
release.stress_oisa.max_rss_kib=65536
optimizer.source_smoke.max_rss_kib=6291456

Baselines live under perf/memory-baselines/. A budget increase is allowed only

with a checked baseline update and a concrete explanation of the feature or

input size that made the increase necessary.

The optimizer source-smoke budget is intentionally much higher than the release

compiler budgets because it exercises the source runner importing

compiler/main.0x0 and emitting multiple optimizer dump reports in one

process-group measurement. The release compiler chain remains governed by the

release.* budgets above.

Measurement Semantics

tools/memory-check.sh samples /proc/<pid>/status and records VmHWM, so the

gate measures peak resident set size rather than final RSS. When setsid is

available, each measured compiler command runs in its own process group and the

gate samples every live member of that group; if the group exceeds its budget,

the group is terminated before it can keep growing. The report is written to

build/release/v<VERSION>/memory.txt and is included in release metadata and

SHA256SUMS.

make memory-architecture-check is the narrow source-level regression gate for

Milestone 1 subwork. It does not compile; it checks that the normal public

compiler entry parses the root file once, token and node constructors store

compact kind IDs and packed spans, measure-only layout remains in the compiler

builder, and the normal compiler-builder path still returns chunked hex values

to the file writer. It also checks that the released linker and archiver write

large outputs through stream writes instead of whole-output byte concatenation,

and that archive members stay lazy until selected for writing or linking. The

gate also checks that archive symbol discovery does not read full object files.

It also source-checks the version-2 memory summary shape and the symbol-heavy

and diagnostic-heavy summary gate hooks.

The deterministic report is written to

build/memory-architecture/memory-architecture.report and its hashes are

written to build/memory-architecture/SHA256SUMS, including the audited

tools/zero-link.py, tools/zero-ar.py, and memory-summary fixture sources.

make normal-path-audit-check verifies the normal-source diet for Milestone

1.2. It dry-runs make all, requires all: trusted-chain, verifies that

COMPILER_SOURCE is compiler/main.0x0, verifies that

compiler/compat-main.0x0 remains the separate compatibility source, and

rejects non-comment references to seed, zero-native, generated C, GAS object,

or compatibility compiler hooks in the normal compiler source.

make memory-summary-check runs the compiler-owned summary on an import-using

program and on a symbol-heavy duplicate-identifier fixture. It verifies that the

reported fields are present and internally consistent, and that duplicate

symbol pressure is visible as symbols > unique_symbols. The same gate

generates a diagnostic-heavy malformed source and verifies the exact parser

line/column for the unterminated form, proving compact packed spans still feed

precise user diagnostics.

tools/rss-limit.sh is the reusable guard for explicit heavy gates that are not

safe to run unbounded during incremental work. When setsid is available, it

samples VmHWM for every live process in the child process group, writes the

same peak-RSS/budget/elapsed/status/scope fields used by the memory reports,

and kills the child process group if it exceeds the configured cap. This keeps

shell wrappers from hiding memory-heavy compiler children. Optimizer source

smokes use optimizer.source_smoke.max_rss_kib from

perf/memory-budgets.txt. make rss-limit-check verifies both the pass and

over-budget paths, including an over-budget spawned child process, and runs a

fake compiler chain through tools/memory-check.sh to prove the release memory

report records process-group scope and over-budget kill fields without invoking

the real compiler chain.

The report includes:

• compiler and OISA compiler artifact paths;
• budget file path;
• peak RSS, budget, elapsed time, status, RSS scope, over-budget kill status,

and command for each measured step;

• hashes for every generated measured output.

Elapsed time is diagnostic only. Peak RSS and output hashes are the release

contract.

Implementation Rules

• Use phase-local allocation for parse/import, validation, lowering, layout,

emission, and packaging.

• Store identifiers, token kinds, module names, field names, and short literals

through interning where the language/runtime slice supports it.

• Store spans as compact fields on token and node records; diagnostics may

expand them to text only at reporting time.

• Prefer dense tables and vectors for production-size data. Linked lists are

acceptable only for small front-end structures with measured bounds.

• Use a measure-only probe layout before emission so probe code is not retained

beside final code.

• Use chunked or binary writers for every large output path. Hex text is an

audit/debug bridge, not the desired final production representation.

• Linker and archive packaging paths must compute deterministic sizes and

offsets before writing, then stream payload chunks to disk.

• Archive readers must parse indexes separately from member payloads and load

only members selected by symbol resolution.

• Do not use the seed path to hide a regression in the released compiler path.

Any change that knowingly retains another full compiler-sized representation

must either remove an older retained representation in the same change or lower

the memory budget with evidence.