Compiler architecture

The official Norm compiler is implemented in Java. .norm files are the project's authoring source. After name resolution and type checking, they produce deterministic, content-addressed Core IR. Truffle is the sole execution backend for Core. The implementation strategy decision fixes the technology stack, and the toolchain development rules define engineering dependencies.

dev.w0fv1.norm.abi is a stateless leaf layer shared by the frontend, Core, and backend. stdlib-abi.json is the single declaration source for built-in types, call signatures, capabilities, exception layout, and runtime shapes. The build generates neutral contracts and a fingerprint. BuiltinContracts derives a validation view; BuiltinSemanticView derives a semantic view. Core depends on neither semantic nor builtin. The general pattern-coverage algorithm belongs to pattern; source identity and locations belong to source.

ProjectEnvironment
  → module.norm + hidden entry → Core → Truffle → ModuleDescriptor
  → ProjectSourceSet
  → Lexer / Parser
  → Analyzer / SemanticModel ──→ authoring snapshot
  → Binder
  → CoreBuilder
  → CoreCanonicalizer
  → CompilationOutput / CompilationResultCache
  → CompilationOutput.artifact
  → ApplicationCompiler → CompiledApplication
  → Lowerer
  → Truffle CallTarget

Project and frontend

ProjectEnvironment first evaluates the standard library's module.norm through the bootstrap protocol, then establishes the shared standard-library prelude. ProjectLoader recursively evaluates the exact dependency graph returned by Module module() and establishes an immutable ProjectSourceSet. Module configuration and business programs compile separately; configuration artifacts do not enter the business Core dependency graph. project owns discovery and input capture. The CLI and Polyglot share application compilation and resource preparation. The Language Server manages document analysis through workspace. CompilationScope carries each source document's module name, version, relative path, and module direct-read edges in one place; Analyzer and language services use this same visibility model. See the module system for module rules.

Lexer and Parser build the syntax tree. Language-service type fragments also enter the same syntax implementation through TypeSyntaxParser. Completion context consumes Lexer tokens for literals, interpolation, and unterminated literals directly. Analyzer assembles declaration, type, and body analysis. TypeResolver, DeclarationAnalyzer, and DeclarationPolicyResolver own type resolution, declaration signatures, and compile-time policy, respectively. SemanticModelBuilder is the only write entry to semantic output. Scoped handles in TypeResolutionState and BodyAnalysisState manage state entry and restoration. Speculation rolls back its writes through AnalysisJournal. See AnalysisTransaction and AnalysisTransactionTest for transaction boundaries, and FrontendBoundaryTest for component boundaries. Loop-flow joins remain owned by FlowAnalyzer.

Declaration analysis precedes incremental planning and function-body analysis. DeclarationAnalysis holds frozen declaration facts. DeclarationAnalysisTest verifies boundaries among type resolution, declaration diagnostics, and local analysis.

DeclarationContract forms an incremental contract from resolved symbols and declaration attributes. Implementation changes and declaration-contract changes trigger local analysis and dependent invalidation separately. DeclarationInvalidationTest covers cross-session results, default-implementation changes, and signature diagnostics. CoreReusePlan still decides whether Core content links are reusable.

CoreBuildHistory stores relocatable Core units and exact declaration references. It relinks those units when implementation links change; build reports count conversions, links, and direct reuse separately. CoreRelinkingTest and PersistentCompilationTest cover equal content with different targets, recursive groups, and cross-process execution.

CallResolver unifies candidate inference and selection for ordinary functions, methods, interfaces, constructors, built-ins, and enum constructors. CallArguments maps arguments; TypeArguments fills default type arguments. Diamond construction and owner type parameters participate in candidate solving. Candidate applicability and final parameter validation share type relations; SemanticProbeTest constrains behavior. ResolvedCall keeps the exact target and instantiated signature for Binder, signature help, and navigation. AnalyzerTypeArchitectureTest constrains analysis-stage boundaries.

TypeRelations.DeclarationGraph unifies nominal type projection, assignability, and common-type solving. The frontend supplies declaration relations; SemanticModel supplies frozen relations. Both resolve type applications through TypeApplication, while generic inference shares TypeConstraintSolver. BuiltinCatalog instantiates ABI contracts for built-in protocol queries. SemanticArchitectureTest constrains semantics. MemberRelations derives member relations from overrides and witnesses; navigation, references, and rename use the same relations. AuthoringArchitectureTest constrains editor behavior.

Workspace manages analysis batches per project. AnalysisScheduler coalesces pending edits and cancels obsolete work. All open project documents publish the same snapshot atomically. ProjectSession prepares standard-library source identity translation and the full overlay; WorkspaceTest checks batch behavior. Both analysis commit and diagnostic delivery validate document versions and batch ownership. LSP DocumentService only translates protocol data. See workspace for state and scheduling entry points.

Binder freezes verified semantics into an internal resolved representation. It fixes global call targets, interface requirements and witnesses, field owners and ordinals, argument-to-parameter mappings, source evaluation order, closure targets and captures, runtime generic arguments, and value/identity copy semantics. Later stages consume those resolved targets directly.

CompilationRequest.Kind distinguishes application and library compilation. Both share analysis, binding, and Core construction. Libraries select no application entry; even a library without declarations may form a Core artifact. An application entry is required only at the execution boundary, and compilation kind is part of cache identity. LibraryCompilationTest verifies the contract and persistence.

Application and artifact boundaries

Source methods retain implementation presence through Syntax.FunctionDecl.implementation; an empty body and a declaration without a body have distinct syntax identities. Formatting and language services preserve the signature of a declaration-only method. Until its implementation provider resolves, execution compilation reports a diagnostic rather than reducing it to an empty body. See MethodDeclarationCompilerTest.

Bound implementation presence matches source. A declaration-only method becomes a Core MethodSignature, with no executable body allocated. ManagedMethodLoweringTest checks lowering for class and method generics.

All bodyless methods use CoreDefinition.MethodSignature in Core. The signature includes the nominal receiver, type parameters, parameter types, and return type. Interface, class, and value receivers share signature validation. Interface inheritance and calls still require the signature to belong to the corresponding interface. See CoreDefinition and CoreIdentityVersion for structure and version entry points.

A dispatch target refers to its target declaration; the declaration's type determines whether it has an implementation. Class dispatch may point to a managed signature, while inheritance and overriding still validate receiver and generic ABI. The execution plan includes only executable targets in the local call graph. See CoreManagedDispatchTest.

ApplicationCompiler returns a closeable ApplicationCompilation. On success, CompiledApplication owns temporary directories, the selected and captured Java classpath, annotation-processing outputs, and the execution plan. Each run opens its own runtime resources. Closing the compiler or runner does not invalidate a delivered application; the consumer closes the artifact. Annotation processing creates the Java method index once, and execution, tests, and Native builds reuse that result. See the application entry point.

Application delivery uses ApplicationBuilder. It borrows the caller's ApplicationRunner, closes this compilation's artifact and staging, and returns only a delivery location or compilation diagnostics. ApplicationProgramPlan decides application-execution retention; original applications and retained artifacts use matching execution plans. Building does not recompile the project or repeat annotation processing. See the build tests for lifecycle and archive checks.

ProjectResources defines value equality from module ownership and resource content and derives the classpath resource view. Application-cache reuse follows value semantics for complete inputs; PolyglotProjectTest checks lifecycle. Packaging reads captured resources rather than rescanning the working directory. FileSnapshot validates module-archive and JAR content identity and rechecks after copying, rejecting stale identities for changed files. See application builds for archive and release contracts.

Java binding scans and inheritance relations share JavaTypeProjector. JarApiScanner alone produces a complete schema that includes effective inherited members; binding planning only consumes that schema. BindingPlanner produces an immutable BindingPlan fixing exports, names, signatures, and call tables; BindingSourceRenderer generates source from the plan. See the JAR binding generator.

The sole Norm-to-Java generation chain is JavaStubPlanner → JavaStubPlan → JavaStubRenderer. The plan fixes type projections, signatures, inheritance, annotations, and bridge targets. The renderer accesses neither Core nor the project. JavaAnnotationProcessorPipeline consumes generated source and produces the method index once. JavaStubPlannerTest and JavaAnnotationBindingIntegrationTest constrain byte equivalence, immutable plans, and actual annotation processing.

Identity boundaries

Identity	World	Purpose
DocumentId / SymbolId	authoring	Document revisions, diagnostics, and editor operations
DefinitionId	semantic Core	Immutable definition and its fixed dependencies
PublicAbiId	namespace	Exported names, visibility, and public signatures
CoreCodeId / MetadataId	executable	Core code and runtime companion metadata
DebugInfoId	authoring	Source, locations, and occurrence routing
ExecutableId	backend	Code, runtime metadata, and backend ABI
ArtifactId	bundle	Published combination of code, links, public ABI, debug information, and metadata

DefinitionId derives only from versioned canonical encoding. Parameter names belong to observable call and ParameterContext contracts. Callable definition names, local names, source locations, and whitespace lie outside semantic Core. Local bindings and type parameters use dense indices within a definition.

DeclarationIdentity forms declaration identity from module coordinates, module-relative source path, declaration kind, name, and canonical signature. It excludes the disk root, declaration ordinal, and source offset. Actual document URIs remain in diagnostics and source maps. Member identities derive from owners, while type-parameter identities derive from owning declarations and parameter indices. Overload-resolution families are grouped by visibility scope; public declarations can share a family across documents, while private declarations stay document-local.

Stable BuiltinTypeId values identify built-in types, while CoreDefinitionLink identifies user types. A nominal type key includes module name and version, package, type name, and visibility. Private types additionally include the module-relative source path. Renaming a type or moving a private type within a module changes nominal identity; moving an entire project root does not. Semantic content includes class/value aggregate kind, generic parameters, parent types, field layout, constructor entries, method dispatch, and interface conformances; interface generic parameters, parents, and requirements; and stable enum-variant keys and payload types.

CoreNamespace stores authoring names, signatures, visibility, export state, and exact occurrences. ParameterPolicy shares the parameter-call policy between semantic declarations and Core signatures. Default-argument markers, label policy, and callback parameter names participate in public ABI; see ParameterContractTest. For every canonical definition, CoreAuthoringMap stores source-stable DefinitionOccurrenceId values, declaration roles, CoreDefinitionOrigin, and reference-occurrence routing. CoreArtifact.metadata stores companion metadata targeted to occurrences. Lowerer selects the corresponding origin from the call occurrence, so multiple source definitions sharing one DefinitionId still retain their own roles, names, locations, call stacks, and annotations.

Canonical Core

Through CoreCompilationInput, CoreBuilder receives declarations to convert and already compiled definitions through one interface. Source declarations become strongly typed CoreDefinition values from the resolved representation; reused declarations retain content groups while rebinding current authoring information. Callables, aggregates, enums, interfaces, interface methods, and built-in conformances share one content-definition model. Calls, construction, enum variants, interface witnesses, user types, and field owners first become PendingDefinitionReference values. CoreCanonicalizer walks signatures, generic bounds, interface relations, local and runtime types, fields, and executable expressions to establish the full dependency graph. It canonicalizes strongly connected components: references within a component use member indices, while external references use complete DefinitionId values. A DefinitionGroupId identifies the recursive group as a whole; group identity plus canonical member index identifies each member.

BoundCoreBodyConverter converts Bound to Core by exhaustive matching over a sealed hierarchy. A new node must be converted before compilation can succeed; Core traversal fixtures constrain codec, walker, and rewriter coverage.

Canonical refinement uses positioned outgoing and incoming edges and skips search branches proven to belong to the same automorphism. A search budget remains as a resource boundary for adversarial graphs. CoreBuildReport records component sizes, refinement, search, memoization, and automorphism pruning together.

DefaultArgumentDeclarations indexes default expressions. Binder generates a separate Core callable in the declaration's own type and receiver context. A caller invokes that implementation with explicit arguments; Let ensures the receiver evaluates only once. A resulting reference must prove that it points to long-lived storage; borrowed parameter references and temporary local addresses cannot escape. CoreLetTest verifies this. DefaultArgumentExecutionTest and DefaultArgumentReuseTest cover execution and persistent reuse. DefaultArgumentContractTest covers default-implementation links, public-signature versus execution-link identity, and occurrence relocation after persistent reuse. CoreBindingShape maps declaration contracts in one place. See startup performance for archived-declaration import boundaries.

CoreCodec is the sole encoder of canonical bytes. CoreIdentityVersion defines the identity version. Encoding fixes version, domain separation, node tags, byte order, collection order, and string encoding. Java object serialization, Truffle ASTs, and runtime profiles do not participate in semantic hashes.

Overall verification enters through CoreProgramVerifier. Declaration verification combines CoreCallableVerifier, CoreIntrinsicVerifier, and read-only CoreVerificationTypes; each callable owns its control-flow and reference state. See CoreVerifierBoundaryTest.

Before content enters storage, CoreProgram verifies the entire closure: nominal types and generic bounds, callable receivers and reified ABI, interface inheritance and complete witnesses, local and runtime types, call and construction targets, field and enum references, built-in protocols and operations, and namespace bindings must agree. Runtime type captures are canonically ordered by type-parameter index, so semantically equal descriptors have one canonical encoding.

Standard-library source follows the same Core pipeline and uses module coordinates from module.norm. CoreBuilder produces Core artifacts only. CompilationResultCache manages consumable compilation results and declaration history, using the bounded, content-validated, atomically published FileArtifactCache throughout. Authoring snapshots are not written into the compilation artifact cache.

Incremental boundaries

CompilerSession reuses parsing results by document content and retains the preceding CompilationOutput by stable CompilationUnitId. Definition dependencies are encoded in identity, so changing a leaf definition produces new identities for its dependent closure while unrelated definitions reuse their existing identities and content groups. A module uses the root module.norm URI to identify its compilation unit; an independent file uses its own URI.

Declaration-level analysis detects changes from lexical structure rather than absolute offsets. Token anchors map reused contributions to current source after whitespace edits and declaration reordering. Declaration additions, removals, and signature-family changes invalidate only the corresponding resolution family and semantic dependents. Package, import, or compilation-scope changes still invalidate at the document boundary.

A persistent session stores exact compilation results and CompilationHistory. History references Core artifacts by content key and retains semantic contributions, declaration identities, and local-variable mappings. Core is rebuilt when a result referenced by history has been evicted. Analysis and Core share TokenSpanMapping for source-location updates. CoreReusePlan propagates invalidation along actual declaration references; equal-content but distinct declarations keep their own call relationships. PersistentCompilationTest constrains separate JVMs, default arguments, local annotations, and corruption recovery. CoreBuildReport supplies actual conversion counts. Binder still processes the whole resolved program. See startup performance for published-package declaration import boundaries.

Operations on one compilation unit commit sequentially in history order; different units may analyze and build in parallel. Parse caches and compilation history are read or written only under short state locks. Invalidation and closing use exclusive lifecycle boundaries and do not interleave with in-flight compilation.

A published module stores its own semantic contributions and Core units in CompiledModule. A consumer uses ImportedCompilation to verify declaration contracts, module read relations, and source policies. Import and incremental relinking reuse relocation tables from CoreBuildHistory. A change in dependency implementation does not require reconverting its caller. PublishedCoreImportTest covers directory moves, multiple published modules, default arguments, closures, and cross-session reuse. PortableObjectCodecTest covers compact source and location serialization.

CoreDependencyIndex provides direct dependencies, reverse dependencies, and transitive dependents. CoreCompilationDelta reports definitions added, reused, or detached from the current source set. Type references and executable references use the same dependency-propagation rule. Downstream caches invalidate by these strongly typed identities; a content-store hit only verifies integrity and reads content.

JarBindingClasspath holds application Java dependencies as a content file set, and CompiledApplication manages its release. Changes to application source do not change the dependency file set's identity. DirectoryArtifactCache coordinates production by content key, performing validation and file production outside the directory-metadata lock. Directories in production are also protected by process ownership.

Truffle backend

CompilationResult holds CompilationOutput directly. ExecutionBackend and Lowerer consume only CoreArtifact. Lowerer uses resolved Core to generate function CallTarget values, frame slots, control-flow nodes, fixed-target calls, and dispatch tables keyed by static method or interface-requirement DefinitionId. Class and interface calls share the dynamic-dispatch entry. Iterator-style for works through Iterable<T> and Iterator<T> requirements; built-in collections return an internal NativeIterator<T> runtime value.

ExecutionContext is passed along fixed call edges as a hidden root parameter; executable nodes do not capture per-run state. A standalone TruffleExecutionBackend caches context-free executable programs by ExecutableId in a bounded cache. Whitespace, position, and source-URI changes alter only DebugInfoId; runtime errors map locations through the current artifact's authoring sidecar. The Polyglot path requiring Truffle source instrumentation includes DebugInfoId at its instantiation boundary. The built-in ABI fingerprint is part of the backend ABI key.

Guest runtime errors carry stable error codes and SourceSection at Truffle nodes. Crossing the public boundary converts them into structured NormExecutionException values. The self-contained CLI packages the same Core and Truffle execution chain with the platform runtime.

Verification

Declaration-reference operators preserve navigation and compile-time binding. SemanticModelBuilder records their authoring reference roles and relocates them with semantic contributions. RenamePreviewTest covers rename, source capture, and operator boundaries.

DependencyArchitectureTest is an executable constraint on package boundaries and acyclic dependencies.

ApplicationProgramArchiveTest verifies writing, reading, and executing a native application archive on both classpath and distribution module path.

Identity tests cover source moves, generic alpha-renaming, explicit versus inferred type arguments, module versions, nominal types, declaration reordering, recursive strongly connected components, type-dependency propagation, and authoring-occurrence routing. Storage tests cover admission policy, read-only verification, corruption recovery, concurrent publication and cleanup, and cross-instance reads. Boundary tests cover Core type and operation ABI, namespace shape, and duplicate groups. Backend tests cover Core-only dependencies, DefinitionId enum identity, artifact reuse, separate execution contexts, the Polyglot entry, source locations, and guest call stacks.