> ## Documentation Index
> Fetch the complete documentation index at: https://ricardovelit.com/axon-docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Compilation Pipeline

> How AXON source code transforms from text to executable prompts

AXON is a **compiled language** with a multi-stage transformation pipeline. Unlike interpreted languages that execute source directly, AXON transforms `.axon` files through multiple representations before generating backend-specific prompts for LLMs.

## Pipeline Overview

```mermaid theme={null}
graph LR
    A[.axon source] --> B[Lexer]
    B --> C[Token Stream]
    C --> D[Parser]
    D --> E[AST]
    E --> F[Type Checker]
    F --> G[IR Generator]
    G --> H[AXON IR]
    H --> I[Backend]
    I --> J[Runtime]
    J --> K[Typed Output]
    
    style A fill:#e1f5ff
    style E fill:#fff4e1
    style H fill:#f0e1ff
    style K fill:#e1ffe1
```

<Steps>
  <Step title="Lexer — Source → Tokens">
    Character stream becomes structured tokens
  </Step>

  <Step title="Parser — Tokens → AST">
    Token stream becomes cognitive syntax tree
  </Step>

  <Step title="Type Checker — Semantic Validation">
    AST validated for type correctness
  </Step>

  <Step title="IR Generator — AST → IR">
    Cognitive AST lowered to intermediate representation
  </Step>

  <Step title="Backend — IR → Prompts">
    IR compiled to model-specific prompts
  </Step>

  <Step title="Runtime — Execution + Validation">
    Prompts executed, validated, traced
  </Step>
</Steps>

***

## Stage 1: Lexer (Tokenization)

### Purpose

Convert raw `.axon` source text into a **stream of tokens** — the atomic units of the language.

### Implementation

<Info>
  **Location:** `/axon/compiler/lexer.py:1`\
  **Type:** Hand-written, single-pass character scanner
</Info>

```python lexer.py theme={null}
class Lexer:
    """Tokenizes AXON source code into a stream of Token objects."""
    
    def tokenize(self) -> list[Token]:
        """Scan the entire source and return all tokens."""
        while not self._at_end():
            self._skip_whitespace()
            if self._at_end():
                break
            self._scan_token()
        self._tokens.append(Token(TokenType.EOF, "", self._line, self._column))
        return self._tokens
```

### Token Types

The lexer recognizes **35 keywords** (cognitive primitives) and various symbols:

<CodeGroup>
  ```python Cognitive Keywords theme={null}
  KEYWORDS = {
      "persona": TokenType.PERSONA,
      "context": TokenType.CONTEXT,
      "intent": TokenType.INTENT,
      "flow": TokenType.FLOW,
      "reason": TokenType.REASON,
      "anchor": TokenType.ANCHOR,
      "validate": TokenType.VALIDATE,
      "refine": TokenType.REFINE,
      "memory": TokenType.MEMORY,
      "tool": TokenType.TOOL,
      "probe": TokenType.PROBE,
      "weave": TokenType.WEAVE,
      # ... and more
  }
  ```

  ```python Literals theme={null}
  TokenType.STRING    # "text"
  TokenType.INTEGER   # 42
  TokenType.FLOAT     # 3.14
  TokenType.BOOL      # true/false
  TokenType.DURATION  # 10s, 5m, 2h
  TokenType.IDENTIFIER # variable_name
  ```

  ```python Symbols theme={null}
  TokenType.LBRACE    # {
  TokenType.RBRACE    # }
  TokenType.ARROW     # ->
  TokenType.RANGE     # ..
  TokenType.COLON     # :
  TokenType.COMMA     # ,
  ```
</CodeGroup>

### Features

<CardGroup cols={2}>
  <Card title="Comment Stripping" icon="comment">
    Removes `//` line comments and `/* */` block comments
  </Card>

  <Card title="String Escapes" icon="quote-left">
    Handles `\n`, `\t`, `\"`, `\\` in string literals
  </Card>

  <Card title="Keyword Discrimination" icon="key">
    Distinguishes `flow` (keyword) from `flow_name` (identifier)
  </Card>

  <Card title="Location Tracking" icon="location-dot">
    Tracks line and column for error messages
  </Card>
</CardGroup>

### Example

```axon Input source theme={null}
persona Expert {
  domain: ["AI", "ML"]
  confidence_threshold: 0.85
}
```

```python Output tokens theme={null}
[
  Token(PERSONA, "persona", line=1, col=1),
  Token(IDENTIFIER, "Expert", line=1, col=9),
  Token(LBRACE, "{", line=1, col=16),
  Token(IDENTIFIER, "domain", line=2, col=3),
  Token(COLON, ":", line=2, col=9),
  Token(LBRACKET, "[", line=2, col=11),
  Token(STRING, "AI", line=2, col=12),
  Token(COMMA, ",", line=2, col=16),
  Token(STRING, "ML", line=2, col=18),
  Token(RBRACKET, "]", line=2, col=22),
  # ...
]
```

***

## Stage 2: Parser (AST Construction)

### Purpose

Transform the **flat token stream** into a hierarchical **Abstract Syntax Tree (AST)** representing the program's cognitive structure.

### Implementation

<Info>
  **Location:** `/axon/compiler/parser.py:1`\
  **Algorithm:** Recursive descent parser with one method per grammar rule
</Info>

```python parser.py theme={null}
class Parser:
    """Recursive descent parser for the AXON language."""
    
    def parse(self) -> ProgramNode:
        """Parse the full program → ProgramNode."""
        program = ProgramNode(line=1, column=1)
        while not self._check(TokenType.EOF):
            decl = self._parse_declaration()
            if decl is not None:
                program.declarations.append(decl)
        return program
```

### AST Node Types

Each cognitive primitive has a corresponding AST node:

<Tabs>
  <Tab title="Declarations">
    * `PersonaDefinition`
    * `ContextDefinition`
    * `AnchorConstraint`
    * `FlowDefinition`
    * `ToolDefinition`
    * `MemoryDefinition`
    * `TypeDefinition`
  </Tab>

  <Tab title="Statements">
    * `RunStatement`
    * `IntentNode`
    * `StepNode`
    * `ValidateGate`
    * `RefineBlock`
  </Tab>

  <Tab title="Expressions">
    * `ReasonChain`
    * `ProbeDirective`
    * `WeaveNode`
    * `UseToolNode`
    * `RememberNode`
    * `RecallNode`
  </Tab>
</Tabs>

### Example AST

```axon Input theme={null}
flow Analyze(doc: Document) -> Summary {
  step Extract {
    probe doc for [entities, dates]
    output: EntityMap
  }
}
```

```python Output AST (simplified) theme={null}
FlowDefinition(
  name="Analyze",
  parameters=[
    ParameterNode(name="doc", type_expr="Document")
  ],
  return_type="Summary",
  steps=[
    StepNode(
      name="Extract",
      body=[
        ProbeDirective(
          target="doc",
          extract_fields=["entities", "dates"]
        )
      ],
      output_type="EntityMap"
    )
  ]
)
```

<Tip>
  **Cognitive AST:** Unlike traditional ASTs with mechanical nodes (e.g., `BinaryExpression`), AXON's AST uses **semantic nodes** (`ReasonChain`, `ProbeDirective`) that map directly to cognitive operations.
</Tip>

***

## Stage 3: Type Checker (Semantic Validation)

### Purpose

Validate the **semantic correctness** of the program using AXON's epistemic type system.

### Implementation

<Info>
  **Location:** `/axon/compiler/type_checker.py:1`\
  **Type System:** Epistemic partial order lattice
</Info>

```python type_checker.py theme={null}
class TypeChecker:
    """Semantic type validation for AXON programs."""
    
    def check(self) -> list[AxonTypeError]:
        """Validate the entire program and return type errors."""
        self._build_symbol_table()
        self._check_declarations()
        self._check_references()
        return self._errors
```

### Validation Rules

<Steps>
  <Step title="Symbol Table Construction">
    Build registry of all personas, contexts, flows, anchors, types
  </Step>

  <Step title="Type Compatibility">
    Check `Opinion ≰ FactualClaim` and other subsumption rules
  </Step>

  <Step title="Reference Resolution">
    Ensure `run Analyze(doc)` references a defined `flow Analyze`
  </Step>

  <Step title="Uncertainty Propagation">
    Track `Uncertainty` taint through operations
  </Step>

  <Step title="Range Validation">
    Verify `RiskScore(0.0..1.0)` only accepts values in range
  </Step>
</Steps>

### Type Errors

The type checker returns a list of `AxonTypeError` objects:

```python theme={null}
class AxonTypeError(AxonError):
    def __init__(self, message: str, line: int, column: int):
        self.message = message
        self.line = line
        self.column = column
```

<Warning>
  **Compile-Time Safety:** Type errors prevent compilation. AXON guarantees that well-typed programs satisfy epistemic constraints.
</Warning>

***

## Stage 4: IR Generator (Lowering)

### Purpose

Lower the **cognitive AST** into the **AXON Intermediate Representation (IR)** — a JSON-serializable format ready for backend compilation.

### Implementation

<Info>
  **Location:** `/axon/compiler/ir_generator.py:1`\
  **Pattern:** Visitor pattern with explicit dispatch
</Info>

```python ir_generator.py theme={null}
class IRGenerator:
    """Transforms a type-checked AST into AXON IR."""
    
    def generate(self, program: ast.ProgramNode) -> IRProgram:
        """Generate a complete IR program from validated AST."""
        self._reset()
        
        # Phase 1: Convert declarations
        for decl in program.declarations:
            self._visit(decl)
        
        # Phase 2: Resolve cross-references
        self._resolve_references()
        
        return IRProgram(
            personas=list(self._personas.values()),
            flows=list(self._flows.values()),
            runs=self._runs,
            # ...
        )
```

### IR Node Types

The IR uses simplified, backend-agnostic nodes:

```python ir_nodes.py theme={null}
@dataclass
class IRFlow:
    name: str
    parameters: list[IRParameter]
    return_type: str
    steps: list[IRStep]
    data_edges: list[IRDataEdge]  # Dependency graph

@dataclass
class IRStep:
    name: str
    directives: list[IRNode]  # IRReason, IRProbe, IRWeave, etc.
    output_type: str
    refinement: IRRefine | None
```

### Cross-Reference Resolution

The IR generator links symbolic references:

```axon Source theme={null}
run AnalyzeContract(doc)
  as LegalExpert              // Resolves to IRPersona
  within LegalReview           // Resolves to IRContext
  constrained_by [NoHallucination]  // Resolves to IRAnchor
```

```python Generated IR theme={null}
IRRun(
  flow_name="AnalyzeContract",
  flow_ref=IRFlow(...),         # Direct reference
  persona_ref=IRPersona(...),   # Direct reference
  context_ref=IRContext(...),   # Direct reference
  anchors=[IRAnchor(...)],      # Direct references
)
```

<Check>
  **Why IR?** The IR decouples language design from backend implementation. New backends (e.g., Gemini, Llama) only need to compile IR, not parse AXON source.
</Check>

***

## Stage 5: Backend (Prompt Compilation)

### Purpose

Compile the **backend-agnostic IR** into **model-specific prompts** for LLM providers.

### Supported Backends

<CardGroup cols={2}>
  <Card title="Anthropic" icon="robot">
    Claude 3.x (Opus, Sonnet, Haiku)
  </Card>

  <Card title="OpenAI" icon="brain">
    GPT-4, GPT-4 Turbo, GPT-3.5
  </Card>

  <Card title="Gemini" icon="sparkles">
    Gemini 1.5 Pro, Flash
  </Card>

  <Card title="Ollama" icon="server">
    Local models (Llama, Mistral)
  </Card>
</CardGroup>

### Backend Interface

<Info>
  **Location:** `/axon/backends/base_backend.py:1`
</Info>

```python base_backend.py theme={null}
class BaseBackend(ABC):
    """Abstract interface for all AXON compiler backends."""
    
    @abstractmethod
    def compile_flow(self, ir_flow: IRFlow) -> str:
        """Compile an IR flow into a model-specific prompt."""
        pass
    
    @abstractmethod
    def compile_persona(self, ir_persona: IRPersona) -> str:
        """Compile an IR persona into system prompt."""
        pass
```

### Example: Anthropic Backend

```python anthropic.py theme={null}
class AnthropicBackend(BaseBackend):
    def compile_persona(self, ir_persona: IRPersona) -> str:
        """Generate Claude-compatible system prompt."""
        prompt = f"You are {ir_persona.name}.\n"
        prompt += f"Your expertise: {', '.join(ir_persona.domain)}\n"
        prompt += f"Tone: {ir_persona.tone}\n"
        if ir_persona.confidence_threshold:
            prompt += f"Confidence threshold: {ir_persona.confidence_threshold}\n"
        return prompt
```

<Tip>
  **Backend Customization:** Each backend optimizes prompts for its model's strengths. Anthropic uses XML tags, OpenAI prefers JSON, Gemini uses structured examples.
</Tip>

***

## Stage 6: Runtime (Execution + Validation)

### Purpose

Execute the compiled prompts, validate outputs, handle failures, and trace execution.

### Runtime Components

<Tabs>
  <Tab title="Executor">
    **Location:** `/axon/runtime/executor.py:1`

    Orchestrates flow execution:

    * Resolves step dependencies (DAG)
    * Invokes model with compiled prompts
    * Passes outputs between steps
  </Tab>

  <Tab title="SemanticValidator">
    **Location:** `/axon/runtime/semantic_validator.py:1`

    Validates outputs at runtime:

    * Classifies epistemic type of LLM output
    * Checks `actual_type ≤ declared_type`
    * Raises `ValidationError` on mismatch
  </Tab>

  <Tab title="RetryEngine">
    **Location:** `/axon/runtime/retry_engine.py:1`

    Handles failures adaptively:

    * Retries with backoff (none, linear, exponential)
    * Injects failure context into next attempt
    * Raises `RefineExhaustedError` after max attempts
  </Tab>

  <Tab title="Tracer">
    **Location:** `/axon/runtime/tracer.py:1`

    Records execution:

    * 14 event types (step start/end, validation, retry, etc.)
    * JSON trace output for debugging
    * Performance metrics
  </Tab>
</Tabs>

### Execution Flow

```mermaid theme={null}
sequenceDiagram
    participant E as Executor
    participant B as Backend
    participant V as Validator
    participant R as RetryEngine
    
    E->>B: Compile step prompt
    B-->>E: Compiled prompt
    E->>B: Execute (LLM API call)
    B-->>E: Raw output
    E->>V: Validate output type
    alt Valid
        V-->>E: Success
        E->>E: Continue to next step
    else Invalid
        V-->>R: ValidationError
        R->>E: Retry with failure context
    end
```

***

## CLI Usage

### Check (Lex + Parse + Type Check)

```bash theme={null}
axon check program.axon
```

Outputs type errors without executing.

### Compile (Generate IR)

```bash theme={null}
axon compile program.axon -o program.ir.json
```

Produces JSON IR for inspection or caching.

### Run (End-to-End)

```bash theme={null}
axon run program.axon -b anthropic --trace
```

Executes with chosen backend and saves trace.

***

## Pipeline Comparison

| Stage          | Traditional Compiler   | AXON                             |
| -------------- | ---------------------- | -------------------------------- |
| **Input**      | Source code            | `.axon` source                   |
| **Lexer**      | Tokens                 | Tokens (with cognitive keywords) |
| **Parser**     | AST (mechanical)       | AST (cognitive nodes)            |
| **Type Check** | Memory safety          | Epistemic correctness            |
| **IR**         | LLVM IR, bytecode      | AXON IR (JSON)                   |
| **Backend**    | Assembly, machine code | Model-specific prompts           |
| **Runtime**    | CPU execution          | LLM invocation + validation      |
| **Output**     | Binary executable      | Validated semantic result        |

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Cognitive Primitives" icon="brain" href="/axon-docs/axon-docs/concepts/cognitive-primitives">
    Learn what gets compiled
  </Card>

  <Card title="Type System" icon="sitemap" href="/axon-docs/axon-docs/concepts/type-system">
    Understand type checking
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/axon-docs/axon-docs/concepts/error-handling">
    See runtime behavior
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/axon-docs/axon-docs/cli/overview">
    Use the compiler tools
  </Card>
</CardGroup>