Rust Project Specifications — Detailed Reference

Companion to: The Rust Learning Program (1-Year Guide)

Ashwin Hebbar | March 2026

This document contains the full specifications, expectations, concepts covered, and success criteria for every project in the Rust Learning Program. Use this as your reference while building.

Table of Contents

1. Phase 1 — csvq: Command-Line CSV Query Tool
2. Phase 2A — schemaguard: Streaming Data Contract Validator
3. Phase 2B — logr: Terminal-Native Activity Tracker
4. Phase 3A — HTTP/1.1 Server From Scratch — Option A
5. Phase 3A (Alternative) — naivechain: Blockchain From the Opcode Up — Option B
6. Phase 3B — ticknorm: Real-Time Market Data Feed Normalizer ⚠️ COI risk — see notice
7. Phase 3B (Replacement) — airpost: Real-Time Air Quality Aggregator
8. Phase 4A — riskbook: Real-Time Portfolio Risk Engine
9. Phase 4B — xlsxfmt: High-Fidelity Bidirectional Excel Library ⚠️ replaced — see notice
10. Phase 4B (Replacement) — dataprof: Fast Data Profiler
11. Phase 5 — docforge: Document Conversion Library (Capstone) — Option A — retired, see notice
12. Phase 5 — formulaengine: Spreadsheet Formula Computation Engine (Capstone) — Option B
13. Phase 5 — ruleforge: Embeddable Business Rules Engine (Capstone) — Option C
14. Phase 5 — inferd: ML Model Inference Server (Capstone) — Option D — ML/DL, recommended

ML/DL track (added June 2026): Applied ML extensions now run through the program: schemaguard drift (feature drift detection, Week 14), riskbook volatility forecasting + Monte Carlo VaR (Weeks 34–35), an optional forecasting endpoint in airpost (stretch), and a new capstone Option D — inferd, an ML model inference server with an embedding-serving endpoint. All from-scratch math stays from-scratch; ort/candle appear only where FFI and serving are the lesson.

Career-fit revision (June 10, 2026): xlsxfmt is replaced by dataprof (a fast data profiler — same Phase 4 learning goals, directly usable in day-to-day data science work), and docforge is retired as a capstone option (it depended on xlsxfmt and pulled the program toward document-infrastructure engineering). formulaengine remains an active capstone option and now integrates with XLSX via calamine + rust_xlsxwriter instead. Timelines re-baselined — see the program guide’s calendar.

Phase 1 — csvq

Command-Line CSV Query Tool

Timeline: Weeks 1–6 (primarily weekend work in weeks 4–6)

The Problem It Solves

Data engineers and analysts regularly need to do quick, ad-hoc analysis on CSV files: filter rows, select columns, compute aggregates. The current options are all unsatisfying:

• Python/Pandas: Requires a full Python environment, slow startup (2–5 seconds even for small files), heavy memory usage
• awk/sed: Fragile, no header awareness, painful syntax for anything beyond trivial operations
• DuckDB CLI: Excellent but is a full database engine — overkill for “show me the top 10 rows where age > 25”
• xsv: The original Rust CSV tool by BurntSushi, but it’s been unmaintained since 2018
• qsv: A maintained fork of xsv, but complex and not beginner-friendly in API

csvq fills the space of “fast, single-binary, zero-dependency CLI for CSV analysis” — think ripgrep for tabular data.

Functional Requirements

Core CLI Interface:

csvq <file> [options]

Options:
  --select <columns>        Comma-separated column names to display
  --filter <expression>     Filter rows (e.g., "age > 25", "name == 'Alice'")
  --sort <column> [asc|desc]  Sort by column
  --limit <n>               Show first n results
  --group-by <column>       Group rows by column
  --agg <expressions>       Aggregations: count(*), sum(col), avg(col), min(col), max(col)
  --stats                   Show per-column statistics (min, max, mean, std, null count)
  --head <n>                Show first n rows (default 10)
  --tail <n>                Show last n rows
  --count                   Count total rows
  --output <format>         Output format: table (default), csv, json

Example Usage:

# Basic filtering and selection
csvq employees.csv --filter "department == 'Engineering'" --select "name,salary" --sort "salary desc" --limit 10

# Aggregation
csvq sales.csv --group-by "region" --agg "sum(revenue),avg(deal_size),count(*)"

# Quick statistics
csvq dataset.csv --stats

# Pipe-friendly
cat large_file.csv | csvq --filter "status == 'active'" --count

Technical Requirements

Must implement yourself (before reaching for crates):

• CSV parsing: handle quoted fields, escaped commas, newlines within quotes, different delimiters
• Type inference: detect integer, float, string, date columns from data
• Expression parsing: build a simple parser for filter expressions (comparison operators, AND/OR)
• Sorting: implement at least one sorting algorithm yourself before using .sort()

Allowed crates:

• clap — CLI argument parsing
• csv — you may use this AFTER you’ve written your own parser and understand the edge cases
• serde — for JSON output format

Not allowed:

• polars, datafusion, or any dataframe library (defeats the purpose)

Expected Concepts Covered

Definition of Done

• Can parse a 1M-row CSV file and filter it in under 2 seconds
• Handles malformed CSV gracefully (quoted fields, embedded commas, missing values)
• --stats produces correct min/max/mean/std for numeric columns
• --group-by with --agg produces correct grouped aggregations
• Error messages are clear and helpful (not just “Error: something went wrong”)
• Has at least 10 unit tests covering edge cases
• README.md with usage examples
• Published as a binary you can install with cargo install

Stretch Goals

• Support reading from stdin (piped input)
• Support TSV and custom delimiters (--delimiter '\t')
• Column type override (--types "name:string,age:int")
• Basic --join support (join two CSV files on a key column)

Phase 2A — schemaguard

Streaming Data Contract Validator

Timeline: Weeks 11–13 (primary project work on weekends)

The Problem It Solves

Data pipelines break silently. An upstream schema changes — a field is renamed, a type changes from integer to string, a required field becomes nullable — and the downstream pipeline produces garbage without crashing. This is the #1 source of data quality issues in production systems.

Current solutions and their gaps:

• Great Expectations (Python): Powerful but extremely slow at scale. Validating 1M rows takes minutes. Configuration is verbose.
• Pandera (Python): Lighter than Great Expectations but still Python-speed. Tied to Pandas DataFrames.
• Kafka Schema Registry: Enterprise-grade but requires the full Confluent platform. Overkill for simple validation.
• JSON Schema validators: Exist in every language but validate individual documents, not streaming data at high throughput.
• dbt tests: SQL-based, post-load only. Cannot validate data in-flight.

The gap: There is no fast, standalone, language-agnostic CLI tool that validates streaming data (NDJSON, CSV) against a schema contract at high throughput and produces actionable error reports. schemaguard fills this gap.

Functional Requirements

Core CLI Interface:

schemaguard validate --schema contract.json --input data.ndjson [--report errors.csv]
schemaguard validate --schema contract.json --input data.csv --format csv
schemaguard diff schema_v1.json schema_v2.json
schemaguard generate --input sample.ndjson --output inferred_schema.json

Schema Contract Format (JSON):

{
  "name": "user_events",
  "version": "1.2.0",
  "fields": [
    {
      "name": "user_id",
      "type": "string",
      "required": true,
      "pattern": "^USR-[0-9]{6}$"
    },
    {
      "name": "event_type",
      "type": "string",
      "required": true,
      "allowed_values": ["click", "view", "purchase", "signup"]
    },
    {
      "name": "amount",
      "type": "float",
      "required": false,
      "min": 0.0,
      "max": 100000.0
    },
    {
      "name": "timestamp",
      "type": "datetime",
      "required": true,
      "format": "ISO8601"
    }
  ],
  "constraints": [
    {
      "type": "conditional_required",
      "condition": "event_type == 'purchase'",
      "required_fields": ["amount"]
    }
  ]
}

Validation Behavior:

• Read input line-by-line (streaming — never load entire file into memory)
• For each record: check all field types, required fields, value ranges, regex patterns, conditional constraints
• Emit violations to stderr or a structured report file
• Exit code 0 = all valid, exit code 1 = violations found (CI/CD friendly)
• Print summary at end: Validated 1,000,000 records in 2.3s. 47 violations found.

Diff Mode:

• Compare two schema versions
• Report: added fields, removed fields, type changes, constraint changes
• Classify each as “breaking” or “non-breaking”
• Example output:

Breaking changes:
  - Field 'user_id' type changed: string → integer
  - Field 'email' removed (was required)

Non-breaking changes:
  - Field 'preferences' added (optional)
  - Field 'amount' max changed: 100000.0 → 500000.0

Schema Inference:

• Given a sample data file, infer a schema contract
• Detect types, required/optional status, value ranges, common patterns
• Output a draft schema that the user can refine

Technical Requirements

Performance target: Validate at least 100,000 NDJSON records per second on a single core.

Required crates:

• serde + serde_json — JSON parsing and schema deserialization
• csv — CSV format support
• clap — CLI
• regex — pattern validation
• chrono — datetime validation
• thiserror — structured error types

Architecture:

main.rs          → CLI entry point (clap)
schema/
  mod.rs         → Schema types (structs, enums for field types)
  parser.rs      → Schema file parser
  diff.rs        → Schema diff engine
validator/
  mod.rs         → Core validation engine (trait-based)
  json.rs        → NDJSON validator
  csv.rs         → CSV validator
report/
  mod.rs         → Error report generation
  formats.rs     → CSV, JSON, plain text report formatters
infer/
  mod.rs         → Schema inference from sample data

Expected Concepts Covered

Definition of Done

• Validates NDJSON at 100k+ records/second
• Validates CSV files with header detection
• Schema diff correctly identifies breaking vs non-breaking changes
• Schema inference generates a usable draft from sample data
• Exit codes are CI/CD compatible (0 = pass, 1 = fail)
• Error report includes line numbers and specific violation details
• 20+ unit tests, 5+ integration tests with test fixture files
• README.md with clear examples for each command
• Clean public library API (not just a CLI binary)

Real-World Value

This tool would be immediately useful in:

• CI/CD pipelines (validate data contracts before deployment)
• Kafka consumer validation layers
• Data warehouse ingestion checks
• ML feature store quality gates
• Any team that has ever been bitten by a silent schema change

ML Extension — schemaguard drift: Feature Drift Detection

Timeline: Week 14 (one weekend, after the core validator works)

Schema validation catches the loud failure (a field changed type). Drift detection catches the quiet one: the data is still schema-valid, but its distribution has shifted, and the ML model consuming it is silently degrading. This is the #1 unmonitored failure mode in production ML. The existing tools (Evidently, NannyML, whylogs, Alibi-Detect) are all Python — there is no fast CLI that answers “does production data still look like training data?” in one command.

CLI:

schemaguard drift --baseline train.ndjson --current prod.ndjson --schema contract.json [--report drift.json]
# Field 'amount':       PSI 0.31  KS 0.18 (p < 0.001)   → MAJOR DRIFT
# Field 'event_type':   chi² 4.2  (p = 0.24)            → stable
# Drift check: 1 major, 0 moderate, 11 stable. Exit code 2.

Statistics to implement yourself — no stats crates:

1. Population Stability Index (PSI) per field: bin the baseline into 10 quantile bins, compute PSI = Σ (pᵢ - qᵢ)·ln(pᵢ/qᵢ) over current-vs-baseline bin proportions (smooth zero bins with ε). Thresholds: < 0.1 stable, 0.1–0.25 moderate, > 0.25 major. PSI is the standard metric in credit-risk model monitoring — directly relevant domain vocabulary.
2. Two-sample Kolmogorov–Smirnov test for numeric fields: D = max|F₁(x) - F₂(x)| via a sorted-merge walk of both samples, p-value from the asymptotic Kolmogorov distribution approximation.
3. Chi-squared test for categorical fields against baseline category frequencies.

Behavior: streams both files (reuse the validator’s reader), two passes max, CI-friendly exit codes (0 stable / 1 moderate / 2 major), JSON report with per-field verdicts.

Extra concepts covered: sorting and merged iteration over two streams, quantile computation, floating-point hygiene (log of ratios, ε-smoothing), reusing your own library API — the validator and drift checker share the ingestion layer.

Definition of Done (extension):

• PSI matches a Python reference implementation within 1e-6 on fixture data
• KS statistic and p-value verified against scipy.stats.ks_2samp on 5+ fixtures
• Chi-squared verified against scipy.stats.chisquare
• 1M-row baseline/current comparison completes in seconds, streaming
• Exit codes usable as a CI quality gate for ML feature pipelines

This turns schemaguard from a data-engineering tool into an ML-platform tool: a feature-store quality gate that checks not just “is the data valid” but “does it still look like what the model was trained on.”

Phase 2B — logr

Terminal-Native Personal Activity Tracker

Timeline: Weeks 15–16 (two focused weekends + evenings)

The Problem It Solves

Your own Quantified Self Flask app requires a running server, a browser, and a database daemon. For a daily habit tracker, that is massive overhead. logr is the terminal-native, zero-dependency replacement you will actually use every day.

Functional Requirements

# Log an activity
logr log "study rust" --duration 90 --tags "phase2,lifetimes" --notes "finally understood lifetimes"

# View today's log
logr today

# View weekly summary
logr stats --week

# View streak for an activity
logr streak "study rust"

# List all tracked activities
logr activities

# Export data
logr export --format csv --range "2026-03-01..2026-03-31" > march.csv

# Edit last entry
logr edit --last

# Delete an entry by ID
logr delete <id>

Data Storage: Local JSON files in ~/.local/share/logr/ (XDG compliant). One file per month: 2026-03.json. No database, no server.

Entry format:

{
  "id": "a7f3b2c1",
  "activity": "study rust",
  "duration_minutes": 90,
  "tags": ["phase2", "lifetimes"],
  "notes": "finally understood lifetimes",
  "timestamp": "2026-03-15T21:30:00+05:30"
}

Expected Concepts Covered

Definition of Done

• All CRUD operations work (log, view, edit, delete)
• Stats show correct totals, averages, and streaks
• Data persists between sessions in XDG-compliant directory
• CSV export works with configurable date range
• Handles empty state gracefully (first run, no data yet)
• 10+ unit tests
• You are actually using it daily to track your Rust study time

Phase 3A — HTTP/1.1 Server From Scratch

TWO OPTIONS FOR PHASE 3A: Both are “from scratch, no magic” projects — the point is to see underneath a technology you use every day. Option A (HTTP Server) goes deep on networking and concurrency. Option B (naivechain) goes deep on cryptographic data structures and a stack-based VM. Pick whichever feels more interesting at the time. Both teach roughly the same Rust fundamentals and take the same 3-weekend slot.

Option A: No Frameworks, No Libraries — Raw TCP

Timeline: Weeks 21–23 (spread across 3 weekends — one per week)

Why This Project Is Non-Negotiable

When Flask, FastAPI, or Axum handle a request, there are ~15 layers of abstraction between your handler function and the raw TCP bytes. This project opens them all. After building this, every web framework will feel transparent rather than magical.

Functional Requirements

Build a minimal HTTP/1.1 server using ONLY std::net::TcpListener. It must:

1. Accept TCP connections on a configurable port
2. Parse raw HTTP/1.1 request bytes into a structured type:

   struct Request {
       method: Method,      // GET, POST, etc.
       path: String,        // "/api/logs"
       headers: HashMap<String, String>,
       body: Option<String>,
   }
   

3. Route requests to handler functions based on method + path
4. Return proper HTTP responses with status line, headers, and body:

   HTTP/1.1 200 OK\r\n
   Content-Type: application/json\r\n
   Content-Length: 42\r\n
   \r\n
   {"message": "hello"}
   

5. Handle concurrent connections using std::thread::spawn
6. Serve your logr data as a JSON API:
   • GET /logs — list all entries
   • GET /logs?activity=rust — filter by activity
   • GET /stats/week — weekly summary
   • POST /logs — add a new entry (parse JSON body)

Technical Constraints

Allowed:

• std::net (TcpListener, TcpStream)
• std::thread
• std::sync (Arc, Mutex)
• std::io (Read, Write, BufReader)
• serde_json (for JSON serialization only — you parse HTTP yourself)

NOT allowed:

• Any HTTP framework (axum, actix, warp, hyper, rocket)
• Any HTTP parsing library
• tokio or any async runtime

Expected Concepts Covered

Definition of Done

• Server accepts concurrent HTTP connections on a port
• Correctly parses GET and POST requests with headers and body
• Returns properly formatted HTTP/1.1 responses
• Serves logr data through at least 4 endpoints
• Handles malformed requests with 400 Bad Request
• Returns 404 for unknown paths
• Does not crash on connection drop or malformed input
• Can be tested with curl and a web browser

Phase 3A (Alternative) — naivechain: Blockchain From the Opcode Up

Option B: No Libraries, No Magic — Raw Cryptography and a Stack-Based VM

Timeline: Weeks 21–23 (spread across 3 weekends — one per week)

Why This Project Exists Here

Phase 3A’s purpose in the program is simple: pick something you use constantly without thinking about, strip away every abstraction, and build the core of it yourself. The HTTP server does that for web networking. naivechain does it for blockchains.

By the time most developers talk about blockchains, they’re talking about wallets, DeFi, or smart contract deployments — all layers that sit on top of a surprisingly small core. That core is: a linked list of hashed blocks, a proof-of-work puzzle, a set of unspent transaction outputs, and a tiny stack-based scripting VM that enforces spending conditions. This project builds all four from scratch, in that order. When you’re done, you’ll be able to read Bitcoin’s whitepaper and a real Bitcoin Core source file and recognise exactly what is happening.

This is not a production blockchain and is not meant to be. It is a working, inspectable, testable implementation of every concept that matters — built small enough that you can hold the whole thing in your head.

The Scope: What “From Scratch” Means Here

• No cryptography crates for the core hash function — you implement SHA-256 from the FIPS 180-4 spec (the compression function is 64 rounds of bitwise ops; it is the best possible exercise in working with u32 arithmetic and bit manipulation in Rust)
• No blockchain framework — no substrate, no near-sdk, nothing
• No P2P networking — this is a single-node chain. Consensus between peers is explicitly out of scope.
• The sha2 crate is permitted only after you have implemented your own SHA-256 and verified it produces identical hashes — use it to cross-check, not as a shortcut
• serde + serde_json are permitted for serialization in the CLI/inspection tools
• hex is permitted for encoding/decoding byte arrays to hex strings in output

What You Are Building

A minimal but complete blockchain with four layers, built in order:

Layer 1 — The Hash Engine

Implement SHA-256 from the FIPS 180-4 specification. No crates. This is approximately 80 lines of Rust — sixteen u32 message schedule words, sixty-four rounds of the compression function, eight hash state registers. You will understand exactly why &[u8] → [u8; 32] is the right type signature for a hash function, and why the output cannot be reversed.

// Your implementation — no sha2 crate
fn sha256(input: &[u8]) -> [u8; 32]

// Double-SHA256 (used for block hashing, like Bitcoin)
fn hash256(input: &[u8]) -> [u8; 32] {
    sha256(&sha256(input))
}

Layer 2 — The Block and Chain

A block is a header (previous block hash + merkle root of transactions + timestamp + difficulty target + nonce) plus a list of transactions. Chaining is enforced by including the previous block’s hash in the current block’s header — if anyone tampers with block 5, block 6’s prev_hash no longer matches, and the chain is invalid from that point forward.

struct BlockHeader {
    prev_hash: [u8; 32],
    merkle_root: [u8; 32],
    timestamp: u64,
    difficulty_bits: u32,
    nonce: u64,
}

struct Block {
    header: BlockHeader,
    transactions: Vec<Transaction>,
}

Implement the Merkle tree yourself: pair up transaction hashes, hash each pair together, repeat until one root hash remains. An odd number of transactions: duplicate the last one (Bitcoin does this). The Merkle root is what makes it possible to prove a transaction is in a block without downloading the whole block.

Proof of Work: Mining a block means finding a nonce such that hash256(serialized_header) starts with enough zero bytes to satisfy the current difficulty_bits. Implement adjustable difficulty (a target threshold, not just “N leading zeros”). Keep difficulty low (say, 16 bits = 2 leading zero bytes) so mining during testing is near-instant.

Layer 3 — The UTXO Model

Bitcoin (and this chain) does not have accounts with balances. It has unspent transaction outputs (UTXOs). A transaction consumes some UTXOs as inputs and creates new UTXOs as outputs. Your balance is the sum of all UTXOs that your key can unlock.

struct TxOutput {
    value: u64,        // in "natooshis", the smallest unit
    lock_script: Script,  // spending condition (a program)
}

struct TxInput {
    prev_txid: [u8; 32],  // which transaction created the UTXO
    prev_output_index: u32,   // which output of that transaction
    unlock_script: Script,    // proof that you can spend it
}

struct Transaction {
    inputs: Vec<TxInput>,
    outputs: Vec<TxOutput>,
}

Maintain a UTXOSet — a HashMap<(TxId, u32), TxOutput> — that the chain validator updates as each block is processed. A transaction is only valid if every input references a real, unspent output in the UTXO set, and the total input value ≥ total output value.

Layer 4 — The Script VM

This is the heart of the project. Each TxOutput has a lock_script — a small program that defines the spending condition. Each TxInput has an unlock_script — a program that must satisfy the lock. To validate a transaction input, you concatenate the unlock script and the lock script, then execute them on a stack. If the stack ends with a single truthy value on top, the input is valid.

Implement a stack-based VM with the following opcodes:

#[derive(Debug, Clone, PartialEq)]
enum Opcode {
    // Stack operations
    OpPushBytes(Vec<u8>),   // push raw bytes onto the stack
    OpDup,                  // duplicate the top stack item
    OpDrop,                 // discard the top stack item
    OpSwap,                 // swap top two items

    // Arithmetic (items treated as little-endian i64)
    OpAdd,
    OpSub,
    OpNegate,

    // Comparison
    OpEqual,                // pop two items, push 1 if equal, else 0
    OpEqualVerify,          // like OpEqual, but fail the script if not equal
    OpNumEqual,
    OpGreaterThan,
    OpLessThan,

    // Crypto
    OpHash256,              // pop top item, push hash256 of it (your SHA-256!)
    OpCheckSig,             // simplified: pop pubkey and sig, verify (see below)

    // Control
    OpIf,
    OpElse,
    OpEndIf,
    OpReturn,               // immediately fail the script
    OpVerify,               // pop top; if not truthy, fail the script
}

The OpCheckSig opcode in real Bitcoin involves ECDSA elliptic curve signature verification — which is genuinely complex cryptography. For naivechain, simplify it: treat a “signature” as a SHA-256 HMAC of the transaction hash with a “private key” that is just a secret byte string. A “public key” is the SHA-256 hash of the private key. OpCheckSig pops the public key and signature from the stack, recomputes the expected HMAC, and pushes 1 if they match. This preserves the structure of real script verification (unlocking requires knowledge of a secret that corresponds to the locked public key) without requiring you to implement secp256k1.

Standard script patterns to implement and test:

P2PK (Pay-to-Public-Key) — simplest spending condition
  lock:   <pubkey> OP_CHECKSIG
  unlock: <sig>

P2PKH (Pay-to-Public-Key-Hash) — what most Bitcoin outputs use
  lock:   OP_DUP OP_HASH256 <pubkey_hash> OP_EQUALVERIFY OP_CHECKSIG
  unlock: <sig> <pubkey>

Puzzle (pure arithmetic — no keys needed)
  lock:   OP_DUP OP_ADD <target> OP_EQUAL
  unlock: <n>                              (where 2n == target)

Timelock (demonstrates conditional logic)
  lock:   <secret_hash> OP_HASH256 OP_EQUAL
  unlock: <preimage>

CLI Interface

# Initialize a new chain (creates genesis block)
naivechain init

# Show chain info
naivechain info

# Mine a new block (picks up pending transactions from the mempool)
naivechain mine

# Create a transaction (outputs a signed transaction to stdout or a file)
naivechain tx create \
    --from <privkey_hex> \
    --to <pubkey_hash_hex> \
    --amount 5000 \
    --fee 100

# Submit a transaction to the mempool
naivechain tx submit <tx_hex>

# Inspect a transaction by txid
naivechain tx show <txid>

# Inspect a block by height or hash
naivechain block show <height|hash>

# Validate the entire chain from genesis
naivechain validate

# Run a script directly (for debugging the VM)
naivechain script run --unlock "<hex>" --lock "<hex>"

Technical Requirements

Must implement yourself:

• SHA-256 compression function (from FIPS 180-4)
• Merkle tree construction
• Proof of Work mining loop
• UTXO set management
• The full script VM (all opcodes listed above)
• Chain validation (every block, every transaction, every input script)

Allowed crates:

• serde + serde_json — for persisting the chain to disk and CLI output
• hex — encoding byte arrays as hex strings in output
• clap — CLI
• sha2 — only for cross-checking your own implementation. Use assert_eq!(sha256(data), sha2::Sha256::digest(data).as_slice()) in tests.

NOT allowed:

• Any crate that implements SHA-256 as the actual hashing path (your code must call your sha256() function for all chain/script operations)
• Any crate that implements a UTXO model, script VM, or blockchain data structure
• bitcoin, bitcoincash, near-sdk, substrate, or any blockchain SDK

Chain persistence: Serialize the chain (blocks + UTXO set + mempool) to a local JSON file (~/.naivechain/chain.json). No database. Keep it simple.

Expected Concepts Covered

Definition of Done

• sha256() produces identical output to sha2::Sha256::digest() for 20+ test vectors including empty input, single bytes, and multi-block messages
• Merkle root changes when any single transaction in a block is modified (proven by test)
• Mining produces a valid block header whose hash satisfies the difficulty target
• Difficulty adjustment: adjustable difficulty_bits with a test that lower difficulty mines faster
• All four standard script patterns (P2PK, P2PKH, Puzzle, Timelock) execute correctly
• OpCheckSig correctly rejects a signature made with the wrong private key
• A double-spend attempt (spending the same UTXO twice in the same block) is rejected
• naivechain validate detects a tampered block anywhere in the chain and reports which block failed and why
• naivechain script run lets you test VM execution interactively for debugging
• 25+ tests: SHA-256 vectors, Merkle proofs, script execution (valid + invalid), chain tamper detection, UTXO double-spend rejection
• README.md with a worked example showing the full lifecycle: init → create tx → mine → inspect → validate

What You Will Be Able to Say Afterwards

After building this, you can genuinely say you understand how a blockchain works — not at the API level, but at the implementation level. You will know:

• Why changing one transaction in a block invalidates every block after it (you built the hash chain)
• Why miners can’t fake proof of work (you ran the loop yourself)
• Why Bitcoin Script is Turing-incomplete by design (you built a VM with no loops)
• What “unspent transaction output” actually means in memory (you maintain the UTXO set)
• Why “losing your private key” is catastrophic (your simplified OpCheckSig shows the one-way dependency)

This is not a production blockchain. It is a learning tool — and a remarkably effective one.

Phase 3B — ticknorm

⚠️ CONFLICT OF INTEREST NOTICE

This project poses a potential conflict of interest with your employment at LSEG. LSEG’s core business is market data normalization and distribution (via Refinitiv / LSEG Data & Analytics). Building an open-source market data feed normalizer — even a simplified one — while employed at LSEG could violate IP assignment clauses, non-compete clauses, or moonlighting policies in your employment contract. The optics of an LSEG data scientist publishing an “open-source market data normalizer” are also problematic regardless of legal technicalities.

This project is retained for reference only. Use airpost (below) instead. It teaches identical Rust concepts (Tokio async, multi-source TCP ingestion, channels, sqlx, axum, error isolation) in a domain with zero LSEG conflict.

Real-Time Market Data Feed Normalizer

Timeline: (Retained for reference only — see airpost below. Not on the active schedule.)

The Problem It Solves

Financial market data arrives from multiple exchanges in incompatible formats. Exchange A sends price as an integer in basis points. Exchange B sends it as a decimal string. Timestamps come in Unix milliseconds from one, ISO8601 from another, and epoch nanoseconds from a third. Field names differ (last_price vs lastTrade vs lt). Normalizing this in real-time is critical infrastructure at every financial data company, and it is directly relevant to LSEG.

Current landscape:

• Python solutions are too slow for tick-by-tick real-time normalization (GIL + overhead per record)
• Java solutions (common in finance) work but are verbose and memory-heavy
• Existing Rust financial tools (like hftbacktest, barter-rs) focus on backtesting, not feed normalization
• Most firms build this as internal proprietary infrastructure. There is no good open-source version.

Functional Requirements

Architecture:

[Mock Exchange Feed A] ──TCP──┐
[Mock Exchange Feed B] ──TCP──┤──→ ticknorm ──→ [PostgreSQL]
[Mock Exchange Feed C] ──TCP──┘        │
                                       └──→ [JSON API via axum]

Mock Exchange Feeds (you build these too): Three simulated exchange feeds, each sending tick data in a different format:

Feed A (JSON over TCP):

{"sym": "AAPL", "px": 18750, "px_denom": 100, "qty": 200, "ts": 1711234567890, "side": "B"}

Feed B (CSV over TCP):

AAPL,187.50,200,2026-03-15T14:30:00.123Z,BUY

Feed C (JSON over TCP, different schema):

{"ticker": "AAPL", "lastTrade": "187.50", "volume": 200, "timestamp": "1711234567", "direction": "buy"}

Canonical Output Format:

struct NormalizedTick {
    symbol: String,          // "AAPL"
    exchange: Exchange,      // FeedA, FeedB, FeedC
    timestamp_ns: i64,       // nanoseconds since epoch, always UTC
    bid: Option<f64>,
    ask: Option<f64>,
    last_price: f64,         // always as f64 decimal
    volume: u64,
    side: Side,              // Buy, Sell, Unknown
}

Service Behavior:

• Connect to all three feeds simultaneously using Tokio async tasks
• Parse each feed’s proprietary format
• Normalize to the canonical NormalizedTick struct
• Write normalized ticks to PostgreSQL via sqlx (async, compile-time checked)
• Expose a JSON API via axum:
  • GET /ticks/:symbol — last N ticks for a symbol
  • GET /stats/:symbol — VWAP, spread, volume over configurable window
  • GET /feeds/status — health check for each connected feed
• Log throughput: print ticks/second every 10 seconds

Performance target: Process and normalize at least 10,000 ticks/second across all feeds combined.

Technical Requirements

Required crates:

• tokio — async runtime (tasks, TCP, channels)
• serde + serde_json — JSON parsing
• sqlx — async PostgreSQL with compile-time query checking
• axum — HTTP API (you’ve earned the right to use a framework now)
• chrono — timestamp normalization
• tracing — structured logging

Architecture (Rust modules):

main.rs              → Tokio runtime setup, task orchestration
feeds/
  mod.rs             → Feed trait definition
  feed_a.rs          → Feed A parser (JSON, basis points)
  feed_b.rs          → Feed B parser (CSV, ISO timestamps)
  feed_c.rs          → Feed C parser (JSON, different schema)
normalizer.rs        → Core normalization logic
storage.rs           → PostgreSQL writer (sqlx)
api/
  mod.rs             → Axum router setup
  handlers.rs        → API endpoint handlers
mock_exchange.rs     → Mock feed generators (separate binary)

Expected Concepts Covered

Definition of Done

• Three mock exchange generators produce tick data at configurable rates
• ticknorm connects to all three simultaneously and normalizes in real-time
• Normalized data is written to PostgreSQL with correct types
• JSON API returns correct data for all three endpoints
• If one feed disconnects, the others continue processing
• Achieves 10k+ ticks/second throughput
• sqlx queries are compile-time verified (not runtime string SQL)
• 15+ tests (unit tests for each parser, integration test for the pipeline)
• README with architecture diagram and setup instructions

Real-World Value

This is a simplified version of infrastructure that runs in production at LSEG, Bloomberg, ICE, and every financial data vendor. Building it teaches you the core challenges of real-time financial data: schema normalization, timestamp alignment, feed resilience, and high-throughput database writes. Directly applicable to your current role.

Phase 3B (Replacement) — airpost

Real-Time Air Quality Aggregator

Timeline: Weeks 24–28 (primary weekend work + evening coding)

This project replaces ticknorm to avoid conflict of interest with LSEG. It teaches identical Rust concepts in the domain of public environmental data.

The Problem It Solves

Air quality data is fragmented across dozens of incompatible sources. Researchers, journalists, public health advocates, and concerned citizens cannot easily answer the question: “What is the air quality in my city right now, according to all available sensors?”

Each data source uses different formats, different pollutant measurements, different units, different scales, and different update frequencies:

• OpenAQ (global): JSON REST API. Reports PM2.5, PM10, O3, NO2, SO2, CO in µg/m³. Timestamps in ISO 8601 UTC. Station IDs are alphanumeric strings.
• India CPCB (Central Pollution Control Board): India’s government sensor network. Reports AQI (a composite index, not raw pollutant concentrations). Different station naming conventions. Data published via web scraping or XML feeds.
• PurpleAir (citizen science): JSON API. Reports raw PM2.5 particle counts and applies correction factors. Uses sensor IDs (integers). Timestamps in Unix epoch seconds. Data quality varies wildly (sensors are installed by individuals, some are indoors, some are malfunctioning).
• WAQI (World Air Quality Index): Another JSON API. Reports AQI values. Different station identifiers. Rate-limited.

A researcher in Bengaluru who wants to compare government CPCB readings with nearby PurpleAir citizen sensors and OpenAQ global data currently needs to write custom scripts for each source, handle unit conversions manually, and maintain their own storage.

Functional Requirements

Architecture:

[OpenAQ API] ──HTTP──┐
[CPCB Data]  ──HTTP──┤──→ airpost ──→ [PostgreSQL]
[PurpleAir]  ──HTTP──┤        │
[WAQI API]   ──HTTP──┘        └──→ [JSON API via axum]

Data Sources (you implement adapters for each):

Source A — OpenAQ (JSON REST):

{
  "location": "Silk Board Junction, Bengaluru",
  "parameter": "pm25",
  "value": 85.3,
  "unit": "µg/m³",
  "date": {"utc": "2026-03-15T14:30:00.000Z"}
}

Source B — CPCB India (different JSON structure):

{
  "station_name": "BTM Layout, Bangalore",
  "pollutant_id": "PM2.5",
  "pollutant_avg": "142",
  "last_update": "15-Mar-2026 20:00"
}

Source C — PurpleAir (different JSON, raw particle counts):

{
  "sensor_index": 131075,
  "name": "Koramangala Sensor",
  "pm2.5": 45.2,
  "pm2.5_cf_1": 52.1,
  "humidity": 65,
  "temperature": 82,
  "last_seen": 1710512400
}

Canonical Output Format:

struct AirReading {
    station_id: String,           // normalized unique ID
    station_name: String,         // human-readable name
    source: DataSource,           // OpenAQ, CPCB, PurpleAir, WAQI
    latitude: f64,
    longitude: f64,
    city: String,
    pollutant: Pollutant,         // PM25, PM10, O3, NO2, SO2, CO
    value_ugm3: f64,              // always in µg/m³ (converted if needed)
    aqi: Option<u32>,             // computed AQI if raw value available
    timestamp_utc: DateTime<Utc>, // always UTC
    quality_flag: QualityFlag,    // Good, Suspect, Calibrating, Unknown
}

Service Behavior:

• Spawn one Tokio task per data source, each polling on its own schedule (OpenAQ every 5 min, PurpleAir every 2 min, CPCB every 15 min)
• Each adapter parses its source’s proprietary format and converts to AirReading
• Apply unit conversions: AQI → µg/m³ (using EPA breakpoint tables), raw PM counts → corrected µg/m³ (PurpleAir correction factor)
• Quality flagging: mark PurpleAir sensors with humidity > 70% as “Suspect” (high humidity inflates particle counts)
• Write normalized readings to PostgreSQL via sqlx
• Expose JSON API via axum:
  • GET /stations?city=bengaluru — list all stations in a city
  • GET /readings/:station_id?hours=24 — last 24 hours for a station
  • GET /aqi/current?city=bengaluru — current AQI for all stations in a city
  • GET /compare?sources=openaq,purpleair&city=bengaluru — compare readings across sources for the same location
  • GET /sources/status — health check for each data source connection
• Log throughput: print readings ingested/second every polling cycle

Performance target: Handle 50+ concurrent source polls, normalize and store 1,000+ readings per polling cycle, API response time < 50ms.

Technical Requirements

Required crates:

• tokio — async runtime (tasks, timers, HTTP client via reqwest)
• reqwest — async HTTP client for API polling
• serde + serde_json — JSON deserialization from different schemas
• sqlx — async PostgreSQL with compile-time query checking
• axum — HTTP API server
• chrono — timestamp normalization and timezone conversion
• tracing — structured logging

Architecture (Rust modules):

main.rs              → Tokio runtime setup, task orchestration
sources/
  mod.rs             → DataSource trait definition
  openaq.rs          → OpenAQ adapter (JSON, µg/m³)
  cpcb.rs            → CPCB India adapter (different JSON, AQI scale)
  purpleair.rs       → PurpleAir adapter (raw particle counts)
  waqi.rs            → WAQI adapter (AQI values)
normalizer.rs        → Unit conversion, AQI calculation, quality flagging
storage.rs           → PostgreSQL writer (sqlx)
api/
  mod.rs             → Axum router setup
  handlers.rs        → API endpoint handlers
  responses.rs       → JSON response types
config.rs            → Source URLs, polling intervals, API keys

Expected Concepts Covered

Definition of Done

• Four data source adapters implemented (OpenAQ, CPCB, PurpleAir, WAQI)
• All readings normalized to common schema with correct unit conversions
• AQI calculation from raw µg/m³ values using EPA breakpoint tables
• PurpleAir readings flagged when humidity > 70%
• Normalized data written to PostgreSQL with correct types
• JSON API returns correct data for all five endpoints
• If one data source API is down, the others continue polling
• sqlx queries are compile-time verified
• 15+ tests (unit tests for each adapter + unit conversion, integration test for pipeline)
• README with architecture diagram, setup instructions, and API examples

ML Stretch Goal — Short-Horizon Forecasting (optional)

Add GET /forecast/:station_id?hours=6 — a next-hours PM2.5 forecast per station.

• From-scratch multiple linear regression: lag features (t-1, t-2, t-3, t-24), hour-of-day, humidity. Solve the normal equations with your own Gauss elimination or Cholesky — no nalgebra, no linfa.
• Walk-forward evaluation: report MAE against a naive persistence baseline (“next hour = this hour”). The endpoint must report both numbers honestly; if the model doesn’t beat persistence, that’s a finding, not a failure.
• Strictly a stretch goal. The Tokio/axum/sqlx learning is the priority in Phase 3. Do this only if the core DoD is finished early — or defer it to a post-program weekend. (You’ll build serious regression machinery in riskbook and inferd anyway.)

Real-World Value

Why this matters:

• Bengaluru’s air quality has deteriorated significantly in recent years. Citizens and researchers need access to unified, real-time data from all available sources.
• India’s CPCB network has limited coverage. PurpleAir citizen sensors fill gaps but report in incompatible units.
• No open-source tool currently unifies these fragmented sources into a single, queryable, self-hostable service.
• This could be deployed by environmental NGOs, journalism organizations investigating pollution, or citizen science communities.

Why it teaches the same concepts as ticknorm: The architecture is structurally identical — multiple concurrent data sources with incompatible formats, normalization to a canonical schema, async database writes, and a query API. The domain is different; the Rust learning is the same.

Phase 4A — riskbook

Real-Time Portfolio Risk Engine

Timeline: Weeks 31–35 (primary weekend project; the ML/statistical extension — EWMA/GARCH volatility forecasting and Monte Carlo VaR — lands in weeks 34–35)

The Problem It Solves

Computing portfolio risk metrics — Value-at-Risk (VaR), Conditional VaR (CVaR), P&L attribution, and options Greeks — in real-time requires processing thousands of positions against continuously updating market data. Every bank, asset manager, and hedge fund needs this. The landscape:

• Python: Too slow for real-time. Even with NumPy vectorization, per-call overhead and the GIL make tick-by-tick risk updates infeasible. Used only for end-of-day batch risk.
• Java/C++: Common in production trading systems but heavy and proprietary.
• Commercial solutions: Bloomberg PORT, MSCI RiskMetrics, Axioma — all extremely expensive.
• Open-source: QuantLib (C++) is comprehensive but monstrous in complexity. There is no lean, focused, modern open-source risk engine. Rust has nothing in this space.

riskbook fills this gap: a fast, focused, well-documented Rust library for portfolio risk computation, with optional PyO3 bindings.

Functional Requirements

CLI Interface:

# One-shot risk computation
riskbook compute --portfolio positions.csv --prices market_data.csv --var-confidence 0.99

# Streaming mode: update risk as prices change
riskbook stream --portfolio positions.csv --price-feed stdin

# Options Greeks for a single position
riskbook greeks --type call --strike 185.0 --spot 187.50 --vol 0.25 --rate 0.05 --expiry 30d

# Volatility forecasting — EWMA or GARCH(1,1) fitted by your own MLE optimizer (ML extension)
riskbook vol --returns returns.csv --method garch --horizon 10

# Monte Carlo VaR — correlated scenarios via your own Cholesky + Box-Muller (ML extension)
riskbook compute --portfolio positions.csv --prices market_data.csv --var-method montecarlo --simulations 10000

Portfolio File Format (positions.csv):

symbol,type,quantity,entry_price,strike,expiry,option_type
AAPL,equity,1000,185.00,,,
MSFT,equity,500,420.00,,,
AAPL,option,10,5.50,190.00,2026-06-19,call
TSLA,option,5,12.00,250.00,2026-04-17,put

Output — Risk Report:

Portfolio Risk Report (2026-03-15 14:30:00 UTC)
================================================
Positions:     4
Total Notional: $587,500.00
Mark-to-Market: $591,250.00
Unrealized P&L: +$3,750.00 (+0.64%)

Value-at-Risk (99%, 1-day):
  Historical VaR:    -$8,420.00
  Parametric VaR:    -$7,890.00

Options Greeks (Portfolio):
  Delta:  1,342.5
  Gamma:     12.3
  Theta:   -145.2
  Vega:     890.4

Concentration:
  AAPL: 63.4% of notional
  MSFT: 35.6% of notional
  TSLA:  1.0% of notional

Top Risk Contributors:
  1. AAPL equity    (-$5,200 marginal VaR)
  2. AAPL call      (-$1,890 marginal VaR)
  3. MSFT equity    (-$1,120 marginal VaR)

Mathematical Implementations (YOU MUST IMPLEMENT THESE FROM THE FORMULAS — NO LIBRARIES)

1. Black-Scholes Option Pricing:

C = S·N(d1) - K·e^(-rT)·N(d2)
P = K·e^(-rT)·N(-d2) - S·N(-d1)

where:
  d1 = (ln(S/K) + (r + σ²/2)·T) / (σ·√T)
  d2 = d1 - σ·√T
  N(x) = cumulative normal distribution function

You must implement:

• The Gaussian CDF N(x) yourself (use the Abramowitz & Stegun rational approximation or the Horner form polynomial)
• ln(), exp(), sqrt() from std::f64 are allowed (these are CPU instructions)
• The full Black-Scholes formula for calls and puts

2. Greeks (partial derivatives of Black-Scholes):

• Delta: ∂C/∂S = N(d1) for calls, N(d1) - 1 for puts
• Gamma: ∂²C/∂S² = n(d1) / (S·σ·√T) where n(x) is the standard normal PDF
• Theta: ∂C/∂t (time decay per day)
• Vega: ∂C/∂σ = S·n(d1)·√T

3. Historical VaR:

• Given N days of historical returns, compute portfolio return for each day
• Sort returns, take the (1-α) percentile as VaR
• For 99% VaR with 252 days: the 2.52nd worst day (interpolate)

4. Parametric VaR:

• Assume returns are normally distributed
• VaR = μ - z_α · σ (where z_0.99 ≈ 2.326)
• Compute portfolio σ from individual position volatilities and correlations

5. EWMA Volatility (RiskMetrics) — ML extension, weeks 34–35:

σ²_t = λ·σ²_{t-1} + (1-λ)·r²_{t-1}    with λ = 0.94 (daily)

The industry-standard exponentially weighted estimator. ~20 lines, but it makes parametric VaR responsive to recent volatility instead of assuming it’s constant.

6. GARCH(1,1) Volatility Forecasting — ML extension, weeks 34–35:

σ²_t = ω + α·r²_{t-1} + β·σ²_{t-1}

• Fit (ω, α, β) by maximum likelihood: implement the Gaussian log-likelihood over the return series, and maximize it with a Nelder–Mead simplex optimizer you write yourself (~100 lines — reflection, expansion, contraction, shrink)
• Enforce constraints ω > 0, α, β ≥ 0, α + β < 1 via penalty terms or parameter transforms
• Forecast h-day-ahead volatility via the σ² recursion; feed it into parametric VaR
• This is real applied statistics: an objective function, an optimizer, convergence criteria, and local-minimum pitfalls — the same machinery under every ML .fit() call you’ve ever made, with nothing hidden

7. Monte Carlo VaR — ML extension, weeks 34–35:

• Estimate the covariance matrix Σ of position returns from historical data
• Cholesky decomposition from scratch (find L such that L·Lᵀ = Σ) to generate correlated shocks
• Box–Muller transform for standard normal sampling (the rand crate is allowed for the underlying uniform RNG — RNG quality is not the lesson)
• Simulate 10,000 correlated return scenarios, fully revalue the portfolio in each (options through your own Black-Scholes), take the loss percentile
• You now have VaR three ways — historical, parametric, Monte Carlo — and can explain when and why they disagree

Technical Requirements

Required crates:

• clap — CLI
• csv + serde — data parsing
• chrono — date handling for option expiry
• criterion — benchmarking
• pyo3 (optional) — Python bindings
• rand — allowed only as the uniform RNG source for Monte Carlo sampling (Box-Muller, Cholesky, and everything downstream is yours)

NOT allowed for math:

• statrs, nalgebra, ndarray, or any statistics/math/optimization library for the core computations — including the GARCH MLE fit and Cholesky decomposition
• You implement the math. That’s the point.

Performance target:

• Compute full risk report for 1,000 positions in < 100ms
• Update P&L for a single price change in < 1μs

Expected Concepts Covered

Definition of Done

• Black-Scholes pricing matches known test values to 4 decimal places
• Greeks match known test values (use options pricing calculators as reference)
• Put-call parity holds: C - P = S - K·e^(-rT) (within floating-point tolerance)
• Historical VaR computation is correct against a hand-calculated example
• Full risk report for 1,000 positions runs in < 100ms
• CLI produces the formatted report shown above
• Streaming mode updates on new price data from stdin
• criterion benchmarks for all core computations
• 20+ tests including mathematical identity checks
• PyO3 binding works: from riskbook import Portfolio, compute_risk
• (ML extension) EWMA volatility matches a pandas ewm(alpha=0.06).var() reference on fixture data
• (ML extension) GARCH(1,1) parameters fitted on a reference return series land within tolerance of Python’s arch package estimates
• (ML extension) Cholesky verified: L·Lᵀ reconstructs Σ within 1e-10; fails cleanly on non-positive-definite input
• (ML extension) Monte Carlo VaR converges toward parametric VaR on synthetic normal data as simulation count grows (proven by test)

Real-World Value

This is directly applicable to your LSEG role. A well-documented, tested, open-source Rust risk library with Python bindings would attract real users from quantitative finance. The PyO3 path means Python quants can use it without learning Rust. The SaaS path: a risk computation API endpoint.

Phase 4B — xlsxfmt

⚠️ REPLACED — June 10, 2026

This project has been replaced by dataprof (next section) for career-fit reasons, not gap reasons. The bidirectional-XLSX gap is real and well-documented below — but building it (and docforge on top of it) was pulling the program toward document-infrastructure engineering, away from Ashwin’s actual professional center of gravity as a data scientist. dataprof preserves every Phase 4B learning goal (binary formats, performance engineering, PyO3, library design, publishing) in an artifact that gets used in day-to-day DS work immediately. This section is retained for reference; the gap remains open for anyone who wants it.

High-Fidelity Bidirectional Excel Processing Library

Timeline: (Retained for reference only — replaced by dataprof below. Not on the active schedule.)

The Problem It Solves

You lived this problem at LSEG. The Rust Excel ecosystem today:

The gap: No production-quality, pure-Rust library can read an existing XLSX file with full formatting fidelity, modify cell values, and write it back with formatting preserved. This is the core operation in document processing pipelines, report generation, and template-based systems.

xlsxfmt fills this gap.

Functional Requirements

Library API:

use xlsxfmt::Workbook;

// Read with full formatting
let wb = Workbook::open("report.xlsx")?;
let sheet = wb.sheet("Q1 Revenue")?;

// Inspect cell with formatting
let cell = sheet.cell("B5")?;
println!("Value: {}", cell.value());           // "1,234.56"
println!("Font: {} {}pt", cell.font_name(), cell.font_size()); // "Calibri 11pt"
println!("Bold: {}", cell.is_bold());          // true
println!("Fill: {}", cell.fill_color());       // "#4472C4"

// Modify value, preserve ALL formatting
sheet.set_value("B5", 1500.00)?;

// Save — formatting is preserved
wb.save("report_updated.xlsx")?;

CLI Tool:

xlsxfmt inspect report.xlsx                    # List sheets, dimensions, formatting summary
xlsxfmt cell report.xlsx "Sheet1!B5"           # Show cell value + all formatting attributes
xlsxfmt set report.xlsx "Sheet1!B5" "1500.00"  # Update cell, preserve formatting
xlsxfmt diff original.xlsx modified.xlsx       # Show cell-by-cell differences

Technical Deep Dive: XLSX Format

XLSX is a ZIP archive containing XML files (OOXML standard). Key files inside the ZIP:

[Content_Types].xml          — file type declarations
xl/workbook.xml              — sheet names and order
xl/worksheets/sheet1.xml     — cell values and references to shared strings/styles
xl/sharedStrings.xml         — deduplicated string table
xl/styles.xml                — all formatting (fonts, fills, borders, number formats)
xl/theme/theme1.xml          — color themes
_rels/.rels                  — relationship definitions

The critical insight: cell values and cell formatting are stored separately. sheet1.xml has cell references like <c r="B5" s="12"> where s="12" points to style index 12 in styles.xml. To preserve formatting, you must:

1. Parse and retain the full styles.xml
2. When modifying a cell value, keep its style reference (s attribute) unchanged
3. When writing back, ensure the shared strings table is updated correctly
4. Preserve all XML elements and attributes you don’t understand (round-trip fidelity)

Expected Concepts Covered

In-Program Scope (Weeks 37–40 — Basic Implementation)

This is what gets built during the program. Honest about what it covers.

Week 37: Understand the format before writing code. Open a .xlsx file with unzip, read the XML by hand. Implement reading: parse cell values (strings, numbers, booleans), handle the shared string table correctly (values are stored by index reference — getting this wrong is the most common mistake in XLSX parsers).

Week 38: Implement writing — create a new workbook with one or more sheets, write cell values, write basic formatting (bold, font size, background color) through correct styles.xml construction.

Week 39: Cell range reading (A1:D50 → Vec<Vec<Value>>), formula string preservation (read a formula cell, store the formula string, write it back — do not evaluate it, that is formulaengine’s job), row height and column width preservation.

Week 40: PyO3 binding, integration tests (read → modify → write → re-read → assert), README.md that explicitly states what is and is not supported, publish as 0.1.0-alpha.

Definition of Done (In-Program)

• Can read XLSX files created by Excel, LibreOffice, and Google Sheets (cell values, shared strings, basic formatting)
• Basic formatting preserved after read-modify-write: bold, font size, fill color
• Shared strings table is updated correctly when cell values change
• Formula string preservation: formulas read and written back unchanged (not evaluated)
• Row height and column width preserved on write
• CLI inspect shows useful summary of workbook structure
• 15+ tests including round-trip tests
• README explicitly documents what is NOT supported
• Published as 0.1.0-alpha to signal scope
• Benchmarks showing read/write performance vs. calamine + rust_xlsxwriter separately

What is explicitly NOT in scope for Weeks 37–40: merged cells, charts, images, conditional formatting, pivot tables, full number format string parsing (the Excel custom format language is its own mini-language), and complex border/alignment fidelity.

Post-Program Extension Track (Weeks 53–60+ — Advanced Implementation)

Pursue if xlsxfmt becomes a serious OSS project after the program ends. This is outside the 1-year program.

• Weeks 53–54: Full formatting fidelity — borders, alignment, wrap text, number format string parsing (dates, currency, percentages, custom formats)
• Weeks 55–56: Merged cell handling — mergeCells in sheet1.xml, reading/writing merged ranges correctly
• Weeks 57–58: Named ranges, cross-sheet formula references (Sheet2!A1), sheet visibility and protection
• Weeks 59–60: Streaming reader for large files (>100MB) using quick-xml’s streaming API, formulaengine integration as optional feature flag, benchmark vs. Python openpyxl, upgrade from 0.1.0-alpha to 0.2.0

Real-World Value

Every Python developer doing Excel automation hits this wall. LibreOffice is slow and fragile. openpyxl is slow at scale. A fast, safe, pure-Rust library for bidirectional Excel processing would be immediately useful for document pipelines, report generators, and template systems. Directly connects to your LSEG work. Publishable to crates.io. SaaS path: an API that accepts XLSX modifications and returns the result.

Phase 4B (Replacement) — dataprof

Fast Data Profiler for CSV and Parquet

Timeline: Weeks 37–40 (four focused weekends + evenings)

This project replaces xlsxfmt. Same learning goals — binary formats, performance engineering, PyO3, library design, publishing — relocated into the tool a data scientist actually opens every week.

The Problem It Solves

The first hour of every data science task is the same: what’s in this file? Types, missingness, cardinality, distributions, suspicious columns. The current options:

• df.describe() / manual pandas: fast to type, shallow, and you rebuild the same EDA cells in every notebook
• ydata-profiling (formerly pandas-profiling): the standard “full report” tool — and notoriously slow. Minutes on datasets pandas loads in seconds; frequent OOM on wide data; the docs themselves recommend minimal=True for anything large
• Sweetviz / DataPrep / D-Tale: same Python-speed family, same scaling wall
• qsv stats: genuinely fast (Rust) — but CSV-only, CLI-only, no Parquet, no Python API, no report output
• DataProfiler (Capital One): capable, but a heavyweight Python library with a TensorFlow dependency

The gap: a native-speed profiler that reads CSV and Parquet, profiles a gigabyte in seconds, and meets you where you work — as a CLI for the terminal and a PyO3 module for the notebook (import dataprof), emitting a self-contained HTML report or JSON.

Functional Requirements

CLI:

dataprof profile data.parquet --output report.html
dataprof profile data.csv --format json | jq '.columns[] | select(.missing_pct > 0.2)'
dataprof compare baseline.parquet current.parquet     # drift mode — reuses your schemaguard PSI/KS math

Python (PyO3):

import dataprof
p = dataprof.profile("data.parquet")
p.to_dict()              # full profile as nested dict
p.to_html("report.html") # same report as the CLI

Per-column profile:

• Type inference (int / float / string / date / bool), count, missing count + %
• Distinct count: exact for low cardinality, HyperLogLog beyond a threshold
• min / max / mean / std via Welford’s one-pass algorithm (numerically stable — naive sum-of-squares fails on real data, and you’ll prove it in a test)
• Quantiles (exact for small data, reservoir sampling beyond)
• Histogram (streaming, fixed bins after a first-pass range estimate or two-pass)
• Top-k frequent values via Misra–Gries heavy hitters
• Pearson correlation matrix across numeric columns (one pass)

Drift mode (compare): per-column PSI and KS statistics between two files — the same math you wrote for schemaguard drift, now over Parquet too.

Technical Requirements

Must implement yourself (the sketching algorithms are the point):

• Welford’s online mean/variance
• HyperLogLog (the 2007 paper’s algorithm; ±2% at 16KB of state — understand why it works)
• Reservoir sampling (Algorithm R)
• Misra–Gries heavy hitters
• The HTML report template (hand-rolled, single self-contained file — no template engine)

Allowed crates:

• arrow + parquet — Parquet reading and the Arrow columnar model (learning Arrow’s memory layout is an explicit goal: it’s the Rust that already lives inside your Python stack via Polars/PyArrow)
• csv, serde + serde_json, clap, rayon, pyo3, criterion

NOT allowed:

• polars, datafusion (defeats the purpose)
• Any sketching/statistics crate (hyperloglog, streaming_algorithms, etc.) — the sketches are yours

Parallelism: per-column profiling via rayon — this is the program’s dedicated rayon exercise. Measure the scaling curve (1, 2, 4, 8 threads) and explain its shape in the README.

Performance targets:

• Profile a 1GB Parquet file in < 10 seconds on a laptop
• ≥ 20x faster than ydata-profiling on the same 1M-row dataset (benchmark honestly: same stats compared)

Week-by-Week Plan

• Week 37: CSV profiling core — type inference, Welford, missing counts, exact distinct/quantiles. CLI skeleton.
• Week 38: The sketches — HyperLogLog, reservoir sampling, Misra–Gries — each verified against exact computation on fixtures. rayon parallelism + scaling measurements.
• Week 39: Parquet/Arrow reading, correlation matrix, HTML report, compare drift mode.
• Week 40: PyO3 binding, benchmarks vs ydata-profiling and qsv, README, publish 0.1.0.

Expected Concepts Covered

Definition of Done

• Profiles CSV and Parquet with identical output schema
• Welford results match pandas mean()/std() within 1e-9 on fixtures
• HyperLogLog within 2% of exact distinct count on 10M-distinct synthetic data
• Misra–Gries top-k verified against exact counts on skewed fixtures
• rayon scaling curve measured and documented
• 1GB Parquet profiled in < 10s; ≥ 20x ydata-profiling on the benchmark dataset
• compare mode produces PSI/KS drift output consistent with schemaguard drift
• PyO3 binding works in a real notebook; HTML report opens self-contained
• 20+ tests including sketch-vs-exact property tests
• Published to crates.io (and PyPI via maturin if time allows)

Real-World Value

Unlike xlsxfmt, this needs no imagination to justify: you will use it yourself, at work, within a month of building it — and so will any data scientist colleague you show it to. ydata-profiling’s slowness is a meme in the field; qsv proves the appetite for Rust-speed tabular tooling. The sketching algorithms are simultaneously the most “fundamental CS” content in Phase 4 and the exact techniques running inside every production analytics system (Redis, BigQuery, Spark all ship HyperLogLog). Zero COI: nothing here touches market data or LSEG’s product space.

THREE ACTIVE CAPSTONE OPTIONS (June 10, 2026 revision): Choose one of formulaengine (Option B — spreadsheet formula evaluator), ruleforge (Option C — business rules engine), or inferd (Option D — ML model inference server). B/C share a parsing + trait-architecture core; D trades the parser for async serving, real FFI, and ML systems engineering. All end with benchmarking, Python interop, and publishing.

docforge (Option A) is retired — it depended on xlsxfmt (now replaced) and pointed the program at document-infrastructure engineering rather than data science. Its spec is retained below for reference.

Recommendation: Option D (inferd) for ML-engineering career capital; Option B (formulaengine) if the parser/interpreter itch is what excites you most — it’s the most “computer science” project in the program, and intellectual pull is what gets capstones finished. Both are legitimate; pick by honest self-assessment in week 40, not now.

Phase 5 — docforge

⚠️ RETIRED — June 10, 2026

Retired as a capstone option. Two reasons: (1) it builds directly on xlsxfmt, which has been replaced by dataprof; (2) a year that ends in “I built a document converter” optimizes for a document-tooling career the owner of this program doesn’t want. The LibreOffice gap is real and stays documented in the gap analysis — this is a fit decision, not a quality one. Retained for reference.

Document Conversion Library (Capstone — Option A)

Timeline: (Retained for reference only — retired. Not on the active schedule.)

The Problem It Solves

Converting documents (XLSX → PDF, DOCX → PDF) with formatting fidelity is universally needed and universally painful. The open-source standard is LibreOffice in headless mode:

• 500MB+ install size (includes a full office suite you don’t need)
• 3–8 second startup time per conversion (JVM-like cold start)
• Documented formatting bugs: pagination breaks, font substitution issues, merged cell rendering errors (hundreds of Stack Overflow threads)
• Sequential only: one conversion at a time per process instance
• 700 seconds for a batch — your own LSEG measurement

wkhtmltopdf is abandoned. Headless Chrome is 200MB+ and only handles HTML. Python wrappers (python-docx + reportlab) are slow and lose formatting.

You already proved this can be done better — your LSEG tool does it in 3.5 seconds with parallel processing. docforge takes that proof-of-concept and makes it a proper, open-source, publishable Rust library.

Phased Build Plan

Phase 5a — XLSX → PDF (Weeks 21–22):

Build on xlsxfmt from Phase 4B. The rendering pipeline:

XLSX file
  → xlsxfmt reads cells + formatting
  → Layout engine computes column widths, row heights, page breaks
  → PDF renderer draws cells, borders, text with correct fonts/colors
  → Output PDF file

Must handle:

• Column widths and row heights (from XLSX metadata)
• Cell borders (thin, medium, thick, dashed, dotted)
• Fill colors (solid fills)
• Font styles (bold, italic, size, color)
• Number formatting (currency $1,234.56, percentages 45.2%, dates)
• Merged cells
• Multi-sheet documents (one PDF page per sheet, or configurable)
• Page orientation (portrait/landscape from XLSX print settings)

Phase 5b — DOCX → PDF (Weeks 23–24):

DOCX is also ZIP + XML (OOXML). The XML structure is different from XLSX but the parsing approach is similar.

Must handle:

• Paragraphs with alignment (left, center, right, justified)
• Headings (H1–H6 mapped from DOCX heading styles)
• Bold, italic, underline, strikethrough
• Font size and color
• Tables with borders and cell formatting
• Page breaks (explicit and auto)
• Basic list items (numbered and bulleted)

Phase 5c — Polish and Publish (Weeks 25–26):

• Clean, ergonomic public API for both XLSX and DOCX conversion
• CLI: docforge convert input.xlsx --output output.pdf
• Comprehensive documentation with examples
• Benchmarks: conversion time and file size vs. LibreOffice headless
• PyO3 binding: docforge.xlsx_to_pdf("input.xlsx", "output.pdf")
• Publish to crates.io
• Build CI/CD with GitHub Actions

Technical Requirements

Required crates:

• xlsxfmt (your Phase 4B library) — XLSX reading
• quick-xml — XML parsing for DOCX
• zip — ZIP archive handling
• printpdf — PDF generation (or lopdf for lower-level control)
• rusttype or ab_glyph — font rendering and text measurement
• clap — CLI
• pyo3 — Python binding
• criterion — benchmarking

Architecture:

docforge/
  src/
    lib.rs           → Public API: convert_xlsx_to_pdf(), convert_docx_to_pdf()
    xlsx/
      reader.rs      → Uses xlsxfmt for XLSX reading
      layout.rs      → Page layout computation (column widths, pagination)
    docx/
      reader.rs      → DOCX XML parsing
      layout.rs      → Paragraph/table layout computation
    pdf/
      renderer.rs    → PDF rendering engine (shared between XLSX and DOCX)
      fonts.rs       → Font loading and text measurement
      page.rs        → Page model (dimensions, margins)
    cli.rs           → CLI entry point
  python/
    src/lib.rs       → PyO3 bindings
  benches/
    conversion.rs    → Benchmark suite vs LibreOffice
  tests/
    fixtures/        → Test XLSX and DOCX files created in Excel/Word
    round_trip.rs    → Convert and verify output

Expected Concepts Covered

Definition of Done

• XLSX → PDF conversion handles all listed formatting features
• DOCX → PDF conversion handles paragraphs, headings, tables, basic styles
• CLI works: docforge convert input.xlsx -o output.pdf
• Benchmarks show measurable speed improvement over LibreOffice headless
• PyO3 binding works: docforge.xlsx_to_pdf(input_path, output_path)
• Published on crates.io with version 0.1.0
• README with architecture overview, usage examples, and benchmark results
• 30+ tests including fixture-based visual verification
• CI/CD with GitHub Actions (test on Linux, macOS, Windows)
• At minimum 3 real-world test files (created in Excel/Word/Google Docs) convert correctly

Real-World Value and SaaS Potential

Open-source impact: Every data engineering team, every report generation system, every document pipeline hits the LibreOffice problem. A fast, native-code, well-documented alternative would get significant adoption.

SaaS path: A REST API that accepts document uploads and returns PDFs:

POST /convert
Content-Type: multipart/form-data
file: report.xlsx

→ 200 OK
Content-Type: application/pdf
[PDF bytes]

Hosted on a single server, this could serve hundreds of concurrent conversions (unlike LibreOffice which handles one at a time). Real businesses pay $50–500/month for document conversion APIs. You already proved the concept at LSEG. This is the productizable version.

Career impact: A published crate that solves a real problem, with benchmarks proving it outperforms the incumbent, with Python bindings for broad accessibility — this is the kind of project that transforms a resume from “I used frameworks” to “I built infrastructure.”

Phase 5 (Alternative) — formulaengine

Spreadsheet Formula Computation Engine (Capstone — Option B)

Timeline: Weeks 41–52

The Problem It Solves

Every product with a “calculated field,” “formula column,” or “computed cell” needs a formula evaluation engine under the hood — Airtable, Notion databases, Google Sheets, internal pricing tools, financial models, report builders. Building one from scratch is hard: you need a parser, a dependency resolver, circular reference detection, type coercion, and hundreds of built-in functions.

Today, companies solve this by:

• Embedding full Excel via COM automation — Windows-only, fragile, requires a licensed Excel installation, impossible to deploy in containers
• Wrapping LibreOffice Calc — same 500MB+ footprint problems as document conversion, slow, unstable
• Reimplementing every formula in application code — no reuse, error-prone, impossible to let end-users define their own formulas
• Using JavaScript-based engines (HyperFormula, FormulaJS) — single-threaded, slow on large sheets, no systems-level performance

There is no standalone, embeddable formula evaluation library in Rust, Go, or C++. The closest things are full spreadsheet applications (LibreOffice Calc, Gnumeric) where the formula engine is deeply coupled to the UI and cannot be extracted.

XLSX integration note (June 2026): with xlsxfmt replaced by dataprof, the XLSX round-trip uses the established crates instead — calamine to read formulas and values from existing files, rust_xlsxwriter to write computed results. This costs nothing pedagogically (the engine is the project; file I/O is plumbing) and removes a dependency on a project you’re no longer building.

Phased Build Plan

Phase 5a — Parser and Core Engine (Weeks 21–22):

Build the formula language parser and basic evaluation engine.

"=IF(AND(A1>100, B1<50), A1*0.15, A1*0.10)"
  → Tokenizer: [EQUALS, FUNC("IF"), LPAREN, FUNC("AND"), ...]
  → AST: FunctionCall("IF", [FunctionCall("AND", [...]), Multiply(...), Multiply(...)])
  → Evaluator: walks the AST, resolves cell references, returns Value::Number(15.0)

Must handle:

• Tokenizer/Lexer: Numbers, strings, booleans, cell references (A1, $A$1, A1:B10), operators (+, -, *, /, ^, &, =, <>, <, >, <=, >=), function names, parentheses, commas
• Parser: Recursive descent parser producing an AST. Operator precedence (PEMDAS). Nested function calls.
• Cell reference resolution: Absolute ($A$1), relative (A1), ranges (A1:B10), cross-sheet (Sheet2!A1)
• Type system: Value enum — Number(f64), Text(String), Boolean(bool), Error(FormulaError), Empty
• Type coercion: “123” + 1 = 124 (Excel-compatible coercion rules)
• Core functions (20+): SUM, AVERAGE, COUNT, COUNTA, MIN, MAX, IF, AND, OR, NOT, CONCATENATE, LEFT, RIGHT, MID, LEN, TRIM, UPPER, LOWER, ROUND, ABS, MOD

Phase 5b — Dependency Graph and Advanced Functions (Weeks 23–24):

Cells reference other cells. You must evaluate them in the right order.

A1 = 10
B1 = =A1 * 2          → depends on A1
C1 = =B1 + A1          → depends on B1 and A1
D1 = =SUM(A1:C1)       → depends on A1, B1, C1

Dependency graph:
  A1 → B1 → C1 → D1

Topological sort → evaluate A1 first, then B1, then C1, then D1

Must handle:

• Dependency graph: Build a directed graph of cell → cell dependencies
• Topological sort: Evaluate cells in dependency order
• Circular reference detection: A1 = =B1, B1 = =A1 → detect and return Error::CircularReference
• Incremental recalculation: When A1 changes, only recalculate cells that depend on A1 (not the entire sheet). Use the dependency graph to find the dirty set.
• Range functions at scale: SUM(A1:A100000) must not clone 100k values
• Advanced functions (20+): VLOOKUP, HLOOKUP, INDEX, MATCH, SUMIF, COUNTIF, AVERAGEIF, IFERROR, ISBLANK, ISNA, DATE, YEAR, MONTH, DAY, NOW, TODAY, TEXT, VALUE, INDIRECT, OFFSET
• Array formulas: Basic support for functions that return ranges
• Named ranges: revenue → A1:A12

Phase 5c — Polish and Publish (Weeks 25–26):

• Clean, ergonomic public API with builder pattern
• XLSX integration: load formulas via calamine, evaluate, write results back via rust_xlsxwriter
• CLI: formulaengine eval spreadsheet.xlsx --recalculate --output computed.xlsx
• CLI: formulaengine eval --formula "=SUM(1,2,3)" --format value
• Benchmarks: evaluate 100k cells with dependencies, compare to Python (openpyxl + manual eval)
• PyO3 binding: engine.set_cell("A1", 100); engine.set_formula("B1", "=A1*2"); engine.evaluate("B1")
• Comprehensive documentation with examples
• Publish to crates.io
• Build CI/CD with GitHub Actions

Technical Requirements

Required crates:

• calamine (read) + rust_xlsxwriter (write) — XLSX integration
• clap — CLI
• pyo3 — Python binding
• criterion — benchmarking
• petgraph — dependency graph (or hand-roll with HashMap + topological sort)
• chrono — date/time functions
• tracing — debugging complex evaluation chains

Architecture:

formulaengine/
  src/
    lib.rs              → Public API: Engine::new(), set_cell(), set_formula(), evaluate()
    parser/
      tokenizer.rs      → Lexer: input string → token stream
      ast.rs            → AST node types: Expr, CellRef, Range, FunctionCall
      parser.rs         → Recursive descent parser: tokens → AST
    eval/
      evaluator.rs      → AST walker: resolves references, applies operators, calls functions
      value.rs          → Value enum (Number, Text, Boolean, Error, Empty)
      coercion.rs       → Type coercion rules (Excel-compatible)
    functions/
      registry.rs       → FunctionRegistry: trait-based, extensible
      math.rs           → SUM, AVERAGE, ROUND, ABS, MOD, etc.
      text.rs           → CONCATENATE, LEFT, RIGHT, MID, LEN, TRIM, etc.
      logical.rs        → IF, AND, OR, NOT, IFERROR, etc.
      lookup.rs         → VLOOKUP, HLOOKUP, INDEX, MATCH, etc.
      date.rs           → DATE, YEAR, MONTH, DAY, NOW, TODAY, etc.
      statistical.rs    → COUNT, COUNTA, COUNTIF, SUMIF, AVERAGEIF, etc.
    graph/
      dependency.rs     → Build dependency graph from cell formulas
      topo.rs           → Topological sort for evaluation order
      incremental.rs    → Dirty-set tracking for incremental recalc
    sheet.rs            → Sheet model: cells, values, formulas
    cli.rs              → CLI entry point
  python/
    src/lib.rs          → PyO3 bindings
  benches/
    evaluation.rs       → Benchmark: 100k cells, complex dependency chains
    functions.rs        → Benchmark: individual function performance
  tests/
    excel_compat.rs     → Test against known Excel outputs
    circular.rs         → Circular reference detection tests
    incremental.rs      → Incremental recalculation tests
    fixtures/           → Test XLSX files with formulas and expected results

Expected Concepts Covered

Definition of Done

• Parser handles all standard Excel formula syntax (operators, functions, cell refs, ranges)
• 40+ built-in functions across math, text, logical, lookup, date, and statistical categories
• Dependency graph correctly resolves evaluation order via topological sort
• Circular references detected and reported as Error::CircularReference
• Incremental recalculation: changing one cell only recalculates its dependents
• XLSX integration: load formulas via calamine, evaluate, write results back via rust_xlsxwriter
• CLI works: formulaengine eval spreadsheet.xlsx --recalculate --output computed.xlsx
• Benchmarks: 100k cell evaluation under 1 second
• PyO3 binding works: engine.evaluate("B1") returns correct value from Python
• Published on crates.io with version 0.1.0
• README with architecture overview, usage examples, and benchmark results
• 50+ tests including Excel-compatibility verification
• CI/CD with GitHub Actions (test on Linux, macOS, Windows)
• Custom functions: users can register their own via the Function trait

Real-World Value and SaaS Potential

Open-source impact: Every “smart spreadsheet” product, every internal tool with calculated fields, every reporting engine that needs user-defined formulas would benefit from an embeddable, high-performance formula engine. This is infrastructure-level software — the kind of dependency that gets millions of downloads.

SaaS path: A REST API for formula evaluation:

POST /evaluate
{
  "cells": {
    "A1": {"value": 1000},
    "A2": {"value": 2000},
    "A3": {"formula": "=SUM(A1:A2)"},
    "B1": {"formula": "=A3 * 0.15"}
  },
  "evaluate": ["A3", "B1"]
}

→ 200 OK
{
  "results": {
    "A3": 3000,
    "B1": 450
  }
}

This powers pricing calculators, financial models, and report builders without requiring each product to implement their own formula engine. Companies like Rows.com, Causal.app, and Equals.app are building exactly this — they’d pay for a reliable, fast engine they can embed.

Career impact: Building a parser + evaluator + dependency graph + incremental computation engine is the most “computer science” project in this entire program. It demonstrates compiler knowledge, graph algorithm fluency, and systems-level thinking. For a data scientist, it’s also closer to home than it looks: dependency graphs with topological sort and incremental recomputation are exactly the machinery inside dbt, Airflow, and every feature-pipeline DAG you’ve ever debugged.

Phase 5 (Alternative) — ruleforge

Embeddable Business Rules Engine (Capstone — Option C)

Timeline: Weeks 41–52

The Problem It Solves

Every enterprise application is full of business logic that changes frequently: pricing rules, eligibility checks, approval workflows, risk thresholds, compliance policies, discount tiers, routing logic. Today, this logic lives scattered across application code:

# This is in 47 different files across 3 microservices
if customer.tier == "gold" and order.total > 5000:
    discount = 0.15
elif customer.tier == "gold" and order.total > 1000:
    discount = 0.10
elif customer.loyalty_years > 5:
    discount = 0.08
else:
    discount = 0.0

When these rules change — and they change constantly — a developer must find every instance, modify code, get it reviewed, test it, and redeploy. A rules engine externalizes this logic into a simple, readable format that non-developers can modify:

rule "Gold tier high-value discount"
  when customer.tier == "gold" AND order.total > 5000
  then set discount = 0.15

rule "Gold tier standard discount"
  when customer.tier == "gold" AND order.total > 1000
  then set discount = 0.10

rule "Loyalty discount"
  when customer.loyalty_years > 5
  then set discount = 0.08

Rules are loaded at runtime. No redeployment. No developer needed for business logic changes. This is how banks process loan applications, how insurance companies evaluate claims, how e-commerce platforms manage pricing, and how healthcare systems enforce compliance.

The current landscape:

• Drools (Java): The dominant open-source rules engine. 20+ years old, massive codebase, requires JVM. Powerful but heavyweight — overkill for embedding in a microservice.
• Open Policy Agent (OPA) (Go): Policy engine from CNCF. Excellent for authorization and infrastructure policy. Uses Rego, a purpose-built query language. Not designed for general business rules (no arithmetic, no stateful rule chaining).
• json-rules-engine (JavaScript): Lightweight JSON-based engine. Decent for simple conditions but no DSL, no rule chaining, no conflict resolution, limited operator set.
• Grule (Go): Go rules engine inspired by Drools. Active but limited documentation and smaller ecosystem.

There is no embeddable rules engine in Rust. Not even an experimental one. A search for “rules engine” on crates.io returns nothing meaningful. This is a genuine, total gap.

Phased Build Plan

Phase 5a — Rule Language and Parser (Weeks 21–22):

Design and implement the rule DSL (Domain-Specific Language). This is the core CS challenge — you’re building a small programming language.

Rule language grammar (simplified):

ruleset     = rule+
rule        = "rule" STRING priority? condition action
priority    = "priority" INTEGER
condition   = "when" expression
action      = "then" statement ("AND" statement)*
expression  = comparison (("AND" | "OR") comparison)*
comparison  = accessor operator value
accessor    = IDENTIFIER ("." IDENTIFIER)*
operator    = "==" | "!=" | ">" | "<" | ">=" | "<=" | "contains" | "matches"
value       = STRING | NUMBER | BOOLEAN | accessor
statement   = "set" accessor "=" expression
            | "emit" STRING
            | "flag" STRING

Must handle:

• Tokenizer/Lexer: Keywords (rule, when, then, set, AND, OR, NOT), identifiers, operators, literals (strings, numbers, booleans)
• Parser: Recursive descent parser producing a Rule AST
• Rule types: Conditions with AND/OR/NOT combinators, nested field access (customer.address.country), comparison operators including contains (for lists) and matches (for regex)
• Actions: set (modify a field), emit (produce an output event/message), flag (mark for downstream processing)
• Priority system: Rules have optional priority (higher runs first). When multiple rules match, priority determines order.
• Rule validation: Catch syntax errors with clear error messages (“Line 3: expected operator after ‘customer.tier’, found ‘gold’”)

Phase 5b — Evaluation Engine and Conflict Resolution (Weeks 23–24):

Build the engine that evaluates rules against data (called “facts” in rules engine terminology).

Facts (input):
{
  "customer": { "tier": "gold", "loyalty_years": 7 },
  "order": { "total": 6200, "items": 3 }
}

Rules evaluated → matching rules:
  1. "Gold tier high-value discount" (priority 10) → set discount = 0.15
  2. "Loyalty discount" (priority 5) → set discount = 0.08

Conflict resolution (highest priority wins):
  → discount = 0.15

Output:
{
  "discount": 0.15,
  "matched_rules": ["Gold tier high-value discount", "Loyalty discount"],
  "applied_rule": "Gold tier high-value discount"
}

Must handle:

• Fact model: Accept JSON-like data structures. Support nested field access via dot notation.
• Expression evaluator: Walk the AST, resolve field references against facts, apply operators, short-circuit AND/OR
• Conflict resolution strategies: Configurable — priority (highest wins), first_match (stop at first match), all (apply all matching rules in order), custom (user-defined via trait)
• Rule chaining: When a rule’s action modifies a fact, re-evaluate other rules that depend on that fact. Detect infinite loops (rule A triggers rule B which triggers rule A).
• Type coercion: “123” == 123 (configurable strict/loose mode)
• Built-in functions: len(), sum(), avg(), min(), max(), now(), contains(), matches() (regex)
• Audit trail: Log which rules fired, in what order, what they changed — critical for compliance

Phase 5c — Polish and Publish (Weeks 25–26):

• Clean, ergonomic public API with builder pattern
• CLI: ruleforge eval --rules pricing.rules --facts order.json
• CLI: ruleforge validate --rules pricing.rules (syntax check without evaluation)
• CLI: ruleforge explain --rules pricing.rules --facts order.json (show which rules matched and why)
• Benchmarks: evaluate 10k rule sets against complex facts, compare to json-rules-engine (JS) and Grule (Go)
• PyO3 binding: engine.load_rules("pricing.rules"); result = engine.evaluate(facts)
• Comprehensive documentation with examples
• Publish to crates.io
• Build CI/CD with GitHub Actions

Technical Requirements

Required crates:

• serde + serde_json — JSON fact parsing and result serialization
• clap — CLI
• pyo3 — Python binding
• criterion — benchmarking
• regex — matches operator support
• chrono — date/time comparisons and now() function
• tracing — audit trail and rule execution logging

Architecture:

ruleforge/
  src/
    lib.rs              → Public API: Engine::new(), load_rules(), evaluate()
    parser/
      tokenizer.rs      → Lexer: rule DSL text → token stream
      ast.rs            → AST node types: Rule, Condition, Action, Expr, Accessor
      parser.rs         → Recursive descent parser: tokens → Vec<Rule>
      validator.rs      → Semantic validation (type checking, undefined field warnings)
    engine/
      evaluator.rs      → Evaluate conditions against facts
      resolver.rs       → Conflict resolution strategies (priority, first_match, all)
      chaining.rs       → Rule chaining with loop detection
      audit.rs          → Execution trace: which rules fired, what changed
    facts/
      model.rs          → Fact data structure (JSON-like nested key-value)
      accessor.rs       → Dot-notation field access (customer.address.country)
      coercion.rs       → Type coercion rules (strict/loose mode)
    functions/
      registry.rs       → Built-in function registry (len, sum, contains, matches, etc.)
      builtins.rs       → Built-in function implementations
    cli.rs              → CLI entry point (eval, validate, explain)
  python/
    src/lib.rs          → PyO3 bindings
  benches/
    evaluation.rs       → Benchmark: 10k rules × complex facts
    parsing.rs          → Benchmark: parse large rule files
  tests/
    fixtures/           → Test rule files and fact JSON files
    evaluation.rs       → Rule evaluation correctness tests
    chaining.rs         → Rule chaining and loop detection tests
    conflict.rs         → Conflict resolution strategy tests
    edge_cases.rs       → Type coercion, null handling, nested access

Expected Concepts Covered

Definition of Done

• Rule DSL parser handles conditions, actions, priorities, nested field access, AND/OR/NOT
• 10+ built-in functions (len, sum, avg, min, max, contains, matches, now, etc.)
• Three conflict resolution strategies: priority, first_match, all
• Rule chaining works: action modifying a fact re-triggers dependent rules
• Infinite chain loop detection with clear error message
• Audit trail: every evaluation produces a log of which rules fired and what changed
• CLI works: ruleforge eval --rules pricing.rules --facts order.json
• CLI explain mode: shows human-readable reasoning for each matched rule
• Benchmarks: 10k rule evaluation under 100ms for simple rules
• PyO3 binding works: engine.evaluate(facts) returns results from Python
• Published on crates.io with version 0.1.0
• README with rule language reference, usage examples, and benchmark results
• 50+ tests including edge cases (null fields, type mismatches, empty rules)
• CI/CD with GitHub Actions (test on Linux, macOS, Windows)
• Custom functions: users can register their own via the Function trait
• Custom conflict resolvers: users can implement ConflictResolver trait

Real-World Value and SaaS Potential

Open-source impact: Every enterprise that processes decisions — loan approvals, insurance claims, pricing, compliance checks, access control — needs a rules engine. The only serious open-source option (Drools) requires JVM. A lightweight, embeddable Rust engine with Python bindings would fill a massive gap in the cloud-native ecosystem.

SaaS path: A Rules-as-a-Service API:

POST /evaluate
{
  "ruleset_id": "pricing-v3",
  "facts": {
    "customer": { "tier": "gold", "loyalty_years": 7 },
    "order": { "total": 6200 }
  }
}

→ 200 OK
{
  "results": {
    "discount": 0.15,
    "matched_rules": ["Gold tier high-value discount"],
    "audit_trail": [
      { "rule": "Gold tier high-value discount", "matched": true, "action": "set discount = 0.15" },
      { "rule": "Loyalty discount", "matched": true, "action": "set discount = 0.08", "superseded_by": "Gold tier high-value discount" }
    ]
  }
}

Companies would pay to externalize their business rules into a managed service with versioning, audit trails, and A/B testing of rule sets. This is exactly what LaunchDarkly does for feature flags — but for business logic.

Career impact: Building a DSL parser + evaluation engine + conflict resolution system demonstrates language design thinking — the kind of work that senior engineers and compiler teams do. The audit trail feature shows compliance awareness. The embeddable architecture shows systems thinking. This is a project that speaks directly to enterprise engineering leadership.

Phase 5 (Alternative) — inferd

ML Model Inference Server (Capstone — Option D — ML/DL)

Timeline: Weeks 41–52

The Problem It Solves

The standard ML deployment story is: train in Python, then serve it… somehow. The “somehow” is in bad shape:

• TorchServe: was the default answer for PyTorch — archived in August 2025, no maintenance, no security patches
• NVIDIA Triton: production-grade but a heavyweight GPU-first C++ system; massive operational overhead for serving a few small CPU models
• BentoML / Ray Serve: Python frameworks — convenient, but the serving path inherits Python latency, GIL throughput limits, and a fat container image
• KServe / Seldon: Kubernetes platforms, not libraries — you adopt a cluster architecture to serve a logistic regression
• FastAPI + onnxruntime (what most teams actually do): works, but no batching, no model registry, p99 latency at the mercy of the event loop, and ~10x the memory of a native binary

The gap: a single-binary, CPU-friendly inference server with dynamic batching — drop it on a VPS or in a scratch container, point it at an ONNX file, get a fast, observable prediction API.

Why it’s the right capstone for you specifically: it sits exactly on the boundary between your day job (training models in Python) and what this program teaches (async Rust, FFI, performance). You train and export in Python; Rust owns everything after that. It is also the strongest portfolio piece for a “data scientist → ML engineer with systems depth” positioning.

Functional Requirements

Architecture:

[Python: train model → export ONNX + preprocessing params JSON]
        │
        ▼
inferd serve --config models.toml --port 8080
        │
  ┌─────┴───────────────────────────────────────┐
  │  axum HTTP API                              │
  │    POST /v1/models/:name/predict            │
  │  dynamic batcher (mpsc + tokio timers)      │
  │  trait InferenceBackend                     │
  │    ├── MlpBackend   (your own forward pass) │
  │    ├── OrtBackend   (ONNX Runtime via ort)  │
  │    └── CandleBackend (pure Rust)            │
  │  feature pipeline (scaler/one-hot, yours)   │
  │  input validation (schemaguard-style)       │
  │  GET /metrics (Prometheus text, hand-rolled)│
  └─────────────────────────────────────────────┘

Endpoints:

• POST /v1/models/:name/predict — JSON feature map in, prediction(s) out
• POST /v1/embed — text (or batch of texts) in, normalized embedding vectors out
• GET /v1/models — loaded models, versions, backends
• POST /v1/models/:name/load / unload — hot loading without dropping in-flight requests
• GET /metrics — Prometheus exposition format: request counts, latency histograms, batch-size distribution
• GET /healthz

Model registry (models.toml): per model — name, version, artifact path, backend, preprocessing spec path, batching config (max_batch_size, max_wait_ms).

Dynamic batching — the heart of the project:

• Each request sends (features, oneshot::Sender) into an mpsc channel
• A batcher task collects up to max_batch_size requests or waits at most max_wait_ms (tokio::select! over channel + timer)
• One batched inference call; results routed back through the oneshot channels
• This is the same mechanism Triton uses internally — you build it from channels yourself

From-Scratch Components (program philosophy applies here too)

1. MLP forward pass with zero dependencies: before touching ort or candle, implement inference for a small multi-layer perceptron yourself — matrix multiply, bias add, ReLU/sigmoid — loading weights from JSON exported by your Python training script. You will have done neural-network inference with nothing but Vec<f32> once before any library does it for you.
2. The feature preprocessing pipeline: StandardScaler, MinMaxScaler, one-hot encoding implemented in Rust, parameters (means, stds, vocabularies) loaded from JSON exported at training time. Golden test: Rust pipeline output must match sklearn’s .transform() within 1e-12 on fixture data. Train/serve skew is one of the most common real production ML failures — you’ll now understand it at the byte level.
3. The dynamic batcher: channels and timers only, no library.
4. Prometheus exposition format: render the text format yourself, including histogram buckets — no metrics crate.

Where libraries ARE the lesson: ort wraps the ONNX Runtime C library — real FFI, real linking, real “who owns this memory” questions. candle shows you what a pure-Rust tensor library looks like. Both sit behind your trait InferenceBackend, and both must pass the same conformance test suite as your hand-rolled MLP backend.

Demo Models (public data, zero LSEG conflict)

1. Realized-volatility forecaster (finance): a small MLP (or gradient-boosted model) trained in Python on lagged realized volatility computed from public daily index closes (NIFTY 50 / S&P 500 from a public source). Exported to ONNX. Regression task.
2. Credit default classifier (finance): trained on the public German Credit (or Lending Club) dataset. Classification task with categorical features — exercises one-hot encoding and class probabilities.
3. Sentence-embedding endpoint (the “2026 relevance” anchor): serve a small open embedding model (e.g., all-MiniLM-L6-v2 exported to ONNX) at POST /v1/embed — text in, normalized vector out. Tokenization via Hugging Face’s tokenizers crate (Rust-native); mean pooling and L2 normalization implemented yourself over the raw model output. Embeddings are the workhorse of RAG, semantic search, deduplication, and clustering — this is the single most immediately-deployable endpoint you can build in 2026, and embedding serving is exactly where Python overhead hurts most (high request volume, small compute per request). Bonus: your server now produces the vectors that Qdrant (your OSS-track target) stores — the two halves of a semantic search stack, one of which you built.

COI note: all three use public datasets/models. Nothing touches real-time market data feeds, LSEG data products, or anything resembling your employer’s commercial offering. Daily-frequency public data + standard models is the safe lane; stay in it. (Check the moonlighting/IP clause before publishing regardless — same as every project here.)

Phased Build Plan

Phase 5a — From-scratch inference (Weeks 41–44): Python training scripts for both demo models + parameter export. Rust: MLP forward pass, feature pipeline, golden tests against sklearn/PyTorch outputs. A minimal synchronous axum server serving the hand-rolled backend. (This phase is deliberately library-free — it’s the “implement Black-Scholes yourself” of ML.)

Phase 5b — Backends and batching (Weeks 45–48): trait InferenceBackend; ort integration (ONNX Runtime FFI); the dynamic batcher; model registry + hot load/unload; candle as the third backend; backend conformance test suite (all backends, same fixtures, same outputs within tolerance). The embedding endpoint lands here too: tokenizers integration, within-batch padding + attention masks, your own mean pooling and L2 normalization — variable-length inputs make the batcher earn its keep.

Phase 5c — Observability, benchmarks, publish (Weeks 49–52): Metrics endpoint, structured tracing, graceful shutdown (drain the batch queue on SIGTERM). Load testing with oha/k6. Benchmark vs a FastAPI + onnxruntime-python baseline serving the identical ONNX model on the same hardware — publish the latency/throughput plots. README, docs, CI/CD, release.

Technical Requirements

Required crates:

• tokio — runtime, channels, timers
• axum — HTTP API
• ort — ONNX Runtime bindings (the FFI lesson)
• candle-core / candle-nn — pure-Rust backend
• tokenizers — Hugging Face’s Rust-native tokenizer (for the embedding endpoint; pooling/normalization stays yours)
• serde + serde_json — config, requests, preprocessing params
• clap, tracing, criterion

NOT allowed:

• Any pre-built serving framework or batching library
• Metrics crates (prometheus, metrics) — render the exposition format yourself
• Any preprocessing/feature library — the pipeline is yours

Performance targets:

• p99 latency < 10ms for the credit model at 1,000 req/s on CPU (batched)
• Throughput ≥ 5x the FastAPI + onnxruntime-python baseline, same model, same hardware
• Batching adds < 1ms p50 overhead at low load (the max_wait_ms tradeoff, measured)

Expected Concepts Covered

Definition of Done

• From-scratch MLP forward pass matches PyTorch output within 1e-6 on 10+ fixtures
• Feature pipeline matches sklearn.transform() on fixture data (golden tests)
• ort, candle, and hand-rolled backends all pass the same conformance suite
• Dynamic batching demonstrably works: batch-size distribution shifts up under load (shown via /metrics during a load test)
• Hot model load/unload without dropping in-flight requests
• All three demo models served end-to-end: Python train → export → inferd serve → correct predictions
• Embedding endpoint: cosine similarity of inferd vectors vs Python sentence-transformers output > 0.999 on a fixture corpus; semantically similar sentences rank correctly
• Benchmark report vs FastAPI+onnxruntime baseline with plots, honest methodology
• Graceful shutdown drains the queue (proven by test)
• 30+ tests including the backend conformance suite
• Published (crates.io and/or release binaries) with CI/CD
• README good enough that a Python-only data scientist can train, export, and serve a model without reading the source

Real-World Value and Career Impact

Open-source impact: TorchServe’s archival left a genuine hole in the “simple, self-hosted model serving” space. The Rust inference ecosystem (ort, candle, burn, tract) is growing fast but is all libraries — there is no established lightweight server built on them. Even as a learning project, inferd is the kind of repo that gets stars because the README answers a question many teams have.

Career impact: this capstone is the program’s strongest bridge back to your actual profession. docforge/formulaengine/ruleforge make you “a data scientist who can also build infrastructure.” inferd makes you “an ML engineer who owns the full train-to-production path” — a rarer and better-paid profile. It also gives you informed opinions about every serving system you’ll ever evaluate at work, because you’ve built the core mechanisms (batching, registries, backends, observability) yourself.

Project Summary Table

ticknorm is retained for reference but replaced by airpost (COI with LSEG). xlsxfmt is retained for reference but replaced by dataprof (career fit). docforge is retired (depended on xlsxfmt; document-infra direction). See notices in each section.

Two options are provided for Phase 3A (4A/4B). Pick one — both are pure “from scratch” learning exercises that occupy the same 3-weekend slot.

Active capstone options: 8B (formulaengine), 8C (ruleforge), 8D (inferd). Choose one in week 40. B/C center on parsing and trait-based architecture; D centers on async serving, FFI, and ML systems. All include benchmarking, Python interop, and publishing.

ML/DL track: schemaguard drift (PSI/KS drift detection, week 14) · riskbook EWMA/GARCH(1,1) MLE + Monte Carlo VaR (weeks 34–35) · airpost forecasting endpoint (stretch) · inferd capstone (Option D)

Real-world problem solvers: 6 of 10 Finance/ML projects: riskbook + drift detection in schemaguard + inferd (finance demo models) Public good projects: 1 of 10 (airpost) Open-sourceable: 7 of 10 SaaS potential: 5 of 10