Skip to main content

Build pylon from scratch

Reader: someone with a fresh checkout on a clean macOS 14 / Ubuntu 22.04 box who wants a working build/pylon.wasm and a way to prove that the bytes on their disk match what the maintainers shipped.

The reproducibility contract this doc backs up:

Given the pinned commit of this repo and a fresh machine, the sequence below produces a build/pylon.wasm whose sha256 matches the pylon_sha256 recorded in caps-manifest.toml — bit-for-bit.

If your build's digest drifts, make verify prints exactly which cap's artifact diverged so you know where to look.

Overview​

Pylon = CPython 3.14 compiled to wasm32-wasip2 + N sibling WebAssembly capability components composed against it. Each cap lives in its own git repo (~/git/<name>-wasm/ by convention) with its own build. The authoritative index of caps + their pinned SHAs + artifact sha256s is caps-manifest.toml.

The build has four discrete stages:

  1. Install prereqs — one-time host toolchain setup.
  2. make caps-sync — clones + pins + builds every sibling cap.
  3. make build — cross-builds CPython, wires C-extension shims, emits deps/cpython-3.14/cross-build/wasm32-wasip2/python.wasm, then wac plugs every cap into it → build/pylon.wasm.
  4. make verify — recomputes sha256 of build/pylon.wasm + each cap artifact and diffs against caps-manifest.toml.

1. Prerequisites​

macOS 14 (Sonoma)​

# Base toolchains — wasmtime, wac, wasm-tools, cargo-component
brew install wasmtime pkg-config coreutils gnu-sed cmake ninja

# Rust with the wasm32 targets
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
rustup target add wasm32-wasip1 wasm32-wasip2

# Component-model tooling (versions must match, else WIT drift bites)
cargo install --locked wac-cli --version 0.10.0
cargo install --locked wit-bindgen-cli --version 0.57.1
cargo install --locked wasm-tools --version 1.216.0
cargo install --locked cargo-component --version 0.20.0

# Python 3.11+ for the wasi-build driver + manifest helpers
brew install python@3.12

Ubuntu 22.04​

sudo apt-get update
sudo apt-get install -y build-essential pkg-config cmake ninja-build \
    python3.12 python3.12-venv curl git

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
rustup target add wasm32-wasip1 wasm32-wasip2

cargo install --locked wac-cli --version 0.10.0
cargo install --locked wit-bindgen-cli --version 0.57.1
cargo install --locked wasm-tools --version 1.216.0
cargo install --locked cargo-component --version 0.20.0

# wasmtime (pin to a matching runtime — the WIT ABI moves fast)
curl -L https://github.com/bytecodealliance/wasmtime/releases/download/v46.0.1/wasmtime-v46.0.1-x86_64-linux.tar.xz \
    | tar -xJ && sudo mv wasmtime-v45.0.0-x86_64-linux/wasmtime /usr/local/bin/

Version pins matter: mismatches between wac, wit-bindgen, and wasm-tools show up as opaque compose-time errors ("resource types are not the same"). Prefer the versions above until a reproducibility bump.

2. Clone the repo​

mkdir -p ~/git && cd ~/git
git clone https://github.com/tegmentum/python-wasm.git
cd python-wasm

Sibling caps get cloned into ~/git/ by caps-sync — the repo layout is intentionally flat rather than nested so that shared caps (e.g. zlib-wasm) are single-sourced across every consumer.

3. make caps-sync​

make caps-sync

This reads caps-manifest.toml and, for each [[cap]] entry:

  1. Clones the sibling repo into ~/git/<name>/ if not present.
  2. git fetch origin + git checkout <sha> (idempotent).
  3. Builds the cap:
    • target = "wasm32-wasip1" → cargo build --release --target wasm32-wasip1, then wraps the resulting module into a component via wasm-tools component new --adapt.
    • target = "wasm32-wasip2" → cargo build --release --target wasm32-wasip2.
    • blank target → runs make if a Makefile is present (arrow-core, arrow-*-typed use custom Makefiles).
  4. Recomputes sha256 of the artifact and compares against the manifest. Any drift is FAIL'd loudly.

Expected runtime: ~15–30 min on a warm cargo cache, ~90 min cold.

Per-cap reporting:

  • OK — sha256 matches.
  • SKIP — manifest has placeholder 0000... sha (a sibling that wasn't pinned on the maintainer's dev machine; you'll need to clone
    • rev-parse manually and PR the pin).
  • FAIL — clone/build failed.
  • MISMATCH — the artifact was rebuilt but its bytes don't match the pinned sha256. This is the interesting case: either your toolchain differs from the maintainer's or a cap has been rebuilt at a non-reproducible timestamp.

Subset a run with CAPS_ONLY=zlib,zstd make caps-sync. Verify without rebuilding via CAPS_SKIP_BUILD=1 make caps-sync.

4. make build​

make build

This does the CPython cross-build. Roughly:

  • scripts/fetch-sdk.sh — downloads wasi-sdk 33 into deps/.
  • scripts/fetch-cpython.sh — clones CPython 3.14 into deps/cpython-3.14.
  • Builds openssl-wasm + zlib-wasm archives (for the static paths that aren't cap-backed on the current profile).
  • scripts/wire-cpython-ext.sh — symlinks every cpython-ext/* into the CPython tree and appends the cap C extensions to Modules/Setup.local.
  • python3 Tools/wasm/wasi build ... — actual CPython cross-build.
  • Second wire-cpython-ext.sh pass + make Modules/config.c + a targeted relink to bake cap imports into the wasm.
  • scripts/compose-python-component.sh — runs wac plug over python.wasm with every cap plug in ~/git/*/target/component/*.wasm, writes build/<profile>/pylon.wasm, symlinks build/pylon.wasm.

Cold: ~10 min on Apple Silicon, ~20 min on x86_64 Linux. Warm-cache incremental: ~30 seconds if you only touched a shim.

5. make verify​

make verify

Read-only. Prints:

caps OK:            74
caps placeholder:   1
caps missing:       0
caps drifted:       0

pylon_sha256 status: OK

Passes when:

  • Every non-placeholder cap's on-disk artifact sha256 matches the manifest.
  • build/pylon.wasm's sha256 matches pylon_sha256.

Fails when either drifts — the report lists exactly which caps mismatched so you know which sibling repo to re-sync (or which upstream change to attribute the drift to).

6. make smoke​

make smoke

Runs tests/*_smoke.py under pylon-launch — a sanity check that the composed wasm loads, imports its cap extensions, and exercises each capability against the sibling wasm. Green here + green verify = you're reproducing what upstream ships.

Troubleshooting​

caps-sync reports MISMATCH for one cap​

This is the most common surprise. Order of investigation:

  1. Did you build with a different cargo/rustc version? Check rustc --version — pin to the version listed in the pylon CI (.github/workflows/*.yml). Cargo's incremental output is reproducible per rustc version, not across.
  2. Does the cap's Cargo.lock differ from what was pinned? Fresh clones will get whatever is in the pinned SHA — if that SHA was built with a stale lockfile, cargo will now pick different transitive deps. Fix: rebuild the manifest against a fresh cap clone (python3 scripts/regen-caps-manifest.py).
  3. Did the cap's build embed a timestamp? Rare but possible; check strings cap.wasm | grep -E '20[0-9]{2}' — if a build date leaked in, pin a SOURCE_DATE_EPOCH env var and rebuild.

pylon_sha256 status: MISMATCH​

Every cap OK but the root drifted. Almost always a wac-cli version mismatch — the composed wasm's byte layout is very sensitive to it. Pin wac-cli 0.10.0 and rebuild.

A cap's sibling repo isn't at github.com/tegmentum​

Some caps (e.g. v86-posix) live under an org-owned name that differs from what the local clone dir is called. caps-manifest.toml records this via local_dir = "v86" on the [[cap]] entry.

For CI​

The four commands above are the CI recipe:

- run: make caps-sync CAPS_SKIP_BUILD=1  # download only, no cargo builds
- run: make build
- run: make verify
- run: make smoke

CAPS_SKIP_BUILD=1 keeps CI fast when the sibling artifacts are cached under ~/git/*/target/. For nightly reproducibility jobs, drop CAPS_SKIP_BUILD so every cap is rebuilt from source.