Introduction

Built with AI. Murmer is built and maintained with the help of AI coding tools, primarily Claude. I drive the design and review every change, but a lot of the implementation, tests, and docs are written in collaboration with AI. I’d rather you know that going in.

Murmer is a distributed actor framework for Rust, built on tokio and QUIC.

It provides typed, location-transparent actors that communicate through message passing. Whether an actor lives in the same process or on a remote node across the network, you interact with it through the same Endpoint<A> API.

Murmer in 1 minute

Install:

[dependencies]
murmer = "0.4"
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["full"] }

Write an actor, send it messages:

use murmer::prelude::*;

// ① Define your actor — state lives separately
#[derive(Debug)]
struct Counter;
struct CounterState { count: i64 }

impl Actor for Counter {
    type State = CounterState;
}

// ② Handlers become the actor's API
#[handlers]
impl Counter {
    #[handler]
    fn increment(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut CounterState,
        amount: i64,
    ) -> i64 {
        state.count += amount;
        state.count
    }

    #[handler]
    fn get_count(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut CounterState,
    ) -> i64 {
        state.count
    }
}

#[tokio::main]
async fn main() {
    // ③ Create a local actor system
    let system = System::local();

    // ④ Start an actor — returns a typed Endpoint<Counter>
    let counter = system.start("counter/main", Counter, CounterState { count: 0 });

    // ⑤ Send messages via auto-generated extension methods
    let result = counter.increment(5).await.unwrap();
    println!("Count: {result}"); // → Count: 5

    // ⑥ Look up actors by label — works for local and remote
    let found = system.lookup::<Counter>("counter/main").unwrap();
    let count = found.get_count().await.unwrap();
    println!("Looked up: {count}"); // → Looked up: 5
}

cargo run

That’s it — a complete, working actor system. The rest of this page explains what’s happening under the hood. The Getting Started chapter goes deeper into each component.

What it gives you

• Send messages without caring where the actor lives. counter.increment(5) (line ⑤) works identically whether the actor is local or on a remote node — the Endpoint<A> API abstracts the difference away.
• Test distributed systems from a single process. System::local() (line ③) runs everything in-memory. Swap to System::clustered() when you’re ready for real networking — your actor code stays identical.
• Define actors with minimal boilerplate. The #[handlers] macro (line ②) auto-generates message structs (Increment, GetCount), dispatch tables, serialization, and the extension methods you call on line ⑤.
• Get networking and encryption handled for you. QUIC transport with automatic TLS, SWIM-based cluster membership, and mDNS discovery — all configured, not hand-rolled.
• Supervise actors like OTP. Restart policies (Temporary, Transient, Permanent) with configurable limits and exponential backoff keep your system running through failures.
• Orchestrate applications across a cluster. The app module adds placement strategies, leader election, and crash recovery — so you can declare what should run and where, and the framework handles the rest.

What’s happening: line by line

① Actor + State — Counter is a zero-sized struct. All mutable state lives in CounterState, passed as &mut to every handler. This keeps the actor lightweight and the state threading explicit.

② #[handlers] — The macro reads your method signatures and generates:

• Message structs — Increment { pub amount: i64 } and GetCount (unit struct)
• Handler<Increment> and Handler<GetCount> trait implementations
• RemoteDispatch — a wire-format dispatch table so remote nodes know how to route messages to the right handler
• CounterExt — an extension trait on Endpoint<Counter> that gives you .increment(amount) and .get_count() methods

③ System::local() — Creates the actor runtime and boots the Receptionist — the internal actor registry that tracks all actors by label and type.

④ system.start(...) — Wraps Counter in a Supervisor that manages its lifecycle, mailbox, and restart behavior. Registers it with the Receptionist under the label "counter/main". Returns an Endpoint<Counter> — your typed send handle.

⑤ counter.increment(5) — The extension method constructs an Increment { amount: 5 } message and sends it through the Endpoint. Since this is a local actor, the message is dispatched as a zero-copy envelope through the Supervisor’s mailbox to the handler. The result comes back through a oneshot channel.

⑥ system.lookup(...) — Queries the Receptionist for an actor of type Counter at label "counter/main". Returns the same Endpoint<Counter>. In a clustered system, this could return a proxy endpoint that transparently serializes messages over QUIC to a remote node.

Core concepts

Architecture

Every layer in this diagram is touched by the example code:

• System — created at line ③, runs the entire runtime
• Receptionist — populated at line ④ (start), queried at line ⑥ (lookup)
• Supervisor Layer — wraps Counter at line ④, manages its mailbox, would handle restarts if configured
• Endpoint‹A› — returned at line ④, used at line ⑤ to send messages; local dispatch here, but swap to System::clustered() and the same endpoint transparently routes over QUIC
• Cluster Layer — not active in System::local(), but requires zero code changes to enable (see Clustering)

Key design decisions

• Endpoint<A> is the only API — callers never know if an actor is local or remote.
• Receptionist is non-generic — stores type-erased entries internally, uses TypeId guards for safe downcasts at lookup time.
• Supervisors are flat — each actor has its own supervisor, no parent-child hierarchy.
• Labels are paths — "cache/user", "worker/0", "thumbnail/processor/3". Hierarchical naming for organizational clarity.
• Fail-fast networking — if a QUIC stream fails, all pending responses error immediately instead of hanging.

From primitives to applications

Murmer works at two levels:

The core (Actors, Discovery, Supervision, Clustering) gives you the building blocks — everything you saw in the example above. You can build complete services with just these primitives.

The application layer (Application Orchestration) builds on top of the core to manage real, running applications across a cluster. You declare what actors should run, where they should be placed (with constraints like “must have GPU” or “must be a Worker node”), and what happens when a node fails — and the Coordinator handles placement, spawning, and crash recovery automatically.

Other libraries to consider

Murmer is still v0 and an experiment. For anything real, you should probably consider using one of these instead. They inspired Murmer and most of them are far more mature:

• Erlang/Elixir OTP — the actor model and supervision tree that started it all for me. If you can run on the BEAM, OTP is decades of battle-tested production experience.
• Apple Swift Distributed Actors — the typed, location-transparent endpoint design that shaped Murmer’s API. Worth reading even if you’re not writing Swift.
• Actix — the most established actor framework in Rust. Mature, fast, and great for in-process actors.
• ractor — a newer Rust actor library inspired by Erlang, with a clustering story of its own. Well worth a look if you’re comparing options.

Learn more

• Getting Started — deeper walkthrough of each component
• API Reference on docs.rs
• Source on GitHub

Getting Started

This chapter goes deeper into the components you saw in the introduction. If you haven’t seen the 1-minute example yet, start there — this chapter assumes you’ve seen the basics and want to understand more.

Dependencies

[dependencies]
murmer = "0.4"
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["full"] }

murmer is the core framework. The macros feature (on by default) re-exports #[handlers], #[handler], and #[derive(Message)] from murmer-macros — no separate dependency needed. Both serde and tokio are required — serde for message serialization (even in local mode, the types need to be Serialize + Deserialize for remote readiness), and tokio as the async runtime.

The Actor trait

Every actor implements the Actor trait, which has one required associated type: State.

use murmer::prelude::*;

#[derive(Debug)]
struct ChatRoom;

struct ChatRoomState {
    room_name: String,
    messages: Vec<ChatEntry>,
    max_messages: usize,
}

struct ChatEntry {
    from: String,
    text: String,
    timestamp: u64,
}

impl Actor for ChatRoom {
    type State = ChatRoomState;
}

Why state lives separately

This is a deliberate design choice. In many actor frameworks, state lives directly on the actor struct. In murmer, the actor struct is typically empty (zero-sized) and all mutable state lives in the associated State type.

This gives you:

• Explicit state threading — every handler receives &mut State, making it clear what data is being read and modified.
• Clean restarts — when a supervisor restarts an actor, the factory creates a fresh (Actor, State) pair. No hidden state carried over from a crashed instance.
• Separation of identity and data — the actor struct can carry configuration or immutable context (like a database pool handle), while State holds the mutable per-instance data.

You can put fields on the actor struct — they just won’t be part of the restart cycle:

struct ChatRoom {
    db: DatabasePool,  // immutable, shared across restarts
}

struct ChatRoomState {
    messages: Vec<ChatEntry>,  // mutable, reset on restart
}

Defining handlers

Handlers are methods on the actor that process incoming messages. The #[handlers] macro on the impl block and #[handler] on individual methods does the heavy lifting:

#[handlers]
impl ChatRoom {
    #[handler]
    fn post_message(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
        from: String,
        text: String,
    ) -> usize {
        state.messages.push(ChatEntry {
            from,
            text,
            timestamp: now(),
        });
        // Trim if over limit
        if state.messages.len() > state.max_messages {
            state.messages.remove(0);
        }
        state.messages.len()
    }

    #[handler]
    fn get_history(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
    ) -> Vec<String> {
        state.messages.iter()
            .map(|e| format!("[{}] {}: {}", e.timestamp, e.from, e.text))
            .collect()
    }

    #[handler]
    fn room_name(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
    ) -> String {
        state.room_name.clone()
    }
}

Handler signature rules

Every handler method must follow this pattern:

fn method_name(
    &mut self,                          // always &mut self
    ctx: &ActorContext<Self>,           // always second — the actor's context
    state: &mut YourStateType,         // always third — mutable state access
    // ... additional parameters become message fields
) -> ReturnType {
    // ...
}

• &mut self — the actor instance.
• ctx: &ActorContext<Self> — provides access to the system, receptionist, the actor’s own label, and methods like ctx.watch() for actor monitoring. Prefix with _ if unused.
• state: &mut State — the actor’s mutable state.
• Additional parameters — each one becomes a field on the generated message struct. fn increment(... amount: i64) generates Increment { pub amount: i64 }.
• Return type — becomes the message’s Result type. Handlers must return a value (not ()).

What gets generated

For the ChatRoom example above, the macro generates:

// Message structs
struct PostMessage { pub from: String, pub text: String }
struct GetHistory;  // unit struct — no extra params
struct RoomName;

// Trait implementations
impl Handler<PostMessage> for ChatRoom { /* ... */ }
impl Handler<GetHistory> for ChatRoom { /* ... */ }
impl Handler<RoomName> for ChatRoom { /* ... */ }

// Remote dispatch table
impl RemoteDispatch for ChatRoom { /* ... */ }

// Extension trait for ergonomic sends
trait ChatRoomExt {
    fn post_message(&self, from: String, text: String) -> impl Future<Output = Result<usize>>;
    fn get_history(&self) -> impl Future<Output = Result<Vec<String>>>;
    fn room_name(&self) -> impl Future<Output = Result<String>>;
}

impl ChatRoomExt for Endpoint<ChatRoom> { /* ... */ }

This means you can call endpoint.post_message("alice".into(), "hello".into()) directly on any Endpoint<ChatRoom>, without ever constructing a message struct yourself.

Async handlers

For handlers that need to perform async work (I/O, database queries, HTTP calls), use async fn:

#[handlers]
impl ChatRoom {
    #[handler]
    async fn fetch_and_store(
        &mut self,
        ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
        url: String,
    ) -> Result<usize, String> {
        // You can .await here — the supervisor handles scheduling
        let response = reqwest::get(&url).await
            .map_err(|e| e.to_string())?;
        let body = response.text().await
            .map_err(|e| e.to_string())?;

        state.messages.push(ChatEntry {
            from: "system".into(),
            text: body,
            timestamp: now(),
        });
        Ok(state.messages.len())
    }
}

Async handlers generate AsyncHandler<FetchAndStore> instead of Handler<FetchAndStore>. The supervisor processes async handlers cooperatively — while one handler is awaiting, no other messages are processed for that actor (preserving the single-writer invariant).

Explicit message types

The auto-generated messages from #[handlers] cover most cases, but sometimes you want to define a message type explicitly — for example, when multiple actors handle the same message:

use murmer::Message;
use serde::{Serialize, Deserialize};

#[derive(Debug, Clone, Serialize, Deserialize, Message)]
#[message(result = String)]
struct Ping {
    payload: String,
}

To use an explicit message in a handler, name the parameter msg:

#[handlers]
impl ChatRoom {
    #[handler]
    fn ping(
        &mut self,
        _ctx: &ActorContext<Self>,
        _state: &mut ChatRoomState,
        msg: Ping,          // "msg" signals: use this type directly
    ) -> String {
        format!("pong: {}", msg.payload)
    }
}

The msg parameter name tells the macro to use Ping as-is instead of generating a new message struct.

For messages that need to cross the network, add remote:

#[derive(Debug, Clone, Serialize, Deserialize, Message)]
#[message(result = String, remote = "my_app::Ping")]
struct Ping {
    payload: String,
}

The remote value is a unique string identifier used for wire-format routing. See the Proc Macro Reference for the full details.

The System

The System is the entry point for everything — it runs the actor runtime, manages the receptionist, and optionally handles clustering.

// Local mode — in-memory, no networking
let system = System::local();

// Clustered mode — QUIC networking, SWIM membership
let system = System::clustered_auto(config).await?;

Both modes expose the same API. Your actor code doesn’t change.

Starting actors

// Returns Endpoint<ChatRoom>
let room = system.start(
    "room/general",                   // label — unique within the cluster
    ChatRoom,                         // actor instance
    ChatRoomState {                   // initial state
        room_name: "general".into(),
        messages: vec![],
        max_messages: 1000,
    },
);

The label "room/general" is a path-like string that uniquely identifies this actor in the system. Labels are how actors find each other through the receptionist.

Looking up actors

// Type-safe lookup — returns Option<Endpoint<ChatRoom>>
let room = system.lookup::<ChatRoom>("room/general");

if let Some(ep) = room {
    let history = ep.get_history().await?;
}

Lookups are type-checked at compile time. If the label exists but the type doesn’t match, None is returned.

Endpoints in depth

Endpoint<A> is the central abstraction. It’s:

• Typed — Endpoint<ChatRoom> can only send messages that ChatRoom handles
• Cloneable — lightweight handle, share freely across tasks
• Location-transparent — local endpoints dispatch through in-memory channels; remote endpoints serialize over QUIC

// All of these work identically
let room: Endpoint<ChatRoom> = system.start("room/1", ChatRoom, state);
let room: Endpoint<ChatRoom> = system.lookup::<ChatRoom>("room/1").unwrap();

// Clone and pass to another task
let room2 = room.clone();
tokio::spawn(async move {
    room2.post_message("bot".into(), "background task".into()).await.unwrap();
});

// Send via extension methods (ergonomic)
room.post_message("alice".into(), "hello".into()).await?;

// Or send a message struct directly
room.send(PostMessage { from: "alice".into(), text: "hello".into() }).await?;

Both .post_message(...) (extension method) and .send(PostMessage { ... }) (direct) do the same thing. The extension methods are more ergonomic for the common case.

Actor watches

Monitor other actors and get notified when they terminate — inspired by Erlang’s monitor/2:

struct Watchdog;
struct WatchdogState { terminated: Vec<String> }

impl Actor for Watchdog {
    type State = WatchdogState;

    fn on_actor_terminated(
        &mut self,
        state: &mut WatchdogState,
        terminated: &ActorTerminated,
    ) {
        tracing::warn!(
            "Actor {} terminated: {:?}",
            terminated.label,
            terminated.reason
        );
        state.terminated.push(terminated.label.clone());
    }
}

#[handlers]
impl Watchdog {
    #[handler]
    fn watch(
        &mut self,
        ctx: &ActorContext<Self>,
        _state: &mut WatchdogState,
        label: String,
    ) -> bool {
        ctx.watch(&label);
        true
    }
}

The on_actor_terminated callback on the Actor trait fires when any watched actor stops, crashes, or is killed. The ActorTerminated struct tells you which actor and why.

Going from local to clustered

The entire point of murmer’s design is that this transition requires zero changes to your actor code. Only the system construction changes:

// Before: local
let system = System::local();

// After: clustered
let config = ClusterConfig::builder()
    .name("my-node")
    .listen("0.0.0.0:7100".parse()?)
    .cookie("my-cluster-secret")
    // Seeds are dialed by key. Run `murmer id` on the seed node to get its
    // endpoint id, then build its address from the id plus host:port.
    .seed_nodes([iroh::EndpointAddr::from_parts(
        seed_endpoint_id,
        [iroh::TransportAddr::Ip("192.168.1.1:7100".parse()?)],
    )])
    .build()?;

let system = System::clustered_auto(config).await?;

Everything else — system.start(...), system.lookup(...), endpoint.send(...) — stays identical. Actors on remote nodes appear in your local receptionist automatically via registry replication.

A node is now identified by a persistent key (its iroh endpoint id), not by its address. See the Clustering chapter for the full walkthrough, and Administration & Security for key management and the cluster allowlist.

Build and test

cargo build
cargo nextest run
cargo clippy -- -D warnings

Next steps

Now that you understand the components, dive into the specific areas:

• Actors and Messages — the complete actor model: state, handlers, endpoints, location transparency
• Discovery — labels, reception keys, listings, and routing
• Supervision — restart policies, backoff, actor factories
• Clustering — QUIC networking, SWIM membership, multi-node deployment
• Proc Macro Reference — everything #[handlers] and #[derive(Message)] generate
• Application Orchestration — placement, leader election, crash recovery

Actors and Messages

This chapter covers the core building blocks: actors, state, messages, handlers, and endpoints.

Actors

An actor in murmer is a small server defined as a Rust struct with a set of handler methods, persisting state, and message types. You define an actor by implementing the Actor trait, then implementing Handler methods for each message type it can handle.

use murmer::prelude::*;

#[derive(Debug)]
struct ChatRoom;

struct ChatRoomState {
    room_name: String,
    messages: Vec<ChatEntry>,
}

impl Actor for ChatRoom {
    type State = ChatRoomState;
}

Key points:

• The actor struct itself is typically a zero-sized type (no fields). All mutable state lives in the associated State type.
• State is passed as &mut State to every handler, keeping the actor struct itself lightweight and the state explicitly threaded.
• Each actor runs inside a supervisor that manages its lifecycle, mailbox, and restart behavior.

Messages

A message is a type that can be sent to an actor. Every message defines a result type and can optionally be serializable for remote delivery.

There are two ways to define messages:

Auto-generated messages (recommended)

When using #[handlers] with #[handler], the macro generates message structs automatically from your method signatures:

#[handlers]
impl ChatRoom {
    #[handler]
    fn post_message(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
        from: String,
        text: String,
    ) -> usize {
        state.messages.push(ChatEntry { from, text });
        state.messages.len()
    }

    #[handler]
    fn get_history(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut ChatRoomState,
    ) -> Vec<String> {
        state.messages.iter()
            .map(|e| format!("{}: {}", e.from, e.text))
            .collect()
    }
}

This generates PostMessage { pub from: String, pub text: String } and GetHistory unit struct, plus all the trait implementations and an extension trait ChatRoomExt on Endpoint<ChatRoom>.

Explicit messages

For messages shared across multiple actors, define them manually with #[derive(Message)]:

use murmer::Message;
use serde::{Serialize, Deserialize};

#[derive(Debug, Clone, Serialize, Deserialize, Message)]
#[message(result = Vec<String>, remote = "orchestrator::ListDir")]
struct ListDir {
    path: String,
}

Then reference it in a handler with the msg parameter name:

#[handlers]
impl StorageAgent {
    #[handler]
    fn list_dir(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut StorageState,
        msg: ListDir,
    ) -> Vec<String> {
        state.dirs.get(&msg.path).cloned().unwrap_or_default()
    }
}

The remote = "..." attribute is optional — omit it for local-only messages that don’t need wire serialization.

Async handlers

Handlers that need to await use async fn:

#[handlers]
impl MyActor {
    #[handler]
    async fn fetch_data(
        &mut self,
        ctx: &ActorContext<Self>,
        state: &mut MyState,
        url: String,
    ) -> Vec<u8> {
        some_async_call(&url).await
    }
}

This generates an AsyncHandler<FetchData> implementation instead of Handler<FetchData>.

Endpoints

Endpoints are opaque handles to actors used to send messages and receive responses. The Endpoint<A> type abstracts where the actor lives — whether local or remote — and handles serialization in the background.

// Start returns a typed endpoint
let counter = system.start("counter/main", Counter, CounterState { count: 0 });

// Send via auto-generated extension trait
let result = counter.increment(5).await.unwrap();

// Or send a message struct directly
let result = counter.send(Increment { amount: 5 }).await.unwrap();

Location transparency

The key design principle: Endpoint<A> hides whether the actor is local or remote.

• Local actors use the envelope pattern — zero serialization cost, direct in-memory dispatch through a type-erased trait object.
• Remote actors serialize messages with bincode, send them over QUIC streams, and deserialize responses on return.

The caller’s code is identical in both cases:

let result = endpoint.send(Increment { amount: 5 }).await?;

How endpoints work under the hood

Under the hood, an endpoint wraps a tokio channel that sends messages to the actor’s supervisor:

• Local endpoint: The channel connects directly to the actor’s supervisor (mpsc). Messages are dispatched as type-erased envelopes with zero serialization cost.
• Remote endpoint: The channel connects to a proxy supervisor that serializes the message in bincode format, sends it through a QUIC stream, awaits the response, deserializes it, and returns it to the caller.

Endpoints are lightweight and Clone — share them freely across tasks.

Actor watches

Erlang-style actor monitoring — get notified when a watched actor terminates:

impl Actor for Monitor {
    type State = MonitorState;

    fn on_actor_terminated(
        &mut self,
        state: &mut MonitorState,
        terminated: &ActorTerminated,
    ) {
        match &terminated.reason {
            TerminationReason::Panicked(msg) => {
                tracing::error!("{} panicked: {}", terminated.label, msg);
            }
            _ => {}
        }
    }
}

#[handlers]
impl Monitor {
    #[handler]
    fn watch_actor(
        &mut self,
        ctx: &ActorContext<Self>,
        _state: &mut MonitorState,
        label: String,
    ) {
        ctx.watch(&label);
    }
}

The ActorContext provides the watch() method, and termination notifications arrive via the on_actor_terminated callback on the Actor trait.

Discovery

Murmer provides a unified actor discovery system through the Receptionist — a type-erased registry that handles registration, lookup, and subscription for both local and remote actors.

Labels

Actors are identified by path-like labels: "cache/user", "worker/0", "thumbnail/processor/3".

let ep = system.start("service/auth", AuthActor, AuthState::new());
let ep = system.lookup::<AuthActor>("service/auth").unwrap();

Labels serve as the primary routing key in the system:

• No two actors can have the same label in the clustered system.
• The path structure ("group/subgroup/instance") enables hierarchical organization of actors.
• Wildcards can be used in subscriptions on the receptionist (e.g., subscribe to all actors under "worker/*").
• Labels are only used for actor discovery and routing — they are a feature of the receptionist, not intrinsic to actors themselves.

The Receptionist

The Receptionist is a special internal actor started with every actor system. It manages the lookup and registration of both local and remote actors.

When an actor is started, it registers itself with the receptionist. The receptionist maintains a mapping of labels and actor types to their corresponding endpoints. When you want to communicate with an actor, you query the receptionist with the actor type and label to receive an endpoint.

Typed lookup

Lookups are type-safe — you specify the actor type and get back a typed endpoint:

// Returns Option<Endpoint<AuthActor>>
let endpoint = system.lookup::<AuthActor>("service/auth");

If the label doesn’t exist or the type doesn’t match, None is returned. For remote actors, the receptionist returns an endpoint that transparently handles connection management when accessed.

State observation

When an actor is registered, the receptionist becomes an observer of its state. If the actor stops or crashes, the receptionist is notified and removes the actor from its registry. This ensures the receptionist always has an up-to-date view of the actors in the system.

If an actor restarts (via supervision), the receptionist is notified but the actor remains available — the state is non-negative (alive or restarting, not dead).

Reception keys and listings

Group actors by type and subscribe to changes:

let worker_key = ReceptionKey::<Worker>::new("workers");

// Check actors into the group
receptionist.check_in("worker/0", worker_key.clone());
receptionist.check_in("worker/1", worker_key.clone());

// Subscribe — get existing actors immediately + live updates
let mut listing = receptionist.listing(worker_key);
while let Some(endpoint) = listing.next().await {
    endpoint.send(Work { task: "process".into() }).await?;
}

A Listing<A> is an async stream that yields endpoints as actors register and deregister against a ReceptionKey. It provides both backfill (existing actors) and live updates (new registrations), making it ideal for dynamic pool management.

Lifecycle events

Subscribe to all actor registrations and deregistrations across the system:

let mut events = receptionist.subscribe_events();
while let Some(event) = events.recv().await {
    match event {
        ActorEvent::Registered { label, actor_type } => { /* ... */ }
        ActorEvent::Deregistered { label, actor_type } => { /* ... */ }
    }
}

Other actors can subscribe to the receptionist to receive notifications about actors being added, removed, or updated. Subscriptions can be broad (all actor updates, useful for clustering) or specific (a particular actor type or label).

Routing

Distribute messages across actor pools:

let router = Router::new(
    vec![ep1, ep2, ep3],
    RoutingStrategy::RoundRobin,
);

// Each send goes to the next endpoint in sequence
router.send(Increment { amount: 1 }).await?;

// Or broadcast to all
let results = router.broadcast(GetCount).await;

The Router<A> takes a set of endpoints and a RoutingStrategy to distribute messages. Current strategies include round-robin and broadcast.

How discovery works across nodes

When running in clustered mode, the receptionist automatically synchronizes actor registrations across nodes:

1. A local actor registers with its node’s receptionist.
2. The node broadcasts an ActorAdd notification to all connected peers.
3. Remote nodes register the actor in their local receptionists with a lazy endpoint factory.
4. When a client looks up the remote actor, the endpoint factory creates a proxy that handles network transport.
5. If the actor’s node fails, all remote registrations are cleaned up automatically.

This means system.lookup::<MyActor>("some/label") works identically whether the actor is local or on a remote node — the receptionist handles the difference transparently.

Supervision

Every actor in murmer runs inside a supervisor. The supervisor manages the actor’s lifecycle — starting it, processing its mailbox, and restarting it when things go wrong. This model is directly inspired by Erlang/OTP’s supervision trees, adapted for Rust’s ownership and type system.

How supervisors work

Each actor gets its own supervisor. The supervisor is responsible for:

• Starting the actor and registering it with the receptionist.
• Mailbox processing — ingesting messages and passing them to the actor’s handlers in order of arrival.
• Crash detection — catching panics and deciding what to do next based on the restart policy.
• Restarting the actor using a factory if the policy allows it.
• State notifications — informing the receptionist of state changes (started, stopped, dead, etc.).
• Context — providing the actor with access to the system, receptionist, and other actors via ActorContext.

Supervisors are flat — there is no parent-child hierarchy between actors. Each actor is independent and can be stopped or restarted without affecting others.

Actor lifecycle

The supervisor manages an actor through a well-defined set of states:

Restart policies

Actors can be started with restart policies that control behavior on failure:

• Temporary — the actor runs once. If it panics or stops, it’s gone. This is the default.
• Transient — the actor restarts if it panics, but a clean shutdown is respected. Use this for actors that should survive crashes but can be intentionally stopped.
• Permanent — the actor always restarts, whether it panicked or stopped cleanly. Use this for critical services that must always be running.

Configuration

To use restart policies, you provide an ActorFactory (which knows how to create fresh instances) and a RestartConfig:

use murmer::{RestartPolicy, RestartConfig, BackoffConfig, ActorFactory};
use std::time::Duration;

struct MyFactory;
impl ActorFactory for MyFactory {
    type Actor = Counter;
    fn create(&mut self) -> (Counter, CounterState) {
        (Counter, CounterState { count: 0 })
    }
}

let endpoint = receptionist.start_with_config(
    "counter/resilient",
    MyFactory,
    RestartConfig {
        policy: RestartPolicy::Permanent,  // Always restart
        max_restarts: 5,                   // Max 5 restarts...
        window: Duration::from_secs(60),   // ...within 60 seconds
        backoff: BackoffConfig {
            initial: Duration::from_millis(100),
            max: Duration::from_secs(30),
            multiplier: 2.0,
        },
    },
);

Restart limits

The max_restarts and window fields prevent infinite restart loops. If the actor exceeds the restart limit within the time window, the supervisor gives up and the actor is permanently stopped. This prevents a persistent bug from consuming all your resources.

Exponential backoff

The BackoffConfig controls the delay between restarts:

• initial — delay before the first restart attempt.
• max — maximum delay (the backoff caps here).
• multiplier — each subsequent restart delay is multiplied by this factor.

For example, with initial: 100ms, max: 30s, multiplier: 2.0, restarts happen at 100ms, 200ms, 400ms, 800ms, … up to 30s.

Actor factories

The ActorFactory trait gives the supervisor a way to create fresh actor instances for restarts:

trait ActorFactory {
    type Actor: Actor;
    fn create(&mut self) -> (Self::Actor, <Self::Actor as Actor>::State);
}

The factory is called each time the supervisor needs a new instance. It can carry its own state if needed — for example, incrementing a generation counter or loading configuration from disk.

Interaction with the receptionist

When a supervised actor restarts:

1. The old actor instance is dropped.
2. The supervisor creates a new instance via the factory.
3. The new instance is registered with the receptionist under the same label.
4. Any actors watching the old instance receive a termination notification, but the label remains routable.

This means endpoints held by other actors remain valid through restarts — messages sent during the brief restart window are queued in the supervisor’s mailbox and delivered to the new instance.

Deregistration on termination

When a supervisor exits, whatever the reason, a DeregisterGuard removes the actor from the receptionist and fires any watches. That guard runs in Drop, so in normal operation deregistration is instantaneous. There is no moment where an actor has stopped serving messages but is still discoverable.

Real deployments are messier. An actor can accept its stop and then take time to drain a mailbox or tear down an external resource, staying registered but not serving for a while. Code that waits on deregistration has to tolerate that window, and the tolerance is hard to test when the window is zero-width by construction.

Receptionist::set_terminate_hook opens it:

receptionist.set_terminate_hook(Some(Arc::new(my_hook)));  // None clears it

The hook is awaited between the supervisor exiting and the guard firing. While it is pending, lookup still finds the actor, listings still include it, and its watches have not fired. It returns a future rather than taking a duration, so under simulation you build the delay from the runtime seam and the window runs on virtual time, reproducible from the seed.

It is a fault-injection seam. Do not do real work in it. It runs on the supervisor’s task and holds the registry entry for as long as it stays pending, so a hook that never completes leaks the entry. It fires for every actor terminating on the node, including murmer’s internal ones, so a hook aimed at one actor has to filter on the label it is handed. It does not fire on the restart-limit-exceeded path, where the receptionist deregisters directly instead of through a supervisor exit.

See Extending Simulation for a worked example that holds one named actor in that state for five seconds of virtual time.

Watching actors

Any actor can monitor another actor for termination using ctx.watch(). When the watched actor terminates (for any reason), the watcher receives an ActorTerminated notification via on_actor_terminated.

impl Actor for Supervisor {
    type State = SupervisorState;

    fn on_actor_terminated(&mut self, state: &mut SupervisorState, event: &ActorTerminated) {
        tracing::warn!("Actor {} terminated: {:?}", event.label, event.reason);
    }
}

#[handlers]
impl Supervisor {
    #[handler]
    fn start_worker(&mut self, ctx: &ActorContext<Self>, state: &mut SupervisorState) {
        ctx.receptionist().start("worker/0", MyWorker, WorkerState::default());
        ctx.watch("worker/0");
    }
}

Watches are one-shot: they fire once when the watched actor terminates and are not re-armed.

Erlang semantics: If ctx.watch() is called for an actor that doesn’t exist, on_actor_terminated fires immediately. This avoids the race condition where the actor dies between the lookup and the watch.

Tagged watches

When a supervisor manages multiple children with different roles, parsing label strings in on_actor_terminated is fragile. Use ctx.watch_with_tag() to attach a role tag that is delivered alongside the termination notification:

fn start_children(&mut self, ctx: &ActorContext<Self>, state: &mut SupervisorState) {
    let r = ctx.receptionist();

    state.writer = Some(r.start("writer/0", WriterActor, WriterState::default()));
    ctx.watch_with_tag("writer/0", "writer");

    state.reader = Some(r.start("reader/0", ReaderActor, ReaderState::default()));
    ctx.watch_with_tag("reader/0", "reader");
}

fn on_actor_terminated(&mut self, state: &mut SupervisorState, event: &ActorTerminated) {
    match event.tag.as_deref() {
        Some("writer") => {
            tracing::error!("writer died — restarting");
            state.writer = None;
        }
        Some("reader") => {
            tracing::warn!("reader died — clearing slot");
            state.reader = None;
        }
        _ => {}
    }
}

Plain ctx.watch() delivers ActorTerminated with tag: None — fully backward compatible.

Scheduling

Actors can send delayed or periodic messages to themselves using the scheduling API. The returned ScheduleHandle auto-cancels when dropped.

schedule_once

Send a message to the actor after a delay:

fn handle(&self, ctx: &ActorContext<Self>, state: &mut MyState, _msg: StartTimeout) -> () {
    state.timeout = Some(ctx.schedule_once(Duration::from_secs(30), TimeoutExpired));
    // Drop state.timeout to cancel before it fires.
}

schedule_repeat

Send a message to the actor on a repeating interval:

fn on_start(&self, ctx: &ActorContext<Self>, state: &mut MyState) {
    state.heartbeat = Some(ctx.schedule_repeat(Duration::from_secs(60), Heartbeat));
}

The first tick fires after one full interval elapses (not immediately). Missed ticks are skipped — if the actor is slow, it receives one tick per interval boundary rather than a burst of catch-up ticks.

ScheduleHandle

ScheduleHandle is returned by both scheduling methods. Hold it in actor state to keep the schedule alive. Drop it (or call .cancel()) to stop the timer:

struct MyState {
    maintenance: Option<ScheduleHandle>, // Some = running, None = cancelled
}

// Cancel from a handler:
fn handle(&self, _ctx: &ActorContext<Self>, state: &mut MyState, _msg: StopMaintenance) -> () {
    state.maintenance = None; // drop cancels the timer
}

When the actor stops, all ScheduleHandles in its state are dropped and their timers are cancelled automatically.

Clustering

One of murmer’s core design goals is that your actor code doesn’t change when you go from a single process to a multi-node cluster. The same Endpoint<A> API works in both cases.

Step 1: Run everything locally

Create a System::local() — no networking, no config. Your actors communicate through in-memory channels with zero serialization cost:

use murmer::prelude::*;

let system = System::local();

let room = system.start("room/general", ChatRoom, ChatRoomState {
    room_name: "general".into(),
    messages: vec![],
});

// Send messages via extension trait — works instantly
room.post_message("alice".into(), "Hello!".into()).await?;

// Look up actors by label
let ep = system.lookup::<ChatRoom>("room/general").unwrap();
let history = ep.get_history().await?;

Step 2: Go distributed

When you’re ready for real networking, swap System::local() for System::clustered(). Your actor code stays identical — only the system construction changes:

use murmer::prelude::*;
use murmer::cluster::config::ClusterConfig;

let config = ClusterConfig::builder()
    .name("alpha")
    .listen("0.0.0.0:7100".parse()?)
    .advertise("192.168.1.5:7100".parse()?)
    .cookie("my-cluster-secret")
    // Seeds are dialed by endpoint id, not bare address. Get the seed's id by
    // running `murmer id` on it (or read the `seed:` line it prints at startup).
    .seed_nodes([iroh::EndpointAddr::from_parts(
        seed_endpoint_id,
        [iroh::TransportAddr::Ip("192.168.1.1:7100".parse()?)],
    )])
    .build()?;

// clustered_auto() discovers all #[handlers]-annotated actor types automatically
let system = System::clustered_auto(config).await?;

// Same API as local — start, lookup, send
let room = system.start("room/alpha", ChatRoom, state);
room.post_message("alice".into(), "Hello!".into()).await?;

// Actors on other nodes appear automatically via registry replication
let remote_room = system.lookup::<ChatRoom>("room/beta").unwrap();
remote_room.get_history().await?;  // transparently serialized over QUIC

Each node gets a single QUIC connection to every peer, multiplexed over per-actor streams. The OpLog replication protocol uses version vectors for efficient, idempotent sync.

Step 3: Test it interactively

The cluster_chat example lets you try both modes with an interactive CLI:

# Local mode — all actors in one process
cargo run -p murmer-examples --bin cluster_chat -- --local

=== murmer cluster_chat (local mode) ===
  Started room: #general
  Started room: #random

> post general alice Hello everyone!
  [1 messages in #general]
> post general bob Hey alice!
  [2 messages in #general]
> history general
  --- #general ---
  alice: Hello everyone!
  bob: Hey alice!
> rooms
  Known rooms:
    #general — 2 messages
    #random — 0 messages

Same binary, same commands — just add cluster config:

# Terminal 1: seed node. On startup it prints a line like:
#   seed: 5e9c...f0a8@127.0.0.1:7100  (pass to another node with --seed)
cargo run -p murmer-examples --bin cluster_chat -- --node alpha --port 7100

# Terminal 2: joins via the seed's id@address (copy alpha's `seed:` line)
cargo run -p murmer-examples --bin cluster_chat -- \
    --node beta --port 7200 --seed 5e9c...f0a8@127.0.0.1:7100

Each node now has a persistent identity key (its iroh endpoint id). The cluster_chat example writes one to <node-name>.key so the id is stable across restarts. A seed is <endpoint-id>@<host:port>, because iroh dials by key and the address alone is no longer enough. See Administration & Security.

Step 4: Deploy with Docker

The docker-compose.yml in the repo runs a 3-node cluster with no manual setup:

docker compose up --build

Nodes are dialed by key, so the joiners need the seed’s endpoint id. The container entrypoint (docker-entrypoint.sh) handles that automatically. Each node generates its own persistent key in a shared ./keys volume, publishes its public endpoint id to /keys/<node>.id, and the joiners wait for the seed’s id file and dial it by key. The compose file just names each node and points the joiners at the seed:

services:
  alpha:
    build: .
    environment:
      MURMER_NODE: alpha          # seed node (no MURMER_SEED)
    volumes:
      - ./keys:/keys

  beta:
    build: .
    environment:
      MURMER_NODE: beta
      MURMER_SEED: alpha          # join via alpha's published id
    volumes:
      - ./keys:/keys

  gamma:
    build: .
    environment:
      MURMER_NODE: gamma
      MURMER_SEED: alpha
    volumes:
      - ./keys:/keys

Beta and gamma seed from alpha and mesh together. Keys persist in ./keys, so node identities are stable across docker compose restart. Only the public .id files are read by other nodes; each node’s secret key stays in its own .key file.

On a flat LAN you can skip seeds entirely and let mDNS discover peers (it now advertises each node’s endpoint id). Multicast across a Docker bridge network is unreliable, which is why the compose demo uses id-based seeds instead.

How clustering works

Auto-discovery

When an actor system starts in clustered mode, it runs a server that listens for incoming connections. New nodes connect to existing ones via seed nodes and begin exchanging information about their actors. Nodes can be configured to gossip this information, allowing the network to mesh together organically.

Networking layer

The networking layer is built on iroh (a QUIC stack) and SWIM (via the foca crate):

• iroh provides a reliable, low-latency QUIC transport where each peer is identified and authenticated by an ed25519 endpoint id (a public key), not by IP address. host:port becomes an addressing hint iroh uses to establish the direct connection. Each node pair shares a single connection, multiplexed over per-actor streams. The authenticated endpoint id is what makes the zero-trust allowlist possible.
• SWIM handles cluster membership — failure detection, protocol-level heartbeats, and member state dissemination. Membership is keyed on the endpoint id.
• mDNS provides optional zero-configuration discovery for LAN environments, advertising each node’s endpoint id so peers can dial it by key.

Stream architecture

When a remote actor’s endpoint is accessed:

1. A dedicated QUIC stream is opened to the remote node for that actor.
2. The stream stays open as long as it’s active (not idle).
3. On the receiving end, a stream handler deserializes incoming messages, looks up the target actor via the receptionist, and forwards them.
4. Each stream binds to a single actor — messages for other actors result in an error and stream closure.
5. The stream subscribes to the actor’s lifecycle via the receptionist. If the actor enters a negative state (stopped, dead), the stream closes with an error.

An actor on a node might have multiple inbound streams, but the mailbox system ensures messages are processed in order of arrival.

Registry replication

Actor registrations are replicated across the cluster using an OpLog with version vectors:

• When a local actor registers, its node broadcasts an ActorAdd operation to all peers.
• Remote nodes create lazy endpoint factories in their local receptionists.
• Version vectors ensure operations are idempotent and ordering is preserved.
• When a node leaves, its registrations are pruned from all other nodes.

This gives every node an eventually consistent view of the entire cluster’s actor topology.

Edge Clients

Not everything that talks to a murmer cluster needs to be a cluster member. A REST API gateway, a CLI tool, a monitoring dashboard, or an integration test runner just needs to call actors — it doesn’t need to run any, participate in SWIM gossip, or store a registry.

Edge clients fill that role. A MurmerClient connects to any cluster node, pulls the set of public actors, and exposes the same Endpoint<A> API you already know — without any of the cluster machinery behind it.

Like every connection in murmer, an edge client dials the server by key: you give it the server node’s iroh endpoint address (its endpoint id plus a host:port hint), not a bare socket address. Get the server’s endpoint id by running murmer id on it.

use murmer::MurmerClient;
use std::time::Duration;

// The server's endpoint address = its endpoint id + a host:port hint.
let server = iroh::EndpointAddr::from_parts(
    server_endpoint_id,                       // from `murmer id` on the server
    [iroh::TransportAddr::Ip("10.0.0.5:9000".parse()?)],
);

let client = MurmerClient::connect(server, "cluster-cookie").await?;
let ep = client.lookup::<UserService>("api/users").unwrap();
let user = ep.send(GetUser { id: 42 }).await?;
client.disconnect().await;

The edge client generates a fresh ephemeral key on each run. If the server runs with an enforced allowlist, that ephemeral key won’t be admitted. Either run edge clients against Open-mode nodes, or give the client a persistent, allowlisted key.

Visibility: controlling what Edge clients see

Every actor has a visibility that controls who can discover it. You set this at startup time on the server side:

// Public — visible to Edge clients and all cluster members
let api = system.start_public("api/users", UserService, state);

// Internal — visible to cluster members only (default)
let router = system.start("routing/shard-0", ShardRouter, state);

// Private — node-local only, never replicated
let metrics = system.start_private("node/metrics", MetricsCollector, state);

Private is a zero-overhead choice: the actor is never written to the OpLog, never serialized, and never sent over the wire. Use it for utility actors that are purely node-local implementation details — connection managers, per-node caches, local metrics collectors.

Connecting

Edge clients connect to any node in the cluster. The node you connect to acts as your sync source — it responds to pull requests with the current set of public actors.

// Short-lived: connect, call, disconnect. `server` is an iroh::EndpointAddr
// (endpoint id + host:port), built as shown above.
let client = MurmerClient::connect(server, "cluster-cookie").await?;

The cluster cookie must match the server’s cookie or the handshake will be rejected.

Requirements

The server must be started with System::clustered() — Edge clients connect via QUIC and need a listening endpoint. System::local() has no network layer and cannot accept Edge client connections.

Looking up actors

lookup — instant, returns None if not synced yet

if let Some(ep) = client.lookup::<UserService>("api/users") {
    let user = ep.send(GetUser { id: 42 }).await?;
}

Returns None if the actor hasn’t been synced to the client yet. Use this after an initial sync has had time to complete.

lookup_wait — blocks until the actor appears

let ep = client
    .lookup_wait::<UserService>("api/users", Duration::from_secs(5))
    .await?;

Triggers an immediate pull, then waits for the actor to appear — either from that pull’s response or a subsequent one. Re-polls the server every 500 ms. Returns ClusterError::Timeout if the actor doesn’t appear within the deadline.

Fast path: if the actor is already synced, lookup_wait returns after one pull round-trip (typically sub-millisecond on LAN).

Usage patterns

Pattern 1: Short-lived client (pull once)

Ideal for CLI tools, integration tests, and one-off queries. Pulls on connect, uses the snapshot, disconnects.

let client = MurmerClient::connect(addr, cookie).await?;

// Give the initial pull a moment to arrive
tokio::time::sleep(Duration::from_millis(50)).await;

let ep = client.lookup::<UserService>("api/users").unwrap();
let result = ep.send(GetUser { id: 1 }).await?;

client.disconnect().await;

Pattern 2: Long-lived gateway (periodic pull)

Ideal for REST/gRPC API servers, dashboards, and proxies. Use sync_interval to re-pull periodically and pick up new actor registrations as the cluster evolves.

use murmer::ClientOptions;

let client = MurmerClient::connect_with_options(
    addr,
    cookie.into(),
    ClientOptions {
        sync_interval: Some(Duration::from_secs(30)),
        ..Default::default()
    },
).await?;

// client.lookup() stays fresh — re-pulled every 30 seconds

Pattern 3: Wait for a specific actor

Useful when you connect before the target actor is registered — for example, a gateway that starts before the cluster has finished placing its actors.

let ep = client
    .lookup_wait::<PaymentService>("payments/processor", Duration::from_secs(10))
    .await?;

How sync works

Edge clients use pull-based sync — the server never pushes unsolicited updates. The client sends a RegistrySyncRequest with its current version vector; the server responds with only the delta (new public actor registrations since that version).

Edge client                     Cluster node
    │                                │
    │── RegistrySyncRequest(vv) ────▶│
    │◀── RegistrySync(delta ops) ────│
    │                                │
    │  ... time passes ...           │
    │                                │
    │── RegistrySyncRequest(vv') ───▶│  (periodic or lookup_wait re-poll)
    │◀── RegistrySync(new ops) ──────│

After the first sync, subsequent pulls return only the delta — O(new ops), not O(all ops). 1000 idle Edge clients add near-zero server overhead: no SWIM membership, no server-initiated fan-out, no per-client state.

Scalability characteristics

Full example

The edge_client example demonstrates all three patterns:

cargo test -p murmer-examples --bin edge_client

It covers:

1. Public actors visible to Edge clients, internal actors hidden
2. lookup_wait resolving when an actor registers after connect
3. Long-lived client with periodic sync_interval

Simulation Testing

murmer can run your actors under a deterministic, single-threaded runtime with a virtual clock. You drive the world by hand, and time only moves when you advance it. The scheduling, the timer firings, and the seeded randomness all replay from the seed. This is the murmer side of the FoundationDB approach to testing: instead of hoping a race shows up under load, you replay it from a seed.

Enable the sim feature:

[dev-dependencies]
murmer = { version = "0.4", features = ["sim"] }

The idea

Normally a System runs on Tokio. Tasks spawn on real threads, timers fire on the real clock, and Instant::now() reads wall time. None of that is reproducible. The sim feature swaps the runtime under the whole actor framework for a SimRuntime: one thread, a seeded scheduler, and a clock that only advances when you tell it to. Your actor code does not change. You write the same handlers, send the same messages. Only the test harness differs.

SimWorld is that harness. It owns a System built on the SimRuntime plus the executor that drives it.

A first test

use std::time::Duration;
use murmer::SimWorld;

#[test]
fn worker_drains_on_a_timer() {
    let mut world = SimWorld::new(0xC0FFEE);

    let worker = world.system().start("worker/0", Worker, WorkerState::default());

    world.send(&worker, Submit { jobs: 5 }).unwrap();
    world.send(&worker, StartDraining { every_ms: 100 }).unwrap();

    // Move 250ms of virtual time. The drain timer fires at 100ms and 200ms.
    // This returns immediately. Nothing sleeps in real time.
    world.advance(Duration::from_millis(250));

    assert_eq!(world.send(&worker, Completed).unwrap(), 2);
}

Three methods do most of the work:

• world.send(&endpoint, msg) sends a message and drives the world until the reply comes back, then returns it. Use this instead of endpoint.send(..).await.
• world.advance(duration) moves virtual time forward, firing every timer that comes due and running the tasks they wake.
• world.block_on(future) drives the world until any future completes, advancing time as needed. send is built on top of it.

For lower-level control, world.pump() runs ready tasks to quiescence without moving the clock, and world.now() reads the current virtual instant.

Determinism and seeds

SimWorld::new(seed) seeds the scheduler and a PRNG. Draw from that PRNG with world.rng_u64() to make your test’s own choices reproducible:

fn run(seed: u64) -> u64 {
    let mut world = SimWorld::new(seed);
    let worker = world.system().start("worker/0", Worker, WorkerState::default());
    world.send(&worker, StartDraining { every_ms: 10 }).unwrap();

    for _ in 0..20 {
        let jobs = world.rng_u64() % 4;        // deterministic "random" load
        world.send(&worker, Submit { jobs }).unwrap();
        world.advance(Duration::from_millis(10));
    }
    world.advance(Duration::from_secs(1));
    world.send(&worker, Completed).unwrap()
}

assert_eq!(run(1), run(1)); // same seed, same outcome, every run

When a seed surfaces a bug, it stays surfaced. You keep the seed, you keep the repro.

What works under sim

Everything on a single node’s local path: actor lifecycle, the supervisor loop, ctx.spawn, ctx.schedule_once, ctx.schedule_repeat, message send and reply, and restart backoff. Timers run on virtual time, so a test can fast-forward an hour of heartbeats in microseconds.

block_on will panic with a clear message if the future it is driving can never complete (every task is parked, no timer is pending). That almost always means the future is awaiting a reply or message that no actor will ever send, which is a real bug worth seeing.

Actor teardown has a fault seam of its own. Receptionist::set_terminate_hook holds an actor in the “accepted its stop, not yet deregistered” state for as long as you like on the virtual clock. That state is zero-width in normal operation, because deregistration rides on a synchronous Drop, so without the hook there is no way to test the code that waits on it. See Supervision for the contract and Extending Simulation for a worked example.

Running a whole actor stack

The first test showed one actor. The thing you will actually do most is stand up a whole stack of actors that talk to each other and run it deterministically. This is the heart of what the sim gives you. Your application is some set of actors passing messages and scheduling work. The sim lets you boot all of them on one virtual clock, drive them with a seeded workload, fast-forward time instead of sleeping through it, and replay the exact run from the seed.

cargo run -p murmer-examples --bin sim_app_demo is a worked example. It is a small job pipeline: a dispatcher round-robins a batch of jobs across three workers, each worker drains its queue at its own rate on a virtual-time timer, and every finished job is reported to a ledger.

dispatcher --Process--> worker (x3) --Completed--> ledger

The driver stands the stack up, submits a seeded batch, advances ten seconds of virtual time until the queues drain, and reads the ledger. Ten seconds of pipeline work runs in a few milliseconds of real time, and the ledger comes out identical every run at the same seed. Nothing about the actors is special. They use the ordinary start, send, ctx.spawn, and ctx.schedule_repeat you would write anyway. The only thing the sim changes is that time is virtual and the schedule is seeded.

One thing to know about scheduled work: the ScheduleHandle returned by ctx.schedule_once/schedule_repeat cancels the timer when it drops. Keep it in the actor’s state if you want the timer to keep firing. The workers in the example hold their drain ticker in state for exactly this reason.

SimWorld or SimCluster

Both run your actor stack. The difference is how many nodes.

SimWorld boots one System. It is the right tool when your stack lives on a single node, which is most application logic. The example above uses it.

SimCluster boots several Systems on the same virtual clock and wires them over an in-memory network, so you can put actors on different nodes and inject crash, partition, and latency. Your actor code does not change. The same actors that run on SimWorld run on SimCluster. You reach for SimCluster when the behavior you are testing involves the cluster itself: discovery, failover, the singleton fence, or how your actors behave when a node they depend on goes away.

So the progression is: write your actors, test the stack on SimWorld, then move to SimCluster when you need to fault the cluster underneath it. The next sections are about that second step.

Multi-node testing with SimCluster

A single SimWorld boots one System. To test membership, failure detection, and the cluster’s reaction to faults, you need several nodes talking to each other over a wire you control. SimCluster gives you that. It boots N ClusterSystems on one shared SimRuntime, wires them over an in-memory fabric, and gives you a small set of verbs to inject faults and read what each node saw.

You build a cluster, mesh the nodes, and pump until they converge:

use std::time::Duration;
use murmer::cluster::net::sim_cluster::SimCluster;

let mut cluster = SimCluster::builder(1)
    .node("node-a")
    .node("node-b")
    .node("node-c")
    .build();

cluster.mesh();   // inject a full mesh of discovery edges
cluster.pump();   // converge membership

builder(seed) takes the same kind of seed SimWorld does, and the same seed replays the same schedule across every node. .node(name) adds one node. Its endpoint id is "<name>-id" and it gets its own port, so .node("node-a") is reachable later as "node-a". .nodes(3) is shorthand for node-a, node-b, node-c.

mesh() injects a discovery edge between every pair of nodes. pump() runs the ready tasks to quiescence. Convergence here is pump-only. You do not advance the clock, because foca applies membership synchronously when the handshake lands. A node coming up is seen right away, no probe round needed.

Reading what each node saw

Every node holds an event receiver from the moment it boots. events(name) drains everything that node has seen since the last time you called it, sorted into three buckets:

• joined: the identities that came up, in arrival order. Each carries its full identity, so you can read the incarnation.
• failed: the endpoint ids the failure detector declared down.
• pruned: the endpoint ids removed from the registry.

The drain-since-last behavior is the whole trick. You drain once after pump to throw away the convergence joins, then you drain again after a fault to read only the failure phase. The two phases never mix in one read.

Crashing a node

crash(name) cancels a node’s shutdown token. Its event loop, readers, writers, accept loop, and foca timer manager all stop. The node goes silent without broadcasting a departure, so it looks like an abrupt crash rather than a graceful leave. The survivors only notice once you advance the clock past foca’s detection budget. Thirty seconds is comfortable at any seed.

The real test, crash_is_detected_failed_and_pruned_exactly, checks that the survivors detect exactly the crashed node and nothing else:

let only_a = BTreeSet::from(["node-a-id".to_string()]);
for seed in [1u64, 2, 0xC0FFEE] {
    let mut c = converged_trio(seed);
    // Discard the convergence joins so the next drain is the failure phase.
    let _ = (c.events("node-b"), c.events("node-c"));

    c.crash("node-a");
    c.advance(Duration::from_secs(30));

    for survivor in ["node-b", "node-c"] {
        let ev = c.events(survivor);
        assert_eq!(ev.failed, only_a, "{survivor} fails exactly A (seed {seed})");
        assert_eq!(ev.pruned, only_a, "{survivor} prunes exactly A (seed {seed})");
    }
}

converged_trio is the three-node mesh from above, pumped to convergence. The test runs at three seeds and asserts the same outcome at each one.

Partitioning a link

partition(a, b) severs the link between two nodes at the byte level. Both nodes keep running. The streams between them fail in both directions. One call cuts both ways. It returns false if there was no live connection to cut, which catches a test that forgot to mesh first.

A single cut between two nodes in a healthy mesh should heal itself. foca probes indirectly through the third node, so the membership never drops anyone. The test single_link_partition_is_masked_by_indirect_probing asserts exactly that:

for seed in [1u64, 2, 0xC0FFEE] {
    let mut c = converged_trio(seed);
    for n in ["node-a", "node-b", "node-c"] {
        let _ = c.events(n); // drain convergence
    }
    assert!(c.partition("node-a", "node-b"), "A–B link is live");
    c.advance(Duration::from_secs(30));
    for n in ["node-a", "node-b", "node-c"] {
        assert!(
            !c.events(n).any_failed(),
            "{n} saw a failure — a single A–B cut must be masked by C (seed {seed})"
        );
    }
}

any_failed() is a quick “did this node suspect anyone” check on the drained events. After thirty seconds of probing, nobody should have.

Rejoining at a higher incarnation

rejoin(name) brings a crashed node back as the same endpoint id at a strictly higher incarnation. The old cancelled system is dropped and a fresh one is bound in its place. It does not re-establish links, so you dial or mesh it back to the survivors and then advance. foca’s conflict resolution readmits the returning node because its incarnation outranks its own down instance.

crashed_node_rejoins_at_higher_incarnation walks the full cycle:

let mut c = converged_trio(1);
for n in ["node-a", "node-b", "node-c"] {
    let _ = c.events(n);
}
// Crash A; B and C detect it failed.
c.crash("node-a");
c.advance(Duration::from_secs(30));
let _ = (c.events("node-b"), c.events("node-c")); // clear the failure phase

// A returns as itself at incarnation 2 and re-dials the survivors.
c.rejoin("node-a");
assert_eq!(c.identity("node-a").incarnation, 2, "rejoin bumps the incarnation");
c.dial("node-a", "node-b");
c.dial("node-a", "node-c");
c.advance(Duration::from_secs(30));

// B and C readmit the returned A at incarnation 2 (higher incarnation wins).
let readmitted = |ev: &DrainedEvents| {
    ev.joined.iter().any(|id| id.endpoint_id.0 == "node-a-id" && id.incarnation == 2)
};
assert!(readmitted(&c.events("node-b")), "B readmits A@2");
assert!(readmitted(&c.events("node-c")), "C readmits A@2");

Notice the rhythm. Crash, advance to detect, drain the failure phase, rejoin, re-dial, advance again, then drain the readmission. Each drain reads one phase.

Adversarial scheduling

The default scheduler runs ready tasks in FIFO order, the order they were woken. That is deterministic, but it is only one order. A bug that only shows up under a different interleaving will not show up under FIFO no matter how many seeds you throw at it.

Adversarial scheduling fixes that. Instead of FIFO, the executor picks a random ready task each step. The choice comes from a seeded stream, so the interleaving is still reproducible from the seed, but it is a deliberately shuffled order rather than the natural one. On SimWorld you turn it on with world.use_random_scheduling(). On a cluster you set it at build time with .random_scheduling().

The scheduler draws from its own seed stream, derived off the root seed, so it never consumes from the actor RNG. Turning random scheduling on does not change the value sequence your actors draw. What it changes is which actor gets to run first when several are ready, so it changes which actor draws from the shared RNG first. That is the point. You are reordering who runs when, while the randomness stays the same.

The oracle pattern

Random scheduling does not give you identical outcomes to compare against. A different task order can change the cross-actor draw order, so you cannot just assert “random run equals FIFO run” on every detail. What you assert is that the property you care about holds either way. Run the same workload under FIFO and under shuffled scheduling, and check that the observable outcome is invariant.

That is the oracle. The outcome you assert is the invariant. The scheduling is the thing trying to break it. A green result under both is much stronger than a green result under FIFO alone, because the property survived the interleaving space instead of one lucky order.

convergence_is_invariant_under_adversarial_scheduling is the worked example. The converged membership set must not depend on task order:

fn converged(seed: u64, adversarial: bool) -> BTreeSet<(String, String)> {
    let mut b = SimCluster::builder(seed)
        .node("node-a")
        .node("node-b")
        .node("node-c");
    if adversarial {
        b = b.random_scheduling();
    }
    let mut c = b.build();
    c.mesh();
    c.pump();
    let mut pairs = BTreeSet::new();
    for me in ["node-a", "node-b", "node-c"] {
        for id in c.events(me).joined_ids() {
            pairs.insert((me.to_string(), id));
        }
    }
    pairs
}

for seed in [1u64, 2, 0xC0FFEE] {
    let fifo = converged(seed, false);
    assert_eq!(fifo.len(), 6, "full mesh: each of 3 nodes sees 2 peers");
    assert_eq!(
        converged(seed, true),
        fifo,
        "convergence is invariant under adversarial scheduling (seed {seed})"
    );
}

The converged helper runs the real cluster substrate: the event loop, the handshake and accept path, foca’s SWIM protocol, the control streams. It collects the membership each node ended up with, as a set of (observer, peer) pairs. The test asserts the FIFO run produced the full mesh of six pairs, then asserts the shuffled run produced the same six. The membership converged regardless of order.

This pattern carries past convergence. The same approach validates the failure detector under shuffled order, and the cluster’s coordination layer too. A short coordination example: with one shared generation source behind every node’s coordinator, a partition that splits the leader off does not split the brain. The new leader rebuilds the singleton’s spec from the shared backend and adopts it at a strictly higher term, so the fence rejects the stale owner. The send/send_async calls in those tests are local to each node’s own coordinator. What crosses the cluster is membership and the shared source, not application-actor messages between nodes.

The layer model

It helps to see the simulation stack as three layers.

Layer 1 is the deterministic runtime engine: the seeded scheduler, the virtual clock, the seeded RNG. That is SimRuntime and the executor inside SimWorld. It makes one node’s actor logic reproducible.

Layer 2 is the multi-node fault rig: SimCluster, the in-memory fabric, and the crash, partition, and rejoin verbs. It makes a cluster’s membership and failure handling reproducible.

Both of those are murmer’s, and you get them with the sim feature.

Layer 3 is yours. Disk faults, your own workloads, and the oracles that say whether a run was correct belong to the code under test. The framework does not own them. murmer hands you derive_seed(label), which forks an independent reproducible seed off the root, so your storage layer and your fault injector can each seed their own stream without colliding with the actor RNG or with each other. The whole stack descends from one seed.

What the network models

The network model is faithful to murmer’s transport, which is reliable ordered streams. iroh is QUIC underneath, so a stream never drops a byte and never reorders within itself. The fault injection respects that.

You inject latency with .network_latency(base, jitter) on the cluster builder. It delays delivery on the virtual clock and adds seeded jitter, so a slow or variable network replays from the seed like everything else. Order within a stream is preserved, because a reliable stream preserves it. Across streams the differing delays let messages to different actors arrive in a different relative order, which is the faithful form of reordering. A lost connection is the partition verb, a clean cut in both directions. What you will not find is byte-level packet loss or in-stream reordering, because the real transport does neither, and modeling them would surface bugs that cannot happen in production.

Determinism, and what to assert

Outcomes are deterministic and replay from the seed. The scheduler is seeded, the clock is virtual, the RNG is seeded, and the decision-bearing tokio::select! branches on the sim path use biased polling, a fixed branch order, the same policy the supervisor loop uses. A step with several ready branches resolves the same way every run, so the converged set and the final state are reproducible.

The guidance that follows is to assert on the outcome a run produced rather than on the incidental order unrelated events happened to arrive in. Membership converges to the same set every run, so the membership tests assert on that set. That habit is also what makes adversarial scheduling work: the outcome is the invariant you pin, and the interleaving is the thing trying to break it.

Runnable starting points

A few files in the repo are the place to start. Two you run, one you copy.

cargo run -p murmer-examples --bin sim_app_demo runs the whole-actor-stack example from above: a dispatcher, three workers, and a ledger on one virtual clock, drained deterministically and replayed from the seed. Start here to see a realistic stack run.

cargo run -p murmer-examples --bin sim_cluster_demo boots a three-node cluster on the sim runtime, crashes a node, shows the survivors detect it, and replays the same scenario from the same seed to prove it is reproducible. The thirty seconds of detection time it advances elapse in a few milliseconds of real time. This is the multi-node, fault-injection side.

murmer/tests/sim_cluster.rs is the template to copy. It is an external consumer of murmer that uses only the public API, the way your own crate would. It covers convergence, crash detection, partition tolerance, a remote actor reached across nodes, a slow-network run, and the same-seed replay check. Its single-node sibling, murmer/tests/sim_world.rs, is the same idea for one node’s actors.

How it fits together

The sim runtime is built on the Runtime seam (murmer::runtime::Runtime). TokioRuntime is the default and production is unchanged. SimRuntime implements the same trait with a seeded scheduler and virtual clock, and System::with_runtime(..) is the injection point. SimCluster boots its nodes on that same runtime through ClusterSystem::start_with_net, over an in-memory fabric in place of the real QUIC transport. You will not normally touch those directly. SimWorld::new and SimCluster::builder wire them for you.

The Determinism Contract

The previous chapter showed you how to drive a SimWorld. This one is about the rule that makes those green checks mean something. If you are writing tests on top of the sim, read this first. It is short, and the whole thing rests on one promise and one rule.

The promise

Same seed, same run. SimWorld::new(seed) seeds the scheduler, the virtual clock, and the PRNG, and from there the run is a pure function of the seed. Poll order, timer firings, every randomness draw: all of it replays.

That promise is the entire point of the harness. It buys you two things.

A green run you can trust. If a test passes under sim, it passed because the logic is correct, not because the scheduler happened to interleave tasks in a lucky order this time. There is no lucky order. There is one order per seed.

A red run you can replay. When a seed surfaces a bug, you keep the seed and you keep the repro. You can rerun it as many times as you need, add logging, step through it, and it lands in the same place every time. A flaky failure you cannot reproduce is worthless. A deterministic failure is a fixed target.

Both of those depend on the run actually being a function of the seed. The moment anything in your code pulls from a source that is not the seed, both break at once. The green run stops being trustworthy and the red seed stops replaying. So there is a rule.

The rule

Your code goes through the seams.

For concurrency and time and randomness, route through the Runtime seam. Inside an actor that means ctx.spawn, ctx.schedule_once, ctx.schedule_repeat, and for randomness the world’s seeded RNG. The sim swaps a SimRuntime under the whole framework, so when your actor calls ctx.spawn the task lands on the deterministic executor, and when it sleeps it sleeps on virtual time.

Never reach around the seam to touch the real runtime, the real clock, or the global RNG on a path that runs under sim. Concretely, never:

• tokio::spawn or any tokio::time::* timer
• Instant::now() or SystemTime::now() feeding a decision
• rand::random(), rand::thread_rng(), or any unseeded draw

Each of those is a source the seed does not control. Each one is a hole in the promise.

There is a second seam, and it is already here. The Net seam makes the transport between nodes deterministic the same way Runtime does for one node, which is what lets SimCluster boot a whole cluster on a single virtual clock. The same rule applies to code on that path. A raw spawn or an unseeded draw on the cross-node path breaks replay just as it does on a single node, and the multi-node suites run over the Net seam to keep it honest.

Two ways to break it, one that is dangerous

Break the rule and you land in one of two failure modes. They are not equally bad, and the difference is the whole reason this chapter exists.

The loud one is a raw tokio::spawn or tokio::time call on a sim path. There is no Tokio runtime under the sim executor, so the call panics the instant it runs. You will see a panic about there being no reactor running. It is jarring the first time, but it is the safe failure. The run stops, you see exactly where, and you route the call through the seam. Nothing silently rots. The panic is the sim telling you it found a corner of your code it cannot make deterministic yet. That is a coverage gap, and you found it on the spot.

The quiet one is the enemy. An unseeded rand::random(), a wall-clock read that steers a branch, or iteration over a HashMap on a decision path. None of these panic. They run fine. They just draw from a source the seed does not own, so the run quietly stops being a function of the seed. The test still goes green, because most of the time the nondeterminism does not change the observable outcome. Then one day it does, and now you have a green test that was lying and a red seed that will not replay. You debug it, the repro evaporates, and you have lost the one thing the harness was built to give you.

So the silent failure is the one to fear. The loud one tells on itself. The quiet one corrupts the promise while every check stays green.

The HashMap case is worth calling out because it does not look like randomness. A HashMap iterates in per-process random order. Iterate one on a path that decides something, an order of operations, who wins a tie, and the decision changes from run to run even though no rand call is in sight. The fix is a BTreeMap: it iterates in sorted key order, so the iteration is part of the seeded run. The receptionist’s actor table (entries) is a BTreeMap for exactly this reason. Keep decision-path registries sorted.

How the gate enforces it

The discipline is backed by a check script, scripts/check-determinism.sh. It scans the core modules that are routed through the seam and fails if any of the banned tokens appears unmarked:

tokio::spawn | tokio::time:: | Instant::now | SystemTime::now | rand::(rng|random|thread_rng)

A comment line is fine. So is a line carrying an explicit marker:

#![allow(unused)]
fn main() {
// determinism-gate: allow — <reason>
}

The marker is for the handful of sites that genuinely do not affect determinism, or that are consciously deferred. Two kinds show up in the tree today. One is measurement-only instrumentation, like a monitor timing a remote send, where the Instant::now() feeds a metric and never a branch. The other is a deferred path, like a debounce that still spawns on Tokio because it has not been routed yet. Both carry the marker with a reason so the exception is visible and reviewable, not silent.

Run the gate from a pre-commit hook or as a CI step. The script is honest about its reach: it gates only where it is invoked, so wire it in. It is also honest about what it cannot see. It catches Tokio, clock, and RNG escape hatches by token, the loud-adjacent class. It does not catch a new decision-bearing HashMap iteration, because a blanket map ban has too many false positives. That half is on you in review, guarded by one sim test (listing_backfill_order_is_deterministic).

Read that together with the previous section and the shape is clear. The runtime catches the loud failure for you, it panics. The gate catches the escape-hatch tokens. The dangerous failure, the silent decision-path HashMap, is precisely the one the gate cannot see by token, which is why the BTreeMap rule is a rule and not a suggestion.

One seed, one root

You will often have sub-systems that need their own randomness: the scheduler when you run adversarial interleavings, a fault injector, a consumer’s simulated disk faults. You do not want each of them seeded separately, because then a repro needs N seeds instead of one, and two consumers drawing from the same stream perturb each other’s draw order.

derive_seed(label) is the primitive that keeps it to one seed. Each sub-stream seeds itself off the one root by label:

let scheduler_seed = world.derive_seed("scheduler");
let disk_seed      = world.derive_seed("consumer/disk-faults");

Each call is a pure function of the root seed and the label, so the whole stack descends from a single u64. The streams are independent, so the fault injector drawing does not shift what the scheduler draws. The adversarial scheduler is built on exactly this: world.use_random_scheduling() seeds its RandomPolicy from derive_seed("scheduler"), a stream separate from the actor RNG, so turning random scheduling on does not change the value sequence the actors draw. One root seed reproduces the entire run, every sub-stream included.

Troubleshooting

Panic about no reactor running. A raw tokio::spawn or tokio::time call reached a sim path. There is no Tokio runtime under the sim executor. Find the call and route it through the seam: ctx.spawn, ctx.schedule_once, or ctx.schedule_repeat. This is the safe failure. You found a path that is not sim-ready yet.

Not reproducible across runs. Something is drawing from a source the seed does not control. Look for an unseeded rand call, a wall-clock read (Instant::now / SystemTime::now) that steers a decision, or a HashMap iterated on a decision path. Run the gate to catch the first two. For the third, switch the registry to a BTreeMap. Replace any test randomness with world.rng_u64() so it draws from the seeded stream.

block_on panics with a sim deadlock. Every task is parked, no timer is pending, and the future you are driving still has not completed. That almost always means the future is awaiting a reply or a message that no actor will ever send. It is a real bug worth seeing, not a harness quirk. Check what the awaited future is waiting on and whether anything in the run actually produces it.

Extending Simulation

murmer gives you a deterministic world. It does not give you every fault you will want to test. The clock, the scheduler, and the in-memory network belong to murmer. The disk faults and the oracles that check them are yours to write. This chapter is the handoff guide. It walks through building your own simulation layer on top of the seams murmer exposes, using a fault-injecting filesystem as the running example.

The ownership boundary

murmer owns the engine. The SimRuntime is the seeded scheduler and the virtual clock. SimWorld drives one node. SimCluster boots N nodes over an in-memory wire and injects crash, partition, and rejoin. All of that replays from one seed. You build on top of it.

You own your faults. murmer ships no disk dependency. There is no storage trait in the framework, no file abstraction, nothing for a disk-fault simulator to hook. That is on purpose. A storage application has its own idea of what durable state looks like and how it should fail, and murmer does not want to guess. So the disk-fault layer is yours to write. murmer’s job is to hand you a seeded deterministic world to hang it on.

Call this Layer 3. Layer 1 is the runtime and the network seams. Layer 2 is the harness (SimWorld, SimCluster). Layer 3 is your domain: the faults murmer does not know about and the oracles that check your application’s invariants.

The seam primitives you build on

Four primitives carry the whole Layer-3 pattern.

One seed, one root. world.derive_seed("vfs") gives you an independent u64 seeded off the world’s root seed. You feed it into whatever PRNG your fault stream uses. Because it descends from the one root seed, your faults replay alongside everything else murmer does, from a single u64. Distinct labels give distinct streams, so your fault stream and murmer’s own draws do not perturb each other:

let mut world = SimWorld::new(0xC0FFEE);
let fs_seed = world.derive_seed("vfs");    // independent, reproducible
let other  = world.derive_seed("faults");  // a different sub-stream
assert_ne!(fs_seed, other);

derive_seed does not draw from the world’s main RNG stream. Calling it does not shift the values world.rng_u64() will return. Two consumers can each derive their own seed without stealing each other’s draw order. This is the same property the sim tests assert in derive_seed_is_reproducible_and_label_independent.

One honesty note. derive_seed is reproducible only under the sim runtime. Under the default TokioRuntime it hands back fresh entropy on every call, because production is not meant to replay. Reproducibility is a sim-only property.

Synchronous storage calls. Real storage code blocks. You read a file, you fsync, you wait on the kernel. The Runtime trait has run_blocking for exactly this. In production it offloads to a blocking thread so real I/O never stalls the actor pool. In sim it runs the work inline on the deterministic thread as one atomic step, because sim “blocking” work returns promptly from in-memory state and a sim scheduler cannot observe an uncontrolled thread pool.

The signature matters. run_blocking takes a closure returning () and gives back a future that resolves to ():

fn run_blocking(&self, work: Box<dyn FnOnce() + Send + 'static>) -> BoxFuture<'static, ()>;

To get a value back out of the closure, capture a channel or a shared cell and write into it from inside the work, then read it after the future resolves. Do not expect run_blocking to return your bytes directly.

The virtual clock. Anything in your Layer-3 code that waits on time goes through the runtime, not through tokio::time or Instant::now. The runtime’s sleep and now are virtual under sim, so a fault that fires after a delay (a slow disk, a write that lands late) advances when the test advances the clock. If you reach for the real clock, your fault stops being deterministic and you have a hole.

The terminate hook. Actor deregistration is normally instant. The supervisor loop breaks and the DeregisterGuard fires in Drop, which is synchronous, so there is no moment where an actor is stopped but still in the registry. Real systems are messier. An actor can accept its stop and then take time to drain a mailbox or tear down external resources, staying registered but not serving for a while. Callers have to tolerate that window, and you cannot test the tolerance if the window is zero-width.

Receptionist::set_terminate_hook opens it. The hook is awaited between the supervisor exiting and the guard firing, so while it is pending the actor still shows up in lookup, still appears in listings, and its watches have not fired:

struct SlowToDie {
    label: &'static str,
    delay: Duration,
    runtime: Arc<dyn Runtime>,
}

impl TerminateHook for SlowToDie {
    fn before_deregister(&self, label: &str, _r: &TerminationReason) -> BoxFuture<'static, ()> {
        if label == self.label {
            self.runtime.sleep(self.delay)   // virtual time, like every other fault
        } else {
            Box::pin(std::future::ready(()))
        }
    }
}

world.system().receptionist().set_terminate_hook(Some(Arc::new(SlowToDie {
    label: "cache/user",
    delay: Duration::from_secs(5),
    runtime: Arc::new(world.runtime().clone()),
})));

Two things to know. The hook fires for every actor terminating on that node, including murmer’s internal ones, so filter on the label you are handed. And it is a fault seam, not a place to do work: it runs on the supervisor’s task and it holds the registry entry for as long as it stays pending, so a hook that never completes leaks the entry. It does not fire on the restart-limit-exceeded path, where the receptionist deregisters directly instead of through a supervisor exit.

The shape of a Layer-3 add-on

Here is the shape, with a fault-injecting filesystem as the example. Three pieces.

First, a trait your application code calls. Your code never touches std::fs directly. It goes through this seam, the same way murmer’s actor code never touches tokio::spawn directly.

// Your trait. murmer does not ship this. You write it.
trait Filesystem: Send + Sync {
    fn read(&self, path: &str) -> std::io::Result<Vec<u8>>;
    fn write(&self, path: &str, bytes: &[u8]) -> std::io::Result<()>;
    fn fsync(&self, path: &str) -> std::io::Result<()>;
}

Second, a real implementation for production. It does the obvious thing: reads and writes actual files.

struct RealFs;

impl Filesystem for RealFs {
    fn read(&self, path: &str) -> std::io::Result<Vec<u8>> {
        std::fs::read(path)
    }
    // ...write, fsync against the real disk
}

Third, a deterministic fault implementation for sim, seeded off derive_seed. It holds an in-memory store and a seeded PRNG. Before each operation it draws from the PRNG and decides whether to inject a fault: a torn write, a short read, an fsync that reports success but drops the data, a disk-full error. Same seed, same faults, every run.

struct SimFaultFs {
    store: Mutex<HashMap<String, Vec<u8>>>,
    rng: Mutex<ChaCha8Rng>,
}

impl SimFaultFs {
    fn new(seed: u64) -> Self {
        Self {
            store: Mutex::new(HashMap::new()),
            rng: Mutex::new(ChaCha8Rng::seed_from_u64(seed)),
        }
    }
}

impl Filesystem for SimFaultFs {
    fn write(&self, path: &str, bytes: &[u8]) -> std::io::Result<()> {
        // Draw from the seeded stream, decide whether to fault.
        if self.rng.lock().unwrap().next_u64() % 100 < 5 {
            return Err(std::io::Error::other("simulated disk full"));
        }
        self.store.lock().unwrap().insert(path.into(), bytes.to_vec());
        Ok(())
    }
    // ...read, fsync with their own fault draws
}

You build the SimFaultFs with a seed from the world, then hand it to your actors:

let mut world = SimWorld::new(0xC0FFEE);
let fs = Arc::new(SimFaultFs::new(world.derive_seed("vfs")));
let rt: Arc<dyn Runtime> = Arc::new(world.runtime().clone());
let store = world.system().start("store/0", StoreActor::new(fs, rt), StoreState::default());

Inside the actor, the Filesystem calls are synchronous, so they go through run_blocking to run as one atomic sim step instead of stalling the deterministic thread. Give the actor a handle to the Runtime (you constructed the actor, so you wire it in), and call run_blocking on that. The closure returns (), so capture a channel to carry the result out:

// `rt: Arc<dyn Runtime>` and `fs: Arc<dyn Filesystem>` are held by the actor.
let fs = fs.clone();
let (tx, rx) = tokio::sync::oneshot::channel();
rt.run_blocking(Box::new(move || {
    let _ = tx.send(fs.read("store/segment-0"));
})).await;
let bytes = rx.await.unwrap();

The Filesystem trait, RealFs, and SimFaultFs are yours. murmer does not ship them, and it should not. They are illustrations of the pattern you write for your own domain. Swap “filesystem” for whatever your application’s durable surface actually is.

This mirrors how murmer seeds its own scheduler

You are not inventing this pattern. murmer already uses it on itself, and the parallel is exact.

murmer’s scheduler has a ReadyPolicy trait, the seam that decides which ready task to poll next. The default is FifoPolicy, which draws no randomness. The adversarial implementation is RandomPolicy, which picks a random ready task each step. And RandomPolicy is seeded off the root seed through derive_seed. When you call world.use_random_scheduling(), murmer does this:

let seed = self.derive_seed("scheduler");
self.set_policy(Box::new(RandomPolicy::new(seed)));

Line that up against your filesystem:

• ReadyPolicy is the seam trait. Your Filesystem is the seam trait.
• FifoPolicy is the plain implementation. RealFs is the plain implementation.
• RandomPolicy is the fault implementation. SimFaultFs is the fault implementation.
• RandomPolicy is seeded from derive_seed("scheduler"). SimFaultFs is seeded from derive_seed("vfs").

murmer seeds its scheduler’s fault stream off derive_seed("scheduler"). You seed your disk’s fault stream off derive_seed("vfs"). Same root seed, a different label. Copy the pattern.

The discipline travels with it

Layer 3 only stays deterministic if your code obeys the same seam rules murmer’s core does. The discipline does not stop at the boundary. It comes with the pattern.

No tokio::spawn in your fault layer. No tokio::time. No Instant::now. No unseeded rand. No HashMap iteration in a path that drives a decision your oracle checks, because HashMap order is per-process random and will desync your replay. Every one of these is a way to smuggle nondeterminism past the seed, and any one of them turns a reproducible failure back into a flaky one.

This is the same set of rules the determinism chapter spells out for murmer’s own code, and the same rules scripts/check-determinism.sh enforces on the core path. See the determinism chapter for the full list and the reasoning. When you build Layer 3, you are signing up for that discipline in your own code.

End-to-end oracles

The other half of Layer 3 is the oracle: the check that your application’s invariant held across the whole run. murmer’s sim tests already do this. The cluster oracles run a workload under FIFO and again under adversarial scheduling and assert the observable outcome is the same either way. You write the equivalent for your domain.

With the fault filesystem in place, an oracle looks like a property you assert after driving the world. Crash an actor mid-write, advance the clock, restart it, then read back and assert no acknowledged write was lost. Run the same workload across a range of seeds. The faults change with the seed, the invariant does not. When a seed breaks it, you keep the seed and you keep the repro, the same way you would for any sim failure.

Where the boundary could move

Today the boundary is clean. murmer holds no durable state of its own, so a disk-fault simulator touches only your code and never murmer’s. The framework keeps its actors and registries in memory, and persistence is the application’s concern.

That could change. If murmer ever takes on durable state of its own, a Raft log for the singleton fence being the obvious candidate, then disk faults would start touching murmer too. At that point murmer would owe you a storage seam of its own, so its log could run against your fault filesystem the way its scheduler runs against your fault policy. It does not today. As long as murmer’s state lives in memory, the disk layer is entirely yours, and the seeded deterministic world is what murmer gives you to build it on.

Administration & Security

A murmer node is identified by a cryptographic key, not by its address. This page covers what that means for running a cluster: generating and storing node keys, using the murmer CLI, and operating the zero-trust allowlist that controls which nodes are allowed to join.

If you have used earlier versions where a node was just a name@host:port with a shared cookie, the short version is: the cookie still exists as a coarse gate, but the real identity and authorization now come from a per-node key.

Node identity keys

Every clustered node has a secret key on disk. The public half of that key is the node’s endpoint id (an ed25519 public key), and that id is the node’s stable identity. It stays the same across restarts and even if the node’s IP address changes. The host:port you configure is now just a hint that helps other nodes find it.

The key lives in a file. By default that file is murmer-node.key in the working directory. Set an explicit path in config:

let config = ClusterConfig::builder()
    .name("alpha")
    .key_path("/etc/murmer/alpha.key")   // persist the identity here
    .listen("0.0.0.0:7100".parse()?)
    .cookie("my-cluster-secret")
    .build()?;

On first start the file is created (with 0600 permissions on Unix) and a new key is generated. On every start after that the existing key is loaded, so the endpoint id is stable.

Three things to keep in mind:

• Back the key up. If the file is lost, the node returns with a different endpoint id. Every allowlist entry and seed reference that pointed at the old id is now stale, and the node has to be re-admitted.
• One key per node. Because the default path is relative to the working directory, two nodes started in the same directory load the same key and end up with the same identity, which breaks the cluster. Give each node its own key_path.
• The key is sensitive. Anyone with the key file can impersonate that node. Treat it like an SSH private key.

The murmer CLI

The murmer binary manages keys and the allowlist. It works on files and does not talk to a running node, so you can use it during deployment or from a shell. Build it with the cli feature:

cargo run -p murmer --features cli --bin murmer -- <command>
# or install it: cargo install --path murmer --features cli

Print a node’s endpoint id

murmer id --key /etc/murmer/alpha.key
# 5e9c2da5...f0a8

This loads the key file (creating it if it does not exist) and prints the endpoint id. This is how you find the id to put in another node’s allowlist, or to hand to a joiner as a seed.

Manage the allowlist

murmer allow add  <endpoint-id> --file /etc/murmer/allow.txt
murmer allow rm   <endpoint-id> --file /etc/murmer/allow.txt
murmer allow list               --file /etc/murmer/allow.txt

The allowlist file is plain text, one endpoint id per line, with # comments. You can edit it by hand or with the CLI.

The allowlist

The allowlist is murmer’s zero-trust authorization layer. There are two gates a peer has to pass to join a cluster:

1. The cookie, a shared secret checked during the handshake. This is a coarse “are you even talking to this cluster” check. It has not changed.
2. The allowlist, a set of endpoint ids. This is the real authorization. A node admits a peer only if that peer’s endpoint id is on its list.

Because iroh authenticates each connection against the peer’s key, a peer cannot lie about its identity to get past the allowlist. The cookie alone is no longer enough to become a member.

The allowlist has two modes:

• Open (the default). Any peer with the correct cookie is admitted. This matches the old cookie-only behavior, and is convenient for local development.
• Enforced(path). Only peers whose endpoint id appears in the file at path are admitted.

let config = ClusterConfig::builder()
    .name("alpha")
    .key_path("/etc/murmer/alpha.key")
    .listen("0.0.0.0:7100".parse()?)
    .cookie("my-cluster-secret")
    .allowlist("/etc/murmer/allow.txt")   // switch to Enforced mode
    .build()?;

Enforcement runs in both directions. A node checks the allowlist when it accepts an inbound connection and before it dials out to a peer it learned about through discovery or gossip. Gossip carries addressing hints only. A node never treats a gossiped peer as authorized just because another node mentioned it. The rule is simple: a node is a member if, and only if, it is on the local allowlist.

Hot reload and revocation

The allowlist file is watched. When you change it, the running node picks up the change within about a second, with no restart:

• Adding an id lets a previously rejected peer connect.
• Removing an id drops any live connection to that peer right away and refuses to reconnect. In-flight requests to a revoked peer fail fast.

This means you manage membership by editing files (by hand, with the CLI, or through your config management tooling), and the cluster reacts on its own.

Operational workflows

Adding a node to a running cluster

Adding a node X means every existing node has to trust X’s key, and X has to trust theirs. Trust is mutual.

1. On X, get its endpoint id: murmer id --key /etc/murmer/x.key.
2. Add X’s id to every existing node’s allowlist file (or to a single shared file on a mount). Use murmer allow add <x-id> --file ... or your config management.
3. Make sure X’s own allowlist contains the nodes it should reach.
4. Start X with a seed pointing at one of the existing nodes.

No restarts are needed. The existing nodes pick up X’s id within a second and admit it when it connects.

Removing or revoking a node

Remove the node’s endpoint id from the allowlist files. Every node that drops the id closes its live connection to that node within a second and refuses to let it back in. This is how you evict a compromised or decommissioned node.

Distributing the allowlist

murmer does not gossip the allowlist between nodes. Distributing the file is your job, which usually means it is already solved by whatever you use to ship config: Ansible, a Kubernetes ConfigMap, a shared mount, or a baked image. Point every node at the same logical file, and updates roll out the way the rest of your config does.

What the cookie is still for

The cookie has not gone away. It is a cheap first check that keeps unrelated clusters and obvious noise from getting as far as the key-level authorization. Set it to a real secret. Treat the allowlist as the layer that decides membership, and the cookie as the layer that scopes which cluster a node is trying to reach.

Edge clients

Edge clients (see Edge Clients) generate a fresh ephemeral key on each run, since they only dial out and never accept connections. If the server runs with an enforced allowlist, that ephemeral key is not admitted. Either point edge clients at nodes running in Open mode, or give the client a persistent key and add it to the server’s allowlist like any other peer.

Monitoring

Murmer provides production-grade metrics for your actor systems via the monitor feature. It follows a facade pattern — your actor code records metrics through thin instrumentation calls, and you choose the backend (Prometheus, StatsD, etc.) at startup.

Enabling monitoring

Add the monitor feature to your Cargo.toml:

[dependencies]
murmer = { version = "0.4", features = ["monitor"] }

When the feature is off, all instrumentation compiles to nothing — zero overhead, zero dependencies.

Quick start with Prometheus

One call installs the metrics recorder and starts an HTTP endpoint for scraping:

use murmer::monitor::start_prometheus;

#[tokio::main]
async fn main() {
    // Start serving metrics on :9000/metrics
    start_prometheus(9000).expect("failed to start prometheus exporter");

    // Now start your actor system as usual
    let system = System::local();
    let counter = system.start("counter/main", Counter, CounterState { count: 0 });

    // Every message send, handler invocation, and lifecycle event
    // is automatically recorded. Scrape with:
    //   curl http://localhost:9000/metrics
}

What gets measured

Murmer instruments five categories of metrics automatically. You don’t need to add any code — everything is recorded as actors run.

Actor lifecycle

Message processing

Endpoint sends

Networking

Cluster membership

Spawn drain loop

Receptionist

Label cardinality

All actor metrics use actor_type (the Rust type name, e.g. my_app::ChatRoom) rather than actor_label (e.g. "room/general"). This keeps cardinality bounded — you typically have fewer than 20 actor types, but could have thousands of labels.

Architecture: how it works

The instrumentation uses a facade pattern inspired by how tracing works:

1. instrument.rs (always compiled) — Contains thin pub(crate) functions like instrument::message_processed(actor_type). When monitor is on, these call metrics::counter!(...). When off, they’re empty #[inline(always)] functions that the compiler eliminates entirely.

2. Call sites (supervisor, receptionist, endpoint, etc.) — Call instrument functions unconditionally. No #[cfg] attributes scattered across the codebase.

3. Your application — Installs a metrics recorder at startup (e.g., start_prometheus(9000)). All recorded metrics flow to the backend you chose.

This means adding a new metric requires touching exactly two places: the instrument function and the call site. The facade keeps the #[cfg] logic in one file.

ClusterMonitor actor

In addition to Prometheus metrics, murmer provides a ClusterMonitor actor that maintains a queryable in-memory view of cluster health:

use murmer::monitor::{ClusterMonitor, ClusterMonitorState, run_monitor_bridge, GetClusterHealth};

// Start the monitor actor
let monitor = system.start("murmer/monitor", ClusterMonitor, ClusterMonitorState::new());

// Bridge cluster events into the monitor
tokio::spawn(run_monitor_bridge(&cluster_system, monitor.clone()));

// Query health at any time
let health = monitor.send(GetClusterHealth).await?;
println!("Alive: {}, Joins: {}, Failures: {}",
    health.alive_nodes, health.total_joins, health.total_failures);

The ClusterMonitor tracks:

• Which nodes are alive and when they joined
• Cumulative counters for joins, failures, and departures
• Per-node uptime

Grafana dashboard

A typical Prometheus + Grafana setup might query:

# Message throughput by actor type
rate(murmer_messages_processed_total[5m])

# 99th percentile handler latency
histogram_quantile(0.99, rate(murmer_message_processing_duration_seconds_bucket[5m]))

# Actor crash rate
rate(murmer_messages_failed_total[5m])

# Remote call latency
histogram_quantile(0.95, rate(murmer_network_roundtrip_duration_seconds_bucket[5m]))

# Cluster size over time
murmer_cluster_nodes{status="active"}

# Spawn dispatch latency (time requests wait in the drain queue)
histogram_quantile(0.95, rate(murmer_spawn_drain_dispatch_seconds_bucket[5m]))

# Spawn factory execution time by locality
histogram_quantile(0.99, rate(murmer_spawn_drain_factory_seconds_bucket{locality="local"}[5m]))

# Current spawn queue backpressure
murmer_spawn_drain_queue_depth

Proc Macro Reference

Murmer provides two proc macros to reduce boilerplate when defining actors: #[handlers] + #[handler] for handler generation, and #[derive(Message)] for explicit message types.

#[handlers] + #[handler]

Place #[handlers] on an impl block containing actor message handlers. Mark each handler method with #[handler].

Auto-generated messages (recommended)

#[handlers]
impl MyActor {
    #[handler]
    fn do_thing(
        &mut self,
        ctx: &ActorContext<Self>,
        state: &mut MyState,
        name: String,
        count: u32,
    ) -> String {
        format!("{name}: {count}")
    }

    #[handler]
    fn get_status(
        &mut self,
        _ctx: &ActorContext<Self>,
        state: &mut MyState,
    ) -> bool {
        state.is_active
    }

    #[handler]
    async fn fetch_data(
        &mut self,
        ctx: &ActorContext<Self>,
        state: &mut MyState,
        url: String,
    ) -> Vec<u8> {
        some_async_call(&url).await
    }
}

What gets generated

From the above, the macro produces:

Message structs — method name converted to PascalCase, parameters after ctx and state become fields:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DoThing { pub name: String, pub count: u32 }

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GetStatus;  // no extra params → unit struct

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct FetchData { pub url: String }

Message + RemoteMessage impls — each struct implements Message (with the handler’s return type as Result) and RemoteMessage (with a TYPE_ID of "ActorName::method_name"):

impl Message for DoThing { type Result = String; }
impl RemoteMessage for DoThing { const TYPE_ID: &'static str = "MyActor::do_thing"; }

Handler / AsyncHandler impls — dispatches message fields as method arguments:

impl Handler<DoThing> for MyActor {
    fn handle(&mut self, ctx: &ActorContext<Self>, state: &mut MyState, message: DoThing) -> String {
        self.do_thing(ctx, state, message.name, message.count)
    }
}

// async fn → AsyncHandler
impl AsyncHandler<FetchData> for MyActor { /* ... */ }

RemoteDispatch — a wire-format dispatch table that routes serialized messages by their TYPE_ID. This enables cross-node delivery without the sender knowing the concrete handler:

impl RemoteDispatch for MyActor {
    async fn dispatch_remote(&mut self, ctx, state, message_type: &str, payload: &[u8])
        -> Result<Vec<u8>, DispatchError>
    {
        match message_type {
            "MyActor::do_thing" => { /* deserialize DoThing, call handler, serialize result */ }
            "MyActor::get_status" => { /* ... */ }
            "MyActor::fetch_data" => { /* ... */ }
            _ => Err(DispatchError::UnknownMessageType(..))
        }
    }
}

Extension trait — ergonomic methods directly on Endpoint<MyActor>:

pub trait MyActorExt {
    fn do_thing(&self, name: String, count: u32) -> impl Future<Output = Result<String, SendError>>;
    fn get_status(&self) -> impl Future<Output = Result<bool, SendError>>;
    fn fetch_data(&self, url: String) -> impl Future<Output = Result<Vec<u8>, SendError>>;
}

impl MyActorExt for Endpoint<MyActor> { /* ... */ }

This lets you call handlers directly:

let result = endpoint.do_thing("hello".into(), 42).await?;
let status = endpoint.get_status().await?;
let data = endpoint.fetch_data("https://...".into()).await?;

Auto-registration — a linkme distributed slice entry for the TypeRegistry. At cluster startup, TypeRegistry::from_auto() collects all #[handlers]-annotated actor types automatically, enabling the cluster to route messages to the correct deserializer without manual registration.

Handler method signature

Each #[handler] method must follow this pattern:

fn method_name(&mut self, ctx: &ActorContext<Self>, state: &mut State, ...params) -> ReturnType

• &mut self — the actor instance.
• ctx: &ActorContext<Self> — provides access to the system, receptionist, and lifecycle operations like watch().
• state: &mut State — the actor’s mutable state.
• Remaining parameters become message struct fields.
• Use async fn for handlers that need to .await.

Explicit messages (backward compatible)

For messages shared across multiple actors, name the last parameter msg (or _msg) and the macro will use the type directly instead of generating a struct:

#[handlers]
impl MyActor {
    #[handler]
    fn increment(
        &mut self,
        ctx: &ActorContext<Self>,
        state: &mut MyState,
        msg: Increment,
    ) -> i64 {
        state.count += msg.amount;
        state.count
    }
}

Here Increment must already exist and implement Message. The extension trait method will take the message as a parameter: endpoint.increment(msg).

#[derive(Message)]

Derives Message (and optionally RemoteMessage) for a struct or enum.

Basic usage (local only)

#[derive(Debug, Clone, Serialize, Deserialize, Message)]
#[message(result = i64)]
struct Increment { amount: i64 }

This implements Message with type Result = i64.

With remote support

Add remote = "..." to also implement RemoteMessage with a wire-stable type ID:

#[derive(Debug, Clone, Serialize, Deserialize, Message)]
#[message(result = i64, remote = "counter::Increment")]
struct Increment { amount: i64 }

The TYPE_ID string is used for wire dispatch — it must be unique across all message types in the cluster and stable across code changes (don’t use std::any::type_name which can change between compiler versions).

Attributes

Application Orchestration

Murmer’s core gives you actors, messages, and clustering primitives. The app module (enabled via the app feature flag) builds on top of these to provide the application-level abstractions you need for real, working distributed applications: placement strategies, leader election, crash recovery, and a Coordinator actor that ties them all together.

Think of it this way: murmer gives you the building blocks, and the app module helps you assemble them into a running system that manages actor lifecycles across a cluster automatically.

Enable the feature in your Cargo.toml:

[dependencies]
murmer = { version = "0.4", features = ["app"] }

Overview

The orchestration layer answers three questions:

1. Where should this actor run? — Placement strategies score nodes based on load, capabilities, metadata, and constraints.
2. Who decides? — Leader election picks one node to run the Coordinator, which makes all placement decisions.
3. What happens when a node fails? — Crash strategies define recovery behavior: redistribute immediately, wait for the node to return, or let the actor die.

Actor specifications

An ActorSpec describes an actor that the orchestrator should place and manage. It captures what to run, how to recover from crashes, and where to place it.

use murmer::app::spec::{ActorSpec, CrashStrategy, PlacementConstraints};
use murmer::cluster::config::NodeClass;
use std::time::Duration;

let spec = ActorSpec::new("storage/photos", "orchestrator::StorageAgent")
    .with_state(serialized_state)
    .with_crash_strategy(CrashStrategy::WaitForReturn(Duration::from_secs(30)))
    .with_constraints(PlacementConstraints {
        required_classes: vec![NodeClass::Worker],
        required_metadata: [("volume".into(), "photos".into())].into(),
        ..Default::default()
    });

Fields

Crash strategies

Placement constraints

Constraints filter eligible nodes before the placement strategy scores them:

PlacementConstraints {
    required_classes: vec![NodeClass::Worker],        // must be a Worker node
    required_metadata: [("gpu".into(), "true".into())].into(), // must have gpu=true
    anti_affinity_labels: vec!["db/primary".into()],  // repel: don't co-locate with this actor
    colocate_with: Some("writer/c1".into()),          // attract: only where "writer/c1" already runs
    required_node_id: None,                            // or hard-pin to exactly one node id
    ..Default::default()
}

• required_classes — empty means any class is acceptable.
• required_metadata — the node must have all specified key-value pairs.
• anti_affinity_labels — avoid placing on nodes already running these actors (repel).
• colocate_with — place only on a node already running the named anchor actor (attract — the inverse of anti-affinity). Use it to pin a member next to a partner (e.g. a reader next to its writer). Because it is a hard filter, if no node runs the anchor the spec gets NoEligibleNodes rather than landing elsewhere.
• required_node_id — hard-pin to exactly one node id; if that node is not alive the spec gets NoEligibleNodes (it is never silently relocated — unlike the soft Pinned strategy).

colocate_with and required_node_id are hard filters, not preferences — they are how you build co-located actor groups and node-pinned placement on top of the Coordinator.

Placement strategies

The PlacementStrategy trait defines a fitness function that scores nodes for hosting a given actor spec. The Coordinator evaluates all eligible nodes (after constraint filtering) and picks the highest scorer.

trait PlacementStrategy {
    fn fitness(&self, node: &NodeInfo, spec: &ActorSpec, view: &ClusterView) -> f64;
}

• Return 0.0 or negative to indicate “do not place here”.
• Higher values mean stronger preference.
• The full ClusterView is available for global-aware decisions (e.g., load balancing).

Built-in strategies

Leader election

The LeaderElection trait is pluggable. The Coordinator only runs on the elected leader node.

trait LeaderElection {
    fn elect(&self, view: &ClusterView) -> Option<String>;
}

The default implementation, OldestNode, picks the node with the lowest incarnation counter. This is deterministic — all nodes independently compute the same answer without a consensus round.

use murmer::app::election::OldestNode;
use murmer::cluster::config::NodeClass;

// Any alive node can be leader
let election = OldestNode::any();

// Only Edge nodes can be leader
let election = OldestNode::with_class(NodeClass::Edge);

The Coordinator

The Coordinator is itself a murmer actor (dogfooding the framework). It maintains a ClusterView, accepts SubmitSpec messages, and handles crash recovery when nodes fail.

Lifecycle

1. The Coordinator starts on the elected leader node.
2. It subscribes to cluster events to track node joins and failures.
3. Users send SubmitSpec messages to declare what actors should run.
4. The Coordinator evaluates placement constraints and strategies, then sends SpawnActor control messages to target nodes.
5. When a node fails, the Coordinator re-places affected actors according to each spec’s CrashStrategy.

The cluster event bridge

The bridge (murmer::app::bridge) connects the raw cluster machinery to the Coordinator. It subscribes to ClusterEvents and translates them into Coordinator messages (NotifyNodeJoined, NotifyNodeFailed, NotifyNodeLeft, NotifySpawnAck). This keeps the Coordinator decoupled from the transport layer.

The recommended setup uses bridge::start_coordinator():

use murmer::app::bridge;
use murmer::app::coordinator::CoordinatorState;
use murmer::app::placement::LeastLoaded;
use murmer::app::election::OldestNode;

let cluster = system.cluster_system().unwrap();
let state = CoordinatorState::new(
    cluster.identity().node_id_string(),
    Box::new(LeastLoaded),
    Box::new(OldestNode::with_class(NodeClass::Edge)),
);

let coordinator = bridge::start_coordinator(cluster, state);

This wires up everything: the Coordinator actor, the event bridge loop, and the spawn drain loop.

The spawn drain loop

The spawn drain loop reads placement decisions from the Coordinator and dispatches them — either invoking a local spawn factory or sending a SpawnActor control message to a remote node. Each request is dispatched as an independent tokio::spawn task so factories run concurrently; acks arrive at the Coordinator in any order (keyed by request_id).

An AckGuard ensures that every spawn request receives an acknowledgement, even if the factory panics or the task is cancelled. On the happy path the factory calls ack(true, None). If the guard is dropped without an explicit ack (panic, cancellation), it fires a detached task to deliver a failure ack so the Coordinator’s pending_spawns map never leaks stale entries.

The cluster view

The ClusterView is the Coordinator’s world model — a snapshot of all nodes with their capabilities and running actors:

// Query the Coordinator's view
let view = coordinator.send(GetClusterView).await?;
println!("Alive nodes: {}", view.alive_count);
println!("Total nodes: {}", view.total_count);

// Query managed specs
let specs = coordinator.send(GetSpecs).await?;
for spec in &specs {
    println!("{}: {:?} on {}", spec.label, spec.state, spec.node_id);
}

Each node in the view carries:

• Identity — name, host, port, incarnation counter
• Class — Worker, Edge, Coordinator, etc.
• Metadata — arbitrary key-value pairs (e.g., "volume" = "photos")
• Running actors — labels of actors currently hosted
• Liveness — whether the node is reachable

Cluster singletons

Some actors must have exactly one instance across the whole cluster — a catalog owner, a sequence minter, a lock manager. A cluster singleton is a Coordinator-managed actor pinned to an anchor, with a fenced handoff so two instances never run at once.

use murmer::app::singleton::{SingletonSpec, SingletonAnchor};
use murmer::app::coordinator::{StartSingleton, GetSingleton};

// Declare the singleton via the Coordinator.
let ownership = coordinator.send_async(StartSingleton {
    spec: SingletonSpec::new("catalog", "app::Catalog", SingletonAnchor::Leader)
        .with_state(boot_bytes),
}).await??;
assert_eq!(ownership.generation.term, 1);

Or, as a convenience when the Coordinator is registered under the well-known "coordinator" label:

let ownership = system.start_singleton(
    SingletonSpec::new("catalog", "app::Catalog", SingletonAnchor::Leader),
).await?;

Anchors

SingletonAnchor decides which node owns the instance:

The generation fence

Every ownership grant carries a monotone SingletonGeneration { term, seq }:

• term bumps once per ownership change (a failover or move) — an FDB-style recovery epoch.
• seq bumps on a re-grant to the same owner (liveness renew / idempotent re-assert).

SingletonGeneration orders lexicographically (term-major) and packs into a single u64 via packed() — so a downstream fence that compares one integer (e.g. a write generation stamped on disk) rejects a stale ex-owner with no change to that comparison. A node that loses ownership and later returns necessarily holds a strictly-lower generation than its successor, so its first fenced write is rejected.

Failover

When the owner node leaves or fails, the Coordinator re-places the singleton on a surviving node that still satisfies the anchor, minting a strictly-higher term. The old owner is gone, so there is no drain — the generation fence is what guarantees a zombie ex-owner cannot double-write:

// Owner departs → the Coordinator re-drives to a survivor with term N+1.
let after = coordinator.send(GetSingleton { label: "catalog".into() }).await?.unwrap();
assert!(after.generation > before.generation); // strictly higher — the fence

The coordination backend

Singleton correctness across nodes comes from one swappable authority — the coordination backend, the GenerationSource trait — which owns two durable facts per singleton: the monotone fence generation (so two owners can never hold an equal token) and the spec (so a newly-elected leader can rebuild the managed set). Because correctness lives in the backend being a single authority, it works at 1, 2, or N nodes with no quorum among the nodes themselves.

By default the Coordinator uses an in-memory source (CoordinatorGenerationSource) — fine for a single node or tests, but as a per-node source it does not share monotonicity across writers and is not durable across a restart. For multiple nodes, inject one shared durable source that every node reaches:

let state = CoordinatorState::new(local_id, Box::new(LeastLoaded), Box::new(OldestNode::any()))
    .with_generation_source(my_shared_durable_source);

The trait’s methods: claim_term (begin a new ownership epoch — higher term), claim_seq (re-grant within the term), current (read ownership), put_spec (persist the spec), and list (all specs + ownership, for leader rebuild). put_spec/list default to no-ops, so a source that only fences still compiles; implement them to also close the amnesia gap — without them a leader that fails takes its singletons’ specs with it and the new leader silently orphans them. With a shared backend the new leader’s load_singletons_from_backend reads the persisted specs and re-places each at a strictly-higher term.

Pick a backend by deployment:

• In-memory — single node, dev, tests.
• Durable shared store — multi-node: a file-backed store for dev, or a real store (a database row, or appdata’s catalog, which already has the atomic compare-and-swap this needs) for production. One linearization point; no node-to-node consensus.
• Raft — opt-in for 3+ node, survive-a-split, no-external-store deployments. Not the default: a 2-node Raft cluster tolerates zero failures, so it does not fit the 1-to-N (including 2) goal. The trait stays async/fallible/intent-based with an opaque ticket precisely so a Raft backend can drop in later. See the coordination backend decision record.

Full example: Filesystem RPC

The orchestrator example demonstrates the full orchestration loop:

1. Three nodes form a cluster: a gateway (Edge class) and two workers (store-a, store-b).
2. Each worker advertises capabilities via metadata ("volume" = "photos" or "volume" = "docs").
3. The gateway runs a Coordinator that places StorageAgent actors on workers matching their placement constraints.
4. Clients query storage agents for directory listings and file reads — transparently routed across the cluster.
5. Node failure triggers crash strategy handling.

// Submit a spec with placement constraints
let result = coordinator.send(SubmitSpec {
    spec: ActorSpec::new("storage/photos", "orchestrator::StorageAgent")
        .with_state(photos_state_bytes)
        .with_crash_strategy(CrashStrategy::WaitForReturn(Duration::from_secs(30)))
        .with_constraints(PlacementConstraints {
            required_classes: vec![NodeClass::Worker],
            required_metadata: [("volume".into(), "photos".into())].into(),
            ..Default::default()
        }),
}).await?;

// The Coordinator placed the actor — now query it from any node
let photos = system.lookup::<StorageAgent>("storage/photos").unwrap();
let entries = photos.send(ListDir { path: "/".into() }).await?;

Run the example:

cargo test -p murmer-examples --test orchestrator