> ## Documentation Index > Fetch the complete documentation index at: https://personal-9eca1d6c-claude-nifty-bohr-567oke.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > Understanding the whatsapp-rust project structure, modules, and workspace layout ## Overview WhatsApp-Rust is a high-performance, async Rust library for the WhatsApp Web API. The project follows a modular, layered architecture that separates protocol concerns from runtime concerns, enabling platform-agnostic core logic with pluggable backends. ## Workspace Structure The project is organized as a Cargo workspace with multiple crates: ``` whatsapp-rust/ ├── src/ # Main client library ├── wacore/ # Platform-agnostic core │ ├── binary/ # WhatsApp binary protocol │ ├── libsignal/ # Signal Protocol implementation │ ├── appstate/ # App state management │ ├── noise/ # Noise Protocol handshake │ └── derive/ # Derive macros ├── waproto/ # Protocol Buffers definitions ├── storages/sqlite-storage/ # SQLite backend ├── transports/tokio-transport/ # Tokio WebSocket transport ├── http_clients/ureq-client/ # HTTP client for media ├── tests/e2e/ # End-to-end test suite └── examples/ # Example applications (benchmarks) ``` ## Three main crates ### wacore - platform-agnostic core **Location:** `wacore/` **Purpose:** Contains core logic for the WhatsApp binary protocol, cryptography primitives, IQ protocol types, runtime abstraction, and state management traits. **Key Features:** * **Zero runtime dependencies** — no Tokio, no async-std, only `futures`, `async-trait`, `async-lock`, and `async-channel` * **32-bit target support** — uses `portable-atomic` for 64-bit atomics with a software fallback on platforms without native `AtomicU64` (ARM32, MIPS, etc.) * `Runtime` trait for pluggable async executors (Tokio, async-std, WASM, etc.) * `Transport`, `TransportFactory`, and `HttpClient` traits for pluggable networking * `Backend` trait for pluggable storage * Cryptographic operations (Signal Protocol, Noise Protocol) * Type-safe protocol node builders **Key Modules:** ```rust theme={null} wacore/ ├── binary/ // Binary protocol encoding/decoding ├── libsignal/ // E2E encryption ├── noise/ // Noise Protocol handshake ├── appstate/ // App state sync protocol ├── derive/ // Derive macros (EmptyNode, ProtocolNode, StringEnum) ├── iq/ // Type-safe IQ protocol types ├── net.rs // Transport, HttpClient trait definitions ├── runtime.rs // Runtime trait + AbortHandle ├── protocol/ // ProtocolNode trait, keepalive, retry ├── time.rs // Pluggable time provider + portable Instant ├── types/ │ ├── events.rs // Event definitions │ └── message.rs // Message types └── store/ ├── traits.rs // Storage trait definitions (Backend, SignalStore, etc.) └── device.rs // Device state model ``` ### waproto - protocol buffers **Location:** `waproto/` **Purpose:** Houses WhatsApp's Protocol Buffers definitions compiled to Rust structs. **Build Process:** `build.rs` always runs, reads the committed binary descriptor (`src/whatsapp.desc`), verifies its SHA-256, and writes `whatsapp.rs` and `tags.rs` into `OUT_DIR`. Neither generated file is committed. To regenerate after modifying `whatsapp.proto`: ```bash theme={null} scripts/regenerate-proto-desc.sh # updates whatsapp.desc + .sha256 cargo build -p waproto ``` **Generated Types:** * `Message` - All message types * `WebMessageInfo` - Message metadata * `HistorySync` - Chat history * `SyncActionValue` - App state mutations ### Whatsapp-rust - main client **Location:** `src/` **Purpose:** Integrates `wacore` with concrete implementations (Tokio runtime, SQLite storage, ureq HTTP, Tokio WebSocket), provides the high-level `Bot` builder and `Client` API. **Key Features:** * `TokioRuntime` — default `Runtime` implementation (gated on `tokio-runtime` feature) * Typestate `BotBuilder` — compile-time enforcement that all 4 required components are provided * SQLite persistence (pluggable via `Backend` trait) * Event bus system * Feature modules (groups, media, newsletters, communities, etc.) ## Runtime abstraction The library is fully runtime-agnostic. All async operations go through four pluggable trait abstractions defined in `wacore`: | Concern | Trait | Default implementation | Crate | | ----------------- | -------------------------------- | -------------------------------- | ------------------------------------------ | | Async runtime | `Runtime` | `TokioRuntime` | `whatsapp-rust` (gated on `tokio-runtime`) | | Network transport | `TransportFactory` + `Transport` | `TokioWebSocketTransportFactory` | `whatsapp-rust-tokio-transport` | | HTTP client | `HttpClient` | `UreqHttpClient` | `whatsapp-rust-ureq-http-client` | | Storage | `Backend` | `SqliteStore` | `whatsapp-rust-sqlite-storage` | The `Runtime` trait requires four methods plus one optional method with a default: ```rust theme={null} pub trait Runtime: Send + Sync + 'static { fn spawn(&self, future: Pin + Send + 'static>>) -> AbortHandle; fn sleep(&self, duration: Duration) -> Pin + Send>>; fn spawn_blocking(&self, f: Box) -> Pin + Send>>; fn yield_now(&self) -> Option + Send>>>; /// How often to yield in tight loops (every N items). Defaults to 10. fn yield_frequency(&self) -> u32 { 10 } } ``` `AbortHandle` is `#[must_use]` — dropping the handle aborts the spawned task. Call `.detach()` on the handle for fire-and-forget tasks that should run to completion independently. See [custom backends — AbortHandle](/guides/custom-backends#aborthandle) for implementation details. The `yield_frequency()` method controls how often the client cooperatively yields during tight async loops (such as processing incoming frames). It returns the number of items to process before yielding. The default value is `10`. Single-threaded runtimes should return `1` to avoid starving the event loop, while multi-threaded runtimes can use higher values or rely on `yield_now()` returning `None`. On WASM targets, `Send` bounds are automatically removed via `#[cfg(target_arch = "wasm32")]`. The `BotBuilder` uses a typestate pattern with four type parameters `` (Backend, Transport, HttpClient, Runtime). The `build()` method is only callable when all four are `Provided`, making missing-component errors compile-time instead of runtime. See [custom backends](/guides/custom-backends) for implementing your own runtime, transport, HTTP client, or storage backend. ## Key Components ### Client **Location:** `src/client.rs` **Purpose:** Orchestrates connection lifecycle, event bus, and high-level operations. Uses `async-lock` (runtime-agnostic) for all internal synchronization instead of Tokio-specific primitives. ```rust theme={null} pub struct Client { pub(crate) core: wacore::client::CoreClient, pub(crate) persistence_manager: Arc, pub(crate) media_conn: Arc>>, pub(crate) noise_socket: Arc>>>, // ... connection state, caches, locks } ``` **Responsibilities:** * Connection management * Request/response routing * Event dispatching * Session management ### PersistenceManager **Location:** `src/store/persistence_manager.rs` **Purpose:** Manages all state changes and persistence. ```rust theme={null} pub struct PersistenceManager { device: Arc>, backend: Arc, dirty: Arc, save_notify: Arc, } ``` **Critical Pattern:** * Never modify `Device` state directly * Use `DeviceCommand` + `process_command()` * For read-only: `get_device_snapshot()` ### Signal Protocol **Location:** `wacore/libsignal/` & `src/store/signal*.rs` **Purpose:** End-to-end encryption via Signal Protocol implementation. **Features:** * Double Ratchet algorithm * Pre-key bundles * Session management * Sender keys for groups ### Socket & Handshake **Location:** `src/socket/`, `src/handshake.rs` **Purpose:** WebSocket connection and Noise Protocol handshake. **Flow:** 1. WebSocket connection 2. Noise handshake (XX pattern) 3. Encrypted frame exchange ## Module Interactions ```mermaid theme={null} graph TB A[Client API] --> B[Client] B --> C[NoiseSocket] B --> D[PersistenceManager] B --> E[Event Bus] C --> F[Transport] D --> G[Backend Storage] D --> H[Device State] B --> I[Signal Protocol] I --> G B --> J[wacore] J --> K[Protocol Types] J --> L[Crypto] ``` ## Layer Responsibilities ### wacore layer (platform-agnostic) * Protocol logic * State traits * Cryptographic helpers * Data models **Example: IQ Protocol** ```rust theme={null} // wacore/src/iq/groups.rs pub struct GroupQueryIq { group_jid: Jid, } impl IqSpec for GroupQueryIq { type Response = GroupInfoResponse; fn build_iq(&self) -> InfoQuery<'static> { /* ... */ } fn parse_response(&self, response: &NodeRef<'_>) -> Result { /* ... */ } } ``` ### Whatsapp-rust layer (runtime) * Runtime orchestration * Storage integration * User-facing API **Example: Feature API** ```rust theme={null} // src/features/groups.rs impl<'a> Groups<'a> { pub async fn get_metadata(&self, jid: &Jid) -> Result { // Use wacore IqSpec for protocol self.client.execute(GroupQueryIq::new(jid)?).await } } ``` ## Protocol entry points ### Incoming Messages **Flow:** `src/message.rs` → Signal decryption → Event dispatch Incoming stanzas are decoded as `Arc` (zero-copy from the network buffer) and routed through per-chat message queues: ```rust theme={null} // src/message.rs pub async fn handle_message(client: &Arc, node: &Arc) { // 1. Extract encrypted message from NodeRef // 2. Decrypt via Signal Protocol // 3. Commit (or accumulate into a batch during the offline drain) // 4. Dispatch Event::Messages } ``` ### Outgoing Messages **Flow:** `src/send.rs` → Signal encryption → Socket send Outgoing stanzas are built as owned `Node` values via `NodeBuilder`: ```rust theme={null} // src/send.rs pub async fn send_message(client: &Arc, msg: &Message) { // 1. Encrypt via Signal Protocol // 2. Build protocol node (owned Node) // 3. Send via NoiseSocket } ``` ### Socket Communication **Flow:** `src/socket/` → Noise framing → Transport ```rust theme={null} // src/socket/mod.rs impl NoiseSocket { pub async fn send_node(&self, node: Node) -> Result<()> { // 1. Marshal to binary // 2. Encrypt with Noise // 3. Frame and send } } ``` ## Connection Lifecycle ### Auto-Reconnection The client implements robust reconnection handling with stream error awareness: ```rust theme={null} // Client fields for connection and reconnection management is_connected: Arc, // Lock-free connection state (Acquire/Release) pub enable_auto_reconnect: Arc, // Toggle auto-reconnect pub auto_reconnect_errors: Arc, // Error count for backoff pub(crate) expected_disconnect: Arc, // Expected vs unexpected pub(crate) connection_generation: Arc, // Detect stale tasks ``` The `is_connected` field uses an `AtomicBool` to track whether the noise socket is established. This avoids a TOCTOU race that previously occurred when `try_lock()` on the noise socket mutex failed under contention, causing false-negative connection checks and silent ack drops. **Connection timeout:** Both the transport connection and version fetch are wrapped in a 20-second timeout (`TRANSPORT_CONNECT_TIMEOUT`), matching WhatsApp Web's MQTT and DGW defaults. This prevents dead networks from blocking on the OS TCP SYN timeout (\~60-75s). Both operations run in parallel via `tokio::join!`. **Reconnection flow:** 1. Connection lost → `cleanup_connection_state()` (see [disconnect cleanup](#disconnect-cleanup)) 2. Check `enable_auto_reconnect` → exit if disabled (401, 409, 516 disable this) 3. Check `expected_disconnect` → immediate reconnect if expected (e.g., 515) 4. Stability-gated backoff reset: the counter resets to base only if the connection was authenticated (``) for at least 30s (`STABLE_CONNECTION_RESET_MS`, WA Web's `resetDelay`) **and** no explicit penalty (429, manual `reconnect()`) is pending for this cycle — a penalty survives even a stable connection (WA Web `cancelReset`) 5. Calculate Fibonacci backoff delay (1s, 1s, 2s, 3s, 5s, 8s... max 900s with +/-10% jitter) 6. Wait → attempt reconnection (with 20s connect timeout) See [WebSocket & Noise Protocol - Fibonacci backoff](/advanced/websocket-handling#fibonacci-backoff) for the stability-reset and penalty-survival mechanics. ### Disconnect cleanup When a connection is lost or `disconnect()` is called, `cleanup_connection_state()` resets all connection-scoped state to prevent stale data from leaking into the next connection. It runs from `run()` after the message loop exits (and `disconnect()` also invokes it directly); the function is idempotent and race-tolerant, so it is not duplicated inside the message loop on transport disconnect events and resetting twice is harmless: | Resource | Action | Reason | | --------------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Transport, events, noise socket | Set to `None` | Release connection resources | | `is_connected` | Set to `false` (Release ordering) | After socket is `None` so no task sees connected with a cleared socket | | `chat_lanes` | Invalidated | Drop per-chat queue senders so workers exit via channel close — prevents stale workers from the old connection surviving reconnects with outdated signal/crypto state | | `pending_retries` | Cleared | Stale keys from detached scope guard cleanup would otherwise suppress the first retry after reconnect | | `signal_cache` | **Flushed, then cleared** | Pending identity / sender-key writes are persisted to the backend *before* the cache is dropped. If the flush fails, the cache is kept (not cleared) so dirty state survives. | | `response_waiters` | Drained | Pending IQ waiters fail fast with `InternalChannelClosed` instead of hanging until the 75s timeout | | Offline sync state | Reset | Counters, timing, and semaphore replaced with fresh single-permit instance | | Dead-socket timestamps | Reset to 0 | Prevents stale values from triggering an immediate reconnect on the next connection | | `app_state_key_requests`, `app_state_syncing` | Replaced with empty maps | Prevents unbounded growth across reconnections | Chat lane invalidation is critical for correctness. Without it, stale message processing workers from the previous connection survive reconnects, holding outdated Signal session state that causes decryption failures on the new connection. **Flush-before-clear (v0.6).** The signal cache is now flushed to the backend before being cleared on disconnect. Previously the cache was dropped immediately, so a just-advanced sender-key chain that hadn't been persisted yet was lost — on the next send the client would treat the chain as fresh and re-distribute the SKDM to every group device unnecessarily. Disconnect is therefore no longer a "forget all Signal state" operation; it's a "persist, then forget" one. This is a behavior change for anyone relying on the old drop-everything semantics. **Stream error behavior:** * **401 (unauthorized)**: Disables auto-reconnect, emits `LoggedOut` * **409 (conflict)**: Disables auto-reconnect, emits `StreamReplaced` * **429 (rate limited)**: Adds 5 extra Fibonacci steps to backoff, then reconnects; also suppresses the next stability-gated backoff reset (see above) * **515 (expected)**: Immediate reconnect without backoff * **516 (device removed)**: Disables auto-reconnect, emits `LoggedOut` * **``** (no numeric code): Force-closes the socket and reconnects with standard backoff — WA Web treats a malformed frame as unrecoverable for the current stream ### Message loop (read loop) The `read_messages_loop` runs on the `run()` caller's task and uses `select_biased!` to multiplex shutdown signals with transport events. Frame decryption is sequential (noise counter ordering), but node processing uses a hybrid inline/concurrent strategy: * **Inline**: `success`, `failure`, `stream:error` (connection state), `message` (arrival order for per-chat queues), `ib` (offline sync tracking) * **Spawned concurrently**: all other stanzas (receipts, notifications, presence, etc.) After processing a batch of multiple frames, the loop refreshes `last_data_received_ms` so the keepalive loop sees the batch completion time rather than the arrival time — preventing false-positive dead-socket triggers during large offline sync batches. The loop also cooperatively yields every `yield_frequency()` frames to avoid starving other tasks. See [WebSocket & Noise Protocol - Message loop](/advanced/websocket-handling#2-message-loop-read-loop) for implementation details. ### Keepalive loop The keepalive loop runs as a **separate spawned task**, fully decoupled from the read loop. This ensures keepalive pings are never blocked by frame processing — even during large offline sync batches that take seconds to drain. The two loops communicate solely through atomic timestamps (`last_data_received_ms`, `last_data_sent_ms`). ```rust theme={null} const KEEP_ALIVE_INTERVAL_MIN: Duration = Duration::from_secs(15); const KEEP_ALIVE_INTERVAL_MAX: Duration = Duration::from_secs(30); const KEEP_ALIVE_RESPONSE_DEADLINE: Duration = Duration::from_secs(20); const DEAD_SOCKET_TIME: Duration = Duration::from_secs(20); ``` **Behavior:** * Sends ping every 15-30 seconds (randomized, matching WA Web's `15 * (1 + random())`) * Skips ping if data was received within the minimum interval (connection proven alive) * Sends ping *before* dead-socket check to prevent false-positive reconnects on idle-but-healthy connections * Waits up to 20s for response * Checks dead socket on **every** tick (not just after failures) — catches scenarios where pending IQs caused the ping to be skipped, or where the ping succeeded but the connection died immediately after * Detects dead socket if no data received for 20s after a send, triggering immediate reconnection * Fatal errors (`Socket`, `Disconnected`, `NotConnected`, `InternalChannelClosed`) cause the keepalive loop to exit immediately * Error classification is exhaustive and compile-time enforced — adding a new error variant without handling it causes a build failure See [WebSocket & Noise Protocol - Keepalive](/advanced/websocket-handling#keepalive-and-dead-socket-detection) for detailed keepalive internals. ### Offline sync When reconnecting, the client tracks offline message sync progress: ```rust theme={null} pub(crate) struct OfflineSyncMetrics { pub active: AtomicBool, pub total_messages: AtomicUsize, pub processed_messages: AtomicUsize, pub start_time: Mutex>, } ``` **Sync flow:** 1. Receive `` → start tracking, reset counters 2. Process messages with `offline` attribute → increment counter 3. Receive `` → sync complete 4. Emit `OfflineSyncCompleted` event **Timeout fallback:** If the server advertises offline messages via `offline_preview` but never sends the end marker (``), the client applies a 60-second timeout (matching WhatsApp Web's `OFFLINE_STANZA_TIMEOUT_MS`). On timeout, the client logs a warning, marks sync as complete, and resumes normal operation so startup is not blocked indefinitely. **Concurrency gating:** During offline sync, the client restricts message processing to a single concurrent task (1 semaphore permit) to preserve ordering. Once sync completes — either by the server end marker, all expected items arriving, or timeout — the semaphore is expanded to 64 permits, switching to parallel message processing. The drain→live transition also flushes the tail of the [inbound commit batch](/advanced/inbound-durability#batching): the last accumulated batch of decrypted messages commits (buffer → Signal flush → durability hook → acks → `Event::Messages`) before the semaphore widens, so no live-mode message is processed ahead of it. If that tail commit fails, the transition is deferred and retried every 3 seconds while the client stays in single-permit drain mode; `OfflineSyncCompleted` still fires immediately so startup waiters are not blocked on the retry. **Semaphore transition safety:** When the semaphore is swapped from 1 to 64 permits, tasks that were already waiting on the old semaphore must not be silently dropped. The client uses a **generation-checked re-acquire loop** to handle this transition safely: 1. Each semaphore swap increments an atomic `message_semaphore_generation` counter 2. When a task acquires a permit, it checks whether the generation has changed since it started waiting 3. If the generation changed (meaning the semaphore was swapped while the task was blocked), the task drops the stale permit and re-acquires from the new semaphore 4. This loop continues until the task holds a permit from the current-generation semaphore This prevents a critical issue where `pkmsg` messages (which carry Sender Key Distribution Messages for group chats) could be silently dropped during the offline-to-online transition. Without this safety mechanism, a dropped `pkmsg` would cause all subsequent `skmsg` messages from that sender to fail with `NoSenderKeyState`, since the SKDM they depended on was never processed. **State reset:** On reconnect or cleanup, all offline sync state is reset (counters, timing, and the semaphore is replaced with a fresh single-permit instance) so stale state does not leak into the next connection attempt. **Pull-batch backlog drain (v0.6):** Offline resume now drives the same pull-batch loop WA Web uses to drain the backlog: stanzas that the client can't process (unrecognized ``, known-but-empty `` content, duplicates, ciphertexts that decrypt-fail terminally) are transport-acked alongside their retry receipt so the server stops re-delivering them. Before this, such a stanza fell through `classify_incoming_message` silently, so the server kept replaying it from the offline queue every reconnect until `` closed the stream. The drain logic also acks duplicate-message PDOs that previously hit the silent-drop branch in `handle_decrypted_plaintext`, and preserves the original `recipient` attribute (via `Client::spawn_node_transport_ack`, which echoes the raw `NodeRef` instead of rebuilding from `MessageInfo`) so LID-routed offline stanzas don't trigger `` on the ack. The Meta AI bot's `msmsg` (``) encryption type was the original motivating case for this drain — it could not be decrypted, only acked. Since the bot-secret decryption landed (see [Bot message decryption](/api/signal#bot-message-decryption-msmsg)), `msmsg` stanzas are decrypted and dispatched as normal `Event::Messages`; the drain still covers genuinely unrecognized or undecryptable enc types. **Nack on terminal decrypt failure:** When a ciphertext exhausts retries (`max-retry` reached in the PDO recovery state machine), the client now emits a `` carrying a structured `NackReason` code instead of silently dropping the message. The full set of 21 codes (`ParsingError`, `InvalidProtobuf`, `MissingMessageSecret`, etc.) mirrors WA Web's reason set so the server stops retransmitting once it sees a terminal nack — see [Protocol → Nack reasons](/advanced/binary-protocol#nack-reasons). **Ack SKDM-only session decrypts:** A `pkmsg`/`msg` that decrypts successfully but carries only a Sender Key Distribution Message (no user-facing content to dispatch) is now explicitly acked. Previously it could decrypt, skip event dispatch, and leave no ack — so the server kept replaying it from the offline queue. The fix closes that gap so SKDM-only stanzas drain like any other processed message. ### Critical app-state sync (pairing bootstrap) Right after a fresh pairing (and on any reconnect before the account's critical app-state collections have synced), the client fetches the `CriticalBlock` and `CriticalUnblockLow` collections — blocked contacts and push name — via a batched IQ, before dispatching `Connected`. Decoding those snapshots requires the app-state **sync-key-share**, an E2E message the primary phone sends automatically, which can arrive late if a heavy history sync is saturating the stream at the same time. **A single 180-second deadline** (`CRITICAL_SYNC_TIMEOUT_SECS`, matching WhatsApp Web's `WAWebSyncBootstrap`) bounds the whole critical-sync path: 1. A watchdog task is armed first against this deadline, before anything else runs. If critical sync hasn't completed by then, the client reconnects to retry — WhatsApp Web instead logs out (`socketLogout`) at this point, but reconnecting preserves credentials and lets auto-reconnect retry the sync. 2. The client waits up to 10 seconds (`KEY_SHARE_GRACE_SECS`) for the auto-shared key before running the batched critical-collections IQ. This grace period is purely an optimization to skip a redundant explicit key request in the common fast case — it does not gate correctness. 3. If a collection still can't be decoded because its key hasn't landed, the client sends an explicit `AppStateSyncKeyRequest` and waits for the re-share. For this initial critical bootstrap, the wait is bounded by whatever time remains on the shared 180s deadline (rather than a short fixed wait), so a key that arrives late — or is never auto-shared at all — still has a chance to recover on the same connection instead of failing the sync outright. Non-critical app-state sync (background regular collections, group `server_sync`, and the `ib` dirty-resync path) is unaffected by this deadline — those callers keep waiting a fixed 10 seconds for a missing key before giving up and re-syncing on a later cycle. The net effect: a key-share that's delayed by a saturated stream during pairing no longer strands the critical sync until the 180s watchdog forces a reconnect — it recovers via the explicit request within the same window, so contacts and push name sync reliably on the first connection. ### Deferred device sync During offline sync, the client may receive group messages from devices not yet present in the local device registry (for example, a companion device that was paired while the client was offline). Rather than firing a network request for each unknown device individually, the client batches these into a `PendingDeviceSync` set. **Flow:** 1. During offline message processing, `is_from_known_device()` detects an unrecognized sender device 2. The sender's user JID is added to `PendingDeviceSync` (deduplicated — each user is queued at most once) 3. A retry receipt is sent so the sender will redeliver the message after the device list is updated 4. When `` arrives (offline sync complete), the client waits 2 seconds (`OFFLINE_DEVICE_SYNC_DELAY`, matching WhatsApp Web) 5. All batched user JIDs are flushed in a single bulk usync request via `flush_pending_device_sync()` 6. If the flush fails, the JIDs are re-enqueued for the next attempt This batching approach minimizes network overhead — instead of N individual usync requests for N unknown devices, a single bulk request resolves all pending users. When online (not during offline sync), unknown devices trigger an immediate background usync request instead of being batched. ```rust theme={null} // src/pending_device_sync.rs pub(crate) struct PendingDeviceSync { pending: async_lock::Mutex>, } ``` The `PendingDeviceSync` state is cleared on reconnect to prevent stale entries from leaking across connections. Location: `src/pending_device_sync.rs`, `src/handlers/ib.rs`, `src/usync.rs` See also: [Unknown device detection](/advanced/signal-protocol#unknown-device-detection) for the detection mechanism during group message decryption. ## History sync pipeline History sync transfers chat history from the phone to the linked device. The pipeline is designed for minimal RAM usage through a multi-layered zero-copy strategy. ### Processing flow ```mermaid theme={null} graph LR A[Phone uploads
compressed blob] --> B[Download &
stream-decrypt] B --> C[Decompress
zlib] C --> D[Manual protobuf
field walking] D --> E[Bounded channel
cap=4] E --> F[LazyHistorySync
lazy blob] F --> G[Event dispatch] ``` ### RAM optimization layers 1. **Heuristic pre-allocation with `compressed_size_hint`** — the decompression buffer is pre-allocated using a 4x multiplier on the compressed blob's `file_length` (clamped to 256 bytes – 8 MiB). When the notification provides `file_length`, this avoids repeated `Vec` reallocation during decompression. The hint comes from the decrypted (but still compressed) blob size, which is a better estimate than the encrypted size that includes MAC/padding overhead 2. **Compressed bytes retained** — after streaming extraction, the original compressed input is handed back as the event payload. Queued events cost O(compressed) rather than O(decompressed). Peak extraction memory is approximately the largest single conversation 3. **Hand-rolled protobuf parser** — instead of decoding the entire `HistorySync` message tree (which allocates every nested message), the core walks varint tags manually and only extracts field 2 (conversations) and field 7 (pushnames) 4. **`Bytes` zero-copy slicing** — decompressed data is wrapped in a reference-counted `Bytes` buffer; each conversation is extracted as `buf.slice(pos..end)`, which is an Arc refcount increment with no per-conversation heap allocation 5. **Bounded channel streaming** — an `async_channel::bounded::(4)` streams conversation bytes from the blocking parser thread to the async event dispatcher, providing backpressure with only \~4 conversations in-flight 6. **`LazyHistorySync` wrapper** — the compressed payload is wrapped in a `LazyHistorySync` with cheap metadata (sync type, chunk order, progress) available without decoding. Full protobuf decoding only happens if the event handler calls `.get()`. Clone is a refcount bump (no cache carried over). Consumers use `.stream()` for memory-bounded incremental decoding, `.decompress()` for one-shot inflation, or `.compressed_bytes()` for zero-copy access to the stored payload 7. **Compile-time callback elimination** — when no event handlers are registered, the callback is `None`, causing the parser to skip conversation extraction entirely at the protobuf level 8. **Secret-presence pre-scan** — before running buffa decode on each `HistorySyncMsg`, a shallow varint walk checks whether the message carries `message_secret` at any level (`WebMessageInfo.message_secret` or `Message.message_context_info.message_secret`). Messages without a secret (the majority in production blobs) are discarded immediately — no struct decode, no allocation. The scan mirrors protobuf merge semantics so repeated field occurrences and malformed bytes are handled identically to a full decode 9. **Shared conversation id** — `HistoryMsgSecretRecord.chat_id` is `Arc`, allocated once per conversation and reference-counted into every record within that conversation (10k clones → 500 on the bench fixture). `msg_id` uses `CompactString` (inline for typical 20–22 char WA IDs) and `secret` uses `SecretBytes` (inline for secrets ≤32 bytes) ### Skip mode For bots that don't need chat history, `skip_history_sync()` sends a receipt so the phone stops retrying uploads but downloads nothing. See [Bot - History Sync](/api/bot#history-sync). ## Concurrency Patterns ### Per-Chat Lanes Prevents race conditions where a later message is processed before the PreKey message. Each chat gets a lane combining an enqueue lock and a bounded channel (capacity **500** messages) into a single cached entry, providing backpressure to prevent memory amplification when many chats are active simultaneously: ```rust theme={null} pub(crate) chat_lanes: Cache, // ChatLane { enqueue_lock, queue_tx } // Each queue: async_channel::bounded::>(500) ``` ### Per-device session locks Prevents concurrent Signal protocol operations on the same session. Each device JID gets its own lock, keyed by protocol address strings generated by `to_protocol_address_string()` (format: `user[:device]@server.0`): ```rust theme={null} pub(crate) session_locks: Cache>>, // Key examples: "5511999887766@c.us.0", "123456789:33@lid.0" ``` The DM send path resolves all known recipient devices and own companion devices from the local device registry, filters out hosted devices, excludes the sender device, and deduplicates for self-DMs — matching WA Web's `WAWebSendUserMsgJob` and `WAWebDBDeviceListFanout` behavior. The local registry is checked first; a network fetch is only triggered on a cache miss to avoid unnecessary LID-migration side effects. Session locks are acquired for all involved devices in sorted order to prevent deadlocks. The `build_session_lock_keys()` helper resolves encryption JIDs (normalizing the recipient to bare form via `to_non_ad()`), sorts by `(server, user, device)` using `cmp_for_lock_order()`, and deduplicates. The `session_mutexes_for()` helper then converts the sorted JIDs to session mutexes, reusing a single `String` buffer to avoid per-JID heap allocations. The peer message path (single-device) acquires a single lock for the resolved encryption JID. **Group SKDM fan-out now shares the DM per-device session locks (v0.6).** `prepare_group_stanza`'s pairwise SKDM fan-out (`encrypt_for_devices_with_sessions`) mutates each target device's pairwise Signal session the same way the DM path does, but it was previously only covered by the per-`(group, sender)` sender-key chain lock — a disjoint key from the DM path's per-device session locks. A concurrent DM (or another group send) sharing a device could therefore race that device's pairwise ratchet: both sides load chain index *N* and both store *N+1*, silently dropping one advance. If the lost advance carried the SKDM, that member never received the sender key and every subsequent `skmsg` was undecryptable for it until a retry re-distributed. `prepare_group_stanza` now acquires the SKDM targets' per-device session locks — via the new `SendContextResolver::lock_device_sessions()` hook, whose `Client` implementation reuses `build_session_lock_keys()` + `session_mutexes_for()` so both paths serialize on the identical mutexes — before taking the sender-key chain lock, and releases them right after the SKDM fan-out (the `skmsg` chain encrypt that follows only touches the sender-key chain, not any pairwise session). Lock order is always session locks → chain lock on every path, so this cannot introduce a deadlock. The hook defaults to a no-op, so custom `SendContextResolver` implementations (tests, benches) are unaffected unless they opt in. Group stanza preparation uses `sort_dedup_by_user()` to deduplicate participants before device resolution, and `sort_dedup_by_device()` to deduplicate resolved device JIDs after LID conversion — both operate in-place on sorted `Vec` without `HashSet` allocations. ### Background Saver Periodic persistence with dirty flag optimization: ```rust theme={null} impl PersistenceManager { pub fn run_background_saver(self: Arc, runtime: Arc, interval: Duration) { runtime.spawn(Box::pin(async move { loop { // Wait for notification or interval self.save_to_disk().await; } })); } } ``` ## Feature Organization **Location:** `src/features/` ``` features/ ├── mod.rs // Feature exports ├── blocking.rs // Block/unblock contacts ├── chat_actions.rs // Archive, pin, mute, star ├── chatstate.rs // Typing indicators ├── community.rs // Community management ├── contacts.rs // Contact operations ├── groups.rs // Group management ├── media_reupload.rs // Media re-upload for retry ├── mex.rs // GraphQL MEX queries ├── newsletter.rs // Newsletter/channel operations ├── polls.rs // Poll creation and voting ├── presence.rs // Presence updates ├── profile.rs // Profile management ├── signal.rs // Signal protocol feature operations ├── status.rs // Status updates └── tctoken.rs // Trusted contact tokens ``` Media upload and download operations are in `src/download.rs` and `src/upload.rs` as separate top-level modules. **Pattern:** Features are accessed through accessor methods on `Client`: ```rust theme={null} // Access features through the client let metadata = client.groups().get_metadata(&group_jid).await?; let result = client.groups().create_group(options).await?; client.presence().set_available().await?; ``` ## State management flow ```mermaid theme={null} sequenceDiagram participant User participant Client participant PM as PersistenceManager participant Device participant Backend User->>Client: Operation Client->>PM: process_command(DeviceCommand) PM->>Device: modify_device() PM->>PM: Set dirty flag PM->>PM: Notify saver Note over PM: Background task PM->>Backend: save() Backend->>Backend: Persist to disk ``` ## Best Practices ### State Management ```rust ✅ Correct theme={null} // Use DeviceCommand for state changes client.persistence_manager .process_command(DeviceCommand::SetPushName(name)) .await; ``` ```rust ❌ Wrong theme={null} // Never modify Device directly let mut device = client.device.write().await; device.push_name = name; // DON'T DO THIS ``` ### Async Operations ```rust ✅ Correct theme={null} // Wrap blocking I/O in spawn_blocking let result = tokio::task::spawn_blocking(move || { // Heavy crypto or blocking HTTP expensive_operation() }).await?; ``` ```rust ❌ Wrong theme={null} // Never block the async runtime let result = expensive_operation(); // Stalls all tasks ``` ### Error Handling ```rust theme={null} use thiserror::Error; use anyhow::Result; #[derive(Debug, Error)] pub enum SocketError { #[error("socket is closed")] SocketClosed, #[error("noise cipher operation failed")] Cipher(#[from] NoiseError), } // Use anyhow::Result for functions with multiple error types pub async fn complex_operation() -> Result<()> { // Automatically converts errors with ? socket_operation()?; storage_operation()?; Ok(()) } ``` Error variants across the workspace preserve typed sources (via `#[from]` or `#[source]`) instead of stringifying inner errors. Callers can walk `std::error::Error::source()` to downcast to the original cause. ## Related Sections Learn about QR code and pair code flows Understand the event system and handlers Explore storage backends and state management Build your first WhatsApp bot