> ## Documentation Index > Fetch the complete documentation index at: https://personal-9eca1d6c-claude-nifty-bohr-567oke.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Storage > Storage backends, PersistenceManager, and state management in whatsapp-rust ## Overview WhatsApp-Rust uses a layered storage architecture with pluggable backends. The `PersistenceManager` manages all state changes, while the `Backend` trait defines storage operations for device data, Signal protocol keys, app state sync, and protocol-specific data. ## Architecture ```mermaid theme={null} graph TB A[Client] --> B[PersistenceManager] B --> C[Device State] B --> D[Backend Trait] D --> E[SignalStore] D --> F[AppSyncStore] D --> G[ProtocolStore] D --> H[DeviceStore] E --> I[SqliteStore] F --> I G --> I H --> I I --> J[SQLite Database] ``` ## PersistenceManager **Location:** `src/store/persistence_manager.rs` ### Purpose Manages all device state changes and persistence operations. Acts as the gatekeeper for state modifications. ### Structure ```rust theme={null} pub struct PersistenceManager { device: Arc>, device_snapshot: std::sync::RwLock>, backend: Arc, dirty: Arc, save_notify: Arc, // event_listener::Event (runtime-agnostic) } ``` **Fields:** * `device`: In-memory device state (protected by `tokio::sync::RwLock`; write-locked only during mutations) * `device_snapshot`: Cached `Arc` rebuilt under the write guard on every mutation — read by `get_device_snapshot()` with no contention against writers * `backend`: Storage backend implementation * `dirty`: Flag indicating unsaved changes * `save_notify`: Notification channel for background saver (uses `event_listener::Event` for runtime-agnostic operation) ### Initialization ```rust theme={null} impl PersistenceManager { pub async fn new(backend: Arc) -> Result { // Ensure device row exists let exists = backend.exists().await?; if !exists { backend.create().await?; } // Load existing data or create new let device_data = backend.load().await?; let device = if let Some(data) = device_data { let mut dev = Device::new(backend.clone()); dev.load_from_serializable(data); dev } else { Device::new(backend.clone()) }; let snapshot = Arc::new(device.clone()); Ok(Self { device: Arc::new(tokio::sync::RwLock::new(device)), device_snapshot: std::sync::RwLock::new(snapshot), backend, dirty: Arc::new(AtomicBool::new(false)), save_notify: Arc::new(Event::new()), }) } } ``` ### Key Methods #### get\_device\_snapshot **Purpose:** Read-only access to device state ```rust theme={null} pub fn get_device_snapshot(&self) -> Arc { self.device_snapshot.read().unwrap().clone() } ``` **Usage:** ```rust theme={null} let snapshot = client.persistence_manager.get_device_snapshot(); println!("Device ID: {:?}", snapshot.pn); println!("Push Name: {}", snapshot.push_name); ``` #### modify\_device **Purpose:** Modify device state with automatic dirty tracking ```rust theme={null} pub async fn modify_device(&self, modifier: F) -> R where F: FnOnce(&mut Device) -> R, { let mut device_guard = self.device.write().await; let result = modifier(&mut device_guard); // Rebuild the cached snapshot before releasing the write lock *self.device_snapshot.write().unwrap() = Arc::new(device_guard.clone()); self.dirty.store(true, Ordering::Relaxed); self.save_notify.notify(1); result } ``` **Usage:** ```rust theme={null} client.persistence_manager.modify_device(|device| { device.push_name = "New Name".to_string(); }).await; ``` #### process\_command **Purpose:** Apply state changes via `DeviceCommand` ```rust theme={null} pub async fn process_command(&self, command: DeviceCommand) { self.modify_device(|device| { apply_command_to_device(device, command); }).await; } ``` **Usage:** ```rust theme={null} client.persistence_manager .process_command(DeviceCommand::SetPushName("New Name".to_string())) .await; ``` ### Background Saver **Purpose:** Periodically persist dirty state to disk ```rust theme={null} pub fn run_background_saver(self: Arc, runtime: Arc, interval: Duration) { runtime.spawn(Box::pin(async move { loop { let listener = self.save_notify.listen(); // Race: wake on notification or after interval futures_lite::future::or(listener, async { // sleep using runtime abstraction runtime.sleep(interval).await; }).await; if let Err(e) = self.save_to_disk().await { error!("Error saving device state: {e}"); } } })); } async fn save_to_disk(&self) -> Result<(), StoreError> { if self.dirty.swap(false, Ordering::AcqRel) { let device_guard = self.device.read().await; let serializable_device = device_guard.to_serializable(); drop(device_guard); self.backend.save(&serializable_device).await?; } Ok(()) } ``` **Behavior:** * Wakes up when notified or after interval * Only saves if dirty flag is set * Uses optimistic locking (dirty flag) **Start background saver:** ```rust theme={null} let persistence_manager = Arc::new(PersistenceManager::new(backend).await?); persistence_manager.clone().run_background_saver(runtime.clone(), Duration::from_secs(30)); ``` ## Backend Trait **Location:** `wacore/src/store/traits.rs` ### Overview The `Backend` trait is automatically implemented for any type that implements all four domain-specific traits: ```rust theme={null} pub trait Backend: SignalStore + AppSyncStore + ProtocolStore + DeviceStore + Send + Sync {} ``` ### Domain Traits ## SignalStore **Purpose:** Signal protocol cryptographic operations ```rust theme={null} use bytes::Bytes; // from the `bytes` crate #[async_trait] pub trait SignalStore: Send + Sync { // Identity Operations async fn put_identity(&self, address: &str, key: [u8; 32]) -> Result<()>; async fn load_identity(&self, address: &str) -> Result>; async fn delete_identity(&self, address: &str) -> Result<()>; // Session Operations async fn get_session(&self, address: &str) -> Result>; async fn put_session(&self, address: &str, session: &[u8]) -> Result<()>; async fn delete_session(&self, address: &str) -> Result<()>; async fn has_session(&self, address: &str) -> Result; // has default impl using get_session // PreKey Operations async fn store_prekey(&self, id: u32, record: &[u8], uploaded: bool) -> Result<()>; async fn store_prekeys_batch(&self, keys: &[(u32, Bytes)], uploaded: bool) -> Result<()>; // has default impl looping over store_prekey async fn load_prekey(&self, id: u32) -> Result>; async fn load_prekeys_batch(&self, ids: &[u32]) -> Result>; // has default impl looping over load_prekey async fn remove_prekey(&self, id: u32) -> Result<()>; /// Mark already-stored prekeys as uploaded. UPDATE semantics — a key /// consumed (deleted) between the upload snapshot and this call must stay /// deleted, never resurrected by an upsert. async fn mark_prekeys_uploaded(&self, ids: &[u32]) -> Result<()>; async fn get_max_prekey_id(&self) -> Result; // Signed PreKey Operations async fn store_signed_prekey(&self, id: u32, record: &[u8]) -> Result<()>; async fn load_signed_prekey(&self, id: u32) -> Result>>; async fn load_all_signed_prekeys(&self) -> Result)>>; async fn remove_signed_prekey(&self, id: u32) -> Result<()>; // Sender Key Operations (for groups) async fn put_sender_key(&self, address: &str, record: &[u8]) -> Result<()>; async fn get_sender_key(&self, address: &str) -> Result>>; async fn delete_sender_key(&self, address: &str) -> Result<()>; } ``` **Usage Example:** ```rust theme={null} // Store identity key for a contact backend.put_identity( "15551234567@s.whatsapp.net:0", identity_key ).await?; // Load session for decryption if let Some(session) = backend.get_session("15551234567@s.whatsapp.net:0").await? { // Decrypt message using session } ``` ### SignalStoreCache **Location:** `wacore/src/store/signal_cache.rs` (re-exported from `src/store/signal_cache.rs`) The `SignalStoreCache` provides an in-memory cache layer for Signal protocol state, matching WhatsApp Web's `SignalStoreCache` implementation. All crypto operations read and write through this cache, with database writes deferred to explicit `flush()` calls. Sessions and sender keys are cached as deserialized objects (`SessionRecord` and `SenderKeyRecord` respectively), matching WhatsApp Web's pattern where the JS object IS the cache. Serialization only happens during `flush()` — not on every `store_session` or `put_sender_key` call. Identity stores use `Arc<[u8]>` byte caches. ```rust theme={null} pub struct SignalStoreCache { sessions: Mutex, identities: Mutex, sender_keys: Mutex, } // Session object cache — no per-message serialize/deserialize struct SessionStoreState { cache: HashMap, Option>, // None = known-absent dirty: HashSet>, // Modified keys pending flush deleted: HashSet>, // Deleted keys pending flush } // Sender key object cache (same pattern as sessions) struct SenderKeyStoreState { cache: HashMap, Option>, // None = known-absent dirty: HashSet>, // Modified keys pending flush } // Byte cache for identities struct ByteStoreState { cache: HashMap, Option>>, // None = known-absent dirty: HashSet>, deleted: HashSet>, } ``` **Key features:** * **Session object cache**: Sessions are stored as `SessionRecord` objects, eliminating protobuf encode/decode from the per-message path. Cold loads deserialize from backend bytes once and cache the object; subsequent reads return the cached object directly * **Sender key object cache**: Sender keys are stored as `SenderKeyRecord` objects (same pattern as sessions), eliminating serialize/deserialize from the per-message group encryption path. Cold loads deserialize from backend bytes once and cache the object * **`Arc` previous sessions**: `SessionRecord.previous_sessions` is wrapped in `Arc>`, making clone O(1) for the \~40 archived sessions. Only rare paths (archive, promote, take/restore) trigger `Arc::make_mut` * **Owned `store_session`**: The `store_session` method takes `SessionRecord` by value, enabling zero-cost moves from the protocol layer. The compiler enforces no use-after-store * **Deferred writes**: Changes are accumulated in memory and batch-written on `flush()`. Sessions and sender keys are serialized only during `flush()`, not on every store * **Redundant write elimination**: For identities (which rarely change), `put_dedup()` compares incoming bytes against the cached value and skips if identical * **Negative caching**: Known-absent keys are cached as `None` to avoid repeated DB lookups * **Independent locking**: Sessions, identities, and sender keys each have their own mutex * **O(1) key cloning**: Keys stored as `Arc` so cloning a key is a refcount bump instead of a heap allocation. The `key_for()` method reuses existing `Arc` keys from the HashMap via `get_key_value()`, avoiding heap allocation on the hot path * **Single-allocation keys**: Session lock keys use `to_protocol_address_string()` (format: `user[:device]@server.0`) which builds the key string in one allocation, avoiding the two-allocation overhead of constructing a `ProtocolAddress` then calling `.to_string()`. See [Signal Protocol performance](/advanced/signal-protocol#single-allocation-session-lock-keys) for details **Cache operations:** ```rust theme={null} // Read (loads from backend if not cached, returns SessionRecord object) let session = cache.get_session(address, &backend).await?; // Check existence (cache hit is O(1), cold load caches for subsequent get) let exists = cache.has_session(address, &backend).await?; // Write (takes ownership, marks as dirty, doesn't hit backend) cache.put_session(address, record).await; // Delete (marks for deletion on flush) cache.delete_session(address).await; // Persist all dirty state to backend (sessions serialized here) cache.flush(&backend).await?; // Clear cache (on disconnect/reconnect, retains allocated capacity) cache.clear().await; ``` **Flush behavior:** * Acquires all three mutexes to ensure consistency * Sessions are serialized to bytes only during flush (not on every `put_session`) * Only clears dirty tracking after ALL writes succeed * On failure, dirty state is preserved for retry on next flush ## AppSyncStore **Purpose:** WhatsApp app state synchronization ```rust theme={null} #[async_trait] pub trait AppSyncStore: Send + Sync { // Sync Keys async fn get_sync_key(&self, key_id: &[u8]) -> Result>; async fn set_sync_key(&self, key_id: &[u8], key: AppStateSyncKey) -> Result<()>; // Version Tracking async fn get_version(&self, name: &str) -> Result; async fn set_version(&self, name: &str, state: HashState) -> Result<()>; // Mutation MACs async fn put_mutation_macs( &self, name: &str, version: u64, mutations: &[AppStateMutationMAC], ) -> Result<()>; async fn get_mutation_mac(&self, name: &str, index_mac: &[u8]) -> Result>>; // Batched previous-MAC lookup (v0.6); default loops over get_mutation_mac. // index_macs are inline [u8; 32] (an index MAC is always a full // HMAC-SHA256 output) — no per-MAC heap allocation on either side. async fn get_mutation_macs( &self, name: &str, index_macs: &[[u8; 32]], ) -> Result>>; async fn delete_mutation_macs(&self, name: &str, index_macs: &[Vec]) -> Result<()>; async fn get_latest_sync_key_id(&self) -> Result>>; } ``` **AppStateSyncKey Structure:** ```rust theme={null} #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct AppStateSyncKey { pub key_data: Vec, pub fingerprint: Vec, pub timestamp: i64, } ``` **Collections:** * `critical_block` - Blocked contacts, push names * `regular_high` - Mute settings, starred messages, contact info * `regular_low` - Archive settings, pin settings * `regular` - Other chat settings ## ProtocolStore **Purpose:** WhatsApp Web protocol-specific storage ```rust theme={null} #[async_trait] pub trait ProtocolStore: Send + Sync { // Per-Device Sender Key Tracking (matches WA Web's participant.senderKey Map) // Updated only AFTER the server ACKs the message stanza, // preventing stale entries from network failures mid-send. async fn get_sender_key_devices(&self, group_jid: &str) -> Result>; async fn set_sender_key_status(&self, group_jid: &str, entries: &[(&str, bool)]) -> Result<()>; async fn clear_sender_key_devices(&self, group_jid: &str) -> Result<()>; // Clear all sender key device tracking across ALL groups. // Provided for completeness; not invoked on identity change — WhatsApp Web's // `WAWebUpdateLocalSignalSession` only calls `markForgetSenderKey` per-group/per-device // on retry receipts, so a global wipe here would empty the tracker too aggressively. async fn clear_all_sender_key_devices(&self) -> Result<()>; // LID-PN Mapping (Long-term ID to Phone Number) async fn get_lid_mapping(&self, lid: &str) -> Result>; async fn get_pn_mapping(&self, phone: &str) -> Result>; async fn put_lid_mapping(&self, entry: &LidPnMappingEntry) -> Result<()>; async fn get_all_lid_mappings(&self) -> Result>; // Base Key Collision Detection async fn save_base_key(&self, address: &str, message_id: &str, base_key: &[u8]) -> Result<()>; async fn has_same_base_key(&self, address: &str, message_id: &str, current_base_key: &[u8]) -> Result; async fn delete_base_key(&self, address: &str, message_id: &str) -> Result<()>; // Device Registry async fn update_device_list(&self, record: DeviceListRecord) -> Result<()>; async fn get_devices(&self, user: &str) -> Result>; // Delete a device list record, forcing a network re-fetch on next query. async fn delete_devices(&self, user: &str) -> Result<()>; // TcToken Storage (Trusted Contact Tokens) async fn get_tc_token(&self, jid: &str) -> Result>; async fn put_tc_token(&self, jid: &str, entry: &TcTokenEntry) -> Result<()>; async fn delete_tc_token(&self, jid: &str) -> Result<()>; async fn get_all_tc_token_jids(&self) -> Result>; // Pruned only when BOTH the received-token cutoff and the sender-bucket // cutoff have passed, so recent sender-side state outlives an expired token. async fn delete_expired_tc_tokens(&self, token_cutoff: i64, sender_cutoff: i64) -> Result; // Advance-only upsert of the sender-side issuance timestamp; inserts a // byte-less placeholder if no entry exists yet. Defaulted (read-modify-write). async fn touch_tc_token_sender_timestamp(&self, jid: &str, sender_timestamp: i64) -> Result<()> { /* default */ } // Store a token received from a contact without clobbering an existing // sender_timestamp. Newer-wins: a stale write (older token_timestamp, // unless the existing token is a byte-less placeholder) is a no-op. // Defaulted (read-modify-write); SqliteStore/InMemoryBackend enforce it atomically. async fn store_received_tc_token(&self, jid: &str, token: &[u8], token_timestamp: i64) -> Result<()> { /* default */ } // Sent Message Store (retry support, matches WA Web's getMessageTable) async fn store_sent_message(&self, chat_jid: &str, message_id: &str, payload: &[u8]) -> Result<()>; async fn take_sent_message(&self, chat_jid: &str, message_id: &str) -> Result>>; async fn delete_expired_sent_messages(&self, cutoff_timestamp: i64) -> Result; } ``` **LidPnMappingEntry:** ```rust theme={null} #[derive(Debug, Clone, Serialize, Deserialize)] pub struct LidPnMappingEntry { pub lid: String, pub phone_number: String, pub created_at: i64, pub updated_at: i64, pub learning_source: String, } ``` ### LidPnCache **Location:** `src/lid_pn_cache.rs` The `LidPnCache` provides an in-memory cache for LID to phone number mappings, used for Signal address resolution. WhatsApp Web uses LID-based addresses for Signal sessions when available. ```rust theme={null} pub struct LidPnCache { lid_to_entry: TypedCache, Arc>, pn_to_entry: TypedCache, Arc>, } ``` **Defaults match `WAWebLidPnCache`:** * **Time-based expiry**: none — entries do not idle out * **Capacity**: effectively unbounded (`u64::MAX`; no dedicated `unbounded()` builder) A custom `CacheEntryConfig` can impose a capacity bound if memory pressure requires it. Be aware that capacity-LRU eviction silently downgrades Signal addresses from `@lid` (or `@hosted.lid`) back to `@c.us`, which can cause `SessionNotFound` decryption failures. Notably, this keeps `status@broadcast` participants resolving to `@lid` for the entire session, matching WA Web behavior. **Bidirectional lookups:** ```rust theme={null} // Get LID for a phone number let lid = cache.get_current_lid("15551234567").await; // Get phone number for a LID let phone = cache.get_phone_number("100000012345678").await; ``` Since v0.6 `get_current_lid` (and the `SendContextResolver::get_lid_for_phone` trait method) returns `Option` instead of `Option`. This avoids a heap allocation on the hot group-send path, since LID identifiers fit inline in a `CompactString`. Call `.as_deref()` for `&str` comparisons, and update any custom `SendContextResolver` implementation to the new return type. **Timestamp conflict resolution:** When multiple LIDs exist for the same phone number, the entry with the most recent `created_at` timestamp wins for the PN → LID lookup: ```rust theme={null} // Add mapping (only updates PN map if newer timestamp) cache.add(entry).await; ``` **Initialization:** ```rust theme={null} // Warm up cache from persistent storage on client init let entries = backend.get_all_lid_mappings().await?; cache.warm_up(entries).await; ``` **Session migration on LID discovery:** When a new LID-PN mapping is added to the cache, the client automatically migrates any Signal sessions stored under the PN address to the corresponding LID address. The migration reads and writes through the `SignalStoreCache` (not the backend directly) to avoid stale reads when the cache has unflushed mutations, then flushes the migrated state to the backend. This prevents `SessionNotFound` decryption failures when the phone switches from PN to LID addressing. See [Signal Protocol — PN→LID session migration](/advanced/signal-protocol#pnlid-session-migration) for details. **Learning sources:** * `usync` - User sync responses * `peer_pn_message` / `peer_lid_message` - Peer messages * `pairing` - Device pairing * `device_notification` - Device notifications * `blocklist_active` / `blocklist_inactive` - Blocklist operations **DeviceListRecord:** ```rust theme={null} #[derive(Debug, Clone, Serialize, Deserialize)] pub struct DeviceListRecord { pub user: String, pub devices: Vec, pub timestamp: i64, pub phash: Option, /// ADV raw_id from ADVKeyIndexList — used to detect identity changes. /// When this changes, all sessions and sender keys for the user must be cleared. pub raw_id: Option, } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct DeviceInfo { pub device_id: u32, pub key_index: Option, } ``` **TcTokenEntry:** ```rust theme={null} #[derive(Debug, Clone, Serialize, Deserialize)] pub struct TcTokenEntry { pub token: Vec, pub token_timestamp: i64, pub sender_timestamp: Option, } ``` ## DeviceStore **Purpose:** Device data persistence ```rust theme={null} #[async_trait] pub trait DeviceStore: Send + Sync { async fn save(&self, device: &Device) -> Result<()>; async fn load(&self) -> Result>; async fn exists(&self) -> Result; async fn create(&self) -> Result; async fn snapshot_db(&self, _name: &str, _extra_content: Option<&[u8]>) -> Result<()>; } ``` **Device Structure:** ```rust theme={null} // wacore/src/store/device.rs #[derive(Clone, Serialize, Deserialize)] pub struct Device { pub pn: Option, // Phone number JID pub lid: Option, // Long-term identifier pub push_name: String, // Display name pub registration_id: u32, // Signal registration ID pub adv_secret_key: [u8; 32], // Advertisement secret #[serde(with = "key_pair_serde")] pub identity_key: KeyPair, // Signal identity keypair #[serde(with = "key_pair_serde")] pub noise_key: KeyPair, // Noise protocol keypair #[serde(with = "key_pair_serde")] pub signed_pre_key: KeyPair, // Signal signed pre-key pub signed_pre_key_id: u32, #[serde(with = "BigArray")] pub signed_pre_key_signature: [u8; 64], #[serde(with = "account_serde", default)] pub account: Option>, pub app_version_primary: u32, pub app_version_secondary: u32, pub app_version_tertiary: u32, pub app_version_last_fetched_ms: i64, #[serde(skip)] pub device_props: wa::DeviceProps, // Not persisted #[serde(skip)] pub client_profile: ClientProfile, // Noise-handshake identity, not persisted #[serde(default)] pub edge_routing_info: Option>, // Optimized reconnection #[serde(default)] pub props_hash: Option, // A/B experiment tracking #[serde(default)] pub next_pre_key_id: u32, // Monotonic counter; advances at generation time (WA Web NEXT_PK_ID) #[serde(default)] pub first_unupload_pre_key_id: u32, // First generated-but-unuploaded key (WA Web FIRST_UNUPLOAD_PK_ID); 0 = unset/legacy #[serde(default)] pub server_has_prekeys: bool, // Whether server has prekeys uploaded #[serde(default)] pub nct_salt: Option>, // NCT salt for cstoken computation #[serde(skip)] pub nct_salt_sync_seen: bool, // Runtime flag: authoritative sync seen #[serde(default)] pub server_cert_chain: Option, // Cached for Noise IK resume #[serde(default)] pub login_counter: i32, // Per-connect counter sent as ClientPayload.lc #[serde(default)] pub lid_migrated: bool, // Account is 1:1-LID-migrated; gates DM wire addressing #[serde(default)] pub last_signed_pre_key_rotation_ms: i64, // WA Web RotateKeyJob cadence clock; 0 = pre-feature device, baselined (not rotated) on first connect } /// Cached form of the server's two-cert chain. `leaf.key` is the server /// static public key consumed by Noise IK; the intermediate is kept solely /// to mirror WA Web's expiry checks. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub struct CachedServerCertChain { pub intermediate: CachedNoiseCert, pub leaf: CachedNoiseCert, } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub struct CachedNoiseCert { pub key: [u8; 32], // 32-byte X25519 public key pub not_before: i64, // Validity window (Unix epoch seconds) pub not_after: i64, } ``` The `server_cert_chain` field is populated after a successful XX (or XXfallback) handshake and consumed on the next connect to attempt **Noise IK**. It is cleared via `DeviceCommand::ClearServerCertChain` if a resumed IK handshake fails crypto-fatally. See [Authentication — Noise Protocol Handshake](/concepts/authentication#noise-protocol-handshake). The `account` field uses a custom `account_serde` module to bridge buffa-generated protobuf types (which lack `serde::Deserialize`) into serde. It encodes `ADVSignedDeviceIdentity` to protobuf bytes on serialization and decodes them back on deserialization. The `#[serde(default)]` attribute ensures backward compatibility — old data missing this field deserializes as `None`. ## SqliteStore implementation **Location:** `storages/sqlite-storage/src/lib.rs` As of v0.5, the `whatsapp-rust-sqlite-storage` crate bundles SQLite by default via the `bundled-sqlite` feature. You no longer need SQLite installed as a system dependency. To link against a system SQLite instead, disable the default features on the crate. ### BLOB encoding Three BLOB columns use **protobuf (via `buffa`)** for on-disk encoding: | Table | Column | Domain type | | -------------------- | ------------------- | ----------------------- | | `device` | `server_cert_chain` | `CachedServerCertChain` | | `app_state_keys` | `key_data` | `AppStateSyncKey` | | `app_state_versions` | `state_data` | `HashState` | Protobuf's field-tagged wire format tolerates additive changes (adding or reordering fields) without corrupting persisted rows. The encode/decode helpers live in `storages/sqlite-storage/src/wire.rs`; `wacore` domain types are untouched — conversion happens only at the storage boundary. #### Upgrading from a bincode-encoded database No manual migration is needed. The storage layer self-heals lazily on first startup after upgrade: * A BLOB that cannot be decoded as protobuf (a legacy bincode row or genuine corruption) is treated as **absent** rather than an error. * App-state sync keys: the client re-requests the missing key from the primary device, which re-shares it. The next `set_sync_key` call overwrites the row in protobuf format. * App-state versions: the collection resets to version 0 and re-syncs from the server. The first `set_version` call overwrites the row in protobuf format. * Server cert chain: rebuilt on the next Noise XX handshake. The following `save_device_data` call overwrites the row in protobuf format. Net effect: settings re-sync automatically on the first connection after upgrade, with no data loss and no manual DB intervention required. ### Database schema The device table uses named columns for each field, making the schema self-documenting and reducing the risk of column mix-ups when fields are added: ```sql theme={null} -- Device table (named columns, not a serialized blob) CREATE TABLE device ( id INTEGER PRIMARY KEY AUTOINCREMENT, lid TEXT NOT NULL, pn TEXT NOT NULL, registration_id INTEGER NOT NULL, noise_key BLOB NOT NULL, identity_key BLOB NOT NULL, signed_pre_key BLOB NOT NULL, signed_pre_key_id INTEGER NOT NULL, signed_pre_key_signature BLOB NOT NULL, adv_secret_key BLOB NOT NULL, account BLOB, push_name TEXT NOT NULL DEFAULT '', app_version_primary INTEGER NOT NULL DEFAULT 0, app_version_secondary INTEGER NOT NULL DEFAULT 0, app_version_tertiary BIGINT NOT NULL DEFAULT 0, app_version_last_fetched_ms BIGINT NOT NULL DEFAULT 0, edge_routing_info BLOB, props_hash TEXT, next_pre_key_id INTEGER NOT NULL DEFAULT 0, first_unupload_pre_key_id INTEGER NOT NULL DEFAULT 0, -- WA Web FIRST_UNUPLOAD_PK_ID; 0 = unset/legacy server_has_prekeys BOOLEAN NOT NULL DEFAULT 0, nct_salt BLOB, server_cert_chain BLOB, -- Cached two-cert chain for Noise IK resume login_counter INTEGER NOT NULL DEFAULT 0, -- Per-connect counter; sent as ClientPayload.lc lid_migrated BOOLEAN NOT NULL DEFAULT 0, -- Account is 1:1-LID-migrated; gates DM wire addressing last_signed_pre_key_rotation_ms BIGINT NOT NULL DEFAULT 0 -- WA Web RotateKeyJob cadence clock; 0 = unset/legacy, baselined on first connect ); -- Signal protocol tables CREATE TABLE identities ( address TEXT NOT NULL, key BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (address, device_id) ); CREATE TABLE sessions ( address TEXT NOT NULL, record BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (address, device_id) ); CREATE TABLE prekeys ( id INTEGER NOT NULL, key BLOB NOT NULL, uploaded BOOLEAN NOT NULL DEFAULT FALSE, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (id, device_id) ); CREATE TABLE signed_prekeys ( id INTEGER NOT NULL, record BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (id, device_id) ); CREATE TABLE sender_keys ( address TEXT NOT NULL, record BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (address, device_id) ); -- App state sync tables CREATE TABLE app_state_keys ( key_id BLOB NOT NULL, key_data BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (key_id, device_id) ); CREATE TABLE app_state_versions ( name TEXT NOT NULL, state_data BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (name, device_id) ); CREATE TABLE app_state_mutation_macs ( name TEXT NOT NULL, version BIGINT NOT NULL, index_mac BLOB NOT NULL, value_mac BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (name, index_mac, device_id) ); -- Protocol tables -- Unified per-device sender key tracking (matches WA Web's participant.senderKey Map) CREATE TABLE sender_key_devices ( group_jid TEXT NOT NULL, device_jid TEXT NOT NULL, has_key INTEGER NOT NULL DEFAULT 0, device_id INTEGER NOT NULL DEFAULT 1, updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), PRIMARY KEY (group_jid, device_jid, device_id) ); CREATE TABLE lid_pn_mapping ( lid TEXT NOT NULL, phone_number TEXT NOT NULL, created_at BIGINT NOT NULL, learning_source TEXT NOT NULL, updated_at BIGINT NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, PRIMARY KEY (lid, device_id) ); CREATE TABLE base_keys ( address TEXT NOT NULL, message_id TEXT NOT NULL, base_key BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), PRIMARY KEY (address, message_id, device_id) ); CREATE TABLE device_registry ( user_id TEXT NOT NULL, devices_json TEXT NOT NULL, timestamp INTEGER NOT NULL, phash TEXT, device_id INTEGER NOT NULL DEFAULT 1, updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), PRIMARY KEY (user_id, device_id) ); CREATE TABLE tc_tokens ( jid TEXT NOT NULL, token BLOB NOT NULL, token_timestamp INTEGER NOT NULL, sender_timestamp INTEGER, device_id INTEGER NOT NULL DEFAULT 1, updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), PRIMARY KEY (jid, device_id) ); -- Sent message store for retry handling CREATE TABLE sent_messages ( chat_jid TEXT NOT NULL, message_id TEXT NOT NULL, payload BLOB NOT NULL, device_id INTEGER NOT NULL DEFAULT 1, created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), PRIMARY KEY (chat_jid, message_id, device_id) ); CREATE INDEX idx_sent_messages_created ON sent_messages (created_at, device_id); ``` ### Multi-Account Support **Each device has unique `device_id`:** ```rust theme={null} use whatsapp_rust::store::SqliteStore; // Account 1 let backend1 = Arc::new(SqliteStore::new_for_device("whatsapp.db", 1).await?); let pm1 = PersistenceManager::new(backend1).await?; // Account 2 let backend2 = Arc::new(SqliteStore::new_for_device("whatsapp.db", 2).await?); let pm2 = PersistenceManager::new(backend2).await?; ``` **All tables scoped by `device_id`:** ```sql theme={null} SELECT * FROM sessions WHERE device_id = 1 AND address = ?; ``` ### Memory and Thread Tuning (`SqliteStoreConfig`) Each session owns its own `SqliteStore`, so the store's fixed costs (thread-pool threads, pooled connections, page cache) multiply with session count. On a host running many sessions the defaults from `SqliteStore::new` / `new_for_device` are deliberately low-memory; `SqliteStoreConfig` lets you override them when you need more throughput on a single session. **Default (low-memory) profile:** | Setting | Default | Notes | | ---------------- | ------------------------ | -------------------------------------------------------------- | | `pool_size` | `1` | Single connection; store serializes via its internal semaphore | | `cache_size_kib` | `512` | Per-connection page cache (\~512 KiB) | | `busy_timeout` | `30 s` | SQLite busy-timeout | | `synchronous` | `NORMAL` | WAL-safe durability | | `thread_pool` | shared process-wide pool | One lazy pool across all stores; \~0 idle threads per store | **`SqliteStoreConfig` struct:** ```rust theme={null} pub struct SqliteStoreConfig { /// Connection pool size. Drives both r2d2 `max_size` and the /// internal semaphore permits, so they always stay in sync. pub pool_size: u32, /// Per-connection SQLite page cache in KiB. pub cache_size_kib: u32, /// SQLite busy-timeout. pub busy_timeout: Duration, /// SQLite PRAGMA synchronous level. pub synchronous: Synchronous, /// Optional shared r2d2 thread pool. When `None`, the store uses the /// process-wide lazy pool. Inject a custom pool (e.g. for tests) via this field. pub thread_pool: Option>, } ``` **Constructors:** ```rust theme={null} use whatsapp_rust::store::{SqliteStore, SqliteStoreConfig}; // Low-memory default — same as SqliteStore::new() let backend = Arc::new(SqliteStore::with_config("whatsapp.db", SqliteStoreConfig::default()).await?); // Low-memory default, specific device_id — same as SqliteStore::new_for_device() let backend = Arc::new( SqliteStore::with_config_for_device("whatsapp.db", 1, SqliteStoreConfig::default()).await?, ); // High-throughput profile for a single busy session let backend = Arc::new( SqliteStore::with_config_for_device( "whatsapp.db", 1, SqliteStoreConfig { pool_size: 4, cache_size_kib: 4096, ..SqliteStoreConfig::default() }, ) .await?, ); ``` `SqliteStore::new` and `SqliteStore::new_for_device` are unchanged — they delegate to `SqliteStoreConfig::default()` internally. Raising `pool_size` above `1` also raises the internal semaphore permits in lockstep, so real concurrency is available without the store serializing every operation. **Multi-session measurements (from PR #926):** | | before | after (default config) | | ------------------------- | --------- | ---------------------- | | threads / store | \~3.0 | \~0 (shared, flat) | | 50 stores → total threads | 164 | 16 | | peak heap, 25 stores | 11.47 MiB | 9.69 MiB | At \~100 sessions the r2d2 thread count drops from \~300 to 2. ## DeviceCommand Pattern **Location:** `src/store/commands.rs`, `wacore/src/store/commands.rs` ### Purpose Provide type-safe, centralized state mutations. ### Command Enum ```rust theme={null} pub enum DeviceCommand { SetId(Option), SetLid(Option), SetPushName(String), SetPlatform(String), SetAccount(Option), /// Update both prekey watermarks atomically. Both must always move /// together — split updates are how the pre-watermark model lost track /// of generated keys. Replaces the removed `SetNextPreKeyId` variant. SetPreKeyWatermarks { next_pre_key_id: u32, first_unupload_pre_key_id: u32, }, SetNctSalt(Option>), SetNctSaltFromHistorySync(Vec), // ... more commands } ``` ### Command Application ```rust theme={null} pub fn apply_command_to_device(device: &mut Device, command: DeviceCommand) { match command { DeviceCommand::SetId(id) => { device.pn = id; } DeviceCommand::SetLid(lid) => { device.lid = lid; } DeviceCommand::SetPushName(name) => { device.push_name = name; } DeviceCommand::SetPlatform(platform) => { device.platform = platform; } DeviceCommand::SetAccount(account) => { device.account = account.map(std::sync::Arc::new); } DeviceCommand::SetPreKeyWatermarks { next_pre_key_id, first_unupload_pre_key_id } => { device.next_pre_key_id = next_pre_key_id; device.first_unupload_pre_key_id = first_unupload_pre_key_id; } // ... } } ``` ### Usage ```rust theme={null} // ✅ Correct: Use DeviceCommand client.persistence_manager .process_command(DeviceCommand::SetPushName("New Name".to_string())) .await; // ❌ Wrong: Direct modification let mut device = client.device.write().await; device.push_name = "New Name".to_string(); // DON'T DO THIS ``` ## State management best practices ### Read-Only Access ```rust theme={null} // Cheap snapshot for read-only access — sync, no .await let snapshot = client.persistence_manager.get_device_snapshot(); println!("JID: {:?}", snapshot.pn); ``` ### Modifications ```rust theme={null} // Use process_command for type-safe mutations client.persistence_manager .process_command(DeviceCommand::SetPushName(name)) .await; ``` ### Bulk Operations ```rust theme={null} // Use modify_device for multiple changes client.persistence_manager.modify_device(|device| { device.push_name = "New Name".to_string(); device.platform = "Chrome".to_string(); }).await; ``` ### Critical Errors ```rust theme={null} // Create snapshot for debugging crypto failures client.persistence_manager .create_snapshot("decrypt_failure", Some(&failed_message_bytes)) .await?; ``` ## Custom backend implementation ### Example: PostgreSQL backend ```rust theme={null} use async_trait::async_trait; use wacore::store::traits::*; pub struct PostgresStore { pool: sqlx::PgPool, device_id: i32, } #[async_trait] impl SignalStore for PostgresStore { async fn put_identity(&self, address: &str, key: [u8; 32]) -> Result<()> { sqlx::query( "INSERT INTO identities (device_id, address, key) VALUES ($1, $2, $3) ON CONFLICT (device_id, address) DO UPDATE SET key = $3" ) .bind(self.device_id) .bind(address) .bind(&key[..]) .execute(&self.pool) .await?; Ok(()) } // ... implement other methods } #[async_trait] impl AppSyncStore for PostgresStore { // ... implement methods } #[async_trait] impl ProtocolStore for PostgresStore { // ... implement methods } #[async_trait] impl DeviceStore for PostgresStore { // ... implement methods } // Backend trait automatically implemented ``` ### Usage ```rust theme={null} let backend = Arc::new(PostgresStore::new(pool, device_id)); let pm = PersistenceManager::new(backend).await?; ``` ## Migration & Debugging ### Database Snapshots **Feature flag:** `debug-snapshots` ```toml theme={null} [dependencies] whatsapp-rust = { version = "0.6", features = ["debug-snapshots"] } ``` **Usage:** ```rust theme={null} // Trigger snapshot on critical errors if let Err(e) = decrypt_message(&msg).await { client.persistence_manager .create_snapshot( &format!("decrypt_failure_{}", msg.id), Some(&serialized_message) ) .await?; return Err(e); } ``` **Output:** ``` snapshots/ ├── decrypt_failure_1234567890_whatsapp.db └── decrypt_failure_1234567890_extra.bin ``` ## Pluggable cache store **Location:** `src/cache_store.rs`, `src/cache_config.rs`, `wacore/src/store/cache.rs` ### Overview By default, whatsapp-rust uses in-process `PortableCache` caches for group metadata, device lists, device registry, and LID-PN mappings. The pluggable cache store adapter lets you replace any of these with an external backend (Redis, Memcached, etc.) by implementing the `CacheStore` trait. ### PortableCache `PortableCache` is the client's sole in-process cache backend on every target, including wasm32. It is a platform-agnostic, runtime-independent implementation. `PortableCache` supports: * **Maximum capacity** with oldest-inserted (FIFO) eviction * **Time-to-live (TTL)** — entries expire a fixed duration after insertion * **Time-to-idle (TTI)** — entries expire after a fixed duration of no access * **Monotonic expiry** — TTL/TTI use `wacore::time::Instant` (not the wall clock), so system-clock jumps cannot expire entries early * **Single-flight `get_with`** — concurrent initializations for the same key coalesce into a single call, which is critical for caches storing coordination primitives (mutexes, channels) * **Eager init-lock reclamation** — `get_with`/`get_with_by_ref` drop a key's init lock once no other caller holds it, preventing unbounded `init_locks` growth in high-cardinality caches (session locks, chat lanes, dedup) * **Reliable async `clear()`** — awaits the write lock; prefer this over the best-effort sync `invalidate_all()` in async contexts * **`snapshot_entries()`** — reliable awaited snapshot of `(Arc, V)` pairs for invalidation passes; `iter()` is a best-effort sync spin that can yield an empty snapshot under write contention ```mermaid theme={null} graph TB A[Client] --> B[TypedCache] B -->|Default| C[PortableCache In-Process] B -->|Custom| D[CacheStore Trait] D --> E[Redis] D --> F[Memcached] D --> G[Your Backend] ``` ### CacheStore trait ```rust theme={null} #[async_trait] pub trait CacheStore: Send + Sync + 'static { async fn get(&self, namespace: &str, key: &str) -> anyhow::Result>>; async fn set(&self, namespace: &str, key: &str, value: &[u8], ttl: Option) -> anyhow::Result<()>; async fn delete(&self, namespace: &str, key: &str) -> anyhow::Result<()>; async fn clear(&self, namespace: &str) -> anyhow::Result<()>; async fn entry_count(&self, namespace: &str) -> anyhow::Result { Ok(0) } } ``` Each logical cache uses a unique namespace string (e.g., `"group"`, `"device"`, `"lid_pn_by_lid"`). Implementations should partition keys by namespace — for example, a Redis implementation might prefix keys as `{namespace}:{key}`. Cache operations are best-effort. The client falls back gracefully when cache reads fail (treats as miss) and logs warnings on write failures. ### TypedCache `TypedCache` is a generic wrapper that dispatches to either the in-process `PortableCache` or a custom `CacheStore` backend. The in-process path has zero extra overhead — values are stored in memory without any serialization. The custom-store path serializes values with `serde_json` and keys via `Display`. ```rust theme={null} // In-process path (zero overhead) let cache = TypedCache::from_local(portable_cache); // Custom store path (serde_json serialization) let cache = TypedCache::from_store(store, "group", Some(Duration::from_secs(3600))); ``` `CacheEntryConfig` provides a `build_typed_ttl` convenience method that automatically selects the right backend: if a custom `CacheStore` is provided, it creates a `TypedCache` backed by the store; otherwise it falls back to an in-process `PortableCache`. ```rust theme={null} let cache: TypedCache> = config.group_cache.build_typed_ttl( config.cache_stores.group_cache.clone(), "group", ); ``` The group cache stores `Arc` so that warm sends and repeated `query_info` calls reuse the same snapshot instead of deep-cloning the participants list. #### `invalidate_all` and `clear` `TypedCache` provides two ways to remove all entries: * **`invalidate_all()`** — synchronous. For the in-process backend this is a best-effort spin; under sustained write contention it can silently skip the clear. For custom `CacheStore` backends, it spawns a fire-and-forget task via `tokio::runtime::Handle::try_current()`, which **requires the `tokio-runtime` feature**. Without `tokio-runtime` enabled, the clear is skipped and a warning is logged. * **`clear()`** — async. Awaits completion for custom backends and is the recommended approach when you need to ensure all entries are removed. If you disable the `tokio-runtime` feature and use a custom `CacheStore` backend, `invalidate_all()` will silently skip clearing the external store. Use the async `clear()` method instead. ### CacheStores configuration The `CacheStores` struct controls which caches use custom backends: ```rust theme={null} pub struct CacheStores { pub group_cache: Option>, pub device_registry_cache: Option>, pub lid_pn_cache: Option>, } ``` Fields left as `None` keep the default in-process `PortableCache` behaviour. Use `CacheStores::all(store)` to set the same backend for all pluggable caches at once. Coordination caches (`session_locks`, `chat_lanes`), the signal write-behind cache, and `pdo_pending_requests` always stay in-process — they hold live Rust objects (mutexes, channel senders) that cannot be serialized to an external store. ### Usage with Bot builder ```rust theme={null} use whatsapp_rust::{CacheConfig, CacheStores}; use std::sync::Arc; let redis = Arc::new(MyRedisCacheStore::new("redis://localhost:6379")); let bot = Bot::builder() .with_backend(backend) .with_transport_factory(transport) .with_http_client(http_client) .with_cache_config(CacheConfig { cache_stores: CacheStores { group_cache: Some(redis.clone()), device_registry_cache: Some(redis.clone()), ..Default::default() }, ..Default::default() }) .build() .await?; ``` Or use `CacheStores::all()` to route all pluggable caches to the same backend: ```rust theme={null} let config = CacheConfig { cache_stores: CacheStores::all(redis.clone()), ..Default::default() }; ``` See [Custom backends guide](/guides/custom-backends#custom-cache-store) for a full implementation example. ## Granular cache patching Instead of invalidating a cache entry and re-fetching from the server on every change notification, whatsapp-rust applies granular patches to cached values in place. This eliminates an extra IQ round-trip per update and keeps caches consistent in real time. ### How it works All patching follows the same pattern: read the cached value, mutate it, and write it back. There is no atomic compare-and-swap — the `get` → mutate → `insert` sequence can race with concurrent notifications, but this is acceptable because the cache is best-effort and a full server fetch on the next read corrects any drift. ```rust theme={null} // Pseudocode for the patching pattern if let Some(mut cached) = cache.get(&key).await { cached.apply_change(notification_data); cache.insert(key, cached).await; } // If the key isn't cached, the patch is a no-op — // the next read fetches authoritative state from the server. ``` ### Patched cache domains Granular patching is implemented in three areas: **Device registry** (`src/client/device_registry.rs`) When a device notification arrives, the client patches the `device_registry_cache` (keyed by user string, stores `DeviceListRecord`): * `patch_device_add` — appends a new device JID to the cached list and persists the updated `DeviceListRecord` to the backend store * `patch_device_remove` — removes a device by ID using `retain`, then persists * `patch_device_update` — updates `key_index` on an existing device entry, then persists All three methods iterate over every known PN/LID alias for the user so stale alternate-key entries stay consistent. Device patches also persist to the backend store immediately, so changes survive cache eviction and restarts. The `patch_device_add` method also performs ADV (Account Device Verification) key index filtering. When `KeyIndexInfo` is present, the signed bytes are decoded via `wacore::adv::decode_key_index_list` to extract valid device indexes, and stale devices not in the valid set are filtered out. If the `raw_id` in the notification differs from the stored value, the client detects an identity change and clears all Signal sessions for that user's non-primary devices. When a notification arrives with only a hash (no device list), the client falls back to full invalidation and lets the next access re-fetch from the server. **LID migration**: When a new LID-PN mapping is discovered, the device registry re-keys entries from the PN key to the LID key via `migrate_device_registry_on_lid_discovery()`. This ensures lookups by either addressing scheme resolve to the same canonical record. The migration also invalidates stale PN-keyed entries from both cache and database. **Group metadata** (`src/handlers/notification.rs`, `src/features/groups.rs`) When participant add/remove notifications arrive, the cached `GroupInfo` is patched in place: * Participant adds use `GroupInfo::add_participants()`, which deduplicates by user and backfills LID-to-PN maps for LID-addressed groups * Participant removes use `GroupInfo::remove_participants()`, which also cleans up both LID-to-PN and PN-to-LID maps bidirectionally * API calls (`client.groups().add_participants()`, `client.groups().remove_participants()`) also patch the cache after a successful server response, filtering to only participants the server accepted (status 200) Group patches are cache-only — they are not persisted to the backend. If the cache entry is evicted, the next `query_info()` call re-fetches from the server. Only `leave()` uses full invalidation. **LID-PN mappings** (`src/lid_pn_cache.rs`) The `LidPnCache` uses timestamp-based conflict resolution when adding new mappings. The PN → entry map only updates if the new entry's `created_at` is newer than or equal to the existing entry's, preventing older stale mappings from overwriting newer ones. ### Patching vs. invalidation summary | Behavior | Granular patch | Invalidate + refetch | | ---------------- | --------------------------------------------------------------------- | ------------------------------------ | | **Network cost** | Zero — applies diff locally | One IQ round-trip per cache miss | | **Latency** | Immediate update | Stale until next access | | **Used when** | Device add/remove/update, group participant changes, LID-PN discovery | Group leave, hash-only device update | | **Atomicity** | Not atomic (can race, corrected on next full fetch) | N/A | | **Persistence** | Device patches persist immediately; group patches are cache-only | Backend remains authoritative | ## Related Sections Understand PersistenceManager's role Learn how session data is persisted Implement your own storage backend Complete storage API reference