Private message history sync: how it works and how to operate it

Related: requirements · plan · implementation and evidence · storage findings · sender-assistance implementation.

What the feature does

Private message history sync helps approved devices read older messages using encrypted access supplied by a device that can already read them. Your own devices can help with sent and received messages. A sender’s devices can also help you recover messages that sender originally sent to you. It runs automatically during authenticated visits, without a sync banner or a separate sharing prompt. This guide includes the sender-assisted extension; merge and production deployment are separate steps.

For example, Alice's laptop can read her old conversations, but her newly approved phone cannot. Alice visits the site on the laptop, then later opens Messages on the phone. The laptop leaves encrypted recovery material for the phone, allowing the phone to read and verify the original messages. The two devices do not need to be online together.

If Alice has lost every old device, Bob may still be able to help: after Alice’s phone is approved, Bob visits the site on a device that can read his messages to Alice. That visit leaves encrypted access for Alice’s phone. Bob can help restore his messages to Alice, not Alice’s messages to him. Bob receives no recovery report or read receipt. Any currently approved Bob key with verified access can contribute; it need not be the key that originally signed the message.

How recovery happens

1. A key is approved for the account. The application treats currently approved keys sharing the normalized username as that account's keys. All those keys are authorized for its history, including keys approved before this feature existed. Original senders may supply access to their outgoing messages to those approved recipient keys; no additional sharing prompt is required.

2. A device with access visits the site. It can contribute from an ordinary authenticated page; opening Messages on that device is unnecessary. The browser checks a bounded portion of history and verifies messages it can already decrypt.

3. The browser prepares encrypted access. Each original message has a session key that unlocks its encrypted contents. The contributing browser signs a bundle of these keys and encrypts it specifically for an approved target device. The bundle identifies the destination account, source and target keys, and the exact original messages. Sender-assisted bundles additionally bind the source account and sender-assistance purpose.

4. The server retains the encrypted bundle. This is private recovery data, separate from conversations. It creates no chat messages and does not mark messages read. Original encrypted messages remain unchanged.

5. The target reads the originals. Its browser decrypts the bundle with its own private key, checks the donor and message bindings, then decrypts each original message and verifies the original sender's signature before displaying trusted text. A donor's signature alone is insufficient.

Transfers stay available after successful recovery, so reloading the target does not require another donor visit. Multiple devices can supply different parts of history, and a recovered device can help later approved devices. Large histories progress across bounded visits and visible returns; recovery is not guaranteed to finish in one visit. Existing per-message unavailable states and reading retries remain available.

What is protected, and what remains limited

• Private identity keys stay in their browsers. Message plaintext and unwrapped session keys are not sent to the server or added to persistent recovery caches.

• The server stores encrypted messages, encrypted recovery bundles and synchronization metadata. It can see account, key and message relationships; this does not hide that metadata from the operator.

• Only currently approved keys participate. Each contribution is either within one account or from the original sender’s account to the original recipient’s account. Authorization is checked for each message during upload, retrieval and confirmation. Removing approval prevents future eligible transfer use, but cannot retract secrets a device already received.

• Recovery requires an eligible device with suitable access or a retained eligible transfer. If neither exists, approval alone cannot restore history. Original messages must also still exist on the server.

• Sender assistance restores incoming messages only. It does not grant unrelated conversation access or allow a recipient to give history back to the sender. Sender-visible work and upload responses do not depend on recipient confirmations; the sender receives no recipient recovery counts or read receipts. Ordinary replies can still reveal that someone read a message.

• Messages/unread data and recovery data use separate private databases. A sync-store outage can interrupt historical recovery while ordinary sending and direct reading continue. Both stores need coordinated private backups; neither belongs in public downloads or offline snapshots.

The remaining sections document the API, exact limits, deployment, backup and rollback for technical readers and operators.

Storage and deployment

Deploy PHP and browser assets together. Messages and unread state remain in PRIVATE_MESSAGE_DATABASE_PATH (default state/private/messages.sqlite3). Sync uses its own lazy connection to PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH (default state/private/message_history_sync.sqlite3). Configure both paths outside the document root, canonical repository, static releases, offline snapshots, and public backups. Identical paths, symlinks and hardlinks to the message database are rejected. New directories use mode 0700 and the sync file 0600; preserve private permissions for backups and SQLite sidecars too.

The first authenticated sync request creates history_sync_transfers, history_sync_items, history_sync_coverage, history_sync_scan, and history_sync_sender_scan. The sender-assisted upgrade adds only the last table, with independent per-source sender checkpoints; existing transfer rows, account checkpoints and the original-message schema are preserved. No original-message migration or approval-event replay is required: existing approved keys are discovered too. The sync connection uses a one-second busy timeout and FULL synchronous durability, retaining SQLite's default rollback journal on newly created files. Operators should preserve this policy; separate files isolate writer locks, not disk load or original-message reads. There are no attached databases, cross-store foreign keys, or cross-store transactions.

Ordinary messaging initializes only its existing store. A missing/unwritable/unavailable sync path yields a generic retryable 503; Messages can still send and directly read messages. Never repair a sync error by deleting the retained transfers. A missing file can be recreated, but old coverage alone cannot recover the lost keys.

API contract

All endpoints require the existing authenticated, currently approved identity and return Cache-Control: no-store on successes and errors. Source authority derives from the authenticated account/key. An upload may name a destination account, but every original message must authorize that sender-to-recipient relationship and the target must currently belong to that destination. POST requires JSON, X-Requested-With: ForumPrivateMessages, and rejects cross-site/same-site fetches. Request bodies are limited to 100,000 bytes; unknown methods/actions return 405, unavailable authentication 401/403, invalid input 400, and unavailable storage 503. Responses use {status:"ok", ...} or {status:"error", error:...}.

Account work scans up to ten originals and about 2 MiB per page, advancing past covered/unavailable rows. A message contains its original encrypted envelope, sender keys and SHA-256 digest. Checkpoints rotate through current approved targets; a missing original anchor resets the scan. total counts account messages, not recovered messages. Retrieval returns up to ten candidates with source/target/account, source public key and server-derived source_account, transfer ID and ciphertext. Pagination advances through revoked-source rows without exposing them.

Sender work scans only outgoing originals, batches one recipient at a time, rotates that batch through approved recipient keys, then advances history. Its checkpoints are separate from account scans. Recipients without approved keys are skipped and revisited; missing anchors restart scanning. No recipient receipt is consulted and no recipient recovery total is returned. Legacy GET discovery deliberately revisits contributions even after recipient recovery. Combined POST discovery omits contributions already retained for this donor, target and original digest, independently of recipient recovery. The legacy GET and upload APIs remain available for explicit same-donor repair of an invalid contribution; other donors remain independent. A retained upload is evidence of contribution, never proof that its recipient can read it.

Uploads have at most ten distinct current message/digest bindings and 65,536 bytes of armored ciphertext. Exact upload retries deduplicate. Unconfirmed candidates from the same donor can be replaced; verified retained candidates stay reachable. Changed digests and missing retained blobs reopen work. Different donors remain independent. Acknowledgments always report the authenticated device’s own verified access, including while it is donating to another account; an empty transfer ID reports direct access. They are scheduling hints and do not mark messages seen. Acknowledgment never deletes ciphertext. Original IDs/digests are rechecked on reads and writes, tolerating capture skew between the two databases.

Version 1's signed, target-encrypted JSON is {version:1,account,source,target,entries:[{message_id,digest,keys:[{algorithm,data}]}]}. Same-account writes remain v1. Sender assistance uses {version:2,purpose:"sender-history",source_account,account,source,target,entries} with the same entry structure; account is the original recipient. A v1 bundle cannot be used as a cross-account transfer, and a v2 bundle cannot be rebound as same-account recovery. Fingerprints are full lowercase OpenPGP fingerprints; data is base64 session-key bytes. Browsers verify signer and target fingerprints, current account/key bindings and original envelope digest, then decrypt and verify the original sender signature before showing plaintext or reporting coverage. Bad candidates, unsigned originals and failed signatures cannot establish verified reading. Server indexes expose account/key/message relationships and encrypted bundles, never plaintext or unwrapped keys. Both bundled OpenPGP versions have real-key protocol coverage.

Browser behavior and limits

The existing unread response includes has_history, derived from any sent or received message for the authenticated account with a bounded existence query. The browser waits for that status: unused accounts make no dedicated synchronization calls. First send/receipt is picked up on the next normal unread refresh; visiting Messages and browser-local usage are not eligibility requirements. The gate does not block foreground reading/retry.

An eligible live visit makes one combined POST work check, without opening Messages. A genuinely idle response ends the run without transfers or acknowledgments. Each check scans at most eight pages per mode, returns pending bundles directly, and advances only safe empty/covered discovery pages. An empty batch with unfinished discovery returns continuation; large histories may need further checks even without new transfers. Completed modes are carried only for the current run, so new messages and keys are discovered on later runs. No access receipt is inferred from scan progress.

The browser processes at most eight bundles and makes at most eight combined checks per run for approximately ten seconds, with a fifteen-second cancellation deadline. Account and sender priority alternates, including across short visits using session storage; no recovered secrets are persisted. If storage is unavailable, priority still alternates within the page. Individual HTTP requests time out after five seconds and retry once for network failures or 503, with at most three sync requests in flight. Terminal HTTP errors are not retried; a 405 on combined work selects legacy GET discovery for that page. Focus/visible return continues work (five-second throttle); hidden tabs, navigation and identity changes cancel stale work. Synchronization runs without a global notice or Retry history button; existing per-message reading retries remain available. An interrupted sender batch retains its checkpoint for retry. There is no continuous polling, job queue, lease service, permanent failure row or persistent secret cache.

Devices need not be online together. Retained bundles permit reading after reload and allow a restored device to help later approved keys. Recovery depends on at least one eligible device retaining access; a status check does not prove every historical message is recoverable. Only successful original-message verification establishes readable content; uploads alone do not. Normal message ordering, drafts, sending, signature warnings, compact unavailable groups and unread/seen boundaries remain controlled by their existing components. Revocation prevents future eligible transfer use, but cannot retract already delivered secrets.

Coordinated private backup and restore

1. Resolve and record both configured absolute paths and the corresponding account/approval repository state. Preserve file ownership and private permissions. Take private backups outside release artifacts.

2. Quiesce message sends and sync writes and drain active requests. Use SQLite's backup API or an equivalent consistent snapshot for each file while writes remain stopped. A live bare cp can miss transactions or journal/WAL state. Keeping writes stopped across both captures gives a common logical backup point; the application does not provide cross-database atomic snapshots.

3. Verify each backup with PRAGMA integrity_check; retain the capture times/configuration. Back up the canonical identity/approval repository through its existing procedure. Resume writes after both private captures finish.

4. For restore, stop/drain writers again, restore the matching pair and compatible identity/approval state, and handle journals using SQLite's supported restore procedure. Preserve private permissions, then start the matching code/assets and run the smoke journey below.

5. With mismatched captures, stable message IDs and envelope digests reject orphaned or changed references. Missing transfer ciphertext reopens coverage; missing scan anchors restart discovery. Allow donor visits to reconcile and inspect remaining unavailable messages. Older sync captures may permanently lose the only retained transfer if no eligible donor can return; receipts cannot reconstruct it. Existing unread metadata has its own restore constraints—see unread rollout.

Rollback code and assets together while retaining both databases. Rolling back sender assistance to the same-account release preserves v1 recovery and ignores the additive sender checkpoint table; its backend excludes foreign-account donors and its browser rejects v2 bundles. History dependent only on sender assistance becomes unavailable until re-upgrade. Rolling back further, to pre-sync code, disables all transfer-based recovery. These are code limitations, not reasons to delete transfers. Re-upgrading reuses retained bundles/current membership and resumes scans. Do not copy sync tables into the message database.

Mixed versions fail conservatively: old clients retain v1 same-account recovery against the new backend but cannot consume v2 sender bundles; new clients against the old backend detect unsupported sender work and continue account work. Older unread responses without the new eligibility field permit legacy synchronization rather than permanently disabling recovery. Deploy PHP and assets together for the full feature. A rollback does not revoke secrets already delivered to browsers.

Idle synchronization deployment checks

After authorized deployment, use the production browser network panel on an ordinary forum page:

1. For an approved account with no sent or received messages, wait for unread status and focus/reload the page. Expect zero requests under /api/private_messages/history_sync/.

2. Send or receive the first message, then return/focus or reload normally. The next successful unread refresh enables synchronization without a separate setup step.

3. For settled history, observe one POST work per successful idle run and no transfer/acknowledgment. The existing unread call is separate. Large histories can report bounded scan continuation; record that separately from genuine idle.

4. Approve another device, let eligible donors visit ordinary pages, and verify older content through Messages. Repeat after an interrupted visit. Existing per-message retry remains available; failed unread status uses Retry on Messages. Missing access still requires an eligible donor or retained valid transfer.

5. Disable messaging and verify sync stops, then re-enable and confirm recovery resumes. Do not infer production acceptance from local test results.

Local evidence and remaining checks for this change: idle synchronization summary.

Deployment smoke and evidence boundary

In a controlled environment, publish and approve a new same-account key through the normal signed approval path. Verify pending access is denied, an authenticated donor visit on a non-Messages page contributes, and the target later reads sent/received originals across at least three batches with the donor closed. Repeat for 31 incoming originals with all recipient donors absent and sender visits occurring before recipient reading. Exercise partial donors, multiple recipient accounts/keys, retry, reload, mobile/keyboard, restored-sender contributions and restored-recipient forwarding. Confirm originals are unchanged and uploads alone do not clear unavailable state or unread counts.

Temporarily make only the sync store unavailable: sync APIs should return a retryable error while ordinary sends/direct reads remain usable, without a global sync banner. Restore it and verify retained recovery. Rehearse paired and mismatched restore plus rollback, inspect no-store/auth/origin checks, and scan logs/public artifacts for private data. Run:

php tests/run.php PrivateMessageHistoryCryptoTest PrivateMessageHistorySyncTest PrivateMessageReaderTest PrivateMessageListTest PrivateMessageReleaseIsolationTest
node tests/browser/private_message_history_sync_browser.mjs
node tests/browser/private_message_history_sync_browser.mjs --sender
node tests/browser/private_message_list_browser.mjs

Automated local Chromium covers same-account and sender-assisted signed-approval journeys, partial donors, retained reload/forwarding, outage, paired restore, and existing messaging regressions. PHP tests cover concurrent writers, stale-key messages, aliases, revoked/foreign keys and mismatched restores. These are local checks, not production capacity or deployment evidence. Physical mobile keyboards and other browser engines remain manual follow-ups. No production deployment was performed.