GitHub

Restore Large Firebase RTDB Backups Losslessly

A memory-efficient, stream-based Python toolkit designed to bypass console file-size limits, OOM crashes, and PUT overwrite traps.

$
Run Tests Status PyPI Version Latest GitHub Release License: MIT
PyPI package Current site version: 0.2.3 Latest release

The Overwrite Trap vs Lossless Restore

Standard Firebase backup restoration methods fail when file sizes scale.

Standard Console Import

  • Overwrites Siblings: UI uses a full PUT operation, erasing all other data at the target node.
  • REST Limit Block: Firebase limits payload sizes; giant backups are rejected.
  • Out-Of-Memory (OOM): Loading multi-gigabyte JSON files in-memory crashes browser or scripts.

Toolkit Lossless Restore

  • Additive Merging: Uses PATCH updates to safely integrate data without erasing existing records.
  • Stream-Based: Processes files in tiny 128 KB chunks, keeping memory footprint < 20 MB.
  • Oversized Key Splitting: Automatically recurses into giant nodes, writing key-by-key.

Core Features

Built with robustness, speed, and safety in mind for large-scale operations.

Stream Splitting

Iteratively parses pretty-printed or minified JSON backups, writing N-user files without memory leaks.

SHA-256 Validation

Losslessly checks and compares every fingerprint to guarantee that chunk files are an exact 100% split.

Size-Based Uploads

Accumulates entries into safe ≤ 4 MB batches, writing them in parallel to protect network buffer pipelines.

Auto-Recovery

Encountered an oversized node? The single-user tool splits keys recursively so they fit within the limits.

Documentation Hub

Everything you need to safely plan, execute, and troubleshoot a production restore.

Production Runbook

Safety checklist, dry-run guides, and step-by-step production restore procedure.

CLI Reference

Complete list of parameters, flags, and programmatic API endpoints.

Release Notes

Changelog, new features, and upgrade instructions.

Security Policy

Guidance on reporting vulnerabilities and maintaining data safety.

Launch Kit

Ready-to-use project positioning, social posts, and community submission drafts.

Interactive Restore Workflow

Step-by-step interactive simulator showing how the toolkit processes and uploads your backup.

Production Warning: These are simplified examples. Before restoring real production data, consult the Production Restore Runbook to prevent accidental data loss.
1
Split Backup
2
Validate Chunks
3
Upload to RTDB
4
Single Node Recovery

Step 1: Stream-split the giant backup file

Reads the massive JSON backup in 128 KB blocks and splits it into smaller JSON chunk files (default 1,000 users per chunk).

$ firebase-rtdb-split backup.json -o ./chunks -n users -c 1000
firebase-rtdb-split
Input: backup.json Output dir: ./chunks Chunk size: 1000 entries Split node: users [2048 entries parsed | 100% read] chunk_0000.json (1000 entries) chunk_0001.json (1000 entries) chunk_0002.json (48 entries) Done: 2048 entries → 3 chunk files.

Step 2: Verify the split without data loss

Compares SHA-256 fingerprints of every entry in both the original file and the chunks to guarantee that no data was lost.

$ firebase-rtdb-validate backup.json ./chunks -n users
firebase-rtdb-validate
Loading chunks: 3/3 — 2048 entries loaded Streaming original: 2048 entries | 100% read ─────────────────────────────────────────────────────── Original entries : 2,048 Chunk entries : 2,048 ─────────────────────────────────────────────────────── OK No duplicate keys across chunks OK All original entries present in chunks OK No extra keys in chunks OK All values match exactly (SHA-256 verified) ─────────────────────────────────────────────────────── RESULT: PASSED — chunks are a 100% lossless split!

Step 3: Upload chunks in safe batches

Uploads the validated chunks in safe ≤ 4 MB batches. Uses PATCH requests to merge data without erasing anything else.

$ firebase-rtdb-upload ./chunks -p /users --wipe
firebase-rtdb-upload
Project: production-db Database: https://production-db.firebaseio.com Target Path: /users Chunks dir: ./chunks Chunks: 3 Wipe first: YES — target path (/users) will be wiped Wiping /users ... done [ 1/3] 33% chunk_0000.json req#1 (1000 entries, 3810 KB) . [ 2/3] 66% chunk_0001.json req#2 (1000 entries, 3844 KB) . [ 3/3] 100% chunk_0002.json req#3 (48 entries, 185 KB) . Done in 0m14s Uploaded: 2048 entries across 3 chunks Restore complete. Verify in Firebase Console.

Step 4: Handle giant entries recursively

If a single entry (like one giant user) is too large, the tool automatically splits it into sub-keys and uploads them recursively.

$ firebase-rtdb-upload-single oversized_uid ./chunks/chunk_0000.json -p /users
firebase-rtdb-upload-single
Project : production-db Key : oversized_uid Size : 18.4 MB Sub-keys: 4 /name (0 KB) ... done /profile_settings (210 KB) ... done /analytics_logs (18.1 MB) ... splitting (24 sub-keys) /analytics_logs/logs_a (2400 KB) ... done /analytics_logs/logs_b (3100 KB) ... done ... /role (0 KB) ... done Entry 'oversized_uid' fully restored under /users/oversized_uid.

CLI Command Generator

Customize the parameters below to dynamically generate the exact commands you need for your restoration.

Production Warning: Do not blindly copy-paste destructive commands (--wipe, --wipe-root). See the Production Runbook first.
1. Split Command 
make split BACKUP=my_rtdb_backup.json CHUNKS=./rtdb-chunks NODE=users
2. Validate Command 
make validate BACKUP=my_rtdb_backup.json CHUNKS=./rtdb-chunks NODE=users
3. Upload Command 
make upload CHUNKS=./rtdb-chunks SA=serviceAccountKey.json DBPATH=/users