Email is the backbone of our digital identity. Invoices, contracts, travel tickets, and decades of personal correspondence all sit inside our inboxes. Yet, despite its critical importance, most people treat email backups as an afterthought—often relying on proprietary exports or opaque, monolithic archive files.
In this post, we will explore why file-level .eml backups are superior for long-term data preservation, and how to build an industrial-grade, fully automated incremental daily backup pipeline using Python, a robust Windows Batch wrapper with Conda support, and modern laptop-resilient PowerShell Task Scheduling.
1. Why Searchable, File-Based Email Backups Matter
The Problem with Monolithic Archives (.pst, .mbox, .zip)
By default, clients like Mozilla Thunderbird store thousands of emails inside large, monolithic text files known as the MBOX format (or .pst in Microsoft Outlook). While efficient for the mail client during daily browsing, these massive database files present severe challenges for backup routines:
- Catastrophic Corruption Risk: If a single byte inside a 15 GB MBOX file is corrupted during disk synchronization, you risk losing thousands of messages.
- Terrible Backup Efficiency: Even if you only received one single newsletter today, your backup tool (Veeam, Borg, Restic, OneDrive) has to re-read and re-deduplicate a massive multi-gigabyte file.
- No Direct Searchability: You cannot inspect a message without loading the entire archive back into an active email client.
- Vendor Lock-in: Opening an archive five or ten years from now might require legacy software that is no longer maintained on modern operating systems.
The Superiority of .eml (RFC 822)
An .eml file is an open, plaintext representation of a single email message including headers, body text, and embedded MIME attachments.
- Universal Compatibility: An
.emlfile can be opened natively by Windows Mail, Apple Mail, Outlook, Thunderbird, or any text editor. - Instant OS Search: Windows Search, macOS Spotlight, or command-line utilities like
grepandripgrepcan index and search across message subjects, senders, and body text in milliseconds. - True Incremental Backups: When each email is an isolated file, backing up 50 new emails today means transferring exactly 50 small files. Your daily differential backups finish in seconds.
2. The Extraction Engine: Python with Native Windows API Integration
Thunderbird stores account configurations, index summaries (.msf), and actual message data inside profile subdirectories: Mail/ (POP3 & Local Folders) and ImapMail/ (IMAP accounts). Folders and subfolders are structured on disk using directories ending in .sbd.
To extract these messages reliably without crashing or locking during live use, our production Python script implements five critical engineering features:
- Intelligent Profile Discovery: Evaluates profile disk usage across modern
[Install<HASH>]and traditional[Profile...]sections inprofiles.inito automatically target the profile that actually contains active mail data. - Windows Kernel File Timestamp Preservation: By default, writing a new file assigns the current time as its creation date. Using 64-bit
ctypeswith Windowskernel32.SetFileTime, the script reads each email's RFC 822Dateheader and sets the exported.emlfile's filesystem timestamp accordingly. Windows Explorer and search indexers sort emails by their genuine send/receive date. - Transient Lock Retry Mechanism: If Thunderbird is open, active folders (like
INBOX) may be temporarily locked by the client. The script implements an exponential retry loop (MAX_RETRIES = 3) to wait out momentary locks before safely proceeding. - Memory Optimization for Large Datasets: Parsing 50,000+ messages in a single run can trigger high memory consumption in Python's
mailboxmodule. The script invokes explicit garbage collection (gc.collect()) every 3,000 messages and cleanly closes file handles. - Optional Orphan Cleanup: With
DELETE_ORPHANS = True, emails that have been permanently deleted from Thunderbird will also be purged from the backup directory, keeping storage synchronized.
src/thunderbird_backup.py
import os
import re
import sys
import time
import ctypes
import hashlib
import mailbox
import configparser
import traceback
from pathlib import Path
from ctypes import wintypes
from email.header import decode_header, make_header
from email.utils import parsedate_to_datetime
import gc
# Force UTF-8 and unbuffered line logging for console and log file redirection
if sys.stdout and hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace", line_buffering=True)
if sys.stderr and hasattr(sys.stderr, "reconfigure"):
sys.stderr.reconfigure(encoding="utf-8", errors="replace", line_buffering=True)
# ========================================================
# CONFIGURATION
# ========================================================
BACKUP_DESTINATION = Path(r"C:\Backups\Thunderbird_EML")
DELETE_ORPHANS = False # True = delete .eml files in backup if deleted in client
MAX_RETRIES = 3 # Retry attempts for locked folders
RETRY_DELAY_SEC = 3 # Pause between retry attempts in seconds
# ========================================================
FILE_WRITE_ATTRIBUTES = 0x0100
OPEN_EXISTING = 3
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
# Define 64-bit type safety for Windows API (prevents C-level memory crashes)
ctypes.windll.kernel32.CreateFileW.argtypes = [
wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD,
ctypes.c_void_p, wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE
]
ctypes.windll.kernel32.CreateFileW.restype = wintypes.HANDLE
ctypes.windll.kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
def set_windows_file_times(filepath: Path, timestamp: float):
"""Preserve original email date on exported .eml file in Windows filesystem."""
try:
wintime = int((timestamp + 11644473600) * 10000000)
low = wintime & 0xFFFFFFFF
high = (wintime >> 32) & 0xFFFFFFFF
ft = wintypes.FILETIME(low, high)
handle = ctypes.windll.kernel32.CreateFileW(
str(filepath), FILE_WRITE_ATTRIBUTES, 3, None, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, None
)
if handle and handle != wintypes.HANDLE(-1).value:
ctypes.windll.kernel32.SetFileTime(handle, ctypes.byref(ft), ctypes.byref(ft), ctypes.byref(ft))
ctypes.windll.kernel32.CloseHandle(handle)
except Exception:
pass
def ensure_destination_dir(dest: Path):
"""Ensure the target destination directory exists."""
try:
dest.mkdir(parents=True, exist_ok=True)
print(f"[OK] Destination ready: {dest}", flush=True)
except Exception as e:
print(f"[ERROR] Could not create destination: {dest}\nDetails: {e}", flush=True)
sys.exit(1)
def get_profile_mail_size(profile_path: Path) -> int:
"""Calculate the total size of mail data to evaluate profile activity."""
total_size = 0
for folder_name in ("Mail", "ImapMail"):
mail_dir = profile_path / folder_name
if mail_dir.exists():
for root, _, files in os.walk(mail_dir):
for f in files:
if not f.endswith(".msf"):
try:
total_size += (Path(root) / f).stat().st_size
except OSError:
pass
return total_size
def find_best_thunderbird_profile() -> Path:
"""Locate the active profile containing actual mail data."""
appdata = os.environ.get("APPDATA")
tb_base = Path(appdata) / "Thunderbird"
profiles_ini = tb_base / "profiles.ini"
if not profiles_ini.exists():
raise FileNotFoundError(f"profiles.ini not found at {tb_base}")
config = configparser.ConfigParser()
config.read(profiles_ini, encoding="utf-8")
candidate_paths = []
for section in config.sections():
if section.startswith("Install") and config.has_option(section, "Default"):
p = tb_base / config.get(section, "Default")
if p.exists() and p not in candidate_paths:
candidate_paths.append(p)
for section in config.sections():
if section.startswith("Profile") and config.has_option(section, "Path"):
raw_path = config.get(section, "Path")
is_rel = config.get(section, "IsRelative", fallback="1") == "1"
p = (tb_base / raw_path) if is_rel else Path(raw_path)
if p.exists() and p not in candidate_paths:
candidate_paths.append(p)
profiles_dir = tb_base / "Profiles"
if profiles_dir.exists():
for p in profiles_dir.iterdir():
if p.is_dir() and p not in candidate_paths:
candidate_paths.append(p)
print(f"[INFO] Discovered profile candidates ({len(candidate_paths)}):", flush=True)
best_profile = None
max_size = -1
for p in candidate_paths:
size = get_profile_mail_size(p)
mb = size / (1024 * 1024)
print(f" - {p.name} ({mb:.1f} MB mail data)", flush=True)
if size > max_size:
max_size = size
best_profile = p
return best_profile
def sanitize_filename(name: str, max_len: int = 40) -> str:
"""Sanitize subject string for safe filesystem usage."""
name = re.sub(r'[\\/*?:"<>|]', '_', name)
name = re.sub(r'\s+', ' ', name).strip()
return name[:max_len] if name else "No_Subject"
def parse_header_text(header_val):
"""Decode RFC 2047 encoded email headers."""
if not header_val:
return ""
try:
return str(make_header(decode_header(header_val)))
except Exception:
return str(header_val)
def process_mbox(mbox_path: Path, dest_dir: Path, delete_orphans: bool = False):
"""Extract individual messages from MBOX and save as .eml files."""
dest_dir.mkdir(parents=True, exist_ok=True)
box = mailbox.mbox(mbox_path)
new_count = 0
skip_count = 0
deleted_count = 0
valid_filenames = set()
for i, msg in enumerate(box, start=1):
if i % 3000 == 0:
print(f" ... {i} emails verified ...", flush=True)
gc.collect() # Release memory back to OS periodically
try:
msg_id = msg.get("Message-ID", "").strip()
if not msg_id:
raw_bytes = msg.as_bytes()
msg_id = hashlib.sha256(raw_bytes[:1024]).hexdigest()
id_hash = hashlib.md5(msg_id.encode('utf-8', errors='ignore')).hexdigest()[:10]
subject = parse_header_text(msg.get("Subject", "No_Subject"))
filename = f"{id_hash} - {sanitize_filename(subject)}.eml"
valid_filenames.add(filename)
file_path = dest_dir / filename
# Skip existing messages (incremental backup)
if file_path.exists():
skip_count += 1
continue
with open(file_path, "wb") as f:
f.write(msg.as_bytes())
# Synchronize filesystem timestamp with genuine email date
date_str = msg.get("Date")
if date_str:
try:
dt = parsedate_to_datetime(date_str)
if dt and 1970 <= dt.year <= 3000:
set_windows_file_times(file_path, dt.timestamp())
except Exception:
pass
new_count += 1
except Exception as err:
print(f" [Exception] msg_id {msg_id} error: {err}", flush=True)
box.close() # Explicitly close file handle
gc.collect()
if delete_orphans:
try:
for existing_file in dest_dir.glob("*.eml"):
if existing_file.name not in valid_filenames:
try:
existing_file.unlink()
deleted_count += 1
except Exception:
pass
except Exception:
pass
return new_count, skip_count, deleted_count
def run_backup():
ensure_destination_dir(BACKUP_DESTINATION)
profile_dir = find_best_thunderbird_profile()
print(f"\n[INFO] Selected Active Profile: {profile_dir}", flush=True)
print(f"[INFO] Backup Destination: {BACKUP_DESTINATION}\n", flush=True)
storage_roots = [profile_dir / "Mail", profile_dir / "ImapMail"]
total_new = 0
total_skipped = 0
total_deleted = 0
folders_processed = 0
ignore_extensions = {".msf", ".dat", ".json", ".sqlite", ".ini", ".txt", ".bak"}
for root in storage_roots:
if not root.exists():
continue
for dirpath, _, filenames in os.walk(root):
dirpath = Path(dirpath)
rel_path = dirpath.relative_to(root)
if not rel_path.parts:
continue
account_name = rel_path.parts[0]
clean_subparts = [p.removesuffix(".sbd") for p in rel_path.parts[1:]]
clean_all_parts = [p.removesuffix(".sbd") for p in rel_path.parts]
folder_target_dir = BACKUP_DESTINATION / root.name / Path(*clean_all_parts)
for file in filenames:
file_path = dirpath / file
if file_path.suffix.lower() in ignore_extensions:
continue
display_folder = "/".join(clean_subparts + [file]) if clean_subparts else file
# Retry loop to handle transient file locks
success = False
for attempt in range(1, MAX_RETRIES + 1):
try:
file_size = file_path.stat().st_size
if file_size == 0:
success = True
break # Ignore empty placeholder files
new_cnt, skip_cnt, del_cnt = process_mbox(
file_path,
folder_target_dir / file,
delete_orphans=DELETE_ORPHANS
)
status_parts = [f"+{new_cnt} new", f"{skip_cnt} existing"]
if DELETE_ORPHANS and del_cnt > 0:
status_parts.append(f"-{del_cnt} deleted")
print(f" -> [{account_name}] '{display_folder}': {', '.join(status_parts)}.", flush=True)
folders_processed += 1
total_new += new_cnt
total_skipped += skip_cnt
total_deleted += del_cnt
success = True
break
except Exception as err:
if attempt < MAX_RETRIES:
print(f" [WARN] Attempt {attempt}/{MAX_RETRIES} on '{display_folder}' failed: {err}", flush=True)
print(f" Waiting {RETRY_DELAY_SEC}s before retry...", flush=True)
time.sleep(RETRY_DELAY_SEC)
else:
print(f" [ERROR] Folder '{display_folder}' unreadable after {MAX_RETRIES} attempts: {err}", flush=True)
print(f" -> Skipping folder, backup continues!", flush=True)
print("\n----------------- SUMMARY -----------------", flush=True)
print(f"Mail Folders Processed : {folders_processed}", flush=True)
print(f"New Emails Exported : {total_new}", flush=True)
print(f"Existing Emails Skipped: {total_skipped}", flush=True)
if DELETE_ORPHANS:
print(f"Deleted Emails Purged : {total_deleted}", flush=True)
print("-------------------------------------------", flush=True)
if __name__ == "__main__":
try:
run_backup()
except Exception as fatal_err:
print(f"\n[UNEXPECTED ERROR]: {fatal_err}", flush=True)
traceback.print_exc()
sys.exit(1)3. The Windows Conda Wrapper: run_backup.cmd
Many Python development workflows rely on Anaconda or Miniconda environments. However, Conda is typically not placed on the global system PATH to avoid dynamic library collisions with core system utilities.
Our batch wrapper dynamically resolves the absolute path to conda.bat via where.exe, uses %~dp0 to work portably relative to its own location, logs everything to backup.log, and executes cleanly inside the dedicated environment:
scripts/run_backup.cmd
@echo off
setlocal EnableExtensions EnableDelayedExpansion
:: 1. Derive script directory dynamically
set "BASE_DIR=%~dp0"
set "SCRIPT_PATH=%BASE_DIR%thunderbird_backup.py"
set "LOG_PATH=%BASE_DIR%backup.log"
set "CONDA_ENV_NAME=base"
echo ======================================================== >> "%LOG_PATH%"
echo [START] Backup started on %date% at %time% >> "%LOG_PATH%"
:: 2. Resolve absolute path to Conda
set "CONDA_CMD="
for /f "delims=" %%I in ('where.exe conda.bat 2^>nul') do (
set "CONDA_CMD=%%~fI"
goto :FOUND_CONDA
)
for /f "delims=" %%I in ('where.exe conda.exe 2^>nul') do (
set "CONDA_CMD=%%~fI"
goto :FOUND_CONDA
)
:: Fallback: search common Conda install locations
for %%P in (
"%USERPROFILE%\miniconda3\condabin\conda.bat"
"%USERPROFILE%\anaconda3\condabin\conda.bat"
"%LOCALAPPDATA%\miniconda3\condabin\conda.bat"
"%LOCALAPPDATA%\anaconda3\condabin\conda.bat"
"C:\ProgramData\miniconda3\condabin\conda.bat"
"C:\ProgramData\anaconda3\condabin\conda.bat"
) do (
if exist %%P (
set "CONDA_CMD=%%~fP"
goto :FOUND_CONDA
)
)
echo [ERROR] Conda executable could not be located! >> "%LOG_PATH%"
exit /b 1
:FOUND_CONDA
echo [INFO] Conda located at: %CONDA_CMD% >> "%LOG_PATH%"
:: 3. Validate Python script presence
if not exist "%SCRIPT_PATH%" (
echo [ERROR] Python script not found at: %SCRIPT_PATH% >> "%LOG_PATH%"
exit /b 1
)
:: 4. Execute inside Conda Environment
echo [INFO] Running Python inside Conda environment '%CONDA_ENV_NAME%'... >> "%LOG_PATH%"
call "%CONDA_CMD%" run -n "%CONDA_ENV_NAME%" python "%SCRIPT_PATH%" >> "%LOG_PATH%" 2>&1
if %errorlevel% neq 0 (
echo [ERROR] Backup execution failed with error code %errorlevel%. >> "%LOG_PATH%"
exit /b %errorlevel%
)
echo [SUCCESS] Backup completed successfully at %time%. >> "%LOG_PATH%"
exit /b 04. Laptop-Resilient Automation with PowerShell Scheduled Tasks
Traditional scheduled tasks registered via schtasks.exe suffer from critical flaws on mobile workstations and laptops:
- If your laptop is asleep or closed in your bag at 21:00, the task simply fails to run.
- Windows default task configurations disallow task execution when operating on battery power.
- Unplugging the charger mid-backup will abort the process by default.
By creating the scheduled task with PowerShell using New-ScheduledTaskSettingsSet, we enable three vital flags:
-StartWhenAvailable: If the laptop was in standby or turned off at the scheduled trigger time, Windows automatically executes the missed backup run as soon as you wake or open the device.-AllowStartIfOnBatteries: Permits the backup to start even when running unplugged.-DontStopIfGoingOnBatteries: Prevents task cancellation if you unplug the charger while the backup is active.
Run the following command in an administrative PowerShell window:
scripts/register_task.ps1
# 1. Define action (path to your .cmd wrapper)
$Action = New-ScheduledTaskAction -Execute "C:\Scripts\ThunderbirdBackup\run_backup.cmd"
# 2. Define schedule (daily at 21:00)
$Trigger = New-ScheduledTaskTrigger -Daily -At 21:00
# 3. Configure laptop-resilient settings:
# - Catch up missed runs immediately on wake/unfold
# - Allow execution on battery power
# - Do not terminate if power cable is disconnected
$Settings = New-ScheduledTaskSettingsSet `
-StartWhenAvailable `
-AllowStartIfOnBatteries `
-DontStopIfGoingOnBatteries
# 4. Register the task in Windows Task Scheduler
Register-ScheduledTask -TaskName "DailyThunderbirdBackup" -Action $Action -Trigger $Trigger -Settings $SettingsPro Tip: Completely Silent Execution
To eliminate terminal window popups while you are actively working, place a simple one-line VBScript file named silent_run.vbs in your script directory:
scripts/silent_run.vbs
Set WshShell = CreateObject("WScript.Shell")
WshShell.Run "C:\Scripts\ThunderbirdBackup\run_backup.cmd", 0, FalseThen, update your scheduled task action to execute:
powershell
$Action = New-ScheduledTaskAction -Execute "wscript.exe" -Argument "C:\Scripts\ThunderbirdBackup\silent_run.vbs"5. Verification & Log Output
When the script runs, check backup.log to see the execution breakdown:
logs/backup.log
========================================================
[START] Backup started on 2026-03-15 at 21:00:01,12
[INFO] Conda located at: C:\Miniconda3\condabin\conda.bat
[INFO] Running Python inside Conda environment 'base'...
[OK] Destination ready: C:\Backups\Thunderbird_EML
[INFO] Discovered profile candidates (2):
- xxxxxxxx.default (0.0 MB mail data)
- xxxxxxxx.default-release (4120.5 MB mail data)
[INFO] Selected Active Profile: C:\Users\User\AppData\Roaming\Thunderbird\Profiles\xxxxxxxx.default-release
[INFO] Backup Destination: C:\Backups\Thunderbird_EML
-> [Personal] 'INBOX': +4 new, 18420 existing.
-> [Personal] 'Sent': +1 new, 4310 existing.
-> [Work] 'Archives/2025': +0 new, 9210 existing.
----------------- SUMMARY -----------------
Mail Folders Processed : 12
New Emails Exported : 5
Existing Emails Skipped: 31940
-------------------------------------------
[SUCCESS] Backup completed successfully at 21:00:04,45.Notice that inspecting over 31,000 messages and exporting the 5 newest ones with synchronized file creation timestamps took under 4 seconds.
Summary and Key Takeaways
- Granular .eml Preservation: By breaking away from monolithic mail databases and treating emails as individual
.emlfiles, backups become resilient against single-file corruption. - Filesystem Timestamp Synchronization: Utilizing Windows
SetFileTimeensures exported files preserve their original email delivery dates, enabling instant chronological sorting in Windows Explorer. - Robust Concurrency Handling: Exponential retries and unbuffered I/O gracefully handle transient locks when Thunderbird is actively checking mail during a scheduled run.
- Laptop-Grade Task Scheduling: Registering tasks with PowerShell's
-StartWhenAvailableand battery flags guarantees missed runs execute as soon as your device wakes from sleep.