When a production database, a critical config file, or a VM volume shows signs of corruption, the first question isn’t “what caused this?” It’s “do I rebuild from scratch or restore from backup?” For small-to-mid-size technical teams running cloud infrastructure, that choice can make or break your recovery time—and your credibility. This isn’t about vendor promises or best-practice platitudes. It’s about reading the evidence in your own environment and picking the path that actually gets you back to a trustworthy state.
Define the Shape of the Corruption First
Corruption isn’t one thing. It might be silent bit rot in a block storage volume, a logical inconsistency in a database index, an overwritten configuration file, or a cascading failure where one bad artifact poisons downstream consumers. Each shape calls for a different response. Before you commit to a path, pin down three things:
- Scope. Is the damage limited to a single file, a table, a volume, or an entire environment?
- Timeline. When did the corruption first appear, and how far back do your clean backups go?
- Dependencies. What other systems, pipelines, or configurations rely on the corrupted component?
A corrupted index on a non-critical reporting table might be fixed with a quick REINDEX. But a corrupted pg_wal segment on a primary database that feeds five microservices? That’s a different animal. The scope and dependency chain often push you toward a full restore or a rebuild from a known-good source. Skip this triage step and you’ll likely restore a backup only to find the same corruption lurking inside—just waiting to bite you again.
When Restore Makes Sense
Restoring from backup is the right call when three things line up: your backups are tested, the corruption is recent and contained, and the system’s state is hard to recreate programmatically. For many smaller teams, the database is the prime candidate. A PostgreSQL instance with years of accumulated data, hand-tuned parameters, and complex materialized views isn’t something you can just spin up from a Terraform module. If your point-in-time recovery tooling is solid and you can roll back to a moment before the corruption hit, restore is the pragmatic choice.
But “solid” means more than “we set up pg_dump and forgot about it.” You need proof that your backups are logically consistent. A restore test in a sandbox that passes application-level checks is worth a thousand green dashboard icons. We’ve seen teams discover—during an actual incident—that their nightly pg_dump had been silently truncating large objects for months. The backups existed. They were just hollow. If you haven’t run a full restore drill recently, your backup is a wish, not a strategy. For a practical checklist, see our Recovery Checklist Before You Need It.
When Restore Turns into a Trap
Restore fails when the corruption is older than your retention window, when the backup itself is the corrupted artifact, or when the restored system drags forward hidden technical debt. A classic story: a team restores a database, confirms the application starts, and calls it done. Two weeks later, a subtle foreign key inconsistency—born from the same corruption event—triggers a billing error. The restore “worked,” but the decision was wrong because the blast radius was bigger than anyone diagnosed.
Configuration drift is another trap. Restore a server image from a three-month-old backup and you also restore three months of unpatched vulnerabilities, expired TLS certificates, and stale IAM roles. In cloud environments, an old state can break integrations with services that have since rotated credentials or deprecated API versions. The restore path is only safe when your state is fully self-contained or you have a tested process for reconnecting it to the current ecosystem.
When Rebuilding Is the Safer Bet
Rebuilding means recreating the system from source: infrastructure as code, configuration management, application artifacts, and then repopulating data from a known-good state or letting the system regenerate it. This path wins when your environment is highly codified and the corrupted state is either untrusted or easy to reproduce.
For stateless or semi-stateful services—web servers, API gateways, CI/CD runners—rebuilding is almost always the right answer. If an EC2 instance acts up after a bad package update, terminate it and let an Auto Scaling group launch a fresh one from a golden AMI. It’s faster than forensic analysis. Same logic for Kubernetes pods: if a pod’s filesystem is corrupted, delete it and let the ReplicaSet recreate it. The rebuild path leans hard on the maturity of your infrastructure as code and your ability to redeploy without manual fiddling.
Rebuilding Stateful Systems
Stateful systems like databases and message queues are tougher to rebuild, but not impossible. The decision hinges on whether you can reconstruct the data from an authoritative source. If your primary database is corrupted but you maintain a logical replication stream to a clean secondary, promote the secondary and rebuild the primary from it. That’s a rebuild, not a restore—you’re constructing a new instance from a known-good state rather than rolling back to a snapshot.
Another rebuild pattern uses event sourcing. If your application logs all state changes as an immutable event stream, you can replay those events into a fresh database to reconstruct the current state. This is common in systems built around Apache Kafka or AWS Kinesis. The trade-off is time: replaying millions of events can take hours, but the result is a database free of any corruption that might have been baked into older backups. Teams that practice this often keep a “hot spare” database that continuously replays the event stream, ready for promotion at a moment’s notice.
A Decision Framework for Your Team
When the pressure is on, a simple decision tree can prevent analysis paralysis. Here’s a framework you can adapt to your runbooks:
- Is the corrupted component stateless? If yes, rebuild. There’s no data to lose, and a fresh deployment eliminates any lingering corruption.
- Is the corruption older than your oldest clean backup? If yes, you cannot safely restore. You must rebuild from source data or accept data loss.
- Can you isolate and validate the corrupted data? If you can identify exactly which rows, files, or objects are affected and confirm the rest is clean, a partial restore or selective rebuild may work. Otherwise, assume the corruption is pervasive.
- Is your infrastructure fully codified? If you can run a single command to recreate the entire environment, the cost of rebuilding is low. If your infrastructure relies on manual steps, restoring may be faster—but document that technical debt immediately.
- What is the recovery time objective (RTO) and recovery point objective (RPO)? If restoring a backup meets both, restore. If not, rebuild from a clean source or promote a standby.
This framework assumes you have current answers for RTO and RPO. If those are aspirational numbers in a document nobody reads, your first step after this incident should be to measure your actual recovery capabilities and update your runbooks.
Testing Your Decision Before You Commit
In cloud environments, you can test both paths in parallel without touching production. Spin up a restored volume from a snapshot and a fresh volume from your infrastructure-as-code pipeline. Run your validation suite against both. This side-by-side comparison often reveals that the “faster” path isn’t what you expected. A restore might complete in minutes but require hours of manual reconfiguration to reconnect to current IAM roles, security groups, and monitoring agents. A rebuild might take longer to provision but come out fully integrated and operational.
One team we observed used this approach after a corruption event in their MongoDB cluster. The restore path brought back a working database in 20 minutes, but it was missing two days of data due to a backup gap. The rebuild path—replaying the oplog from a known-good secondary—took 90 minutes but recovered all data. The parallel test gave them the evidence to choose the slower, more complete path without guessing.
Documenting the Decision for Future Incidents
Every corruption event is a learning opportunity. After the system is healthy, document the decision process: what you knew at the time, which path you chose, and what the outcome was. This post-incident analysis becomes a reference for the next event and helps refine your decision framework. Over time, patterns emerge. You might notice that database corruption from storage-level bit rot always leads to a rebuild from a replica, while application-level logical corruption is usually handled by a point-in-time restore. These patterns can be codified into runbooks that reduce decision fatigue during future incidents.
If you haven’t yet written a recovery runbook, start with the basics. Our Recovery Checklist Before You Need It walks through the minimum set of procedures every team should have documented before an incident occurs.
FAQ
How do I know if my backup is clean?
You don’t, until you test it. A clean backup is one that has been restored to a sandbox environment and passed application-level consistency checks. For databases, this means running integrity checks like pg_verify_checksums or mysqlcheck. For file systems, compare checksums against known-good values. Automate these tests and run them at least weekly. A backup that hasn’t been tested is a Schrödinger’s backup: it’s simultaneously good and corrupt until you observe it.
What if the corruption is in my infrastructure as code repository?
If your Terraform modules or CloudFormation templates are corrupted, rebuild is your only option—but you need a clean source. This is where Git’s history becomes your recovery tool. Check out a commit from before the corruption was introduced, validate the infrastructure it produces in a sandbox account, and then apply it to production. If the corruption is in your remote state file (e.g., an S3 backend with versioning disabled), you may need to reconstruct state manually. This is a painful process that underscores why state file backups and versioning are non-negotiable.
How do I handle partial corruption where only some data is affected?
Partial corruption is the hardest case because it tempts you to “just fix the broken parts.” If you can definitively identify the corrupted records and your application logic allows for surgical repairs, a partial restore from a logical backup (e.g., pg_restore with specific tables) can work. However, you must verify that no other data depends on the corrupted records. Foreign key cascades, materialized views, and application caches can all propagate corruption silently. When in doubt, assume the corruption is wider than it appears and choose the path that gives you the highest confidence in a clean state.