“Web services are supposed to run continuously, 24 hours a day, 365 days a year.” When did we start taking this for granted?
The theme of this series is whether the value gained from 24/365 uptime is worth the security risk. Rather than rejecting continuous operation outright, we will examine its benefits and trade-offs from the perspectives of security, operations, and business.
It is impossible to prevent every attack. That is why we need to consider not only how to prevent intrusions, but also how reliably we can return to normal operation when something goes wrong.
One idea often discussed in this context is returning to a “Known-Good State”—a state that has been verified as trustworthy. If an environment can be rebuilt through a defined process and the system can be started with the intended software and configuration, its ability to recover can improve.
There is, however, an important caveat: restarting a system and recovering it to a safe state are not the same thing.
1. What a Restart Removes—and What It Does Not
When a computer restarts, temporary state that exists only in memory is usually lost. A restart can also resolve some process problems. In operational terms, this can be useful.
But a restart does not remove everything. Malicious files stored on disk, altered configuration, mechanisms that run at startup, stolen credentials, and unauthorized changes left in a database may remain. If the software being used or the source from which updates are obtained has itself been compromised, the same problem may return even after a restart or rebuild.
It is therefore dangerous to assume that “stopping once makes the system safe.” What matters is defining what can be trusted and being able to verify the basis for that judgment.
A trustworthy state might be one in which an environment can be rebuilt using software obtained from an authentic source and verified, along with intended configuration, managed settings, and known dependencies. Even then, business data, external services, and credentials do not automatically become safe. They must be checked, protected, and recovered separately.
2. Not Just “Restarting,” but “Being Able to Rebuild from Trusted Components”
When planning for safe recovery, the focus should not be limited to whether a server can be restarted.
First, manage which software and configuration represent the correct state, and make it possible to reproduce an environment from that state. Then verify, after updates or rebuilding, that the intended versions and settings are in place. Data integrity and the security of credentials must be checked separately as well.
When a compromise is suspected, an immediate restart is not always the best response. Depending on the situation, evidence needed for an investigation may have to be preserved and the scope of the impact assessed before taking action. Routine planned maintenance and response to a security incident should be treated as different activities.
The value of scheduled downtime, then, is not that stopping makes an attack disappear. It is being able to stop or switch over when necessary, and recover through a process that can be trusted.
3. Disaster Recovery Requires More Than Having a Procedure Document
Many organizations keep backups. But having a backup is not the same as being able to restore business operations when needed. Can the data be restored correctly? Can the software be rebuilt? Can the required credentials and dependent services be made available? Some problems only become visible when these procedures are actually tested.
Planned maintenance and environment rebuilding can create opportunities to verify startup procedures and reproducibility as part of normal operations. However, this alone does not complete disaster-recovery testing. Restoring from backups, recovering in a separate environment, and checking data after recovery must be tested separately.
To say that a system “can be recovered,” it is important to establish what can be restored, how long restoration takes, and how much data can be recovered. If a test fails, the cause should be documented and the procedures or architecture improved.
These test records are not just for engineers. They can also help explain to executives and auditors how much of the organization’s recovery capability has actually been verified. The evidence is not simply that “the system is restarted regularly,” but the tests performed, the scope checked, the issues found, and the corrective actions completed.
Conclusion: The Value Lies Not in Stopping, but in Being Able to Return
Scheduled downtime can provide opportunities to reset temporary memory state and verify rebuilding procedures. But on its own, it neither prevents compromise nor returns the entire system to a safe state.
What matters is being able to rebuild from trusted software and configuration, check data and credentials, and actually test recovery procedures.
In the next article, we will consider the complexity associated with uninterrupted operations and the benefits of simpler design. To what extent can scheduled maintenance make systems and operations simpler?
References