AWS says data kept only in Bahrain and one UAE zone is gone for good
Six months after the March 1 drone strikes, AWS told customers it has exhausted its options for resources hosted only in the Bahrain region and in the UAE zone mec1-az2. Multi-AZ design was built for a different failure.
Multi-AZ did not cover this. AWS has now told customers what it could not restore, and what to do about it.
An AWS region is built from availability zones, separate sites far enough apart that a fire or a flood in one leaves the others running, and the standard advice for two decades has been that a workload spread across zones is safe. On September 15, AWS told customers that in Bahrain the damage from March's drone strikes "exceeded what our regional and multi-AZ services are designed to withstand," and that data hosted only there is not coming back.
The strikes hit three AWS data centers in Bahrain and the United Arab Emirates on Sunday, March 1, in the opening days of the Iran war, knocking two of the UAE region's three zones offline and taking EC2, S3, DynamoDB, Lambda, and RDS down with them. Tuesday's update on the AWS Health Dashboard was the company's first public word on the sites since April, Reuters reported via CNBC. For Bahrain it read: "After a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in this region." A separate post said the same of one zone in the UAE region, mec1-az2, while the region's other two zones continue to operate.
What AWS said, and what it did not
AWS says it has exhausted all options for restoring Bahrain data that customers had not already migrated, and that it has notified the authorities in both countries, CircleID reports. The instruction to customers is to move workloads to other regions and restore from their own backups. The company has not said how much data is gone, how many customers are affected, or which services held it. The Wall Street Journal, which broke the story, calls it the first confirmed permanent loss of customer data at a hyperscale cloud caused by a military attack.
The design assumption that failed is the interesting part, and AWS named it. Multi-AZ replication protects against one site going down. It does not protect against every site in a region going down at once, and a region's zones are, by design, in the same metropolitan area. Cross-region replication and off-cloud backups exist for exactly that case. They are opt-in, they cost money, and a great many production systems do not have them.
One check takes a minute. Ask S3 whether a bucket replicates anywhere else, and if there is no replication rule the CLI answers with an error, which is itself the answer.
aws s3api get-bucket-replication --bucket your-bucket-nameWhy a build studio cares
The backup and restore reality check in our Deep Audit exists because vendors say "it is on AWS" as if the sentence ended there. After this week it does not. The question we now ask a vendor is not whether the data is replicated but where the second copy lives, in which region, and when someone last restored from it, since a backup nobody has restored is a hope. For clients with users in the Gulf, residency now has a second half: the law says where the data must sit, and March says where the copy must not.
Next step: read CNBC's report carrying the Reuters wire, CircleID's summary of the dashboard posts, and the Wall Street Journal's story. If your production system's disaster plan is the phrase "it is on AWS," write to us at hello@gattyworks.com.