I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside.
I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:
I take the typical formulation (e.g., [1]), and translate it into:
- Keep 3 copies of your data: production + 2.
- Keep 2 snapshots, stored separately.
- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.
It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.
I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.
However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.
Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.
From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.
Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player.
> But nowadays they are just as mainstream industrial as their competitors
Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google.
It's slow slow slow slow slow and horribly buggy, did I mention it's slow?
People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...
> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
Most of the "big" cloud providers aren't as negligent as OVH though. That entire datacenter was just one blunder after the other (wood structures, barely functional fire suppression, no proper power shut down, insane replication strategy that caused most of the replicas to burn in the very same fire in the same building...)
I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.
another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.
Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready.
Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.
You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.
> It's not like there is a mirror datacenter just two blocks away
Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)
It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region
If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it.
OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.
Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.
As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.
I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:
https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...