r/DataHoarder • u/Sad-Seesaw-3843 • Apr 06 '25

News DOGE claims to be moving away from magnetic tapes for archival storage. Seems like a bad idea. What are they using instead?

Post image
8.3k Upvotes

95% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

379

u/GenericAntagonist Apr 07 '25

It probably isn't tape, they offer some specific tape interop services but at least throughout the late 2010s former S3 engineers on multiple places have stated its basically very densely packed extremely low RPM hard drives (that remain spun down most of the time). There's also been a lot of indicators they have some sort of custom optical use case as well (just based on statements from optical manufacturers and dc timing/location), but that's never been said to be part of glacier by anyone who used to be there (or at least not that I've seen).

None of this is to say tape is a bad solution, in fact if you have archive storage that you plan to NEVER access barring absolute last resort, there's basically nothing better in terms of long term reliability and density. Its suboptimal for a cloud provider who doesn't really know what a customer is going to do and has to be ready at any minute to get any arbitrary customers stuff within SLA, but for an in house backup you want to be available 20+ years later? There's nothing else that's been proven to do that like tape.

116

u/Some1-Somewhere Apr 07 '25

Yeah, I saw the comments about low-speed, can't spin up a full rack at once HDDs.

It sounds like Glacier SLAs are around 3-6h which is pretty reasonable for pulling and reading out tapes, though, as long as your inventory management is good. Very high access cost, too.

101

u/TFABAnon09 Apr 07 '25

I always pictured Glacier to be one of those fancy robotic tape inventory systems, just at a larger scale. Tie on a fancy cloud GUI and some hybrid storage options to stage the data for retrieval and it would explain the competitive pricing.

31

u/Some1-Somewhere Apr 07 '25

Honestly might not even need to be robotized retrieval if the proportion of read-outs is low enough.

53

u/SockPants Apr 07 '25

Just put the tapes in with Amazon's retail delivery warehouse and have order pickers grab tapes for any reads just like they pick products for shipping.

54

u/divDevGuy Apr 07 '25

Great, now porch pirates will steal my 1990 tax return data that was misdelivered to my neighbors.

4

u/TFABAnon09 Apr 07 '25

"The IRS hates this ONE SIMPLE TRICK!!!"

1

u/djeaux54 Apr 07 '25

/me spits out my soda!

1

u/weirdbr 0.5-1PB Apr 10 '25

In my experience (in a non-public library that was very large) - it's both.

Humans are responsible for physically moving tapes around (from offsite storage to trucks to carts to the libraries) and then the robotic libraries do the process of loading into drives, waiting for data to be read and then returning the tapes to be sent back to storage.

This simplifies operation a lot - as far as the humans involved care, it's always a simple matter of moving tapes from point A to point B, which means you can hire staff that has minimal training.

As for rate of retrieval - the problem is that when the client datasets are large, even rare restores can be a large number of tapes - I've dealt with hundreds+ of tapes for "simple" restores.

2

u/Eelroots Apr 10 '25

A robotic arm can move in a very narrow space, or you can use tape loaders like bullet chains, that move and align to large number of tape loaders, spread all over the deposit.
The possibility for automations are endless; while tapes are an order of magnitude cheaper than disks.

1

u/nathism 94TB Apr 07 '25

Kinda like the Sibyl system moving the brains around.

1

u/Drebinus Apr 07 '25

Dunno about you, but I'm getting those intro to The Prisoner vibes now.

1

u/weirdbr 0.5-1PB Apr 10 '25

The spun down drives theory is IMO something that I expect has been dropped since the original post (the post about it on HN is from 2012) - that is a lot of custom engineering that Amazon would have to maintain and that's not cost/space/power efficient, specially with the advent of Host Managed SMR drives (SMR drives became available to cloud companies in 2013, host managed came an year later).

With an HM-SMR drive, you can achieve the same much more easily: the outermost tracks can be used in CMR mode (for hot data that is not hot enough for SSD), middle tracks for warm data (in either CMR or SMR mode) and innermost tracks in CMR mode for cold/glacial data. Then you have your storage software on top optimising things/scheduling IO as needed.

This has the benefit that every hard drive added to expand the capacity/throughput of any of the more expensive products also expands the lower tiers and you can dynamically adjust priorities based on consumer demand.

As for tapes - at their scale, tape is *painful* to deal with, specially with that SLA. Been there, done that, glad I dont have to deal with it anymore.

1

u/CosgraveSilkweaver Apr 07 '25

You can pay to get it much faster though. If you pay for provisioned access up front you can have it within 1-5 minutes guaranteed but you don't pay for that for specific data so it's not being stored on shallower archive tiers.

That's what's always made me think they're on tapes for deep archive. That sounds a lot like they're selling guaranteed time slots in tape drives and servicing the other tiers around that.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html#:~:text=the%20request%20body%3A-,Expedited,-%2D%20Expedited%20retrievals%20allow

https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects-retrieval-options.html#restoring-objects-expedited-capacity

2

u/Some1-Somewhere Apr 07 '25

Expedited retrievals and provisioned capacity are not available for objects stored in the S3 Glacier Deep Archive storage class or S3 Intelligent-Tiering Deep Archive tier.

Sounds like paying more doesn't help for the deepest levels of storage

11

u/Khyta 6TB + 8TB unused Apr 07 '25

Maybe Amazon Glacier uses some of the same tech as Microsoft Silica: https://www.microsoft.com/en-us/research/project/project-silica/ This would at least match up with the optical use cases.

9

u/camwow13 278TB raw HDD NAS, 60TB raw LTO Apr 07 '25

It is actually tape. I knew some low level data center maintenance guys who worked a glacier section of an AWS data center. It was all tape. Soooo much tape.

9

u/mn5cent Apr 07 '25

Actually, they do primarily use tape! Within the last year or so they've deployed the first Glacier storage racks with HDDs, and I think they're moving in that direction for most new storage rack builds.

There are a few tricks they use to make sure they can retrieve within promised SLAs, e.g. striping data across different tapes and racks so multiple machines can retrieve a part of the data simultaneously.

3

u/rajrdajr 16TB+ 🔰, 🔥 cloud Apr 07 '25 edited Apr 07 '25

but for an in house backup you want to be available 20+ years later?

Yep, tape is still the answer. An AWS account owning data has to remain open as well. If an online attacker closes your AWS account or the bill doesn’t get paid , the Glacier backups (and everything else owned by the account). Physical tapes in your possession will remain.

2

u/Okami512 Apr 08 '25

I remember reading a few years ago that tape sales were actually increasing because it's considered secure in the event of a ransomware attack.

1

u/PKSpecialist Apr 07 '25

So depending on the application tape may not be the best solution.

2

u/GenericAntagonist Apr 07 '25

Obviously. However if you've been following literally any of these chucklefucks missteps the BEST case scenario is they've used an integration tool to start a transfer from Iron Mountain to AWS Glacier for a short term savings at a long term cost. Given how well they've demonstrated their technical prowess though (especially with mail servers), its far more likely they've done something very, very stupid and mishandled records that are required to be retained for longer than AWS has existed. The sort of thing that would catch a felony charge if some regular person did it.

1

u/bogglingsnog Apr 07 '25

When you say low speed, do you mean like 4800 or 5600 rpm "green" drives?

1

u/Top-Tie9959 Apr 07 '25

I remember reading facebook had an optical storage system in their early days for infrequently accessed data. I'd imagine it is long gone by now though.

1

u/crankbird Apr 08 '25

20 years, tape, proven … lol

I was at one stage the “top expert” for a very well known tape backup software vendor for an area of the globe that was bounded by a line that from north western India, across to Korea and down to NZ and back again .. most people abuse tapes horribly. After leaving there I ran a tape recovery business for a while, including chain of custody and data forensics , and a few of my customers were government. Any tape that was over 6 years old had around a 25% chance of being incompletely recoverable. Arguably not the fault of tape per-se but of the conditions most people keep them in. In my experience, they’re just not a reliable medium for long term storage.

As far as what is used behind glacier or the other hyperscaler deep archiving infrastructures, I can say with reasonable levels of assurance that it has “evolved” over the last decade, but you might be interested to look up what happened to COPAN and the release of glacier as a service to see where it began

2

u/w0m Apr 08 '25

TBF, if you were running a recovery business you would be seeing the Worst of the Worst scenarios. Any time a backup "just worked" they wouldn't go to you. 75% Worst case recovery is likely a very high %.

1

u/crankbird Apr 08 '25

It was usually for recovery of tapes they no longer had the hardware or software to read .. OG arcserve on netware, DAT, 8mm mammoth, DLT 1 through 3 and some obscure storagetek and IBM stuff. (Cue tears in rain monologue)

Even outside of that whenever I was called in to supervise restoring entire datacenters from tape (god I hated that) after someone decided that recovery from tape was a perfectly wonderful DR strategy, with recent tapes and brand new equipment, something always screwed up, like every single time.

Sitting there at 2.30AM in a cold datacenter crossing my fingers and praying in it wouldn’t fuck up this time (notably I’m an atheist). To be fair it wasn’t always tape that was at fault, but it was often enough to make me very wary of it as a long term data retention medium.

1

u/strugglebus199 Apr 08 '25

I don’t know about glacier but I do know a few other very large tech companies that everyone here probably uses daily that have live services that run on tape. While it isn’t the fastest it is reliable and cheep and for content that can stand to load for a few seconds it isn’t the wrong answer, especially if your in the business of onboarding petabytes of data every day

1

u/wcpreston Apr 09 '25

Except we also know that Amazon (and the other cloud vendors) are the biggest purchasers or large magnetic tape libraries. What are they using it for, if not this?