How to thrive in the era of minimal data deletion

According to the latest global estimates, up to 400 million terabytes of data are created every single day. However, unlike the past, where users were happy to delete data once they felt it was no longer required, today’s data rarely gets deleted anymore, which is creating a growing number of issues.

It has become a cliché to say, “data is the new oil”, but its value has never been higher. Companies across all industries now collect as much as they can from every customer and process, so it can be mined for valuable insights and/or sold to generate new revenue streams. But to extract greater value from data, you need to keep it. To keep it, you need space, security, and money. Consequently, organizations are increasingly looking for the best ways to store their data without breaking the bank (or damaging the environment, if they can help it).  The question is, what’s the best way to go about doing this?

The era of indefinite data retention

The concept of big data has evolved rapidly over time. The enterprise “data pools” of the early 2000s became “data lakes” by 2010 and grew to become “data oceans” by 2015. From there they began to morph into vast multiform, million-petabyte interconnected networks and have only got larger since. The active installed base of enterprise petabytes exceeded one zettabyte (a million petabytes) in 2016 and grew to 5.2 zettabytes (5.2 million petabytes) by 2022. It’s estimated this figure could be as high as 36 zettabytes by 2030.

In tandem with this, the amount of data being deleted is getting smaller and smaller. A key driver is the aforementioned desire to extract as many valuable insights from it as possible, especially as the explosion in AI requires constant data uploads to receive accurate outputs, but there is also a growing fear amongst businesses about deleting data that will one day become useful, further fueling the trend of storing it indefinitely.

These trends are combining to raise significant questions around the best way of storing the huge amount of data involved. In a typical data center, the volume of data that is classified as ‘frozen’ -- meaning it hasn’t been accessed in weeks or years -- is expected to grow from 30 percent in 2020 to 35 percent by 2030. This means over a third of all data stored in data centers will rarely ever be accessed, but still needs to be stored appropriately.

The importance of data security and immutability

As the volume of frozen data grows, its security and immutability -- meaning all aspects of the original data remain unchanged -- is also becoming an increasing concern amongst IT managers.

At present, the most popular way to store data is on enterprise HDDs and SSDs, which are designed for 100 percent online duty cycles and have limited encryption and immutability attributes. As a result, they are seldom deployed for more than five years and often for only three years, which means the data must be continuously re-mastered. They are also prone to inevitable failure over time and do not fare well when powered down and left sitting on shelves. As a result, they offer limited or no “offline airgap” protection from data tampering or data loss.

Tape storage offers a cost effective, sustainable solution

For this reason, a growing number of IT managers are exploring the alternative option of tape storage. Tape offers “offline airgap” data security as well as guaranteed Write Once Read Many (WORM) immutability and a specified 30-year shelf life, with much lower initial acquisition costs/GB than SSDs or HDDs. Although tape will also be plagued with remastering requirements, they should occur over a longer time span at significantly less cost than for HDDs and SSDs.

Tape also offers other benefits over SSD and HDD solutions when used for frozen data storage, particularly where sustainability is concerned. This is because the power consumption required by tape is significantly lower than SSD or HDD alternatives. The vast majority of tape cartridges will not be mounted in the tape drive itself, but will instead reside offline, either in a slot of the robotic library, or in an off-premises location, drawing little to no power. Tape systems also require minimal cooling, further saving costs over long periods of time.

As the volumes of data being produced and stored indefinitely continue to grow at an exponential rate, the need for ultra-low-cost, secure storage solutions has become greater than ever.  Tape storage offers a fantastic alternative to HDD and SSD solutions, not only from a cost perspective, but also from a sustainability point of view. More and more companies are waking up to this reality. In fact, over the last few years, tape storage demand from the world’s biggest hyperscalers has started to eclipse that from traditional enterprise accounts. Hyperscalers rarely get it wrong, so if it’s good enough for them, it’s probably worth investigating for you too.

Image credit: monsit/depositphotos.com

Laura Loredo is HPE Worldwide Product Management Manager, the LTO Program.