bodyType

customContents

entityId

entityUuid

articlePageQuery

filter

article

articleBaseAdvertisingInfoArticle

articleListArticle

articleSeoArticle

helpersCheckIsPostiesArticle

hooksArticleAcknowledgementGateOverlayArticle

hooksTrackPageViewArticle

hooksContentInterestArticle

articleLinkingDataContent

articleSchemaContent

liveArticle

advertZone

sentiment

contentLock

topics

types

sections

moreOnThisArticles

urlAlias

commentCount

sponsorType

authorLocations

authors

paywallTypes

corrections

displaySlideShow

multimediaEmbed

printHeadline

seriesContainer

socialHeadline

headline

flag

firstTopic

glossary

flattenTypeIds

publishedDate

additionalInfoBoxes

images

image

relatedLinks

relatedNewsletters

__typename

createdDate

sponsor

factSheets

series

disableOffPlatformContent

published

updatedDate

summary

subHeadline

body

keywords

copyrighted

customTimezone

liveContents

Stuart Biggs

With the world creating more terabytes of data, large firms are not so much worried about capacity nowadays as they are manageability

The digital era is throwing up some mind-boggling statistics:  62 billion e-mails are sent every day;  1 billion songs are shared via peer-to-peer every month; 1.5 billion terabytes of data is created every year.

These numbers have conspired to do what few people could ever have thought possible - make the storage industry look sexy.

Sounds absurd? Look at it this way: all this data (it's no longer just boring spreadsheets and databases, but also cool stuff like digital music, e-mails and instant messenger files) requires somewhere to store it. And storage firms are coming up with some seriously powerful kit to do just that.

'The world generates 25 per cent more information every year,' said EMC chief executive Joe Tucci.

'This will go on forever because it is now a self-connected world - the people creating this information are individuals like you and me.'

EMC is talking about a world in which the average home stores around one terabyte (1000 gigabytes) of data.

Yet that's nothing compared with the corporate-class one-petabyte systems it began selling in January.

'In our lifetime we will see a terabyte of information in the home, but it will also need to be stored centrally so that people can have ubiquity of access,' Mr Tucci said.

EMC's system breaks the one petabyte (1 million gigabyte) threshold thanks to 2400 linked 500-gigabyte hard drives, which together are capable of storing a staggering 250 million songs or 212,765 DVD quality movies.

This would seem excessive were it not for other companies like Fujitsu  rolling out similar systems in recent months, illustrating just how much the industry has developed in the decade since storage capacity was predominantly measured in megabytes.

There once was a time when only the world's largest institutions had to deal with large volumes of data,  but these days almost every company is increasing the volume of e-mails, Word documents and customer records they store.

Video is another space gobbler, and popular sports franchises such as the Boston Red Sox  are talking about how storing their players' at-bats helps win them games. According to the club's director of IT, Steve Conley, the club now stores five terabytes of video, which will morph to about 20 terabytes by the end of this season.

The Red Sox are not alone.

Globally, the total capacity of hard drives shipped in 2002 was about 8.5 million terabytes. Last year, that figure rose to 15 million terabytes and will continue to rise as digital camera sensors get larger  and the number of photos zipping between colleagues on a slow work day increases. In 2003, 10 per cent of the world's population created 610 billion e-mails.

Although the monster capacities of today's storage systems understandably grab the headlines, the reality is that, thanks to Moore's Law, building the physical hardware is the smallest part of the storage challenge facing companies and their vendors.

'The ability to build a very large array is not difficult,' says Hitachi Data Systems south Asia vice-president Tom Zack.

'What matters is how storage vendors make sure it is delivered at the service levels an organisation or application requires.'

Put simply, it means giving companies the tools to find that critical e-mail lost somewhere in the proverbial data stack.

With new regulations like Sarbanes-Oxley taking effect, companies are finding that diligently storing every e-mail and word document is no longer enough. At some point, somebody may actually ask to see it.

'The inability to satisfy compliance requirements can be damaging to a company, both financially and otherwise,' said CA's Asia-Pacific marketing director Phoebe Hsu.

'Storage forms the foundation for companies to better manage risks associated with business continuance, litigation and provide better corporate governance.'

The leading storage companies recently went on a multibillion-dollar spending spree to acquire and develop software that will help banish their image as black box vendors once and for all.

This includes programmes for e-mail archiving, virtualisation technology to let roaming users access files regardless of where they are physically stored, and tiering systems to ensure that data nobody will ever need is not taking up prime location on a company's most expensive servers. Not every firm will be hoarding data like Google or a financial institution, or even be prepared to fork out the estimated US$4 million for a one-petabyte system.

But it seems that nobody will entirely escape this trend, either. In the future, even security will take up more disk space.

Companies like EMC are looking at ways to use metadata - the information which tracks all the changes made to a document - to restrict access to documents and track their usage across a network. It sounds logical, but in some cases could lead to the metadata growing larger than the actual file it is there to protect.

'People are not worried about capacity nowadays nearly as much as they are about manageability,' said EMC chief technology officer Mark Lewis.

The petabyte

(1 million gigabytes)

A one-petabyte iPod can store 250million songs

A one-petabyte hard drive can store 212,765 DVD quality movies

The Internet Archive  Wayback Machine,  a repository for old websites, contains almost two petabytes of data and is growing at a rate of 20 terabytes a month

A 2004 press report estimated that search engine Google had stored between 1.8 and five petabytes of data

In October 2005, files traded over peer-to-peer network Kazaa equalled 54 petabytes



Big numbers add up to make storage sexy