The new metrics for unstructured data management Komprise Director Apply on EasyApply Create a free account to apply in seconds Randy Hopkins, Komprise8:20 am, PT, December 4, 2022The rate of data growth worldwide in the past few years has been greater than in the previous two decades. Data is predicted to more than double again over the next few years — reaching 175 zettabytes in 2025, according to IDC. Most of this data is not structured and includes documents, video, images, instrument and sensor data, text and chats and more. Unstructured data is harder to find, move and manage because it doesn’t live in rows and columns in a database but is dispersed across countless applications and storage locations inside and outside of the enterprise.The explosion of data and the diversity in data types today is bringing a host of new challenges for enterprise IT departments and data storage professionals. These include escalating storage and backup costs, management complexity, security risks, and an opportunity gap from hindered visibility. To solve these issues, we need new, smart analytics and metrics. These must go beyond legacy storage metrics, to focus on understanding data and involving application owners and department and business stakeholders in data management decisions. These metrics should also include measures to track and improve energy consumption to meet broader sustainability goals, which are becoming critical in this age of cyclical energy shortages and climate change. First, let’s review what storage metrics IT departments have traditionally tracked:Legacy storage IT metricsOver the last 20-plus years, IT professionals in charge of data storage tracked a few key metrics primarily related to hardware performance. These include:The new metrics: Data-centric versus storage-centric The traditional IT infrastructure metrics above are table stakes today for any enterprise IT organization. In today’s world, where data is the center of decisions, there are a host of new data-centric measures to understand and report. Departments and business unit leaders are increasingly responsible for monitoring their own data usage — and often paying for it. Discussions with IT organizations can be contentious when, while IT is trying to conserve spend and free up capacity, business leaders are uneasy about archiving or deleting their own data. These metrics help bridge the gap:Visibility into data-centric versus storage-centric metrics helps IT and departments make better decisions together. However, these metrics have been historically difficult to gather because of the prevalence of data silos in enterprises, with data spread across many applications and storage environments, from on-premises to the edge and cloud.Getting this data requires a way to find and index data across vendor boundaries, including cloud providers, using a single pane of glass. Collating data between all your storage providers to get these metrics is possible yet manually intensive and error-prone. Independent data management solutions can help achieve these deeper and broader analytics goals.The new metrics: Sustainable data managementThe global energy crisis, worsened by the war in Ukraine and the surge in demand from the post-pandemic economic recovery, is fueling corporate sustainability programs as well as investment in new green technologies worldwide. Managing data responsibly is no small part of this overall initiative. Most organizations have hundreds of terabytes of data which can be deleted but are hidden and/or not understood well enough to manage appropriately. Storing rarely used and zombie data on top-performing Tier 1 storage (whether on-premises or in the cloud) is not only expensive but consumes the most energy resources. Data centers must reduce their climate footprints if we are to mitigate climate change. The sustainability-related data management metrics below can help measure and reduce energy consumption as relates to data storage:The new data managementIt is true: Investing in new initiatives to expand metrics programs requires time, resources and money. So why do it?For one thing, having better and more extensive metrics on data can inform cost-effective and sustainable data management decisions — easily cutting spending and energy usage by 50% or more. But there is more: Your users (data consumers) will also benefit from having detailed insights into their data. Understanding data and being able to quickly search on data characteristics such as file type or metadata tags (like a project keyword) can drastically reduce the amount of time spent searching for data. An estimated 80% of the time spent conducting AI and data mining projects is spent finding the right data and moving it to the right place.In critical sectors like healthcare, agriculture, government, utilities and manufacturing, there is always a need for faster insights to solve hard problems like creating a new treatment for a chronic condition; improving electric car batteries or wind turbine propulsion; or adjusting soil nutrients to produce a larger yield of crops. In today’s data-driven economy, basic storage metrics are no longer enough to be competitive and meet vital marketplace and operational goals.Randy Hopkins is VP of global systems engineering and enablement at Komprise.Welcome to the VentureBeat community!Our guest posting program is where technical experts share insights and provide neutral, non-vested deep dives on AI, data infrastructure, cybersecurity and other cutting-edge technologies shaping the future of enterprise.Read more from our guest post program — and check out our guidelines if you’re interested in contributing an article of your own!More