DCIG’s soon-to-be-released report on Unstructured Data Management (UDM) offers an in-depth look at how to understand and categorize UDM solutions. However, one aspect of UDM that this report only loosely addresses has to do with data curation’s role in UDM.
Unlike the components of UDM identified in this report, the need for organizations to curate data depends on multiple factors. Further, the process each organization employs will also differ depending on their data curation requirements.
Defining Data Curation
On one hand, defining data curation represents an easy task in today’s world. All one must do is google “data curation” or ask an AI engine and multiple answers appear. Here are three definitions out of the many I quickly found worthy of highlighting:
- A recent article on TechTarget’s site defined data curation as “the process of creating, organizing and maintaining data sets so people looking for information can access and use them.”
- ChatGPT provided this simple definition. Data curation is the process of collecting, organizing, and maintaining data so that it stays accurate, useful, and accessible over time.
- I particularly liked this definition that appears on Monte Carlo Data’s website. It defines data curation as “the process of transforming and enriching larger amounts of raw data into smaller, more widely accessible subsets of data that provide additional value to the organization or the intended use case.”
Data Curation’s Lowest Common Denominator: Process
While all three data curation definitions above have many similarities, they all include “process” in their definition. That merits further examination.
Many organizations want to derive value from their unstructured data using AI as quickly and affordably as possible. After all, who doesn’t? But organizations may assume the unstructured data they plan to use already exists in a usable state. That assumption may not hold true in many circumstances, if not all.
Organizations must minimally consider the current state of their unstructured data. They must ask questions such as:
- How old is the data?
- How was the data created?
- Who or what created the data?
- Did the data come from a trustworthy source?
- Where is the data physically stored? Locally? Remotely? In the cloud? All the above?
- Is the data physically stored on media with sufficient performance to meet our needs?
- Was the data stored in an immutable, secure format?
- Do we need access to all the data or just a subset of it? If so, what subset?
- Does the data constantly change or is it static?
- How much value do they hope to derive from the unstructured data once curated?
Answering questions like these and others like it help organizations determine both what their data curation process should look like and how to best implement it.
Implementing a Data Curation Process
Organizations will find they need to define and implement their own data curation process to address their specific UDM requirements. Admittedly, organizations in some verticals may find some of their duration curation processes more well-defined then others.
For instance, financial, healthcare, and government organizations subject to specific regulatory requirements will have some data curation requirements built in. Further, all organizations must do annual taxes, and financial reporting will also find it easier to create data curation processes for data related to these tasks. However, these more well-defined data curation requirements often impact only limited amounts of unstructured data.
Properly curating large amounts of poorly understood unstructured data or unstructured data originating from dubious sources becomes more subjective. DCIG has put together the following preliminary chart that highlights the level of effort and process required to curate other types of unstructured data. This list goes from lowest level of effort and processing associated with data curation to higher levels of data curation and processing.
Again, readers should note this diagram only represents an admittedly very small subset of all types of unstructured data. Further, DCIG has only begun to examine where different types of unstructured data fit into this diagram. However, this diagram illustrates that different types of unstructured data will require different processes with various levels of effort to properly curate.
Data Curation’s Role in UDM
The forthcoming DCIG report on Unstructured Data Management provides a structure for UDM and how to map current solutions to this structure. However, DCIG has yet to explore and identify solutions that help organizations properly curate their data.
DCIG recognizes data curation serves an important and even critical role in UDM. However, the responsibility for creating and carrying out data curation processes will currently still largely lay upon organizations to perform.
KEEP UP-TO-DATE WITH DCIG
To be notified of new DCIG articles, reports, and webinars, sign up for DCIG’s free weekly Newsletter.
To learn about DCIG’s future research and publications, see the DCIG Editorial Calendar.
Technology providers interested in licensing DCIG TOP 5 reports or having DCIG produce custom reports on their behalf, please contact DCIG for more information.