4 min read
3 min read
Wondering what's up with data catalogs lately? We were part of a group of data and governance experts at a virtual event hosted by Eckerson Group to talk about exactly that. The event focused on how data catalogs are changing and the crucial role that next generation data catalogs play in managing data effectively.
Joining ThinkData Works were companies like Promethium, data.world, Snaplogic, and Data Integrity First, sharing insights, tips, and trends to help you choose and use data catalog tech.
If you missed the event, don't worry. Here are the main things you need to know:
Data catalogs need to be understood
Wayne Eckerson, President of Eckerson Group, kicked off the event by talking about data catalogs and how they’ve evolved since 2015. They started with basic functionality of organizing data information. Now, they offer advanced features like data management assistance, self-service options, and even machine learning and AI.
Despite their potential, not many companies have adopted data catalogs (successfully, anyway). Complicated structures and compatibility issues with other systems are making it tough. To overcome these challenges, Eckerson suggested getting clear on what we want from data catalogs, involving everyone, learning and evolving continuously, and critically, choosing a technology that can keep up.
Nitin Singhal from Snaplogic and David Hendrawirawan from Data Integrity First also weighed in. They agreed data catalogs are essential tools, but we need to understand the data for them to work well. In big organizations, you might have multiple catalogs, so it's important to find one that works with all your tools.
"Define the problem statement very well and what goals you have for the catalogs. They're fantastic for what they do, but they still depend upon your knowledge and your data to be successful,” Singhal shared.
In a nutshell: When choosing a data catalog, it’s important to have clear goals, make sure it works with your other tools, and think about how it can grow and adapt to your needs.
AI and cloud make it easy
During the panel discussion, Byron Jacob from data.world, Kaycee Lai from Promethium, and Lewis Wynne-Jones from ThinkData Works talked about how far data catalogs have come over the years – and how far past the dark ages of hostile interfaces and glorified spreadsheets we are.
They all agreed on the pivotal role data catalogs play in modern data management and emphasized the importance of cloud-native solutions for scalable, flexible, and accessible data catalogs.
They talked about how AI is making data catalogs even more powerful. Users can search for data using plain language and how, generally, AI allows us to offload time-wasting tasks. There are limits though, especially with complex tasks and specialized information.
One of the keys to a successful implementation, the panel suggested, is an intuitive and user-friendly interface. Making sure the catalog actually makes it easier to get more data to more people is key to broader adoption.
In a nutshell: data catalogs are keeping up with the times, even including AI in their workflows, and they’re a pivotal piece of tech for organizations that aim to be data-driven.
Data catalogs are doing better
During the breakout sessions, two key topics were discussed: AI (again) and bringing data into catalogs (not just metadata).
ThinkData Works experts, Lewis Wynne-Jones, VP of Product, and Connor Wilson, Solutions Engineer, started by highlighting the advantages of adding the actual data to data catalogs as a central control plane.
They shared three ways to make data catalogs better:
- first, data virtualization lets you gather and use data from different sources through one platform, no matter where it's stored;
- second, using tools to check data quality helps to make better decisions since they’re based on reliable data;
- finally, granting data access through the catalog unlocks easy and secure collaboration.
Having one catalog that does it all saves time and money. “By virtualizing data and generating data views within the catalog, you can conduct most of the data evaluation and analysis within the same platform, reducing the need to switch between multiple applications and workflows,” Wilson shared.
In another session, Kaycee Lai spoke about how advanced technologies like AI and natural language processing (NLP) are making a big impact on data catalogs. Businesses want to make money from their data, but with so many data sources, it's tough.
He suggested that next-gen catalogs would combine the best of both worlds. It should be easy to find the data we need and work quickly for people who want to analyze it, really leaning into the concept of a data fabric.
"We believe that leveraging the power of technology, such as AI and data analytics, can simplify the workflow and make it easier for people to find answers," Lai said.
In the last break-out, Bryon Jacob illustrated the impact and growth of AI using examples like ChatGPT to show how it can transform data work. He envisioned a world where bots could take over data-related tasks for unprecedented levels of productivity and efficiency.
With effective data cataloging and metadata utilization — using automation —we’re in for a whole new slew of benefits.
In a nutshell: modern data catalogs are finding ways of meeting the needs of modern organizations and staying ahead of the technology curve.
But humans aren’t off the hook
To wrap up the event, Wayne Eckerson left attendees with three key takeaways for managing data catalogs.
- We need to take care of our metadata.
- We need to involve people in curating the catalog.
- We need to be careful with AI tools to avoid quality, bias, and IP risks.
“You need to curate what your catalog is, and it involves people and processes. Spend equal time on people and processes to ensure success, " Eckerson suggested.
Add data to your day
If you want to make the most of your data, stay up to date on data catalogs — they’re not what they used to be, and that’s a really good thing. The extra capabilities they offer now will help you get in the best position to win the data game.
Want to learn more? Contact our team or watch the event recording.
4 min read
How to better leverage data for risk management and crisis response
It’s becoming increasingly difficult to manage risk in a global climate that’s growing in complexity. On the other side of this coin, however, is an...
3 min read