The traditional approach to data governance needs an overhaul. Centralised teams have long been the norm in most of the organisations I work with, but it’s becoming increasingly clear that these rigid structures create bottlenecks and can’t satisfy the demands of modern data processing. They often lack agility, stifle innovation and in particular can hinder the progress of AI and the creation of Machine Learning (ML) tooling.
Here’s where federated governance can provide the answer. This decentralised approach to managing data and making decisions, gives a new perspective; focused on principles rather than processes. It reminds me of when software architecture changed from monolithic mainframes to micro-services, allowing organisations to make use of centralised data platforms whilst embracing decentralised decision-making.
It prompts you to consider how you want your data to be processed, governed, and shared, setting the stage for a more agile and collaborative approach. A move to this way of working has now become essential if you want to introduce transformational technologies.
Who can you trust?
In the world of AI and ML, data quality rules. Federated governance acknowledges this and relies on all parties sticking to the same data standards. It’s not just about having data; it’s about having data of a known quality. If you want to use information from external sources you need to be able to trust it. Using a decentralised approach it’s crucial to minimise the risks associated with inconsistent data and ensure integrity throughout the process.
The two phases of data science – training and inference, where you’re using models to make predictions or draw conclusions, are heavily reliant on data quality. If the data you’ve used for training is flawed, it can compromise the entire process. Therefore, establishing stable and reliable data sources is key. Understanding the sources of your data is also vital. In my opinion, unknown data quality is the worst scenario, as it leaves you in the dark about which techniques to adopt for improvement.
Data owners know best
Centralised teams often struggle to keep expertise across diverse domains, but with federated governance, the data expertise stays aligned with the domain knowledge. In this way, each team governs its own data according to agreed-upon principles. Identifying the data with meta tags then becomes all important, allowing those closest to the data to decide what the access policies should look like. I believe that all of this helps to foster a more agile and inclusive approach to data management, moving away from bureaucratic processes.
Cultural shift
To get buy-in for this new approach, it’s important to understand that you’ll also need to support a cultural change. As the teams who own the data begin to define the access principles and take on the responsibility for that data, data management starts to become a wider team responsibility. This shift from leaning on a central team means that some stream-aligned teams will need on the job training and support to be helped through the initial learning phase. By gradually decentralising responsibilities, domain experts will ultimately feel more empowered and enabled, and you’ll rapidly see the rewards of this transition.
Unlocking data freedom
So if you’re looking to take advantage of the full potential of AI and ML technologies, moving to a federated governance model is essential . By decentralising, encouraging collaboration, and driving cultural change, you’ll be able access the full potential of your data, and drive innovation for your clients and end users.
At Made Tech, we recently helped Hackney council to harness the power of their data, creating a new platform that would allow them to store their information, consolidate and analyse it all in one place. We worked closely with the team to design and deliver a cloud platform to help the council get better visibility of their data and deliver the best possible service for local residents.
I’m hosting an upcoming event with Government Transformation Magazine on the theme of federated data governance and how it will be crucial for AI integration in the Public Sector. Sign up for our newsletter if you want to be one of the first to read the post-event round-up and in the meantime visit our web page to find out more about our data capabilities.
About the Author
Jim Stamp
Head of Data and AI at Made Tech
Jim has over two decades of experience working in and around software development, spanning many disciplines. He has led the way in establishing Made Tech’s data capability while also leading several major data platform projects.
