When planning your data platform, it’s easy to think that the latest technology (with all the bells and whistles) is the answer to everything. But before your budget rapidly disappears, it’s important to remember that technology is only one piece of the puzzle. In my whitepaper ‘Laying the groundwork for AI’, I cover how a successful data strategy balances investment across 3 key areas: technology, culture, and use cases.
So although this blog focuses on the technology pillar, I’d urge you not to lose sight of the other 2 pillars – because they are inextricably linked.
Designing for platform evolution
When you’re building a data platform from scratch, it’s important to accept one truth early on. Most of the technology you choose will eventually need to be replaced. This doesn’t mean you’ve made the wrong decisions. It just means your platform should be designed to embrace change from the start. Evolving architecture and emerging design principles are key. They allow you to build with the expectation of future transformation. Designing for change also means choosing tools and technologies that don’t tie you to a single provider.
How to avoid vendor lock-in
One of the biggest risks when choosing technology is relying so heavily on one provider that switching later becomes difficult or extremely expensive.
Wherever possible, opt for open-source tools and technologies, specifically ones that have managed instances in the main cloud provider platforms. Apache Spark is a great example. Multiple engines support its code execution, so you’re not tied to a specific cloud or compute provider. Similarly, tools like Airflow offer open-source and managed options across the 3 major cloud platforms, allowing you to balance flexibility with convenience.
The goal is to minimise vendor lock-in while maintaining agility. Yes, open-source technologies can bring their own form of dependency, but they provide far more freedom than proprietary solutions.
How do you ensure your platform remains practical and adaptable as your organisation grows?
Start simple, scale when ready
Choosing the right technology isn’t easy. It needs to reflect where your organisation is now and where you’re heading. I often tell clients that over 50% of the technologies they implement will likely be replaced in 2-3 years as their data maturity evolves and their use cases change. Instead of sticking with familiar tools, I assess both their technical and data maturity, as well as the complexity of their use cases. The sweet spot where these factors intersect is where the ideal technology choice lies.
For example, your ingestion mechanisms might vary based on complexity. If you’re just copying CSV files, a hand-crafted Spark script might suffice. But if you’re pulling data from systems like Salesforce or SAP, it’s often better to invest in pre-built connectors rather than writing your own. The same principle applies across all services: start simple, address the immediate need and scale complexity as your platform and team mature.
Avoiding the gold-plated trap
It’s easy to get carried away with the advanced, feature-packed technology solutions that promise to do it all. But if you go all out and invest everything in one perfect solution upfront, you may end up with a gold-plated platform – one that’s overly complex, expensive and possibly redundant in 2 years time.
Instead, focus on your use case and the cultural maturity of your organisation. You can spend a lot of money on any service, but it’s critical to consider where that investment is actually needed. In many cases, a simpler, ‘good enough’ approach will work.
Take one public sector client, for instance. Their primary focus was on cataloguing, discovery and metadata contracts. Recognising that their work revolved around sharing data with other organisations, they prioritised building a robust data catalogue and metadata management system. This meant that their analysts could easily find and use the data they ingested. They designed everything else – ingestion, storage, and transformation – to meet the smallest possible functional, secure and safe standards while keeping costs in check.
By aligning their technology investments to these key priorities, they avoided over-engineering their platform. They delivered exactly what their use case demanded – nothing more, nothing less.
It’s an important reminder to design a platform that prioritises what’s really important to your organisation.
Keep flexibility at the heart of your platform design
I’ve looked at how starting simple, avoiding vendor lock-in, and aligning technology with your organisation’s needs sets the stage for a data platform that can evolve over time. But this is just the start.
In part 2 (coming soon): How to choose a flexible tech stack for your data platform, I’ll cover how to design a platform that’s flexible at every level. From managing data ingestion and storage to embedding DevOps principles like automation, observability and security by design. All these elements work together to ensure your platform stays future-ready.
But technology is only part of the picture. Building for the future means designing for change from the start. Embracing evolving principles and keeping your investments focused on what truly matters to you. For more insights into balancing technology, culture and use cases, check out my whitepaper ‘Laying the Groundwork for AI’.
About the Author
Jim Stamp
Head of Technology at Made Tech
Jim has over two decades of experience working in and around software development, spanning many disciplines. He has led the way in establishing Made Tech’s data capability while also leading several major data platform projects.
