From feedback to forecast at the Met Office
Boosting the Met Office’s insights from their users feedback with a modern analytics data platform.
Challenge
Forecasting weather requires deep scientific knowledge and vast amounts of computing power. World-leading experts, the Met Office are constantly improving their forecasting process – to help us stay safe and thrive. But how can they measure the success of a new weather model? It’s one thing to know that your model is more scientifically accurate. But when it comes to data-driven insights on how users respond to and understand that information – it’s a whole different ball game.
By using data and AI to automate the sentiment analysis of vast amounts of user feedback, we allowed the Met Office to extract patterns and trends with an efficiency that was previously impossible.
We knew we could then combine this with traditional methods of user research that provide in-depth, qualitative insights into user behaviours.
Our data science team helped the Met Office interrogate hundreds of thousands of individual pieces of user feedback. These came from from millions of daily users of the Met Office public website and mobile apps. Our challenge lay in extracting a needle of insight from this haystack of information. Together we set out to demonstrate if a new weather model was more trusted than previous iterations.
Our approach
When we joined, a sole team member was reading hundreds of comments every day on top of their day job. They were trying to extract insight from multiple spreadsheets and data sources. Our data scientists quickly recognised this tiresome task was a perfect opportunity for AI.
To begin, there were conversations around which technology to use. Should we jump on the large language model (LLM) bandwagon, or use a simpler, more tailored solution?
Despite the hype in the industry, we knew an LLM was not appropriate for this use case. Due to the complexity of these systems, their unpredictable behaviour and the large cost associated with the compute. We recommended in this early stage to use well-established natural language processing (NLP).
“Following the advice of our Made Tech colleagues who are experts in this field, we opted for a tried and tested NLP model. As this would be best to categorise the thousands of items of accuracy feedback we receive. The efficiency of this system and the amount of time it has saved extracting insight from our feedback has been invaluable.”
To get started, we broke this task into 3 problems:
- how do we feed this insight back to stakeholders?
- how do we collect and process so much data?
- what insight can we extract from user comments?
In order to process the hundreds of thousands of pieces of user feedback, we built an automated analytics pipeline. This pipeline had to be easy to maintain and robust enough to support computationally heavy machine learning (ML) tasks. It also had to safely strip any personal information that users might have submitted.
Building this platform centralised a single source of truth for all user feedback. We were then able to begin sifting through comments and developing analytics processes. It was in this space that the majority of data cleaning, enriching and preparing processes were carried out.
This central analytics environment instantly began closing the feedback loop between stakeholders and users, empowering the Met Office to begin asking for bespoke pieces of analysis and digging into feedback following feature releases. We presented their user feedback in a completely new light. We helped visualise user distributions, target specific use cases and report on different weather scenarios as they happened. Something that had not been possible before.
We brought in junior developers to the team to provide valuable experience on at greenfield project. By aligning to the Met Office’s strong culture of excellent people, culture and developing capabilities, the team were able to provide a supportive space to learn, upskill and share knowledge across the team and wider Consumer Digital Programme.
Results
“Through their hard work and dedication, our colleagues at Made Tech have created a robust data pipeline. And in doing so, a more efficient way for the Met Office to not only monitor accuracy and cost reporting, but open the door to other areas where accuracy and potentially more general feedback can be monitored. This increasing our validity as an organisation where user-centred experience is at the heart of everything we do.”
Improving the response to user feedback
Thanks to the development of the feedback analytics platform, the Met Office can now actively monitor user feedback and measure the perceived success of new weather models in real time. Stakeholders and product owners can track user sentiment throughout the rollout and precisely quantify the impact of any changes they make to the service.
They’re able to monitor the impact of different weather systems, check model performance across different regions and environments and rapidly respond and iterate on issues as they arise – reducing the feedback-to-development loop from weeks to days. All focused on providing the best service possible for users.
User feedback analytics platform
The new analytics platform empowers the Met Office to conduct more types of data analysis. We’re now able to use powerful NLP techniques to categorise comments, allowing a shift from anecdotal to actual evidence. We can visualise and monitor feedback as it lands, and slice it however needed. For example, if the weather science team needs to know how users in Region X responded to forecast Y, this is now achievable with the click of just a few buttons.
By presenting this analysis in responsive, live dashboards, Made Tech has enabled an interaction with Met Office user data like never before.
Moving forward
AI technology continues to progress at a lightning pace. Smaller, open-source generative AI solutions are now far more available, with performance and reliability nipping at the heels of the larger closed-source models. The next steps for the Met Office will be to further enhance this analysis platform by generating automated, summary reports, condensing a never-ending river of user feedback into easy-to-understand, direct points of action.
This is an excellent example of delivering value to our public sector clients. By working in partnership with us, we have led the Met Office to the adoption of new technologies that harness the power of their own data.
Case studies
Enhancing resilience to overcome website downtime during severe weather events
Collaborating with the Met Office on the challenge of the National Severe Weather Warning Service’ website downtime, caused by spikes in internet traffic during severe weather events.
A strategic data platform review with the Department for Business and Trade
Working closely with the Department for International Trade to review their Data Hub, a self-built customer relationship management system.
