We’ve watched UK public sector organisations struggle with the same problem for years now.
They know they need data and AI capabilities. They see the potential. They’ve read the strategy documents. But when it comes to building the actual team, everything gets murky.
Should you hire a Chief Data Officer? What’s the difference between a data engineer and a data scientist? Do you really need AI developers when you haven’t even sorted out your data architecture?
Here’s what we’ve learned: you need all of these roles. But you don’t need to employ all of them full-time.
The UK public sector faces a dramatic capability gap. Nearly 9 in 10 public sector workers admit their organisation can’t fully leverage AI today. Around 65% are experimenting with AI, but only 30% have actually integrated it into how they work.
That gap between experimentation and execution? It exists because organisations don’t have the right mix of specialised roles working together.
The Strategic Leadership Layer: Chief Data Officers and Chief AI Officers
Let us start at the top.
Your Chief Data Officer and Chief AI Officer aren’t optional anymore. The UK government just hired Kalbir Sohi as the country’s first chief artificial intelligence officer, the most senior AI leadership role in the public sector. That tells you everything about where this is heading.
These roles do something your other leaders can’t do. They translate between technical possibility and organisational reality. They set governance frameworks. They make decisions about ethics, privacy, and risk that protect your organisation and the citizens you serve.
Chief Data Officers own your data strategy. They determine what data you collect, how you store it, who can access it, and how you ensure quality across systems. In the public sector, where you’re handling sensitive citizen information and operating under strict regulatory requirements, this role becomes critical.
Chief AI Officers focus on AI strategy and responsible deployment. They answer questions like: Where should we apply AI? How do we ensure our AI systems are fair and transparent? How do we build public trust in our AI initiatives?
Trust matters more than you think. The NHS is trusted by 63% of citizens to use AI responsibly. Central government? Only 39%. Local councils? 44%. That trust deficit makes strategic AI leadership absolutely essential.
Do you need these roles full-time? If you’re a large organisation with complex data ecosystems and ambitious AI plans, yes. If you’re smaller or just starting your journey, you can bring in interim Chief Data Officers or AI consultants to establish your foundations and governance frameworks.
The Architecture Layer: Data Architects Who Design Your Foundation
Data architects are the people who stop your organisation from building a mess.
They design how data flows through your systems. They create the blueprints for databases, data warehouses, and integration points. They make decisions about technology platforms and ensure everything can actually talk to each other.
Without data architects, you end up with what we call “bolt-on chaos.” You have 15 different systems that don’t connect. Your data lives in silos. Every new project becomes a custom integration nightmare.
Sound familiar? 45% of government AI applications are implemented as bolt-on experiments or standalone tools rather than being embedded into core workflows. That’s what happens when you skip the architecture phase.
Data architects prevent this. They think long-term. They design systems that scale. They make sure your data infrastructure can support both today’s analytics needs and tomorrow’s AI ambitions.
You need data architects during major system implementations, platform migrations, or when you’re establishing new data capabilities. But once the architecture is designed and implemented, you don’t need them every day. This makes them perfect candidates for project-based outsourcing or contract work.
The Engineering Layer: Data Engineers Who Build the Pipelines
Data engineers turn architectural blueprints into working systems.
They build the data pipelines that move information from source systems into your data warehouse. They automate data quality checks. They optimise performance. They maintain the infrastructure that keeps everything running.
This is hands-on technical work. Your data engineers write code, configure databases, troubleshoot integration issues, and ensure data flows reliably 24/7.
The demand for these skills has exploded. Demand for workers with specialist data skills like data engineers has more than tripled over five years, up 231%.
Here’s the thing about data engineering: the workload isn’t constant.
You need heavy engineering effort when you’re building new pipelines, integrating new data sources, or migrating platforms. Once systems are stable and running, the work shifts to maintenance and occasional enhancements.
This makes data engineering another strong candidate for flexible staffing. Keep a small core team for ongoing operations, then scale up with contractors or outsourced specialists when you have major projects.
The Analysis Layer: Data Analysts Who Turn Data Into Decisions
Data analysts are your translators.
They take raw data and turn it into insights that people can actually use. They build dashboards, create reports, identify trends, and answer the questions your organisation asks every day.
In the public sector, analysts might track service delivery metrics, identify patterns in citizen complaints, measure program effectiveness, or support budget planning with data-driven forecasts.
The difference between analysts and other data roles? Analysts focus on understanding what happened and why. They work closely with business stakeholders. They communicate findings to non-technical audiences. They drive day-to-day operational decisions.
You need data analysts as permanent staff. Their work is continuous. They build institutional knowledge. They understand your organisation’s context and can spot anomalies that automated systems might miss.
But you might still outsource specialised analytical work. If you need advanced statistical modelling for a specific policy evaluation, or if you’re conducting a one-time citizen survey analysis, bringing in external analytical expertise makes sense.
The Science Layer: Data Scientists Who Find Hidden Patterns
Data scientists go deeper than analysts.
They build predictive models. They apply machine learning algorithms. They find patterns in complex datasets that traditional analysis would miss. They answer questions like: Which citizens are most at risk of needing intervention? How can we predict service demand six months from now? What factors actually drive program outcomes?
The skills gap here is severe. Universities produce fewer than 10,000 data scientists per year in the UK, but there are potentially at least 178,000 unfilled data specialist roles across the economy. 97% of organisations report at least one AI skills gap, with more than half identifying technical shortages in areas like data science.
This scarcity makes data scientists expensive and hard to retain.
You need data science capabilities, but you might not need full-time data scientists. Many public sector organisations benefit from project-based data science work. Bring in specialists to build predictive models, validate approaches, or solve specific analytical challenges. Then hand off the results to your analysts and engineers for ongoing operation.
The exception? If you’re running continuous experimentation, building multiple models, or data science is core to your mission, you need permanent data science capacity.
The AI Development Layer: AI and GenAI Developers Who Build Intelligent Systems
AI developers and GenAI developers represent the newest layer in this ecosystem.
AI developers build and deploy machine learning models into production systems. They take the experimental work from data scientists and turn it into reliable, scalable applications that serve real users.
GenAI developers specifically work with large language models and generative AI technologies. They build chatbots, document processing systems, content generation tools, and other applications powered by models like GPT or similar technologies.
The public sector is just starting to figure out how to use these capabilities effectively. While 60% of UK public servants say AI use has increased over the past year, adoption remains largely confined to basic tasks like drafting and analysis. Fewer than one in three use AI to improve workflows.
AI developers bridge that gap. They embed AI into core processes instead of leaving it as a standalone experiment.
Do you need permanent AI developers? It depends on your AI maturity and ambitions. If you’re just starting to experiment with AI, you probably don’t. Bring in contractors or work with specialised AI development firms to build your initial applications and establish best practices.
As your AI usage grows and you start building multiple AI-powered services, you’ll want permanent AI engineering capacity. But even then, you’ll likely supplement with external expertise for specialised projects or emerging technologies.
The Consultant Layer: External Expertise When You Need It
Throughout this discussion, we’ve mentioned consultants and external specialists.
Data and AI consultants serve several purposes. They bring specialised expertise you don’t have in-house. They provide objective outside perspectives. They help you avoid expensive mistakes by learning from what’s worked (and failed) elsewhere.
Consultants are particularly valuable for:
- Strategy development when you’re defining your data or AI roadmap
- Technology selection when you’re choosing platforms or tools
- Implementation guidance during major projects
- Capability assessment when you need to understand your current state
- Training and upskilling when you’re building internal capabilities
- Governance frameworks when you need to establish policies and processes
The key with consultants is knowing when to use them and when to build internal capacity. Use consultants to establish foundations, tackle specialised challenges, or provide surge capacity. But don’t outsource everything. You need internal expertise to sustain and evolve your capabilities over time.
Why You Need All of These Roles (But Not All at Once)
Here’s what we’ve seen happen when organisations skip roles or try to make one person do everything:
Without strategic leadership, you get tactical chaos. Teams build things that don’t align with organisational goals. You waste money on pilots that go nowhere. You create governance and ethical risks that blow up later.
Without data architects, you build systems that don’t scale. Every new project becomes a custom integration. Your data quality degrades. You can’t answer basic questions about where data comes from or whether you can trust it.
Without data engineers, your brilliant architecture stays on paper. Data doesn’t flow. Pipelines break. Nobody can access the data they need when they need it.
Without analysts, you have data but no insights. The information sits there unused. Decisions get made on gut feel instead of evidence.
Without data scientists, you miss opportunities for prediction and optimisation. You’re always reacting to what already happened instead of anticipating what’s coming.
Without AI developers, your AI experiments never become real services. You’re stuck in perpetual pilot mode while your potential impact remains theoretical.
Each role solves different problems. Each role enables the others.
Data architects design the foundation that engineers build. Engineers create the pipelines that feed analysts and data scientists. Data scientists develop models that AI developers deploy. Strategic leaders ensure all of this work aligns with organisational goals and public trust.
The UK has a £400 billion AI opportunity by 2030, but 28% of organisations say skills gaps are already impacting their ability to achieve business goals. You can’t capture that opportunity with half the team.
The Smart Staffing Strategy: Permanent Core Plus Flexible Specialists
You need all these capabilities, but you don’t need to recruit everyone permanently.
Think about your staffing strategy in three layers:
Your permanent core should include the roles you need continuously. For most public sector organisations, that means:
- Data analysts who support ongoing operations and decision-making
- Data engineers who maintain your core data infrastructure
- A data or AI leader (or both) who sets strategy and governance
Your project-based roles are capabilities you need intensely during specific initiatives but not all the time:
- Data architects during major system implementations
- Additional data engineers during platform migrations or new integrations
- Data scientists for specific modeling projects
- AI developers for building new AI-powered services
Your strategic advisors provide specialised expertise and outside perspective:
- Consultants for strategy development and governance frameworks
- Interim executives to establish new capabilities or lead transformation
- Specialised experts for emerging technologies or niche applications
This approach gives you the capabilities you need while managing costs and avoiding the recruitment challenges that come with trying to hire scarce specialists.
What This Looks Like in Practice
Let us give you a concrete example of how this might work for a mid-sized local council.
You hire a permanent Chief Data Officer (or share one across multiple councils). You employ two data analysts who work with departments daily and one data engineer who maintains your core systems.
When you decide to modernise your citizen services platform, you bring in a data architect on contract to design the new data architecture. You augment your engineering team with two additional contract engineers for the 18-month implementation.
You want to build a predictive model for social care demand. You engage a data science consultancy for a three-month project to develop and validate the model. Your permanent analyst then monitors and reports on the model’s outputs.
You’re exploring AI for processing planning applications. You work with an AI development firm to build a proof of concept. If it works, you might hire a permanent AI engineer or continue working with external specialists depending on your broader AI plans.
Throughout all of this, you periodically engage strategy consultants to help you refine your roadmap, assess your progress, and learn from what’s working elsewhere in the public sector.
This hybrid model gives you comprehensive capabilities without the cost and complexity of recruiting and retaining specialists you only need intermittently.
Moving Forward: Building Your Team Strategically
The ‘Public Sector AI Adoption Index 2026’ ranks the UK as sixth out of ten, scoring 47 out of 100. Despite having one of the most ambitious AI agendas in the world, the UK is falling behind in execution.
The gap between ambition and delivery comes down to having the right people doing the right work at the right time.
You need strategic leaders who set direction and manage risk. You need architects who design scalable foundations. You need engineers who build reliable systems. You need analysts who generate daily insights. You need data scientists who find hidden patterns. You need AI developers who turn experimental models into production services.
You need all of these roles. But you can be smart about how you access them.
Start by assessing your current state. What capabilities do you have? What gaps are blocking your progress? What work is continuous versus project-based?
Build your permanent core around the capabilities you use every day. Supplement with contractors and consultants for specialised work and surge capacity.
Most importantly, stop thinking you need to figure this out alone. The skills shortage is real. The competition for talent is intense. But you have options beyond traditional recruitment.
The organisations that succeed will be the ones that build flexible, comprehensive teams that combine permanent staff with strategic use of external expertise.
You don’t need to hire everyone. But you do need access to everyone’s skills.
That’s how you close the gap between experimentation and execution. That’s how you move from ambitious strategy documents to real transformation. That’s how you deliver better services to the citizens who depend on you.