Contact

Why Many Public Sector Data Projects Stall Before They Deliver

Across councils, local authorities and government departments, there’s a pattern many leaders recognise – even if it’s rarely stated quite this plainly.

Data projects often begin with genuine intent and ambition. Skilled people are involved. Proven technologies are available. And yet, many initiatives still struggle to deliver exactly what they originally set out to achieve – whether that’s on time, on budget, or at the level of value hoped for.

This isn’t a reflection of apathy or poor capability. More often, it’s the result of how complex, high‑stakes work naturally unfolds in large public organisations. Studies of public‑sector IT and data programmes consistently show that under‑delivery is common even where technical capability exists.

In our experience, the conditions that lead to difficulty are usually present long before any tools are selected or any data is moved. They emerge quietly, early on – often in the way the work is framed.

That’s where the story usually begins.

When the Brief Tries to Do Too Much

Public sector data initiatives often start with the best of intentions. The brief is designed to be inclusive, future‑proof and comprehensive. It aims to account for current challenges, anticipated policy shifts, organisational change and long‑term transformation – all at once.

The result is often a scope that’s wide, ambitious, and difficult for any one team to truly own.

Research into large public‑sector IT and data programmes shows that expansive scope and scale are among the strongest predictors of delivery risk. Multi‑year programmes promise significant transformation before any value has been proven. Discovery phases extend to incorporate every perspective, every dependency, every edge case.

Insight is generated – but over time, the marginal benefit declines.

In one recent engagement, a discovery phase ran for six months. With hindsight, most of the meaningful understanding of the current state could have been surfaced in four to six weeks. The remaining time wasn’t wasted – it just didn’t materially change the decisions that followed.

When More Time Creates Less Momentum

It’s understandable to assume that more time leads to better outcomes. In practice, extended timelines often produce different dynamics.

Longer projects invite more discussion, deeper analysis and broader consultation. More stakeholders quite rightly want visibility and involvement. Attention shifts toward rare scenarios rather than everyday operational decisions.

What often reduces, however, is momentum.

Evidence from public‑sector IT “megaprojects” shows that the longer initiatives run, the more likely they are to exceed cost and schedule expectations – with each additional year significantly increasing the risk of overruns.

As programmes expand, decisions become harder to land. Conversations repeat. Complexity increases. The work continues, but progress becomes harder to point to.

This isn’t about individuals doing the wrong thing. It’s a structural effect. Research consistently shows that the majority of risk in large public‑sector programmes comes from management and governance, rather than from technology itself.

When Everyone Is Involved, But Decisions Still Drift

Longer timelines tend to bring larger stakeholder groups. Inclusivity matters – but accountability can quietly diffuse.

Meetings are held to gather views, but not always to conclude them. Discussions are rarely grounded in a single, current source of data. Figures are quoted from historic reports, remembered presentations, or assumptions that once felt reasonable.

No‑one is deliberately misleading anyone. But without a tight scope and a clearly named decision owner, estimates and anecdotes can harden into “facts” that are difficult to question – because questioning them feels disruptive, politically sensitive, or simply time‑consuming.

Oversight bodies and large‑scale project reviews consistently identify diffuse accountability and unclear ownership as recurring causes of delay and under‑delivery in public‑sector initiatives.

Over time, projects remain active, but direction becomes less clear.

A More Reliable Way to Build Traction

The alternative isn’t radical. It’s practical.

Start smaller. Focus narrowly. Make early success achievable.

Instead of attempting organisation‑wide transformation, begin with:

  • One data source
  • One priority question
  • One output that supports a real operational decision

A short, time‑boxed initiative – often around 90 days – is usually enough to demonstrate value, uncover genuine constraints and build confidence across stakeholders. This mirrors the approach advocated in research on iterative and agile delivery in the public sector, which finds that shorter cycles encourage clearer decision‑making and faster progress – delivering services up to 50% faster and improving satisfaction by up to 25%.

Proving Value Before Promising Transformation

When everyone is involved, accountability can thin.

When timelines stretch into years, urgency often fades.

The most successful data programmes we see rarely start with larger budgets or broader mandates. They start with a disciplined question, a contained scope, and the intent to learn quickly.

This aligns with wider findings in data and AI research, where a majority of initiatives fail to reach sustained value precisely because ambition outpaces organisational readiness and delivery structure.

A useful rule of thumb is this:

Design your next data initiative around what you can prove in three months – not what you hope to deliver in three years.

That shift alone moves the conversation from ambition to traction – and gives leaders something solid to build on.

Sources & Further Reading

The observations in this article align closely with findings from academic research, government oversight bodies and large‑scale public‑sector delivery reviews, including:

Other Articles

What do you need us to simplify, automate or innovate for you?