A new approach to Engineering in DWP Digital

By Sean Luke, DWP Digital engineering

We’ve come a long way since first introducing Site Reliability Engineering (SRE) as a discipline in DWP Digital last year, but we’re not standing still. We recently launched our new engineering strategy which aims to deliver automatic, agile, end-to-end pipelines for our product development.

Some of the key things the new strategy focuses on are:

• experimenting with automated code
• embracing open source software
• breaking down boundaries
• trialling 8-week product cycles
• a new product catalogue

We’ve recognised the need to simplify our governance model for deployment of code, making it easier for developers to distribute code to production as autonomously as possible and without putting the department at unnecessary risk.

So we’re taking an experimental approach to transformation and our first major experiment with intelligent automation is now underway. We want to test code delivery methods that no longer require human intervention once the developer has committed changes.

We’re building in safety valves of course, but we’re looking to automate these as far as possible. We’re working closely with colleagues who operate our governance processes and involving them in our experiments.

It’s early days, but there’s a tangible excitement to see how work progresses and what we can learn.

This work is closely associated with development of a new product catalogue that reflects our engineering needs and provides predictable pathways to product consumption. We’ve been bringing engineers together from across DWP Digital to build a more cohesive and collaborative engineering community, and so far our new strategy has been well received.

We’ve defined a direction of travel for our shared services that looks set to, not only adopt more open source products, but an open source culture to maintain services. We’ve been testing these ideas for a while and demonstrated more open and collaborative culture speeds things up and reduces mistakes.

To help with this, we’ve created ‘Chorlton’ – a dynamic operations product, and knowledge platform using open source products. It was built in response to some frustration in the wider team at having to go to the commercial software market to solve basic problems when we already had the engineering talent in-house.

Our engineers developed a prototype within 24 hours, using GDS templates and open APIs for our data repositories. From that prototype we made some major improvements to the code and added lots more operational data.

We also defined some strict principles at the outset that would pay off for us later. We insisted any data we used had to be gathered automatically and that everything on the platform be fully codified.

Applying our principles meant we delivered something of value that required virtually no engineering time to maintain. We focused on open source tools and libraries that were freely available and didn’t use any commercial, off-the-shelf products. This was a major step forward for our team and helped to build strong cross-team engineering relationships which have proved to be both fulfilling and sustainable.

Our maintenance squad is under no more pressure now, than they were, when we launched the prototype. So building scalable services like Chorlton using open source software has been a lot easier than we imagined.

We’ve recently expanded the use of Chorlton to our engineering practice, and they’ve adopted it as their new digital home.

Our focus as an organisation is shifting rapidly towards engineering and we’re trialling alternative methods for delivering our engineering products. We’re developing a true end-to-end code pipeline that will standardise the route to production for our product development units.

Early indications from our engagement with engineers suggest that this is a much-needed shift towards a more automated and intelligent way of getting things done.

We’ve removed silos as much as possible to build a common engineering capability. These changes have given us a more focused team and helped to unlock creativity.

We’re experimenting by working closer with product development teams at an engineering level, and recently had a positive result with the launch of a new shared service in record time. And a collaborative project underway to implement automatic code pipelines. This makes delivery of our shared services faster, more relevant, and more scalable to business needs.

We’ve been trialling an 8-week product cycle which provides engineers with a 6-week product delivery and a 2-week decompression.

We currently run continuous 2-week sprint cycles with formal agile ceremonies. But we wanted to see what would happen if we dialled back the ceremonies, enabled a longer delivery cycle, and provided some kind of break for engineers to take a breath before the next product delivery.

The approach is based on many years of research undertaken by our friends at Basecamp. They settled on a 6-week delivery period because it’s long enough to build something meaningful from start-to-finish, and short enough that everyone can feel the deadline looming from the start, so they use time wisely.

We’re part way through this experiment, but early indications are that we’ve reduced ceremony time from around 35 per cent to just 5 per cent. And we’re monitoring to see if time spent on manual processes reduces as a result of a more creative mindset.

This means that when we do spend time on agile ceremonies it doesn’t feel like we’re just going through the motions – every ceremony has meaning.

We’ve prototyped a new product catalogue that supports the delivery of engineering infrastructure and services for consumption by our product delivery teams.

We played with various approaches to this long standing problem but after many conversations with our engineering community we settled on ‘engineering needs’ as the primary stem from which to build out our catalogue of products.

The biggest challenge for any product catalogue is finding the right balance between usability and architectural integrity. We think we’ve found a good balance by orienting the catalogue around engineering needs and the end-to-end engineering experience.

We’re continuing to build our engineering capability with more creative engineers. If you want to make a big difference to the quality of our products, the usefulness of our digital services, and the lives of our citizens, join us.