Author: Steve Smith (Page 10 of 10)

Continuous Delivery and organisational change

10 September, 2012 / Steve Smith

Continuous Delivery unaccompanied by organisational change will not reduce cycle time

Our Continuous Delivery value proposition describes a goal of reducing cycle time – the average time for a software release to propagate through to Production – in order to improve our time-to-market, saving time and money that can be invested back into product development and growing revenues. However, it is important to bear in mind that like any cross-organisation transformational programme Continuous Delivery is susceptible to Conway’s Law:

Any organisation that designs a system (defined broadly) will produce a design whose structure is a copy of the organisation’s communication structure

This extraordinary sociological observation predicts that multiple teams working on the same problem will produce disparate solutions, and that the structure of an organisation must be adaptable if product development is to remain sustainable. As a Continuous Delivery pipeline will likely traverse multiple organisational units (particularly in silo-based organisations), these are pertinent warnings that were addressed by Dave Farley and Jez Humble in the principles of Continuous Delivery:

Repeatable Reliable Process
Automate Almost Everything
Keep Everything In Version Control
Bring The Pain Forward
Build Quality In
Done Means Released
Everybody Is Responsible
Continuous Improvement

The majority of these principles are clearly focussed upon culture and behaviours, yet some Continuous Delivery implementations are entirely based upon Reliable Repeatable Process and Automate Almost Everything at the expense of more challenging principles such as Everybody Is Responsible.

For example, in our siloed organisation we are asked to improve the cycle time of an application from 28 days to 14 days, with the existing deployment and migration mechanisms manual processes that each take 20 minutes to perform. We introduce a Continuous Delivery pipeline in which we Automate Almost Everything, we Keep Everything In Version Control, and we establish our Repeatable Reliable Process. However, despite deployment and migration now taking only 5 minutes each, our cycle time is unaffected! How is this possible?

To explain this disheartening situation, we need to use Lean Thinking and examine the value stream of our application. While our new release mechanism has reduced the machine time of each pipeline stage (i.e. time releasing an artifact), the process lead time (i.e. time required to release and sign off a artifact) is largely unaffected. This is because process lead time includes wait time, and in a siloed organisation there are likely to be significant handoff periods both during and between pipeline stages which are “fraught with opportunities for waste“. If the deployment and migration mechanisms have each been reduced to 5 minutes but a 3 hour handoff from server administrator to database administrator remains, our Repeatable Reliable Process will never affect our cycle time.

To accomplish organisational change alongside Continuous Delivery, the most effective method of breaking down silo barriers is to visualise your value stream and act upon waste. Donella Meadows recommended that to effect organisational change you must “arrange the structures and conditions to reduce the probability of destructive behaviours and to encourage the possibility of beneficial ones“, and a pipeline containing a Repeatable Reliable Process is an excellent starting point – but it is not the end. Visualise your pipeline, educate people on the unseen inefficiencies caused by your organisational structure, and encourage an Everybody Is Responsible mentality.

Updating a Pipeline

28 August, 2012 / Steve Smith

Pipeline updates must minimise risk to protect the Repeatable Reliable Process

We want to quickly deliver new features to users, and in Continuous Delivery Dave Farley and Jez Humble showed that “to achieve these goals – low cycle time and high quality – we need to make frequent, automated releases“. The pipeline constructed to deliver those releases should be no different and frequently, automatically released into Production itself. However, this conflicts with the Continuous Delivery principle of Repeatable Reliable Process – a single application release mechanism for all environments, used thousands of times to minimise errors and build confidence – leading us to ask:

Is the Repeatable Reliable Process principle endangered if a new pipeline version is released?

To answer this question, we can use a risk impact/probability graph to assess if an update will significantly increase the risk of a pipeline operation becoming less repeatable and/or reliable.

This leads to the following assessment:

An update is unlikely to increase the impact of an operation failing to be repeatable and/or reliable, as the cost of failure is permanently high due to pipeline responsibilities
An update is unlikely to increase the probability of an operation failing to be repeatable, unless the Published Interface at the pipeline entry point is modified. In that situation, the button push becomes more likely to fail, but not more costly
An update is likely to increase the probability of an operation failing to be reliable. This is where stakeholders understandably become more risk averse, searching for a suitable release window and/or pinning a particular pipeline version to a specific artifact version throughout its value stream. These measures may reduce risk for a specific artifact, but do not reduce the probability of failure in the general case

Based on the above, we can now answer our original question as follows:

A pipeline update may endanger the Repeatable Reliable Process principle, and is more likely to impact reliability than repeatability

We can minimise the increased risk of a pipeline update by using the following techniques:

Change inspection. If change sets can be shown to be benign with zero impact upon specific artifacts and/or environments, then a new pipeline version is less likely to increase risk aversion
Artifact backwards compatibility. If the pipeline uses a Artifact Interface and knows nothing of artifact composition, then a new pipeline version is less likely to break application compatibility
Configuration static analysis. If each defect has its root cause captured in a static analysis test, then a new pipeline version is less likely to cause a failure
Increased release cadence. If the frequency of pipeline releases is increased, then a new pipeline version is more likely to possess shallow defects, smaller feedback loops, and cheaper rollback

Finally, it is important to note that a frequently-changing pipeline version may be a symptom of over-centralisation. A pipeline should not possess responsibility without authority and should devolve environment configuration, application configuration, etc. to separate, independently versioned entities.

Pipeline Pattern: Artifact Container

6 August, 2012 / Steve Smith

A pipeline should be decoupled from artifact content

Note – this pattern was previously known as Binary Interface

In a Continuous Delivery pipeline, a simple Commit stage implementation may equate an application artifact with the compiled artifact(s) e.g. a JAR or a WAR:

This approach may suffice for a single application pipeline, but the coupling between start/stop behaviour and artifact file type means that details of java -jar, $CATALINA_HOME/bin/startup.sh, etc. seep into the pipeline start/stop stages and Operations documentation for manually starting/stopping artifacts. This becomes more of an issue when a pipeline manages multiple applications comprised of different web server technologies, different build tools, and/or different programming languages:

Each new artifact type introduced into the pipeline requires a notable increase in complexity, as conditional behaviour must be incorporated into different pipeline stages and Operations must retain knowledge of multiple start/stop methods. This threatens the Continuous Delivery principle of Repeatable Reliable Process and is a significant barrier to pipeline scalability.

The solution is to introduce a Artifact Container as the output of the Commit Stage, so that artifacts appear identical to the pipeline:

The advantage of this strategy is that it minimises the amount of application-specific knowledge that can leak into the pipeline, empowering development teams to use whatever tools they deem necessary regardless of release management. A change in web server, build tool, or programming language should not necessitate a new pipeline version.

Pipeline Antipattern: Deployment Build

18 July, 2012 / Steve Smith

Continuous Integration “Has A” Continuous Delivery is the wrong way around

Eric Minick has written a thought-provoking assessment of Continuous Delivery and Continuous Integration tooling, which includes a variant of The Golden Hammer:

“When all you have is a Continuous Integration system, everything looks like a build”

This leads to an antipattern Eric and I refer to as Deployment Build, in which application deployments are tacked onto a Continuous Integration system by treating them as pseudo-builds. While this approach may be cheap to set up, it creates a number of problems:

Ambiguous language – mis-communication is more likely when a deployment button is mis-labelled as a build
Noisy user interface – endless buttons such as “Deploy Apples To QA”, “Deploy Apples To Production”, and “Deploy Oranges To QA” hinder feedback
Lax security – all downstream servers must be accessible including Production
Increased risk – a system failure will impede Operations as well as Development

Eric describes how Deployment Build drove UrbanCode to create uDeploy independent of AntHillPro, and ThoughtWorks Go has Continuous Delivery at its heart. Jenkins now has a Continuous Delivery plugin, although to say Continuous Integration “has a” Continuous Delivery capability is incorrect. The correct relationship is the inverse.

Pipeline Pattern: Aggregate Artifact

25 June, 2012 / Steve Smith

Aggregate Artifacts can incrementally deliver complex applications

When pipelining inter-dependent applications, the strength of the pipeline architecture directly correlates to the assembly cost and scalability of the packaging solution. If the Uber-Artifact approach is tacitly accepted as a poor implementation choice, is there an alternative?

The inherent value of any packaging solution is the version manifest mapping of package name/version to constituent artifacts, and there is no reason why that manifest cannot be managed as an artifact itself. In terms of Domain-Driven Design a version manifest is a naturally occurring Aggregate, with the package name/version equating to an Aggregate Root and the constituent artifacts represented as Entities, suggesting a name of Aggregate Artifact.

In an Aggregation Pipeline, the multiple pipelines of an Integration Pipeline are collapsed into a single pipeline with multiple commit stages. A successful commit of a constituent artifact triggers the commit of an Aggregate Artifact containing the new constituent version to the binary repository. At a later date the release stage fetches the aggregate artifact and examines the pipeline metadata for each constituent. Each constituent already known to the target environment is ignored, while the previously unknown constituents are released.

There are a number of advantages to this approach:

Consistent release mechanism. Whether a artifact is released independently or as part of an aggregate, the same process can be used
No duplication of artifact persistence. Committing an aggregate artifact to the binary repository does not necessitate the re-persistence of its constituents
High version visibility. An aggregate artifact is human and machine readable and can be published in multiple formats e.g. email, PDF/HTML release notes
Lightweight incremental release process. As an aggregate artifact is a manifest a version diff with earlier releases is easy to implement

As Aggregate Artifact persistence can be as low-tech as a properties file, the cost of the aggregate commit stage is extremely low. This means that a single Aggregate Artifact can scale to support many constituents (of which some may be Aggregate Artifacts themselves), and that failure scenarios can be easily handled.

For example, if a release of Fruit Basket 1.0 fails with the successful constituent Apples 23 and the unsuccessful constituent Oranges 49, then Stop The Line applies to Fruit Basket 1.0 and Oranges 49. Once a fix has been committed for Oranges 49, a new Fruit Basket 1.1 aggregate containing Oranges 50 and the previously successful Apples 23 can be quickly created and incrementally released to the environment.

Pipeline Antipattern: Uber-Artifact

17 June, 2012 / Steve Smith

Pipelining inter-dependent applications as uber-artifacts is unscalable

Achieving the Continuous Delivery of an application is a notable feat in any organisation, but how do we build on such success and pipeline more complex, inter-dependent applications? In Continuous Delivery, Dave Farley and Jez Humble suggest an Integration Pipeline architecture as follows:

In an Integration Pipeline, the successful commit of a set of related application artifacts triggers their packaging as a single releasable artifact. That artifact then undergoes the normal test-release cycle, with an increased focus upon fast feedback and visibility of binary status.

Although Eric Minick’s assertion that this approach is “broken for complex architectures” seems overly harsh, it is true that its success is predicated upon the quality of the tooling, specifically the packaging implementation.

For example, a common approach is the Uber-Artifact (also known as Build Of Builds or Mega Build), where an archive is created containing the application artifacts and a version manifest. This suffers from a number of problems:

Inconsistent release mechanism. The binary deploy process (copy) differs from the uber-binary deploy process (copy and unzip)
Duplicated artifact persistence. Committing an uber-artifact to theartifact repository re-commits the constituent artifacts within the archive
Lack of version visibility. The version manifest must be extracted from the uber-artifact to determine constituent versions
Non-incremental release mechanism. An uber-artifact cannot easily diff constituent versions and must be fully extracted to the target environment

Of the above, the most serious problem is the barrier to incremental releases, as it directly impairs pipeline scalability. As the application estate grows over time in size and/or complexity, an inability to identify and skip the re-release of unchanged application artifacts can only increase cycle time.

Returning to the intent of the Integration Pipeline architecture, we simply require a package that expresses the relationship between the related application artifacts. In an uber-artifact, the value resides in the version manifest – so why not make that the artifact?

The Strangler Pipeline – Introduction

18 May, 2012 / Steve Smith

Continuously Delivering greenfield and legacy applications en masse

I recently gave a talk at Agile Horizons 2012 on behalf of my amazing employer Sky Network Services, detailing our yearlong Continuous Delivery journey and the evolution of our Strangler Pipeline. As a follow-up I intend to write a series of articles on our pipeline, as it is a narrative far removed from the “pipelining a single greenfield application” model often found in Continuous Delivery experience reports.

Sky Network Services is an agile, innovative technology company that produces telecommunications middleware for BSkyB. Despite a plethora of talented technical/non-technical staff and an enviable reputation for delivering quality software, an in-house analysis in mid-2011 identified a number of problems:

Many applications used different methods of deployment, start, stop, and database migration in different environments
There was little visibility of which application versions were progressing through the test environments at any given time
Releasing a minor bug fix for an application necessitated a re-release of the parent platform
Development teams and Operations teams were constrained to separate silos

At this point we were attracted to the Continuous Delivery value proposition, albeit with the additional challenge of scaling our pipeline to manage an estate of legacy/greenfield applications that in the past year has doubled in size.

In this series of articles I aim to cover:

Challenges – how we solved the more common Continuous Delivery challenges
Scaling Up – how we scaled our pipeline to manage our ever-growing application estate
Legacy and Greenfield – how we simultaneously release legacy and greenfield applications
Autonomation – how we established a Continuous Delivery transformation across a Waterscrumfall organisation

The Merit of Metadata

19 April, 2012 / Steve Smith

Metadata increases feedback and ensures value stream integrity

In Continuous Delivery, Dave Farley and Jez Humble describe the Lean production principles that underpin Continuous Delivery, and how a pipeline encapsulates a value stream – the journey a customer feature undertakes from discovery to real world consumption.

In a pipeline each stage represents a step in the value stream, meaning that for application XYZ an example value stream of [Development -> Acceptance -> UAT -> Performance -> Production] could be defined as follows:

Pipeline with Metadata

In the above pipeline, each stage ends with a discrete piece of metadata (“created XYZ 2.1”, “XYZ 2.1 passed acceptance tests”, etc.) being written back to the binary repository, indicating that one or more new customer features have progressed in the value stream.

Unfortunately, pipelines are often constructed without metadata support:

Pipeline without Metadata

In this situation the lack of activity data reduces each stage to a fire-and-forget operation, constraining feedback and unnecessarily exposing the value stream to obtuse, time-consuming errors. For example, QA could mistakenly test new features that have not passed automated regression tests, or Operations could mistakenly release features that have not been signed off.

With metadata the following safeguards can be easily implemented:

Check if binary actually exists e.g. “can XYZ 1.2 be retrieved for deploy to Production”
Prevent binary re-entering a previous stage e.g. “once XYZ 1.2 has passed or failed Acceptance, the result is final”
Ensure binary has successfully passed sufficient dependencies to enter a stage e.g. “XYZ 1.2 can only enter Production once it has successfully passed UAT and Performance”
Introduce a manual sign-off process for critical environments e.g. “XYZ 1.2 can only pass UAT when exploratory testing is complete”
Visualise pipeline activity e.g. “I can see XYZ 1.2 was successfully released to Production a week ago, and that 1.4 is the next viable candidate as 1.3 failed Acceptance”

These features ensure fast feedback is always available and that the pipeline is an accurate representation of the underlying value stream. An absence of metadata unnecessarily hinders these goals and suggests a failure to understand the core values of Continuous Delivery.

Pipeline Antipattern: Deficient Deployer

14 March, 2012 / Steve Smith

A badly-defined Deployer abstraction impairs Continuous Delivery

As Continuous Delivery patterns are gradually establishing themselves, antipatterns are also surfacing – and a common antipattern is the Deficient Deployer.

When we talk about a Deployer, we are referring to a pipeline task that can be invoked by any post-Commit stage to deliver an application binary to an environment. A Deployer interface should look like:

Deployer#Deploy(Application a, Version v, Environment e)

There are a couple of ways Deficient Deployer can creep into a Deployer implementation:

Anemic implementation – this is where the Deployer behaviour is solely specified as “download binary and copy to environment”, ignoring any one-time operations deemed necessary for a binary to be in a valid state. For example, environment configuration should be filtered into the application binary during deployment, as it is a one-time operation decoupled from the application lifecycle. If this configuration step is omitted from the Deployer, then additional manual effort(s) will be required to make the binary ready for use.
Over-specified interface – this is where environment-specific parameters are added to the Deployer interface e.g. Deployer#Deploy(Application a, Version v, Environment e, Server s). The server(s) associated with an environment and their credentials should be treated as an implementation detail of the Deployer via a per-environment server mapping. Application version, and environment are the only consistent mandatory parameters across all environments.

The root cause of the Deficient Deployer seems to be a reluctance to treat deployment scripts as first class citizens.

The Value Proposition for Continuous Delivery

13 March, 2012 / Steve Smith

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

In November 2011, Dave West argued that “Water-Scrum-Fall is the reality of Agile for most organisations today“. Dave suggested that in many Agile organisations the Agile process in Development (“Scrum”) is constrained by culture and governance in both Business (“Water”) and Operations (“Fall”), and that such an organisation “fails to realise Agile’s business benefits, such as faster time-to-market, increased business value, and improved flexibility and responsiveness“. This can be summarised as:

Agile has transformed software development, but not product development

By definition, software can only generate revenue when it is in Production. However, Dave Farley and Jez Humble assert in Continuous Delivery that such common sense is not common practice in the software industry, saying that “in most organisations the release of software into Production is a manually intensive, error-prone, and risky process“. The high-cost, high-risk nature of this scenario is the root cause of the low frequency release policy so common in the industry, where stability is favoured over flexibility.

Dave F and Jez go on to argue that such a release policy is inherently expensive, stating that “every week of delay between having an idea and releasing the code that implements it can represent millions of dollars in opportunity costs“. The elapsed time between feature inception and feature delivery is the cycle time of an organisation, and by establishing a regular cadence of Production releases we can reduce cycle time and as a consequence dramatically improve throughput.

Enter Continuous Delivery, which is defined as “releasing high quality software fast through build, test and deployment automation“, and has been adopted by organisations such as Facebook, LMAX, and TheTrainLine to create reliable, repeatable, low-cost delivery mechanisms that release software to Production on a weekly or even daily basis. Such significant reductions in cycle time mean that more time can be invested back into product development and revenue growth, while the reduction in batch size provides faster feedback and reduces risk.

This is the value proposition of Continuous Delivery – firstly, “that we now have the tools, patterns and practices to drive down the transaction cost of releasing a change enormously“, and secondly that by fostering closer Development/Operations partnerships (such as DevOps) the artificial boundaries within “Water-Scrum-Fall” organisations can be dismantled.