Why Continuous Delivery?

There are many ways to deploy software but at Hornbill we deploy continuously to remove the need for our customers to ever have to perform an upgrade. Doing a software upgrade often means a project that includes planning, evaluating, testing, training, and roll-out. This approach is really a relic of the 80s that was born out of the fact that software is complicated and users of the software were often expected to have some degree of technical expertise. In today’s world, mass adoption of complex software systems has become commonplace where users no longer need the technical expertise they once did. The idea then is software is delivered As-A-Service, and just like other services you take (for example your home cable TV service) you simply consume the service. From time to time your service provider will update your service, update your set-top box with new firmware, add or remove channels, etc.

As the consumer of the service you don’t care what version you are on, but you are happy to avail of the new channels so long as your service continues to work. Another example might be Facebook, if you use Facebook (or Gmail or LinkedIn or Twitter etc) and I ask you the question “What version are you on?” I would expect your answer to be “I have no idea” simply because you don’t need to know, or care, you just expect the service to work when you log in. That’s how we think about Hornbill, and we believe all software should be deployed and consumed like this so that’s the foundation upon which we built our platform.

The Quality of Our Service is our Reputation

Just because we do continuous delivery does not mean we don’t have a formal process, in fact, the complete opposite is true. Our delivery process is embodied in our fabric and our very culture and runs right through the core of everything we do. There are a number of key characteristics to the way we work that makes continuous delivery possible.

• Hornbill is a loosely coupled, layered, and highly distributed architecture.
• Hornbill is made up of a large number of independent micro-services.
• Hornbill has 100% automated functional test coverage for functions used in production for all micro-services.
• Hornbill includes fast regression tests for elements (such as user experience) where it is not possible (or cost effective) to automate.
• Hornbill has a fully automated deployment pipeline, all packaging and deployment is done without human intervention.
• Hornbill is organized into small development teams that assume responsibility for feature and fix deployment.

Hornbill Update Deployment Process

When we write code for a fix or a minor feature/enhancement we follow a strict methodology to facilitate both agility and continuous deployment. Each time we do a build, which is fully automated, we have four stages of release before the code makes it into production. These are known as test, dev, beta, and live and every push of code has to transition through these four stages. Code can be blocked from progressing at any stage and for any reason.

• test
  When we initiate a build the code is compiled and the releasable content is created and published into our deployment system on the test stream. All of our automated test services are sitting on this stream, so when a new build is available our test instances will upgrade themselves and initiate the automated testing. As the tests are run our central build controller is notified of the test results so we have a dashboard that allows our development teams to see the test progress. If any test fails, the stream is blocked and the build is no longer deployable. Once all automated and regression tests have passed the release is marked as successful and the build is published to the dev stream.
• dev
  All development instances used by Hornbill’s development teams sit on this stream. Developers use many instances every day in the process of developing both our platform and our applications. Once a new build is available in dev all developer instances are automatically updated. This not only enables developers to validate recent code changes in a production environment but it’s our first line of defense to pick up anything unexpected. This is also where we run any manual regression testing that needs to be completed. The project build master will make a judgment about when a given build on dev is fit for promotion to the beta stream. A build will typically sit for 24 hours on dev before being pushed forward. While on this stream we monitor performance, API, and system logs to make sure there are no unexpected error conditions. If there are any problems then the build master for the project in question will reject the build which will prevent it moving forward.
• beta
  Our beta steam is used by all pre-production instances. Hornbill’s own production instances run on this stream, as do our beta test customers, some sandboxes, sales demo instances, and partner instances. At this point, we do not expect to see any issues but if anything does show up the build is blocked from progressing any further. While on this stream we monitor performance, API, and system logs to make sure there are no unexpected error conditions. Once the build has been on beta for 48 hours without issue the build master for that project will promote the build to live.
• live
  Our live stream is production. Once a build is pushed to live, all production instances will update themselves to that build. Hornbill is an always up-to-date solution using continuous development and updates with 0 downtime occurring throughout the day. However, the platform and applications that may require up to 2 minutes of downtime, occur at 0500 UTC for Platform and 0530 UTC for applications Mon-Fri.

Hornbill Major “New” Feature Update Process

If we add something major that is new, for example, introduce a new application, a new report,or a new view that did not previously exist then we simply follow the above Hornbill Update Process and leave the In-App training to guide our users to the new functionality.

Hornbill Major “Change” Update Process

Our major change update process is different from our continuous update process set out above. We are always conscious of our customer’s workflows and general usage expectations and do not just change the user experience or the way our system works. However, we do have to evolve and take our customers with us and we love adding great new features or sometimes significant step changes. In order to introduce such changes/features we provide Feature Option Switches that will allow users to switch between the new and existing behavior/experience for a period of time known as the preview period, typically this is between 30 and 90 days depending on the feature and the feedback received during the feature preview. A typical workflow for this type of release would be as follows:

• deploy
  In order to get the preview to our customers we follow the Hornbill Update Process set out above. However, instead of enabling the feature, we would typically provide an in-app notice indicating the presence of the new feature along with a feature switch right within the affected view(s) that are being changed. For example, at the time of writing this page, we just introduced an all-new email interface which was substantially different (and improved) but as a result, the user workflow was also substantially different. So we provided a “Try the new Xyz Feature” button which when pressed will switch the user’s session to use the new view, they would also have a button to switch back should they wish. Individual users can switch between the old and new ways of working as they see fit. When the new feature is enabled, we will generally also provide an in-app link to documentation and/or a conversation to solicit feedback and allow users to ask us questions about the new feature.
• preview
  We will leave the feature in preview for a period of time and react to any feedback by enhancing the new major feature using the Hornbill Update Process described above. notice - once we are satisfied that the feature is fit for purpose and we have addressed any concerns from our user community we will issue notice to change, this is typically 30 days and will be shown in the application in the form of a countdown giving our users time to make the switch and adjust their workflow.
• finalize
  On the planned date the option switch will be removed and all users will be switched over to the new feature.
• stand-off
  We leave the now depreciated feature in the codebase for a stand-off period, typically 30 days, and in an emergency, Hornbill can revert to the old feature during this time. However, we would only do this in exceptional circumstances on a case-by-case basis and only to provide a short-term workaround should a serious problem be found that’s impacting a specific customer.

Patch Management

Application patches can be deployed seamlessly without the need for end-user involvement or downtime and will be applied when needed.

Platform Patches will either be Critical (Affect the number of instances or security) or Emergency (Cannot wait until the next scheduled release). Critical Patches will be applied when needed and may cause up to 1 minute of instance unavailability, Emergency Patches will be applied within the next update window (5 AM UTC).

Infrastructure Changes

Hornbill is provisioned on fully redundant hardware and most changes can be made by our teams without any downtime. All infrastructure changes are conducted in the same cycle as software (Dev->Beta->Live) with the same time delays. Any Scheduled Maintenance that is expected to take longer than 2 minutes will be announced at least 7 days in advance (unless critical) via email to the nominated contacts. Scheduled maintenance of this type is usually performed before 0600 UTC Sat-Sun.