Continuous Build with the Open Build Service

Reproducible build has long been a known (and mostly solved) problem. Many automation steps can exist between authorship of a new feature or a bug fix and the delivery of that update to an infrastructure. Automation is important to allow shrinking this temporal gap between developer and infrastructure. Here, I highlight one approach to continuous delivery and the supporting components. I identify the Open Build Service (OBS), as one of the good solutions for continuous build.

1   Reproducible Build (the long-solved problem)

The term reproducible build describes the repeatable creation of an identical output piece of software, if the input source code, compiler, processor and environment are the same. If all of the circumstances surrounding the build (source, libraries, processor, etc.) are the same and the resulting software is different, that can pose a problem.

1.1   Deterministic build

The term deterministic neatly expresses this concept of reproducibility. It's like a mathematical function—if the inputs to the function are always the same, then the result will always be the same.

In order to ensure reproducible build most build systems provide a facility for describing the steps for building the software in sequence. The need to generate a sequence of build steps is fundamental both to building individual pieces of software (object files, libraries) and larger interdependent components (packages). Below, I will describe the logic behind creating a build sequence in the section When to rebuild: the directed acyclic graph.

To begin, though, let's just trust the venerable and extremely well-known tools like GNU make, which is a great example of a build tool. It calculates and executes a deterministic set of build steps for building a piece of software. This notion of deterministic build based on dependencies is absolutely fundamental to the capacity for reproducible build.

1.2   Dependencies

In order to build, every piece of software has some dependency or another. Preparing a system (computer) to build a piece of software means installing the requirements, for example, libraries and development libraries. The well-known ./configure script shipped with many software packages will try to locate and report on missing dependencies.

For many of the early years of widely distributed cross-platform software, the combination of ./configure && make served adequately. But, over time, the complexity of dependencies grew and the number of nodes grew. Some software distributors decided to ship both a source package (e.g. a tarball, .tar.gz) and some sort of binary package (e.g. .rpm or .deb).

In order to ship such a binary package, it is important to line up a predictable set of dependencies for the build.

Before we dive into continuous integration, build and delivery, let's wind the clock back to an earlier time and revisit earlier attempts at reproducible build for different target platforms (architectures) and also for software projects with possibly conflicting dependencies on the same computer system.

1.3   The gold server

In the early days of open source and infrastructures (let's say the 1990s), development groups would often have a single gold server (or one gold server per target platform) on which the software would build. Often there would be a single person, build master whose responsibility it was to maintain this system and the successful build of all software.

1.4   Problems of the gold server

Among a number of other shortcomings, the two biggest were the occasional absence of the build master and, secondarily, the constant tension between the need to try newer libraries and to maintain a predictable gold platform.

Unfortunately, the software load on the gold system sometimes differed slightly from the systems on which the developers would write and build their software. Sometimes the gold system would get an update and that would break a piece of software. Sometimes, a new library release would be required for a particular developer's software, but that would break some other build on the gold server.

So, the build master, often had some work to do to ensure clean builds when new software was being developed.

This latter problem is an inevitable part of dynamic software. In fact, it is an absolute necessity and a boon. Most developers are (by nature) interested in novel approaches, better libraries and, of course, bugfixes (while some of them may not be so excited about security maintenance). But, even more importantly, there are many reasons to encourage the use of new software (features, bugfixes, less development effort by using libnewgadget).

The build masters of the time paid the headache of knitting together stable reproducible build while trying to support newer software. Many organizations necessarily shouldered the burden of multiple gold target platforms. But, this was largely not automated. (In every case that I saw and heard of, this led to a spaghetti of shell and other half-automated tools.) While there was good reproducible build at this time, there was also significant labor cost and a bottleneck, the build master.

Oh, and of course, that pesky problem of absent key employees was easily solved by simply forbidding all employees or contributors from ever taking vacation.

The shortcomings of the gold server were quite obvious, even at the time. Since this problem touched so many organizations, the intervening two decades has seen many approaches to shrinking the gap between the time that a developer writes a piece of software and the time it can be deployed.

Other trends, for example, the move to distributed infrastructure and the related Death of the Monolith: microservices, have both provided impetus to automate the delivery of updated software.

2   What's all this about continuity?

The proliferation of continuous processes highlights the industry-wide effort to move authored software smoothly and predictably into production. This often involves many steps. Each piece of software must build and should pass its own tests. In many cases, it must integrate with (or be used by) another piece of software. The resulting software product be installable on the end target system. Then, it has to be deployed to the end system.

I will focus on delivery to software infrastructures for the remainder of this article.

Given the need to support regular and irregular (arbitrarily-frequent) software updates to software services, the industry has, since the fin de siècle, come up with many tools to address some of these problems. These are the various processes, each labelled with the word continuous.

  • continuous integration: rebuild software and run any automated tests, reporting success or failure; sometimes can produce build artifact
  • continuous build: rebuild software for configured target platforms; might run automated testing but usually not by default
  • continuous delivery: provide a readily available set of built software representing a working application for installation at any time for all target platforms (e.g. Ubuntu-14.04, CentOS-7.0) and environments (e.g. production, staging, testing)
  • continuous deployment: automated updates of infrastructure or software (often depending on a continuous delivery system)

One of the key elements to all of these continuous processes is automation. In short, the adjective continuous implies automation. Any of these processes can be completed by a human, but then, it is not continuous.

Since the machines are not yet writing the software nor handling the deployment process, there are always breakpoints, where a person can look at the output from the Jenkins continuous integration system or the OBS continuous build system and make a choice about whether to trigger the next step.

2.1   Components supporting continuous delivery

There are logical steps to implement in order to realize any of the continuous solutions. I divide the problem set into the following class of logical steps:

base-build:deterministic build (a long-solved problem)
testing:automated testing of the built software
platforms:(re-)build and test on each target platform
autorebuild:(re-)build the software dependency tree (DAG; see below)
binary:want to produce runnable software package or image
microservices:small, self-contained and (horizontally scalable) services
reversion:must be able to roll back to earlier software
monitoring:tracking of availability, performance and behaviour over time

The last two are important tasks for operational management of an infrastructure, and may factor less importantly in continuous delivery.

Nonetheless, I mention monitoring and reversion because they become important in a continuous deployment scenario. Without monitoring, there's no way to know that the software is doing what it should as well as it should. Without the capability for reversion, there's no way to back out of a mistake (fortunately, they never happen).

A directed acyclic graph (DAG) is used here to express of the relationships between things, let's say software and its many layers of dependencies. A DAG is often (inaccurately) called a tree (as I did above). See the discussion below in OBS solves autorebuild need.

3   Continuous Integration

Continuous integration (CI) swept broadly across the software industry in the 2000s and there were a large number of tools available for development groups to install and use. Today, like version control systems and chat systems, many continuous integration tools are available for operational integration directly online, some for free.

This was the first of the continuous processes that was widely adopted.

Projects like Jenkins, Hudson, Buildbot Travis and Semaphore CI all provide an answer to the continuous integration question, usually on a specified set of target platforms.

Let's review the components that these CI systems depend on before they can provide their value.

3.1   Addressing the base-build question

The venerable and extremely well-known autotools has addressed the per-platform, multiple architecture (and cross compilation) build questions (base-build) for C and C++ for the last 25 years. A recent cross platform C and C++ build managament tool called CMake has gained a significant following.

Of course, there are others (e.g. SCons), but the most widely used are autotools and CMake. These are both geared a bit more toward building C and C++ software, both can be (and have been) used by projects in Java, shell, Python and other languages.

In some cases, a programming language will provide its own cross-platform build tooling. For example, cross platform support is an integral part of the Go Programming Language. In the Java world, the target architecture and platform is the Java Virtual Machine, so the base-build problem is effectively minimal for these two languages. Perl and Python each provide their own build and packaging system, as well.

In short, though, almost all languages provide good tooling to address the base-build problem. (Otherwise, how would people share software written in that language?)

So, most continuous integration systems (see below) assume that each language would provide its cross-platform and/or build tooling.

Always, the base-build must be a reproducible build.

3.2   All about testing

The commercial software industry and large parts of the open-source software world have inexorably adopted conventions around automated testing. Usually the software source code includes or is shipped with a testing suite. Now that the software can be built on a different architecture or platform, the built result can then be tested.

Testing tools can be written (and often are written) in the same language as the software under test (SUT), but that is not always necessary. There are a plethora of options for writing, running and summarizing test suites. Here are a few examples: JUnit in Java, Python's py.test or unittest. Often the developer or team is expected to turn over a testing suite with a set minimum amount of code coverage (say 70% of the production software must be exercised by the testing code).

The adoption of some sort of automated testing is quite important to the next step, where we can see how continuous integration fits in.

3.3   Continuous Integration and platforms

The main point behind continuous integration is first to attempt a build of the software (base-build) like a user would on all of the supported platforms and architectures.

Assuming successful build, the next step is running the testing, and this is one of the most important outputs of any continuous integration system. A developer can configure the preferred CI tool to watch for newly available software, and immediately try to build it and run the test suite.

To watch for newly available software means that the CI tool will either look for a software package (a zipfile, a tarball, an RPM) or, more commonly, for evidence that a developer has updated the source code repository (this is automated checkout from a version control system). When the CI tool sees the new code (or tarball), it kicks into gear, building the software and running the tests.

The outputs from a CI system are usually two-fold, first, a report of success or failure and, second, some sort of build artifact (this is the most common term used across the different CI systems).

Without automated testing, a CI tool can prove only that the software builds properly, but cannot offer any assurance that the software operates correctly. This is why some form of automated testing is usually a prerequisite for continuous integration.

For the last decade or so, the common developer pattern is to identify a bug, write a test to trigger the bug (notice that it fails), fix the bug and then see that the test succeeds. This usage of testing dovetails very nicely with a CI system, because the CI system will now run the full build and full testing suite and the developer and users can be pleased that there was no feature regression.

Digression: In conjunction with the widespread adoption of distributed version control systems (DVCS), it has become socially questionable, in the developer community, to commit software that does not build: "Don't break the build!". To commit broken code (to a main branch, master or trunk) is considered bad form, especially since each developer is able to (and, now expected to) test changes locally before committing to the shared development tree.

Automation, in some organizations, stops with CI. For these, there is no process after the reproducible build. For such an outfit, it would be customary to publish or ship the software or ask the operations/deployment team to deploy the software to production. (If the number of packages or applications is small, this can be perfectly manageable for an operations team.)

In my estimation, however, this leaves a gap between the reproducible development test/build cycle and the installable software image.

Side note: Some CI systems allow the generation of an artifact. It is, therefore, possible to produce a binary deliverable out of the CI system in addition to the base build success or failure on the platforms. In many organizations, continuous build does not occur.

In other organizations, the function is performed in a different system, probably a build system. So, now, let's look at a continuous build system.

4   Continuous Build, featuring OBS

A continuous build service takes software from any number of sources and, for each target platform attempts to build a complete set of binary-installable outputs.

While there are a number of such tools (Fedora's Koji or the Debian project's Autobuilder), my favorite is the Open Build Service (OBS).

4.1   What is the Open Build Service (OBS)

The OBS is a tool for creating independent or interdependent software distributions. Initially conceived in the early 2000s, OBS takes advantage of virtualized machines to perform reproducible builds in pristine environments. The resulting build output is a package (e.g. RPM and Debian), which is published in a visible software repository from which the package can then be installed for the next pristine build.

This allows a developer or builder to do the following (see below for terminology):

  • define a distribution: repository (e.g. CentOS-5.10, x86_64)
  • create a grouping of similar packages: project (e.g. production)
  • configure dependency inheritance build rules for a project (get all software dependencies from CentOS-5.10 upstream distribution + updates)
  • create a package in a project (e.g. frobnitz-1.2)

Once the above steps are complete, OBS will build the package to produce consistent installable binary outputs every time the package (or any dependency) changes.

4.2   OBS terminology

As with every system, there are terms to understand the internal system componentry. While there's a great deal of richness available, the simplest explanation of the relationship of OBS inputs and outputs requires understanding the following three concepts and terms.

package:a source tarball and spec file; optionally a VCS URL to watch and/or multiple source files; a package can (and usually does) depend on other packages
project:a container grouping a set of packages intended to be delivered together; a project can also depend on other projects
repository:a collection of binary deliverable software, built for a specific target platform and architecture (e.g. CentOS-7.1, x86_64)
binary:a binary installable package, like an RPM or a deb file, a Docker image, a VMDK or a warfile, any result of a build that is intended to be run in production

(A package can also include Debian-specific build instructions.)

Terminology nit: Generally speaking, people refer to the term package to mean both a source package and a binary package. It is such a common term that I shall point out that in this document, I mean package to describe the source package. I will use the term binary to mean a binary package.

OBS will rebuild a piece of software (a package) if there's a change to:

  • the package source code
  • the package specfile (which are the build instructions)
  • any dependency of package (recursively)
  • or any change to the project configuration

I call the result of this system autorebuild with binary delivery.

4.3   OBS solves autorebuild need

The inputs to the OBS are the software source and an RPM specfile (and/or Debian build instructions). The output(s) are binary software repositories with installable images (see below).

The instructions for building one source package can generate multiple binary RPM-installable packages, but in the simplest case a single source package results is a single binary package in each repository.

All resulting binary packages are available in the output repository and have been consistently built (or rebuilt) according to the rules established by the OBS user/administrator.

4.3.1   When to rebuild: the directed acyclic graph

OBS uses concepts familiar from make for dependency tracking and software recency to determine build order. Though it is generally understood as a tree, the concept is the directed acyclic graph (DAG).

In a Makefile, any produced output file has an unambiguous set of dependencies that are encoded in the rules. The make program performs an operation called a topological sort to identify the order in which to build any intermediate files and final (desired) outputs.

For example, if a frobnitz.c source file has been updated, then the dependent frobnitz.o object file must be rebuilt. If the shared library depends on frobnitz.o, then it, too, must be rebuilt. This dependency-tracking is integral to make when generating individual pieces of software (command-line utilities, shared libraries, applications).

Analagously, within OBS, every package has a clear set of dependencies that are listed in the specfile. The OBS system performs the same topological sort to identify the order in which intermediate packages and final (desired) packages should be created.

A developer writing a Makefile specifies the dependencies on source and intermediate files. In OBS, the developer specifies the dependencies on other software packages.

4.3.2   OBS target platforms

A developer using OBS can also configure the desired platform and depend on tools provided from that distribution (e.g. OpenSUSE, CentOS, Ubuntu). Thus, an organization's software stack can be built at any time against any chosen distribution. And, of course, any distribution-supported architecture (x86_64, i386, arm) is available as well.

4.3.3   Example

For example, let's suppose that a Python library (python-frobnitz) has C bindings to the OpenCV (Computer Vision) library, and the upstream (distributor-supplied) package for libopencv has been updated, then it would be wise to rebuild the Python library. OBS calculates this dependency, realizing that libopencv must be built first, and then python-frobnitz. If the target platforms (repositories) for python-frobnitz are Ubuntu-14.04 (x86_64 and i386) and OpenSUSE-LEAP (x86_64 and arm), then OBS will build libopencv and python-frobnitz four times.

4.3.4   Predictability, at a cost

It may be obvious that the result of a single update of a core library (for example glibc) could be a deep rebuild chain, but if the rebuilds are successful, then each piece of software is guaranteed to have been rebuilt against the current version of its dependency.

This can use a good deal of compute resource—and is sometimes regarded as unnecessary, especially because many library authors go to great lengths to ensure library compatibility between minor library releases (and almost always, micro-releases).

Even so, the autorebuild approach leaves no doubt about whether the rebuild was needed or not. (And, for those who are deeply concerned about possible CPU-wastefulness, let me assure you that OBS includes several mechanisms to control sensible suppression of rebuild chains.)

4.4   Delivering binary packages with continuous build

Using OBS like this, allows the user to generate complete software distributions in an RPM-MD or Debian repository that behaves just like any upstream platform. Here's an example:

  • You create a project with 50 distinct software packages.
  • You configure repository targets for 3 different build platforms and architectures (CentOS-6.7, x86_64; Scientific-Linux-7.1, x86_64; Scientific-Linux-7.2, x86_64).
  • OBS will run all permutations of the build to satisfy generating all of the binary output RPMs for each of the configured target platforms.

Thus, when the work is done, you can simply execute::

zypper install mypackage


apt-get install mypackage

The use of continuous build now enables the possibility of continuous delivery.

5   Conclusion

In a future posting, I hope to include some more detail about using OBS and also show how to use OBS to apply the concept of continuous delivery.