Continuous Build with the Open Build Service

1.1   Deterministic build

The term deterministic neatly expresses this concept of reproducibility. It's like a mathematical function—if the inputs to the function are always the same, then the result will always be the same.

In order to ensure reproducible build most build systems provide a facility for describing the steps for building the software in sequence. The need to generate a sequence of build steps is fundamental both to building individual pieces of software (object files, libraries) and larger interdependent components (packages). Below, I will describe the logic behind creating a build sequence in the section When to rebuild: the directed acyclic graph.

To begin, though, let's just trust the venerable and extremely well-known tools like GNU make, which is a great example of a build tool. It calculates and executes a deterministic set of build steps for building a piece of software. This notion of deterministic build based on dependencies is absolutely fundamental to the capacity for reproducible build.

1.2   Dependencies

In order to build, every piece of software has some dependency or another. Preparing a system (computer) to build a piece of software means installing the requirements, for example, libraries and development libraries. The well-known ./configure script shipped with many software packages will try to locate and report on missing dependencies.

For many of the early years of widely distributed cross-platform software, the combination of ./configure && make served adequately. But, over time, the complexity of dependencies grew and the number of nodes grew. Some software distributors decided to ship both a source package (e.g. a tarball, .tar.gz) and some sort of binary package (e.g. .rpm or .deb).

In order to ship such a binary package, it is important to line up a predictable set of dependencies for the build.

Before we dive into continuous integration, build and delivery, let's wind the clock back to an earlier time and revisit earlier attempts at reproducible build for different target platforms (architectures) and also for software projects with possibly conflicting dependencies on the same computer system.

1.3   The gold server

In the early days of open source and infrastructures (let's say the 1990s), development groups would often have a single gold server (or one gold server per target platform) on which the software would build. Often there would be a single person, build master whose responsibility it was to maintain this system and the successful build of all software.

1.4   Problems of the gold server

Among a number of other shortcomings, the two biggest were the occasional absence of the build master and, secondarily, the constant tension between the need to try newer libraries and to maintain a predictable gold platform.

Unfortunately, the software load on the gold system sometimes differed slightly from the systems on which the developers would write and build their software. Sometimes the gold system would get an update and that would break a piece of software. Sometimes, a new library release would be required for a particular developer's software, but that would break some other build on the gold server.

So, the build master, often had some work to do to ensure clean builds when new software was being developed.

This latter problem is an inevitable part of dynamic software. In fact, it is an absolute necessity and a boon. Most developers are (by nature) interested in novel approaches, better libraries and, of course, bugfixes (while some of them may not be so excited about security maintenance). But, even more importantly, there are many reasons to encourage the use of new software (features, bugfixes, less development effort by using libnewgadget).

The build masters of the time paid the headache of knitting together stable reproducible build while trying to support newer software. Many organizations necessarily shouldered the burden of multiple gold target platforms. But, this was largely not automated. (In every case that I saw and heard of, this led to a spaghetti of shell and other half-automated tools.) While there was good reproducible build at this time, there was also significant labor cost and a bottleneck, the build master.

Oh, and of course, that pesky problem of absent key employees was easily solved by simply forbidding all employees or contributors from ever taking vacation.

The shortcomings of the gold server were quite obvious, even at the time. Since this problem touched so many organizations, the intervening two decades has seen many approaches to shrinking the gap between the time that a developer writes a piece of software and the time it can be deployed.

Other trends, for example, the move to distributed infrastructure and the related Death of the Monolith: microservices, have both provided impetus to automate the delivery of updated software.

2.1   Components supporting continuous delivery

There are logical steps to implement in order to realize any of the continuous solutions. I divide the problem set into the following class of logical steps:

The last two are important tasks for operational management of an infrastructure, and may factor less importantly in continuous delivery.

Nonetheless, I mention monitoring and reversion because they become important in a continuous deployment scenario. Without monitoring, there's no way to know that the software is doing what it should as well as it should. Without the capability for reversion, there's no way to back out of a mistake (fortunately, they never happen).

A directed acyclic graph (DAG) is used here to express of the relationships between things, let's say software and its many layers of dependencies. A DAG is often (inaccurately) called a tree (as I did above). See the discussion below in OBS solves autorebuild need.

3.1   Addressing the base-build question

The venerable and extremely well-known autotools has addressed the per-platform, multiple architecture (and cross compilation) build questions (base-build) for C and C++ for the last 25 years. A recent cross platform C and C++ build managament tool called CMake has gained a significant following.

Of course, there are others (e.g. SCons), but the most widely used are autotools and CMake. These are both geared a bit more toward building C and C++ software, both can be (and have been) used by projects in Java, shell, Python and other languages.

In some cases, a programming language will provide its own cross-platform build tooling. For example, cross platform support is an integral part of the Go Programming Language. In the Java world, the target architecture and platform is the Java Virtual Machine, so the base-build problem is effectively minimal for these two languages. Perl and Python each provide their own build and packaging system, as well.

In short, though, almost all languages provide good tooling to address the base-build problem. (Otherwise, how would people share software written in that language?)

So, most continuous integration systems (see below) assume that each language would provide its cross-platform and/or build tooling.

Always, the base-build must be a reproducible build.

3.2   All about testing

The commercial software industry and large parts of the open-source software world have inexorably adopted conventions around automated testing. Usually the software source code includes or is shipped with a testing suite. Now that the software can be built on a different architecture or platform, the built result can then be tested.

Testing tools can be written (and often are written) in the same language as the software under test (SUT), but that is not always necessary. There are a plethora of options for writing, running and summarizing test suites. Here are a few examples: JUnit in Java, Python's py.test or unittest. Often the developer or team is expected to turn over a testing suite with a set minimum amount of code coverage (say 70% of the production software must be exercised by the testing code).

The adoption of some sort of automated testing is quite important to the next step, where we can see how continuous integration fits in.

3.3   Continuous Integration and platforms

The main point behind continuous integration is first to attempt a build of the software (base-build) like a user would on all of the supported platforms and architectures.

Assuming successful build, the next step is running the testing, and this is one of the most important outputs of any continuous integration system. A developer can configure the preferred CI tool to watch for newly available software, and immediately try to build it and run the test suite.

To watch for newly available software means that the CI tool will either look for a software package (a zipfile, a tarball, an RPM) or, more commonly, for evidence that a developer has updated the source code repository (this is automated checkout from a version control system). When the CI tool sees the new code (or tarball), it kicks into gear, building the software and running the tests.

The outputs from a CI system are usually two-fold, first, a report of success or failure and, second, some sort of build artifact (this is the most common term used across the different CI systems).

Without automated testing, a CI tool can prove only that the software builds properly, but cannot offer any assurance that the software operates correctly. This is why some form of automated testing is usually a prerequisite for continuous integration.

For the last decade or so, the common developer pattern is to identify a bug, write a test to trigger the bug (notice that it fails), fix the bug and then see that the test succeeds. This usage of testing dovetails very nicely with a CI system, because the CI system will now run the full build and full testing suite and the developer and users can be pleased that there was no feature regression.

Digression: In conjunction with the widespread adoption of distributed version control systems (DVCS), it has become socially questionable, in the developer community, to commit software that does not build: "Don't break the build!". To commit broken code (to a main branch, master or trunk) is considered bad form, especially since each developer is able to (and, now expected to) test changes locally before committing to the shared development tree.

Automation, in some organizations, stops with CI. For these, there is no process after the reproducible build. For such an outfit, it would be customary to publish or ship the software or ask the operations/deployment team to deploy the software to production. (If the number of packages or applications is small, this can be perfectly manageable for an operations team.)

In my estimation, however, this leaves a gap between the reproducible development test/build cycle and the installable software image.

Side note: Some CI systems allow the generation of an artifact. It is, therefore, possible to produce a binary deliverable out of the CI system in addition to the base build success or failure on the platforms. In many organizations, continuous build does not occur.

In other organizations, the function is performed in a different system, probably a build system. So, now, let's look at a continuous build system.

4.1   What is the Open Build Service (OBS)

The OBS is a tool for creating independent or interdependent software distributions. Initially conceived in the early 2000s, OBS takes advantage of virtualized machines to perform reproducible builds in pristine environments. The resulting build output is a package (e.g. RPM and Debian), which is published in a visible software repository from which the package can then be installed for the next pristine build.

This allows a developer or builder to do the following (see below for terminology):

• define a distribution: repository (e.g. CentOS-5.10, x86_64)
• create a grouping of similar packages: project (e.g. production)
• configure dependency inheritance build rules for a project (get all software dependencies from CentOS-5.10 upstream distribution + updates)
• create a package in a project (e.g. frobnitz-1.2)

Once the above steps are complete, OBS will build the package to produce consistent installable binary outputs every time the package (or any dependency) changes.

4.2   OBS terminology

As with every system, there are terms to understand the internal system componentry. While there's a great deal of richness available, the simplest explanation of the relationship of OBS inputs and outputs requires understanding the following three concepts and terms.

(A package can also include Debian-specific build instructions.)

Terminology nit: Generally speaking, people refer to the term package to mean both a source package and a binary package. It is such a common term that I shall point out that in this document, I mean package to describe the source package. I will use the term binary to mean a binary package.

OBS will rebuild a piece of software (a package) if there's a change to:

• the package source code
• the package specfile (which are the build instructions)
• any dependency of package (recursively)
• or any change to the project configuration

I call the result of this system autorebuild with binary delivery.

4.3   OBS solves autorebuild need

The inputs to the OBS are the software source and an RPM specfile (and/or Debian build instructions). The output(s) are binary software repositories with installable images (see below).

The instructions for building one source package can generate multiple binary RPM-installable packages, but in the simplest case a single source package results is a single binary package in each repository.

All resulting binary packages are available in the output repository and have been consistently built (or rebuilt) according to the rules established by the OBS user/administrator.

4.4   Delivering binary packages with continuous build

Using OBS like this, allows the user to generate complete software distributions in an RPM-MD or Debian repository that behaves just like any upstream platform. Here's an example:

• You create a project with 50 distinct software packages.
• You configure repository targets for 3 different build platforms and architectures (CentOS-6.7, x86_64; Scientific-Linux-7.1, x86_64; Scientific-Linux-7.2, x86_64).
• OBS will run all permutations of the build to satisfy generating all of the binary output RPMs for each of the configured target platforms.

Thus, when the work is done, you can simply execute::

zypper install mypackage

or::

apt-get install mypackage

The use of continuous build now enables the possibility of continuous delivery.