Keeping a project build in a manageable time range is an important prerequisite for an efficient software development workflow. The more complex projects are the more time, automation effort and tools required to develop and maintain an efficient build pipeline. Regardless of a scale and type of a project the core principle of avoiding unnecessary work is worth applying.

This tutorial presents an approach to simplify a build process by creating a package of a third-party software component upfront and hosting it in an external package repository. It is an alternative to building a component from sources on a development or continuous integration machine.

In the example presented below the Google Operation Research Tools library is compiled on a clean virtual machine and then packaged by the DEB CPack Package Generator. The process is automated using Vagrant. Then, a local Debian package repository is setup to host the package created in the previous step. Finally, the package repository is made public by uploading it to GitHub and hosting using GitHub Pages.

Although the tutorial is built around a specific example, the overall approach and selected tools are generic. In particular, CPack offers a unified interface for creating packages that supports the most popular formats including ZIP, DEB and RPM.

Create a DEB package

There is not a single established way to create DEB packages. This section demonstrate DEB CPack Package Generator which should be suitable for building packages for internal use, especially if you are already familiar with CMake or it is used as the project build tool. If you aspire to applying for hosting the package in the official Debian/Ubuntu repositories refer to the Debian New Maintainers’ Guide. That document explains a complex process of packaging that allows for applying custom patches and more flexibility.

CPack is the package generator used by CMake. Apart from DEB and RPM popular in Linux community, CPack supports also cross platform formats such as ZIP. CPack is integrated with CMake build process in the similar vein as packaging plugins with Maven. Package generation is executed after compilation and tests. By default one package is created for a build project.

Package configuration is typically described in a single text file that contains a list of GLOB patterns capturing files and directories to be distributed with the package. Furthermore, a specific package output format may impose extra requirements on information that should be included. For example, the DEB package format requires metadata such as package version, processor architecture and maintainer.

The listing below contains a CPack configuration for the Google Operational Research Tools library.

cmake_minimum_required(VERSION 3.2 FATAL_ERROR)
project(or_tools_package LANGUAGES CXX VERSION 6.0)

include(./Utils.cmake)

m_get_filepath(OR_TOOLS_DIR ~/dev/or-tools REALPATH)
m_get_filepath(OR_TOOLS_DEP_INSTALL_DIR ${OR_TOOLS_DIR}/dependencies/install REALPATH)

add_custom_target(dist COMMAND ${CMAKE_MAKE_PROGRAM} package)

# Dependencies

# Protobuf = 3.3.0
m_check_filepath(${OR_TOOLS_DEP_INSTALL_DIR}/include/google/protobuf/api.pb.h)
m_check_filepath(${OR_TOOLS_DEP_INSTALL_DIR}/lib/libprotobuf.a)
m_check_filepath(${OR_TOOLS_DEP_INSTALL_DIR}/bin/protoc)

install(DIRECTORY ${OR_TOOLS_DEP_INSTALL_DIR}/include/google/ DESTINATION include/google)

install(DIRECTORY ${OR_TOOLS_DEP_INSTALL_DIR}/lib/ DESTINATION lib
  FILES_MATCHING PATTERN "libproto*"
                 PATTERN "cmake" EXCLUDE
                 PATTERN "pkgconfig" EXCLUDE)

install(FILES ${OR_TOOLS_DEP_INSTALL_DIR}/bin/protoc DESTINATION bin)

# gflags = 2.2.0
# Glog = 0.3.5
# cbc = 2.9.8
# CoinUtils = 2.10.13
# Osi = 0.107.8
# Clp = 1.16.10
# Cgl = 0.59.9
# ...

# OR Tools 6.0
m_check_filepath(${OR_TOOLS_DIR}/lib/libcvrptw_lib.so )
m_check_filepath(${OR_TOOLS_DIR}/lib/libdimacs.so)
m_check_filepath(${OR_TOOLS_DIR}/lib/libfap.so)
m_check_filepath(${OR_TOOLS_DIR}/lib/libortools.so)
m_check_filepath(${OR_TOOLS_DIR}/ortools/linear_solver/linear_solver.h)
m_check_filepath(${OR_TOOLS_DIR}/ortools/gen/ortools/linear_solver/linear_solver.pb.h)

install(DIRECTORY ${OR_TOOLS_DIR}/lib/ DESTINATION lib
  FILES_MATCHING PATTERN "lib*")

foreach (dir algorithms base bop constraint_solver flatzinc glop graph linear_solver lp_data sat util)
  install(DIRECTORY ${OR_TOOLS_DIR}/ortools/${dir} DESTINATION include/ortools FILES_MATCHING PATTERN "*.h")
  install(DIRECTORY ${OR_TOOLS_DIR}/ortools/gen/ortools/${dir} DESTINATION include/ortools FILES_MATCHING PATTERN "*.h")
endforeach ()

if(EXISTS "${CMAKE_ROOT}/Modules/CPack.cmake")
  include(InstallRequiredSystemLibraries)

  set(CPACK_SET_DESTDIR "on")
  set(CPACK_PACKAGING_INSTALL_PREFIX "/tmp")

  set(CPACK_PACKAGE_NAME "or-tools")
  set(CPACK_PACKAGE_VENDOR "google")
  set(CPACK_PACKAGE_CONTACT "mateusz.polnik at strath.ac.uk")
  set(CPACK_PACKAGE_DESCRIPTION "Google Optimization Tools")
  set(CPACK_PACKAGE_DESCRIPTION_SUMMARY ${CPACK_PACKAGE_DESCRIPTION})

  set(CPACK_RESOURCE_FILE_LICENSE ${OR_TOOLS_DIR}/LICENSE-2.0.txt)
  set(CPACK_RESOURCE_FILE_README ${OR_TOOLS_DIR}/README)
  set(CPACK_PACKAGE_VERSION ${PROJECT_VERSION})

  if(UNIX)
    list(APPEND CPACK_GENERATOR "DEB")

    m_get_architecture(_OS_ARCH)
    mark_as_advanced(_OS_ARCH)

    set(CPACK_DEBIAN_REVISON "ubuntu-trusty-a1")
    set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}-${CPACK_DEBIAN_REVISON}_${_OS_ARCH}")
    set(CPACK_DEBIAN_PACKAGE_PRIORITY "optional")
    set(CPACK_DEBIAN_COMPRESSION_TYPE "gzip")
    set(CPACK_DEBIAN_PACKAGE_DEPENDS "git (>= 1:2.1.4), bison (>= 2:3.0.2), flex (>= 2.5.39), python-setuptools (>= 3.3.8), python-dev (>= 2.7.9), autoconf (>= 2.69), libtool (>= 2.4.2), zlib1g-dev (>= 1:1.2.8), texinfo (>= 5.2.0), help2man (>= 1.46.4), gawk (>= 1:4.1.1), g++ (>= 4:4.9), curl (>= 7.38.0), texlive, cmake (>= 3.0.2), subversion (>= 1.8.10)")
  endif()

  include(CPack)
endif()

Third-party dependencies required by the library except Protocol Buffers were skipped for brevity. See GitHub for the full content of the file. Important elements of the CPack configuration are explained below.

Declare the CMake header. It should mention the minimum supported CMake version, because some commands or their options may not be available in older versions of CMake.
Declare the project name and the version. The name and the version of the project will be reused to set the package name and version respectively.
Import custom utility functions. The script uses two non-standard functions that operate on file paths. m_check_filepath asserts that a file exists in the specified location. Otherwise a fatal error is raised. m_get_filepath is an extension of m_check_filepath that assigns the absolute filepath to a local variable. Implementation of both helper functions is available in the Utils.cmake file. The default CMake command to operate on file paths, [get_filename_component](https://cmake.org/cmake/help/v3.9/command/get_filename_component.html, cannot be configured to raise a fatal error. I found the strict variant more useful to prevent creating a package of a project that did not compile correctly or was restructured since previous release.
Declare a custom target dist to build a package.
Specify files that should be included in the package using the install command.
Set the package output format and metadata. CPack is available as a CMake module. It is considered a good practice to ensure that module is available and load it on demand. Available metadata depends on the package format. Some properties may be required to set to build a package. For more information on other output formats and their configuration see the CPack documentation.

Make sure that the include(CPack) statement is the final command of the CPack configuration block. Otherwise, the module may not work correctly.

Once the CPack configuration is ready run the cmake program to generate make files for your platform. This step should succeed if there are no syntax errors in the CPack configuration and all files required to build the package are available in the file system. Then run make dist to build the package using the make files generated in the previous step. It is common to output make files to a separate directory, so they can be easily removed if needed.

The sequence of commands below attempts to remove previous make files, generates new make files and builds a package if previous steps were successful.

rm -Rf build \
&& mkdir -p build \
&& cd build \
&& cmake .. \
&& make dist

It is useful to package software on a clean operating system image. Following this practice is important for several reasons. Firstly, all administrator operations required to build a package are either documented or automated. Secondly, the build is always executed in the same environment, so it is reproducible. Furthermore, package content is not affected by user activity, temporary changes or software updates. Finally, bugs are detected as soon as possible, because they are not masked by undocumented temporary modifications.

Fortunately, using a clean operating system image does not imply that the process has to be manual. In fact, packaging can be automated end-to-end due to the advent of provisioning tools. In this article building a package is implemented as a Vagrant script. Vagrant is an automation tool that provides a unified interface for accessing popular provisioning tools and virtual machine providers. If you are not familiar with Vagrant or would like to keep things simple, execute the shell scripts referenced by the Vagrant configuration below. Otherwise, to learn more about Vagrant and install it in your environment, see the official project site.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.hostname = "ubuntu.server.org"
  config.vm.provider :virtualbox do |vm|
    vm.memory = 4096
    vm.cpus = 4
  end

  config.vm.provision :shell, name: "install-prerequisites", path: "third-party-install.sh"
  config.vm.provision :shell, name: "install-cmake", path: "cmake-install.sh"
  config.vm.provision :shell, privileged: false, name: "compile-or-tools", path: "or-tools-compile.sh"
  config.vm.provision :shell, privileged: false, name: "package-or-tools", path: "or-tools-package.sh"
end

Download the Vagrant and shell scripts from GitHub and save them in the same folder.

Open the folder in terminal and run the vagrant up command.

vagrant up

The workflow consists of the following phases:

Download the operating system image,
Install external tools and some third-party libraries required for compilation,
Install the latest version of CMake from sources,
Download the project sources,
Download and compile remaining third-party libraries required by the project,
Compile the project,
Package the project as a library along with its third-party dependencies.

The whole packaging process may take up to 2 hours on a slow network connection. Fortunately, the process requires no user interaction.

Once the final step completes the package or-tools-6.0-ubuntu-trusty-a1_amd64.deb should be written to the directory that contains the Vagrant script.

Setup a local, secure package repository

Integrity of Debian package repositories is protected by a royalty-free, key-based cryptographic solution alternative to the commercial proprietary Pretty Good Privacy (PGP). Its open standard, known as OpenPGP, is defined in RFC4880. Free implementation of the standard is available on Linux distributions by GNU Foundation as GnuGP Privacy Guard or GnuPG for short. The software is distributed on Debian and Ubuntu as the gnupg package.

Install the gnupg package.

sudo apt-get install gnupg

Successful generation of a new PGP key pair requires sufficient amount of entropy in the system. The exact minimum threshold depends on the size of a key and the algorithm selected in the wizard.

Check the amount of entropy in the system.

cat /proc/sys/kernel/random/entropy_avail

A value larger than 3000 bits is sufficient to create strong 4096 bit keys. If the value is lower than 1000 bits, you certainly should generate more entropy. Otherwise, the gpg program will issue a warning about insufficient entropy in kernel and wait until more entropy is available. That may force you to stop the key generation process if you are connected via SSH, have a single terminal window and no means to generate more entropy.

Install rng-tools to accelerate entropy collection by kernel.

apt-get install rng-tools

The package contains the hardware RNG entropy gatherer daemon rngd. The program uses hardware random number generators if they are available in the system. They are faster than traditional sources of entropy, such as inter-interrupt timings, inter-keyboard timings or disk latency. The rngd daemon should be started automatically after the package is installed.

Generate a PGP key pair. To make things simpler for the first time, provide an empty password while answering the wizard questions. This way you will avoid the password prompt before importing a package to the local repository.

gpg --gen-key

If you would like to use an existing key, import it using the command below.

gpg --allow-secret-key-import --import private.key

For completeness the cheat sheet below contains commands for common operations on PGP keys. The --armor option is used to export a key in ASCII format, so the output can be printed and sent via email. Other options are self-explanatory. For information how to use the gpg utility to sign documents, verify signatures, edit or revoke keys check the man pages.

GPG Cheat Sheet
gpg –gen-key
gpg –output public.key –armor –export email.address
gpg –output private.key –armor –export-secret-key email.address
gpg –import public.key
gpg –allow-secret-key-import –import private.key
gpg –delete-key email.address
gpg –delete-secret-key email.address
gpg –list-keys
gpg –list-secret-keys

Install the reprepro program for local Debian package repository administration. It simplifies and automates file system operations that otherwise would be toilsome and error-prone.

sudo apt-get install reprepro

Create a directory for repository and open it in a terminal window.

mkdir debian && cd debian

Create a directory named conf and move to it.

mkdir conf && cd conf

Create a file named distributions.

touch distributions

The file stores the repository metadata and configuration. The listing below presents an example content.

Origin: pmateusz.github.io/debian
Label: pmateusz.github.io/debian
Codename: jessie
Architectures: amd64
Components: main
Description: Personal repository
SignWith: B2323F7E

The meaning of specific headers is explained in the table below.

Header	Required	Possible Values	Documentation
Origin	No	Any text	Name of maintainer, company name, address of repository
Label	No	Any text	Name of maintainer, company name, address of repository
Codename	Yes	Jessie, Trusty etc.	Unique identifier of a distribution. Fixed and not supposed to change. Used to reference the repository in user configuration.
Architectures	Yes	amd64, i386, source	Multiple values can be provided separated by spaces.
Components	Yes	main, non-free, contrib	Apt repository naming scheme. Used to reference the repository in user configuration.
SignWith	No	yes, default, KEY.ID	The parameter passed to the `libpgpgme` library and controls the selection of a private key used to issue a signature. If the value is set to yes or default the first key will be used. If packages are not to be signed, the SignWith header should be removed.

The KEY.ID value for the SignWith header can be obtained from the gpg --list-secret-keys output.

username@debian:~/dev/debian/conf$ gpg --list-secret-keys
/home/username/.gnupg/secring.gpg
---------------------------------
sec   4096R/B4411E34 2017-05-19 [expires: 2018-05-19]
uid                  Forename Surname (username) <email.address>
ssb   4096R/B2323F7E 2017-05-19

In the example above B2323F7E is the key id to use in configuration. For curious readers sec and ssb are resolved to secret key and secret subkey respectively. For more information on GPG subkeys see the blog post.

Import the package to the local repository. Replace repository path, codename and package with your values.

reprepro --basedir REPOSITORY.PATH includedeb CODENAME PACKAGE

Other common repository operations are listing packages stored in a repository and removing a package from a repository.

reprepro --basedir REPOSITORY.PATH list CODENAME
reprepro --basedir REPOSITORY.PATH remove CODENAME PACKAGE

Host a package repository using GitHub Pages

Create a new repository in GitHub.

Open the Settings tab and move to the GitHub Pages section. Enable GitHub Pages feature, select master branch as the source and save your changes.

Initiate an empty local Git repository in the root folder of the package repository.

git init

Export the GPG public key that will be used by clients to verify package signatures.

gpg --output PUBLIC.KEY --armor --export EMAIL.ADDRESS

Copy the public key to the local Git repository.

mv PUBLIC.KEY REPOSITORY.PATH

Add all repository files to the local repository.

git add --all
git commit -m "Package repository release"
git remote set-url origin git@github.com:USERNAME/REPOSITORY.git
git push origin master

Register an external repository

Add the public GPG key to the apt sources keyring.

wget -qO - https://USERNAME.github.io/REPOSITORY/PUBLIC.KEY | sudo apt-key add -

To list and remove a key from apt sources use the following commands respectively.

apt-key list
sudo apt-key del KEY.ID

sudo vim /etc/apt/sources.list

Add the following line, replace variables with your values and save your changes.

deb http://USERNAME.github.io/REPOSITORY CODENAME COMPONENT

Refresh the apt configuration.

sudo apt-get update

Finally, install the package from your external repository.

sudo apt-get install PACKAGE

Building Pyramids

Personal DEB Package Repository using GitHub Pages

Create a DEB package

Setup a local, secure package repository

Host a package repository using GitHub Pages

Register an external repository

Personal DEB Package Repository using GitHub Pages

Create a DEB package

Setup a local, secure package repository

Host a package repository using GitHub Pages

Register an external repository

Share this post