Building Pyramids

Matt Polnik's blog

CMake Project with Third-party Dependencies

C++ Cmake

CMake project setup

Setting up a build for a new software project requires foresight and experience. Decisions made at this step influence compilation time and effort needed to configure a development machine. Having in mind that there is no replacement for practice, this article aims at providing a baseline configuration for other a C or C++ projects that will be built using CMake. The post targets persons who did not cut their teeth on CMake and consider it as the build tool in their own project.

Introduction

CMake is a cross platform and open source tool for building C/C++ projects. CMake works by generating a collection of make files based on the project definition contained in the CMakeLists.txt file. The project definition is expressed using a domain specific language that CMake interprets. Apart from traditional constructs offered in any programming language, such as assignment instruction, control flow statements, loops or function definition, CMake supports an extensive list of predefined functions. They serve two major purposes: hiding platform specific aspects of a build definition and simplifying common tasks that are often required to build a project. Examples of both could be file system operations and finding libraries. Make files generated by CMake are then processed by the make tool to build a project.

In the following sections we present a basic CMake build configuration for a C/C++ project that depends on third-party libraries. As the use case we will create CMake files to build a program that uses the SCIP Optimization Suite, a collection of libraries for solving constraint integer optimization problems. The exact application of the program itself is not relevant in our context, though the list of engineering tasks we are going to solve is a good representation of what you may face in setting up a build configuration for any C or C++ project yourself. For simplicity we assume that all required dependencies are installed in the system. If this assumption does not hold, we expect CMake to fail with a meaningful error message. Although it is possible to develop a CMake configuration that will download missing dependencies, we will not cover this subject here to maintain the introductory level of the post. If you would like to learn more on that, we refer you to the ExternalProject_Add function in the official CMake documentation.

The CMake configuration that will be developed in this tutorial is available at GitHub. You may either follow the tutorial below step by step or download the sample code upfront. The following command will download all files created in this tutorial and save them into the scip-example directory.

svn export https://github.com/pmateusz/examples/trunk/cmake_sample_project scip-example

This tutorial was created using Debian Stretch. If you use a different operating system you may need to modify commands that install packages and their names in the prerequisites section.

Prerequisites

  1. Install CMake, g++ and external libraries required by SCIP.
    sudo apt-get install --assume-yes bison build-essential cmake flex g++ libgmp-dev libreadline-dev libncurses-dev zlib1g-dev
    
  2. Download and install SCIP.
    wget http://scip.zib.de/download/release/SCIPOptSuite-5.0.1-Linux.deb
    sudo dpkg --install SCIPOptSuite-5.0.1-Linux.deb
    

Tutorial

  1. Create a directory for the project and open it in a command prompt.
    mkdir scip-example && cd scip-example
    
  2. Create subdirectories for the source code and CMake modules.
    mkdir src
    mkdir -p cmake/Modules
    

    Source code and header files will be located in the src directory. CMake modules, which are responsible for finding third-party libraries, will be stored in the cmake/Modules directory.

  3. Download the project source code from GitHub.

    for file in cmain.c relax_lp.c relax_lp.h relax_nlp.c relax_nlp.h
    do
    wget https://raw.githubusercontent.com/pmateusz/examples/master/cmake_sample_project/src/$file -P src
    done
    
  4. Create the CMakeLists.txt file with the following content.

    cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
    project(demo LANGUAGES C CXX VERSION 0.0.1)
    set(CMAKE_VERBOSE_MAKEFILE FALSE)
    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(CMAKE_POSITION_INDEPENDENT_CODE ON)
    if (CMAKE_COMPILER_IS_GNUCXX)
     set(CMAKE_C_FLAGS_DEBUG "-g -ggdb -pg -fsanitize=undefined")
     set(CMAKE_C_FLAGS_RELEASE "-O2")
     set(CMAKE_CXX_FLAGS_DEBUG ${CMAKE_C_FLAGS_DEBUG})
     set(CMAKE_CXX_FLAGS_RELEASE ${CMAKE_C_FLAGS_RELEASE})
    endif ()
    set(CMAKE_BUILD_TYPE RELEASE)
    list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules/")
    find_package(Scip REQUIRED)
    find_package(Threads REQUIRED)
    get_filename_component(HEADERS src REALPATH)
    include_directories(${HEADERS} ${SCIP_INCLUDE_DIRS})
    file(GLOB_RECURSE SOURCES src/*.c)
    add_library(demo STATIC ${SOURCES})
    add_executable(demo-main src/cmain.c)
    target_link_libraries(demo-main demo ${SCIP_LIBRARIES} ${CMAKE_THREAD_LIBS_INIT})
    add_dependencies(demo-main demo)
    

    Comments

    We will now review the file line by line and highlight important concepts.

    cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
    project(demo LANGUAGES C CXX VERSION 0.0.1)
    set(CMAKE_VERBOSE_MAKEFILE FALSE)
    

    Require CMake in version 3.1 or newer. Define a project called demo that requires C and C++ compilers. Project deliverables will have version 0.0.1. Generate non verbose make files. If you change the parameter to TRUE, make files will print the current command before its execution, which could be priceless while troubleshooting build related issues.

    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(CMAKE_POSITION_INDEPENDENT_CODE ON)
    

    Generate make files that compile source code with C++11 standard enabled in a compiler. In case of g++ that boils down to adding the -std=c++11 flag to the compiler arguments. The same technique will not work with MSVC. Always be mindful about possible cross platform issues that hard coding parameters may cause. To avoid them use CMake features whenever possible.

    Code produced by a compiler should be position independent, so the shared libraries built in the project will execute correctly regardless of the memory location where they have been loaded to.

    if (CMAKE_COMPILER_IS_GNUCXX)
     set(CMAKE_C_FLAGS_DEBUG "-g -ggdb -pg -fsanitize=undefined")
     set(CMAKE_C_FLAGS_RELEASE "-O2")
     set(CMAKE_CXX_FLAGS_DEBUG ${CMAKE_C_FLAGS_DEBUG})
     set(CMAKE_CXX_FLAGS_RELEASE ${CMAKE_C_FLAGS_RELEASE})
    endif ()
    set(CMAKE_BUILD_TYPE RELEASE)
    

    We set extra compiler flags that should be used when compiling the project. Files with the .c extension will use the C compiler, while files with .cxx or .cpp extensions will be compiled by the C++ compiler.

    The build target is set to RELEASE, so the -O2 flag, which as of this writing is the recommended optimization level for GCC, will be used. If you change the build type to DEBUG flags -g -ggdb -pg -fsanitize=undefined will be passed instead. Debug symbols will be added, extra debugging information will be produced to improve debugging experience in GDB, extra tracing instructions for gprof will be produced to enable profiling the code. Finally, additional checks will be run to detect situations which lead to undefined behaviors, such as null pointer dereference or arithmetic overflows. To read more about undefined behavior sanitizer see the Red Hat blog post or Clang documentation.

    Notice that these flags are passed only for the GNU GCC compiler, which protects us from running into cross platform issues.

    list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules/")
    

    Instruct CMake where to find additional scripts that find third-party libraries. These scripts are also known as modules and will be developed in the upcoming sections.

    find_package(Scip REQUIRED)
    find_package(Threads REQUIRED)
    

    Look for the Scip library and the system threading model. In case any of the dependencies is not satisfied stop generating make files and fail with an error. While searching for a library CMake first tries to load a FindLibraryName.cmake file, where LibraryName is the name of a library passed in the find_package function, and then interprets its content. Regardless of the operating system library names in CMake are case sensitive. If in doubt double check that you are using a library name in the correct form.

    The CMake scripts interpreted by the find_package function export new variables that point to the library binaries and header files. By convention the path to a symbolic object or an archive is stored in theLIBRARYNAME_LIB variable, where LIBRARYNAME is the name of the library in capital letters. For example, find_package(Scip REQUIRED) should create a variable SCIP_LIB with a binary file of the library. Header file location is saved in the variable LIBRARYNAME_INCLUDE_DIR. Unfortunately, the naming convention for CMake modules is not strictly followed, even inside the CMake project, so before reusing any official module consider reading its documentation(https://cmake.org/cmake/help/latest/manual/cmake-modules.7.html).

    get_filename_component(HEADERS src REALPATH)
    include_directories(${HEADERS} ${SCIP_INCLUDE_DIRS})
    

    Resolve a relative path to the src directory to an absolute path and save it as the value of the HEADERS variable. Then declare header files’ locations. Apart from the local header files, we also declare the header files of the SCIP library and its dependencies saved in the SCIP_INCLUDE_DIRS variable.

    You may notice that values of CMake variables are referenced using the ${...} operator.

    file(GLOB_RECURSE SOURCES src/*.c)
    add_library(demo STATIC ${SOURCES})
    

    Set paths to all local source code files as the value of the SOURCES variable and create a statically linked library with them. Notice that the first argument in the add_library function, which corresponds to a library name, matches the project name. It will be the default build target of the project.

    add_executable(demo-main src/cmain.c)
    target_link_libraries(demo-main demo ${SCIP_LIBRARIES} ${CMAKE_THREAD_LIBS_INIT})
    add_dependencies(demo-main demo)
    

    Declare another project target to build an executable from the src/cmain.c file. The executable will be linked against the project sources, the SCIP library, its dependencies and the threading model.

    A prerequisite for successful linking of the demo-main target is availability of the local library build in the demo target. This dependency is declared using the add_dependencies function.

  5. Develop a CMake modules for the Termcap library and the GNU Multiple Precision Library.

    Create the FindTermcap.cmake file in the cmake/Modules directory with the following content.

    find_path(TERMCAP_INCLUDE_DIR termcap.h)
    find_library(TERMCAP_LIBRARY termcap)
    include(FindPackageHandleStandardArgs)
    find_package_handle_standard_args(TERMCAP DEFAULT_MSG TERMCAP_LIBRARY TERMCAP_INCLUDE_DIR)
    

    Comments

    CMake modules of these libraries will be developed first, because their structure is simple and similar to each other. We will reduce complexity of CMake modules to bare minimum, since you are unlikely to compile these libraries yourself. Both libraries have their official packages available for most Unix-like operating systems. Moreover the first versions of each library was released in 90’, so their API is mature and codebase stable.

    find_path(TERMCAP_INCLUDE_DIR termcap.h)
    

    Search for the termcap.h header file using the find_path function that assigns the absolute path to the directory containing the file name to the first variable passed to the function. Note that we rely on CMake to check common header files’ locations in the operating system by not passing any suggested directories. This behavior should work well as soon as header files can be found in one of common locations.

    find_library(TERMCAP_LIBRARY termcap)
    

    Find a library with the termcap stem. CMake will not have problems with detecting that libtermcap.a or libtermcap.so are an archive and a shared object of the Termcap library respectively. This capability is built into CMake, because library prefixes and their file extensions depend on the operating system. By default if both an archive and a shared object are available, CMake will prefer the shared object.

    include(FindPackageHandleStandardArgs)
    find_package_handle_standard_args(TERMCAP DEFAULT_MSG TERMCAP_LIBRARY TERMCAP_INCLUDE_DIR)
    

    Import the FindPackageHandleStandardArgs module. This module contains the find_package_handle_standard_args function, which should be called with variables denoting the package name, a failure message and variables that should be returned to the caller. The DEFAULT_MSG parameter means that the default failure message format will be used if required.

    We skip the FindGMP.cmake module, because its structure strictly follows the FindTermcap.cmake module.

    Both modules developed in this section can be downloaded using the command.

    for file in FindTermcap.cmake FindGMP.cmake
    do
    wget https://raw.githubusercontent.com/pmateusz/examples/master/cmake_sample_project/cmake/Modules/$file -P cmake/Modules
    done
    
  6. Create the FindReadline.cmake file in the cmake/Modules directory with the following content.

    find_path(READLINE_INCLUDE_DIR readline/readline.h)
    find_library(_READLINE_LIB readline)
    if (NOT _READLINE_LIB)
     message(FATAL_ERROR "Not found libreadline.a")
    endif ()
    find_library(_HISTORY_LIB history)
    if (NOT _HISTORY_LIB)
     message(FATAL_ERROR "Not found libhistory.a")
    endif ()
    set(READLINE_LIBRARY ${_READLINE_LIB})
    list(APPEND READLINE_LIBRARY ${_HISTORY_LIB})
    mark_as_advanced(_READLINE_LIB _HISTORY_LIB)
    find_package(Curses REQUIRED)
    find_package(Termcap REQUIRED)
    include(FindPackageHandleStandardArgs)
    find_package_handle_standard_args(READLINE DEFAULT_MSG READLINE_LIBRARY READLINE_INCLUDE_DIR)
    if (READLINE_FOUND)
     set(READLINE_LIBRARIES ${READLINE_LIBRARY})
     list(APPEND READLINE_LIBRARIES ${CURSES_LIBRARIES})
     list(APPEND READLINE_LIBRARIES ${TERMCAP_LIBRARY})
     set(READLINE_INCLUDE_DIRS ${READLINE_INCLUDE_DIR} ${CURSES_INCLUDE_DIR} ${TERMCAP_INCLUDE_DIR})
    endif ()
    

    Comments

    The Readline library also has an official package with its binaries. Albeit, its CMake module is more complex. Readlines is distributed as 2 files: readline and history, which further depend on libraries: Curses and Termcap.

    Handling a library that is distributed via multiple binary files is a three step process. Firstly, find a file path to every file or fail if it cannot be found. Then create a variable that contains a list of all file paths separated by semicolon. Note that the list function facilitates handling such variables. Finally, mark the temporary variables used to find the binary files as advanced variables. That will instruct third-party tools that offer CMake integration that these variables are helper variables, which should remain hidden and unavailable for an external assignment.

    Find the library dependencies. Export their binaries and header files using extra variables: READLINE_LIBRARIES and READLINE_INCLUDE_DIRS respectively.

  7. Create the FindSoplex.cmake file in the cmake/Modules directory with the following content.

    if (SOPLEX_ROOT_DIR)
     set(_SOPLEX_INCLUDE_LOCATIONS " ${SOPLEX_ROOT_DIR}")
     set(_SOPLEX_LIB_LOCATIONS "${SOPLEX_ROOT_DIR}")
    else ()
     set(SOPLEX_ROOT_DIR "" CACHE PATH "Folder contains Soplex library")
     set(_SOPLEX_INCLUDE_LOCATIONS "")
     set(_SOPLEX_LIB_LOCATIONS "")
    endif ()
    find_path(SOPLEX_INCLUDE_DIR soplex.h HINTS ${_SOPLEX_INCLUDE_LOCATIONS} PATH_SUFFIXES src)
    find_library(SOPLEX_LIBRARY soplex HINTS ${_SOPLEX_LIB_LOCATIONS} PATH_SUFFIXES lib)
    include(FindPackageHandleStandardArgs)
    find_package_handle_standard_args(SOPLEX DEFAULT_MSG SOPLEX_LIBRARY SOPLEX_INCLUDE_DIR)
    mark_as_advanced(_SOPLEX_INCLUDE_LOCATIONS _SOPLEX_LIB_LOCATIONS)
    

    Soplex is an example of a library that does not have an official installation package. In prerequisites of this tutorial we downloaded a package with this library from a third-party source. However, we might as well compile the library from sources and install it in a custom location. To retain such flexibility define an optional SOPLEX_ROOT_DIR variable which, if assigned, will point to the root directory of the Soplex project. This location is passed to find_path and find_library functions as the HINTS parameter. It has a priority over other paths that CMake may otherwise search to find a file or a binary. Note that we also use the PATH_SUFFIXES parameter to pass information about subdirectories of the main location where a file or a library could be found. For example, the Soplex project stores source files in the src subdirectory, which is in turn is used as the path suffix in the find_path function.

    Other structures have been explained already in the previous steps.

  8. Create the FindScip.cmake file in the cmake/Modules directory with the following content.

    if (SCIP_ROOT_DIR)
     set(_SCIP_INCLUDE_LOCATIONS "${SCIP_ROOT_DIR}")
     set(_SCIP_LIB_LOCATIONS "${SCIP_ROOT_DIR}")
    else ()
     set(SCIP_ROOT_DIR "" CACHE PATH "Folder contains SCIP library")
     set(_SCIP_INCLUDE_LOCATIONS "")
     set(_SCIP_LIB_LOCATIONS "")
    endif ()
    find_path(SCIP_INCLUDE_DIR scip/scip.h HINTS ${_SCIP_INCLUDE_LOCATIONS} PATH_SUFFIXES src)
    find_library(SCIP_LIBRARY scip HINTS ${_SCIP_LIB_LOCATIONS} PATH_SUFFIXES lib lib/static)
    find_package(ZLIB REQUIRED)
    find_package(Readline REQUIRED)
    find_package(Soplex REQUIRED)
    find_package(Gmp REQUIRED)
    include(FindPackageHandleStandardArgs)
    find_package_handle_standard_args(SCIP DEFAULT_MSG SCIP_LIBRARY SCIP_INCLUDE_DIR)
    if (SCIP_FOUND)
     set(SCIP_LIBRARIES ${SCIP_LIBRARY})
     list(APPEND SCIP_LIBRARIES ${SOPLEX_LIBRARY})
     list(APPEND SCIP_LIBRARIES ${GMP_LIBRARY})
     list(APPEND SCIP_LIBRARIES ${READLINE_LIBRARIES})
     list(APPEND SCIP_LIBRARIES ${ZLIB_LIBRARY})
     set(SCIP_INCLUDE_DIRS ${SCIP_INCLUDE_DIR}
             ${GMP_INCLUDE_DIR}
             ${SOPLEX_INCLUDE_DIR}
             ${READLINE_INCLUDE_DIR}
             ${ZLIB_INCLUDE_DIR})
    endif ()
    

    All constructs used to develop the file have been already explained in the previous examples.

  9. Finally, generate the make files and build the project. To maintain separation between source files and build products we output the make files and perform the build in the build directory.

    mkdir build && cd build
    cmake ..
    -- The C compiler identification is GNU 6.3.0
    -- The CXX compiler identification is GNU 6.3.0
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compilershell: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.a (found version "1.2.8") 
    -- Found Curses: /usr/lib/x86_64-linux-gnu/libcurses.a  
    -- Found TERMCAP: /usr/lib/x86_64-linux-gnu/libtermcap.a  
    -- Found READLINE: /usr/lib/x86_64-linux-gnu/libreadline.a;/usr/lib/x86_64-linux-gnu/libhistory.a  
    -- Found SOPLEX: /home/pmateusz/Applications/scipoptsuite-5.0.1/soplex/lib/libsoplex.a  
    -- Found GMP: /usr/lib/x86_64-linux-gnu/libgmp.a  
    -- Found SCIP: /home/pmateusz/Applications/scipoptsuite-5.0.1/scip/lib/static/libscip.a;/home/pmateusz/Applications/scipoptsuite-5.0.1/scip/lib/static/libobjscip.a;/home/pmateusz/Applications/scipoptsuite-5.0.1/scip/lib/static/libtpinone.a;/home/pmateusz/Applications/scipoptsuite-5.0.1/scip/lib/static/liblpispx2.a;/home/pmateusz/Applications/scipoptsuite-5.0.1/scip/lib/static/libnlpi.cppad.a  
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Looking for pthread_create
    -- Looking for pthread_create - not found
    -- Looking for pthread_create in pthreads
    -- Looking for pthread_create in pthreads - not foundrque Job Management Cover Image
    -- Looking for pthread_create in pthread
    -- Looking for pthread_create in pthread - found
    -- Found Threads: TRUE  
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /home/pmateusz/dev/cmake_sample_project/build
    
    make
    Scanning dependencies of target demo
    [ 16%] Building C object CMakeFiles/demo.dir/src/cmain.c.o
    [ 33%] Building C object CMakeFiles/demo.dir/src/relax_lp.c.o
    [ 50%] Building C object CMakeFiles/demo.dir/src/relax_nlp.c.o
    [ 66%] Linking CXX static library libdemo.a
    [ 66%] Built target demo
    Scanning dependencies of target demo-main
    [ 83%] Building C object CMakeFiles/demo-main.dir/src/cmain.c.o
    [100%] Linking CXX executable demo-main
    [100%] Built target demo-main
    

Summary

In this post we reviewed a list of CMake functions that are necessary to setup a build of a project that depends on third-party libraries. We then explained the process of finding dependencies by CMake and implemented example modules. The first module handled a library that does not have external dependencies and can be installed using a package downloaded from the official repository. We then switched to more complex scenarios: a library distributed using multiple binaries, a library with third party dependencies and a library built locally whose project root directory should have priority over other locations.

Overall, the material covered here should be sufficient to start development of a new project, but it is definitely not all what CMake has to offer. We mention the most important features below to help you avoid reinventing the wheel. The post is concluded with reference materials on CMake for self-study.

CMake is distributed with additional tools that bundled together form a comprehensive toolbox for building, testing and distributing software. CTest is a test runner independent of a testing framework used to develop the tests. Furthermore, CTest configuration is recognized by software development environments, such as CLion or Visual Studio, which allow to run and debug tests within an editor.

CPack is a generic purpose packaging tool that can create compressed archives or packages with project binaries. CPack supports DEB and RPM package formats, which jointly cover majority of UNIX-like systems.

Apart from finding installed dependencies CMake can automatically download and compile third-party libraries needed to build a project. This feature comes in handy every time a library does not have an installation package available on your system or the package has not been released for the particular version of the library you would like to use. You may consider incorporating this step into build configuration by means of ExternalProject_Add. That upfront investment should pay off and save your time later if you are going to develop the project within a community or use an external continuous integration system.

Finding a high quality learning material on CMake for an entry level person can be difficult, because the official documentation assumes that you are already familiar with the tool. Personally, I ended up reviewing repositories of large open source projects and reading how others solved problems I was going to address. A repository of a deep learning framework Caffe turned out to be very useful to me with this respect. For best practices on writing CMake modules I referred to the official CMake repository which contains modules for the most popular libraries, such as Google Protocol Buffers, libxml2 or ZLIB to name a few.