Diary #16: CPacking debs

I’ve been having fun packing .deb files, and it’s pretty easy to do with CPack once you’ve scrounged up the boilerplate that you have to copy and paste into your CMakeLists.txt. For example, here’s a sample minimal CMakeLists.txt that incorporates Boost filesystem:

find_package( Boost 1.49 COMPONENTS system filesystem )
include_directories( ${Boost_INCLUDE_DIR} )

add_executable( atBoostTest test.cpp )
target_link_libraries( atBoostTest ${Boost_LIBRARIES} )

install( TARGETS atBoostTest
    RUNTIME
    DESTINATION bin
)

set(CPACK_PACKAGE_NAME "atBoostTest")
set(CPACK_PACKAGE_VENDOR "atsui")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "atBoostTest - test package with boost-based command-line program")

set(CPACK_PACKAGE_VERSION "0.1.0")
set(CPACK_PACKAGE_VERSION_MAJOR "0")
set(CPACK_PACKAGE_VERSION_MINOR "1")
set(CPACK_PACKAGE_VERSION_PATCH "0")

set(CPACK_GENERATOR "DEB")
set( CPACK_DEBIAN_PACKAGE_MAINTAINER "atsui" )
set( CPACK_DEBIAN_PACKAGE_DEPENDS "libc6 (>= 2.3.2), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.6), libboost-dev (>= 1.49), libboost-filesystem-dev (>= 1.49)" )

message( STATUS "${CMAKE_SYSTEM_PROCESSOR}" )
set( CPACK_SYSTEM_NAME "${CMAKE_SYSTEM_PROCESSOR}" )

include( CPack )

Lines 1-10 are a typical project where you specify a target, dependent libraries, and installation instructions. Everything afterwards is CPack boilerplate that you can start out with.

The important one is probably the CPACK_DEBIAN_PACKAGE_DEPENDS, where you declare packages that your app depends on. This means that if you manage to write software using standard dev packages, you can just reference them here and users can install and run your deb just fine (see gdebi).

So that means that this works differently from the OSX bundle fixup workflow where a fully distributable, monolithic bundle is created by pulling in all dependent libraries instead of relying on a package manager to be able to fill in the blanks for you.

Another thing that I am excited to try out is that you can actually create a Windows installer. I found this SO question that seems to have the details as well. I will need to try this out with the MinGW build of my app.

Wrap Fortran code to use in C++

There’s a lot of useful code in Fortran, and a lot of people actually prefer to work in Fortran, so as a C++ developer, it would be nice to be able to call their code from C++. Here’s a small CMake example for how to do it so you can follow along below: Download

I put the example together and tested it with the following versions of these tools:

  • g++ 4.7.3
  • gfortran++ 4.7.3
  • cmake 2.8.10.1

From a high level, the example does the following:

  1. Create a one-function Fortran library.
  2. Set up the interface to the Fortran library in C++ code.
  3. Link to the Fortran library and call the Fortran function.

Something to note is that all the parameters to Fortran function is a pointer type, even for non-array data types.

To demystify the part of the example that sets up the interface to Fortran code:

extern "C" 
{
    void average_(int *n, double *a, double *ave);
}

The extern "C" part tells the C++ compiler not to expect a mangled name but rather take the function name as it is. C++ renames function names at compile-time in order to support function overloading automatically, but this behavior should be disabled when interfacing with code from other languages like Fortran.

Note that the average function has an underscore appended to it, whereas it does not have it in the Fortran file where it is defined. This is default behavior of gfortran. I think I read that there is a flag to disable this behavior, but I think it’s nice to make it clear that it is a Fortran function.

Boost.Python hello world example using CMake

Somehow it seems I’m the only one among my labmates who is working mainly in C++/CMake and not Python. I want to do more than help them by discussing code — I want to share my code. So let’s use Boost.Python wrap our C++ library so it can be called from Python code. This tutorial is based on the Hello world example in the Boost.Python docs and uses CMake.

To begin with, here’s a mention of the versions of everything I’m using when I wrote this post so you can judge if stuff is still applicable:

  • Ubuntu 13.04
  • cmake version 2.8.10.1
  • gcc version 4.7.3
  • Boost 1.49 with Python component
  • Python 2.7.4

Here, we create a simple C++ library called libgreet, and we make a wrapper library called greet_ext. The wrapper library can be loaded as a Python module that exposes the function in the C++ library. The files are shown below, and the full example can be downloaded here. To use it, just download it and run these commands:

tar xf BoostPythonHelloWorld.tar.gz
cd BoostPythonHelloWorld
cmake .
make
./test.py

greet.h:

#ifndef GREET_H
#define GREET_H
char const* greet( );
#endif // GREET_H

greet.cpp:

#include "greet.h";

char const* greet( )
{
    return "Hello world";
}

greet_ext.cpp:

#include "greet.h";
#include <boost/python.hpp>;

BOOST_PYTHON_MODULE(greet_ext)
{
    using namespace boost::python;
    def( "greet", greet );
}
  • The parameter to BOOST_PYTHON_MODULE must match the name of the wrapper library we import in Python below. Note where greet_ext appears throughout the different files.

CMakeLists.txt:

cmake_minimum_required( VERSION 2.8 )

project( BoostPythonHelloWorld )

# Find necessary packages
find_package( PythonLibs 2.7 REQUIRED )
include_directories( ${PYTHON_INCLUDE_DIRS} )

find_package( Boost COMPONENTS python REQUIRED )
include_directories( ${Boost_INCLUDE_DIR} )

# Build our library
add_library( greet SHARED greet.cpp )

# Define the wrapper library that wraps our library
add_library( greet_ext SHARED greet_ext.cpp )
target_link_libraries( greet_ext ${Boost_LIBRARIES} greet )
# don't prepend wrapper library name with lib
set_target_properties( greet_ext PROPERTIES PREFIX "" )
  • The line with PythonLibs locates the python includes, which are necessary because the Boost.Python header pulls in a config file from there.
  • By default, CMake builds library targets with the lib prefix, but we actually want our wrapper library to be named greet_ext.so so we can import it with the name below.

test.py:

#!/usr/bin/python

import greet_ext

print greet_ext.greet()
  • This python script works for me if I put it in the same folder as the greet_ext.so wrapper library after it is built.

What I need to figure out
This is a nice, simple example to get started with, but I have to figure out more details.

  • How do I wrap functions that take/return C++ types? How about pointers and references to those types? How about global operators?
  • How does a wrapper library deal with C++ templates? Is it the case that you have to instantiate some of those template classes are just work with those when you’re in Python?
  • Is Boost.Python the right choice? There’s a fairly recent webpage that discusses SWIG versus Boost.Python, and I think I’ve made the right choice for my project.

I’m looking forward to working on this further. This probably will be one of the things I work on this coming Christmas break.

P.S. God WordPress is so annoying when it escapes your code when you toggle between visual and text editing modes. Note to self, don’t switch modes.

Using CMake to create a bundle for a Qt4/VTK/CGAL project

I write a GUI application that uses Qt4 as a frontend as well as other libraries in the backend, like VTK, CGAL, Boost, and what-have-you, and because I use CMake as my build system, it’s quite manageable for developers on either Windows, OS X, or Linux to build from source code. It gets a little more complicated when I need to build distributable executables for non-developers to use, and so far I do not have a good way to do this written into my main CMakeLists.txt. However, I was recently able to build a bundle, which is OS X’s distributable executable that you can pass around, put into your Applications folder, and double-click to run, and this blog has some details that might be useful, especially if you have experience building distributables on a Linux environment.

To begin, note the versions of things that I’m using:

  • OS X 10.8.5
  • CMake 2.8.12
  • CGAL 4.3
  • Qt 4.8.5
  • VTK 5.10.1

The general approach is the following:

  1. Build CMake
  2. Build the library dependencies using our CMake
  3. Build our project
  4. Install our project in a portable way – in Linux, the executable and its library dependencies are copied into an install folder. The executable is hardwired with a path (RPATH) that points to where to look for library dependencies. The executable is portable in the sense that the RPATH can be set to a path relative to the executable (with the @ORIGIN keyword), so you can relocate the executable and still have it find the libraries it needs.
    In OS X, the distributable executable is a bundle, which is basically a folder containing the executable and the libraries in a particular structure. CMake has a module to set this up automatically on install. This module also performs bundle fixup where library dependencies are pointed to the ones in the bundle folder rather than the ones on your system.

Why are we doing it this way?
This approach might seem like overkill: Homebrew is a package manager that lets you install CMake and all the libraries you probably would need. However, there is an issue with the last step in the general approach. Basically, when CMake copies the dependency libraries into the bundle, it messes up. The only way I’ve found how to get this working is not to use Homebrew’s CMake but use our own CMake.

Okay, let’s step through.

1. Build CMake
I’m just going to assume you got the source package, built and installed it, and added the bin folder to your path. One further thing that I did was edit a file that was giving me errors during bundle fixup. In share/cmake-2.8/Modules/BundleUtilities.cmake:

--- BundleUtilities.orig.cmake 2013-11-28 01:11:34.000000000 -0800
+++ BundleUtilities.cmake 2013-11-02 20:52:04.000000000 -0700
@@ -585,9 +585,9 @@
 endif()
 endforeach()

- if(BU_CHMOD_BUNDLE_ITEMS)
+ #if(BU_CHMOD_BUNDLE_ITEMS)
 execute_process(COMMAND chmod u+w "${resolved_embedded_item}")
- endif()
+ #endif()

 # Change this item's id and all of its references in one call
 # to install_name_tool:

2. Build the library dependencies using our CMake
I won’t write anything about this except post the line that I used to configure Qt4. Basically, I lifted the configuration from the brew script for qt4, which seemed to have some OS X specific things in it:

./configure -system-zlib -confirm-license -opensource -nomake demos -nomake     examples -cocoa -fast -release -no-3dnow -no-ssse3 -platform unsupported/macx-  clang -nomake docs -qt3support -prefix ${PREFIX}

3. Build our project
I also won’t write anything about this part since it is just a matter of using find_package to locate and pull in the required libraries, and I assume you’ve got your project configured.

4. Install our project in a portable way
So here we throw in a few extra lines into CMakeLists.txt. A full example is available from the CMake wiki here. Say your build target is named MyProgram, then you would include some install instructions that look like this:

install( TARGETS
MyProgram
BUNDLE DESTINATION .
RUNTIME DESTINATION bin
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib/static
)

Now I already had an install directive, but line 3 was the new line to add. Basically, if you’re building on any other OS other than OS X, your executable will end up in ${CMAKE_INSTALL_PREFIX}/bin, otherwise, it will go into a bundle folder named after the build target, which in this case will be ${CMAKE_INSTALL_PREFIX}/./MyProgram.app. This explains the following lines that come afterwards:

set( APPS "\${CMAKE_INSTALL_PREFIX}/MyProgram.app" )
set( DIRS "\${CMAKE_INSTALL_PREFIX}/lib" )
set(qtconf_dest_dir MyProgram.app/Contents/Resources)
install(CODE "
file(WRITE \"\${CMAKE_INSTALL_PREFIX}/${qtconf_dest_dir}/qt.conf\" \"\")
" COMPONENT Runtime)
install( CODE "
include(BundleUtilities)
fixup_bundle( \"${APPS}\" \"\" \"${DIRS}\" )
" Component Runtime )

Lines 1-3 just define some helper variables that get used in the following lines. ${APPS} points to the bundle folder and ${DIRS} points to where to look for libraries it depends on. I set this to point to where I install my project library build targets.

Lines 4-6 generates a file in the bundle folder that is necessary to get Qt applications working properly.

Lines 7-11 performs the bundle fixup: when the bundle is initially installed, the executable is linked to required libraries via paths specific to your system. These include libraries external to your project and library build targets within your project. fixup_bundle will identify the libraries from the bundle pointed to by ${APPS}, copy them into the bundle folder, then relink the executable to these copies instead of the ones installed on your system.

It’s a little weird how BundleUtilities actually finds out the library dependencies. On the one hand, it is supposed to look in ${DIRS} to find libraries, but it might not be able to, and for my project, I had to manually copy the ones it missed into the bundle folder and rerun the cmake install to get it fixed up. Also, although I didn’t specify VTK and the others in ${DIRS}, it was able to find and copy these in. In short, during bundle fixup, some errors may occur that you should pay attention to, and if you see missing libraries that should have been copied, you can probably get around by manually copying them into the bundle right next to the executable and rerunning the install.

Additional notes

  1. In Linux, you could inspect dynamic library dependencies with ldd. In OS X, you can make use of otool -L to do the same thing.
  2. In OS X, you can do some manual fixup of library dependencies as BundleUtilities does by using install_name_tool.

Hopefully, reporting all of this in one place was helpful to someone. CMake really works wonders, but the only thing is hunting down enough information to be able to do what you want can take more time than expected. Also, this is my first serious take on building projects on OS X so there was a bit of a learning curve coming from Linux, but it’s not so bad now.

Feel free to leave questions in the comments.

Happy thanksgiving!

CMake C++ project template

Here’s a minimal CMake C++ project setup that I rolled together that reflects the project structure that I’ve been working with for a while now. Hopefully you can use it to get started on a project pretty quickly:

https://dl.dropbox.com/u/508241/wordpress/barebones.tar.gz

You’ll need to have a few things to get started.

  1. CMake
  2. Doxygen
  3. GoogleTest

CMake is the build system that generates the Makefiles after you specify the build targets and libraries. The convention is to have a CMakeLists.txt file for each subdirectory of your project, specifying any build targets along the way. The root CMakeLists.txt specifies the libraries, custom compiler flags, and subdirectories containing other CMakeLists.txt files. To build the project, do:


cd build/
cmake ..
make

Doxygen is the typical way to generate documentation in C++. If you just follow Javadoc-style comment convention, then you can generate pretty good documentation pretty trivially by just running doxygen from the root directory. This barebones setup is configured to scan the src and doc directories for documented source code and dedicated documentation files (*.dox).

GoogleTest is a unit testing framework that’s pretty similar to JUnit. The conventional use is to create a TestXXX.cpp file for every class XXX that you write, and add it to the test directory. In this project, there is an AllTests build target that will compile a driver program that will run all unit tests that it was built with.

On Ubuntu, you can grab the GoogleTest library files directly from the package manager:


sudo apt-get install libgtest-dev

The annoying thing though is that you’ll need to build the library yourself. There is a CMakeLists.txt file that you can configure and build the library with. You can copy the resulting library file to /usr/local/lib and you should be able to compile the unit test in this project template.