Tag Archives: automake

Running GitLab CI on autotools projects

Inspired by the talk at FOSDEM, I’ve just enabled GitLab’s continuous integration (CI) for building make distcheck for Walbottle, and it was delightfully easy. The results are on Walbottle’s GitLab page.

Steps

  1. Create a ci branch to contain the mess you’ll make while iterating over the correct compile steps.
  2. Create and push a .gitlab-ci.yml file containing build rules similar to the following:
    image: debian:unstable
    
    before_script:
      - apt update -qq
      - apt install -y -qq build-essential autoconf automake pkg-config libtool m4 autoconf-archive gtk-doc-tools libxml2-utils gobject-introspection libgirepository1.0-dev libglib2.0-dev libjson-glib-dev
    
    stages:
      - build
    
    # FIXME: Re-enable valgrind once running the tests under it doesn’t take forever (it causes timeouts).
    # Re-add valgrind to apt-install line above
    build-distcheck:
      stage: build
      script:
        - mkdir build
        - cd build
        - ../autogen.sh --disable-valgrind
        - make V=1 VERBOSE=1
        - DISTCHECK_CONFIGURE_FLAGS=--disable-valgrind make distcheck V=1 VERBOSE=1
    
      # The files which are to be made available in GitLab
      artifacts:
        paths:
          - build/*
  3. Iterate a few times until you get all the dependencies right.
  4. Fix any problems you find (because this might well find problems with your dependency declaration in configure.ac, or other distcheck problems in your project).
  5. Merge ci to master and profit from CI results on every branch and master commit.

Looking at the .gitlab-ci.yml file

For information on the overall layout of the YAML file, and the phases available, you’re best off looking at the comprehensive GitLab documentation. Here are some notes about the autotools-and-C–specific bits of it:

  • The image is a Docker image; I picked a Debian one from the Docker hub.
  • Package installation seems to need to be done in the before_script phase, or the packages can’t be found (which is presumably a protection against rogue build systems).
  • I chose to build distcheck in my build rule because that runs the build, runs the tests, and tries various srcdir ? builddir configurations. You can add other build targets (like build-distcheck to try other build setups).
  • Pass V=1 VERBOSE=1 to get verbose build and test log output in your CI build logs, otherwise you will struggle to work out what is causing any failures.
  • Note that configure flags passed to ./configure are not automatically passed in again when ./configure is run as part of distcheck — so use DISTCHECK_CONFIGURE_FLAGS for that. Ideally, your project will be less fragile than mine, and hence not need any of this.
  • Export the whole build directory as an artifact on success, so you can look at any of the build objects, or the generated tarball, or documentation. You could limit this (for example, to just the tarball) if you’re sure you’ll never need the rest of it.

gnome-common deprecation, round 2

tl;dr: gnome-common is deprecated, but will be hanging around for a while. If you care about modernising your build system, migrate to autoconf-archive macros.

This GNOME release cycle (3.18), we plan to do the last ever release of gnome-common. A lot of its macros for deprecated technologies (scrollkeeper?!) have been removed, and the remainder of its macros have found better replacements in autoconf-archive, where they can be used by everyone, not just GNOME.

We plan to make one last release, and people are welcome to depend on it for as long as they like. However, if you want new hotness, port to the autoconf-archive versions of the macros; but please do it in your own time. There will be no flag day port away from gnome-common.

Note that, for example, porting to AX_COMPILER_FLAGS is valuable, but will probably require fixing a number of new compiler warnings in your code due to increased warning flags. We hope this will make your code better in the long run.

There’s a migration guide here: https://wiki.gnome.org/Projects/GnomeCommon/Migration.

We’ve tried to make the transition as easy and smooth as possible, but there will inevitably be hiccups. Please let me know about anything which breaks or doesn’t make sense, or discuss it on the desktop development list thread. First person to complain about -Wswitch-enum gets a prize.

For developers

When building from a tarball of a module which uses the new macros, you will no longer need gnome-common installed. (Although you may not have needed it before.)

When building from git, you will need m4-common or autoconf-archive installed.

JHBuild bootstrap installs m4-common automatically, as does gnome-continuous; so you don’t need to worry about that.

For packagers

In the 3.14.0 release, gnome-common installed some early versions of the autoconf-archive macros which conflicted with what autoconf-archive itself installs. It now has a --[with|without]-autoconf-archive configure option to control this. We suggest that all packagers pass --with-autoconf-archive if (and only if) autoconf-archive is packaged on the distribution. See bug #747920.

m4-common must not be packaged. See its README. m4-common is essentially a caching subset of autoconf-archive.

For continuous integrators

Modules which use the new AX_COMPILER_FLAGS macro gain a new standard --disable-Werror configure flag, which should be used in CI systems (and any other system where spurious compiler warnings should not cause total failure of a build) to disable -Werror. The idea here is that -Werror is enabled by default when building from git, and disabled by default when building from release tarballs and in buildbots.

For further discussion

See the thread on the desktop development mailing list.

Automatically Valgrinding code with AX_VALGRIND_CHECK

tl;dr: For `make check` with Valgrind support, add AX_VALGRIND_CHECK to configure.ac and @VALGRIND_CHECK_RULES@ to tests/Makefile.am. Use memcheck, helgrind, drd and sgcheck.

Once you’ve written a unit test suite with near-100% coverage, you can validate software with a lot more confidence than before, since any effort you put into validation is working across the whole code base. ((Normal caveats about the combinatorial complexity of dynamic testing apply.)) It’s quite common to check for memory leaks with Valgrind’s memcheck tool — but Valgrind has several additional tools which are useful in production: helgrind, drd and sgcheck. ((Although currently the threading tools may not work with GLib threads, due to its use of futexes rather than pthreads.)) All of these can be run over unit tests.

How can you run a test suite under these tools? The normal method is some kind of hard-to-remember rune involving libtool, and either something about TESTS_ENVIRONMENT or some kind of LOG_COMPILER (but why would I want to compile my test logs?). For that reason, I’ve written an autoconf macro which abstracts it all a bit: AX_VALGRIND_CHECK. It adds a check-valgrind target to your makefile which handles running the `make check` tests under all four Valgrind tools of interest. Various ancillary variables (such as VALGRIND_SUPPRESSIONS_FILES or VALGRIND_FLAGS) allow the Valgrind options to be changed. `make check-valgrind` will return an error exit code if any problems are found, with the idea that the target can be used for automated testing and continuous integration.

To use it, either add a dependency on autoconf-archive to your project, or copy the ax_valgrind_check.m4 macro in-tree (and add it to EXTRA_DIST), then follow the instructions at the top of the file for adding it to configure.ac and tests/Makefile.am.

With this macro, I’ve tried to bring together existing makefile snippets which exist in various projects to produce one macro which can be reused everywhere. Of course, I’ve probably missed something – some feature or specific use case which someone has – so if anyone has any suggestions for improvement, please let me know or submit a patch to the autoconf-archive! For example, at the moment the macro requires automake’s parallel test harness, and does not support older versions of automake.

Thanks to my employer, Collabora, for enabling me to work on (hopefully useful) stuff like this!

A checklist for writing pkg-config files

tl;dr: Use AX_PKG_CHECK_MODULES to split public/private dependencies; use AC_CONFIG_FILES to magically include the API version in the .pc file name.

A few tips for creating a pkg-config file which you will never need to think about maintaining again — because one of the most common problems with pkg-config files is that their dependency lists are years out of date compared to the dependencies checked for in configure.ac. See lower down for some example automake snippets.

  • Include the project’s major API version ((Assuming this is the number which will change if backwards-incompatible API/ABI changes are made.)) in the pkg-config file name. e.g. libfoo-1.pc rather than libfoo.pc. This will allow parallel installation of two API-incompatible versions of the library if it becomes necessary in future.
  • Split private and public dependencies between Requires and Requires.private. This eliminates over-linking when dynamically linking against the project, since in that case the private dependencies are not needed. This is easily done using the AX_PKG_CHECK_MODULES macro (and perhaps using an upstream macro in future — see pkg-config bug #87154). A dependency is public when its symbols are exposed in public headers installed by your project; it is private otherwise.
  • Include useful ancillary variables, such as the paths to any utilities, directories or daemons which ship with the project. For example, glib-2.0.pc has variables giving the paths for its utilities: glib-genmarshal, gobject-query and glib-mkenums. libosinfo-1.0.pc has variables for its database directories. Ensure the variables use a variable form of ${prefix}, allowing it to be overridden when invoking pkg-config using pkg-config --define-variable=prefix=/some/other/prefix. This allows use of libraries installed in one (read only) prefix from binaries in another, while installing ancillary files (e.g. D-Bus service files) to the second prefix.
  • Substitute in the Name and Version using @PACKAGE_NAME@ and @PACKAGE_VERSION@ so they don’t fall out of sync.
  • Place the .pc.in template in the source code subdirectory for the library it’s for — so if your project produces multiple libraries (or might do in future), the .pc.in files don’t get mixed up at the top level.

Given all those suggestions, here’s a template libmy-project/my-project.pc.in file (updated to incorporate suggestions by Dan Nicholson):

prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@
includedir=@includedir@

my_project_utility=my-project-utility-binary-name
my_project_db_dir=@sysconfdir@/my-project/db

Name: @PACKAGE_NAME@
Description: Some brief but informative description
Version: @PACKAGE_VERSION@
Libs: -L${libdir} -lmy-project-@API_VERSION@
Cflags: -I${includedir}/my-project-@API_VERSION@
Requires: @AX_PACKAGE_REQUIRES@
Requires.private: @AX_PACKAGE_REQUIRES_PRIVATE@

And here’s a a few snippets from a template configure.ac:

# Release version
m4_define([package_version_major],[1])
m4_define([package_version_minor],[2])
m4_define([package_version_micro],[3])

# API version
m4_define([api_version],[1])

AC_INIT([my-project],
        [package_version_major.package_version_minor.package_version_micro],
        …)

# Dependencies
PKG_PROG_PKG_CONFIG
PKG_INSTALLDIR

glib_reqs=2.40
gio_reqs=2.42
gthread_reqs=2.40
nice_reqs=0.1.6

# The first list on each line is public; the second is private.
# The AX_PKG_CHECK_MODULES macro substitutes AX_PACKAGE_REQUIRES and
# AX_PACKAGE_REQUIRES_PRIVATE.
AX_PKG_CHECK_MODULES([GLIB],
                     [glib-2.0 >= $glib_reqs gio-2.0 >= $gio_reqs],
                     [gthread-2.0 >= $gthread_reqs])
AX_PKG_CHECK_MODULES([NICE],
                     [nice >= $nice_reqs],
                     [])

AC_SUBST([PACKAGE_VERSION_MAJOR],package_version_major)
AC_SUBST([PACKAGE_VERSION_MINOR],package_version_minor)
AC_SUBST([PACKAGE_VERSION_MICRO],package_version_micro)
AC_SUBST([API_VERSION],api_version)

# Output files
# Rename the template .pc file to include the API version on configure
AC_CONFIG_FILES([
libmy-project/my-project-$API_VERSION.pc:libmy-project/my-project.pc.in
…
],[],
[API_VERSION='$API_VERSION'])
AC_OUTPUT

And finally, the top-level Makefile.am:

# Install the pkg-config file; the directory is set using
# PKG_INSTALLDIR in configure.ac.
pkgconfig_DATA = libmy-project/my-project-$(API_VERSION).pc

Once that’s all built, you’ll end up with an installed my-project-1.pc file containing the following (assuming a prefix of /usr; note that by default autoconf substitutes in references to variables rather than the values themselves, so pkg-config can continue to be used with --define-variable to override the prefix):

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

my_project_utility=my-project-utility-binary-name
my_project_db_dir=${prefix}/etc/my-project/db

Name: my-project
Description: Some brief but informative description
Version: 1.2.3
Libs: -L${libdir} -lmy-project-1
Cflags: -I${includedir}/my-project-1
Requires: glib-2.0 >= 2.40 gio-2.0 >= 2.42 nice >= 0.1.6
Requires.private: gthread-2.0 >= 2.40

All code samples in this post are released into the public domain.

AM_INCLUDES doesn’t exist

tl;dr: Don’t use AM_INCLUDES in Makefiles. It does nothing. INCLUDES and AM_CPPFLAGS should be avoided in favour of my_program_name_CPPFLAGS.

Once upon a time, there was an automake variable called INCLUDES. It was used to pass flags to the preprocessor, such as -I flags to specify include paths. All was good.

Then the gentle leaders of automake decided that INCLUDES was not a very generic name, so they deprecated it in favour of the *_CPPFLAGS variables, like the target-specific my_program_name_CPPFLAGS. For those who wanted to specify preprocessor flags for all targets in their Makefile, the AM_CPPFLAGS variable was provided. Due to their kind and wise oversight, the transition between these variables (in the time of the coming of automake 1.12) was peaceful, and tranquillity continued to reign in the pastures of automakery.

And yet, some lands were troubled. Some of the citizens of automake had developed a confusion. They had taken the INCLUDES incantations of old, and mixed them with the AM_ prefix of new to create an AM_INCLUDES monster. ‘This is surely better’, they reasoned — ‘we have global preprocessor flags, but allow it to be overridden by the user due to our use of the AM_ prefix’. These citizens went ahead and used their AM_INCLUDES variable, but were confused when it failed to affect their automagic. They thought the runes were not in their favour, so copied its value into AM_CFLAGS as well, just in case.


These citizens are wrong. AM_INCLUDES does not exist. Do not use it. automake does not recognise it. You are wasting your time. It has no effect.

Use per-target CPPFLAGS (e.g. my_program_name_CPPFLAGS) to specify preprocessor flags. Use per-target CFLAGS to specify C-compiler-only flags. Do not use AM_CPPFLAGS unless you’re sure all flags in it are needed for all targets in the Makefile. Do not use INCLUDES unless you want a big deprecation warning from automake((warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS'))) (it is entirely equivalent to AM_CPPFLAGS). Do not use AM_INCLUDES at all.

For the curious, passing -Wall to automake will enable a bunch of warnings which can improve your Makefiles. You can do that using the AUTOMAKE_OPTIONS environment variable, or the AM_INIT_AUTOMAKE macro in configure.ac.


And the citizens of automake lived happily ever after.

Long live gnome-common? Macro deprecation

We’ve worked out the details, and have new recommendations for porting to autoconf-archive, including justifications and a migration guide. The information below is outdated.

gnome-common is shrinking, as we’ve decided to push as much of it as possible upstream. We have too many layers in our build systems, and adding an arbitrary dependency on gnome-common to pull in some macros once at configure time is not helpful — there are many cases where someone new has tried to build a module and failed with some weird autotools error about an undefined macro, purely because they didn’t have gnome-common installed.

So, for starters:

What does this mean for you, a module maintainer? Nothing, if you don’t want it to. gnome-common now contains copies of the autoconf-archive macros, and has compatibility wrappers for them.

In the long term, you should consider porting your build system to use the new, upstreamed macros. That means, for each macro:

  1. Downloading the macro to the m4/ directory in your project and adding it to git.
  2. Adding the macro to EXTRA_DIST in Makefile.am.
  3. Ensuring you have ACLOCAL_AMFLAGS = -I m4 ${ACLOCAL_FLAGS} in your top-level Makefile.am.
  4. Updating the macro invocation in configure.ac; just copy out the shim from gnome-common.m4 and tidy everything up.

Here’s an example change for GNOME_CODE_COVERAGE ? AX_CODE_COVERAGE in libgdata.

It seems from the comments that there’s more discussion to be had about the best way to implement this. So hold off on these changes for the moment!

This is the beginning of a (probably) long road to deprecating a lot of gnome-common. Macros like GNOME_COMPILE_WARNINGS and GNOME_COMMON_INIT are next in the firing line — they do nothing GNOME-specific, and should be part of a wider set of reusable free software build macros, like the autoconf-archive. gnome-common’s support for legacy documentation systems (DocBook, anyone?) is also getting deprecated next cycle.

Comments? Get in touch with me or David King (amigadave). This work is the (long overdue) result of a bit of discussion we had at the Berlin DX hackfest in May.

Notes on Vala and automake

Here are some brief notes about getting automake’s Vala support ((Note: That documentation seems to be a bit out of date regarding VALAFLAGS.)) to play nicely with you when writing a library, mostly for my own future benefit.

Firstly, use target-specific *_VALAFLAGS variables (rather than AM_VALAFLAGS) to specify the flags to pass to the valac compiler. Doing this enables a couple of dist and clean features in automake which wouldn’t otherwise be enabled. It’s also means you can specify multiple targets in a single Makefile.am, which is good for non-recursive automake.

When building a library, explicitly specify --vapi, --header, --internal-vapi, --internal-header and --gir (as appropriate) in the *_VALAFLAGS variable directly, rather than including them through another variable. automake looks for these flags in order to build dist and clean rules for the generated VAPI, C header and GIR files.

automake will automatically delete .stamp, VAPI and generated C header and source files on maintainer-clean, but not as part of any other clean target — this is because automake is designed to ensure generated C files are included in the distribution tarball. Failing to specify --header, --vapi (etc.) directly in *_VALAFLAGS will cause the appropriate maintainer-clean rules to not be generated.

automake knows nothing about .deps files (which accompany .vapi files), and will not generate dist rules for them; that must be done manually using:

vapidir = $(datadir)/vala/vapi
dist_vapi_DATA = my-library.deps

Source: lang_vala_finish_target() in the automake-1.13 script. Read it: it’s quite clear and informative.

Colourful GCC output

For a long time, I’ve used colorgcc as a way to highlight warnings and errors in GCC output. It’s worked wonderfully, but today Travis pointed out that GCC has grown an option to enable colourised output: -fdiagnostics-color. This will be available in GCC 4.9, but the patch seems to have been backported to Fedora 19 so that more people can join in the fun.

This means I can throw away colorgcc from my .jhbuildrc:

os.environ['CC'] = '/usr/bin/color-gcc'

and enable -fdiagnostics-color:

os.environ['CFLAGS'] += ' -fdiagnostics-color=auto'

Even better, adding the following snippet to configure.ac in a standard automake project will automatically enable colourised compiler output if the compiler (is GCC and) supports it:

AS_COMPILER_FLAG([-fdiagnostics-color=auto],
                 [AM_CFLAGS="$AM_CFLAGS -fdiagnostics-color=auto"])

(In actual fact, I used ERROR_CFLAGS for this in folks, to keep unrelated CFLAGS separated, but that’s an implementation detail of folks’ build system.)

The last GUADEC post?

GUADEC's over (and has been so for a while) and was great. The talk videos are up, and still I haven't blogged about the conference. Naughty me.

The conference was great: it was good to meet up with friends (old and new) again, get a bit of hacking done and hear lots of discussion about the future of GNOME. My thanks to the GNOME Foundation and my summer employer, Collabora, for sponsoring my accommodation and travel. Thank you to the local team for organising a brilliant GUADEC in a nice city, and here's to a successful GCDS 2011 in Berlin!

In other news, I've finally got fed up with forgetting to add new files in my project to either the project-wide header file or the documentation index, so I wrote some `make check` rules which will check for my braindeadness:

This one goes in the Makefile.am in the directory which builds your public header (such as gtk/gtk.h), and assumes a list of all the headers you're going to install is in public_headers, and that your main header's path is in main_header.

check-local: check-headers
check-headers:
	@any_missing=0; for header_file in $(public_headers); do \
		if test "x\(\)header_file>" $(main_header) >/dev/null; then \
				echo "$(main_header) doesn't appear to include \"\(\)any_missing"

This one goes in the Makefile.am in your gtk-doc directory (e.g. docs/reference), and only assumes the existence of DOC_MAIN_SGML_FILE, which needs to be defined for gtk-doc anyway.

check-local: check-xml-includes
check-xml-includes:
	@any_missing=0; find $(srcdir) -name "*.xml" | while read x; do \
		xml_file="\(\)xml_file" != "x$(DOC_MAIN_SGML_FILE)"; then \
			if ! grep "\"\(\)xml_file\""; \
				any_missing=1; \
			fi; \
		fi; \
	done; exit "$$any_missing"

I've filed bgo#627920 about adding the second rule to gtk-doc itself. I'm not sure the first rule is general enough to be put anywhere common.

Non-recursive automake

A while back, I was toying with the idea of adding gcov support to libgdata, to give metrics on the code coverage of the test suite. I played around with adding it, but my attempts were thwarted by the fact that I couldn't easily get a list of all C files which were to be compiled, in all source directories, to use in the top-level Makefile.

I recently found a little time to work on this some more and, as usually happens, it ballooned into something bigger, and I ended up redoing the entire build system for libgdata. It now uses non-recursive automake, which makes things a lot simpler in many respects. The diffstat was favourable ("21 files changed, 641 insertions(+), 833 deletions(-)"), and it has made the autogen/clean/build cycle about 20 seconds faster (on average, from some very rough tests). Accomplishing this was largely possible due to the (fairly) recent literature on the subject from Murray Cumming and Daniel Elstner, which proved rather useful; so many thanks to them.

As a result of all this, I now know that only just over 50% of libgdata's code was actually being exercised before by the test suite. I've now pushed this up to 60%, and uncovered a few buglets in the process. I dread to think what other problems lie in the remaining 40%; there's certainly lots of work left to be done.

Anyway, since all the snippets of automake magic I could find were subtly incompatible with non-recursive automake, I ended up with a slightly different one (based on gobject-introspection's gcov support). Hopefully it's useful to someone, though it could probably do with some tidying up:

if GCOV_ENABLED
gcov-report.txt: gcov-clean all check
	$(AM_V_GEN)(rm -f $@; \
	echo -e "Test coverage for libgdata:\n" >> $@; \
	total_covered=0; total_actual=0; \
	for file in $(filter %.c,$(gdata_libgdata_la_SOURCES)); do \
	  file2=\(\){file%/*}; \
	  gcov -o `find -newer \(\){file2/.c/.gcda}" -print0 | sed -e 's/\.gcda/\.o/'` \(\)file2.gcov; then \
	    actual=`grep -v '        -:' \(\)file2.gcov | wc -l`; \
	    covered=\(\)((total_covered + covered)); \
	    total_actual=\(\)file:\t\(\)actual\t(\(\)covered * 100) / \(\)total_actual\nCovered statements: \(\)(((\(\)total_actual))%" >> $@)

gcov: gcov-report.txt
	@cat gcov-report.txt

clean: gcov-clean
gcov-clean:
	@find . -name "*.gcda" -o -name "*.gcov" -delete

MAINTAINERCLEANFILES += gcov-report.txt

.PHONY: gcov gcov-clean gcov-report.txt
else
gcov:
	@echo "Need to reconfigure with --enable-gcov"
endif

It will output a code coverage report of the test suite (`make check`) when `make gcov` is called, and will also save it as "gcov-report.txt" in the root source directory.