Tag Archives: GitLab

Looking at project resource use and CI pipelines in GitLab

While at GUADEC I finished a small script which uses the GitLab API to estimate the resource use of a project on GitLab. It looks at the CI pipeline job durations and artifact storage for the project and its forks over a given period, and totals things.

You might want to run it on your project!

It gives output something like the following:

Between 2022-06-23 00:00:00+00:00 and 2022-07-23 00:00:00+00:00, GNOME/glib and its 20 forks used:

  • 4592 CI jobs, totalling 17125 minutes (duration minimum 0.0, median 2.3, maximum 65.0)
  • Total energy use: 32.54kWh
  • Total artifact storage: 4426 MB (minimum 0.0, median 0.2, maximum 20.9)

This is useful for giving a rough look at the CI resources used by a project, which could be useful for noticing low-hanging fruit for speeding things up or reducing resource waste.

What can I do with this information?

If total pipeline durations are long, either reduce the number of pipeline jobs or speed them up. Speeding them up almost always has no downsides. Reducing the number of jobs is a tradeoff between convenience of development and resource usage. Two ideas for reducing the number of jobs are to make some jobs manual-only, if they are very unlikely to find problems. Or run them on a schedule rather than on every commit, if it’s OK for them to catch problems up to a week after they’re introduced.

If total artifact storage use is high, store fewer artifacts, or expire them after a week (or so). They are likely not so useful after that point anyway.

If artifacts are being used to cache build dependencies, then consider moving those dependencies into a pre-built container image instead. It may be cached better between CI runners.

This script is rubbish, how do I improve it?

Merge requests welcome on https://gitlab.gnome.org/pwithnall/gitlab-stats, or perhaps you’d like to integrate it into cauldron.io so that the data could be visualised over time? The same query code should work for all GitLab instances, not just GNOME’s.

How does it work?

It queries the GitLab API in a few ways, and then applies a very simple model to the results.

It can take a while to run when querying for large projects or for periods of over a couple of weeks, as it needs to make a REST request for each CI job individually.

Easily speed up CI by reducing download size

Every time a CI pipeline runs on GitLab, it downloads the git repository for your project. Often, pipeline jobs are set up to make further downloads (of dependencies or subprojects), which are also run on each job.

Assuming that you’ve built a Docker image containing all your dependencies, to minimise how often they’re re-downloaded (you really should do this, it speeds up CI a lot), you can make further improvements by:

  1. Limiting the clone depth of your repository in the GitLab settings: Settings ? CI/CD, and change it to use a ‘git shallow clone’ of depth 1.
  2. Adding --branch, --no-tags and --depth 1 arguments to every git clone call you make during a CI job. Here’s an example for GLib.
  3. Adding depth=1 to your Meson .wrap files to achieve the same thing when (for example) meson subprojects download is called. See the same example merge request.

For GLib, the difference between git clone https://gitlab.gnome.org/GNOME/glib.git and git clone --depth 1 https://gitlab.gnome.org/GNOME/glib.git is 66MB (reducing from 74MB to 8MB), or a factor of 10. It won’t be as much for younger or smaller projects, but still worthwhile.

Generate NEWS entries for GitLab projects

I’ve just published a small script to help generating NEWS file entries for projects which use GitLab:

https://gitlab.gnome.org/pwithnall/gitlab-changelog

Use it like this:

  1. Generate a GitLab api token at https://gitlab.gnome.org/profile/personal_access_tokens.
  2. Run gitlab-changelog.py GNOME/glib 2.58.2.. -H https://gitlab.gnome.org/ -t ${generated_token}.
  3. Copy the output (below the dashed line) into your NEWS file entry. There are some blanks (indicated by TODO) in the entry which you need to fill in yourself.

The next time you run the script, you don’t need to pass the -H or -t options, as the configuration is saved in ~/.config/gitlab-changelog.ini.

I’ve used it on the last few GLib releases, which have seen a lot of issues fixed and MRs merged, and it sped up writing the NEWS file.

I haven’t tested it on other projects, or other GitLab instances, but it should work with them. Merge requests with improvements are very welcome.

GTK+ hackfest and FOSDEM: outcomes

Last Thursday and Friday was the GTK+ hackfest in Brussels. Matthias and Timm have blogged about GTK+ discussions already. This post is about the GLib side of things.

Firstly, we’re moving to Meson, but with no regressions from autotools. The plan is to target functional compatibility with autotools for 2.58, to keep both build systems in parallel for a release or two, and then drop autotools as soon as we can be satisfied there are no regressions for any of our features or supported platforms. I’d like to encourage distributions and developers to start trying to build GLib with Meson, seeing what breaks, and filing bugs.

Secondly, we’re migrating to gitlab, slowly. The first step is to migrate from cgit to gitlab, which will allow us to set up continuous integration for GLib. This will be a big win. The second step will be to migrate Bugzilla to gitlab. That’s going to take a bit longer, since there are issues with getting the data out of Bugzilla efficiently. All open bugs will be transferred, just like with the other gitlab transitions so far.

The maintainership status of GLib was also discussed. We are short on people power, and would appreciate assistance. If your project or company relies on GLib, please consider helping out with patch review or writing patches. We have three part-time maintainers who can provide guidance and help. We’re particularly interested in finding people to help maintain the platform ports of GLib, like Windows, OS X or BSD, more officially. Find us in #gtk+ on irc.gnome.org.

We also discussed a number of other, smaller features and issues, which might get handled for 2.58 depending on time. If you would like to work on any of them, please do! We can provide guidance and patch review in Bugzilla.

Thanks to Matthias and Emmanuele for organising the hackfest, Allison for turning up and imparting sage GLib wisdom, and Purism for kindly sponsoring dinner on Friday night.

Running GitLab CI on autotools projects

Inspired by the talk at FOSDEM, I’ve just enabled GitLab’s continuous integration (CI) for building make distcheck for Walbottle, and it was delightfully easy. The results are on Walbottle’s GitLab page.

Steps

  1. Create a ci branch to contain the mess you’ll make while iterating over the correct compile steps.
  2. Create and push a .gitlab-ci.yml file containing build rules similar to the following: 
    image: debian:unstable

before_script:
  - apt update -qq
  - apt install -y -qq build-essential autoconf automake pkg-config libtool m4 autoconf-archive gtk-doc-tools libxml2-utils gobject-introspection libgirepository1.0-dev libglib2.0-dev libjson-glib-dev

stages:
  - build

# FIXME: Re-enable valgrind once running the tests under it doesn’t take forever (it causes timeouts).
# Re-add valgrind to apt-install line above
build-distcheck:
  stage: build
  script:
    - mkdir build
    - cd build
    - ../autogen.sh --disable-valgrind
    - make V=1 VERBOSE=1
    - DISTCHECK_CONFIGURE_FLAGS=--disable-valgrind make distcheck V=1 VERBOSE=1

  # The files which are to be made available in GitLab
  artifacts:
    paths:
      - build/*
  3. Iterate a few times until you get all the dependencies right.
  4. Fix any problems you find (because this might well find problems with your dependency declaration in configure.ac, or other distcheck problems in your project).
  5. Merge ci to master and profit from CI results on every branch and master commit.

Looking at the .gitlab-ci.yml file

For information on the overall layout of the YAML file, and the phases available, you’re best off looking at the comprehensive GitLab documentation. Here are some notes about the autotools-and-C–specific bits of it:

  • The image is a Docker image; I picked a Debian one from the Docker hub.
  • Package installation seems to need to be done in the before_script phase, or the packages can’t be found (which is presumably a protection against rogue build systems).
  • I chose to build distcheck in my build rule because that runs the build, runs the tests, and tries various srcdir ? builddir configurations. You can add other build targets (like build-distcheck to try other build setups).
  • Pass V=1 VERBOSE=1 to get verbose build and test log output in your CI build logs, otherwise you will struggle to work out what is causing any failures.
  • Note that configure flags passed to ./configure are not automatically passed in again when ./configure is run as part of distcheck — so use DISTCHECK_CONFIGURE_FLAGS for that. Ideally, your project will be less fragile than mine, and hence not need any of this.
  • Export the whole build directory as an artifact on success, so you can look at any of the build objects, or the generated tarball, or documentation. You could limit this (for example, to just the tarball) if you’re sure you’ll never need the rest of it.

libnice is now mirrored on GitHub

libnice, everyone’s favourite ICE networking library, is now mirrored on GitHub (and GitLab), to make contributing to it easier — just submit a pull request. The canonical git repository is still on freedesktop.org.

Bug tracking is now on phabricator.freedesktop.org, which is where all new bugs should be reported. Existing open bugs have been migrated; existing closed bugs have not, and are archived on bugs.freedesktop.org.

We’ve been slowly working on some interesting changes to how GSockets are handled, so watch out for news on that.