Category Archives: GNOME

Posts about GNOME and programming for it.

Rate limiting of asynchronous calls in Vala

Following on from single-thread synchronisation in Vala, I just used a similar technique to easily implement rate limiting of concurrent calls to the same function in Vala. The use case for this was bug #705742, caused by attempting to write out many avatars to libfolks’ on-disk avatar cache simultaneously. So many files were opened simultaneously that the process’ file descriptor limit was hit and subsequent writes failed.

The principle for fixing this is to maintain a counter of the number of ongoing operations. If this hits a limit, operations which are started subsequently are added to a work queue and yielded. Any ongoing operation which finishes will pop a yielded operation off the work queue (if it’s non-empty) and resume that operation. This forms a FIFO queue which is guaranteed to progress due to each completed operation causing the next to resume (which will, in turn, cause the next to resume, etc., until the queue is empty).

A code example:

using GLib;

public class AvatarCache : Object
{
  private uint _n_ongoing_stores = 0;
  private Queue<DelegateWrapper> _pending_stores =
      new Queue<DelegateWrapper> ();

  /* Change this to change the cap on the number of concurrent
   * operations. */
  private const uint _max_n_ongoing_stores = 10;

  public async string store_avatar (string id, LoadableIcon avatar)
      throws GLib.Error
    {
      string retval = "";

      /* If the number of ongoing operations is at the limit,
       * queue this operation and yield. */
      if (this._n_ongoing_stores >
          AvatarCache._max_n_ongoing_stores)
        {
          /* Add to the pending queue. */
          var wrapper = new DelegateWrapper ();
          wrapper.cb = store_avatar.callback;
          this._pending_stores.push_tail ((owned) wrapper);
          yield;
        }

      /* Do the actual store operation. */
      try
        {
          this._n_ongoing_stores++;
          retval = yield this._store_avatar_unlimited (id, avatar);
        }
      finally
        {
          this._n_ongoing_stores--;

          /* If there is a store operation pending, resume it,
           * FIFO-style. */
          var wrapper = this._pending_stores.pop_head ();
          if (wrapper != null)
            {
              wrapper.cb ();
            }
        }

      return retval;
    }

  private async string _store_avatar_unlimited (string id,
                                                LoadableIcon avatar)
      throws GLib.Error
    {
      return /* the actual computation goes here */;
    }
}

/* See:
 * https://mail.gnome.org/archives/vala-list/2011-June/msg00005.html */
[Compact]
private class DelegateWrapper
{
  public SourceFunc cb;
}

This is all done using Vala’s asynchronous operation support, so runs in a single thread using the global default main context, and no locking or thread synchronisation is needed. If you were to use AvatarCache.store_avatar() from multiple threads, locking would have to be added and things would become more complex.

As with the single-thread synchronisation example from before, the key lines are: wrapper.cb = store_avatar.callback; yield, which stores the current function pointer and its closure (in Vala terminology, a delegate with target for the current method); and wrapper.cb (), which calls that function pointer with the stored closure (in Vala terminology, executes the delegate), effectively resuming computation from the yield statement.

So that’s it: a way to rate-limit concurrent method calls in Vala, ensuring they all block correctly (i.e. calls which are waiting for earlier ones to complete continue to block until they themselves are resumed and complete computation). By changing the scheduling function applied to the GQueue, priorities can be applied to queued calls if desired.

What’s the difference between int and size_t?

Passing variable-sized buffers around is a common task in C, especially in networking code. Ignoring GLib convenience API like GBytes, GInputVector and GOutputVector, a buffer is simply an offset and length describing a block of memory. This is just a void* and int, right? What’s the problem?

tl;dr: I’m suggesting to use (uint8_t*, size_t) to describe buffers in C code, or (guint8*, gsize) if you’re using GLib. This article is aimed at those who are still getting to grips with C idioms. See also: (const gchar*) vs. (gchar*) and other memory management stories.

There are two problems here: this effectively describes an array of elements, but void* doesn’t describe the width of each element; and (assuming each element is a byte) an int isn’t wide enough to index every element in memory on modern systems.

But void* could be assumed to refer to an array of bytes, and an int is probably big enough for any reasonable situation, right? Probably, but not quite: a 32-bit signed integer can address 2 GiB (assuming negative indices are ignored) and an unsigned integer can address 4 GiB1. Fine for network processing, but not when handling large files.

How is size_t better? It’s defined as being big enough to refer to every addressable byte of memory in the current computer system (caveat: this means it’s architecture-specific and not suitable for direct use in network protocols or file formats). Better yet, it’s unsigned, so negative indices aren’t wasted.

What about the offset of the buffer — the void*? Better to use a uint8_t*, I think. This has two advantages: it explicitly defines the width of each element to be one byte; and it’s distinct from char* (it’s unsigned rather than signed2), which makes it more obvious that the data being handled is not necessarily human-readable or nul-terminated.

Why is it important to define the width of each element? This is a requirement of C: it’s impossible to do pointer arithmetic without knowing the size of an element, and hence the C standard forbids arithmetic on void* pointers, since void doesn’t have a width (C11 standard, §6.2.5¶¶19,1). So if you use void* as your offset type, your code will end up casting to uint8_t* as soon as an offset into the buffer is needed anyway.

A note about GLib: guint8 and uint8_t are equivalent, as are gsize and size_t — so if you’re using GLib, you may want to use those type aliases instead.

For a more detailed explanation with some further arguments, see this nice article about size_t and ptrdiff_t by Karpov Andrey.


  1. int doesn’t have a guaranteed width (C11 standard, §6.2.5¶5), which is another reason to use size_t. However, on any relevant modern platform it is at least 32 bits. 

  2. Technically, it’s architecture-dependent whether char is signed or unsigned (C11 standard, §6.2.5¶15), and whether it’s actually eight bits wide (though in practice it is really always eight bits wide), which is another reason to use uint8_t

What is GMainContext?

GMainContext is at the core of almost every GLib application, yet it was only recently that I took the time to fully explore it — the details of it have always been a mystery. Doing some I/O work required me to look a little closer and try to get my head around the ins and outs of GMainContext, GMainLoop and GSources. Here I’ll try and write down a bit of what I’ve learned. If you want to skip to the conclusion, there’s a list of key points for using GMainContexts in libraries at the bottom of the post.

What is GMainContext? It’s a generalised implementation of an event loop, useful for implementing polled file I/O or event-based widget systems (i.e. GTK+). If you don’t know what poll() does, read about that first, since GMainContext can’t be properly understood without understanding polled I/O. A GMainContext has a set of GSources which are ‘attached’ to it, each of which can be thought of as an expected event with an associated callback function which will be invoked when that event is received; or equivalently as a set of file descriptors (FDs) to check. An event could be a timeout or data being received on a socket, for example. One iteration of the event loop will:

  1. Prepare sources, determining if any of them are ready to dispatch immediately.
  2. Poll the sources, blocking the current thread until an event is received for one of the sources.
  3. Check which of the sources received an event (several could have).
  4. Dispatch callbacks from those sources.

This is explained very well in the GLib documentation.

At its core, GMainContext is just a poll() loop, with the preparation, check and dispatch stages of the loop corresponding to the normal preamble and postamble in a typical poll() loop implementation, such as listing 1 from http://www.linux-mag.com/id/357/. Typically, some complexity is needed in non-trivial poll()-using applications to track the lists of FDs which are being polled. Additionally, GMainContext adds a lot of useful functionality which vanilla poll() doesn’t support. Most importantly, it adds thread safety.

GMainContext is completely thread safe, meaning that a GSource can be created in one thread and attached to a GMainContext running in another thread. A typical use for this might be to allow worker threads to control which sockets are being listened to by a GMainContext in a central I/O thread. Each GMainContext is ‘acquired’ by a thread for each iteration it’s put through. Other threads cannot iterate a GMainContext without acquiring it, which guarantees that a GSource and its FDs will only be polled by one thread at once (since each GSource is attached to at most one GMainContext). A GMainContext can be swapped between threads across iterations, but this is expensive.

Why use GMainContext instead of poll()? Mostly for convenience, as it takes all the grunt work out of dynamically managing the array of FDs to pass to poll(), especially when operating over multiple threads. This is done by encapsulating FDs in GSources, which decide whether those FDs should be passed to the poll() call on each ‘prepare’ stage of the main context iteration.

So if that’s GMainContext, what’s GMainLoop? Ignoring reference counting and locking gubbins, it is essentially just the following three lines of code (in g_main_loop_run()):

loop->is_running = TRUE;
while (loop->is_running)
	g_main_context_iteration (context, TRUE);

Plus a fourth line in g_main_loop_quit() which sets loop->is_running = FALSE and which will cause the loop to terminate once the current main context iteration ends. i.e. GMainLoop is a convenient, thread-safe way of running a GMainContext to process events until a desired exit condition is met, at which point you call g_main_loop_quit(). Typically, in a UI program, this will be the user clicking ‘exit’. In a socket handling program, this might be the final socket closing.

It is important not to confuse main contexts with main loops. Main contexts do the bulk of the work: preparing source lists, waiting for events, and dispatching callbacks. A main loop just iterates a context.

One of the important features of GMainContext is its support for ‘default’ contexts. There are two levels of default context: the thread-default, and the global-default. The global-default (accessed using g_main_context_default()) is what’s run by GTK+ when you call gtk_main(). It’s also used for timeouts (g_timeout_add()) and idle callbacks (g_idle_add()) — these won’t be dispatched unless the default context is running!

What are the thread-default contexts then? These are a later addition to GLib (since version 2.22), and are generally used for I/O operations which need to run and dispatch callbacks in a thread. By calling g_main_context_push_thread_default() before starting an I/O operation, the thread-default context has been set, and the I/O operation can add its sources to that context. The context can then be run in a new main loop in an I/O thread, causing the callbacks to be dispatched on that thread’s stack rather than on the stack of the thread running the global-default main context. This allows I/O operations to be run entirely in a separate thread without explicitly passing a specific GMainContext pointer around everywhere.

Conversely, by starting a long-running operation with a specific thread-default context set, your code can guarantee that the operation’s callbacks will be emitted in that context, even if the operation itself runs in a worker thread. This is the principle behind GTask: when a new GTask is created, it stores a reference to the current thread-default context, and dispatches its completion callback in that context, even if the task itself is run using g_task_run_in_thread().

For example, the code below will run a GTask which performs two writes in parallel from a thread. The callbacks for the writes will be dispatched in the worker thread, whereas the callback from the task as a whole will be dispatched in the interesting context.

typedef struct {
	GMainLoop *main_loop;
	guint n_remaining;
} WriteData;

/* This is always called in the same thread as thread_cb() because
 * it’s always dispatched in the @worker_context. */
static void
write_cb (GObject *source_object, GAsyncResult *result,
          gpointer user_data)
{
	WriteData *data = user_data;
	GOutputStream *stream = G_OUTPUT_STREAM (source_object);
	GError *error = NULL;
	gssize len;

	/* Finish the write. */
	len = g_output_stream_write_finish (stream, result, &error);
	if (error != NULL) {
		g_error ("Error: %s", error->message);
		g_error_free (error);
	}

	/* Check whether all parallel operations have finished. */
	write_data->n_remaining--;

	if (write_data->n_remaining == 0) {
		g_main_loop_quit (write_data->main_loop);
	}
}

/* This is called in a new thread. */
static void
thread_cb (GTask *task, gpointer source_object, gpointer task_data,
           GCancellable *cancellable)
{
	/* These streams come from somewhere else in the program: */
	GOutputStream *output_stream1, *output_stream;
	GMainContext *worker_context;
	GBytes *data;
	const guint8 *buf;
	gsize len;

	/* Set up a worker context for the writes’ callbacks. */
	worker_context = g_main_context_new ();
	g_main_context_push_thread_default (worker_context);

	/* Set up the writes. */
	write_data.n_remaining = 2;
	write_data.main_loop = g_main_loop_new (worker_context, FALSE);

	data = g_task_get_task_data (task);
	buf = g_bytes_get_data (data, &len);

	g_output_stream_write_async (output_stream1, buf, len,
	                             G_PRIORITY_DEFAULT, NULL, write_cb,
	                             &write_data);
	g_output_stream_write_async (output_stream2, buf, len,
	                             G_PRIORITY_DEFAULT, NULL, write_cb,
	                             &write_data);

	/* Run the main loop until both writes have finished. */
	g_main_loop_run (write_data.main_loop);
	g_task_return_boolean (task, TRUE);  /* ignore errors */

	g_main_loop_unref (write_data.main_loop);

	g_main_context_pop_thread_default (worker_context);
	g_main_context_unref (worker_context);
}

/* This can be called from any thread. Its @callback will always be
 * dispatched in the thread which currently owns
 * @interesting_context. */
void
parallel_writes_async (GBytes *data,
                       GMainContext *interesting_context,
                       GCancellable *cancellable,
                       GAsyncReadyCallback callback,
                       gpointer user_data)
{
	GTask *task;

	task = g_task_new (NULL, cancellable, callback, user_data);
	g_task_set_task_data (task, data,
	                      (GDestroyNotify) g_bytes_unref);
	g_task_run_in_thread (task, thread_cb);
	g_object_unref (task);
}

From the work I’ve been doing recently with GMainContext, here are a few rules of thumb for using main contexts in libraries which I’m going to follow in future:

  • Never iterate a context you don’t own, including the global-default or thread-default contexts, or you can cause the user’s sources to be dispatched unexpectedly and cause re-entrancy problems.
  • Always remove GSources from a main context once you’re done with them, especially if that context may have been exposed to the user (e.g. as a thread-default). Otherwise the user may keep a reference to the main context and continue iterating it after your code expects it to have been destroyed, potentially causing unexpected source dispatches in your code.
  • If your API is designed to be used in threads, or in a context-aware fashion, always document which context callbacks will be dispatched in. For example, “callbacks will always be dispatched in the context which is the thread-default at the time of the object’s construction”. Users of your API need to know this information.
  • Use g_main_context_invoke() to ensure callbacks are dispatched in the right context. It’s much easier than manually using g_idle_source_new().
  • Libraries should never use g_main_context_default() (or, equivalently, pass NULL to a GMainContext-typed parameter). Always store and explicitly use a specific GMainContext, even if that reduces to being some default context. This makes your code easier to split out into threads in future, if needed, without causing hard-to-debug problems with callbacks being invoked in the wrong context.
  • Always write things asynchronously internally (using the amazing GTask where appropriate), and keep synchronous wrappers to the very top level, where they can be implemented by calling g_main_context_iteration() on a specific GMainContext. Again, this makes future refactoring easier. You can see it in the above example: the thread uses g_output_stream_write_async() rather than g_output_stream_write().
  • Always match pushes and pops of the thread-default main context.

In a future post, I hope to explain in detail what’s in a GSource, and how to implement one, plus do some more in-depth comparison of poll() and GMainContext. Any feedback or corrections are gratefully received!

Notes on Vala and automake

Here are some brief notes about getting automake’s Vala support1 to play nicely with you when writing a library, mostly for my own future benefit.

Firstly, use target-specific *_VALAFLAGS variables (rather than AM_VALAFLAGS) to specify the flags to pass to the valac compiler. Doing this enables a couple of dist and clean features in automake which wouldn’t otherwise be enabled. It’s also means you can specify multiple targets in a single Makefile.am, which is good for non-recursive automake.

When building a library, explicitly specify --vapi, --header, --internal-vapi, --internal-header and --gir (as appropriate) in the *_VALAFLAGS variable directly, rather than including them through another variable. automake looks for these flags in order to build dist and clean rules for the generated VAPI, C header and GIR files.

automake will automatically delete .stamp, VAPI and generated C header and source files on maintainer-clean, but not as part of any other clean target — this is because automake is designed to ensure generated C files are included in the distribution tarball. Failing to specify --header, --vapi (etc.) directly in *_VALAFLAGS will cause the appropriate maintainer-clean rules to not be generated.

automake knows nothing about .deps files (which accompany .vapi files), and will not generate dist rules for them; that must be done manually using:

vapidir = $(datadir)/vala/vapi
dist_vapi_DATA = my-library.deps

Source: lang_vala_finish_target() in the automake-1.13 script. Read it: it’s quite clear and informative.


  1. Note: That documentation seems to be a bit out of date regarding VALAFLAGS

Clang plugin for GLib and GNOME

This past week, I’ve been working on a Clang plugin for GLib and GNOME, with the aim of improving static analysis of GLib-based C projects by integrating GIR metadata and assumptions about common GLib and GObject coding practices. The idea is that if running Clang for static analysis on a project, the plugin will suppress common GLib-based false positives and emit warnings for common problems which Clang wouldn’t previously have known about. Using it should be as simple as enabling scan-build, with the appropriate plugin, in your JHBuild configuration file.

I’m really excited about this project, which I’m working on as R&D at Collabora. It’s still early days, and the project has a huge scope, but so far I have a Clang plugin with two major features. It can:

  • Load GIR metadata1 and use it to add nonnull attributes based on the presence (or absence) of (allow-none) annotations.
  • Detect g_return_if_fail(), assert(), g_return_val_if_fail() (etc.) preconditions in function bodies and add nonnull attributes if the assertions are non-NULL checks (or are GObject type checks which also assert non-nullability).

Additionally, the plugin can use GIR metadata to add other attributes:

  • Add deprecated attributes based on Deprecated: gtk-doc tags.
  • Add warn_unused_result attributes to functions which have a (transfer container) or (transfer full) return type.
  • Add malloc attributes to functions which are marked as (constructor).
  • Constify the return types of (transfer none) functions which return a pointer type.

By modifying Clang’s abstract syntax tree (AST) for a file while it’s being scanned, the plugin can take advantage of the existing static analysis for NULL propagation to function calls with nonnull attributes, plus its analysis for variable constness, use of deprecated functions, etc.

So far, all the code is in two ‘annotaters’, which modify the AST but don’t emit any new warnings of their own. The next step is to implement some ‘checkers’, which examine (but don’t modify) the AST and emit warnings for common problems. Some of the problems checked for will be similar to those handled above (e.g. warning about a missing deprecated attribute if a function has a Deprecated: gtk-doc tag), but others can be completely new, such as checking that if a function has a GError parameter then it also has a g_return_if_fail(error == NULL || *error == NULL) precondition (or something along those lines).

I’ve got several big ideas for features to add once this initial work is complete, but the next thing to work on is making the plugin effortless to use (or, realistically, nobody will use it). Currently, the plugin is injected by a wrapper script around Clang, e.g.:

cd /path/to/glib/source
GNOME_CLANG_GIRS="GLib-2.0 Gio-2.0" gnome-clang -cc1 -analyze … /files/to/analyse

This injects the -load $PREFIX/lib64/clang/3.5/libclang-gnome.so -add-plugin gnome arguments which load and enable the plugin. By specifying GIR namespaces and versions in GNOME_CLANG_GIRS, those GIRs will be loaded and their metadata used by the plugin.

This can be configured in your ~/.jhbuildrc with:

static_analyzer = True
static_analyzer_template = 'scan-build --use-analyzer=$JHBUILD_PREFIX/bin/gnome-clang -v -o %(outputdir)s/%(module)s'
os.environ['GNOME_CLANG_GIRS'] = 'GLib-2.0 Gio-2.0 GnomeDesktop-3.0'

Please note that it’s alpha-quality code at the moment, and may introduce program bugs if used during compilation, so should only be used with Clang’s -analyze option. Furthermore, it still produces some false positives, so please be cautious about submitting bug reports to upstream projects ‘fixing’ problems detected by static analysis.

The code so far is here: http://cgit.collabora.com/git/gnome-clang.git/ and there’s a primitive website here: http://people.collabora.com/~pwith/gnome-clang/. If you’ve got any ideas, bug reports, patches, or just want to criticise my appalling C++, please e-mail me: philip {at} tecnocode.co(.)uk.


  1. Actually, it loads the metadata from GIR typelib files, rather than using the source code annotations themselves, so there is a degree of data loss incurred due to the limitations of the typelib binary format. 

libfolks: BlueZ backend for phone address book access

libfolks now has a backend for accessing address book data from Bluetooth-enabled phones, making use of BlueZ 5. This has been a long time in development, with various developers from Collabora contributing to it (thanks to Matthieu Bouron, Gustavo Padovan, Arun Raghavan and Jeremy Whiting!).

It works using the Bluetooth Phone Book Access Profile (PBAP), which allows a computer to download a phone’s address book as an array of vCards. libfolks will automatically do this for any phone which is paired with the computer (and which isn’t blocked), establishing a connection with any (paired, non-blocked) phone in range.

We’ve only been able to test this with a limited number of phones, so further real-world testing and feedback would be greatly appreciated. To do so, grab the latest version of libfolks from git (https://git.gnome.org/browse/folks/log/), build it and run folks-inspect (making sure that the ‘bluez’ backend is not disabled in ~/.local/share/folks/backends.ini, if you’ve created and customised that file).

As is the design of libfolks, this is not intended as a solution for synchronising contacts between your phone and your computer, but it does provide a handy way of accessing contacts on your phone from GNOME Shell or Contacts, or from other consumers of libfolks data.

Note that the backend only supports BlueZ 5 and explicitly doesn’t support BlueZ 4.

Statically linking using pkg-config and JHBuild

tl;dr: Export PKG_CONFIG="pkg-config --static" or add it to your .jhbuildrc file.

Statically linking libraries which use libtool (specifically, the LT_INIT autoconf macro) is easy: you pass --enable-static --disable-shared to configure, and out pop static .a libraries instead of dynamic .so libraries.

However, things get more complex once you start to consider dependencies of your library. For example, if I’m statically linking libfoo against libbar, and libbar uses libwidgit internally (i.e. consumes the libwidgit symbols but does not expose any of them as part of libbar’s public interfaces), libfoo needs to know about libwidgit and needs to list it in the libfoo linker command. This is because UNIX archives (statically linked .a files) do not list their dependencies (whereas dynamically linked .so files do (on sensible UNIX platforms, including Linux)). Effectively the transitive closure of link dependencies needs to be resolved at link time, rather than at run time.

How can we cope with this when configuring libfoo? By using pkg-config --static instead of normal pkg-config. This can be achieved by setting the PKG_CONFIG environment variable to pkg-config --static or by adding the following lines to .jhbuildrc (which also set the libtool options mentioned above):

autogenargs = '--disable-shared --enable-static'
os.environ['PKG_CONFIG'] = 'pkg-config --static'

pkg-config’s --static option changes the program’s output to include the Libs.private and (recursively the) Requires.private variables. For example:

$ pkg-config --libs gobject-2.0
-L/opt/gnome3/build/lib64 -lgobject-2.0 -lglib-2.0
$
$ pkg-config --static --libs gobject-2.0
-pthread -L/opt/gnome3/build/lib64 -lgobject-2.0 -lffi -lglib-2.0

Here, libffi and pthread are both private dependencies of GObject: they’re used internally in GObject, but don’t appear in its external interface. When statically linking against GObject, though, their symbols must be resolved and hence they must appear in the linker command.

For more about the differences between pkg-config’s Libs and Libs.private variables, see the excellent guide to pkg-config. For more information about static linking with pkg-config, see the Autotools Mythbuster.

(There are probably errors or omissions in this post, since I’m nowhere near an expert on linking. Please comment with any corrections! Also, this is not meant to suggest that statically linking libraries is a good idea in general — there are only a few situations where it’s sensible. But for those situations, hopefully this post makes things a little easier.)

Colourful GCC output

For a long time, I’ve used colorgcc as a way to highlight warnings and errors in GCC output. It’s worked wonderfully, but today Travis pointed out that GCC has grown an option to enable colourised output: -fdiagnostics-color. This will be available in GCC 4.9, but the patch seems to have been backported to Fedora 19 so that more people can join in the fun.

This means I can throw away colorgcc from my .jhbuildrc:

os.environ['CC'] = '/usr/bin/color-gcc'

and enable -fdiagnostics-color:

os.environ['CFLAGS'] += ' -fdiagnostics-color=auto'

Even better, adding the following snippet to configure.ac in a standard automake project will automatically enable colourised compiler output if the compiler (is GCC and) supports it:

AS_COMPILER_FLAG([-fdiagnostics-color=auto],
                 [AM_CFLAGS="$AM_CFLAGS -fdiagnostics-color=auto"])

(In actual fact, I used ERROR_CFLAGS for this in folks, to keep unrelated CFLAGS separated, but that’s an implementation detail of folks’ build system.)

Writing a GNOME thumbnailer

How does GNOME generate thumbnails for files? It uses a collection of programs called thumbnailers, each one generating thumbnails for a specific set of content-types of files. For example, totem-video-thumbnailer generates thumbnails for video files using GStreamer; evince-thumbnailer generates thumbnails for PDFs and other document files.

To generate a thumbnail, an appropriate thumbnailer program is selected then executed, passing it the path of the file to thumbnail, plus a path to write the thumbnail image to. If thumbnailing succeeds, the thumbnailer should have written the image to disk before terminating; but if thumbnailing fails, no image should be written, and the thumbnailer should return a non-zero exit status.

Thumbnailers are chosen by examining a series of .thumbnailer files in $PREFIX/share/thumbnailers. Each is in a simple key-file format:

[Thumbnailer Entry]
TryExec=evince-thumbnailer
Exec=evince-thumbnailer -s %s %u %o
MimeType=application/pdf;application/x-bzpdf;application/x-gzpdf;

The .thumbnailer format supports three keys:

  • TryExec: Optional. The name of the the thumbnailer program in the current $PATH. This allows the calling code to quickly check whether the thumbnailer exists before trying to execute it.
  • Exec: Required. The command to execute the thumbnailer. It supports a few different parameters which are replaced before calling the thumbnailer: %u is the URI of the file being thumbnailed; %i is its filename; %o is the filename of the image file to be written to; %s is the maximum desired size of the thumbnail image (in pixels); and %% is a literal percent character.
  • MimeType: Required. A semicolon-separated list of MIME types which the thumbnailer supports generating thumbnails for.

So in the evince-thumbnailer file above, the command passes the requested thumbnail size, then the input file’s URI, then the path for the output image file to evince-thumbnailer.

The code to examine and call a thumbnailer is in gnome-common’s GnomeDesktopThumbnailFactory class, which handles looking up the right thumbnailer script, building and executing the command for it, and loading the resulting thumbnail image into a GdkPixbuf.

Thumbnail caching is supported by GnomeDesktopThumbnailFactory. When calling a thumbnailer, the path passed for the output image file is in ~/.cache/thumbnails/$SIZE/. The cached image file is given a (probably) unique filename, generated by hashing the original file’s URI, so the thumbnail can be looked up in future. gnome-desktop supports two sizes of thumbnails: ‘normal’ and ‘large’. Normal thumbnails are up to 128×128 pixels, whereas large thumbnails are up to 256×256 pixels. Thumbnails which are larger than this are scaled down before being cached, and non-square thumbnails are scaled so their largest dimension is at most 128 or 256 pixels.

gnome-desktop also handles failed thumbnails. If a thumbnailer can’t generate a thumbnail for a file (e.g. because the file is corrupt or because the right video codecs aren’t available), it returns a non-zero exit status. gnome-desktop then writes an entry to ~/.cache/thumbnails/fail/gnome-thumbnail-factory/ which is named after the hash of the input file URI (just like a successful cached thumbnail). For future queries for thumbnails for that file, GnomeDesktopThumbnailFactory can immediately return an error after looking up the fail entry.

If a file changes content, GnomeDesktopThumbnailFactory knows to generate a new thumbnail because each cached image has associated metadata (stored as PNG tEXt keys) storing the full URI of the thumbnailed file (to check for hash collisions) and its last modification time at the point of thumbnailing. If the stored modification time doesn’t match the file’s current one, a new thumbnail is generated.

I don’t care. Show me the code.

As an example of this, I present gnome-directory-thumbnailer, which generates thumbnails for directories (which is possible with a few small fixes to gnome-desktop). I wrote this as part of a project for my new employer, Collabora. It generates a thumbnail for a directory by heuristically choosing its ‘most interesting’ file (or subdirectory) and taking the thumbnail from that. It works fairly well for generic directories, but there’s great scope for improving the heuristics to cater for directories which match different templates (e.g. a directory of audio files for an album; a directory containing an automake project; a mounted DVD directory; etc.).

Nautilus showing some album directories.

Nautilus showing some album directories.

The code is in gnome-directory-thumbnailer on git.gnome.org and it’s in the process of getting a gnome-directory-thumbnailer product on bugzilla.gnome.org, so please file bugs there. Note that you’ll need the latest release (3.10.1) of gnome-desktop for it to work correctly in Nautilus.

If you want to write your own thumbnailer, a good place to start is with the gnome-thumbnailer-skeleton by Bastien Nocera. It provides the generic wrapper code for command line parsing and output handling which would otherwise get rewritten for each new thumbnailer.

There are actually other ways of generating thumbnails, such as the Tumbler thumbnailing system which is accessed via D-Bus and implements the proposed thumbnailer specification which is used by Rygel. As far as I’m aware, Nautilus doesn’t use D-Bus thumbnailers.

Using libgee from C to access libfolks

If you’re interfacing with Vala code from C, it’s possible that you’ll have to make calls to libgee, a Vala data structure library. libgee is written in Vala, but its C API is perfectly usable once you get your head around the idioms.

A common idiom is that of the GeeIterator, which is the recommended way to iterate through sets, maps and lists (which all implement the GeeIterable interface). For a loop, you create a new GeeIterator, then call its get and next methods to retrieve the object for the current iteration and advance to the next iteration, respectively.

For example, to list all the phone numbers in a FolksIndividual:

GeeSet *phone_numbers;
GeeIterator *iter;
FolksPhoneDetails *details;

details = FOLKS_PHONE_DETAILS (my_individual);
phone_numbers = folks_phone_details_get_phone_numbers (details);
iter = gee_iterable_iterator (GEE_ITERABLE (phone_numbers));

while (gee_iterator_next (iter)) {
	FolksPhoneFieldDetails *details;
	gchar *number;

	/* Get the current iteration’s element.
	 * Note that ownership is transferred. */
	details = gee_iterator_get (iter);
	number = folks_phone_field_details_get_normalised (details);

	g_message ("Phone number: %s", number);

	g_free (number);
	g_object_unref (details);  /* important! */
}

g_object_unref (iter);  /* important! */

One key thing to note here is that the return values from gee_iterator_get() and gee_iterable_iterator() are transfer-full and need to be unreffed. Failing to unref them is a common mistake when using libgee from C, and results in leaked objects — and was the cause of several memory leaks recently fixed in Empathy. When using libgee from Vala this isn’t a problem, since the memory management is handled automatically.

This has been a public service announcement.