Tag Archives: GNOME

How widely is the GNOME stack used?

After a couple of discussions at the DX hackfest about cross-platform-ness and deployment of GLib, I started wondering: we often talk about how GNOME developers work at all levels of the stack, but how much of that actually qualifies as ‘core’ work which is used in web servers, in cross-platform desktop software1, or commonly in embedded systems, and which is security critical?

On desktop systems (taking my Fedora 19 installation as representative), we can compare GLib usage to other packages, taking GLib as the lowest layer of the GNOME stack:

Package Reverse dependencies Recursive reverse dependencies
glib2 4001
qt 2003
libcurl 628
boost-system 375
gnutls 345
openssl 101 1022

(Found with repoquery --whatrequires [--recursive] [package name] | wc -l. Some values omitted because they took too long to query, so can be assumed to be close to the entire universe of packages.)

Obviously GLib is depended on by many more packages here than OpenSSL, which is definitely a core piece of software. However, those packages may not be widely used or good attack targets. Higher layers of the GNOME stack see widespread use too:

Package Reverse dependencies
cairo 2348
gdk-pixbuf2 2301
pango 2294
gtk3 801
libsoup 280
gstreamer 193
librsvg2 155
gstreamer1 136
clutter 90

(Found with repoquery --whatrequires [package name] | wc -l.)

Widely-used cross-platform software which interfaces with servers2 includes PuTTY and Wireshark, both of which use GTK+3. However, other major cross-platform FOSS projects such as Firefox and LibreOffice, which are arguably more ‘core’, only use GNOME libraries on Linux.

How about on embedded systems? It’s hard to produce exact numbers here, since as far as I know there’s no recent survey of open source software use on embedded products. However, some examples:

So there are some sample points which suggest moderately widespread usage of GNOME technologies in open-source-oriented embedded systems. For more proprietary embedded systems it’s hard to tell. If they use Qt for their UI, they may well use GLib’s main loop implementation. I tried sampling GPL firmware releases from gpl-devices.org and gpl.nas-central.org, but both are quite out of date. There seem to be a few releases there which use GLib, and a lot which don’t (though in many cases they’re just kernel releases).

Servers are probably the largest attack surface for core infrastructure. How do GNOME technologies fare there? On my CentOS server:

  • GLib is used by the popular web server lighttpd (via gamin),
  • the widespread logging daemon syslog-ng,
  • all MySQL load balancing via mysql-proxy, and
  • also by QEMU.
  • VMware ESXi seems to use GLib (both versions 2.22 and 2.24!), as determined from looking at its licencing file. This is quite significant — ESXi is used much more widely than QEMU/KVM.
  • The Amanda backup server uses GLib extensively,
  • as do the clustering solutions Heartbeat and Pacemaker.

I can’t find much evidence of other GNOME libraries in use, though, since there isn’t much call for them in a non-graphical server environment. That said, there has been heavy development of server-grade features in the NetworkManager stack, which will apparently be in RHEL 7 (thanks Jon).

So it looks like GLib, if not other GNOME technologies, is a plausible candidate for being core infrastructure. Why haven’t other GNOME libraries seen more widespread usage? Possibly they have, and it’s too hard to measure. Or perhaps they fulfill a niche which is too small. Most server technology was written before GNOME came along and its libraries matured, so any functionality which could be provided by them has already been implemented in other ways. Embedded systems seem to shun desktop libraries for being too big and slow. The cross-platform support in most GNOME libraries is poorly maintained or non-existent, limiting them to use on UNIX systems only, and not the large OS X or Windows markets. At the really low levels, though, there’s solid evidence that GNOME has produced core infrastructure in the form of GLib.


  1. As much as 2014 is the year of Linux on the desktop, Windows and Mac still have a much larger market share. 

  2. And hence is security critical. 

  3. Though Wireshark is switching to Qt. 

What is GMainContext?

GMainContext is at the core of almost every GLib application, yet it was only recently that I took the time to fully explore it — the details of it have always been a mystery. Doing some I/O work required me to look a little closer and try to get my head around the ins and outs of GMainContext, GMainLoop and GSources. Here I’ll try and write down a bit of what I’ve learned. If you want to skip to the conclusion, there’s a list of key points for using GMainContexts in libraries at the bottom of the post.

What is GMainContext? It’s a generalised implementation of an event loop, useful for implementing polled file I/O or event-based widget systems (i.e. GTK+). If you don’t know what poll() does, read about that first, since GMainContext can’t be properly understood without understanding polled I/O. A GMainContext has a set of GSources which are ‘attached’ to it, each of which can be thought of as an expected event with an associated callback function which will be invoked when that event is received; or equivalently as a set of file descriptors (FDs) to check. An event could be a timeout or data being received on a socket, for example. One iteration of the event loop will:

  1. Prepare sources, determining if any of them are ready to dispatch immediately.
  2. Poll the sources, blocking the current thread until an event is received for one of the sources.
  3. Check which of the sources received an event (several could have).
  4. Dispatch callbacks from those sources.

This is explained very well in the GLib documentation.

At its core, GMainContext is just a poll() loop, with the preparation, check and dispatch stages of the loop corresponding to the normal preamble and postamble in a typical poll() loop implementation, such as listing 1 from http://www.linux-mag.com/id/357/. Typically, some complexity is needed in non-trivial poll()-using applications to track the lists of FDs which are being polled. Additionally, GMainContext adds a lot of useful functionality which vanilla poll() doesn’t support. Most importantly, it adds thread safety.

GMainContext is completely thread safe, meaning that a GSource can be created in one thread and attached to a GMainContext running in another thread. A typical use for this might be to allow worker threads to control which sockets are being listened to by a GMainContext in a central I/O thread. Each GMainContext is ‘acquired’ by a thread for each iteration it’s put through. Other threads cannot iterate a GMainContext without acquiring it, which guarantees that a GSource and its FDs will only be polled by one thread at once (since each GSource is attached to at most one GMainContext). A GMainContext can be swapped between threads across iterations, but this is expensive.

Why use GMainContext instead of poll()? Mostly for convenience, as it takes all the grunt work out of dynamically managing the array of FDs to pass to poll(), especially when operating over multiple threads. This is done by encapsulating FDs in GSources, which decide whether those FDs should be passed to the poll() call on each ‘prepare’ stage of the main context iteration.

So if that’s GMainContext, what’s GMainLoop? Ignoring reference counting and locking gubbins, it is essentially just the following three lines of code (in g_main_loop_run()):

loop->is_running = TRUE;
while (loop->is_running)
	g_main_context_iteration (context, TRUE);

Plus a fourth line in g_main_loop_quit() which sets loop->is_running = FALSE and which will cause the loop to terminate once the current main context iteration ends. i.e. GMainLoop is a convenient, thread-safe way of running a GMainContext to process events until a desired exit condition is met, at which point you call g_main_loop_quit(). Typically, in a UI program, this will be the user clicking ‘exit’. In a socket handling program, this might be the final socket closing.

It is important not to confuse main contexts with main loops. Main contexts do the bulk of the work: preparing source lists, waiting for events, and dispatching callbacks. A main loop just iterates a context.

One of the important features of GMainContext is its support for ‘default’ contexts. There are two levels of default context: the thread-default, and the global-default. The global-default (accessed using g_main_context_default()) is what’s run by GTK+ when you call gtk_main(). It’s also used for timeouts (g_timeout_add()) and idle callbacks (g_idle_add()) — these won’t be dispatched unless the default context is running!

What are the thread-default contexts then? These are a later addition to GLib (since version 2.22), and are generally used for I/O operations which need to run and dispatch callbacks in a thread. By calling g_main_context_push_thread_default() before starting an I/O operation, the thread-default context has been set, and the I/O operation can add its sources to that context. The context can then be run in a new main loop in an I/O thread, causing the callbacks to be dispatched on that thread’s stack rather than on the stack of the thread running the global-default main context. This allows I/O operations to be run entirely in a separate thread without explicitly passing a specific GMainContext pointer around everywhere.

Conversely, by starting a long-running operation with a specific thread-default context set, your code can guarantee that the operation’s callbacks will be emitted in that context, even if the operation itself runs in a worker thread. This is the principle behind GTask: when a new GTask is created, it stores a reference to the current thread-default context, and dispatches its completion callback in that context, even if the task itself is run using g_task_run_in_thread().

For example, the code below will run a GTask which performs two writes in parallel from a thread. The callbacks for the writes will be dispatched in the worker thread, whereas the callback from the task as a whole will be dispatched in the interesting context.

typedef struct {
	GMainLoop *main_loop;
	guint n_remaining;
} WriteData;

/* This is always called in the same thread as thread_cb() because
 * it’s always dispatched in the @worker_context. */
static void
write_cb (GObject *source_object, GAsyncResult *result,
          gpointer user_data)
{
	WriteData *data = user_data;
	GOutputStream *stream = G_OUTPUT_STREAM (source_object);
	GError *error = NULL;
	gssize len;

	/* Finish the write. */
	len = g_output_stream_write_finish (stream, result, &error);
	if (error != NULL) {
		g_error ("Error: %s", error->message);
		g_error_free (error);
	}

	/* Check whether all parallel operations have finished. */
	write_data->n_remaining--;

	if (write_data->n_remaining == 0) {
		g_main_loop_quit (write_data->main_loop);
	}
}

/* This is called in a new thread. */
static void
thread_cb (GTask *task, gpointer source_object, gpointer task_data,
           GCancellable *cancellable)
{
	/* These streams come from somewhere else in the program: */
	GOutputStream *output_stream1, *output_stream;
	GMainContext *worker_context;
	GBytes *data;
	const guint8 *buf;
	gsize len;

	/* Set up a worker context for the writes’ callbacks. */
	worker_context = g_main_context_new ();
	g_main_context_push_thread_default (worker_context);

	/* Set up the writes. */
	write_data.n_remaining = 2;
	write_data.main_loop = g_main_loop_new (worker_context, FALSE);

	data = g_task_get_task_data (task);
	buf = g_bytes_get_data (data, &len);

	g_output_stream_write_async (output_stream1, buf, len,
	                             G_PRIORITY_DEFAULT, NULL, write_cb,
	                             &write_data);
	g_output_stream_write_async (output_stream2, buf, len,
	                             G_PRIORITY_DEFAULT, NULL, write_cb,
	                             &write_data);

	/* Run the main loop until both writes have finished. */
	g_main_loop_run (write_data.main_loop);
	g_task_return_boolean (task, TRUE);  /* ignore errors */

	g_main_loop_unref (write_data.main_loop);

	g_main_context_pop_thread_default (worker_context);
	g_main_context_unref (worker_context);
}

/* This can be called from any thread. Its @callback will always be
 * dispatched in the thread which currently owns
 * @interesting_context. */
void
parallel_writes_async (GBytes *data,
                       GMainContext *interesting_context,
                       GCancellable *cancellable,
                       GAsyncReadyCallback callback,
                       gpointer user_data)
{
	GTask *task;

	task = g_task_new (NULL, cancellable, callback, user_data);
	g_task_set_task_data (task, data,
	                      (GDestroyNotify) g_bytes_unref);
	g_task_run_in_thread (task, thread_cb);
	g_object_unref (task);
}

From the work I’ve been doing recently with GMainContext, here are a few rules of thumb for using main contexts in libraries which I’m going to follow in future:

  • Never iterate a context you don’t own, including the global-default or thread-default contexts, or you can cause the user’s sources to be dispatched unexpectedly and cause re-entrancy problems.
  • Always remove GSources from a main context once you’re done with them, especially if that context may have been exposed to the user (e.g. as a thread-default). Otherwise the user may keep a reference to the main context and continue iterating it after your code expects it to have been destroyed, potentially causing unexpected source dispatches in your code.
  • If your API is designed to be used in threads, or in a context-aware fashion, always document which context callbacks will be dispatched in. For example, “callbacks will always be dispatched in the context which is the thread-default at the time of the object’s construction”. Users of your API need to know this information.
  • Use g_main_context_invoke() to ensure callbacks are dispatched in the right context. It’s much easier than manually using g_idle_source_new().
  • Libraries should never use g_main_context_default() (or, equivalently, pass NULL to a GMainContext-typed parameter). Always store and explicitly use a specific GMainContext, even if that reduces to being some default context. This makes your code easier to split out into threads in future, if needed, without causing hard-to-debug problems with callbacks being invoked in the wrong context.
  • Always write things asynchronously internally (using the amazing GTask where appropriate), and keep synchronous wrappers to the very top level, where they can be implemented by calling g_main_context_iteration() on a specific GMainContext. Again, this makes future refactoring easier. You can see it in the above example: the thread uses g_output_stream_write_async() rather than g_output_stream_write().
  • Always match pushes and pops of the thread-default main context.

In a future post, I hope to explain in detail what’s in a GSource, and how to implement one, plus do some more in-depth comparison of poll() and GMainContext. Any feedback or corrections are gratefully received!

Back from the Desktop Summit

The Desktop Summit's over for another year. Berlin was great, the parties were good (thanks to Intel and Collabora!), and it was good to see everyone again — and also meet some new people. My thanks, as always, go to the GNOME Foundation for arranging and sponsoring my accommodation.

Unfortunately, the Summit wasn't all parties and sightseeing. We managed to belatedly push out folks 0.6, with Travis putting in far too much of his holiday time to tar it all up. Raúl somehow got away with sneaking off and doing interesting things with the Sugar people instead.

My libgdata BoF went down well, with a reasonable amount of interest in pushing libgdata's support for Google Documents forward, for use by both GNOME Documents and Zeitgeist. I'll get round to tidying up and pushing forward with the libgdata 0.12 roadmap in the near future.

I'm looking forward to being the lucky recipient of a load of GSoC patches to review for using GXml in libgdata. I love reviewing patches.

A Coruña next year!

Avatars for Google Contacts in Evolution

Following on from group/category support, I've just merged support for setting and retrieving avatars/photos on Google Contacts from Evolution. There are still a few rough edges, but hopefully I'll get time to tidy those up before GNOME 3.2.

“I'm going to Desktop Summit 2011 in Berlin” bannerI'll be arriving on the 5th, spending a day hurriedly seeing absolutely everything there is to see in Berlin, then diving into the conference on Saturday. I'm leaving on the 12th.

For anybody interested in contacts, meta-contacts, individuals, personas and inventing words to make your software sound more impressive, there's a folks BoF happening at some point on Wednesday (not Thursday as the schedule says, since Travis has to leave beforehand). More details will follow, hopefully, in the libfolks talk on Saturday, 11:20–11:50 in Room 2002.

End of the hackfest

The IM, Contacts & Social hackfest is over and people are variously heading back to their respective corners of the world. A lot of good discussion was had, and I think everyone's got a clearer vision of where we need to be with IMs and contacts and what needs to be done to get there.

Travis, Raúl and I clarified a lot of the changes needed to libfolks for this shiny new future, and while I didn't get as much hacking done as I would've liked, there's plenty of time for that later.

Thanks to Collabora for providing space for the hackfest and sponsoring plenty of food, the GNOME Foundation for enabling the hackfest to happen, and Nyan cat for entertaining Bastien.

IM, Contacts & Social hackfest

I'll be heading along to the IM, Contacts & Social hackfest next week at Collabora's offices. There, a plan for world domination by libfolks will be forged, along with plotting around GNOME's new SSO overlord system and work on the much-awaited GNOME Contacts.

Should be fun!

Relatedly, I've just released libgdata 0.9.0, which has sprouted support for OAuth 1.0 — so hopefully some GNOME Online Accounts goodness will soon make it into Evolution's Google Contacts and Calendar backends.

Unicode in Python

Now that exams are finally over, I can spend more time on GNOMEy things. One problem which has been sitting on my to-do list for a while is that of translatable Unicode strings in Python. It appears that my patch in bug #591496 to get Hamster to use Unicode em-dashes inadvertently broke translation of the strings. Whoops.

It turns out that in order for gettext to properly match and translate a C-locale string which contains Unicode characters, the encoding of the Python file must be specified using a coding: line at the top of the file, and the string in question must be a Unicode object. For example:

# -*- coding: utf-8 -*-
…
import gettext
gettext.textdomain('myapp')
…
my_translated_string = gettext.gettext(u'My Unicode string…')
…

I don't think this is too common a problem, and I've checked that it doesn't affect any of the other Python modules I've fiddled with, but hopefully this will be useful to someone. As far as I understand it, all translatable strings in Python modules should be u'Unicode objects rather than normal strings' anyway, ideally, but don't take my word on it because my Python-fu is weak.

Unicode in GNOME

This is something I’ve been meaning to write about for a while and, I must admit, something I should have written about before I started pushing through changes in GNOME applications. I’m talking about the use of Unicode in GNOME: the use of the proper ellipsis character (“…”), proper en- and em-dashes (“–” and “—”, respectively) and fancy quotation marks.

This is something which has been brought up before, so I’ll try not to reignite the old arguments, and instead concentrate on the unresolved issues. Here are the main points:

  • Proper Unicode characters look nicer than the ASCII versions which substitute for them. The ellipsis is correctly spaced (if one were to use full stops instead, they should technically have non-breaking spaces between them), and the quotation marks are pleasantly curved. This looks nicer, to my eye at least. The difference between en- and em-dashes and the ASCII hyphens used to simulate them is considerable.
  • They’re harder to type on a conventional keyboard, though are easily accessible through the use of the compose key.
  • There are questions about the level of font support for such characters. On my Fedora 11 system, all the fonts except one (“PakTypeTehreer”) have the expected characters (ellipsis, dashes and quotation marks) at the right codepoints, although many of the glyphs are ugly and unloved (e.g. in Hershey and Khmer). DejaVu and Bitstream have excellent support for these characters. There is a suggestion that Pango should be extended to support decomposing the Unicode characters into their ASCII equivalents if a font doesn't support them.
  • There was confusion over what exactly was allowed in source code, and whether UTF-8 characters were allowed in C-locale strings (regardless of their representation in source code). It was decided that they were, but that the most portable way to represent them in C was to use octal slash escaping (e.g. “\342\200\246” instead of “…”). We’ve had Unicode characters in source code since GNOME 2.22, and (apparently) there have been no bug reports on the matter, but there was no conclusive answer about how embedded C compilers (and other, less well-known compilers) cope with such things.

Obviously, I’m thoroughly in the pro-Unicode camp. I believe it would make our desktop look more professional, and improve legibility of the interface in places. I’ve spoken to Calum Benson of HIG fame and he has no particular objections to mandating use of the appropriate Unicode characters by the HIG.

In the meantime, I’ve been filing bugs against applications to convert them to using proper Unicode characters; this probably wasn’t the best way to go about things, but at least it is a move in the right direction (in my view anyway). Unfortunately, this has come at the cost of inconsistency in the desktop. Most of the changes have been applied after branching for gnome-2-28, however, so if we can work out some guidelines about use of Unicode characters early in the 2.30 cycle (i.e. now), consistency could be maintained in the desktop for the 2.30 release. We might even be able to brag about nice typography for (dare I say it?) GNOME 3.0!

So, should we be expending effort on dealing with fonts which don’t support various Unicode characters, extending Pango to support the appropriate decompositions? Are there any problems with embedded C compilers and Unicode string literals? If we decide to go with a uniform usage of certain Unicode characters, what guidelines shall we go with, and how can we educate translators in how to type them?

Sources:

Gran Canaria!

Here I am, somehow successfully arrived in Gran Canaria, despite Iberia's best efforts. My first plane was delayed not quite long enough to give me hope that I'd catch my connection, but just too long for me to do so comfortably. Thankfully, the second plane was also delayed, so my running down the entire length of Madrid's Terminal 4 was somewhat unnecessary.

A GNOME Foundation sponsorship badge.

A GNOME Foundation sponsorship badge.

How am I here? I'm only here because of the nice people at the GNOME Foundation, who decided to sponsor me. Thank you, nice people!