Tag Archives: Collabora

Introduction to ICE and libnice

As part of the series of tea time talks we do within Collabora, I recently got to refresh my knowledge of STUN, TURN and ICE (the protocols for NAT traversal) and give an introductory talk on how they all fit together within the context of libnice.

Since the talk might be useful (and perhaps even interesting) to a wider audience, I’ve made it available: slides, handout and source (git). It’s under CC-BY-SA 4.0. Please leave comments if anything is unclear, incorrect, or could do with more in-depth coverage!

Recent improvements in libnice

For the past several months, Olivier Crête and I have been working on a project using libnice at Collabora, which is now coming to a close. Through the project we’ve managed to add a number of large, new features to libnice, and implement hundreds (no exaggeration) of cleanups and bug fixes. All of this work was done upstream, and is available in libnice 0.1.8, released recently! GLib has also gained a number of networking fixes, API additions and documentation improvements.

tl;dr: libnice now has a GIOStream implementation, support for scatter–gather send and receive, and more mature pseudo-TCP support — so its API should be much nicer to integrate; GLib has gained a number of fixes.

Firstly, what is libnice? It’s a GLib implementation of ICE, the standard protocol for NAT traversal. Briefly, NAT traversal is needed when two hosts want to communicate peer-to-peer in a network where there is at least one NAT translator between them, meaning that at least one of the hosts cannot directly address the other until a mapping is created in the NAT translator. This is a very common situation (due to the shortage of IPv4 addresses, and the consequence that most home routers act as NAT translators) and affects virtually all peer-to-peer communications. It’s well covered in the literature, and the rest of this post will assume a basic understanding of NAT and ICE, a topic about which I recently gave a talk.

Conceptually, libnice exists just to create a reliable (TCP-like) or unreliable (UDP-like) socket which connects your host with a remote one in a manner that traverses any intervening NATs. At its core, it is effectively an implementation of send(), recv(), and some ancillary functions to negotiate the ICE stream at startup time.

The biggest change is the addition of nice_agent_get_io_stream(), and the GIOStream subclass it returns. This allows reliable ICE streams to be used via GIOStream, with all the API sugar which comes with GIO streams — for example, g_output_stream_splice(). Unreliable (UDP-like) ICE streams can’t be used this way because they’re not technically streams.

Highly related, the original receive API has been augmented with scatter–gather support in the form of a recvmmsg()-like API: nice_agent_recv_messages(). Along with appropriate improvements to libnice’s underlying socket implementations (the most obscure of which are still to be plumbed in), this allows performance improvements by batching messages, reducing the number of system calls needed for communication. Furthermore (perhaps more importantly) it reduces memory copies when assembling and parsing packets, by allowing the packets to be split across multiple non-contiguous buffers. This is a well-studied and long-known performance technique in networking, and it’s nice that libnice now supports it.

So, if you have an ICE connection (stream 1 on agent, with 2 components) exchanging packets with 20B headers and variable-length payloads, instead of:

nice_agent_attach_recv (agent, 1, 1, main_context, recv_cb, NULL);
nice_agent_attach_recv (agent, 1, 2, main_context, recv_cb, NULL);

…

static void
recv_cb (NiceAgent *agent, guint stream_id, guint component_id,
         guint len, const gchar *buf, gpointer user_data)
{
    if (stream_id != 1 ||
        (component_id != 1 && component_id != 2)) {
        g_assert_not_reached ();
    }

    if (parse_header (buf)) {
        if (component_id == 1)
            parse_component1_data (buf + 20, len - 20);
        else
            parse_component2_data (buf + 20, len - 20);
    }
}

…

static void
send_to_component (guint component_id,
                   const gchar *data_buf, gsize data_len)
{
    gsize len = 20 + data_len;
    guint8 *buf = malloc (len);

    build_header (buf);
    memcpy (buf + 20, data, data_len);

    if (nice_agent_send (agent, 1, component_id,
                         len, buf) != len) {
        /* Handle the error */
    }
}

you can now do:

/* Only set up 1 NiceInputMessage as an illustration. */

static guint8 buf1_1[20];  /* header */
static guint8 buf1_2[1024];  /* payload size limit */
static GInputVector buffers1[2] = {
    { &buf1_1, sizeof (buf1_1) },  /* header */
    { &buf1_2, sizeof (buf1_2) },  /* payload */
};
static NiceInputMessage messages[1] = {
    buffers1, G_N_ELEMENTS (buffers1),
    NULL, 0
};
GError *error = NULL;

n_messages = nice_agent_recv_messages (agent, 1, 1, &messages,
                                       G_N_ELEMENTS (messages),
                                       NULL, &error);
if (n_messages == 0 || error != NULL) {
    /* Handle the EOS or error. */
    if (error != NULL)
        g_error ("Error: %s", error->message);
    return;
}

/* Component 2 can be handled similarly and code paths combined. */
for (i = 0; i < n_messages; i++) {
    NiceInputMessage *message = &messages[i];

    if (parse_header (message->buffers[0].buffer)) {
        parse_component1_data (message->buffers[1].buffer,
                               message->buffers[1].size);
    }
}

…

static void
send_to_component (guint component_id, const gchar *data_buf,
                   gsize data_len)
{
    GError *error = NULL;
    guint8 header_buf[20];
    GOutputVector vec[2] = {
        { header_buf, sizeof (header_buf) },
        { data_buf, data_len },
    };
    NiceOutputMessage message = { vec, G_N_ELEMENTS (vec) };

    build_header (header_buf);

    if (nice_agent_send_messages_nonblocking (agent, 1, component_id,
                                              &message, 1, NULL,
                                              &error) != 1) {
        /* Handle the error */
        g_error ("Error: %s", error->message);
    }
}

libnice has also gained non-blocking variants of its I/O functions. Previously, one had to explicitly attach a libnice stream to a GMainContext to start receiving packets. Packets would be delivered individually via a callback function (set with nice_agent_attach_recv()), which was inefficient and made for awkward control flow. Now, the non-blocking I/O functions can be used with a custom GSource from g_pollable_input_stream_create_source() to allow for more flexible reception of packets using the more standard GLib pattern of attaching a GSource to the GMainContext and in its callback, calling g_pollable_input_stream_read_nonblocking() until all pending packets have been read. libnice’s internal timers (used for retransmit timeouts, etc.) are automatically added to the GMainContext passed into nice_agent_new() at construction time, which you must run all the time as before.

GIOStream *stream = nice_agent_get_io_stream (agent, 1, 1);
GInputStream *istream;
GPollableInputStream *pollable_istream;

istream = g_io_stream_get_input_stream (stream);
pollable_istream = G_POLLABLE_INPUT_STREAM ();

source = g_pollable_input_stream_create_source (pollable_istream, NULL);
g_source_set_callback (source, readable_cb, NULL, pollable_istream);
g_source_attach (main_context, source);

static gboolean
readable_cb (gpointer user_data)
{
    GPollableInputStream *pollable_istream = user_data;
    GError *error = NULL;
    guint8 buf[1024];  /* whatever the maximum packet size is */

    /* Read packets until the queue is empty. */
    while ( (len = g_pollable_input_stream_read_nonblocking (pollable_istream,
                                                             buf, sizeof (buf),
                                                             NULL,
                                                             &error) ) > 0) {
        /* Do something with the received packet. */
    }

    if (error != NULL) {
        /* Handle the error. */
    }
}

libnice also gained much-improved support for restarting individual streams using ICE restarts with the addition of nice_agent_restart_stream(), switching TURN relays with nice_agent_forget_relays(), plus a number of bug fixes.

Finally, FIN/ACK support has been added to libnice’s pseudo-TCP implementation. The code was originally based on Google’s libjingle pseudo-TCP, establishing a reliable connection over UDP by encapsulating TCP-like packets within UDP. This implemented the basics of TCP, but left things like the closing FIN/ACK handshake to higher-level protocols. Fine for Google, but not for our use case, so we added support for that. Furthermore, we needed to layer TLS over a pseudo-TCP connection using GTlsConnection, which required implementing half-duplex close support and fixing a few nasty leaks in GTlsConnection.

Thanks to the libnice community for testing out the changes, and thanks to the GLib developers for patiently reviewing the stream of tiny documentation fixes and several larger GLib patches! All of the libnice API changes are shown on the handy upstream-tracker.org tool.

Clang plugin for GLib and GNOME

This past week, I’ve been working on a Clang plugin for GLib and GNOME, with the aim of improving static analysis of GLib-based C projects by integrating GIR metadata and assumptions about common GLib and GObject coding practices. The idea is that if running Clang for static analysis on a project, the plugin will suppress common GLib-based false positives and emit warnings for common problems which Clang wouldn’t previously have known about. Using it should be as simple as enabling scan-build, with the appropriate plugin, in your JHBuild configuration file.

I’m really excited about this project, which I’m working on as R&D at Collabora. It’s still early days, and the project has a huge scope, but so far I have a Clang plugin with two major features. It can:

  • Load GIR metadata1 and use it to add nonnull attributes based on the presence (or absence) of (allow-none) annotations.
  • Detect g_return_if_fail(), assert(), g_return_val_if_fail() (etc.) preconditions in function bodies and add nonnull attributes if the assertions are non-NULL checks (or are GObject type checks which also assert non-nullability).

Additionally, the plugin can use GIR metadata to add other attributes:

  • Add deprecated attributes based on Deprecated: gtk-doc tags.
  • Add warn_unused_result attributes to functions which have a (transfer container) or (transfer full) return type.
  • Add malloc attributes to functions which are marked as (constructor).
  • Constify the return types of (transfer none) functions which return a pointer type.

By modifying Clang’s abstract syntax tree (AST) for a file while it’s being scanned, the plugin can take advantage of the existing static analysis for NULL propagation to function calls with nonnull attributes, plus its analysis for variable constness, use of deprecated functions, etc.

So far, all the code is in two ‘annotaters’, which modify the AST but don’t emit any new warnings of their own. The next step is to implement some ‘checkers’, which examine (but don’t modify) the AST and emit warnings for common problems. Some of the problems checked for will be similar to those handled above (e.g. warning about a missing deprecated attribute if a function has a Deprecated: gtk-doc tag), but others can be completely new, such as checking that if a function has a GError parameter then it also has a g_return_if_fail(error == NULL || *error == NULL) precondition (or something along those lines).

I’ve got several big ideas for features to add once this initial work is complete, but the next thing to work on is making the plugin effortless to use (or, realistically, nobody will use it). Currently, the plugin is injected by a wrapper script around Clang, e.g.:

cd /path/to/glib/source
GNOME_CLANG_GIRS="GLib-2.0 Gio-2.0" gnome-clang -cc1 -analyze … /files/to/analyse

This injects the -load $PREFIX/lib64/clang/3.5/libclang-gnome.so -add-plugin gnome arguments which load and enable the plugin. By specifying GIR namespaces and versions in GNOME_CLANG_GIRS, those GIRs will be loaded and their metadata used by the plugin.

This can be configured in your ~/.jhbuildrc with:

static_analyzer = True
static_analyzer_template = 'scan-build --use-analyzer=$JHBUILD_PREFIX/bin/gnome-clang -v -o %(outputdir)s/%(module)s'
os.environ['GNOME_CLANG_GIRS'] = 'GLib-2.0 Gio-2.0 GnomeDesktop-3.0'

Please note that it’s alpha-quality code at the moment, and may introduce program bugs if used during compilation, so should only be used with Clang’s -analyze option. Furthermore, it still produces some false positives, so please be cautious about submitting bug reports to upstream projects ‘fixing’ problems detected by static analysis.

The code so far is here: http://cgit.collabora.com/git/gnome-clang.git/ and there’s a primitive website here: http://people.collabora.com/~pwith/gnome-clang/. If you’ve got any ideas, bug reports, patches, or just want to criticise my appalling C++, please e-mail me: philip {at} tecnocode.co(.)uk.


  1. Actually, it loads the metadata from GIR typelib files, rather than using the source code annotations themselves, so there is a degree of data loss incurred due to the limitations of the typelib binary format. 

libfolks: BlueZ backend for phone address book access

libfolks now has a backend for accessing address book data from Bluetooth-enabled phones, making use of BlueZ 5. This has been a long time in development, with various developers from Collabora contributing to it (thanks to Matthieu Bouron, Gustavo Padovan, Arun Raghavan and Jeremy Whiting!).

It works using the Bluetooth Phone Book Access Profile (PBAP), which allows a computer to download a phone’s address book as an array of vCards. libfolks will automatically do this for any phone which is paired with the computer (and which isn’t blocked), establishing a connection with any (paired, non-blocked) phone in range.

We’ve only been able to test this with a limited number of phones, so further real-world testing and feedback would be greatly appreciated. To do so, grab the latest version of libfolks from git (https://git.gnome.org/browse/folks/log/), build it and run folks-inspect (making sure that the ‘bluez’ backend is not disabled in ~/.local/share/folks/backends.ini, if you’ve created and customised that file).

As is the design of libfolks, this is not intended as a solution for synchronising contacts between your phone and your computer, but it does provide a handy way of accessing contacts on your phone from GNOME Shell or Contacts, or from other consumers of libfolks data.

Note that the backend only supports BlueZ 5 and explicitly doesn’t support BlueZ 4.

Writing a GNOME thumbnailer

How does GNOME generate thumbnails for files? It uses a collection of programs called thumbnailers, each one generating thumbnails for a specific set of content-types of files. For example, totem-video-thumbnailer generates thumbnails for video files using GStreamer; evince-thumbnailer generates thumbnails for PDFs and other document files.

To generate a thumbnail, an appropriate thumbnailer program is selected then executed, passing it the path of the file to thumbnail, plus a path to write the thumbnail image to. If thumbnailing succeeds, the thumbnailer should have written the image to disk before terminating; but if thumbnailing fails, no image should be written, and the thumbnailer should return a non-zero exit status.

Thumbnailers are chosen by examining a series of .thumbnailer files in $PREFIX/share/thumbnailers. Each is in a simple key-file format:

[Thumbnailer Entry]
TryExec=evince-thumbnailer
Exec=evince-thumbnailer -s %s %u %o
MimeType=application/pdf;application/x-bzpdf;application/x-gzpdf;

The .thumbnailer format supports three keys:

  • TryExec: Optional. The name of the the thumbnailer program in the current $PATH. This allows the calling code to quickly check whether the thumbnailer exists before trying to execute it.
  • Exec: Required. The command to execute the thumbnailer. It supports a few different parameters which are replaced before calling the thumbnailer: %u is the URI of the file being thumbnailed; %i is its filename; %o is the filename of the image file to be written to; %s is the maximum desired size of the thumbnail image (in pixels); and %% is a literal percent character.
  • MimeType: Required. A semicolon-separated list of MIME types which the thumbnailer supports generating thumbnails for.

So in the evince-thumbnailer file above, the command passes the requested thumbnail size, then the input file’s URI, then the path for the output image file to evince-thumbnailer.

The code to examine and call a thumbnailer is in gnome-common’s GnomeDesktopThumbnailFactory class, which handles looking up the right thumbnailer script, building and executing the command for it, and loading the resulting thumbnail image into a GdkPixbuf.

Thumbnail caching is supported by GnomeDesktopThumbnailFactory. When calling a thumbnailer, the path passed for the output image file is in ~/.cache/thumbnails/$SIZE/. The cached image file is given a (probably) unique filename, generated by hashing the original file’s URI, so the thumbnail can be looked up in future. gnome-desktop supports two sizes of thumbnails: ‘normal’ and ‘large’. Normal thumbnails are up to 128×128 pixels, whereas large thumbnails are up to 256×256 pixels. Thumbnails which are larger than this are scaled down before being cached, and non-square thumbnails are scaled so their largest dimension is at most 128 or 256 pixels.

gnome-desktop also handles failed thumbnails. If a thumbnailer can’t generate a thumbnail for a file (e.g. because the file is corrupt or because the right video codecs aren’t available), it returns a non-zero exit status. gnome-desktop then writes an entry to ~/.cache/thumbnails/fail/gnome-thumbnail-factory/ which is named after the hash of the input file URI (just like a successful cached thumbnail). For future queries for thumbnails for that file, GnomeDesktopThumbnailFactory can immediately return an error after looking up the fail entry.

If a file changes content, GnomeDesktopThumbnailFactory knows to generate a new thumbnail because each cached image has associated metadata (stored as PNG tEXt keys) storing the full URI of the thumbnailed file (to check for hash collisions) and its last modification time at the point of thumbnailing. If the stored modification time doesn’t match the file’s current one, a new thumbnail is generated.

I don’t care. Show me the code.

As an example of this, I present gnome-directory-thumbnailer, which generates thumbnails for directories (which is possible with a few small fixes to gnome-desktop). I wrote this as part of a project for my new employer, Collabora. It generates a thumbnail for a directory by heuristically choosing its ‘most interesting’ file (or subdirectory) and taking the thumbnail from that. It works fairly well for generic directories, but there’s great scope for improving the heuristics to cater for directories which match different templates (e.g. a directory of audio files for an album; a directory containing an automake project; a mounted DVD directory; etc.).

Nautilus showing some album directories.

Nautilus showing some album directories.

The code is in gnome-directory-thumbnailer on git.gnome.org and it’s in the process of getting a gnome-directory-thumbnailer product on bugzilla.gnome.org, so please file bugs there. Note that you’ll need the latest release (3.10.1) of gnome-desktop for it to work correctly in Nautilus.

If you want to write your own thumbnailer, a good place to start is with the gnome-thumbnailer-skeleton by Bastien Nocera. It provides the generic wrapper code for command line parsing and output handling which would otherwise get rewritten for each new thumbnailer.

There are actually other ways of generating thumbnails, such as the Tumbler thumbnailing system which is accessed via D-Bus and implements the proposed thumbnailer specification which is used by Rygel. As far as I’m aware, Nautilus doesn’t use D-Bus thumbnailers.

End of the hackfest

The IM, Contacts & Social hackfest is over and people are variously heading back to their respective corners of the world. A lot of good discussion was had, and I think everyone's got a clearer vision of where we need to be with IMs and contacts and what needs to be done to get there.

Travis, Raúl and I clarified a lot of the changes needed to libfolks for this shiny new future, and while I didn't get as much hacking done as I would've liked, there's plenty of time for that later.

Thanks to Collabora for providing space for the hackfest and sponsoring plenty of food, the GNOME Foundation for enabling the hackfest to happen, and Nyan cat for entertaining Bastien.

IM, Contacts & Social hackfest

I'll be heading along to the IM, Contacts & Social hackfest next week at Collabora's offices. There, a plan for world domination by libfolks will be forged, along with plotting around GNOME's new SSO overlord system and work on the much-awaited GNOME Contacts.

Should be fun!

Relatedly, I've just released libgdata 0.9.0, which has sprouted support for OAuth 1.0 — so hopefully some GNOME Online Accounts goodness will soon make it into Evolution's Google Contacts and Calendar backends.

End of the summer

It's come to the end of the summer, and the end of my internship with Collabora. They've enabled me to spend three months helping work on adding meta-contacts support to Empathy in the form of libfolks; something which I've thoroughly enjoyed. Thank you to all my friends and colleagues at Collabora for letting me do this!

Term starts at university for me again in the next few weeks, so I expect to disappear back into the world of degree-enabling work, but I hope to be able to poke my head up from my little hole every so often, and possibly even submit a patch or two.

The last GUADEC post?

GUADEC's over (and has been so for a while) and was great. The talk videos are up, and still I haven't blogged about the conference. Naughty me.

The conference was great: it was good to meet up with friends (old and new) again, get a bit of hacking done and hear lots of discussion about the future of GNOME. My thanks to the GNOME Foundation and my summer employer, Collabora, for sponsoring my accommodation and travel. Thank you to the local team for organising a brilliant GUADEC in a nice city, and here's to a successful GCDS 2011 in Berlin!

In other news, I've finally got fed up with forgetting to add new files in my project to either the project-wide header file or the documentation index, so I wrote some `make check` rules which will check for my braindeadness:

This one goes in the Makefile.am in the directory which builds your public header (such as gtk/gtk.h), and assumes a list of all the headers you're going to install is in public_headers, and that your main header's path is in main_header.

check-local: check-headers
check-headers:
	@any_missing=0; for header_file in $(public_headers); do \
		if test "x\(header_file" != "x$(main_header)"; then \
			if ! grep "#include <\)header_file>" $(main_header) >/dev/null; then \
				echo "$(main_header) doesn't appear to include \"\(header_file\""; \
				any_missing=1; \
			fi; \
		fi; \
	done; exit "\)any_missing"

This one goes in the Makefile.am in your gtk-doc directory (e.g. docs/reference), and only assumes the existence of DOC_MAIN_SGML_FILE, which needs to be defined for gtk-doc anyway.

check-local: check-xml-includes
check-xml-includes:
	@any_missing=0; find $(srcdir) -name "*.xml" | while read x; do \
		xml_file="\({x#./}"; \
		if test "x\)xml_file" != "x$(DOC_MAIN_SGML_FILE)"; then \
			if ! grep "\"\(xml_file\"" $(DOC_MAIN_SGML_FILE) >/dev/null; then \
				echo "$(DOC_MAIN_SGML_FILE) doesn't appear to include \"\)xml_file\""; \
				any_missing=1; \
			fi; \
		fi; \
	done; exit "$$any_missing"

I've filed bgo#627920 about adding the second rule to gtk-doc itself. I'm not sure the first rule is general enough to be put anywhere common.

Introspectable libgdata

Now that full term at university has finished, I have a little free time (inbetween sleeping, doing all the holiday work, and that celebration thing that people seem to do) to spend on projects. Today, I got round to adding GObject introspection support to libgdata. I think some of the Makefile changes I made were a little hacky, but they seem to work. If anyone wants to use libgdata from an interpreted language which has GIR support, it should now be possible.

In university news, things have gone well this term, and I managed to (precariously) stay on top of the rolling mountain of work. I've now got two weeks' stay in Cambridge to help shepherd interviewees around for their application interviews before I go home and do Christmassy things.

It took me until about half way through the term to notice that I was actually walking past Collabora's UK office every morning (opposite King's College). It was quite a shock when I noticed. It was also a shock to see they weren't at the computer lab's recent careers fair. Perhaps missing a trick there?