D-Bus has just gained some upstream guidelines on how to write APIs which best use its features and follow its conventions. Hopefully they draw together all the best practices from the various articles which have been published about writing D-Bus APIs in the past. If anybody has suggestions, criticism or feedback about them, or ideas for extra guidelines to include, please get in touch!
Before I start, an addendum to my last post about the DX hackfest: I wish to thank Codethink for sponsoring dinner one night of the event. I forgot to include that in my original post, sorry! Thanks again to all the companies who let employees attend: Endless, Codethink, Canonical and Red Hat.
tl;dr: Check out the new GNOME Programming Guidelines and file bugs in Bugzilla. Though it looks like a cronjob may be taking the docs offline occasionally. This will be fixed.
Now, to some of the results of the hackfest. In the last week or so, I’ve been working on expanding the GNOME programming guidelines, upstreaming various bits of documentation which Collabora have been writing for a customer who is using the GNOME stack in a large project. The guidelines were originally written in the early days of GNOME by Federico, Miguel and Morten; Federico updated them in 2013, and now they’ve been expanded again.
It looks like these guidelines can fill one of the gaps we currently have in documentation, where we need to recommend best practices and give tutorial-style examples, but gtk-doc–generated API manuals are not the right place. For example, the new guidelines include recommendations for making libraries parallel-installable (based off Havoc’s original article, with permission); or recommendations for choosing where to store data (in GSettings, a serialised GVariant store, or a full-on GOM/SQLite database?). The guidelines are intended to be useful to all developers, although inherently need to target newer developers more, so may simplify a few things.
I’ve still got some ideas for things to add. For example, I need to rework some of my blog posts about GMainContext into an article, since we should be documenting before blogging. Other ideas are very welcome, as is criticism, feedback and improvements: please file a bug against gnome-devel-docs. Thanks to the documentation team for their help and reviews!
Day 5, and the DX and docs hackfest in Collabora HQ, Cambridge has drawn to a close. It’s been great to have everyone here, and there have been a lot of in-depth discussions over the last few days about the details of app sandboxing, runtimes, Builder integration with various new services, the development of an IDE abstraction layer, approaches for making build systems accessible to Builder, lots of new things to statically analyse, and some fairly fundamental additions to GLib in the form of G_DECLARE_[FINAL|DERIVABLE]_TYPE and general-purpose reference counted memory areas. Whew! We even had a fleeting visit by Richard Hughes to discuss packaging issues for apps.
For all the details, see the blogs by Ryan, Cosimo, Ryan again, Alberto, Christian and Emmanuele.
I can’t do justice to the work of the docs team, who put in consistent, solid effort throughout the hackfest. See the blogs by Petr, Bastian, Kat and Jim for all the details. They even left me with a seemingly endless supply of Mallard balls to throw around the office!
Dave and I have spent a little while working on further deprecating gnome-common. More details to come once the migration guide is finished.
Thank you to Collabora for hosting, Codethink for sponsoring a dinner, and Endless, Codethink (again), Canonical and Red Hat for letting people attend; and thank you to the GNOME Foundation for sponsoring some of the attendees. It would not be a hackfest without the hackers!
It’s a sunny Sunday here in Cambridge, UK, and GNOMErs have been arriving from far and wide for the first day of the 2015 developer experience hackfest. This is a week-long event, co-hosted with the winter docs hackfest (which Kat has promised to blog about!) in the Collabora offices.
Today was a bit of a slow start, since people were still arriving throughout the day. Regardless, there have been various discussions, with Ryan, Emmanuele and Christian discussing performance improvements in GLib, Christian and Allan plotting various different approaches to new UI in Builder, Cosimo and Carlos silently plugging away at GTK+, and Emmanuele muttering something about GProperty now and then.
Tomorrow, I hope we can flesh out some of these initial discussions a bit more and get some roadmapping down for GLib development for the next year, amongst other things. I am certain that Builder will feature heavily in discussions too, and apps and sandboxing, now that Alex has arrived.
I’ve spent a little time finishing off and releasing Walbottle, a small library and set of utilities I’ve been working on to implement JSON Schema, which is the equivalent of XML Schema or RELAX-NG, but for JSON files. It allows you to validate JSON instances against a schema, to validate schemas themselves and, unusually, to automatically generate parser unit tests from a schema. That way, you can automatically test json-glib–based JsonReader/JsonParser code, just by passing the JSON schema to Walbottle’s json-schema-generate utility.
It’s still a young project, but should be complete enough to be useful in testing JSON code. Please let me know of any bugs or missing features!
Tomorrow, I plan to dive back in to static analysis of GObject code with Tartan…
tl;dr: For `make check` with Valgrind support, add AX_VALGRIND_CHECK to configure.ac and @VALGRIND_CHECK_RULES@ to tests/Makefile.am. Use memcheck, helgrind, drd and sgcheck.
Once you’ve written a unit test suite with near-100% coverage, you can validate software with a lot more confidence than before, since any effort you put into validation is working across the whole code base.1 It’s quite common to check for memory leaks with Valgrind’s memcheck tool — but Valgrind has several additional tools which are useful in production: helgrind, drd and sgcheck.2 All of these can be run over unit tests.
How can you run a test suite under these tools? The normal method is some kind of hard-to-remember rune involving libtool, and either something about TESTS_ENVIRONMENT or some kind of LOG_COMPILER (but why would I want to compile my test logs?). For that reason, I’ve written an autoconf macro which abstracts it all a bit: AX_VALGRIND_CHECK. It adds a check-valgrind target to your makefile which handles running the `make check` tests under all four Valgrind tools of interest. Various ancillary variables (such as VALGRIND_SUPPRESSIONS_FILES or VALGRIND_FLAGS) allow the Valgrind options to be changed. `make check-valgrind` will return an error exit code if any problems are found, with the idea that the target can be used for automated testing and continuous integration.
To use it, either add a dependency on autoconf-archive to your project, or copy the ax_valgrind_check.m4 macro in-tree (and add it to EXTRA_DIST), then follow the instructions at the top of the file for adding it to configure.ac and tests/Makefile.am.
With this macro, I’ve tried to bring together existing makefile snippets which exist in various projects to produce one macro which can be reused everywhere. Of course, I’ve probably missed something – some feature or specific use case which someone has – so if anyone has any suggestions for improvement, please let me know or submit a patch to the autoconf-archive! For example, at the moment the macro requires automake’s parallel test harness, and does not support older versions of automake.
Thanks to my employer, Collabora, for enabling me to work on (hopefully useful) stuff like this!
Normal caveats about the combinatorial complexity of dynamic testing apply. ↩
Although currently the threading tools may not work with GLib threads, due to its use of futexes rather than pthreads. ↩
tl;dr: Use g_set_object() to conveniently update owned object pointers in property setters (and elsewhere); see the bottom of the post for an example.
A little while ago, GLib gained a useful function called g_clear_object(), which clears an owned object pointer (a pointer which owns a reference to a GObject). GLib also just gained a function called g_set_object(), which works similarly but can either clear an object pointer or update it to point to a new object.
Why is this valuable? It saves a few lines of code each time an object pointer is updated, which isn’t much in itself. However, one thing it gets right is the order of reference counting operations, which is a common mistake in property setters. Instead of:
/* This is bad code. */
if (object_ptr != NULL)
g_object_unref (object_ptr);
if (new_object != NULL)
g_object_ref (new_object);
object_ptr = new_object;
you should always do:
/* This is better code. */
if (new_object != NULL)
g_object_ref (new_object);
if (object_ptr != NULL)
g_object_unref (object_ptr);
object_ptr = new_object;
because otherwise, if (new_object == object_ptr) (or if the objects have some other ownership relationship) and the object only has one reference left, object_ptr will end up pointing to a finalised GObject (and g_object_ref() will be called on a finalised GObject too, which it really won’t like).
So how does g_set_object() help? We can now do:
g_set_object (&object_ptr, new_object);
which takes care of all the reference counting, and allows new_object to be NULL. &object_ptr must not be NULL. If you’re worried about performance, never fear. g_set_object() is a static inline function, so shouldn’t adversely affect your code size.
Even better, the return value of g_set_pointer() indicates whether the value changed, so we can conveniently base GObject::notify emissions off it:
/* This is how all GObject property setters should look in future. */
if (g_set_object (&priv->object_ptr, new_object))
g_object_notify (self, "object-ptr");
Hopefully this will make property setters (and other object updates) simpler in new code.
tl;dr: Use AX_PKG_CHECK_MODULES to split public/private dependencies; use AC_CONFIG_FILES to magically include the API version in the .pc file name.
A few tips for creating a pkg-config file which you will never need to think about maintaining again — because one of the most common problems with pkg-config files is that their dependency lists are years out of date compared to the dependencies checked for in configure.ac. See lower down for some example automake snippets.
Given all those suggestions, here’s a template libmy-project/my-project.pc.in file (updated to incorporate suggestions by Dan Nicholson):
prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@
includedir=@includedir@
my_project_utility=my-project-utility-binary-name
my_project_db_dir=@sysconfdir@/my-project/db
Name: @PACKAGE_NAME@
Description: Some brief but informative description
Version: @PACKAGE_VERSION@
Libs: -L${libdir} -lmy-project-@API_VERSION@
Cflags: -I${includedir}/my-project-@API_VERSION@
Requires: @AX_PACKAGE_REQUIRES@
Requires.private: @AX_PACKAGE_REQUIRES_PRIVATE@
And here’s a a few snippets from a template configure.ac:
# Release version
m4_define([package_version_major],[1])
m4_define([package_version_minor],[2])
m4_define([package_version_micro],[3])
# API version
m4_define([api_version],[1])
AC_INIT([my-project],
[package_version_major.package_version_minor.package_version_micro],
…)
# Dependencies
PKG_PROG_PKG_CONFIG
PKG_INSTALLDIR
glib_reqs=2.40
gio_reqs=2.42
gthread_reqs=2.40
nice_reqs=0.1.6
# The first list on each line is public; the second is private.
# The AX_PKG_CHECK_MODULES macro substitutes AX_PACKAGE_REQUIRES and
# AX_PACKAGE_REQUIRES_PRIVATE.
AX_PKG_CHECK_MODULES([GLIB],
[glib-2.0 >= $glib_reqs gio-2.0 >= $gio_reqs],
[gthread-2.0 >= $gthread_reqs])
AX_PKG_CHECK_MODULES([NICE],
[nice >= $nice_reqs],
[])
AC_SUBST([PACKAGE_VERSION_MAJOR],package_version_major)
AC_SUBST([PACKAGE_VERSION_MINOR],package_version_minor)
AC_SUBST([PACKAGE_VERSION_MICRO],package_version_micro)
AC_SUBST([API_VERSION],api_version)
# Output files
# Rename the template .pc file to include the API version on configure
AC_CONFIG_FILES([
libmy-project/my-project-$API_VERSION.pc:libmy-project/my-project.pc.in
…
],[],
[API_VERSION='$API_VERSION'])
AC_OUTPUT
And finally, the top-level Makefile.am:
# Install the pkg-config file; the directory is set using # PKG_INSTALLDIR in configure.ac. pkgconfig_DATA = libmy-project/my-project-$(API_VERSION).pc
Once that’s all built, you’ll end up with an installed my-project-1.pc file containing the following (assuming a prefix of /usr; note that by default autoconf substitutes in references to variables rather than the values themselves, so pkg-config can continue to be used with --define-variable to override the prefix):
prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
my_project_utility=my-project-utility-binary-name
my_project_db_dir=${prefix}/etc/my-project/db
Name: my-project
Description: Some brief but informative description
Version: 1.2.3
Libs: -L${libdir} -lmy-project-1
Cflags: -I${includedir}/my-project-1
Requires: glib-2.0 >= 2.40 gio-2.0 >= 2.42 nice >= 0.1.6
Requires.private: gthread-2.0 >= 2.40
All code samples in this post are released into the public domain.
Assuming this is the number which will change if backwards-incompatible API/ABI changes are made. ↩
tl;dr: Don’t use AM_INCLUDES in Makefiles. It does nothing. INCLUDES and AM_CPPFLAGS should be avoided in favour of my_program_name_CPPFLAGS.
Once upon a time, there was an automake variable called INCLUDES. It was used to pass flags to the preprocessor, such as -I flags to specify include paths. All was good.
Then the gentle leaders of automake decided that INCLUDES was not a very generic name, so they deprecated it in favour of the *_CPPFLAGS variables, like the target-specific my_program_name_CPPFLAGS. For those who wanted to specify preprocessor flags for all targets in their Makefile, the AM_CPPFLAGS variable was provided. Due to their kind and wise oversight, the transition between these variables (in the time of the coming of automake 1.12) was peaceful, and tranquillity continued to reign in the pastures of automakery.
And yet, some lands were troubled. Some of the citizens of automake had developed a confusion. They had taken the INCLUDES incantations of old, and mixed them with the AM_ prefix of new to create an AM_INCLUDES monster. ‘This is surely better’, they reasoned — ‘we have global preprocessor flags, but allow it to be overridden by the user due to our use of the AM_ prefix’. These citizens went ahead and used their AM_INCLUDES variable, but were confused when it failed to affect their automagic. They thought the runes were not in their favour, so copied its value into AM_CFLAGS as well, just in case.
These citizens are wrong. AM_INCLUDES does not exist. Do not use it. automake does not recognise it. You are wasting your time. It has no effect.
Use per-target CPPFLAGS (e.g. my_program_name_CPPFLAGS) to specify preprocessor flags. Use per-target CFLAGS to specify C-compiler-only flags. Do not use AM_CPPFLAGS unless you’re sure all flags in it are needed for all targets in the Makefile. Do not use INCLUDES unless you want a big deprecation warning from automake((warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS'))) (it is entirely equivalent to AM_CPPFLAGS). Do not use AM_INCLUDES at all.
For the curious, passing -Wall to automake will enable a bunch of warnings which can improve your Makefiles. You can do that using the AUTOMAKE_OPTIONS environment variable, or the AM_INIT_AUTOMAKE macro in configure.ac.
And the citizens of automake lived happily ever after.
As part of the series of tea time talks we do within Collabora, I recently got to refresh my knowledge of STUN, TURN and ICE (the protocols for NAT traversal) and give an introductory talk on how they all fit together within the context of libnice.
Since the talk might be useful (and perhaps even interesting) to a wider audience, I’ve made it available: slides, handout and source (git). It’s under CC-BY-SA 4.0. Please leave comments if anything is unclear, incorrect, or could do with more in-depth coverage!
For the past several months, Olivier Crête and I have been working on a project using libnice at Collabora, which is now coming to a close. Through the project we’ve managed to add a number of large, new features to libnice, and implement hundreds (no exaggeration) of cleanups and bug fixes. All of this work was done upstream, and is available in libnice 0.1.8, released recently! GLib has also gained a number of networking fixes, API additions and documentation improvements.
tl;dr: libnice now has a GIOStream implementation, support for scatter–gather send and receive, and more mature pseudo-TCP support — so its API should be much nicer to integrate; GLib has gained a number of fixes.
Firstly, what is libnice? It’s a GLib implementation of ICE, the standard protocol for NAT traversal. Briefly, NAT traversal is needed when two hosts want to communicate peer-to-peer in a network where there is at least one NAT translator between them, meaning that at least one of the hosts cannot directly address the other until a mapping is created in the NAT translator. This is a very common situation (due to the shortage of IPv4 addresses, and the consequence that most home routers act as NAT translators) and affects virtually all peer-to-peer communications. It’s well covered in the literature, and the rest of this post will assume a basic understanding of NAT and ICE, a topic about which I recently gave a talk.
Conceptually, libnice exists just to create a reliable (TCP-like) or unreliable (UDP-like) socket which connects your host with a remote one in a manner that traverses any intervening NATs. At its core, it is effectively an implementation of send(), recv(), and some ancillary functions to negotiate the ICE stream at startup time.
The biggest change is the addition of nice_agent_get_io_stream(), and the GIOStream subclass it returns. This allows reliable ICE streams to be used via GIOStream, with all the API sugar which comes with GIO streams — for example, g_output_stream_splice(). Unreliable (UDP-like) ICE streams can’t be used this way because they’re not technically streams.
Highly related, the original receive API has been augmented with scatter–gather support in the form of a recvmmsg()-like API: nice_agent_recv_messages(). Along with appropriate improvements to libnice’s underlying socket implementations (the most obscure of which are still to be plumbed in), this allows performance improvements by batching messages, reducing the number of system calls needed for communication. Furthermore (perhaps more importantly) it reduces memory copies when assembling and parsing packets, by allowing the packets to be split across multiple non-contiguous buffers. This is a well-studied and long-known performance technique in networking, and it’s nice that libnice now supports it.
So, if you have an ICE connection (stream 1 on agent, with 2 components) exchanging packets with 20B headers and variable-length payloads, instead of:
nice_agent_attach_recv (agent, 1, 1, main_context, recv_cb, NULL);
nice_agent_attach_recv (agent, 1, 2, main_context, recv_cb, NULL);
…
static void
recv_cb (NiceAgent *agent, guint stream_id, guint component_id,
guint len, const gchar *buf, gpointer user_data)
{
if (stream_id != 1 ||
(component_id != 1 && component_id != 2)) {
g_assert_not_reached ();
}
if (parse_header (buf)) {
if (component_id == 1)
parse_component1_data (buf + 20, len - 20);
else
parse_component2_data (buf + 20, len - 20);
}
}
…
static void
send_to_component (guint component_id,
const gchar *data_buf, gsize data_len)
{
gsize len = 20 + data_len;
guint8 *buf = malloc (len);
build_header (buf);
memcpy (buf + 20, data, data_len);
if (nice_agent_send (agent, 1, component_id,
len, buf) != len) {
/* Handle the error */
}
}
you can now do:
/* Only set up 1 NiceInputMessage as an illustration. */
static guint8 buf1_1[20]; /* header */
static guint8 buf1_2[1024]; /* payload size limit */
static GInputVector buffers1[2] = {
{ &buf1_1, sizeof (buf1_1) }, /* header */
{ &buf1_2, sizeof (buf1_2) }, /* payload */
};
static NiceInputMessage messages[1] = {
buffers1, G_N_ELEMENTS (buffers1),
NULL, 0
};
GError *error = NULL;
n_messages = nice_agent_recv_messages (agent, 1, 1, &messages,
G_N_ELEMENTS (messages),
NULL, &error);
if (n_messages == 0 || error != NULL) {
/* Handle the EOS or error. */
if (error != NULL)
g_error ("Error: %s", error->message);
return;
}
/* Component 2 can be handled similarly and code paths combined. */
for (i = 0; i < n_messages; i++) {
NiceInputMessage *message = &messages[i];
if (parse_header (message->buffers[0].buffer)) {
parse_component1_data (message->buffers[1].buffer,
message->buffers[1].size);
}
}
…
static void
send_to_component (guint component_id, const gchar *data_buf,
gsize data_len)
{
GError *error = NULL;
guint8 header_buf[20];
GOutputVector vec[2] = {
{ header_buf, sizeof (header_buf) },
{ data_buf, data_len },
};
NiceOutputMessage message = { vec, G_N_ELEMENTS (vec) };
build_header (header_buf);
if (nice_agent_send_messages_nonblocking (agent, 1, component_id,
&message, 1, NULL,
&error) != 1) {
/* Handle the error */
g_error ("Error: %s", error->message);
}
}
libnice has also gained non-blocking variants of its I/O functions. Previously, one had to explicitly attach a libnice stream to a GMainContext to start receiving packets. Packets would be delivered individually via a callback function (set with nice_agent_attach_recv()), which was inefficient and made for awkward control flow. Now, the non-blocking I/O functions can be used with a custom GSource from g_pollable_input_stream_create_source() to allow for more flexible reception of packets using the more standard GLib pattern of attaching a GSource to the GMainContext and in its callback, calling g_pollable_input_stream_read_nonblocking() until all pending packets have been read. libnice’s internal timers (used for retransmit timeouts, etc.) are automatically added to the GMainContext passed into nice_agent_new() at construction time, which you must run all the time as before.
GIOStream *stream = nice_agent_get_io_stream (agent, 1, 1);
GInputStream *istream;
GPollableInputStream *pollable_istream;
istream = g_io_stream_get_input_stream (stream);
pollable_istream = G_POLLABLE_INPUT_STREAM ();
source = g_pollable_input_stream_create_source (pollable_istream, NULL);
g_source_set_callback (source, readable_cb, NULL, pollable_istream);
g_source_attach (main_context, source);
static gboolean
readable_cb (gpointer user_data)
{
GPollableInputStream *pollable_istream = user_data;
GError *error = NULL;
guint8 buf[1024]; /* whatever the maximum packet size is */
/* Read packets until the queue is empty. */
while ( (len = g_pollable_input_stream_read_nonblocking (pollable_istream,
buf, sizeof (buf),
NULL,
&error) ) > 0) {
/* Do something with the received packet. */
}
if (error != NULL) {
/* Handle the error. */
}
}
libnice also gained much-improved support for restarting individual streams using ICE restarts with the addition of nice_agent_restart_stream(), switching TURN relays with nice_agent_forget_relays(), plus a number of bug fixes.
Finally, FIN/ACK support has been added to libnice’s pseudo-TCP implementation. The code was originally based on Google’s libjingle pseudo-TCP, establishing a reliable connection over UDP by encapsulating TCP-like packets within UDP. This implemented the basics of TCP, but left things like the closing FIN/ACK handshake to higher-level protocols. Fine for Google, but not for our use case, so we added support for that. Furthermore, we needed to layer TLS over a pseudo-TCP connection using GTlsConnection, which required implementing half-duplex close support and fixing a few nasty leaks in GTlsConnection.
Thanks to the libnice community for testing out the changes, and thanks to the GLib developers for patiently reviewing the stream of tiny documentation fixes and several larger GLib patches! All of the libnice API changes are shown on the handy upstream-tracker.org tool.