Have you ever written a client for a D-Bus service, then had difficulty in testing it because you need to find a way to set up the state of the D-Bus service, run your test, then set up the service in a completely different manner, rinse and repeat…all while running a modified version of the service’s binary in parallel with your system installation of it, and without doing anything which might cause your personal data to be accidentally eaten?
Having done some hacking on Telepathy and EDS clients I can, unfortunately, say “yes” to all of the above. Since problems are problematic, I’ve been hacking on a tool called Bendy Bus, which will hopefully alleviate some of this pain.
Bendy Bus is a project I’ve been working on as my final year university project, but I intend for it to be most useful outside of (hopefully) getting me marks for my degree. The basic idea of Bendy Bus is that you fuel it up with a description of a nondeterministic finite state automaton which represents the D-Bus service you’re using, plus a D-Bus introspection XML file describing all the relevant D-Bus interfaces. Bendy Bus will use the FSM description to simulate the D-Bus service, and run as a wrapper around the client program you’re trying to test. It’ll set up a private dbus-daemon instance for your client program, and expose all the simulated D-Bus objects on this bus.
Bendy Bus will listen for D-Bus method calls and property changes made by your client program, and execute transitions within the FSM as coded in your FSM description file. These transitions may, for example, change the FSM’s state, change data stored in the FSM (technically making it a nondeterministic DFSM, but that’s immaterial), emit D-Bus signals, throw D-Bus errors, etc. Why do I say it’s a nondeterministic FSM? Because you may specify several transitions between the same pair of states which are triggered by the same (for example) D-Bus method call. Bendy Bus will randomly choose one of the transitions to take. For example, if your client program calls a frobnicate : string → string D-Bus method, you could code one transition which successfully replies to the method call with a string return value, and another transition which simulates a failure in the D-Bus service by throwing a D-Bus error instead.
It’s in this fashion that Bendy Bus is actually designed as a fuzzing tool. You can code up a full description of every possible state and transition in your D-Bus service, then set your client program running in the Bendy Bus wrapper, and it’ll randomly explore the service’s state space until a termination condition is met. For example, the client program could crash (in which case a bug’s been found!), a D-Bus activity timeout could be reached (if your client program hasn’t made any D-Bus calls for a few seconds, for example), or a global timeout could be reached. At this point, the test harness can restart your client program and start the whole thing all over again with a different random seed value, causing different execution paths to be explored, and more of your client’s code to be covered.
Of course, Bendy Bus is still young, so features are missing, there are plenty of bugs, and documentation is basically non-existent. A couple of the big features on the list are to implement support for unit testing (which would tone down the fuzz testing aspect of Bendy Bus, and allow deterministic unit tests to be written for D-Bus client programs), implement better error reporting in the machine description parser and better logging during simulation, write a language specification and GtkSourceView highlighter, and write documentation. Help on any of these (except the unit testing stuff, which I have to keep for myself for my university project) would be greatly appreciated.
More than anything, it would be great if people could play with Bendy Bus and see if it’s useful for them (and if not, what could be done to improve it). In the repository at the moment are a couple of example machine description files for Telepathy. They can be used to get a randomly-generated contact list to appear in Empathy, using the following command:
bendy-bus machines/telepathy-cm.machine machines/telepathy-cm.xml \
-E FOLKS_DEBUG=telepathy -E EMPATHY_DEBUG=all -E G_MESSAGES_DEBUG=all \