Have you ever written a client for a D-Bus service, then had difficulty in testing it because you need to find a way to set up the state of the D-Bus service, run your test, then set up the service in a completely different manner, rinse and repeat…all while running a modified version of the service’s binary in parallel with your system installation of it, and without doing anything which might cause your personal data to be accidentally eaten?
Having done some hacking on Telepathy and EDS clients I can, unfortunately, say “yes” to all of the above. Since problems are problematic, I’ve been hacking on a tool called Bendy Bus, which will hopefully alleviate some of this pain.
Bendy Bus is a project I’ve been working on as my final year university project, but I intend for it to be most useful outside of (hopefully) getting me marks for my degree. The basic idea of Bendy Bus is that you fuel it up with a description of a nondeterministic finite state automaton which represents the D-Bus service you’re using, plus a D-Bus introspection XML file describing all the relevant D-Bus interfaces. Bendy Bus will use the FSM description to simulate the D-Bus service, and run as a wrapper around the client program you’re trying to test. It’ll set up a private dbus-daemon instance for your client program, and expose all the simulated D-Bus objects on this bus.
Bendy Bus will listen for D-Bus method calls and property changes made by your client program, and execute transitions within the FSM as coded in your FSM description file. These transitions may, for example, change the FSM’s state, change data stored in the FSM (technically making it a nondeterministic DFSM, but that’s immaterial), emit D-Bus signals, throw D-Bus errors, etc. Why do I say it’s a nondeterministic FSM? Because you may specify several transitions between the same pair of states which are triggered by the same (for example) D-Bus method call. Bendy Bus will randomly choose one of the transitions to take. For example, if your client program calls a frobnicate : string → string D-Bus method, you could code one transition which successfully replies to the method call with a string return value, and another transition which simulates a failure in the D-Bus service by throwing a D-Bus error instead.
It’s in this fashion that Bendy Bus is actually designed as a fuzzing tool. You can code up a full description of every possible state and transition in your D-Bus service, then set your client program running in the Bendy Bus wrapper, and it’ll randomly explore the service’s state space until a termination condition is met. For example, the client program could crash (in which case a bug’s been found!), a D-Bus activity timeout could be reached (if your client program hasn’t made any D-Bus calls for a few seconds, for example), or a global timeout could be reached. At this point, the test harness can restart your client program and start the whole thing all over again with a different random seed value, causing different execution paths to be explored, and more of your client’s code to be covered.
Of course, Bendy Bus is still young, so features are missing, there are plenty of bugs, and documentation is basically non-existent. A couple of the big features on the list are to implement support for unit testing (which would tone down the fuzz testing aspect of Bendy Bus, and allow deterministic unit tests to be written for D-Bus client programs), implement better error reporting in the machine description parser and better logging during simulation, write a language specification and GtkSourceView highlighter, and write documentation. Help on any of these (except the unit testing stuff, which I have to keep for myself for my university project) would be greatly appreciated.
More than anything, it would be great if people could play with Bendy Bus and see if it’s useful for them (and if not, what could be done to improve it). In the repository at the moment are a couple of example machine description files for Telepathy. They can be used to get a randomly-generated contact list to appear in Empathy, using the following command:
bendy-bus machines/telepathy-cm.machine machines/telepathy-cm.xml \
--test-program-log-file=test-program.log \
--dbus-daemon-log-file=dbus-daemon.log \
--simulator-log-file=simulator.log \
-E FOLKS_DEBUG=telepathy -E EMPATHY_DEBUG=all -E G_MESSAGES_DEBUG=all \
-- empathy
That’s all from me at the moment. The Bendy Bus git repository is on gitorious, and all bug reports should be e-mailed to me: philip {at} tecnocode.co(.)uk.
Question: if you are interested in verifying reachability of a set of states, why not using a Petri Net instead of a simple NFA? They allow for a set of very fast techniques for determining if a state can be reached, for example via a modified version of the Karp-Miller algorithm.
That way you don't need to "generate fuzzyness" and hope to get there statistically. Or maybe I misunderstood your words... but if you go by random, the problem is you might never catch the bad configuration you're looking for, which is not that nice.
If you don't like Petri Nets, there are other models that might be of interest, in the field of WSTSs (Well-Structured Transition Systems).
Those look like interesting models, I’ll have to take a look into them, thanks.
However, the systems which are being modelled by Bendy Bus typically only have a couple of states, and reaching any of those states is statistically quite likely. For example, a simple model of a Telepathy connection manager would just have the states ‘disconnected’, ‘connecting’ and ‘connected’. In my experience so far, a lot of the code coverage comes as a result of fuzzing data in D-Bus method replies, and making random transitions inside a given state.
For example, the Telepathy machine I’ve been testing with will easily reach the ‘connected’ state, then randomly make transitions from ‘connected’ to ‘connected’, adding a contact to the contact list each time. Fuzzing the data on the contact (its alias, avatar, status, etc.) is what gets the code coverage — for Telepathy, at least.
It would be reassuring to be able to guarantee that all states in the simulation will be reached, but I suspect that most D-Bus services will have very few states.
(I omitted from the blog post that Bendy Bus has built-in support for marking any data structure, such as would be returned in a D-Bus method reply, as fuzzable. Whenever that data structure (string, integer, array of structs, a{sv}, etc.) is evaluated, it’ll be fuzzed. This is achieved by using the ‘?’ postfix operator on the data structure in the machine description file. The value of the data structure in the machine description will be used as a default value, which may get through fuzzing unmodified, may be emptied/deleted, or may be mutated. Fairly standard fuzzing stuff, really.)
I see. But I still suspect it would be more useful to have a guarantee of full coverage, were this tool to scale for more complex applications in the future. You could probably translate from one representation to the other automatically, and then look for coverability / reachability. Also because with complex data structures, the number of states in a PN would be likely to increase greatly, so specifying it by hand is not ideal.
Agreed. I guess it would also be good to statically check the reachability of every state in the machine before running it, and warn the user if some states were definitely not reachable. (I suspect that even for the restricted language I’ve allowed in machine descriptions, however, that reachability is undecidable.)